Patent Description:
Document <CIT> describes a device for assembling the upper and the sole of a shoe. The device comprises a conveyor for delivering plates carrying lasts with uppers or soles, and a bench with a series of tracks perpendicular to the conveyor, equipped with transfer mechanisms for moving the plates between the conveyor and a point at some distance from it. The device also includes a robot situated to one side of the bench opposite the conveyor, a press to one side of the robot, and a manual assembly post for uppers and soles at the other side. During operations, the robot is configured to pick up a last with an upper, turn it and transfer it towards a sole which has its edges held open by grippers so that the upper can be inserted and fixed with an adhesive.

The claimed invention is defined by the features set forth in the appended independent claims.

In brief and at a high level, this disclosure describes, among other things, manufacturing of a shoe, such as an automated placement of shoe parts. For example, a part-recognition system analyzes an image of a shoe part to identify the part and determine a location of the part. Once the part is identified and located, the part may be manipulated in an automated manner. For example, a first identified part may be placed at a desired location on second identified part. Identified parts may be placed at desired orientations relative to one another.

Illustrative examples of the present invention are described in detail below with reference to the attached drawing figures, wherein:.

The subject matter of certain aspects of the present invention is described with specificity herein to meet statutory requirements. But the description itself is not intended to define what is regarded as an invention, which is what the claims do. The claimed subject matter may comprise different elements or combinations of elements similar to the ones described in this document, in conjunction with other present or future technologies. Terms should not be interpreted as implying any particular order among or between various elements herein disclosed unless explicitly stated.

Subject matter described herein relates to an automated placement of a shoe part, and <FIG> depicts an exemplary system <NUM> that may perform various actions in a shoe-manufacturing process. For example, a shoe part <NUM> may be provided at a supply station <NUM> together with several other shoe parts. Supply station <NUM> may provide only a single type of part or multiple types of parts that are identified individually by system <NUM>. Supply station <NUM> may comprise a conveyor belt, a table, a robotic arm, or any other device that can make shoe part <NUM> available for identification and/or manipulation in accordance with the present invention. An automated tool <NUM> may pick up the shoe part <NUM> from the supply station <NUM>, and the shoe part <NUM> may be transferred to an assembly station <NUM> by a part-transfer apparatus <NUM>.

A ghost depiction <NUM> of part-transfer apparatus is depicted to illustrate that the part-transfer apparatus may move to various positions. Moreover, various arrows 30a-d are depicted that show possible movement directions or rotations of respective components of part-transfer apparatus <NUM>. Part transfer apparatus <NUM> and the movement directions and rotations depicted by <FIG> are exemplary only. For example, arrows 30a and 30d indicate that respective arms of part-transfer apparatus <NUM> may rotate, whereas arrows 30b and 30c indicate that respective arms may move vertically or horizontally (e.g., in a telescoping manner). Although not depicted, arms of part-transfer apparatus may also be comprised of articulating joints that enable additional ranges of motion of part-transfer apparatus <NUM>. The shoe part <NUM> that is transferred may function as a base shoe part <NUM> at the assembly station <NUM>. Alternatively, the shoe part <NUM> that is transferred may be attached to a base shoe part <NUM> that is already positioned at the assembly station <NUM>.

When identifying and/or placing shoe part <NUM> by part-transfer apparatus <NUM>, one or more cameras 22a-f may record images of the shoe part <NUM> that may be used to recognize the shoe part <NUM>. The cameras 22a-f may be arranged at various positions in system <NUM>, such as above a part supply station (e.g., 22a), on part-transfer apparatus <NUM> (e.g., 22b), along a floor <NUM> (e.g., 22c and 22d), and/or above assembly station <NUM> (e.g., 22e and 22f). In addition, the cameras 22a-f may be arranged at various perspectives, such as vertical (e.g., 22b, 22c, 22d, and 22e), horizontal (e.g., 22f), and angled (e.g., 22a). The number, location, and/or orientation of cameras 22a-f may vary beyond the example illustrated in <FIG>.

The images may be used to determine a position and/or orientation of the shoe part <NUM> relative to part-transfer apparatus <NUM> and a position to which shoe part <NUM> is to be transferred. Once the shoe part <NUM> has been recognized, other shoe-manufacturing processes may be carried out in a manual and/or an automated fashion, such as transferring the shoe part, attaching the shoe part via any attachment method, cutting the shoe part, molding the shoe part, etc..

In a further aspect, information (e.g., shoe-part identity and orientation) obtained by analyzing images of the shoe part <NUM> may be combined with information derived from other shoe-part analysis systems in order to carry out shoe-manufacturing processes. For example, a three-dimensional (<NUM>-D) scanning system may derive information (e.g., shoe-part surface-topography information, shoe-part-size information, etc.) from scans of the shoe part (or from scans of another shoe part that is assembled with the shoe part), and the <NUM>-D-system-derived information may be combined with the shoe-part-identity and/or shoe-part orientation information. That is, the <NUM>-D-system-derived information may be determined upstream and communicated downstream to system <NUM> (or vice versa).

Information that is combined from different systems may be used in various manners. In an exemplary aspect, if system <NUM> is used to attach shoe part <NUM> onto shoe part <NUM>, information obtained from another system may be used to instruct and carry out an attachment method. For example, an amount of pressure may be calculated (based on information provided by another system) that is recommended to be exerted against the shoe part <NUM> in order to sufficiently attach the shoe part to one or more other shoe parts <NUM>. Such pressure measurements may be dependent on various factors determined and/or communicated from another system, such as a size (e.g., thickness) of the shoe part and/or a number of shoe parts (e.g., layers) that are being attached.

Computing device <NUM> may help execute various operations, such as by analyzing images and providing instructions to shoe-manufacturing equipment. Computing device <NUM> may be a single device or multiple devices, and may be physically integral with the rest of system <NUM> or may be physically distinct from other components of system <NUM>. Computing device <NUM> may interact with one or more components of system <NUM> using any media and/or protocol. Computing device <NUM> may be located proximate or distant from other components of system <NUM>.

Light-emitting devices <NUM> may be positioned throughout system <NUM> and may be used to enhance a contrast of shoe part <NUM> that may be useful when an image of shoe part <NUM> is used to recognize shoe part <NUM>. Light-emitting devices may be incandescent bulbs, fluorescent devices, LEDs, or any other device capable or emitting light. A light-emitting device may be positioned in various locations, such as near and/or integrated into supply station <NUM> or part-pickup tool <NUM>. Additionally, a light-emitting device may be positioned near or integrated into assembly station <NUM>. Moreover, light-emitting devices may be positioned throughout the space that surrounds part-transfer apparatus <NUM>, part-pickup tool <NUM>, part supply station <NUM>, assembly station <NUM>, and cameras 22a-f. Varying numbers, types, and positions of light emitting devices may be used in accordance with the present invention. Light emitting devices may be selected based upon the spectrum of light emitted and how that spectrum interacts with spectrums reflected by shoe part <NUM>, supply station <NUM>, assembly station <NUM>, part-pickup tool <NUM>, etc. For example, light-emitting devices may provide full-spectrum light and/or partial-spectrum light (e.g., colored light).

Various aspects of <FIG> have been described that may also be applicable to other systems described in this disclosure, such as systems depicted in <FIG>, <FIG>, <FIG>, and <FIG>. Accordingly, when describing these other systems, reference may also be made to <FIG> and aspects described in <FIG> may also apply in these other systems.

As indicated with respect to <FIG>, some aspects of the invention are directed to using an image of a shoe part to identify certain shoe-part information, such as an identity of the shoe part and an orientation of the shoe part (e.g., position and rotation). The shoe-part identity and shoe-part orientation may then be used to carry out various shoe-manufacturing steps (e.g., placement, attachment, molding, quality control, etc.). Accordingly, certain processes may be executed before the image is recorded in order to facilitate shoe-part-image analysis, and reference is made to <FIG> to describe such aspects.

<FIG> illustrates various depictions 1010a-d, each of which provides one or more exemplary shoe-part reference patterns or models (hereinafter known as shoe-part references). For example, depiction 1010a provides an exemplary shoe-part reference 1012a, and depiction 1010b provides a different shoe-part reference 1014a. Depictions 1010a-d may represent data that is maintained in a computer-storage medium and is retrievable to execute computing functions. For example, depictions 1010a-d may be stored in a computer-storage media as reference models or patterns and retrieved in order to be viewed on a computing output device (e.g., computer display monitor).

Shoe-part references 1012a and 1014a may be determined and/or created using various techniques, such as by using a computer-assisted drawing program, an automatic shape-outlining computer program, or other boundary-determination computer program. For example, an electronic image of a shoe part may be recorded and analyzed by the automatic shape-outlining computer program, which automatically traces boundaries or perimeters of shapes that comprise the shoe part. In another aspect, shapes depicted in an electronic image of a shoe part may be manually traced using a computer-drawing application. In another example, a shoe part and/or a boundary associated therewith may be manually drawn using a computer-drawing application. <FIG> depicts that shoe-part references may be comprised of a shoe-part perimeter or boundary (e.g., <NUM>), as well as an interior portion (e.g., <NUM>) bound by the perimeter <NUM>. As previously indicated, once created, a shoe-part reference may be electronically stored (e.g., item <NUM> in <FIG>) and used in various manners, such as to analyze shoe-part images.

In one aspect, a shoe-part reference (e.g., shoe-part reference 1012a) is created such that it may be scaled to correspond to a multiple of different shoe sizes. For example, a shoe-part reference corresponding to a model size (i.e., a model size for females and a model size for males) is created and all other matching shoe-part references are scaled off of the shoe-part reference corresponding to the model size. A shoe-part reference may be scaled up to, for example, five times to account for the different sizes. Further, the shoe-part reference can be scaled to allow for expansion and/or shrinkage for any particular size.

Continuing, references 1012a and 1014a may be used to determine reference information, which may be subsequently used to assemble shoe parts. For example, an attachment shoe part (e.g., <NUM> in <FIG>) may be positioned relative to a base shoe part (e.g., <NUM> in <FIG>); however, before the attachment shoe part is positioned, it may be helpful to determine a placement location at which the attachment shoe part should be positioned.

As such, in an illustrative aspect, depiction 1010c comprises a reference 1014b, which represents a physical boundary of a base shoe part, and a reference 1012b, which represents a physical boundary of an attachment shoe part. In an exemplary aspect, reference 1012b may be positioned to overlay reference 1014b and may be aligned with at least a portion of the reference 1014b. For example, boundary 1012b may be manually and/or automatically positioned (e.g., drag via input device) in a manner that is consistent with how an attachment shoe part would be arranged onto a base shoe part when the attachment shoe part will be attached to the shoe part. As such, depiction 1010d illustrates a digitally rendered assembly of references <NUM>, which is comprised of reference 1012c aligned with reference 1014c in a position consistent with an attachment position.

In a further aspect of the invention, a reference feature <NUM> may be identified that aligns a portion of reference 1012c with a portion of reference 1014c. As such, each of references 1012c and 1014c comprises respective reference features that are generally aligned with one another. These respective reference features are shown in depiction 1010c and are identified by reference numerals <NUM> and <NUM>. For example, a respective reference feature may be used to determine an orientation (e.g., position and rotation) of a shoe part, as well as a portion of the shoe part that aligns with another shoe part.

Now described is <FIG>, in which an exemplary shoe-manufacturing system <NUM> is depicted. System <NUM> may have a combination of shoe-manufacturing equipment and computing devices, which may assist in determining automated operations of the equipment. Operations carried out in system <NUM> may facilitate manipulation of shoe part <NUM> and shoe part <NUM>, such as by transferring shoe part <NUM> and attaching shoe part <NUM> onto shoe part <NUM>. For example, shoe parts <NUM> and <NUM> may comprise two different pieces of flexible material, which are attached to one another to form part of a shoe upper. Shoe parts <NUM> and <NUM> may comprise the same or different types of flexible material, such as textiles, leathers, TPU materials, etc. Shoe parts <NUM> and <NUM> may be physical structures of the completed shoe and/or a component, such as an adhesive film, that may be used to join shoe components during the shoe manufacturing process.

A part-transfer apparatus <NUM>, cameras 214a and 214b, and conveyor <NUM> are examples of shoe-manufacturing equipment. A grid <NUM> is depicted in <FIG> (in broken lines) to convey that one or more items of the shoe-manufacturing equipment have a known position within a coordinate system (e.g., geometric coordinate system mapping a <NUM>-D space within which the equipment is positioned). Other items, such as shoe parts, may be moved to known distances within the coordinate system. Although for illustrative purposes grid <NUM> only depicts two coordinates, axis arrows <NUM> depict three axes.

Image analyzers 216a and 216b and dimension converter <NUM> represent operations and/or modules that may be carried out by a computing device. Moreover, <FIG> depicts that the shoe-manufacturing equipment may communicate with (i.e., be networked with) computing devices that execute the depicted operations by way of a network connection <NUM>. For example, as will be described in more detail below, image analyzers 216a and 216b may evaluate images recorded by cameras 214a and 214b to recognize shoe parts being utilized in the shoe manufacturing process. In addition, image analyzers 216a-b and dimension converter <NUM> communicate instructions to part-transfers apparatus <NUM>. One example of this type of vision recognition system includes Cognex® machine vision systems.

Components depicted in system <NUM> cooperate in different ways to assist in carrying out various steps of a shoe-manufacturing method. For example, some components of system <NUM> may operate collectively as part of a two-dimensional ("<NUM>-D") part-recognition system, which is used to determine various shoe-part characteristics, such as shoe-part identity and shoe-part orientation (e.g., placement and rotation) relative to part-transfer apparatus <NUM>. For example, a part-recognition system may comprise cameras 214a-b, image analyzers 216a-b, shoe-part datastore <NUM>, dimension converter <NUM>, and some or all of part-transfer apparatus <NUM>.

A part-recognition system may be used in various manners within a shoe manufacturing process. For example, a part-recognition system may be used to execute a method <NUM> that is outlined in <FIG>. Method <NUM> relates to identifying a shoe part and determining an orientation (e.g., geometric position and degree of rotation) of the shoe part. When an identity and orientation of a shoe part is known or determined, the shoe part can be manipulated (e.g., transferred, attached, cut, molded, etc.) in an automated manner. In describing <FIG>, reference will also be made to <FIG> and <FIG>.

At step <NUM>, an image is recorded that depicts a representation of a shoe part. For example, an image may be recorded by camera 214a or 214b and communicated to an image analyzer 216a or 216b. Exemplary images <NUM> and <NUM> are illustrated in image analyzers 216a and 216b (respectively), and each image depicts a two-dimensional ("<NUM>-D") representation <NUM> and <NUM> of a respective shoe part.

In step <NUM>, an outline or perimeter of the representation as depicted in the image is recognized. For example, once image analyzer 216a acquires image <NUM>, image analyzer 216a recognizes a perimeter or outline of the <NUM>-D representation <NUM> depicted in image <NUM>. Perimeter or outline recognition may be enhanced using various techniques, such as by providing a background surface that highly contrasts a part depicted in the image, as well as by positioning various environment lighting elements (e.g., full-spectrum light-emitting devices). For example, if a surface of the shoe part that will be captured in the image is grey, a background surface (e.g., surface of a supply station, a part-pickup tool, or an assembly station) may be colored yellow in order to create a contrast in the image between the outline of the part and the background. In one aspect, shoe-part inward-facing surfaces (i.e., a side of the shoe part that may face inward and towards a wearer's foot when assembled into a shoe) and background surface may be manufactured (i.e., intentionally made) to comprise known contrasting colors.

Additional tools may be used to assist with recognizing a perimeter or outline of a representation. For example, system <NUM> may comprise light-emitting devices 241a and 241b that illuminate the shoe part from various sources. As described with respect to <FIG>, light-emitting devices may be arranged in various positions throughout system <NUM>. For example, surface <NUM> may be illuminated with device 241a or backlit with light 241b, thereby enhancing a contrast between surface <NUM> and part <NUM> to render part <NUM> more recognizable to the <NUM>-D recognition system. That is, if part <NUM> is illuminated or backlit when image <NUM> is captured, a better contrast may appear in image <NUM> between representation <NUM> and other portions of the image. A full-spectrum light may be used for enhancing part recognition of parts having various colors. Alternatively, a color of the light may be customized based on a color of part <NUM> and/or the color of supply station and/or assembly station. For example, a red light may be used to enhance a contrast between parts and a supply assembly station that are black or white.

Next, at step <NUM>, image analyzer 216a may determine a plurality of reference features associated with the <NUM>-D representation <NUM> depicted in image <NUM>. For instance, the reference features may comprise a number of spaced lines and/or points that define the outline or perimeter of the <NUM>-D representation. The spacing between adjacent reference features may be variable. For instance, the spacing between reference features for smaller-sized shoe parts may be less than the spacing between reference features for larger-sized shoe parts to allow for more precision. Each reference feature may be comprised of a variable number of pixels.

An identity of a boundary of the <NUM>-D representation <NUM> may be recognized using various techniques. For example, shoe-part representation <NUM> may be compared to various known or model shoe-part references <NUM>-<NUM>, which are stored in shoe-part datastore <NUM> in order to determine the identity of the shoe-part representation <NUM>.

Shoe-part datastore <NUM> stores information <NUM>, which is shown in an exploded view <NUM> for illustrative purposes. As an example, exploded view <NUM> depicts a plurality of known shoe-part references <NUM>-<NUM> that may be used to recognize the identity of the <NUM>-D representation <NUM>. Shoe-part references <NUM>-<NUM> may be associated with pre-determined reference features (e.g., <NUM> and <NUM>) as outlined above with respect to <FIG>, which may be used when assembling a respective shoe part into a shoe. Such reference features may be pre-determined based on various factors, such as a known position of a shoe part among an assembly of shoe parts. For example, when incorporated into a shoe, shoe part <NUM> is assembled at a position with respect to shoe part <NUM>. As such, this position may be measured and used to instruct shoe-manufacturing equipment on positioning and attachment of shoe part <NUM>.

As depicted in <FIG>, shoe-part references <NUM>-<NUM> form various <NUM>-D shapes. In an aspect of the invention, the pre-determined reference features may comprise any number of features associated with the perimeter or outline of the shoe-part references <NUM>-<NUM>. For example, a reference feature may comprise a specified proportion between different sides of the <NUM>-D shape. As well, a reference feature may comprise a junction point between two adjacent sides of the <NUM>-D shape. Creating pre-determined reference features along a perimeter of the shape can reduce variability that may be created when shoe parts are aligned and connected.

The image analyzer 216a may recognize an identity of the <NUM>-D representation <NUM> by identifying at least one shoe-part reference of the plurality of shoe-part references <NUM>-<NUM> that substantially matches the <NUM>-D shoe-part representation <NUM>. For example, the image analyzer 216a may recognize the identity of the <NUM>-D shoe-part representation <NUM> by identifying at least one pre-determined reference feature of a shoe-part reference that substantially matches the at least one reference feature of the <NUM>-D representation <NUM>.

Once a shoe-part representation (e.g., <NUM>) is substantially matched to a known shoe-part reference (e.g., <NUM>), the pre-determined reference feature(s) may be used to analyze an image that depicts the representation. For example, image analyzer 216a has retrieved a recognized entity <NUM> based on shoe-part reference <NUM>, which was substantially matched to <NUM>-D representation <NUM>. As depicted, recognized entity <NUM> has a boundary and pre-determined reference feature(s). Accordingly, when the descriptions of <FIG> and <FIG> are collectively considered, an exemplary method may comprise various steps. For example, model references (e.g., 1012a and 1014a) and their corresponding pre-determined reference features (e.g., <NUM> and <NUM>) are determined and electronically maintained, such as in datastore <NUM>. A recorded image (e.g., <NUM> and <NUM>) may then be substantially matched to a model reference by substantially matching reference features of the recorded image with pre-determined reference features of the model. This reference information may be mathematically depicted with respect to a known reference system.

At step <NUM>, a rotation of the representation (as depicted in the image) and pixel coordinates of the image are identified. To illustrate one manner in which image analyzer 216a utilizes recognized entity <NUM> to execute step <NUM>, information <NUM> is depicted in an exploded view <NUM>. Exploded view <NUM> depicts image <NUM> that is identical to image <NUM>. For example, image <NUM> and image <NUM> may be the same data, or image <NUM> may be a copy of image <NUM>. Image <NUM> is depicted respective to a coordinate system <NUM>, which maps pixels of image <NUM>. Recognized entity <NUM> is applied to image <NUM>, such as by substantially centering image <NUM> within the boundaries of recognized entity <NUM> and aligning by reference feature(s) <NUM>. As such, pixel coordinates of image <NUM> can be determined that belong to coordinate system <NUM>. In addition, a degree of rotation (i.e., Θ) of the shoe-part representation (as depicted in image <NUM>) is determined by measuring an angle between reference lines <NUM> and <NUM>.

The pixel coordinates and degree of rotation that are extracted from the image may be used to instruct part-transfer apparatus <NUM>. That is, image <NUM> may be recorded by camera 214a when shoe part <NUM> is oriented (i.e., positioned and rotated) somewhere in the <NUM>-D space in which part-transfer apparatus <NUM> operates. Examples of positions at which shoe part <NUM> may be located include a part supply station, an assembly station, and/or held by part-transfer apparatus <NUM>. Accordingly, when certain inputs are provided, pixel coordinates of image <NUM> may be converted by dimension converter <NUM> to a geometric coordinate <NUM> of the system represented by grid <NUM>. Accordingly, in step <NUM> of method <NUM> the pixel coordinates may be converted to a geometric coordinate.

Inputs utilized by dimension converter <NUM> may comprise measurement values describing system <NUM>, camera 214a, and part-transfer apparatus <NUM>. Examples of such measurement values are relative positions (i.e., zero positions) of camera 214a and of part-transfer apparatus <NUM>; a number of pixels of the X and Y coordinates of system <NUM>; a distance between camera 214a and part <NUM>; a chip size of the CCD in camera 214a; a lens focal length; a field of view; a pixel size; and a resolution per pixel. These inputs may vary depending on the capabilities of the equipment used in system <NUM> and some inputs may have a direct bearing on where equipment may be positioned within system <NUM>. For example, the strength of camera 214a may have a bearing on where part <NUM> should be positioned (relative to camera 214a) when camera 214a will record an image of part <NUM>. To further illustrate a relationship between various inputs used to convert a pixel coordinate to a geometric coordinate, <FIG> depicts a schematic diagram of a system with which an image may be recorded and analyzed.

The geometric coordinate generated by dimension converter <NUM> can be used to report a position of shoe part <NUM> to part-transfer apparatus <NUM>. Moreover, the degree of rotation can be used to determine to what extent shoe part <NUM> may need to be rotated by part-transfer apparatus <NUM> in order to be properly aligned for subsequent manipulation (e.g., attachment to another shoe part, cutting, painting, etc.). Accordingly, part-transfer apparatus <NUM> may comprise a part-pickup tool that enables part-transfer apparatus <NUM> to acquire part <NUM> from a part-supply area and hold part <NUM> while transferring part <NUM> to a new location. For example, part-transfer apparatus <NUM> may use a gripping structure, suction, electromagnetic forces, surface tack, or any other methodology to temporarily engage and move a shoe part.

Although the above <NUM>-D recognition process is described by referencing shoe part <NUM> and image <NUM>, a similar analysis may be used to identify shoe part <NUM> and determine its orientation, thereby enabling part-transfer apparatus <NUM> to account for part <NUM> when manipulating part <NUM>. That is, information <NUM> is depicted in image analyzer 216b and is shown in an exploded view <NUM> for illustrative purposes. Exploded view <NUM> conveys that image <NUM> may be analyzed similar to image <NUM> to determine an orientation (i.e., geometric coordinate and degree of rotation) of part <NUM> based on reference feature(s) <NUM> and theta. Any number of shoe parts may be identified and/or positioned, either simultaneously or sequentially in accordance with the present invention.

Once respective geometric coordinates of part <NUM> and part <NUM> are known, part-transfer apparatus <NUM> can pick up part <NUM> and move part <NUM> to a part-position coordinate <NUM> that is relative to the geometric coordinate of part <NUM>. For example, <FIG> depicts multiple broken-line views of part-transfer apparatus <NUM> to illustrate a movement of part-transfer apparatus and a transfer of part <NUM>. A part-position coordinate <NUM> refers to a coordinate in the geometric coordinate system (e.g., the system illustrated by grid <NUM>) to which an attachment part (e.g., part <NUM>) is transferred in order to be attached to a base part (e.g., part <NUM>). For example, part-transfer apparatus <NUM> may transfer part <NUM> to geometric coordinate <NUM> to be attached to part <NUM>.

A part-position coordinate <NUM> may be determined in various ways. For example, part <NUM> may be a base shoe part onto which part <NUM> is attached, such that a position of part <NUM> respective to part <NUM> (when the parts are assembled) is known. As such, the known position may be determined by retrieving a stored reference feature, which was pre-determined using a method similar to that described with respect to <FIG>. However, this position that is known may still be converted to a coordinate that is recognized by part-transfer apparatus <NUM> when part <NUM> has been positioned within a coordinate system of part-transfer apparatus <NUM>. That is, outside of coordinate system <NUM>, a position relative to part <NUM> at which part <NUM> is arranged is known, and is identified by reference numeral <NUM> in datastore <NUM>. This position is also identified in exploded view <NUM> in which the position is identified as "part-position location for part <NUM>. " When an orientation of part <NUM> is determined, such as by executing method <NUM>, the point <NUM> (also depicted in exploded view <NUM>) that is respective to part <NUM> at which part <NUM> is arranged can be converted to a geometric coordinate <NUM> within system <NUM>, thereby calculating part-position coordinate <NUM>. Accordingly, in an exemplary aspect, part-position <NUM> is converted to a geometric coordinate based in part on reference feature <NUM>, which was described with reference to <FIG>.

In a further aspect, once part-position point <NUM> is determined, part <NUM> can be transferred to the part-position coordinate <NUM> based on the reference information determined with respect to part <NUM> (e.g., <NUM> in <FIG>). For example, pixel coordinates and orientation may be derived from image <NUM> (as described above) and may be converted to a geometric coordinate (e.g., <NUM>). Calculations may then be made to transfer part <NUM> to point <NUM>. For example, a virtual robot end effector may be created based on the geometric data (e.g., <NUM> and <NUM>) and may be moved from point <NUM> to point <NUM>. While these steps are depicted graphically in <FIG> for illustrative purposes, these steps could also be executed mathematically by solving sequential conversion algorithms.

Accordingly, the above-described recognition process (e.g., method <NUM>) may be used in many different scenarios within a shoe-manufacturing process. For example, once shoe part <NUM> has been positioned respective to shoe part <NUM>, shoe part <NUM> can be attached to shoe part <NUM>, such as by stitching, adhering, and/or sonic welding. As such, in order to enable automation, a geometric coordinate <NUM> of the attachment point is also determined. That is, once geometric coordinates of parts <NUM> and <NUM> are known within coordinate system <NUM>, geometric coordinates of attachment locations can also be calculated.

An attachment-point coordinate <NUM> may be determined in various ways. For example, part <NUM> may be a base shoe part onto part <NUM> is attached. As such, a point of attachment onto base shoe part is known, but it still may be converted to a coordinate that is recognized by part-transfer apparatus <NUM>. That is, outside of coordinate system <NUM>, a point on part <NUM> at which part <NUM> will be attached is known, and is identified by reference numeral <NUM> in datastore <NUM>. When an orientation of part <NUM> is determined, such as by executing method <NUM>, the point <NUM> (also depicted in exploded view <NUM>) on part <NUM> at which part <NUM> is attached can be converted to a geometric coordinate <NUM> within system <NUM>. As such, an attachment process can be executed at the geometric coordinate <NUM>. As indicated above, although these steps are depicted graphically in <FIG> for illustrative purposes, these steps could also be executed mathematically by solving sequential conversion algorithms.

In one aspect, part-transfer tool <NUM> also may have an attachment device, which operates to attach part <NUM> to part <NUM>. Exemplary attachment devices are an ultrasonic welder, heat press, stitching apparatus, or a device that accomplishes a respective method of attachment.

The components of system <NUM> may be arranged in various configurations to accomplish a wide range of shoe-manufacturing processes. In addition, there may be additional components arranged into a series of stations. For example, system <NUM> may be comprised of cameras in addition to cameras 214a-b, as well as additional part-transfer apparatuses. Different types of cameras and/or part transfer apparatuses may be combined in accordance with the present invention. These additional tools may be arranged at different positions along conveyor <NUM> to allow additional parts to be added (e.g., added to the assembly of parts <NUM> and <NUM>) and to allow additional shoe-part manipulation.

Moreover, the cameras of system <NUM> may be arranged at different positions with respect to a shoe part. For example, as depicted in <FIG>, cameras may be positioned above a shoe part, below a shoe part, horizontal to a shoe part, or at an angle away from a shoe part, so long as the camera position allows the geometric coordinate of the part to be calculated. One such camera position may be perpendicular to (i.e., normal to) a viewing plane. However, the camera could be positioned at an angle from the viewing plane, so long as the angle is provided as an input to the system when converting the representation orientation to a geometric coordinate. Accordingly, system <NUM> may be incorporated into larger shoe-manufacturing processes.

A <NUM>-D recognition system may be used at an initial stage to enable part-transfer apparatus <NUM> to position a base shoe part onto a conveyor or other part-moving apparatus. A base shoe part refers to a shoe part onto which one or more other shoe parts may be attached, and a base shoe part may be constructed of a single part or a plurality of parts that have been assembled. Accordingly, part <NUM> may be deemed a base shoe part onto which part <NUM> is attached. Parts transferred may also be foams, mesh, and/or adhesive layers, such as TPU films, ultimately used to join other parts together. Further, component parts previously affixed to one another in accordance with the present invention may be treated as a single part for subsequent identification transfer, etc..

Referring to <FIG>, a system <NUM> is depicted in which a <NUM>-D part-recognition system may be used at an initial manufacturing stage, such as when the base shoe part <NUM> is initially stored at a part-supply station <NUM>, which may be comprised of various configurations. For example, a part-supply station <NUM> may comprise a set of stacked base shoe parts from which part-transfer apparatus <NUM> acquires a topmost base shoe part. Alternatively, the part-supply station may have a conveyor <NUM> that transfers the base shoe part to a pickup location <NUM> at which part-transfer apparatus <NUM> acquires the base shoe part. As previously described, part-transfer apparatus <NUM> may have a part-pickup tool <NUM>.

Prior to transferring base shoe part <NUM> to conveyor <NUM>, a camera may record an image of the base shoe part <NUM> to allow part-transfer apparatus <NUM> to determine a geometric position and rotation of the base shoe part <NUM>. For example, a camera may record an image of the base shoe part <NUM> when the base shoe part <NUM> is next-in-line to be acquired by part-transfer apparatus <NUM> - i.e., immediately prior to the base shoe part <NUM> being acquired by part-transfer apparatus <NUM> and when the base shoe part <NUM> is at pickup location <NUM>. The camera may be an above-mounted camera 590a-b that is mounted above, and perpendicular to, the base shoe part <NUM>. As depicted in <FIG>, an above-mounted camera 590a-b may be mounted either apart from (e.g., 590a) or onto (e.g., 590b) part-transfer apparatus <NUM>.

Although part-transfer apparatus <NUM> is illustrated to have a certain configuration depicted in <FIG>, part-transfer apparatus may have a different configuration, such as the configuration depicted in <FIG>, in which a camera mounted to the part-transfer apparatus may be positionable directly above and perpendicular to base shoe part <NUM>. Part-transfer apparatus <NUM> may also comprise a plurality of articulating arms that enable movement of a camera (or an acquired shoe part) to a desired angle or position.

Moreover, if the image is recorded while the base shoe part <NUM> is at a part-supply station (i.e., at location <NUM>), a light-emitting device may be arranged at various positions throughout system <NUM>. For example, a light-emitting device 541a may be positioned adjacent to or incorporated into the part-supply station <NUM> to provide a backlight to the base shoe part <NUM>. Also, a light-emitting device 541b may be positioned in a space that surrounds base shoe part, such that the light-emitting device 541b illuminates base shoe part <NUM> from a front side.

Alternatively, part-transfer apparatus <NUM> may acquire base shoe part <NUM> before an image is recorded and position the acquired base shoe part in front of a camera. For example, a below-mounted camera <NUM> may be secured near a floor surface, and part-transfer apparatus <NUM> may position the acquired base shoe part directly above, and perpendicular to, the below-mounted camera <NUM>. Alternatively, part-transfer apparatus <NUM> may position the acquired base shoe part directly below, and perpendicular to, above-mounted cameras 590a or <NUM>. As described above, although part-transfer apparatus <NUM> is illustrated to have a certain configuration depicted in <FIG>, part-transfer apparatus may have a different configuration. For example, part-transfer apparatus <NUM> may have the configuration depicted in <FIG>. In addition, part-transfer apparatus may be comprised of a plurality of articulating arms.

If the image is recorded after the base shoe part <NUM> has been acquired by part-transfer apparatus, a light-emitting device 541c may be arranged at various positions. For example, a light-emitting device 541c may be incorporated into the part-transfer apparatus <NUM>, such as behind (or incorporated into) the part-pickup tool <NUM>, thereby providing a backlight to base shoe part <NUM>. In addition, other light-emitting devices (e.g., 541d) positions throughout system <NUM> may illuminate a front side of a base shoe part that is acquired by part-transfer apparatus <NUM>.

Once an image has been recorded, a geometric position and rotation of the base shoe part may be determined using the previously described methods (e.g., method <NUM>). The geometric position and rotation may then be used to determine a position of the base shoe part when the base shoe part is transferred to conveyor <NUM>. For example, part-transfer apparatus <NUM> may execute a predetermined movement path each time it transfers base shoe part <NUM> from a part-supply station <NUM>, or from in front of a camera (e.g., 590a, <NUM>, or <NUM>), to conveyor <NUM>. As such, once the geometric position and rotation of the base shoe part are known, the part-transfer apparatus may determine where the base shoe part will be positioned when the predetermined movement path is executed. Alternatively, a geometric position on conveyor <NUM> may be predetermined, such that part-transfer apparatus <NUM> (or some computing device associated therewith) calculates a new movement path each time. That is, the new movement path extends from the calculated position of the base shoe part <NUM> (when the image is recorded) to the predetermined position on the conveyor <NUM>. Computing device <NUM> may help execute various operations, such as by analyzing images and providing instructions to shoe-manufacturing equipment.

In another aspect, a <NUM>-D recognition system may be used when base shoe part <NUM> has already been transferred to conveyor <NUM> in order to determine a geometric position and rotation of base shoe part <NUM> as it is arranged on conveyor <NUM>. As such, conveyor <NUM> may move base shoe part along an assembly line and to a position that is beneath an above-mounted camera (e.g., <NUM>). Once an image has been recorded by the above-mounted camera and a position of base shoe part has been determined, other shoe parts may be transferred and attached to the base shoe part.

As such, in a further aspect, a <NUM>-D recognition system may be used after the initial stage to enable a part-transfer apparatus to position an attachment shoe part. An attachment shoe part refers to a shoe part that is to be attached to a base shoe part. Accordingly, in <FIG> part <NUM> may be deemed an attachment shoe part that is to be attached to shoe part <NUM>.

Referring to <FIG>, a system <NUM> is depicted in which a <NUM>-D recognition system may be used to position an attachment part <NUM>, such as when the attachment shoe part <NUM> is initially stored at a part-supply station <NUM>, which may be arranged into various configurations. As previously described, a part-supply station <NUM> may comprise a set of stacked shoe parts from which part-transfer apparatus <NUM> acquires a topmost attachment shoe part. Alternatively, the part-supply station <NUM> may be comprised of a set of conveyors 682a and 682b, one of which transfers the attachment shoe part <NUM> to a pickup location <NUM> at which part-transfer apparatus <NUM> may acquire the attachment shoe part <NUM>.

As previously described, part-transfer apparatus <NUM> may have a part-pickup tool <NUM>. Although part-transfer apparatus <NUM> is illustrated to have a certain configuration depicted in <FIG>, part-transfer apparatus may have a different configuration, such as the configuration depicted in <FIG>, or a configuration comprising a plurality of articulating arms that enable movement of a camera (or an acquired shoe part) to a desired angle or position.

The attachment shoe part <NUM> may be provided at the supply station <NUM> among a plurality of different attachment shoe parts (e.g., <NUM> and <NUM>), each of which may be attached to a respective portion of base shoe part <NUM>. As such, <NUM>-D recognition system may execute a part-selection protocol, which allows the system to identify and select a desired attachment part.

In an exemplary part-selection protocol, the <NUM>-D recognition system may be programmed to follow a predetermined order of attachment parts - i.e., attach first part <NUM>, followed by second part <NUM>, followed by third part <NUM>, etc. Accordingly, the <NUM>-D recognition system may record images of all of the parts arranged among the plurality, identify each part (e.g., based on datastore <NUM>), and determine a geometric location of each part as it is positioned at supply station <NUM>. Once this position information has been determined by the <NUM>-D recognition system, part-transfer apparatus <NUM> may acquire and attach each part in the predetermined order.

In another part-selection protocol, the <NUM>-D recognition system may be programmed to transfer and attach a set of parts, regardless of the order - i.e., attach first, second, and third parts in any order. Accordingly, once images of each part (e.g., <NUM>, <NUM>, and <NUM>) have been analyzed to determine a geometric position, part-transfer apparatus <NUM> may acquire the parts in a variety of orders, as long as all of the parts are transferred to the base part <NUM> at some point. Moreover, the <NUM>-D recognition system may be programmed to retrieve the parts that are positioned in a manner that allows for the most efficient transfer from the supply station <NUM> to base shoe part <NUM>. For example, if two first parts 698a and 698b are provided at the supply station and one of the first parts 698a is closer than the other first part 698b (based on respective geometric coordinates), the part-transfer apparatus <NUM> may be instructed to pick up the closer first part 698a instead of the other first part 698b. Similarly, if a first part 698a is rotated to a degree that may need less adjustment (relative to another first part 698b) in order to be attached to base part <NUM>, the part-transfer apparatus <NUM> may be instructed to pick up the first part 698a. Computing device <NUM> may help execute various operations, such as by executing certain steps in a part-selection protocol, analyzing images, and providing instructions to shoe-manufacturing equipment.

In another exemplary aspect, parts <NUM>, <NUM>, and <NUM> may be arranged at part-pickup location <NUM> in a pre-determined configuration, such that coordinates of the pre-determined configuration may be provided to apparatus <NUM> to assist with part selection. That is, if a coordinate of each part <NUM>, <NUM>, and <NUM> is pre-determined based on how the group of parts are to be arranged (prior to being picked up), then a coordinate may not have to be calculated based on images. Or, a pre-determined coordinate may be used as a check to confirm that a calculated coordinate is accurate (e.g., within a threshold amount away from the pre-determined coordinate).

In a further aspect, a pre-determined arrangement of parts <NUM>, <NUM>, and <NUM> at part-pickup location <NUM> may match an arrangement of the parts <NUM>, <NUM>, and <NUM> when the parts are attached to base part <NUM>. That is, each of parts <NUM>, <NUM>, and <NUM> may be spaced apart from one another and rotated in a manner that matches a spacing and rotation of each part when attached to base part <NUM>. As such, parts <NUM>, <NUM>, and <NUM> may be picked up, placed, and/or attached as a collective group (i.e., more than one at a time) in a manner that maintains the pre-determined arrangement (i.e., maintains the spacing and rotation).

When an image is recorded of an attachment shoe part <NUM> to determine an orientation of the attachment shoe part <NUM>, the camera may be positioned in various locations. As previously described, if the attachment shoe part <NUM> is positioned at the supply station <NUM> when the image is captured, the camera (e.g., 690b) may be coupled directly to part-transfer apparatus <NUM>, or may be an above-mounted camera 690a. Camera 690b or 690a may be perpendicularly oriented from shoe part <NUM> when the image is recorded. For example, part-transfer apparatus <NUM> may be comprised of one or more articulating arms that position camera 690b above and perpendicular to shoe part <NUM>.

Moreover, light-emitting devices may be arranged throughout system <NUM> to illuminate shoe part <NUM> when positioned at part-supply station <NUM>. For example, a light-emitting device 641a or 641b may be positioned adjacent to, or integrated into, the supply station <NUM> in order to backlight the attachment shoe parts positioned on conveyors 682a and 682b. Also, light-emitting devices 641c may be positioned in a space surrounding part-supply station <NUM> to illuminate a front side of shoe part <NUM>.

If the attachment shoe part <NUM> is retained by part-transfer apparatus <NUM> when the image is captured, the camera may be mounted remotely from the part-transfer apparatus <NUM>, such as camera 690a, <NUM>, or <NUM>. In such an arrangement, shoe-transfer apparatus <NUM> may position the attachment shoe part in front of (e.g., perpendicular to a field of view of) camera 690a, <NUM>, or <NUM>. Moreover, a light-emitting device 641d may be integrated into the part-transfer apparatus <NUM>, such as behind the part-pickup tool <NUM>, in order to illuminate the acquired shoe parts when the image is captured.

Although some of the above methods describe analyzing a single image to determine an orientation, multiple images of a single part, which are recorded by one or more cameras, may be analyzed to derive a set of geometric coordinates that are believed to accurately represent a position of a shoe part. In such a system, the set of geometric coordinates may be averaged or otherwise combined to arrive at a final geometric coordinate.

Referring now to <FIG>, a flow diagram is depicted of a method <NUM> for positioning a shoe part in an automated manner during a shoe-manufacturing process. In describing <FIG>, reference is also be made to <FIG>. In addition, method <NUM>, or at least a portion thereof, may be carried out when a computing device executes a set of computer-executable instructions stored on computer storage media.

At step <NUM> an image (e.g., <NUM>) may be received depicting a two-dimensional representation (e.g., <NUM>) of an attachment shoe part (e.g., <NUM>), which is to be attached to a base shoe part (e.g., <NUM>), wherein the two-dimensional representation of the attachment shoe part comprises a plurality of reference features <NUM>. At step <NUM>, pixel coordinates of the image (e.g., coordinate of system <NUM>) are identified that correspond to the reference features. Step <NUM> converts the pixel coordinates of the image to a geometric coordinate (e.g., <NUM>) of a geometric coordinate system (e.g., <NUM>), which maps a three-dimensional space within which the attachment shoe part (e.g., <NUM>) is positioned and a part-transfer apparatus (e.g., <NUM>) operates. Further, at step <NUM>, another geometric coordinate (e.g., <NUM>) of the geometric coordinate system (e.g., <NUM>) is determined by analyzing a different image (e.g., <NUM>) depicting a two-dimensional representation (e.g., <NUM>) of the base shoe part (e.g., <NUM>) to which the attachment shoe part (e.g., <NUM>) will be attached. Step <NUM> transfers, by the part-transfer apparatus (e.g., <NUM>), the attachment shoe part (e.g., <NUM>) to the other geometric coordinate (e.g., <NUM>), thereby moving the attachment shoe part to a location in the three-dimensional space at which the attachment shoe part is to be attached to the base shoe part.

Referring now to <FIG>, another flow diagram is depicted of a method <NUM> for positioning a shoe part in an automated manner during a shoe-manufacturing process. In describing <FIG>, reference is also be made to <FIG>. In addition, method <NUM>, or at least a portion thereof, may be carried out when a computing device executes a set of computer-executable instructions stored on computer storage media.

At step <NUM> an image (e.g., <NUM>) is received depicting a two-dimensional representation (e.g., <NUM>) of an attachment shoe part (e.g., <NUM>), which is to be attached to a base shoe part (e.g., <NUM>), wherein the two-dimensional representation of the attachment shoe part comprises at least one reference feature <NUM>. At step <NUM>, pixel coordinates of the image (e.g., coordinate of system <NUM>) are identified that correspond to the at least one reference feature <NUM>. Step <NUM> converts the pixel coordinates of the image to a geometric coordinate (e.g., <NUM>) of a geometric coordinate system (e.g., <NUM>), which maps a three-dimensional space within which the attachment shoe part (e.g., <NUM>) is positioned and a part-transfer apparatus (e.g., <NUM>) operates. Furthermore, step <NUM> determines a plurality of other geometric coordinates (e.g., <NUM> and <NUM>) in the geometric coordinate system by analyzing a different image (e.g., <NUM>) depicting a two-dimensional representation (e.g., <NUM>) of the base shoe part (e.g., <NUM>) to which the attachment shoe part (e.g., <NUM>) will be attached. The plurality of other geometric coordinates may comprise a part-position coordinate (e.g., <NUM>) and a part-attachment coordinate (e.g., <NUM>). Step <NUM> transfers, by the part-transfer apparatus, the attachment shoe part (e.g., <NUM>) to the part-position coordinate (e.g., <NUM>), and step <NUM> attaches the attachment shoe part to the base part at the part-attachment coordinate (e.g., <NUM>).

The <NUM>-D recognition system described above may also be used for quality control purposes. For instance, the <NUM>-D recognition system may allow for detection of a mismatched attachment part in a set of matching stacked attachment parts. Further, the <NUM>-D recognition system may also enable quality control of shoe-part positioning to ensure position placement accuracy.

As described above, our technology may comprise, among other things, a method, a system, or a set of instructions stored on one or more computer-readable media. Information stored on the computer-readable media may be used to direct operations of a computing device, and an exemplary computing device <NUM> is depicted in <FIG>. Computing device <NUM> is but one example of a suitable computing system and is not intended to suggest any limitation as to the scope of use or functionality of invention aspects. Neither should the computing system <NUM> be interpreted as having any dependency or requirement relating to any one or combination of components illustrated. Moreover, aspects of the invention may also be practiced in distributed computing systems where tasks are performed by separate or remote-processing devices that are linked through a communications network.

Computing device <NUM> has a bus <NUM> that directly or indirectly couples the following components: memory <NUM>, one or more processors <NUM>, one or more presentation components <NUM>, input/output ports <NUM>, input/output components <NUM>, and an illustrative power supply <NUM>. Bus <NUM> represents what may be one or more busses (such as an address bus, data bus, or combination thereof). Although the various blocks of <FIG> are shown with lines for the sake of clarity, in reality, delineating various components is not so clear, and metaphorically, the lines would more accurately be grey and fuzzy. For example, processors may have memory.

Computing device <NUM> typically may have a variety of computer-readable media. By way of example, and not limitation, computer-readable media may comprises Random Access Memory (RAM); Read Only Memory (ROM); Electronically Erasable Programmable Read Only Memory (EEPROM); flash memory or other memory technologies; CDROM, digital versatile disks (DVD) or other optical or holographic media; magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, carrier wave or any other medium that can be used to encode desired information and be accessed by computing device <NUM>.

Memory <NUM> is comprised of tangible computer-storage media in the form of volatile and/or nonvolatile memory. Memory <NUM> may be removable, nonremovable, or a combination thereof. Exemplary hardware devices are solid-state memory, hard drives, optical-disc drives, etc..

Computing device <NUM> is depicted to have one or more processors <NUM> that read data from various entities such as memory <NUM> or I/O components <NUM>. Exemplary data that is read by a processor may be comprised of computer code or machine-useable instructions, which may be computer-executable instructions such as program modules, being executed by a computer or other machine. Generally, program modules such as routines, programs, objects, components, data structures, etc., refer to code that perform particular tasks or implement particular abstract data types.

Presentation component(s) <NUM> present data indications to a user or other device. Exemplary presentation components are a display device, speaker, printing component, light-emitting component, etc. I/O ports <NUM> allow computing device <NUM> to be logically coupled to other devices including I/O components <NUM>, some of which may be built in.

Claim 1:
A method for positioning a shoe part in an automated manner during a shoe-manufacturing process, the method comprising:
receiving (<NUM>; <NUM>) an image (<NUM>) depicting a two-dimensional representation (<NUM>) of a first shoe part;
determining an identity of the first shoe part by substantially matching the image (<NUM>) to a reference image (<NUM>);
determining (<NUM>; <NUM>), from the image (<NUM>), a geometric coordinate (<NUM>) of the first shoe part in a geometric coordinate system (<NUM>), which maps a three-dimensional space within which the first shoe part is positioned and a part-transfer apparatus (<NUM>) operates;
determining (<NUM>; <NUM>), from a different image (<NUM>), a geometric coordinate (<NUM>) of a second shoe part in the geometric coordinate system (<NUM>) by analyzing the different image (<NUM>); and
transferring (<NUM>; <NUM>), by the part-transfer apparatus (<NUM>), the first shoe part to the geometric coordinate (<NUM>) of the second shoe part.