KITTING SHELF MACHINE WITH SELF-LOADING PRESENTATION FACE

A robotic singulation system is disclosed. In various embodiments, sensor data image including data associated with an item present in a workspace in a workspace is received. The sensor data is used to determine and implement a plan to autonomously operate a robotic structure to move and place the item singly in a corresponding location in a singulation conveyance structure. The plan takes into consideration an attribute of the item determined based at least in part on the sensor data.

BACKGROUND OF THE INVENTION

Robots have been used to perform tasks in manufacturing and other fields. For example, robots have been used to perform tasks in environments that may be unhealthy or otherwise dangerous to humans, tasks that require the application of force greater than a human may be able to apply, and tasks that require a high degree of precision and consistency over time.

Autonomous robots perform at least some tasks in an automated manner, without requiring human control or direction. For example, automated robots have been used to perform repetitive and/or otherwise predetermined tasks and sequences of tasks, typically in a controlled environment, such as a factory. More recently, self-driving cars, delivery drones, and other autonomous vehicles have been under development.

Teleoperation in the field of robotics refers to remote operation of a robot by an operator. For example, robots have been used to perform surgery, defuse bombs, and perform other tasks under the control of a skilled human operator.

Kitting and related processes typically involve gathering and packaging individual items together in a single kit or package. For example, an online or mail order retailer may collect individual items included in an order and package them together in a box or other packaging, which is then addressed and shipped to a destination address associated with the order.

Retailers and other vendors typically stock individual items they offer for sale. Such stock may be maintained in a warehouse or other storage facility. A process to fulfill an order may involve locating and selecting the ordered quantity of each individual item; selecting packaging, such as a box of a size and dimensions to accommodate the collection of items in the order; arranging items in the box or other packaging; addressing and shipping the package; processing a sale transaction; etc.

Kitting may be performed manually. For example, employees may collect items from shelves, bins, or other storage locations, each in a corresponding location within a warehouse or other facility. Aspects of kitting operations have been automated in part, such as box assembly. Use of robots or other machines to perform kitting operations has been proposed and explored, but challenges have been encountered, such as the relative complexity associated with using a robotic arm to find arbitrary quantities of an arbitrary set of items, and providing and programming a robot to perform tasks such as reaching into a bin or shelf, picking up items of arbitrary size, fragility, consistency, etc. As a result, large scale kitting operations have continued to be human labor intensive.

DETAILED DESCRIPTION

As used herein, kitting includes the picking of one or more items/objects from corresponding locations and placing the one or more items in a predetermined location in a manner that a set of the one or more items correspond to a kit.

“Kitting machines” or “kitting systems” and their integration into highly automated kitting operations are disclosed. In various embodiments, a kitting machine as disclosed herein comprises an at least partly robotically controlled unit that supplies and positions an item to facilitate the item being located, picked up, and/or positioned in and/or for packaging and/or shipment as part of a kitting operation. In various embodiments, a kitting machine as disclosed herein may comprise one or more kitting shelf machine modules (KSM), each comprising a modular component. A KSM as disclosed herein may comprise one or more shelves, bins, or other receptacles. In some embodiments, the shelves, bins, or other receptacles may be positioned via robotic control to position an item of pick up. A KSM as disclosed herein may be integrated with one or more other KSM's, one or more robotic arms, and/or other components to comprise an at least partly automated kitting system, capable of locating, selecting, and packing prescribed quantities of each of one or more arbitrary individual items, such as items included in an order, invoice, or similar data.

A kitting system configured to perform kitting is disclosed. In some embodiments, the kitting system includes a kitting shelf system that is used in connection with kitting. The kitting shelf system may include one or more shelves on which one or more items are stored for use in a kitting process for assembling one or more kits. The kits may be assembled based at least in part on a corresponding order (e.g., based on a packing slip associated with the order). Various embodiments include one or more robotic systems. A robotic system may include one or more robotic arms that are respectively configured to autonomously operate to pick an item/object from a first location and place the item/object in a second location. A robotic arm included in a kitting system may be controlled to operate (e.g., autonomously) pick and place the item/object according to a plan to assemble a kit.

According to various embodiments, the kitting shelf system controls the flow of items thereon and the kitting shelf system presents an item to a robotic system or human operator for removal of the item. The presentation of an item by the kitting shelf system is more effective and/or efficient than a shelfing used by traditional shelving systems. For example, the item is presented at an orientation (e.g., an angle) that improves line of sight towards the item on the kitting shelf system. A human operator or robotic system can more easily see the item and/or objects within the item while the item is on the kitting shelf system. As another example, the item is presented at a location and/or orientation that improves the ability of a robot arm to grasp the item from the kitting shelf system. In some embodiments, the kitting shelf system comprises a feeder portion, a presentation face, and a gate structure.

According to various embodiments, a feeder portion of the kitting shelf system receives items that are to be presented on at least one shelf. The feeder portion may be configured to receive a succession of items on the receiving end and convey the items to the destination end. Items may be manually loaded on the receiving end of the feeder portion (e.g., by a human operator), or automatically loaded on the receiving end such as via a conveyor system. In some embodiments, each shelf in the kitting shelf system comprises a corresponding feeder portion. The feeder portion may convey the items to the destination based on its orientation or based on a conveyance system. For example, the feeder portion may be oriented in a manner that gravity acts on the various items on the feeder portion to convey the items from the receiving end to the destination end. As another example, the feeder portion may include a conveyor that carries the items from the receiving end to the destination end or other element (e.g., a robotic element) that pushes the items along the feeder portion.

According to various embodiments, the presentation face of the kitting shelf system includes an area on which an item is presented for picking from the corresponding shelf. The kitting shelf system may be configured to control flow of items from the feeder portion to the presentation face such that the items are singly received on (e.g., conveyed to) the presentation face. The presentation face may singly receive an item in response to removal of a preceding item from the presentation face. For example, in response to a preceding item being removed from the presentation face, another item is singly conveyed from the feeder portion to the presentation face. In some embodiments, the presentation face is configured to hold a single item. The item may be a tray that includes one or more objects. In response to a determination that the item (e.g., the tray) is empty, the tray may be removed (e.g., by a robotic arm), and a succeeding item (e.g., a tray having objects therein) is conveyed to the presentation face.

In some embodiments, the kitting shelf system is configured to change an orientation of the presentation face. As an example, the kitting shelf may change the orientation of the presentation face based at least in part on whether the presentation face has an item disposed thereon. As another example, the kitting shelf system may configured to change an orientation of the presentation face based at least in part on a number of items disposed on the presentation face in relation to a threshold number of items. As another example, the kitting shelf system may be configured to change an orientation of the presentation face based at least in part on a weight of one or more items on the presentation face (e.g., a weight or force applied to the presentation face relative to a threshold weight or force). In some embodiments, the presentation face can be toggled between two positions—a position corresponding to the gate structure being in a closed position, and a position corresponding to the gate structure being in an open position. As an example, the presentation face may disposed in a first orientation to facilitate conveyance of an item from the feeder portion to the presentation face (e.g., when the gate structure is in an open position). As another example, the presentation face may be disposed in a second orientation to facilitate the picking of an item from the presentation face (e.g., by a robotic arm) and/or to facilitate capturing an image of the workspace such as an image of at least one or more items (or objects within an item) on the presentation face. According to various embodiments, the kitting shelf system is configured to dampen the movement of the presentation face (e.g., to dampen the movement of the presentation face between the first orientation and the second orientation). Dampening movement of the presentation face may help prevent movement of an item on the presentation face, and/or prevent an item from falling from the presentation face.

According to various embodiments, a gate structure of the kitting shelf system controls conveyance of an item from a feeder portion to a presentation face. The gate structure may be configured to toggle between at least a closed position and a closed position. In some embodiments, the gate structure is positioned in the closed position when an item being present on the presentation face (e.g., in response to an item being moved to the presentation face). In some embodiments, the gate structure is positioned in the open position when no items are on the presentation face (or a number of items on the presentation face is less than a threshold number of item). The gate structure may be moved to an open position and/or to a closed position based on a number of items on the presentation face (e.g., relative to a threshold number such as one). For example, the gate structure may be automatically moved to the open position in response to an item being removed from the presentation face. As another example, the gate structure may be automatically moved to the closed position in response to an item being moved to the presentation face (e.g., from the feeder portion). The gate structure may be moved to an open position and/or to a closed position based at least in part on a weight on (or force applied) the presentation face. For example, the gate structure may be automatically moved to the open position in response to a weight on (or force applied to) the presentation face is less than a threshold weight or a counteracting biasing force (e.g., applied to the presentation face). As another example, the gate structure may be automatically moved to the closed position in response to a weight on (or force applied to) the presentation face being greater than a threshold weight or a counteracting biasing force (e.g., applied to the presentation face). The number of the items on the presentation face or the weight on (or force applied to) the presentation face may be determined based at least in part on information obtained from one or more sensors. For example, the presence or absence of an item or the weight on the presentation face may be determined using an output from one or more weight sensors the measure a weight on the presentation face. As another example, the presence or absence of an item or a number of items on the presentation face can be determined based at least in part an image captured with respect to the workspace (e.g., an image of the presentation face). In some embodiments, a computer system operatively controls the gate structure (e.g., to be placed in an open position or a closed position). The computer system may control the gate structure based at least in part on the information obtained from the one or more sensors.

According to various embodiments, the kitting system picks and places one or more items (or objects from within the items on the presentation faces) according to a plan to assemble a kit. The plan may be determined based at least in part on an order (e.g., a packing slip corresponding to an order). The order may be an order from an electronic commerce website. The plan may be further determined based at least in part on information corresponding to the workspace (e.g., an attribute of an item of the workspace). The information corresponding to the workspace may be determined based at least in part on the sensor data obtained with respect to the workspace. As used herein, a workspace may include a kitting shelf system, a conveyance structure and/or receptacle on which items/objects are placed (e.g., in connection with assembling a kit), and/or a robotic system (e.g., one or more robotic arms that pick one or more items from the chute (or other source) and place the one or more items/objects each in a corresponding location on the conveyance structure/receptacle). The workspace may include a control computer that obtains sensor data associated with the workspace, and/or an on-demand teleoperation device that a human operator can use to control an element within the workspace such as the robotic arm and/or the conveyance structure. In some embodiments, the control computer determines the plan.

In various embodiments, an integrated kitting system as disclosed herein operates in an automated manner unless/until the system gets stuck and has no strategy available to continue automated operation. In some embodiments, in response to entering such a state the system requests human intervention, e.g., by manual assistance, teleoperation, etc.

In various embodiments, a robotic arm and/or a static, rail- or track-mounted, or mobile robot comprising a robotic arm or other appendage capable of retrieving items and placing the items in a destination, such as a box or other packaging, is integrated with one or more KSMs as disclosed herein such as a kitting shelf system. The robotic arm retrieves items, as applicable, from the associated KSM(s) and places them in a box or other packaging for shipment. As used herein, the terms slot, tray, or box, and/or receptacle may be used interchangeably in connection with describing a particular location on the conveyor.

Each item or object within an item on the kitting shelf may have machine readable information, such as text and/or optically or otherwise encoded information, which can be machine read and used in connection with kitting the object and/or item, e.g., via an automated kitting system and/or processing. As an example, to read the information for a given item (or object within the item), one or more sensors may obtain information pertaining to the item while the item is within the kitting shelf system (e.g., on a shelf of the kitting shelf system such as on a presentation face of the shelf, etc.). As another example, to read the information for a given item (or object within the item), one or more sensors may obtain information pertaining to the item while the item is being moved (by the robotic arm) from the kitting shelf system to the corresponding receptacle (e.g., the information pertaining to the item is scanned during a path/trajectory of the item from the kitting shelf system to the receptacle).

FIG. 1is a diagram illustrating a kitting system according to various embodiments. In the example shown, kitting system100includes a kitting shelf system102and a robotic arm110. In some embodiments, a kitting system includes a plurality of kitting shelf systems and/or a plurality of robotic arms110. The robotic arm may operate autonomously to pick an item (or an object from within an item) from a kitting shelf system and place the item (or object) to a predetermined location. In some embodiments, a robotic arm picks and places one or more items to a predetermined location based at least in part on a plan such as a plan for kitting the one or more items (e.g., to assemble a kit based on an order etc.).

Kitting shelf system102includes one or more shelves104,106, and108. In some embodiments, the one or more shelves104,106, and108respectively include one or more items disposed thereon. Different shelves within kitting shelf system102may include different items (e.g., a different type of item, items having one or more different identifiers such as serial numbers, model numbers, lot numbers, or the like).

In some embodiments, robotic arm110is movable with respect to kitting shelf system102and/or with respect to a conveyor or other location at which a receptacle is disposed. In the example shown inFIG. 1, robotic arm110is mounted on carriage120, which is configured to ride along a rail or other linear guide122disposed alongside and substantially parallel to the conveyor118, on a side opposite the kitting shelf system102. In various embodiments, a motor, belt, chain, or other source of motive force is applied via a controller (not shown inFIG. 1) to move the carriage120and attached robotic arm110along the rail or guide122to facilitate the automated retrieval of items from one or more kitting shelf systems, and the placement of items (e.g., object114) in receptacle116(e.g., a box, a tray, etc.) as the receptacle116is moved along conveyor118. Control of robotic arm may be coordinated based at least in part on one or more items to be picked and placed in receptacle116, a location of receptacle116, and/or a path of receptacle116(e.g., based on a determined movement of conveyor118).

In some embodiments, kitting system100includes control computer124. In the example shown, operation of kitting shelf system102, conveyor118; and robotic arm110, and/or carriage120are operated in a coordinated manner under the control of control computer124. In the example shown, control computer124is in communication (e.g., wireless communication) with controllers, not shown inFIG. 1, each configured to control operation of a corresponding element comprising kitting system100, e.g., kitting shelf system102; robotic arm110, conveyor118, carriage120, and/or receptacle source (not shown). Although wireless connections are shown inFIG. 1, in various embodiments wired connections or a combination of wired and wireless connections may be used.

In the example shown inFIG. 1, robotic arm110has an end effector corresponding to a two-digit gripper. In various embodiments, robotic arm110includes one or more other and/or different types of end effectors/retrieval tool, including without limitation a gripper having three or more digits; a gripper having digits with different attributes than as shown, e.g., cushioned digits, smaller digits, larger digits, etc.; and/or a retrieval tool that is not a gripper, such as one configured to pick up items using suction, friction, electrostatic force, magnetic force, etc. In some embodiments, the gripper of robotic arm110may be interchanged with one or more different end effectors, depending on one or more attributes of an item to be retrieved, e.g., weight, fragility, compressibility, rigidity, size, shape, etc. In some embodiments, the gripper of robotic arm110may be used to retrieve and use different end effectors (e.g., gripper-held tools) to pick and place items, depending on one or more attributes of the item to be retrieved, for example. One or more attributes of the items may be determined based at least in part on information obtained from one or more sensors such as camera112.

In various embodiments, control computer124is configured, e.g., by software running on control computer124, to receive data associated with an invoice, order, part list, pick list, or other list of items to be retrieved and packed together; determine a strategy/plan to fulfill the retrieve and pack the required items; and operate elements of kitting system100, e.g., kitting shelf system102; conveyor118; and robotic arm110and/or carriage120, in coordination to fulfill the requirement(s). In some embodiments, kitting system100includes a plurality of kitting shelf systems and/or a plurality of robotic arms, and one or more control computers are controlled to coordinate/operate elements of kitting system100.

For example, in some embodiments, control computer124is configured to receive a list of items to be packed. Control computer124determines which items are associated with which of kitting shelf systems (or which items are associated with a particular shelf of a kitting shelf system such as kitting shelf system102) and makes a plan to retrieve and pack the items. In some embodiments, computer124controls a box assembly machine (not shown) or a receptacle source module, and deposit a receptacle on conveyor118and controls the conveyor118to advance the receptacle to a position to be loaded with a first one or more items. The control computer124controls the carriage120and/or robotic arm110as needed to position the robotic arm110to retrieve the first one or more items from the associated a kitting shelf system (or a shelf104,106, or108of kitting shelf system102). Control computer124may control kitting shelf system102, e.g., to ensure the require item(s) in the required quantities are present in the pickup zone (e.g., a presentation face) at the end of kitting shelf system102(or a shelf104,106, and/or108of kitting shelf system102) nearest to the conveyor118and robotic arm110. Control computer124controls robotic arm110to retrieve the item(s) from the corresponding pickup zone(s) and places the item(s) in the receptacle (e.g., receptacle116) before moving on to perform coordinated retrieval and packing of any further items required to be included in that particular kit. In response to a determination that all items have been retrieved and packed (e.g., according to a plan for kitting one or more items), control computer124controls conveyor118to advance the receptacle (e.g., receptacle116) to a next stage of fulfillment, not shown inFIG. 1(e.g., a station at which the box is sealed, labeled, and sent to be shipped).

In the example shown inFIG. 1, kitting shelf system102comprises angled shelves or angled conveyors (e.g., shelves104,106, and108) that are configured to be loaded, e.g., by human workers, robots, and/or other machines, or some combination thereof, from a back end (upper/left as shown inFIG. 1). Items may be scanned, recognized by computer vision, etc. to determine and store on control computer124data associating the item and/or item type or other items attributes associated with each kitting shelf system. In various embodiments a mix of different types of kitting shelf systems may be included in a kitting system such as system100. For example, the items shown inFIG. 1as being supplied via kitting shelf system102may be supplied in some embodiments via a stationary ramp down which the items are rolled. In some embodiments, a kitting shelf system may comprise any one of a plurality of structures and mechanisms to supply items to an associated pick zone, including without limitation a gravity type conveyor having a plurality of adjacent rollers, a ramp, a conveyor belt, a set of revolving bins, etc.

In some embodiments, kitting shelf system102comprises one or more gating mechanisms that control conveyance of items to or within angled shelves or angled conveyors (e.g., shelves104,106, and108). The one or more gating mechanisms may control the conveyance of one or more items to one or more presentation faces of the kitting shelf system in connection with the kitting of the one or more items (e.g., retrieval of the item(s) by a robotic arm and placement of the item(s) into a corresponding receptacle(s)). The one or more gating mechanisms may be controlled by control computer124or the one or more gating mechanisms may be configured to operate mechanically (e.g., without intervention by a control computer). In various embodiments, the one or more gating mechanisms control conveyance of the one or more items based at least in part on a plan (e.g., a predetermined plan for kitting an item based at least in part on an order).

In various embodiments, kitting system100is initialized by having control computer124determine through automated processing, manual configuration, and/or a combination thereof the placement, type, capabilities, etc. of each kitting shelf system (e.g., kitting shelf system102) and the item(s) associated with each kitting shelf system. In addition, elements of kitting system100may register with the control computer124. Registration may include admitting an element, such as each of the kitting shelf system, to a control network. In some embodiments, operational tests may be performed. For example, control computer124may test an ability to control a newly-registered element, such as by operating the conveyor belt of a kitting machine, such as kitting shelf system102, in the forward and back directions, at various speeds, etc.

In various embodiments, elements of kitting system100may be added, removed, swapped out, etc. In such an instance, control computer124initializes and registers the new element, performs operational tests, and begins/resumes kitting operations, incorporating the newly added element, for example.

Referring further toFIG. 1, in the example shown kitting system100includes a camera112(e.g., a video camera) configured to capture images (e.g., video images) of the elements comprised in kiting system100. Camera112may be one of a plurality of sensors that obtains information pertaining to the workspace (e.g., the workspace corresponding to kitting system100). For example, camera112may be one of a plurality of sensors used by control computer124to control the elements comprising kitting system100. For example, in the example shown, video generated by camera112and sent to control computer124may be used by control computer124to control the speed and/or direction of the conveyor belts comprised in the kitting shelf system102and/or a gating mechanism in the kitting shelf system102to ensure a sufficient and not excessive number of items are available in the pickup zone (e.g., a presentation face of kitting shelf system102) and/or to position or reposition the items for retrieval by robotic arm110. In addition, camera112and/or other cameras and/or other sensors may be used to facilitate robotic arm110picking up an item and/or placing the item in its receptacle (e.g., box). In various embodiments, a plurality of cameras may be deployed in a number of locations, including in the environment and on the respective elements comprising system100, to facilitate automated (and, if needed, human assisted) kitting operations. In various embodiments, sensors other than cameras may be deployed, including without limitation contact or limit switches, pressure sensors, weight sensors, and the like.

In various embodiments, one or more sensors are disposed on a chassis of the robotic arm or a chassis of a robotic structure on which the robotic arm is disposed. System100may include one or more accelerometers operatively connected to or disposed within proximity of the one or more sensors (e.g., the one or more cameras). The control computer may obtain the image data pertaining to the workspace based at least in part on information obtained by the one or more accelerometers. In some embodiments, the one or more sensors may comprise a scanner that obtains data pertaining to a label or identifier on the object as the object is moved from the item to the predetermined structure.

In various embodiments, one or more sensors (e.g., camera112) is used to capture information pertaining to items associated with kitting shelf system102. For example, camera112may capture an image of one or more items on shelf104, shelf106, and/or shelf108. As another example, if an item on a shelf104, shelf106, and/or shelf108is a tray or other receptacle, camera112may capture information pertaining to objects within the tray. Control computer124may use information pertaining to the workspace to determine a plan, and/or to control operation of robotic arm110to pick an item (or an object from within the item) from kitting shelf system102. Control computer124may use the information pertaining to the workspace in connection with determining a location of an object within a tray on shelf104, shelf106, and/or shelf108; a quantity of objects within the tray; a type of object within the tray; an orientation of one or more objects within the tray; etc.

In various embodiments, control computer124is programmed to determine a plan to fulfill a kitting requirement based at least in part on a model of the robotic arm110and other elements comprised in kitting system100, e.g., kitting shelf system102; conveyor118; a receptacle source module (e.g., a box assembly machine) (not shown); robotic arm110; and/or carriage120. The respective models in various embodiments reflect capabilities and limitations of each respective element. For example, the shelves104,106, and108of kitting shelf system102are in fixed positions in this example, but each has a conveyor belt which may be capable of being moved in forward and back directions and/or at different speeds. In addition, the control computer124may use information stored in connection with initialization and/or configuration, e.g., which items are on which location(s) on which kitting shelf system (or on which shelf of which kitting shelf system), where each kitting shelf system and/or its associated pick up zone (e.g., presentation face(s)) is located, etc., to determine a plan to fulfill a requirement. In addition, control computer124may use data determine based at least in part on sensor data, such as video captured by camera112, to make a plan to fulfill a requirement.

According to various embodiments, kitting of items from one or more kitting shelf systems is improved through use of a dynamic kitting method or system that determines a path or trajectory for kitting of an item using one or more of an attribute of the item to be singulated and an attribute of another item within the workspace (e.g., an item and/or receptacle on the conveyor). The dynamic kitting method or system may include an updating of the path or trajectory for kitting of an item in response detecting one or more of an attribute of the item to be kitted and an attribute of another item within the workspace (e.g., an item on the conveyor). The dynamic kitting method or system may include an updating of the conveyance of an item from a feeder portion of a kitting shelf system to a presentation face of the kitting shelf system in connection with kitting of an item (or object from an item on the kitting shelf system) in response to determining an attribute of the workspace. The attribute of the workspace used in connection with updating the plan for kitting the item/object may include an attribute of an item/object to be kitted, a quantity of objects within an item (e.g., a tray on a kitting shelf system), a speed of the conveyor, a characteristic associated with a receptacle on the conveyor, an orientation of the item on the kitting shelf system, an orientation of a presentation face of the kitting shelf system, etc. In some embodiments, the kitting system100(e.g., control computer124) dynamically updates the plan for kitting the item during kitting based on a context of the workspace (e.g., a state or condition of the item, a property of the item, another item within the workspace, a speed of conveyor118, an identifier of receptacle116, etc.).

In various embodiments, control computer124is configured to formulate and/or update or reformulate a plan to fulfill a requirement, and to implement or attempt to implement the plan, by employing strategies to do a (next) task or subtask that have been programmed into and/or learned by control computer124. Examples include, without limitation, strategies to use robotic arm110to pick up a given item (or an object from an item on a presentation face of a kitting shelf system) based on attributes of the item (rigidity, fragility, shape, orientation, etc.). In some embodiments, control computer124is programmed to use a first (e.g., preferred or best) strategy to attempt to perform a task (e.g., pick up an item with robotic arm110), and if the first strategy fails then to determine and use an alternate strategy, if one is available (e.g., use the robotic arm110to nudge the item (or object) then try again, operate the conveyor or other instrumentality of the kitting shelf system102(or of a shelf of kitting shelf system), forward and/or backward a bit and try again, etc.). The preferred or best strategy can be determined based at least in part on a model associated with a likelihood of successful picking and placing the object, etc.

In the example shown inFIG. 1, control computer124is connected to an on demand teleoperation device126operated by a human operator128. While inFIG. 1teleoperation device126is operated by a human operator128, in some embodiments teleoperation device126may be operated by a non-human operator, such as a highly skilled robot. In various embodiments, control computer124is configured to invoke on demand teleoperation based at least in part on a determination by control computer124that control computer124does not have an available strategy to continue/complete a kitting operation and/or a component task thereof through fully automated operation. For example, an item is dropped in a location from which the robotic arm110cannot retrieve the item; or, an item has been attempted to be picked up a prescribed maximum number of attempts and has not successfully been retrieved; operation of and end effector deviates from a defined normal operation (e.g., if a suction cup is broken), etc. Based on such a determination, control computer124sends an alert or other communication to on demand teleoperation device126, prompting human operator128to use teleoperation device126to operate one or more elements of kitting system100—e.g., one or more of kitting shelf system102(or shelf104,106, and/or shelf108); conveyor118; a receptacle source module (e.g., a box assembly machine) (not shown); robotic arm110; and/or carriage118—to perform at least the task or subtask the kitting system100was not able to complete under fully automated control by control computer124.

Examples of teleoperation device126include without limitation a tablet or other mobile device having a graphical user interface to control elements of system100, a desktop or other computer having one or more input devices connected thereto to operate elements of system100remotely, etc.

While in some embodiments control computer124invokes on demand teleoperation, in some embodiments a human operator such as operator128may monitor automated operation of kitting system100, e.g. via a video feed generated by camera124, and may intervene to operate elements of system100by teleoperation if the human operator determines there is a need to intervene or assist.

In various embodiments, teleoperation device126and/or human operator128may be located remotely from a physical site at which kitting system100is located and operates (mostly) in a fully automated mode. Similarly, control computer124may be located remotely from the site at which other elements of system100are located and/or a portion of the work described herein as being performed by control computer124may be performed by a computer located remotely from that site.

In various embodiments, control computer124is configured to learn new or improved strategies to employ elements of kitting system100to perform kitting operations. For example, control computer124in some embodiments is programmed to record actions by a human operator (128) via teleoperation (126) and to update its model(s), strategies, etc., as appropriate to be able to repeat and/or emulate the actions of the human operator to perform the task or subtask for which teleoperation was required.

In various embodiments, control computer124is configured to continue to evaluate, during teleoperation, whether it has a viable plan and/or strategy to resume automated fulfillment of the current requirement. If so, control computer may be configured to resume automated operation, either proactively or by advising the human operator that control can be returned to control computer124.

While in the example shown inFIG. 1receptacles (e.g., receptacle116) are moved by conveyor118into successive positions opposite kitting shelf systems or robotic arms, in other embodiments the receptacles may be place in a single, central position, by conveyor118or in some embodiments manually or by a machine other than conveyor118, and may be filled by moving the robotic arm110into one or more positions to access and pick up items from the kitting shelf systems (or shelves of the kitting shelf system, such as shelves104,106, and108), and place the items in the corresponding receptacles (e.g., receptacle116).

While a “kitting” operation is shown inFIG. 1and described herein with reference toFIG. 1and other Figures, in various embodiments kitting systems (and kitting shelf systems) and integrated systems as disclosed herein may be used to perform the reverse operation, e.g., by stocking shelves, bins, and/or kitting machines with items removed from an initially full or partly full box of items. For example, in the example shown inFIG. 1, the receptacle116may include a plurality of items associated with kitting shelf system102, and robotic arm110may be used to remove items from the receptacle116and place the items on the kitting shelf system102, e.g., from the back or supply end as shown. In some embodiments, a robotic arm provides items (e.g., stocks items) to a feeder portion of a shelf or a feeder portion of a kitting shelf system.

In some embodiments, items on a kitting shelf system, such as kitting shelf system102, or on a shelf accessed by, or comprised in, a kitting system as disclosed herein, may be bins or trays that comprise objects that are to be “kitted”. In some embodiments, the system (e.g., kitting system100ofFIG. 1) is configured to detect, e.g., based on computer vision or other sensors and/or techniques, that a bin is empty, and to clear the bin from the kitting system or shelf by using a robotic arm to pick up the bin and place it in a corresponding empty bin location, e.g., a nearby stack of empty bins, etc. In some embodiments, automated bin clearing makes room from a next bin that is not empty to move into position on the kitting machine or shelf to enable a robotic arm to access and pick up items from the bin. In some embodiments, in response to determining that a bin/tray is empty, a robotic arm (e.g., robotic arm110) removes the bin/tray from the corresponding shelf of kitting shelf system102and uses the empty bin/tray as a subsequent receptacle for kitting other items/objects. For example, robotic arm110can pick up an empty bin/tray from a shelf and place the empty bin/tray on conveyor118for use as a receptacle (e.g., receptacle116).

In various embodiments, a robotic system as disclosed herein includes and/or does one or more of the following, e.g., by operation of a control computer such as control computer124:Computer vision information is generated by merging data from multiple sensors, including one or more of 2D cameras, 3D (e.g., RGBD) cameras, infrared, and other sensors to generate a three-dimensional view of a workspace that includes one or more kitting shelf systems (which may correspondingly include one or more shelves of the kitting shelf system). The robotic system determines characteristics of items and/or debris or other abnormalities in the tree-dimensional view of the workspace.Robotic system coordinates operation of multiple robots to avoid collisions, getting in each other's way, and contending to pick up the same item and/or place an item in the same destination location (e.g., receptacle on the conveyor) as another robot. The robotic system coordinates operation of a plurality of robots operating within a same workspace to perform kitting with respect to a plurality of items/object (e.g., to kit items in different receptacles or within a same receptacle). As an example, in various embodiments, the plurality of robots operate independently to pick and place items. As another example, the plurality of robots operate to independently pick and place items for different orders (e.g., to place different sets of items in different receptacles). If a risk of collision is detected, responsive action is taken to ensure the plurality of robots do not collide with one another during singulation.Robotic system coordinates operation of multiple robots to ensure all items are placed in corresponding receptacles. For example, if robot A drops an item system tasks robot B to pick it up; item placed but with improper orientation is picked up and adjusted or moved to another location by same or another robot; two or more items for different orders are placed in a single receptacle results in a downstream robot picking one of the two or more items from the receptacle and placing the item in a new receptacle; etc.Robotic system continuously updates motion planning for each robot and all of the robots together to achieve a desired collective throughput (e.g., to maximize collective throughput, to attain a predefined threshold of collective throughput, etc.). In response to determining that two or more robots have collided or will collide if moving according to their respective plans for kitting items, the robotic system implements an active measure to ensure that the two or more robots avoid collision or otherwise reset independent operation of the two or more robots.In response to a determination that two robots independently are tasked to acquire the same item, the system picks one at random to get that item and the other moves on to the next item (e.g., identify, select, determine grasp strategy, pick, move according to plan, and place).The robotic system can manage the independent operation of a plurality of robots to ensure that the robots select items at different times to avoid the same item being selected for singulation by two different robots.Conveyor movement and/or speed controlled as needed to achieve a desired robot productivity (throughput) and to permit sufficient time for the robot to place an object into the desired receptacle.In response to a determination that an item is misplaced or dropped, the system assigns a robot or, if needed, a human worker to pick the misplaced item up and place the item back in the applicable kitting shelf system (e.g., on a shelf such as via the feeder portion) or, if available or more optimal, on a receptacle on the conveyor.Upstream robots controlled to intentionally leave some receptacles open for downstream robots to place items on the conveyor (e.g., in the corresponding receptacles).Downstream robots controlled to correct errors from an upstream placing an item in a receptacle on the conveyor (e.g., to correct the placement of an item that rests in more than one receptacle such as a tray, or that rests on the conveyor rather than the applicable receptacle, to update a data structure with an association between an identifier for the item or kit/order with the receptacle in which the upstream robot placed the item, etc.).Failure that cannot be corrected by same or another robot results in an alert being communicated to obtain human (or other robotic) intervention to resolve.In response to a determination that a grip strength (e.g., a pressure attained by the end effector) is abnormal (e.g., less than is expected during normal operation), perform a diagnostic process that includes testing the grip strength on a predefined surface and in connection with determining whether remedial action is necessary with respect to the end effector.Move/remove the debris within the workspace, or to reconfigure an item to be kitted (e.g., to improve the likelihood that the item is successfully picked from the shelf, or the presentation face of the shelf, and placed on the conveyor such as in a receptacle on the conveyor).Use sensor data from the workspace environment state system to detect one or more characteristics (e.g., attributes) of the item selected for kitting, determine that grasping or releasing of the item is expected to improve in response to implementation of an active measure, and implement the active measure to improve the grasping or releasing of the item.Use sensor data to determine that the robotic arm has grasped a plurality of items in connection with kitting of one of the items, determine a plan for releasing the plurality of items in order to place each item singly in a corresponding location in one or more receptacles on the conveyor or to place one of the items back on the shelf/presentation face of the kitting shelf system (e.g., determine a strategy for operating the end effector to release a first subset of the plurality of items at a different time from a second subset of the plurality of items).Select a path for kitting an item to a receptacle on the conveyor based on an attribute of the item (e.g., a size of the selected item, a weight of the item, etc.) and/or one or more attributes (e.g., characteristics) of an item within a receptacle on the conveyor.Determine a movement and speed of the robotic arm that picks an item from the kitting shelf system and places the item in the applicable receptacle based at least in part on a speed of a conveyor belt.Determine a trajectory of the item to be kitted based at least in part on one or more of a characteristic of the item, a characteristic of the workspace environment, and/or a characteristic of the conveyor (e.g., a speed of the conveyor belt).Determine a probability of successful kitting corresponding to one or more paths/trajectories of an item to be kitted, and select a path/trajectory along which the item is to be kitted based on the corresponding probability of success.Determine a positioning of a robot arm and/or an end effector of the robot arm to obtain a successful grasp (e.g., as determined based on a probability of grasp success, a type of packaging of the item, a dimension of the item, an expected grip strength in relation to a threshold value, etc.). The positioning of the end effector can include controlling the robot arm or a wrist of the robot arm to move in order for the end effector be orthogonal to a surface of the item.Update the ability of the robotic system to detect an empty receptacle. For example, the definition of an empty receptacle used by the robotic system to identify an empty receptacle is updated over time.

FIG. 2Ais a diagram illustrating a kitting system according to various embodiments.

In the example shown, kitting system200includes a kitting shelf system202and a robotic arm210. According to various embodiments, kitting shelf system202includes one or more shelves. Each of the one or more shelves comprises a presentation face (e.g., presentation face204b, presentation face206b, and/or presentation face208b). The presentation face corresponds to an area or surface on which an item is disposed on the kitting shelf system. In connection with kitting of one or more items, robotic arm picks an item (or object from within an item) from at least one presentation face. Kitting shelf system202may include one or more feeder portions (e.g., feeder portion204a, feeder portion204b, and/or feeder portion204c). In some embodiments, kitting shelf system204includes a gate structure (e.g., gate structure204c, gate structure206c, and/or gate structure208c) that is configured to control conveyance of an item from a feeder portion to the corresponding presentation face. The gate structure may be coupled to, or integrated with, the presentation face.

In some embodiments, robotic arm210is movable with respect to kitting shelf system200and/or with respect to a conveyor or other location at which a receptacle is disposed. In the example shown inFIG. 2A, robotic arm210is mounted on carriage220, which is configured to ride along a rail or other linear guide222disposed alongside and substantially parallel to the conveyor218. As an example, the robotic arm210may be mounted on a side opposite the kitting shelf system200. As an example, the robotic arm210may be mounted on a same side as the kitting shelf system200. In some embodiments, one or more robotic arms are mounted on a same side of conveyor218as a kitting shelf system and one or more robotic arms are mounted on a side of conveyor218that is opposing to kitting shelf system. In various embodiments, a motor, belt, chain, or other source of motive force is applied via a controller (not shown inFIG. 1) to move the carriage210and attached robotic arm210along the rail or guide222to facilitate the automated retrieval of items from one or more kitting shelf systems, and the placement of items (e.g., object214) in receptacle218(e.g., a box, a tray, etc.) as the receptacle216is moved along conveyor218. Control of robotic arm may be coordinated based at least in part on one or more items to be picked and placed in receptacle216, a location of receptacle216(e.g., a predetermined location at which the item is to be kitted), and/or a path of receptacle216(e.g., based on a determined movement of conveyor218).

In some embodiments, kitting system200includes control computer224and/or on operation teleoperation device226. In the example shown, operation of kitting shelf system202, conveyor218; and robotic arm210, and/or carriage220are operated in a coordinated manner under the control of control computer224. In the example shown, control computer224is in communication (e.g., wireless communication) with controllers, not shown inFIG. 2A, each configured to control operation of a corresponding element comprising kitting system200, e.g., kitting shelf system202; robotic arm210, conveyor218, carriage220, and/or receptacle source (not shown). Although wireless connections are shown inFIG. 2A, in various embodiments wired connections or a combination of wired and wireless connections may be used.

In some embodiments, the gate structure controls flow of items from the feeder portion to the presentation face. The gate structure may serve to isolate an item (e.g., a tray from which objects are picked) to facilitate easier removal of the tray upon a determination that the tray is empty. The isolation of the tray by the gate structure allows for the tray to be removed from the corresponding shelf (e.g., the presentation face) without the tray contacting other trays on the shelf (e.g., buffer trays such as the trays on the feeder portion behind the gate). Incidental contact between an empty tray (e.g., during removal of the tray) with buffer trays may cause the robotic arm to drop the empty try, or may cause one or more of the buffer trays to become misaligned or jammed.

According to various embodiments, the gate structure permits or prevents an item to flow from a feeder portion to the presentation face. The control of flow of items from the feeder portion to the presentation face may prevent items from being cluttered on the presentation face and can assist with providing sufficient space and order of items on the presentation face for a robotic arm to pick and place an item/object from the presentation face. Limiting the number of items on a presentation face (e.g., controlling the flow of items) may also improve the sensing or perception of items on the presentation face, and/or prevent an overflowing of items to the presentation face, which could cause an uncontrolled motion of items. The gate structure may be configured to prevent/inhibit more than a threshold of items to be disposed on a presentation face at any particular time. In some embodiments, the gate structure is toggled (e.g., moved) between an open position and a closed position. As an example, the open position may correspond to an orientation of the gate structure when no item is present on the presentation face. As another example, the open position may correspond to an orientation of the gate structure when a quantity of items on the presentation of items is less than a threshold number of items. As another example, the open position may correspond to an orientation of the gate structure when a weight on (or force applied to) to the presentation face is less than a threshold weight (or threshold force). When the gate structure is oriented in the open position, a flow or conveyance of an item from the feeder portion to the presentation face may be permitted (e.g., unblocked). Conversely, when the gate structure is oriented in the closed position, a flow or conveyance of an item from the feeder portion to the presentation face may be prevented (e.g., the flow of items is blocked). In the example illustrated inFIG. 2A, a portion of the presentation faces204b,206b, and/or208bis used to block conveyance of items from the corresponding feeder portions204a,206a, and/or208a; the corresponding gate portions204c,206c, and/or208cmay be hinges operatively coupled to presentation faces204b,206b, and/or208b. In some embodiments, the gate structure is configured to move an element that is distinct from the corresponding presentation face in with moving to an open position and/or closed portion.

In some embodiments, the gate structure is mechanically operated (e.g., to move between the open position and the closed position) based at least in part on a weight (e.g., a weight of an item, if any) applied to the corresponding presentation face. The gate structure may include a biasing element that biases the gate structure to be oriented in the open position. If a force applied to the presentation face (e.g., corresponding to the weight of the item on the presentation face) exceeds the biasing of the gate structure, the gate structure may operate to orient in the closed position. In some embodiments, the gate structure is electromechanically operated. For example, the gate structure may receive a signal and based at least in part on the signal the gate structure may operate in a closed position or an open position. The signal may be provided by a computer, such as control computer224. The signal to control the gate structure may be determined/provided based at least in part on information obtained by one or more sensors in the workspace. For example, image data obtained by camera212of kitting system200may be used in connection with determining whether to gate structure is to be oriented in the open position or closed position (e.g., based at least in part on a determination of whether an item is disposed on the corresponding presentation face). Information obtained by one or more other sensors may be used in connection with determining whether to control the gate structure is to be oriented in the open position or closed position. Examples of sensors that may be implemented to obtain such information include a weight sensor, a force sensor, a torque sensor, a pressure sensor, an infrared sensor, etc. The information used in connection with determining whether to control the gate structure to be oriented in the open position or closed position may be used to determine whether an item is on the corresponding presentation face, to determine whether an item(s) on the presentation face exceed a threshold value (e.g., a threshold weight, a threshold force), etc. As an example, a model of information obtained by the one or more sensors may be generated. The model may determine/define information that is indicative of whether an item is present on the presentation face, or whether a quantity of items present on the presentation face is less than a threshold quantity of items, etc. The control computer224may use the model to determine whether to control the gate structure to be oriented in the open position or the closed position. For example, in response to determining that an item is present on the presentation face, the control computer224may send a signal to the gate structure to cause the gate structure to move to a closed position (e.g., thereby preventing conveyance of further items to the presentation face).

As illustrated inFIG. 2A, a presentation face (e.g., presentation face204b, presentation face206b, and/or presentation face208b) is movable. For example, the presentation face toggle between an empty position (denoted by204d,206d, and208d) and an occupied position (denoted by204b,206b, and208b). Although the example illustrated inFIG. 2Aillustrates the presentation face toggling between two positions/orientations, the kitting shelf system202may be configured for the presentation face to be moved to orient in one of a plurality of positions/orientations. For example, the presentation face may be oriented in any one of the plurality of positions/orientations based at least in part on a context of the workspace (e.g., an item to be provided on the presentation face, a location of a robotic arm to pick the item, a size of the robotic arm, a location of a sensor or camera, etc.).

In various embodiments, the empty position corresponds to a position at which the presentation face is oriented to facilitate a flow/conveyance of one or more items from the feeder portion to the presentation face. For example, the empty position may correspond to a position at which the presentation face is oriented when a quantity of items on the presentation face is less than a threshold number. As another example, the empty position corresponds to a position at which the presentation face is oriented when the presentation face is empty (e.g., does not have any items disposed thereon). As another example, the empty position corresponds to a position at which the presentation face is oriented when a weight on (or force applied to) the presentation is less than a threshold weight (or threshold force). In some embodiments, each presentation face in a particular kitting shelf system (e.g., kitting shelf system202) may have a same angle of orientation (e.g., such as an angle relative to a normal to the ground) when orientated in corresponding empty positions. In some embodiments, two or more presentation faces in a particular kitting shelf system (e.g., kitting shelf system202) have different angles of orientation (e.g., such as an angle relative to a normal to the ground) when orientated in corresponding empty positions. As an example, a first set of one or more presentation faces has an angle of orientation that is different from an angle of orientation of a second set of one or more presentation faces. As another example, each presentation face within a particular kitting shelf system has a different angle of orientation (e.g., such as an angle relative to a normal to the ground) when orientated in a corresponding empty position. The shelves (e.g., at least one of the corresponding feeder portion and the kitting portion) on a kitting shelf system can be configured at different angles based on a height of the shelf relative to the ground. The configuring of shelves at different angles may permit a better line of sight for sensors in the workspace, such as camera212, and such a configuration may improve the information pertaining to the workspace (e.g., identifiers on the items may be obtained more easily or accurately, etc.).

In some embodiments, a bottom shelf on which items are presented (e.g., a shelf from which an item may be obtained such as the shelf corresponding to feeder portion208a) has a corresponding presentation face208bthat has less of a range of motion than presentation faces of higher shelves such as presentation face204band206b. The presentation face on such a bottom shelf may be fixed relative to the feeder portion. For example, the presentation face on such a bottom shelf may be integrated with the feeder portion or not otherwise change positions relative to the feeder portion. Such a bottom shelf or the presentation face of such bottom shelf (e.g., presentation face208b) may have a shallower pitch/angle than one or more higher shelves or of the presentation faces for the one or more higher shelves (e.g., presentation face204band/or206b). In some embodiments, shelves (e.g., presentation faces of such shelves such as when an item is on the presentation face) has a progressively shallower pitch or angle the lower the shelf is on the kitting shelf system such as kitting shelf system200.

In various embodiments, the occupied position corresponds to a position at which the presentation face is oriented to facilitate picking of one or more items (e.g., by the robotic arm) from the presentation face. The presentation face may be oriented in the occupied position when conveyance/flow of one or more items from the feeder portion to the presentation face is to be prevented. In some embodiments, each presentation face in a particular kitting shelf system (e.g., kitting shelf system202) may have a same angle of orientation (e.g., such as an angle relative to a normal to the ground) when orientated in corresponding empty positions. In some embodiments, two or more presentation faces in a particular kitting shelf system have different angles of orientation (e.g., such as an angle relative to a normal to the ground) when orientated in corresponding occupied positions. As an example, a first set of one or more presentation faces has an angle of orientation that is different from an angle of orientation of a second set of one or more presentation faces. As another example, each presentation face within a particular kitting shelf system has a different angle of orientation (e.g., such as an angle relative to a normal to the ground) when orientated in corresponding occupied positions. As illustrated inFIG. 2A, the shelves (e.g., at least one of the corresponding feeder portion and the kitting portion) on a kitting shelf system can be configured at different angles based on a height of the shelf relative to the ground. The configuring of shelves at different angles may permit a better line of sight for sensors in the workspace, such as camera212, and such a configuration may improve the information pertaining to the workspace (e.g., identifiers on the items may be obtained more easily or accurately, etc.).

In some embodiments, the orientation of the presentation face may have a greater tilt downwards the lower the corresponding shelf is to the ground. Such an orientation of the presentation face may enhance one or more sensors (e.g., camera212) to obtain information pertaining to the presentation shelf or one or more items/objects on the presentation face. In addition, such an orientation may enhance the ability of the robotic arm to engage an item with the end effector of the robotic arm. A robotic arm may have limitations with respect to its wrist extension capabilities/configurations and/or its wrist flexion capabilities/configurations. As an example, the orientation of the presentation face (e.g., at least in the occupied position) is configured based at least in part on an extent of wrist extension that is required by a robotic arm to pick the item/object from the presentation face). The shelf/presentation face may be configured based at least in part on the range of motion of a wrist of a robotic arm in the kitting shelf system (e.g., a range of motion with respect to wrist extension/flexion). An end effector or wrist component of the robotic arm may have size limitations that inhibit the ability of the robotic arm to engage (e.g., at certain angles and heights/locations) an item disposed on a presentation face. Accordingly, the orientation of the presentation face (e.g., at least in an occupied position) of a shelf may be configured to improve the likelihood/ability of the robotic arm to configure its position to engage an item/object on the presentation face with an end effector of the robotic arm at an orthogonal relative to the item/object. The orientation of the presentation face when in the occupied position can correspond to an orientation in which a tray/item disposed thereon is optimally angled (at each level/shelf) for a better vision from an onboard camera (e.g., a camera disposed in the workspace and/or on a robotic arm or chassis thereof). In some embodiments the orientation of the presentation face in the empty position and/or occupied position is based at least in part on a configuration of the corresponding gate structure. For example, if the gate structure is a hinge, the orientation of the presentation face in the empty position and/or occupied position is based at least in part on a range of motion of the hinge.

According to various embodiments, kitting shelf system202includes one or more feeder portions (e.g.,204a,206a, and/or208a). In some embodiments, the kitting shelf system202may have a single feeder portion that conveys one or more items to a plurality of presentation faces. In other embodiments, as illustrated inFIG. 2A, the kitting shelf system202has a single feeder portion for each presentation face (e.g., a one-to-one mapping of feeder portions to presentation faces). The feeder portion may be configured to convey an item to a presentation face. As an example, the conveyance of the item may be passive such as via gravity acting on an item disposed on the feeder portion (e.g., in the case that the feeder portion is configured to tilt towards the presentation face). As another example, the conveyance of the item may be at least partially active based on the feeder portion being configured with a conveyor that carries an item from an input location of the feeder portion to the presentation face. In various embodiments, the feeder portion configured to receive a succession of items on a receiving end (e.g., an input to the feeder portion) and convey the items to a destination end (e.g., an end that is operatively connected/coupled to a presentation face or that otherwise exits the feeder portion to the applicable presentation face). The succession of items may be manually loaded to the feeder portion or kitting shelf system (e.g., via a human operator228), or the succession of items may be automatically loaded to the feeder portion (e.g., via a robotic arm/component, or based at least in part on the feeder portion being coupled to a chute that conveys items from a source flow/pile).

FIG. 2Bis a diagram illustrating a kitting system according to various embodiments. As illustrated inFIG. 2B, kitting shelf system202can be further configured with return portion209. According to various embodiments, an item can returned to a predetermined location via return portion209. The item may be returned in response to a determination that the item is empty (e.g., all objects within item have been kitted to corresponding receptacles); in response to a determination that the item is defective; in response to a determination that the item was not in a correct location (e.g., the item was improperly conveyed to the presentation face such as based at least in part on a type of objects within the item, etc.); etc.

In various embodiments, the return portion209is configured to convey an item (e.g., item230inFIG. 2B) to a predetermined location. The conveyance of the item on the return portion209may be passive such as via gravity acting on an item disposed on the return portion209(e.g., in the case that the return portion is configured to tilt towards the predetermined location). As another example, the conveyance of the item230may be at least partially active based on the feeder portion being configured with a conveyor that carries an item from an input location of the return portion to the predetermined location. In various embodiments, the feeder portion configured to receive a succession of items on a receiving end (e.g., an input to the return portion) and convey the items to a destination end (e.g., an end that is operatively connected/coupled to the predetermined location or that otherwise exits the return portion). The succession of items (to be returned) may be manually loaded to the return portion (e.g., by human operator228), or the succession of items may be automatically loaded to the feeder portion (e.g., via a robotic arm/component).

According to various embodiments, the predetermined location to which items (e.g., item230) are returned corresponds to a system in which the items are recycled or re-populated with objects so that the items may be recirculated into the kitting system (e.g., reloaded to a feeder portion of kitting shelf system202).

FIG. 3Ais a diagram illustrating a shelf system according to various embodiments.FIG. 3Bis a diagram illustrating a shelf system according to various embodiments. According to various embodiments, shelf system300may be implemented by kitting system100ofFIG. 1, kitting system200ofFIGS. 2A and 2B, kitting system600ofFIGS. 6A and 6B, and kitting system700ofFIG. 7.

Shelf system300is similar to the corresponding shelf system illustrated inFIGS. 2A and 2B. For example, as illustrated inFIGS. 3A and 3Billustrate presentation face304oriented in an empty position and an occupied position. Presentation face304may rotate around an axis such as an axis defined by gate structure306. In the example illustrated inFIG. 3A, presentation face304is oriented in an empty position (as denoted by310inFIG. 3B). Conversely, in the example illustrated inFIG. 3B, presentation face304is oriented in an occupied position (e.g., an item is disposed on presentation face304).

In various embodiments, when presentation face is in the occupied position, the extent of a wrist flexion required to engage an end effector with the item on the presentation face (in manner that the end effector engages the item orthogonal a surface of the item) is less than a wrist flexion required if the item were disposed on the presentation face when the presentation is oriented in the empty position310. In some embodiments, a robotic arm uses a wrist extension movement to engage the item when presentation face is oriented in the occupied position; and the robotic arm would be required to use a wrist flexion movement to engage the item when presentation face is oriented in the empty position.

As illustrated inFIGS. 3A and 3B, items are conveyed from feeder portion302to presentation face304(e.g., when gate structure306is in an open position). When an item is on the presentation face304, the gate structure306moves to a closed position according to which conveyance of the items on the feeder portion302is blocked (e.g., by a part of presentation face304). The gate structure306may be biased to return presentation face304to the empty position in response to the item being removed from presentation face304.

FIG. 4Ais a diagram illustrating a shelf system according to various embodiments.FIG. 4Bis a diagram illustrating a shelf system according to various embodiments. According to various embodiments, shelf system400may be implemented by kitting system100ofFIG. 1, kitting system200ofFIGS. 2A and 2B, kitting system600ofFIGS. 6A and 6B, and kitting system700ofFIG. 7.

As illustrated inFIGS. 4A and 4Billustrate presentation face404oriented in an empty position and an occupied position (e.g., an item is disposed on presentation face404inFIG. 4B). The gate structure406may integrated with presentation face404. For example, as illustrated inFIGS. 4A and 4B, presentation face404rotates around an axis such that when the presentation face404has an item disposed thereon, gate structure406is moved to block conveyance/flow of items from feeder portion402to presentation face404. The feeder portion402may be configured to have a space/gap through which the gate structure406is moved to retract/extend based on whether the gate structure406is in a closed position or an open position.

Presentation face404may rotate around an axis based at least in part on whether an item is disposed on the presentation face404. In the example illustrated inFIG. 4A, presentation face404is oriented in an empty position. Conversely, in the example illustrated inFIG. 4B, presentation face404is oriented in an occupied position (e.g., an item is disposed on presentation face404).

In various embodiments, when presentation face is in the occupied position, the extent of a wrist flexion required to engage an end effector with the item on the presentation face (in manner that the end effector engages the item orthogonal a surface of the item) is less than a wrist flexion required if the item were disposed on the presentation face when the presentation is oriented in the empty position. In some embodiments, a robotic arm uses a wrist extension movement to engage the item when presentation face is oriented in the occupied position; and the robotic arm would be required to use a wrist flexion movement to engage the item when presentation face is oriented in the empty position.

As illustrated inFIGS. 4A and 4B, items are conveyed from feeder portion402to presentation face404(e.g., when gate structure406is in an open position). When an item is on the presentation face404, the gate structure406moves to a closed position according to which conveyance of the items on the feeder portion402is blocked (e.g., by a part of gate structure406extending through a gap/space in the feeder portion).

In some embodiments, the gate structure406is a counterweight that is sufficient to return the gate structure to an open position when the weight on the presentation face404is below a threshold weight (e.g., when an item is removed from the presentation face404). In response to the item being removed from the presentation face404, the weight of the gate structure rotates to an open position and the presentation face406may be positioned in a manner for items to convey from feeder portion402to presentation face404. In addition, or alternatively, the gate structure406and/or the presentation face404may be biased to return presentation face404to the empty position in response to the item being removed from presentation face404. In some embodiments, the presentation face is biased by a biasing mechanism such as a torsional spring or the like. The biasing mechanism may bias the presentation face404to return presentation face404to the empty position in response to the item being removed from presentation face404. The presentation face404may pivot around a defined point or axis. For example, shelf system400may include fulcrum408or the like such as a hinge, etc.

In some embodiments, the feeder portion402comprises a roller conveyor comprising one or more rollers that support and/or help guide items from the feeder portion402to the presentation face406. As an example, the conveyance of the item (e.g., along the feeder portion and/or from the feeder portion to the presentation face) may be passive such as via gravity acting on an item disposed on the feeder portion (e.g., in the case that the feeder portion is configured to tilt towards the presentation face). As another example, the conveyance of the item may be at least partially active based on the feeder portion being configured with a conveyor that is driven by a motor or the like and that carries an item from an input location of the feeder portion to the presentation face. In various embodiments, the feeder portion configured to receive a succession of items on a receiving end (e.g., an input to the feeder portion) and convey the items to a destination end (e.g., an end that is operatively connected/coupled to a presentation face or that otherwise exits the feeder portion to the applicable presentation face). The succession of items may be manually loaded to the feeder portion402or kitting shelf system (e.g., via a human operator), or the succession of items may be automatically loaded to the feeder portion (e.g., via a robotic arm/component, or based at least in part on the feeder portion being coupled to a chute that conveys items from a source flow/pile).

FIG. 5Ais a diagram illustrating a shelf system according to various embodiments.FIG. 5Bis a diagram illustrating a shelf system according to various embodiments. According to various embodiments, shelf system500may be implemented by kitting system100ofFIG. 1, kitting system200ofFIGS. 2A and 2B, kitting system600ofFIGS. 6A and 6B, and kitting system700ofFIG. 7.

As illustrated inFIGS. 5A and 5Billustrate presentation face504oriented in an empty position and an occupied position (e.g., an item is disposed on presentation face504inFIG. 5B). The gate structure506may integrated with presentation face504. For example, as illustrated inFIGS. 5A and 5B, presentation face504rotates around an axis such that when the presentation face404has an item disposed thereon, gate structure506is moved to block conveyance/flow of items from feeder portion502to presentation face504. In some embodiments, presentation face504rotate around an axis defined by fulcrum508or the like such as a hinge, etc. The feeder portion502may be configured to have a space/gap through which the gate structure506is moved to retract/extend based on whether the gate structure506is in a closed position or an open position.

Presentation face504may rotate around an axis based at least in part on whether an item is disposed on the presentation face504. In the example illustrated inFIG. 5A, presentation face504is oriented in an empty position. Conversely, in the example illustrated inFIG. 4B, presentation face504is oriented in an occupied position (e.g., an item is disposed on presentation face404). In response to an item being moved to the presentation face504, presentation face504rotates around fulcrum508(e.g., based at least in part on the weight of the item that has moved to presentation face504).

In various embodiments, when presentation face is in the occupied position, the extent of a wrist flexion required to engage an end effector with the item on the presentation face (in manner that the end effector engages the item orthogonal a surface of the item) is less than a wrist flexion required if the item were disposed on the presentation face when the presentation is oriented in the empty position. In some embodiments, a robotic arm uses a wrist extension movement to engage the item when presentation face is oriented in the occupied position; and the robotic arm would be required to use a wrist flexion movement to engage the item when presentation face is oriented in the empty position.

As illustrated inFIGS. 5A and 5B, items are conveyed from feeder portion502to presentation face504(e.g., when gate structure506is in an open position). When an item is on the presentation face504, the gate structure506moves to a closed position according to which conveyance of the items on the feeder portion502is blocked (e.g., by a part of gate structure506extending through a gap/space in the feeder portion). The gate structure506and/or the presentation face504may be biased to return presentation face504to the empty position in response to the item being removed from presentation face504.

In some embodiments, the feeder portion502comprises a roller conveyor comprising one or more rollers that support and/or help guide items from the feeder portion502to the presentation face506. As an example, the conveyance of the item (e.g., along the feeder portion and/or from the feeder portion to the presentation face) may be passive such as via gravity acting on an item disposed on the feeder portion (e.g., in the case that the feeder portion is configured to tilt towards the presentation face). As another example, the conveyance of the item may be at least partially active based on the feeder portion being configured with a conveyor that is driven by a motor or the like and that carries an item from an input location of the feeder portion to the presentation face. In various embodiments, the feeder portion configured to receive a succession of items on a receiving end (e.g., an input to the feeder portion) and convey the items to a destination end (e.g., an end that is operatively connected/coupled to a presentation face or that otherwise exits the feeder portion to the applicable presentation face). The succession of items may be manually loaded to the feeder portion402or kitting shelf system (e.g., via a human operator), or the succession of items may be automatically loaded to the feeder portion (e.g., via a robotic arm/component, or based at least in part on the feeder portion being coupled to a chute that conveys items from a source flow/pile).

In various embodiments, gate structure506is biased to be in an open position and/or presentation face502is biased to an orientation corresponding to the empty position. For example, shelf system500includes one or more of first biasing element510. The first biasing element510may be a spring that exerts a force on the presentation face504(e.g., to bias presentation face504to rotate around the axis defined by fulcrum508and to move the presentation face504to the empty position). In some embodiments, shelf system500includes second biasing element512. As an example, second biasing element512may bias one or more of the gate structure506to the closed position and/or the presentation face504to the occupied position. According to various embodiments, the biasing force exerted by first biasing element510may be greater than the biasing force exerted by the second biasing element512such that the shelf system is biased for gate structure506is biased to be in an open position and/or for presentation face502is biased to an orientation corresponding to the empty position. When an item is moved to be disposed on the presentation face504, the combination of the weight of the item and the biasing force of second biasing element512may be greater than a counter-acting biasing force of the first biasing element510. Accordingly, when an item is disposed on the presentation face504, the gate structure506is moved to the closed position and/or the presentation face504is moved to be oriented in the occupied position. Although the example illustrated inFIGS. 5A and 5Billustrate the first biasing element510and the second biasing element512as being springs, various elements for exerting a bias (e.g., a biasing force) may be implemented. The biasing elements may be a spring mechanism, a pneumatic piston, and the like.

According to various embodiments, movement of the presentation face504and/or gate structure506is dampened by a dampening element. As an example, the dampening element may include one or more first biasing element510and second biasing element512. The movement may be dampened to prevent/inhibit dislodging of an item from presentation face504(e.g., as the presentation face504moves to the occupied position), and/or from the feeder portion502. Various dampening elements may be implemented in the shelf system in connection with dampening movement of the presentation face504and/or gate structure506is dampened by a dampening element.

FIG. 6Ais a diagram illustrating a kitting system according to various embodiments. According to various embodiments, kitting system100may implement shelf system300ofFIGS. 3A and 3B, shelf system400ofFIGS. 4A and 4B, and/or shelf system500ofFIGS. 5A and 5B.

As illustrated inFIG. 6A, in various embodiments a kitting system600includes one or more kitting shelf systems (e.g., kitting shelf system602, kitting shelf system604, and/or kitting shelf system606). In the case of kitting system600including a plurality of kitting shelf systems, the kitting shelf systems may be disposed along one side of a conveyor (e.g., conveyor618), or the kitting shelf systems may be disposed on different sides of the conveyor.

In various embodiments, a kitting system600includes one or more robotic arms (e.g., robotic arm610). In the case of kitting system600including a plurality of robotic arms, the robotic arms may be disposed along one side of a conveyor (e.g., conveyor618), or the robotic arms may be disposed on different sides of the conveyor. Further, a robotic arm may be disposed on a same side of the conveyor as one or more kitting shelf systems, or the robotic arm may be disposed on a side of the conveyor that is opposite to the one or more kitting shelf systems. In various embodiments, in the case of kitting system600including a plurality of robotic arms, a robotic arm is selected from among the plurality of robotic arms to obtain an item (or object from within an item) from a kitting shelf system based on a location of the robotic arm, an effectiveness of the particular robotic arm successfully picking the item (e.g., a computed likelihood of success), a reach of the particular robotic arm, a time within which the particular robotic arm will complete the picking and placing of the item, etc. The particular robotic arm selected to pick and place the item may be selected by control computer624. For example, control computer624may store a model with respect to the workspace of kitting system600and/or one or more of the robotic arms within the workspace. The control computer may use the model to determine a robotic arm to pick and place an item, and/or a robotic arm to pick and place each of one or more items for a particular order (e.g., a single robotic arm may be used to kit the items for an order). In some embodiments, a robotic arm that is disposed on a same side as a kitting shelf system is more effective/efficient at picking and placing items from shelves on the kitting shelf system that are above a threshold height. For shelves below the threshold height, the contortion of the robotic arm required to reach an item on such shelves renders the robotic arm on the same side of the conveyor as the particular kitting shelf system inefficient. Further, the contortion of the robotic arm required to reach an item on such shelves renders the robotic arm with limited range/mobility with which to pick an item from a shelf below the height threshold. Accordingly, in cases of items on shelves below a threshold height, kitting system600may use a robotic arm610on a side of the conveyor618that is opposite to the kitting shelf system from which the item is to be obtained.

Kitting shelf system602includes one or more shelves. In some embodiments, the one or more shelves respectively include one or more items disposed thereon. Different shelves within kitting shelf system602may include different items (e.g., a different type of item, items having one or more different identifiers such as serial numbers, model numbers, lot numbers, or the like). One or more of the kitting shelf system(s) may have a feeder portion, a gate structure, and a presentation face similar to those disclosed with respect to shelf system300ofFIGS. 3A and 3B, shelf system400ofFIGS. 4A and 4B, and/or shelf system500ofFIGS. 5A and 5B. As illustrated inFIG. 6A, the kitting shelf system comprises a shelf having a presentation face608a, a gate structure608b, and a feeder portion608c.

In some embodiments, robotic arm610is movable with respect to one or more of kitting shelf systems602,604, and606and/or with respect to a conveyor (e.g., conveyor618) or other location at which a receptacle is disposed. In the example shown inFIG. 6A, robotic arm610is mounted on carriage620, which is configured to ride along a rail or other linear guide622disposed alongside and substantially parallel to the conveyor618, on a side opposite the kitting shelf system602. In various embodiments, a motor, belt, chain, or other source of motive force is applied via a controller (not shown inFIG. 6A) to move the carriage620and attached robotic arm610along the rail or guide622to facilitate the automated retrieval of items from one or more kitting shelf systems, and the placement of items in receptacle616a(e.g., a box, a tray, etc.) as the receptacle616ais moved along conveyor618. Control of robotic arm may be coordinated based at least in part on one or more items to be picked and placed in receptacle616a, a location of receptacle616a, and/or a path of receptacle616a(e.g., based on a determined movement and/or speed of conveyor618).

In some embodiments, kitting system600includes control computer624. In the example shown, operation of kitting shelf system602, conveyor618; and robotic arm610, and/or carriage620are operated in a coordinated manner under the control of control computer624. In the example shown, control computer624is in communication (e.g., wireless communication) with controllers, not shown inFIG. 6A, each configured to control operation of a corresponding element comprising kitting system600, e.g., one or more of kitting shelf systems602,604, and/or606; robotic arm610, conveyor618, carriage610, and/or receptacle source (not shown). Although wireless connections are shown inFIG. 6A, in various embodiments wired connections or a combination of wired and wireless connections may be used.

In the example shown inFIG. 6A, robotic arm610has an end effector corresponding to a two-digit gripper. In various embodiments, robotic arm610includes one or more other and/or different types of end effectors/retrieval tool, including without limitation a gripper having three or more digits; a gripper having digits with different attributes than as shown, e.g., cushioned digits, smaller digits, larger digits, etc.; and/or a retrieval tool that is not a gripper, such as one configured to pick up items using suction, friction, electrostatic force, magnetic force, etc. In some embodiments, the gripper of robotic arm610may be interchanged with one or more different end effectors, depending on one or more attributes of an item to be retrieved, e.g., weight, fragility, compressibility, rigidity, size, shape, etc. In some embodiments, the gripper of robotic arm610may be used to retrieve and use different end effectors (e.g., gripper-held tools) to pick and place items, depending on one or more attributes of the item to be retrieved, for example. One or more attributes of the items may be determined based at least in part on information obtained from one or more sensors such as camera612.

In various embodiments, control computer624is configured, e.g., by software running on control computer624, to receive data associated with an invoice, order, part list, pick list, or other list of items to be retrieved and packed together; determine a strategy/plan to fulfill the retrieve and pack the required items; and operate elements of kitting system600, e.g., kitting shelf system602,604, and/or606; conveyor618; and robotic arm610and/or carriage620, in coordination to fulfill the requirement(s). In some embodiments, kitting system600includes a plurality of kitting shelf systems and/or a plurality of robotic arms, and one or more control computers are controlled to coordinate/operate elements of kitting system600.

For example, in some embodiments, control computer624is configured to receive a list of items to be packed. Control computer624determines which items are associated with which of kitting shelf systems (or which items are associated with a particular shelf of a kitting shelf system such as kitting shelf system602) and makes a plan to retrieve and pack the items. In some embodiments, computer624controls a box assembly machine (not shown) or a receptacle source module, and deposit a receptacle on conveyor618and controls the conveyor618to advance the receptacle to a position to be loaded with a first one or more items. The control computer624controls the carriage620and/or robotic arm610as needed to position the robotic arm610to retrieve the first one or more items from the associated a kitting shelf system (or a shelf of kitting shelf system602). Control computer624may control kitting shelf system602,604, and/or606, e.g., to ensure the require item(s) in the required quantities are present in the pickup zone (e.g., a presentation face) at the end of kitting shelf system602,604, and/or606(or a shelf of the kitting shelf system) nearest to the conveyor618and robotic arm610. Control computer624controls robotic arm610to retrieve the item(s) from the corresponding pickup zone(s) and places the item(s) in the receptacle (e.g., receptacle616a) before moving on to perform coordinated retrieval and packing of any further items required to be included in that particular kit. In response to a determination that all items have been retrieved and packed (e.g., according to a plan for kitting one or more items), control computer624controls conveyor618to advance the receptacle (e.g., receptacle616a) to a next stage of fulfillment, not shown inFIG. 6A(e.g., a station at which the box is sealed, labeled, and sent to be shipped).

In the example shown inFIG. 6A, kitting shelf system602comprises angled shelves or angled conveyors (e.g., shelf comprising presentation face608a, gate structure608b, and feeder portion608c) that are configured to be loaded, e.g., by human workers, robots, and/or other machines, or some combination thereof, from a back end (upper/left as shown inFIG. 6A). The kitting shelf system602may be loaded at the feeder portion or in a loader element (e.g., a chute) that conveys items to the respective feeder portions of the shelves. Items may be scanned, recognized by computer vision, etc. to determine and store on control computer624data associating the item and/or item type or other items attributes associated with each kitting shelf system. In various embodiments a mix of different types of kitting shelf systems may be included in a kitting system such as system600. For example, the items shown inFIG. 6Aas being supplied via kitting shelf system602may be supplied in some embodiments via a stationary ramp down which the items are rolled. In some embodiments, a kitting shelf system may comprise any one of a plurality of structures and mechanisms to supply items to an associated pick zone, including without limitation a gravity type conveyor having a plurality of adjacent rollers, a ramp, a conveyor belt, a set of revolving bins, etc.

In some embodiments, kitting shelf system602comprises one or more gating mechanisms that control conveyance of items to or within angled shelves or angled conveyors (e.g., gate structure608b). The one or more gating mechanisms may control the conveyance of one or more items to one or more presentation faces (e.g., presentation face608a) of the kitting shelf system in connection with the kitting of the one or more items (e.g., retrieval of the item(s) by a robotic arm and placement of the item(s) into a corresponding receptacle(s)). The one or more gating mechanisms may be controlled by control computer624or the one or more gating mechanisms may be configured to operate mechanically (e.g., without intervention by a control computer). In various embodiments, the one or more gating mechanisms control conveyance of the one or more items based at least in part on a plan (e.g., a predetermined plan for kitting an item based at least in part on an order).

In various embodiments, kitting system600is initialized by having control computer624determine through automated processing, manual configuration, and/or a combination thereof the placement, type, capabilities, etc. of each kitting shelf system (e.g., kitting shelf system602,604, and/or606) and the item(s) associated with each kitting shelf system. In addition, elements of kitting system600may register with the control computer624. Registration may include admitting an element, such as each of the kitting shelf system, to a control network. In some embodiments, operational tests may be performed. For example, control computer624may test an ability to control a newly-registered element, such as by operating the conveyor belt of a kitting machine, such as kitting shelf system602, in the forward and back directions, at various speeds, etc.

In various embodiments, elements of kitting system600may be added, removed, swapped out, etc. In such an instance, control computer624initializes and registers the new element, performs operational tests, and begins/resumes kitting operations, incorporating the newly added element, for example.

Referring further toFIG. 6A, in the example shown kitting system600includes a camera612(e.g., a video camera) configured to capture images (e.g., video images) of the elements comprised in kiting system600. Camera612may be one of a plurality of sensors that obtains information pertaining to the workspace (e.g., the workspace corresponding to kitting system600). For example, camera612may be one of a plurality of sensors used by control computer624to control the elements comprising kitting system100. For example, in the example shown, video generated by camera612and sent to control computer624may be used by control computer624to control the speed and/or direction of the conveyor belts comprised in the kitting shelf system102and/or a gating mechanism (e.g.,608b) in the kitting shelf system602to ensure a sufficient and not excessive number of items are available in the pickup zone (e.g., a presentation face608aof kitting shelf system602) and/or to position or reposition the items for retrieval by robotic arm610. In addition, camera612and/or other cameras and/or other sensors may be used to facilitate robotic arm610picking up an item and/or placing the item in its receptacle (e.g., box). In various embodiments, a plurality of cameras may be deployed in a number of locations, including in the environment and on the respective elements comprising system600, to facilitate automated (and, if needed, human assisted) kitting operations. In various embodiments, sensors other than cameras may be deployed, including without limitation contact or limit switches, pressure sensors, weight sensors, and the like.

In various embodiments, one or more sensors (e.g., camera612) is used to capture information pertaining to items associated with kitting shelf system602,605, and/or606. For example, camera612may capture an image of one or more items on a shelf. As another example, if an item on a shelf is a tray or other receptacle, camera612may capture information pertaining to objects within the tray. Control computer624may use information pertaining to the workspace to determine a plan, and/or to control operation of robotic arm610to pick an item (or an object from within the item) from kitting shelf system602. Control computer624may use the information pertaining to the workspace in connection with determining a location of an object within a tray on a shelf; a quantity of objects within the tray; a type of object within the tray; an orientation of one or more objects within the tray; etc.

In various embodiments, control computer624is programmed to determine a plan to fulfill a kitting requirement based at least in part on a model of the robotic arm610and other elements comprised in kitting system600, e.g., kitting shelf system602,604, and/or606; conveyor618; a receptacle source module (e.g., a box assembly machine) (not shown); robotic arm610; and/or carriage620. The respective models in various embodiments reflect capabilities and limitations of each respective element. For example, the shelves of kitting shelf system602,604, and/or606are in fixed positions in this example, but each has a conveyor belt which may be capable of being moved in forward and back directions and/or at different speeds. In addition, the control computer624may use information stored in connection with initialization and/or configuration, e.g., which items are on which location(s) on which kitting shelf system (or on which shelf of which kitting shelf system), where each kitting shelf system and/or its associated pick up zone (e.g., presentation face(s)) is located, etc., to determine a plan to fulfill a requirement. In addition, control computer624may use data determine based at least in part on sensor data, such as video captured by camera612, to make a plan to fulfill a requirement.

According to various embodiments, kitting of items from one or more kitting shelf systems is improved through use of a dynamic kitting method or system that determines a path or trajectory for kitting of an item using one or more of an attribute of the item to be singulated and an attribute of another item within the workspace (e.g., an item and/or receptacle on the conveyor). The dynamic kitting method or system may include an updating of the path or trajectory for kitting of an item in response detecting one or more of an attribute of the item to be kitted and an attribute of another item within the workspace (e.g., an item on the conveyor). The dynamic kitting method or system may include an updating of the conveyance of an item from a feeder portion of a kitting shelf system to a presentation face of the kitting shelf system in connection with kitting of an item (or object from an item on the kitting shelf system) in response to determining an attribute of the workspace. The attribute of the workspace used in connection with updating the plan for kitting the item/object may include an attribute of an item/object to be kitted, a quantity of objects within an item (e.g., a tray on a kitting shelf system), a speed of the conveyor, a characteristic associated with a receptacle on the conveyor, an orientation of the item on the kitting shelf system, an orientation of a presentation face of the kitting shelf system, etc. In some embodiments, the kitting system600(e.g., control computer624) dynamically updates the plan for kitting the item during kitting based on a context of the workspace (e.g., a state or condition of the item, a property of the item, another item within the workspace, a speed of conveyor618, an identifier of receptacle616, etc.).

In various embodiments, control computer624is configured to formulate and/or update or reformulate a plan to fulfill a requirement, and to implement or attempt to implement the plan, by employing strategies to do a (next) task or subtask that have been programmed into and/or learned by control computer624. Examples include, without limitation, strategies to use robotic arm610to pick up a given item (or an object from an item on a presentation face of a kitting shelf system) based on attributes of the item (rigidity, fragility, shape, orientation, etc.). In some embodiments, control computer624is programmed to use a first (e.g., preferred or best) strategy to attempt to perform a task (e.g., pick up an item with robotic arm610), and if the first strategy fails then to determine and use an alternate strategy, if one is available (e.g., use the robotic arm610to nudge the item (or object) then try again, operate the conveyor or other instrumentality of the kitting shelf system602,604, and/or606(or of a shelf of kitting shelf system), forward and/or backward a bit and try again, etc.). The preferred or best strategy can be determined based at least in part on a model associated with a likelihood of successful picking and placing the object, etc.

In the example shown inFIG. 6A, control computer624is connected to an on demand teleoperation device626operated by a human operator628. While inFIG. 6Ateleoperation device626is operated by a human operator128, in some embodiments teleoperation device626may be operated by a non-human operator, such as a highly skilled robot. In various embodiments, control computer624is configured to invoke on demand teleoperation based at least in part on a determination by control computer624that control computer624does not have an available strategy to continue/complete a kitting operation and/or a component task thereof through fully automated operation. For example, an item is dropped in a location from which the robotic arm610cannot retrieve the item; or, an item has been attempted to be picked up a prescribed maximum number of attempts and has not successfully been retrieved; operation of and end effector deviates from a defined normal operation (e.g., if a suction cup is broken), etc. Based on such a determination, control computer624sends an alert or other communication to on demand teleoperation device626, prompting human operator628to use teleoperation device626to operate one or more elements of kitting system600—e.g., one or more of kitting shelf system602,604, and/or606(or shelf); conveyor618; a receptacle source module (e.g., a box assembly machine) (not shown); robotic arm610; and/or carriage618—to perform at least the task or subtask the kitting system600was not able to complete under fully automated control by control computer624.

FIG. 6Bis a diagram illustrating a kitting system according to various embodiments. As illustrated inFIG. 6B, kitting system600can be further configured with a receptacle source such as box assembly machine630. According to various embodiments, the receptacle source places an empty receptacle onto conveyor618. The receptacle source may obtain the empty receptacle from an inventory of empty receptacles. In some implementations, the receptacle source assembles the empty receptacles (e.g., assembles the box). In some embodiments, the receptacle source places the empty receptacles on the conveyor based at least in part on a predetermined location, the speed of the conveyor618, a distance between a previous receptacle and the empty receptacle being placed, an order for which the items are being kitted (e.g., receptacles pertaining to a particular order may be placed closer together than receptacles pertaining to different orders), a size of the empty receptacle and/or the previous receptacle, etc.

As illustrated inFIGS. 6A and 6B, the receptacle may be a tray such as receptacle616a, or a box such as receptacle616b. Various types of receptacles may be implemented.

FIG. 7is a diagram illustrating a kitting system according to various embodiments.

According to various embodiments, a kitting system may include one or more kitting shelf systems, one or more robotic arms, and/or one or more conveyors. The kitting system may include one or more control computers that are configured to control operation of the kitting shelf system(s), the robotic arm(s), and/or the conveyor(s). The one or more kitting shelf systems, one or more robotic arms, and/or one or more conveyors may be variously arranged. In the example illustrated inFIG. 7, kitting system700includes kitting shelf system704and kitting shelf system724; robotic arm710and robotic arm720; and conveyors718aand718b. In some implementations, kitting system has a plurality of kitting shelf systems, a plurality or robotic arms, and a single conveyor (along which receptacles flow during a kitting process).

In various embodiments, the one or more conveyors of the kitting system700may be arranged in various configurations. The conveyor can be arranged in a straight line (e.g., from an entrance of kitting system700or section of a warehouse, to an exit of the kitting system700or section of the warehouse); an “L”-shape; a “U” shape,

In the case of kitting system700including a plurality of robotic arms, the robotic arms may be disposed along one side of a conveyor (e.g., conveyor718a), or the robotic arms may be disposed on different sides of the conveyor(s). Further, a robotic arm may be disposed on a same side of the conveyor as one or more kitting shelf systems, or the robotic arm may be disposed on a side of the conveyor that is opposite to the one or more kitting shelf systems. In various embodiments, in the case of kitting system700including a plurality of robotic arms, a robotic arm is selected from among the plurality of robotic arms to obtain an item (or object from within an item) from a kitting shelf system based on a location of the robotic arm, an effectiveness of the particular robotic arm successfully picking the item (e.g., a computed likelihood of success), a reach of the particular robotic arm, a time within which the particular robotic arm will complete the picking and placing of the item, etc. The particular robotic arm selected to pick and place the item may be selected by control computer734. For example, control computer724may store a model with respect to the workspace of kitting system700and/or one or more of the robotic arms within the workspace. The control computer may use the model to determine a robotic arm to pick and place an item, and/or a robotic arm to pick and place each of one or more items for a particular order (e.g., a single robotic arm may be used to kit the items for an order). In some embodiments, a robotic arm that is disposed on a same side as a kitting shelf system is more effective/efficient at picking and placing items from shelves on the kitting shelf system that are above a threshold height. For shelves below the threshold height, the contortion of the robotic arm required to reach an item on such shelves renders the robotic arm on the same side of the conveyor as the particular kitting shelf system inefficient. Further, the contortion of the robotic arm required to reach an item on such shelves renders the robotic arm with limited range/mobility with which to pick an item from a shelf below the height threshold. Accordingly, in cases of items on shelves below a threshold height, kitting system700may use a robotic arm710on a side of the conveyor718aor718bthat is opposite to the kitting shelf system724from which the item is to be obtained.

According to various embodiments, a kitting system may determine and/or implement a plan for kitting one or more items/objects based at least in part on information obtained by one or more sensors within the workspace. The one or more sensors may include a camera, a barcode scanner, a QR code scanner, a radio-frequency identification (RFID), a laser sensor, an infrared (IR) sensor, etc. Various other types of sensors or arrays of sensors may be implemented. One or more sensors within the workspace may obtain information (e.g., capture an image) of the workspace such as one or more shelves of a kitting shelf machine.

The kitting system may use information obtained by the one or more sensors within the workspace to determine a plan for one or more items/objects. For example, an image may be generated and an image analysis may be performed with respect to information corresponding to one or more shelves of the kitting shelf machine. One or more images may be cached such as via storing the image in a data structure associated with the kitting shelf machine, a particular shelf of the kitting shelf machine, and/or a particular item/object or type of item/object on the shelf. In some embodiments, rather than obtaining and processing information captured by the one or more sensors in connection with determining a plan for each item/object, the system may use a cached image (or information pertaining to the workspace) in connection with determining the plan.

When the kitting system determines that an item is to be kitted, the kitting system may determine whether a cached image and/or cached information pertaining to the item (e.g., an image/information for the shelf on which the item is determined to be located or for a tray in which the item/object is located, etc.). In response to determining that a cached item or cached information pertaining to the item is stored (e.g., exists), the kitting system may determine whether the kitting of a previous item is likely to have changed a state of the kitting shelf system with respect to the current item to be kitted. For example, if the kitting system determines that a previous item was picked from a first shelf, and the current item is to be picked from the second shelf, the kitting system may determine that the state of the kitting shelf system with respect to the current item has not changed (or is likely to have not changed)—the cached image with respect to an area (e.g., a tray or shelf) on which the current item is located is likely to be the same as a current image generated based on information from the one or more sensors. In response to determining that a cached image or cached information pertaining to the item is stored, a plan for kitting a current item may be determined without capturing a new image/information of the workspace and/or without performing a same level (e.g., full) image analysis with respect to such current image.

The use of the cached image or cached information pertaining to the item in connection with determining the plan to kit the item can improve the efficiency with which the kitting system (e.g., the robotic arm) may pick the item. For example, in some implementations, the image and/or information pertaining to the item/workspace that is used for determining the plan, or in connection with implementing the plan, is obtained by a sensor (e.g., a camera) mounted to the robotic arm or to a pole or other structure attached to the chassis of the robotic arm. Because the robotic arm is moving, the sensor may not be completely (or sufficiently) stabilized when the robotic arm reaches the location from which the item to be kitted is to be picked. Rather than wait for the sensor to stabilize (e.g., for a stabilization of the sensor to reach a threshold level), various embodiments may use the cached image and/or information pertaining to the item/workspace in connection with kitting the item.

In some embodiments, the determination of a change (e.g., a difference) between two images (or areas of two images such as a particular shelf or a particular tray, etc.) is computed more quickly than an image analysis of an image. For example, the computation of a difference between systems (e.g., a difference between a state of the system corresponding to when the cached image was captured, and a current state of the system) is faster than completely modelling the system (according to the current state of the system). Accordingly, if an image with respect to a workspace (e.g., an image of a shelf, an image of a tray on a shelf, etc.) is cached (e.g., before the robotic arm arrives at the shelf to pick the item therefrom), a difference between a current image and a cached image may be computed in connection with determining whether a state of the kitting shelf system with respect to the current item to be kitted, or with determining change of such state. In response to determining that the difference between the current image and the cached image is less than a threshold value (or that there is no difference between the images), then the robotic system may use the cached image in connection determining and/or implementing the plan. If an image is cached, then on a next iteration of picking an item from the kitting shelf machine (e.g., the shelf or tray on a shelf), the system can use an image point cloud to determine whether shapes in the image are in the same positions (or same relative positions) across the images. The determination of whether the shapes in the image are in the same position may be used in connection with determining whether the state of the kitting shelf system has changed or determining that the difference between the current image and the cached image is less than a threshold value (or that there is no difference between the images). In some embodiments, the determining the difference between states of the system comprises running a comparison to see if an expected object is still within an area of the image where it is expected to be. The comparison may be run on a subset of the entire cached image.

In some embodiments, an accelerometer, gyroscope, or other type of sensor may be used in connection with determining whether a sensor (e.g., a camera) is sufficiently stable to obtain the image and/or other pertaining to the workspace. For example, in the case of the camera being mounted to a pole or other structure attached to the chassis of the robot, the camera may require a certain amount of time before becoming sufficiently stable to capture an image. The kitting system may comprise an accelerometer that is used to measure the stability of the camera or the robotic arm, etc. In response to a value from the accelerometer and/or the gyroscope being less than a threshold value (e.g., vibrations being less than a threshold value), the kitting system may determine that the camera is sufficiently stable to capture the image and/or other pertaining to the workspace. In some embodiments, the value(s) from the accelerometer and/or the gyroscope may be used in connection with the processing of the image and/or other information pertaining to the workspace. For example, blurring effects in the image captured by the camera may be removed (or improved) based on an image processing using at least the value(s) from the accelerometer and/or the gyroscope.

FIG. 8A-8Care diagrams illustrating a kitting system according to various embodiments. In the examples shown inFIGS. 8A-8C, system800comprises one or more sensors such as camera815. The one or more sensors may obtain an image of, or information pertaining to, system800such as shelf801, shelf803, and/or shelf805. Accordingly, the image of, or other information pertaining to, system800may indicate that shelf801comprises tray807, shelf803comprises tray809, and/or shelf805comprises tray811.

In some embodiments, system800determines that tray807, tray809, and/or tray811comprises one or more objects. System800may store (e.g., cache) the image and/or the information pertaining to the workspace. As illustrated inFIG. 8B, robotic arm821may move to shelf803and pick up item831. In some embodiments, robotic arm821comprises a camera823. In connection with picking and placing item831from tray809, system800may determine whether the state of the system (or state of the shelf803or tray809in which item831is located) has changed since the cached image was cached. If system800determines that the change with respect to the state of the system, etc. is less than a threshold extent (e.g., a threshold value), then system800may determine the item to pick up (e.g., item831) and/or a plan for picking up item831based at least in part on the cached image. System800may determine the item to pick up (e.g., item831) and/or a plan for picking up item831based at least in part on the cached image and without generating a model (e.g., a complete image model) based on the current image and/or current information obtained by the sensors within the workspace. System800may thus use the cache image to determine the plan and/or in connection with picking up the item831without having to wait for camera823to stabilize or for current information obtained by the one or more sensors in system800to be processed to create a new image comprising at least item831and/or tray809.

As illustrated inFIG. 8C, item831has been removed from tray809. In some embodiments, in connection with picking a subsequent item, system800may use the cache image in connection with picking an item such as item833from tray807and/or item835from811. System800may determine that the states of tray807, tray811, and/or the items within trays807and811has not changed. Accordingly, system800may use the cache image such as the image corresponding to the state illustrated inFIG. 8A. However, because item811has been removed from tray809by robotic arm821, the cache image may not be sufficient to use for picking an item as illustrated inFIG. 8C. In some embodiments, even with item811having been removed from tray809, system800may determine that a change in the state of tray809from before item811was picked and after item811was picked is not greater than a threshold value. In such a case, system800may use the cache image to determine a plan/strategy for picking an item from tray809. Alternatively, in some embodiments, system800may determine that the change in state of tray809(e.g., as a result of item811having been removed) is greater than a threshold value, and in connection with picking an item from tray809, system800may use a current image and/or current information pertaining to the workspace.

According to various embodiments, a kitting system may comprise a plurality of kitting shelf systems (e.g., kitting shelf machines). The plurality of kitting shelf systems may be operatively connected via a conveyance system. Related art systems generally only include a discrete robotic structure to which a receptacle is brought for a kitting of objects. The conveyance system of the kitting system may have an input end at which empty totes (e.g., receptacles such as bins, trays, etc.) are input to the conveyor which carries totes to the various kitting shelf systems comprised in kitting system. Robotic arms disposed at the various kitting shelf systems may pick items from the various kitting shelf systems and place the items in corresponding totes. As the totes are packed with one or more items, the conveyance structure may convey the totes to an output area of the kitting system. The output area may have a packing area or the totes may be removed from the conveyance structure and input to a packing system.

FIG. 9is a diagram illustrating a kitting system according to various embodiments.

As illustrated inFIG. 9, kitting system900comprises a conveyance system that carries (e.g., distributes) totes to various kitting shelf machines. The totes may be input to the conveyance system at input area905. Various ones of the totes are distributed to areas respectively corresponding to kitting shelf machine910, kitting shelf machine915, kitting shelf machine920, kitting shelf machine925, kitting shelf machine930, and kitting shelf machine935.

In some embodiments, conveyance structure terminates at a kitting shelf machine such as at kitting shelf machine910and/or kitting shelf machine935in kitting system900. If the conveyance structure terminates at a kitting shelf machine, the totes that have been packed with one or more items (e.g., by kitting shelf machine910and/or kitting shelf machine935) may be removed from the kitting shelf machine such as by a human operator (or otherwise manually removed) or from a different system.

In some embodiments, a plurality of kitting shelf machines are interconnected via the conveyance structure. For example, a plurality of kitting shelf machines may share a common input area at which totes (e.g., empty totes) are input to the kitting system. As another example, a plurality of kitting shelf machines may share a common output area at which totes (e.g., totes packed with one or more items) are output from the kitting system such as an area from where the totes are sent for packing, labelling, and/or other processing, etc. As another example, plurality of kitting shelf machines may share a common output area and a common input area. As illustrated inFIG. 9, kitting shelf machine915, kitting shelf machine920, kitting shelf machine925, and kitting shelf machine930share a common input area905and a common output area940. Accordingly, in the example shown nFIG. 9, kitting shelf machine915, kitting shelf machine920, kitting shelf machine925, and kitting shelf machine930may be operatively connected by the conveyance structure. In some embodiments, a kitting shelf system may comprise a plurality of kitting shelf machines such as kitting shelf machines disposed on opposing sides of a conveyor. For example, referring to kitting system900, a kitting shelf system may comprise kitting shelf machine915and kitting shelf machine920; and another kitting shelf system may comprise kitting shelf machine925and kitting shelf machine930. In some embodiments, each kitting shelf system may comprise a control computer(s) that controls a plurality of kitting shelf machines and corresponding robotic arms. In other embodiments, each kitting shelf machine may comprise its own control computer that controls the kitting shelf machine and corresponding robot. In other embodiments, kitting system900comprises a control computer that controls a plurality of kitting shelf systems and/or a plurality of kitting shelf machines.

In some embodiments, kitting system900may store and/or manage a data structure pertaining to an association of totes (e.g., trays) with orders or items. For example, a kitting shelf machine (or controller associated therewith) may reserve an empty tote for the kitting shelf machine to place items that a corresponding robotic arm has picked from the kitting shelf. As another example, in response to items being packed in a tote, the data structure may be updated to include a mapping of an identifier of the tote to an order for which the items are kitted and/or a mapping of an identifier of the tote to one or more items comprised in the tote.

Kitting system900may comprise one or more sensors and/or one or more arrays of sensors disposed along the conveyance structure. For example, the kitting system900may comprise one or more sensors at input area905. As another example, the kitting system900may comprise one or more sensors at output area940. As another example, the kitting system900may comprise sensors at areas along the conveyance structure where kitting machines are disposed. The sensors may be used in connection with associating totes with orders and/or items to be kitted (or that are kitted) within the totes. The sensors may also be used in connection with determining one or more attributes associated with the totes and/or items within the totes (e.g., obtaining an identifier on the identifier and/or an identifier on the one or more items, etc.).