Method and system for tele-operated inventory management system

Described herein is an inventory management system, and corresponding methods, in which robotic units are configured to execute at least a portion of a set of instructions in an automated manner. During execution of the set of instructions, robotic units may encounter tasks that they are unable to complete in an automated manner. The robotic units, upon encountering these tasks, may submit a request for manual operation to a control unit. In some embodiments, the request may be assigned to an available operator, which may take over control of the robotic unit using a remote manipulation device, to complete the task manually. Once the task has been completed manually, automated execution of the instructions may be resumed.

BACKGROUND

Modern inventory systems, such as those in mail order warehouses, supply chain distribution centers, and custom-order manufacturing facilities, face significant challenges in responding to requests for inventory items. As inventory systems grow, the challenges of simultaneously completing a large number of packing, storing, and other inventory-related tasks become non-trivial. In inventory systems tasked with responding to large numbers of diverse inventory requests, inefficient utilization of system resources, including space, equipment, and manpower, can result in lower throughput, unacceptably long response times, an ever-increasing backlog of unfinished tasks, and, in general, poor system performance. Additionally, expanding or reducing the size or capabilities of many inventory systems requires significant changes to existing infrastructure and equipment. As a result, the cost of incremental changes to capacity or functionality may be prohibitively expensive, limiting the ability of the system to accommodate fluctuations in system throughput.

DETAILED DESCRIPTION

Techniques described herein are directed to an inventory management system in which one or more robotic units are configured to carry out tasks automatically. In some embodiments, robotic units, during an automated execution of a set of instructions, may submit a request to a control unit for manual operation. The request may be assigned to an operator (e.g., a user having access to a remote manipulation device), which may take over functions of the robotic unit and manually cause the robotic unit to perform an operation. Once the request for manual operation has been completed, the robotic unit may continue with its automated execution of the set of instructions. For the purposes of this disclosure, the term automated or automatic may infer a lack of human interaction.

For example, in the described system, the robotic units may, during execution of an automated procedure, encounter tasks that the robotic units are unable to perform automatically (e.g., the system may have insufficient instructions for performing the task). In this scenario, the robotic units of the described system may submit a request for manual operation to a queue of operators to complete the task. When the request reaches the top of the queue, and when an operator is available to handle the request, the operator is able to take control of the robotic unit to complete the task manually. The operator may be shown image information related to a position of an end-of-arm tool and/or an item during manual operation of the robotic unit. Once the task has been completed, the operator may be assigned the next manual operation request in the queue. Within the described inventory management system, a number of robotic units may be performing automated tasks, such that a number of robotic units may require assistance at any given time. In this way, operators are directed to only those robotic units that need immediate assistance, which minimizes downtime for the operator (e.g., by minimizing the need to wait while a robotic unit traverses to a particular location) and maximizes the number of tasks that may be performed by a single operator.

FIG. 1depicts an illustrative example of an inventory management system in which robotic units are operated in accordance with embodiments of the disclosure. In some embodiments, a number of robotic units102,104, and106may be in communication with a control unit110. Communication between the robotic units and the control unit110may be established via any suitable communication mechanism. In some embodiments, communication may be established via a wireless local area network using a wireless router112.

In accordance with some embodiments, each robotic unit within the inventory management system may be provided, by the control unit110, with a set of instructions to cause that robotic unit to at least partially execute a task in an automated manner (e.g., without user interaction). The provided instructions may also cause the robotic unit to hand over control of the robotic unit to an operator114for execution of a portion of the task. During execution of this portion of the task, the user may control the robotic unit via a remote manipulation device116. Once the manual portion of the task has been completed by the operator, the robotic unit may be caused to continue with automated tasks.

The remote manipulation device116may be any electronic device capable of transmitting input received from an operator to a robotic unit102. In some embodiments, the remote manipulation device116may be configured to receive and relay data received from the robotic unit. For example, the remote manipulation device116may include a display device capable of presenting image data received from the robotic unit102(e.g., video image data). In some embodiments, the remote manipulation device116may provide feedback to a user in a variety of ways. For example, the remote manipulation device116may provide haptic feedback to a user by generating vibrations in order to relay information related to input received from the robotic unit102. In some embodiments, the remote manipulation device may be a combination of devices. For example, the remote manipulation device may be a virtual reality (VR) headset capable of receiving image information (e.g., video augmented with a text overlay) and a pair of sensor gloves capable of receiving hand motion information as user input and translating that user input to computer-executable instructions. In another example, the remote manipulation device may include a display device and a joystick. In another example, a remote manipulation device may include a combination of a VR headset and one or more gloves that are tracked by a system in communication with the VR headset (e.g., using machine vision via a camera system or using sensors on the one or more gloves). In this example, the operator may be provided the ability to guide the robot by moving his or her hand in a projected virtual space.

By way of illustrative example, a robotic unit102may be a robotic picking unit, which is designed to retrieve and/or stow inventory items in storage locations of the inventory management system. In this system, the robotic unit102may receive instructions to fulfill at least part of an order submitted to the inventory management system by retrieving one or more items from storage bin118. In this example, the robotic unit102may automatically traverse to the location of storage bin118(e.g., using routing software) and position itself in front of the item. The robotic unit102may then submit a request to the control unit110for manual manipulation of the requested item. The control unit110may add the request to a queue of such requests that are being worked on by one or more operators. Once the request is at the top of the queue and is selected by or assigned to an operator114, the operator114may use the remote manipulation device116to control the robotic unit102and to cause it to pick the requested item. The operator114may then cause the robotic unit102to deposit the item into a bin120, cart, shipping container, or other suitable placement location. Once the operator114indicates that his portion of the task has been completed, the robotic unit102may be caused to resume automated functions by traversing to the next location.

In some embodiments, each time that the robotic unit102relinquishes control to an operator114, the control unit110may record operations performed by the operator114. In some embodiments, the control unit110may use this recorded information along with one or more machine learning techniques to identify patterns for manipulating particular items. For example, the control unit110may record operations performed by a number of operators with respect to a particular item or type of item. Based on these recorded operations, the control unit110may identify an appropriate set of instructions for manipulating the particular item. In some embodiments, a set of instructions may be particular to a specific pose or position of the item. In future cases in which the robotic unit102is tasked with retrieving the particular item, the retrieval of the item may be entirely automated, as the robotic unit102may be caused to perform the learned set of instructions to manipulate the item. In some cases, the robotic unit102may attempt to manipulate the item, and may subsequently submit a request for an operator upon failure to properly manipulate the item. For example, if the robotic unit102approaches the storage location118and determines that the item is not positioned properly (e.g., using machine vision techniques), the robotic unit102may determine that the set of instructions is no longer relevant to retrieval of the item and may subsequently request operator114support. In some embodiments, the actions recorded with respect to a number of operators may be used to train new operators in grasping an item. For example, new operators may be provided an instructional video for grasping an object recorded from a previous manual operation.

In some embodiments, the inventory management system may receive feedback with respect to performed manual operations. For example, the inventory management system may receive feedback from a quality assurance (QA) unit that indicates a status of the item with respect to the performed manual operations. In this example, the QA unit may determine that the item was broken or otherwise damaged. In some embodiments, an operator may augment image information to assist the robotic unit. For example, an operator may highlight a representation of an item within an image in order to help the robotic unit locate the item. Based on this analysis, the system may learn that the actions recorded with respect to the manual operation used to manipulate the item are inappropriate. In some embodiments, the inventory management system may generate instructions for manipulating an item (e.g., retrieving the item from a storage location) based on the manual operations most commonly performed by operators. Some examples of ways in which an inventory management system may be trained to manipulate items based on human feedback are provided in U.S. patent application Ser. No. 15/384,139, filed Dec. 19, 2016, entitled “Robot Implemented Item Manipulation,” which is herein incorporated by reference in its entirety.

It should be noted that a manual operation may be any suitable operation requiring manual user intervention. For example, in addition to picking and/or stowing of items, a manual operation may include performing quality assurance (QA), bin counting, physical—virtual inventory verification, or any other suitable action. In some embodiments, the manual operation may include setting up the robotic unit to perform a task in an automated fashion. For example, the manual operation may cause an operator to view the current status of the task and provide instructions to the robotic unit (e.g., a type of grasp to use, an amount of force to use in the grasp, or a pose to initiate the grasp from).

For clarity, a certain number of components are shown inFIG. 1. It is understood, however, that embodiments of the invention may include more than one of each component. In addition, some embodiments of the invention may include fewer than or greater than all of the components shown inFIG. 1. In addition, the components inFIG. 1may communicate via any suitable communication medium (including the internet), using any suitable communication protocol.

FIG. 2illustrates an example inventory system architecture that may be implemented in accordance with embodiments of the disclosure. In architecture200, a control unit202may be in communication with inventory management components, such as a robotic picking unit204. A robotic picking unit204may be an example robotic unit102depicted inFIG. 1.

The control unit202may be any type of computing device configured to communicate with and one or more inventory management components and one or more operators. In some embodiments, the control unit202may be executed by one more virtual machines implemented in a hosted computing environment. The hosted computing environment may include one or more rapidly provisioned and released computing resources, which computing resources may include computing, networking, and/or storage devices. A hosted computing environment may also be referred to as a cloud-computing environment.

In one illustrative configuration, the control unit202may include at least one memory206and one or more processing units (or processor(s))208. The processor(s)208may be implemented as appropriate in hardware, computer-executable instructions, firmware or combinations thereof. Computer-executable instruction or firmware implementations of the processor(s)208may include computer-executable or machine executable instructions written in any suitable programming language to perform the various functions described.

The memory206may store program instructions that are loadable and executable on the processor(s)208, as well as data generated during the execution of these programs. Depending on the configuration and type of control unit202, the memory206may be volatile (such as random access memory (RAM)) and/or non-volatile (such as read-only memory (ROM), flash memory, etc.). The control unit202may also include additional storage210, such as either removable storage or non-removable storage including, but not limited to, magnetic storage, optical disks, and/or tape storage. The disk drives and their associated computer-readable media may provide non-volatile storage of computer-readable instructions, data structures, program modules, and other data for the computing devices. In some implementations, the memory206may include multiple different types of memory, such as static random access memory (SRAM), dynamic random access memory (DRAM) or ROM. Turning to the contents of the memory206in more detail, the memory206may include an operating system212and one or more application programs or services for implementing the features disclosed herein including at least a module for managing automated and manual operations of components within the inventory system (inventory management module214). The memory206may also include inventory data216, which provides information associated with items managed by the inventory system. In some embodiments, the inventory data216may be stored in a database.

The memory206and the additional storage210, both removable and non-removable, are examples of computer-readable storage media. For example, computer-readable storage media may include volatile or non-volatile, removable or non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data. As used herein, modules may refer to programming modules executed by computing systems (e.g., processors) that are installed on and/or executed from the control unit202. The control unit202may also contain communications connection(s)218that allow the control unit202to communicate with a stored database, another computing device or server, user terminals, and/or other components of the inventory system. The control unit202may also include input/output (I/O) device(s) and/or ports220, such as for enabling connection with a keyboard, a mouse, a pen, a voice input device, a touch input device, a display, speakers, a printer, etc.

Turning to the contents of the memory206in more detail, the memory206may include an operating system212, a database containing inventory data216and the one or more application programs or services for implementing the features disclosed herein, including an inventory management module214.

In some embodiments, the inventory management module214may be configured to, in conjunction with the processors208, manage inventory retrieval and stowage of items within the inventory management system. In some embodiments, the inventory management module214may be configured to receive a request for an item (e.g., a fulfillment order) and provide instructions to a robotic picking unit204to retrieve the requested item. In some embodiments, at least a portion of the instructions to retrieve the requested item may cause the robotic picking unit204to operate in an automated fashion. The instructions may cause the robotic picking unit204to, at some point during execution of the provided instructions, submit a request to the control unit202for manual execution of one or more actions by a remote operator. In some embodiments, the inventory management module214may be configured to determine whether an item manipulation action should be performed by the robotic picking unit204in a manual or automated fashion based at least in part on whether sufficient automated picking instructions are stored in inventory data216. In some embodiments, the inventory management module214may be configured to provide instructions to a robotic picking unit as each previous set of provided instructions is completed. For example, the control unit202may be configured to provide instructions to a robotic picking unit204to traverse to a particular storage location. Upon arriving at the indicated storage location, the control unit202may then provide a set of instructions for identifying an item within the storage location.

Inventory data216may be predetermined or it may be dynamically generated. For example, manipulation data associated with an item may be updated as new information is received on that item. In some embodiments, the inventory data216may include information related to grasping of items. In some embodiments, the inventory data216may include information related to a pose that is appropriate for one or more items. In some embodiments, the inventory data216may include a set of instructions for manipulating a particular item learned from one or more interactions by one or more operators to cause a robotic picking unit to automatically retrieve the item.

The robotic picking unit204may comprise any robotic device configured to move items from one location to another. The robotic picking unit204may include one or more mechanical arms222with an end-of-arm tool (EOAT)224. This is described in greater detail below. It should be noted that although a particular example robotic picking unit204is depicted inFIG. 2, the robotic picking unit204may be any suitable robotic device, stationary or mobile, which that capability to perform the described functions. In some embodiments, the mechanical arm of a robotics picking unit204may include any number of degrees of freedom.

In one illustrative configuration, the robotic picking unit204may include at least one memory226and one or more processing units (or processor(s))228. The processor(s)228may be implemented as appropriate in hardware, computer-executable instructions, firmware or combinations thereof. Computer-executable instruction or firmware implementations of the processor(s)228may include computer-executable or machine executable instructions written in any suitable programming language to perform the various functions described. Additionally, the robotic picking unit204may comprise a number of input sensors230(A and B), which detect and provide input related to conditions of an environment that the robotic picking unit204is located within.

The memory226may store program instructions that are loadable and executable on the processor(s)228, as well as data generated during the execution of these programs. The memory226may include one or more application programs or services for implementing the features disclosed herein including at least a module for determining how to manipulate an item (item manipulation module232). The memory226may also include item force data234, which provides information associated with forces that may be applied to items managed by the inventory system. The memory226may also include a routing module236, which provides causes the robotic picking unit204to traverse to a specified location within an inventory location.

In accordance with at least some embodiments, the item manipulation module232may be configured to, in conjunction with the processors228, determine an appropriate strategy to be used in manipulating an item. For example, the item manipulation module232may be configured to determine whether an item should be grasped in an automated manner or grasped via manual control by an operator. In this example, the item manipulation module232may assess a pose or position of the item (e.g., using one or more machine vision techniques) to determine whether instructions stored in the inventory data216would support automated manipulation of the item. If the item manipulation module232determines that sufficient instructions are available for manipulating the item, then the robotic picking unit204may select an appropriate grasp to be used on an item based on inventory data216and automatically retrieve the item. In some embodiments, the item manipulation module232may also be configured to determine an appropriate level of force to be applied to an item from force data stored in association with the item. The item manipulation module232may be configured to cause an EOAT224to apply the determined level of force to the item in order to stow or retrieve the item. If the item manipulation module232determines that insufficient instructions are available for manipulating the item, then the robotic picking unit204may submit a request to the control unit202for manual manipulation of the item by a remote operator.

The robotic picking unit204may include a communication interface238that allow the robotic picking unit204to communicate with the control unit202, another computing device or server, user terminals, and/or other components of the inventory system. In some embodiments, the communication interface238may comprise a wireless communication mechanism. In some embodiments, the communication interface238may comprise a physical wired connection with other components of the system. The communication interface238may enable remote operation of the robotic picking unit204.

In some embodiments, the robotic picking unit204may be stationary (e.g., incapable of movement). In some embodiments, the robotic picking unit204may be mobile, in that the robotic picking unit204is configured to traverse from a first location to a second location. For example, the robotic picking unit204may be configured to traverse to a first location, retrieve an item at the first location, traverse to a second location, and place the item at the second location. In these embodiments, the robotic picking unit204may include any suitable mobility means240. For example, the robotic picking unit204may include wheels and a motorized engine, such that the robotic picking unit204is capable of autonomous movement. In this example, the memory226may include a routing module236, which may be configured to perform automated route guidance. In another example, the robotic picking unit204may be located on a track or conveyor belt.

In some embodiments, the inventory system architecture may include one or more remote manipulation devices242. Remote manipulation device242may be an example remote manipulation device116ofFIG. 1. In some embodiments, a remote manipulation device242may include at least a display element244and/or a manipulation element246. Display element244may be any component capable of presenting information received from a robotic picking unit204to a user. In some embodiments, display element244may be a monitor, television, or another suitable display device. Manipulation element246may include any suitable mechanism for receiving input from an operator and using that input to control one or more functions of the robotic picking unit. In some non-limiting examples, a manipulation element246may be a joystick, sensor gloves, or a motion tracking camera system (e.g., Kinect) based system.

FIG. 3depicts an illustrative example of a robotic picking unit300in accordance with embodiments of the disclosure. Robotic picking unit300may be mobile or stationary. In embodiments in which the robotic picking unit300is mobile, the robotic picking unit300may include a mobile conveyance means302(e.g., wheels, tread, conveyor, or any other suitable conveyance mechanism). It should be noted that a number of equivalent conveyance mechanisms are known to one skilled in the art, each of which may be used in embodiments of the disclosure. In some embodiments, the robotic picking unit300may include multiple different mobile conveyance means302. In some embodiments, a robotic picking unit300may be an airborne robotic unit (e.g., a drone).

In some embodiments, the robotic picking unit300may include a mechanical arm304configured to position an EOAT306in front of an item. AlthoughFIG. 3depicts the robotic picking unit300as having a single mechanical arm304, some embodiments of the robotic picking unit300may include multiple mechanical arms. In some embodiments, the EOAT306may be detachable. For example, the EOAT may be configured to be attached to and unattached to the mechanical arm304. In some embodiments, an EOAT306may be selected for a task by a processor within the robotic picking unit300and/or by a control unit, such that the selected EOAT306is selected based on its appropriateness for manipulating a particular item or type of item. In some embodiments, the EOAT306may be permanently affixed to the mechanical arm304. For example, the robotic picking unit300may be a special-purpose robotic picking unit300designed to retrieve and/or stow a particular item or type of item.

In some embodiments, the robotic picking unit300may include a number of alternative EOATs that may be swapped out according to requirements associated with the performance of a particular task. In some embodiments, the EOATs may be configured to be swapped out upon receiving instructions from an operator. For example, the operator, upon initiation of the manual operation, may assess a context in which the manual operation is to be completed and may provide instructions to switch between available EOATs. In some embodiments, the inventory management system may instruct a special tooling unit (either a robotic unit or a person) to navigate to a location of the robotic picking unit300and swap out a current EOAT being used by the robotic picking unit300with one appropriate for the current task.

In some embodiments, the robotic picking unit300may include one or more input sensors308and310. Input sensors may include any sensor device capable of obtaining information related to an environment within which the robotic picking unit300is located. Some examples of input sensors may include cameras (or other imaging devices), global positioning system (GPS) devices, microphones (or other suitable audio input devices), thermometers, haptic sensors, or any other suitable sensing device. Additionally, robotic units may use a number of techniques to navigate throughout an area (e.g., a fulfillment center). For example, in addition to sensors installed on the robotic unit, the robotic units may use sensors located outside of the robotic unit, bar codes, RFID, and/or Real Time Location Service (RTLS) to navigate. In some embodiments, input obtained by the one or more input sensors may be transmitted to a remote manipulation device. In some embodiments, the robotic picking unit may contain multiple input sensors of the same type, some of which may be used for different purposes. For example, the robotic picking unit300may include a first camera device308able to obtain image information used in automated routing of the robotic picking unit300. The robotic picking unit300may also contain a second camera device310able to obtain image information related to a position or status of the EOAT306. In this example, the second camera device310may be mounted on or in close proximity to the mechanical arm304of the robotic picking unit300.

In some embodiments, the robotic picking unit300may include a bin or storage location312. The storage location312may be any area or mechanism on the robotic picking unit300designed to hold an item or another storage bin (e.g., a shipping container). It should be noted that some embodiments of the robotic picking unit300may not include a storage location312. For example, a robotic picking unit300may be accompanied by a separate robotic device designed carry inventory items (e.g., an automated cart). In some embodiments, the robotic picking unit300may be configured to grasp an inventory item, traverse to a different location, and place the inventory item within a shipping container or other storage location.

FIG. 4depicts a first example of a hybrid tele-operated inventory management system in operation. InFIG. 4, an inventory management system may include a number of mobile robotic picking units. The mobile robotic picking units may be in communication with one or more control units, which provide the mobile robotic picking units with instructions for fulfilling a number of fulfillment orders. In at least some of these embodiments, multiple mobile picking units may be provided with instructions for fulfilling multiple item requests at the same time (e.g., in parallel). The instructions provided to the mobile picking units may include at least some instructions that cause the mobile picking unit to operate in an automated manner (e.g., without user interaction). In some cases, a mobile picking unit may reach a point in the received instructions in which manual operation is necessary. For example, the robotic picking unit may need to manipulate an item for which it does not have sufficient automated manipulation information. In this example, the robotic picking unit may position itself in front of the item to be retrieved and submit a request for manual operation by an operator.

Submitted requests for manual operation may be transmitted to a control unit for subsequent assignment. In some embodiments, the control unit may determine which operators, from a number of operators, may be assigned the manual operation. In some embodiments, the control unit may add the request to a queue of such requests that are being worked by the number of operators. In some embodiments, assignment of a manual operation to an operator may be made based on an expertise of that operator. For example, manual operation of a robotic forklift may be assigned to an operator certified to operate a forklift. In another example, manual retrieval of a particular item may be assigned to an operator that is familiar with that particular item (e.g., an operator capable of discerning between like items). In some embodiments, the control unit may obtain statistical information with respect to the operators, upon which it may base future assignment decisions. For example, the control unit may obtain metrics related to a number of manual operations that a particular operator performs within a specified timeframe (e.g., operations per hour). In another example, the control unit may obtain metrics on how many manual operations for an operator are not completed or result in a broken item. These metrics may be used to disqualify the operator from further manual operations or to limit that operator to only non-fragile items.

Once the request has been assigned to an operator or the operator selects the request, the operator may use a remote manipulation device to gain control of the robotic picking unit and to cause the EOAT of the robotic picking unit to grasp the item. In some embodiments, the remote manipulation device may receive image information, which depicts objects in the vicinity of the EOAT. In some embodiments, image information may be augmented to include additional information to be provided to the remote manipulation device.

Once the requested manual operation has been completed (e.g., by receiving an indication of completion from the operator), the remote manipulation device may be disconnected from the robotic picking unit. In some embodiments, the control unit may then select the next request for manual operation in the queue and establish a connection between the remote manipulation device and the robotic picking unit that submitted the request. This creates a system in which operator downtime is minimized during fulfillment of item requests.

By way of illustrative example, a control unit may provide instructions to a number of robotic picking units to retrieve various inventory items for fulfillment of item orders. In this example, a robotic picking unit402may, upon reaching a storage location404associated with an inventory item with which robotic picking unit402is tasked to retrieve, submit a request for manual operation. In some embodiments, the robotic picking unit402may identify the item within the storage location404. For example, the robotic picking unit402may use machine vision along with one or more item identification techniques to determine a precise location of the item within the storage location404. In some embodiments, the robotic picking unit402may be configured to read barcodes or other machine readable code to identify items within the storage location404.

The control unit may receive the request and determine that an operator of a remote manipulation device406is available to perform the manual manipulation operation. The control unit may establish a connection between the robotic picking unit402and the remote manipulation device406that involves relaying image information408obtained by a camera device on the robotic picking unit402to the remote manipulation device406and relaying commands received by the remote manipulation device406as input from the operator to the robotic picking unit402. In some embodiments, the image information408may be augmented to include additional information410. For example, the control unit may overlay a portion of the image information408with an image of the item to be retrieved or other descriptive material related to the manual operation (e.g., a color of the item, a basic description of the item, or any other suitable descriptive material). The operator may use the image information408to identify and retrieve the requested item. To do this, the remote manipulation device may be used to control a grasping function on the EOAT on the robotic picking unit402. Additionally, the remote manipulation device may be used to control a positioning of the EOAT (e.g., by actuating one or more actuators on a mechanical arm or causing the robotic picking unit to move). It should be noted that means for remotely controlling a device in this fashion are well known to those skilled in the art and will not be described in detail here.

Once the operator of remote manipulation device406has completed the requested manual operation, the control unit may sever the connection between the remote manipulation device406and the robotic picking unit402. The control unit may then determine that robotic picking unit412has previously submitted a request for manual operation, for which the operator of the remote manipulation device406is qualified to perform. At some time previous to submission of the request for manual operation, the robotic picking unit412may have traversed, using automated means, to a storage location414within which is stored an item to be retrieved by the robotic picking unit412. The control unit may establish a connection between the robotic picking unit412and the remote manipulation device406that involves relaying image information416obtained by a camera device on the robotic picking unit412to the remote manipulation device406and relaying commands received by the remote manipulation device406as input from the operator to the robotic picking unit412. In some embodiments, the image information416may be processed using one or more machine vision techniques to identify the item within the image information416. Upon identifying the item within the image information, the control unit may augment the image information416with some indication418of the item to be retrieved. In some embodiments, the indication418may comprise a boundary, outline of the item, highlighting of the item, or any other suitable indication of the item's location.

Once the operator of remote manipulation device406has completed the requested manual operation, the control unit may sever the connection between the remote manipulation device406and the robotic picking unit412. The control unit may then identify the next request for manual operation in the queue to be performed by the operator of the remote manipulation device406. The process in this example may be repeated multiple times.

In some embodiments, the operator may be located remote from the inventory fulfillment center in which the disclosed inventory management system is located. For example, the operator may be located in a different country than the fulfillment center. In some embodiments, the control unit may select a particular operator and/or remote manipulation device to perform a particular manual operation. In some embodiments, particular operators may be selected by a control unit based on information associated with the operator. For example, a particular operator may be selected to perform a manual operation based on time zone information, language, experience, labor costs, experience on similar issues, or any other suitable factor. In some embodiments, operators may be compensated for their participation in the disclosed inventory management system. For example, an operator may be employed by an owner of the inventory management system. In another example, an operator may be compensated based on the manual operations performed. In this example, the operator may be a member of a crowdsourcing website in which users are able to select and perform various manual operations.

In some embodiments, multiple operators may use remote manipulation devices to operate a virtual robot that represents the robotic unit within a virtual reality environment. The operators may be given a scenario in which a representation of the item to be manipulated is presented in the virtual environment in a pose similar to that of the actual item. In these embodiments, each operator's approach to retrieving or otherwise manipulating the item may be recorded by the system. The system may then determine which approach used by the operators was most effective. For example, the system may identify a number of metrics to be associated with each approach in order to determine which approach results in the best time for the manipulated item, the best placement pose for the item, the fewest manipulation actions, or any other suitable metric. In some embodiments, each metric may be given a weight, and a “best” approach may be identified by summing each of the metrics according to their assigned weights. In some embodiments, each approach may be recorded and presented to a user or a number of users, who may be given the ability to vote on a particular approach. In at least some embodiments of the inventory management system described herein, a best approach for a manual operation or multiple manual operations may be identified in this manner prior to operation of a robotic unit. Once the robotic unit encounters the manual operation for which the best approach has been identified, the robotic unit may perform the best approach automatically (e.g., without operator intervention) to manipulate the item.

FIG. 5depicts a second example of a hybrid tele-operated inventory management system in operation. InFIG. 5, an inventory management system may include at least one stationary robotic picking unit502and a number of mobile drive units504configured to position storage bins506in front of the robotic picking unit502. The operation of mobile drive units in this fashion is described in U.S. Pat. No. 7,912,574, issued Mar. 22, 2011, entitled “System and Method for Transporting Inventory Items,” which is herein incorporated by reference in its entirety.

In some embodiments, the inventory management system may receive an order for a number of items to be shipped to a consumer. The inventory management system may select a shipping container508that is appropriate for the order (e.g., of sufficient size to hold the requested items) which may be placed in proximity to the stationary robotic picking unit502. During fulfillment of the order, a storage bin506containing at least a first item to be placed in the shipping container508may be transported to the robotic picking unit502automatically by a mobile drive unit504. During the transportation of the storage bin506to the robotic picking unit502, the robotic picking unit may be inactive. Upon arrival of the storage bin506, the robotic picking unit502may locate the requested item and determine whether that item may be retrieved in an automated manner. If the robotic picking unit502is unable to retrieve the item automatically, it may position itself so that an EOAT of the robotic picking unit502is in proximity to the requested item. The robotic picking unit502may then submit a request to the control unit for manual operation to retrieve the item. It should be noted that a robotic picking unit may be mounted in a number of ways. For example, the robotic picking unit may be mounted on the ceiling or a wall. In some embodiments, the robotic picking unit may comprise multiple robotic units. For example, robotic picking unit502may include both a robotic unit mounted on the floor and a robotic unit mounted on the ceiling.

Similar to the embodiment of the system described with respect toFIG. 4above, the request for manual operation may enter a queue of similar requests. Once the request has been assigned to an operator, the operator may take control of the robotic picking unit502to remove the requested item from the storage bin506in a manner similar to that described above. Once this has been completed, the storage bin506may be returned to its original location (or another appropriate location) automatically and the operator may be disconnected from the robotic picking unit502and assigned the next manual operation request in the queue. However, in some embodiments, the operator may continue to operate the robotic picking unit502throughout a period of time (e.g., a shift) or for the duration of an order. For example, a number of storage bins506may have been positioned close to (e.g., lined up at) the robotic picking unit502. As an appropriate item is removed from each storage bin506, that storage bin506may be removed and another storage bin may be moved into its place. This may continue until each of the storage bins506has been dealt with.

In some embodiments, the storage locations506may be fitted with one or more cameras. In these embodiments, the operator may be provided access to image information captured by the one or more cameras in addition to, or instead of, image information received from the robotic unit. For example, the operator may be provided a feed of image information received from the robotic unit that is augmented with a window that depicts a feed of image information received from the storage bin506. It should be noted that the implementation of certain embodiments described herein allow for a much more extended reach than is typically available to a human actor. Accordingly, the storage bins506may be much taller than those utilized in conventional systems. This may result in a more optimal use of available storage space over conventional inventory management systems.

In some embodiments, once each of the items is determined to have been placed in the shipping container508, the inventory management system may send a unit to retrieve the shipping container. In some embodiments, the shipping container508may be replaced by a shipping container associated with the next order in an order fulfillment queue. In some embodiments, a quality assurance (QA) unit or person may assess the items within the shipping container508upon its retrieval. For example, a QA unit may check a status of the items within the shipping container508by ensuring that they are properly packed, all accounted for, unbroken, etc. In some embodiments, a QA unit may provide feedback to the control unit regarding the status of items in the shipping container508. In some embodiments, the status may be used to obtain metrics associated with a particular operator. The control unit may use these obtained metrics to make future manual operation assignments.

FIG. 6depicts an example of a process by which a robotic picking unit may perform a second task while awaiting initiation of a manual operation in accordance with at least some embodiments. InFIG. 6, the second task may be identified by the robotic picking unit602(e.g., the second task may be a task other than the current task in a list of tasks to be completed by the robotic picking unit602) or the second task may be identified by a control unit202as described with respect toFIG. 2.

In some embodiments, the robotic picking unit602may identify a first item604to be retrieved or otherwise manipulated. In some embodiments the robotic picking unit may come into proximity of a storage bin606that includes the item604. For example, the robotic picking unit602may traverse to the storage bin606, or the storage bin606may be transported to the robotic picking unit602. Upon coming into proximity of the storage bin606, the inventory management system may determine that the item604is unable to be manipulated without operator intervention. Upon making that determination, the inventory management system may determine whether it is appropriate to perform a second task. For example, the inventory management system may have insufficient manipulation data regarding a particular pose that the item604is stored in. In this example, upon determining that a manual operation is required with respect to a current item604, the inventory system may calculate an amount of time that will likely transpire before an operator is able to initiate a manual operation (e.g., a wait time). The inventory management system may then determine whether a second task can be performed within this amount of time.

A second task to be performed by the robotic picking unit602may be any task identified by the inventory management system as being able to be performed by the robotic picking unit602. The second task may be a task subsequent to a current task in a list of tasks to be completed by the robotic picking unit602or it may be a task originally intended to be assigned to a different robotic picking unit. In some embodiments, the inventory management system may determine that the robotic picking unit602is proximate to a second item608which can be manipulated and that also needs to be retrieved or otherwise manipulated (e.g., the second item608may be relevant to fulfillment of the same order or a different order). In his example, the second item608may be located in the same storage bin606or it may be location in a second storage bin610. The inventory management system may provide instructions to the robotic picking unit602to manipulate the second item608. Once the second task has been completed by the robotic picking unit602, the robotic picking unit602may be configured to reposition itself in anticipation of the manual operation.

In some embodiments, once the second task has been completed by the robotic picking unit602, the inventory management system may determine that there is sufficient time remaining in the amount of time before initiation of the manual operation to complete a third or subsequent task. In this example, the inventory management system and/or robotic picking unit602may be configured to execute that task. For example, the second storage bin610may be transported away by a mobile drive unit and replaced by a third storage bin that includes a third item to be manipulated. In this example, the robotic picking unit602may be instructed to manipulate the third item.

FIG. 7depicts a flow chart illustrating a process for operating an inventory management system using remote operators in accordance with at least some embodiments. Some or all of the process700(or any other processes described herein, or variations, and/or combinations thereof) may be performed under the control of one or more computer systems configured with executable instructions and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) executing collectively on one or more processors, by hardware or combinations thereof. The code may be stored on a computer-readable storage medium, for example, in the form of a computer program comprising a plurality of instructions executable by one or more processors. The computer-readable storage medium may be non-transitory. Process700may be performed by an example robotic picking unit (RPU)204, a control unit202, and a remote manipulation device242, each of which is depicted with respect toFIG. 2.

Process700may begin at702, when a request is received by a control unit202with respect to an item managed by the inventory management system. In some embodiments, the request for an item may take the form of an order placed by a consumer for an item managed by the inventory management system. In some embodiments, an order placed by a consumer may contain multiple items. In some embodiments, the inventory management system may handle each of the items in the order separately by separating the order into multiple item requests. In some embodiments, the inventory management system may handle the order as a whole by generating an item request including multiple items at potentially multiple storage locations.

At704, the control unit may identify a storage location associated with the item as well as an appropriate RPU. The control unit may identify the storage location by querying inventory data stored in a database. In some embodiments, the control unit may also have stored an indication of the type of EOAT needed to interact with a particular item or type of item. For example, if the item is in a relatively-flat plastic packaging, then it may be appropriate to select a robotic unit that has equipped a suction cup manipulator tool (e.g., a vacuum suction cup pneumatic manipulator). In another example, if the item is relatively elastic then it may be appropriate to select a robotic unit that has equipped a grasping item manipulator tool. In some embodiments, the control unit may consult with an item grasp database to identify an appropriate robotic unit. The selection of an appropriate robotic unit may also be made based on any suitable factor relevant to the item to be manipulated (e.g., a size or weight of the item). In some embodiments, the selection of an appropriate robotic unit may also be made based on a location for each of the available robotic units.

At706, the control unit may generate retrieval instructions to be executed by the selected RPU. In some embodiments, the instructions may include a set of storage locations and items within those storage locations. In some embodiments, the instructions may include routing information to be used by the robotic unit. For example, the control unit may determine a current location of the robotic unit as well as a location of a storage location to which the robotic unit is to traverse. The control unit may then generate routing instructions that, when executed by the robotic unit, will cause the robotic unit to traverse to the identified storage location. In some embodiments, routing instructions generated by the control unit may account for routes being traversed by other robotic units within the inventory management system. In some embodiments, the instructions may include grasp data or information related to manipulation of a particular item. For example, the instructions may include a subset of instructions that, when executed by the robotic unit, causes the robotic unit to retrieve a particular item or type of item from a storage location in an automated fashion.

At708, the RPU204may receive the retrieval instructions from the control unit202. In some embodiments, the instructions may be transmitted to the robotic unit via a wireless network connection. For example, each of the robotic units may be in communication with the control unit via a wireless network. In some embodiments, the robotic unit may receive the set of instructions via a direct physical connection. For example, a stationary robotic unit may share a direct communication channel with a control unit, over which communications may be routed. In another example, a robotic unit may receive instructions from the control unit while the robotic unit is docked (e.g., at a charging station) via a direct physical connection.

At710, the RPU may execute a subset of the instructions intended to be executed in an automated manner. For example, the RPU may receive instructions to traverse to a first storage location and retrieve a first item. Even if the RPU cannot retrieve the item automatically, it may automatically traverse to the storage location. In some embodiments, each RPU may include a separate routing module configured to generate instructions for traversing to the identified storage location. In some embodiments, the RPU may receive routing instructions from the control unit to cause the RPU to traverse to a location.

At712, the RPU may determine whether there are additional instructions to be executed. If no additional instructions are to be executed by the RPU (e.g., the requested item has been retrieved and stowed in the appropriate place), the RPU may end the retrieval process and await new retrieval instructions at714. Otherwise, the RPU may determine whether the additional instructions are to be performed manually at716. If the additional instructions may be automated, then the process700loops back to710. Otherwise, the RPU may generate a request for manual operation and submit the generated request to the control unit202at718. In some embodiments, the request is added to a queue of such requests submitted by other RPUs within the inventory management system. In some embodiments, the control unit may determine that the RPU has arrived at a specified location, or has otherwise fulfilled a predetermined condition, and may generate a request for manual operation based on the RPU having met that predetermined condition. The RPU may, prior to fulfillment of the request for manual operation, position itself in front of the item for which manual operation is required.

At720, the control unit202receives the generated request for manual operation (e.g., when it reaches the top of the queue) and determines whether an operator is available to perform the requested manual operation. In some embodiments, the control unit may assign the received request to the next available operator. In some embodiments, the control unit may select an operator that is appropriate for the type of manual operation requested. For example, the control unit may determine that the EOAT installed on a particular RPU should only be operated by a subset of operators. In this example, the control unit may assign the received request for manual operation to the first available operator from that subset.

At722, the control unit may generate a request to a remote manipulation device242associated with the identified operator. At724, the remote manipulation device242may receive the request for manual operation from the control unit202. Based on the received request, the remote manipulation device242may establish a communication session with the RPU at726. As depicted in process700, the communication session may be routed through, and facilitated by, the control unit202at728. In some embodiments, the control unit202may append or augment information transmitted between the remote manipulation device242and the RPU204. For example, the control unit202may add an overlay to image information received from the RPU which depicts a descriptor (e.g., an image) of an item to be manipulated by the operator of the remote manipulation device242. The augmented image information may then be forwarded to the remote manipulation device242.

At730, the remote manipulation device242may enable manual operation of the RPU by the operator. For example, the remote manipulation device242may receive operator input and may translate that operator input into computer-executable instructions to be executed by the robotic unit. In some embodiments, the computer-executable instructions may activate various actuators (e.g., motors) of the RPU to move the RPU, move a mechanical arm on the RPU, effect an EOAT of the RPU (e.g., open and/or close a grasping tool), or cause the RPU to perform any other suitable action.

At732, the remote manipulation device242may receive an indication that the manual operation has been completed. For example, the operator of the remote manipulation device242may indicate that the request has been completed. In some embodiments, the operator may press a button or take another suitable action to indicate the completion of the manual operation.

Once an indication has been received that the manual operation has been completed at734, the RPU may execute a second subset of the instructions intended to be executed in an automated manner. The process700may be completed upon determining that each of the instructions provided to the RPU have been completed in either an automated or manual fashion. Once this is accomplished, the control unit202may cause a fulfillment system to fulfill the received order at736. For example, the control unit, upon determining that each item to be fulfilled in the order has been retrieved and placed in a shipping container, may cause the shipping container to be sealed and/or placed in an outgoing parcel queue.

FIG. 8depicts a flow diagram illustrating a process for fulfilling an order using a robotic unit in an automated and manual fashion in accordance with at least some embodiments. Process800may be performed by a robotic unit102as depicted inFIG. 1.

Process800may begin at802, when instructions are received from a control unit. In some embodiments, the instructions may include one or more storage locations and one or more items to be retrieved from the one or more storage locations. In some embodiments, the instructions may include instructions for manipulating the item in an automated manner. For example, the control unit may learn (e.g., by observing operator interactions) an appropriate technique for retrieving an item. Once this technique is learned, the control unit may store instructions that cause a robotic unit to act in accordance with that technique.

At804, the robotic unit may automatically perform a subset of the received instructions. For example, the instructions may include an indication of a storage location from which one or more items are to be retrieved. In this example, the robotic unit may automatically traverse to the indicated storage location. The robotic unit may also automatically identify the item within the storage location and position an EOAT proximate to that item. For example, the robotic unit may scan barcodes associated with one or more items within the storage location to identify the item to be retrieved. In another example, the robotic unit may use machine vision with one or more item recognition techniques to identify the item. In some embodiments, the robotic unit may identify a pose, (e.g., a position) of the item within the storage location.

At806, the robotic unit may determine that manual operation of the robotic unit is necessary. For example, the robotic unit may be unable to identify the item within the storage location, which may result in requiring an operator to locate the item. In another example, the robotic unit may identify the item within the storage location, but may not have appropriate instructions for handling the item. In another example, the storage location may be too tightly packed for the robotic unit to manipulate the item without assistance.

At808, the robotic unit may submit a request for performance of the manual operation. In some embodiments, the robotic unit may submit the request to a generic queue for such requests. In some embodiments, the robotic unit may submit the request to a queue based on a type of manual operation to be performed. For example, the robotic unit, upon determining that a manual operation is necessary to retrieve an item from a storage location, may submit a request to a queue intended for retrieval of items. In another example, the robotic unit, upon determining that the item is to be placed within a shipping container or other storage location, may submit the request to a queue intended for packing of items. In some embodiments, operators may specialize in a particular type of manual operation. For example, some operators may specialize in retrieval of items from storage locations whereas other operators may specialize in packing items into storage locations. In this example, operators may be assigned requests for manual operations based on specialties associated with those operators.

At810, the robotic unit may enable manual operation of its EOAT and/or other functions. For example, the robotic unit may receive operator input (e.g., received via a remote manipulation device) which causes various actuators of the robotic unit to be activated. For example, the robotic unit may move, turn, reposition a mechanical arm, effect an EOAT, or perform any other suitable action. At812, the robotic unit may receive an indication that the manual operation has been completed. In some embodiments, this may be a severing of the communication session between the robotic unit and the remote manipulation device.

At814, the robotic unit may automatically traverse to a fulfillment location. The robotic unit may deliver one or more of the retrieved items to the fulfillment location at816. For example, the robotic unit may traverse to the fulfillment location and place the retrieved item into a shipping container. In another example, the robotic unit may carry the shipping container to a number of storage locations that each contain at least one of the items to be fulfilled in an order. The robotic unit may, upon completing the order by retrieving each of the items to be fulfilled, place the shipping container in a fulfillment location for further processing and/or shipment to a consumer. The control unit may, upon receiving an indication that an item has been retrieved from a storage location, update inventory information to reflect that the item is no longer available at that storage location.

FIG. 9depicts a flow diagram illustrating a process for enabling an operator to complete a series of manual operations with multiple robotic units in accordance with at least some embodiments. Process900may be performed by a control unit202as depicted inFIG. 2.

Process900may begin at902, when a request is received by the control unit from a robotic unit for manual operation. At904, the control unit may submit the received request to a request queue, within which the request will be prioritized. The request may be processed according to its determined priority. This process may be repeated each time that a request is received for manual operation from a robotic unit.

During operation of the inventory management system described, the control unit may identify a request that is next in the request queue at906(e.g., a request in the request queue having the highest priority). In some embodiments, requests may be prioritized based on one or more characteristics of the requests. For example, requests of a certain type or category may be prioritized. In another example, requests associated with expedited orders may be prioritized.

Once a request reaches the highest priority in a queue, the control unit may determine an appropriate operator to handle the request at908. In some embodiments, the appropriate operator may be determined based on a skillset or certification of the operator. In some embodiments, the appropriate operator may be determined based on availability of the operator. In some embodiments, an appropriate operator may be selected based on a geographic location or time zone of the operator. In some embodiments, an appropriate operator may be selected based on capabilities, or a type, of remote manipulation device operated by the operator. In some embodiments, the control unit may determine an appropriate operator based on the operator indicating that he or she will handle the request.

Once an appropriate operator has been determined, the control unit may identify a remote manipulation device associated with that operator at910. For example, the control unit may query a database of operator/remote manipulation device mappings to identify an appropriate remote manipulation device. In some embodiments, the control unit may transmit a request to an internet protocol (IP) address associated with the operator.

The control unit may establish a communicative connection between the robotic unit and the remote manipulation device at912. In some embodiments, establishing the communicative connection may comprise relaying image information received by the robotic unit to the remote manipulation device and relaying commands receive by the remote manipulation device (e.g., from the operator) to the robotic unit. The commands may cause the robotic unit to perform the manual operation.

At914, the control unit may receive an indication that the manual operation has been completed. Upon receiving this indication, the control unit may sever the communicative connection between the robotic unit and the remote manipulation device. In some embodiments, the steps described with respect to906through914may be repeated on a continuous basis. In some embodiments, the control unit may constantly identify pending requests, determine which operator is available to complete the request, assign that request to the determined operator (e.g., by establishing a connection with a remote manipulation device for that operator), and move on to the next pending request.

Embodiments of the disclosure provide for a number of technical advantages over conventional inventory management systems. For example, embodiments of the disclosure enable remote operation of an inventory fulfillment center from anywhere in the world, which enables virtualization of tasks which previously required operators to be on-site. Virtualization of these tasks provides the ability to take advantage of differences in labor costs or labor availability based on location. Additionally, virtualization of tasks enhances safety for worker who no longer need to be in the same area as mobile robotic units.

Additionally, the disclosure results in significant improvements to the functionality of the inventory management system as a whole. For example, by automating traversal to and from storage locations, and by switching to manual operation of each device when it is ready to be controlled (e.g., those from which requests have been received), downtime for each operator is minimized or eliminated. Minimizing downtime enables more orders to be fulfilled by each operator of the inventory management system. Additionally, the described system may enable each operator to control each device, so that no particular operator is causing a bottleneck or slowing the system as a whole.

Furthermore, by implementing embodiments of the disclosure, an operator of a fulfillment center maybe able to use taller storage bins relative to fulfillment centers that rely only on human pickers, as robotic picking units may be configured to have higher reach than human pickers. Accordingly, fulfillment centers using embodiments of the disclosure may maximize available storage space by increasing items stored per square foot of fulfillment center relative to fulfillment centers that rely only on human pickers.