Management of sensor failure in a facility

During operation, a facility may utilize many sensors, such as cameras, to generate sensor data. The sensor data may be processed by an inventory management system to track objects, determine the occurrence of events at the facility, and so forth. At any given time, some of these sensors may fail to provide timely data, may fail to provide any data, may generate inaccurate data, and so forth. Described are techniques to determine failure of sensors and adjust operation of the inventory management system to maintain operability during sensor failure.

BACKGROUND

Retailers, wholesalers, and other product distributors typically maintain an inventory of various items that may be ordered, purchased, leased, borrowed, rented, viewed, and so forth, by clients or customers. For example, an e-commerce website may maintain inventory in a fulfillment center. When a customer orders an item, the item is picked from inventory, routed to a packing station, packed, and shipped to the customer. Likewise, physical stores maintain inventory in customer accessible areas (e.g., shopping area), and customers can pick items from inventory and take them to a cashier for purchase, rental, and so forth. Many of those physical stores also maintain inventory in a storage area, fulfillment center, or other facility that can be used to replenish inventory located in the shopping area or to satisfy orders for items that are placed through other channels (e.g., e-commerce). Other examples of entities that maintain facilities holding inventory include libraries, museums, rental centers, and so forth. In each instance, for an item to be moved from one location to another, it is picked from its current location and transitioned to a new location. It is often desirable to monitor the movement of inventory, users, and other objects within the facility.

DETAILED DESCRIPTION

This disclosure describes systems and techniques for maintaining operation at a materials handling facility in the event one or more sensor devices fail. The sensor data may be acquired at a materials handling facility (facility) or other setting from sensor devices. The facility may include, or have access to, an inventory management system. The inventory management system may be configured to maintain information about items, users, condition of the facility, and so forth. For example, the inventory management system may maintain data indicative of what items a particular user is ordered to pick, location of the particular user, availability of a user providing support services to others, requests for assistance, environmental status of the facility, and so forth. The inventory management system, or another system, may generate this data based on sensor data, such as images acquired from imaging sensors such as cameras. For example, the images may be used to identify an object such as a user or item, track the object, and so forth.

Sensor devices include one or more sensors, such as imaging sensors, depth sensors, and so forth. The sensor devices may be located at different points within the facility to acquire sensor data suitable for particular uses. The sensor data may comprise still images, video, depth data, and so forth. For example, sensor devices that include imaging sensors may be used to acquire images that are processed to identify a user. Some sensors may have a field of view (FOV) that encompasses a region at the facility, such as a volume in an aisle, a portion of a shelf, and so forth. In some implementations, the sensor devices may have different optical or electronic characteristics based at least in part on the intended usage. For example, some sensor devices may be equipped with imaging sensors including lenses to acquire close-up images, while others are equipped to acquire images from farther away.

Other sensors may have an area in which they are able to sense or detect input, such as a weight sensor under a particular portion of a floor, an instrumented auto-facing unit providing information about a quantity of items on a shelf, and so forth.

The facility may implement any number of sensor devices to support operation. For example, hundreds or thousands of sensor devices may be mounted within the facility to gather information about items as they are added or removed from inventory locations by users, provide information about the location and identity of users in the facility, and so forth.

The various sensors and computer systems described herein may collect a wide variety of data relating to users of systems or facilities employing such sensors or computer systems. The data relating to users may include, for example, images, video, location information, travel patterns, personal identification information, transaction history information, user preferences, and the like. The sensors, systems, and techniques described herein would be typically configured to process this information in a manner that ensures compliance with all appropriate laws, regulations, standards, and the like.

While the probability of a malfunction of a particular sensor device is low, given the large number of sensor devices that may be deployed in the facility, the probability of failure somewhere in the facility at any given time becomes non-trivial. In the event a sensor device fails, sensor data from the region of the facility serviced by that sensor device may be lost or less reliable. For example, the sensor device may fail by ceasing to send data or may send incorrect data. If the inventory management system attempts to make determination based on incomplete or incorrect data, those determinations may be suspect. For example, incomplete data may result in an incorrect identification of a user, place the user at an incorrect location, identify an incorrect quantity of items picked or placed, and so forth.

Described are systems and processes to mitigate the impact of sensor device failure on the operation of the inventory management system. Failures may be determined based on sensor status data, analysis of the sensor data, or both.

The sensor device may include onboard circuitry configured to generate sensor status data. The sensor status data is indicative of operation of the sensor device, as distinct from the output of the sensor itself that generates sensor data. For example, the sensor status data may indicate one or more of temperature of the sensor device's electronics, processor load, utilization of the processor, memory usage, direction or orientation of the sensor's FOV, and so forth.

A service may access the sensor status data and determine if one or more values of the sensor status data exceed a threshold value. For example, the sensor status data may indicate an average number of frames of video data provided in the past 10 seconds. A threshold value may specify that, in 10 seconds, the sensor device is to deliver 300 frames of video data (10 seconds of frames at 30 frames per second). If the sensor status data indicates that the average number of frames delivered in 10 seconds is 50 frames, that particular sensor device may be deemed to be in a fail state.

The sensor data itself may be processed to determine failure of a sensor device. For example, depth data provided by a sensor device having a depth sensor may indicate a distance of 999 meters. Given that a known maximum distance in the facility is 100 meters, the sensor device that generated the sensor data indicative of “999 meters” may be designated as in a fail state.

Once the sensor device has been determined to exhibit a failure, various techniques may be used to mitigate the impact of the failure on the operation of the facility. For example, based on the use of sensor data from a failed sensor device, a confidence value of output data generated by the inventory management system may be decreased. In another example, data from the failed sensor may be disregarded, and data from other sensors may be used instead to generate output data. For example, during normal operation, sensor devices with depth sensors may be used to generate three-dimensional point cloud data. The three-dimensional point cloud data comprises information about the distance (relative to the sensor) to objects in the FOV of the sensor. This three-dimensional point cloud data may be used to identify objects such as users, items for sale, totes, inventory locations, and so forth. Upon failure, some of this three-dimensional point cloud data may be absent or incorrect. Other sensor devices, such as imaging sensors that provide two-dimensional images, may be used by the inventory management system to mitigate the failure. Continuing the example, image data from the imaging sensors may be processed using object recognition software to identify an object, determine an approximate location within the region, and so forth. This additional processing of the image data may be more computationally intensive than that used to process the three-dimensional point cloud. For example, the three-dimensional point cloud data may be a lower resolution data set (requiring fewer computational resources) compared to a high-resolution image from a high definition camera (requiring more computational resources). In another example, the image data from several imaging sensors at several different locations in the facility having a FOV that covers at least a part of the region affected by the failed sensor device may be processed using stereo-vision techniques to determine the location of the object in the facility. To reduce the overall computer capacity and associated costs, the use of these more computationally intensive techniques may be limited to these failure mitigation situations.

By using these and other techniques described herein, the inventory management system may determine operational status of the sensor devices and may adjust to continue providing services to the users of the facility as sensor devices fail or are restored to normal function. As a result, the overall operation of the facility and the user experience may be improved.

Illustrative System

An implementation of a materials handling system100configured to store and manage inventory items is illustrated inFIG. 1. A materials handling facility102(facility) comprises one or more physical structures or areas within which one or more items104(1),104(2), . . . ,104(Q) may be held. As used in this disclosure, letters in parenthesis such as “(Q)” indicate an integer value. The items104comprise physical goods, such as books, pharmaceuticals, repair parts, electronic gear, and so forth.

The facility102may include one or more areas designated for different functions with regard to inventory handling. In this illustration, the facility102includes a receiving area106, a storage area108, and a transition area110.

The receiving area106may be configured to accept items104, such as from suppliers, for intake into the facility102. For example, the receiving area106may include a loading dock at which trucks or other freight conveyances unload the items104.

The storage area108is configured to store the items104. The storage area108may be arranged in various physical configurations. In one implementation, the storage area108may include one or more aisles112. The aisle112may be configured with, or defined by, inventory locations114on one or both sides of the aisle112. The inventory locations114(1),114(2), . . . ,114(L) may include one or more of shelves, racks, cases, cabinets, bins, floor locations, slatwalls, pegboards, trays, dispensers, or other suitable storage mechanisms. The inventory locations114may be affixed to the floor or another portion of the facility's102structure. The inventory locations114may also be movable such that the arrangements of aisles112may be reconfigurable. In some implementations, the inventory locations114may be configured to move independently of an outside operator. For example, the inventory locations114may comprise a rack with a power source and a motor, operable by a computing device to allow the rack to move from one location within the facility102to another. Continuing the example, the inventory location114may move from one aisle112to another, from one location within an aisle112to another, and so forth. In another example, the inventory locations114may be configured to translate, rotate, or otherwise move relative to the facility102.

One or more users116(1),116(2), . . . ,116(U) and totes118(1),118(2), . . . ,118(T), or other material handling apparatuses, may move within the facility102. For example, the user116may move about within the facility102to pick or place the items104in various inventory locations114, placing them on the tote118for ease of transport. The tote118is configured to carry or otherwise transport one or more items104. For example, the totes118may include carts, baskets, bags, bins, and so forth. In some implementations, the tote118may incorporate one or more inventory locations114. For example, the tote118may include a bin, basket, shelf, and so forth.

Instead of, or in addition to the users116, other mechanisms such as robots, forklifts, cranes, aerial drones, conveyors, elevators, pipes, and so forth, may move items104about the facility102. For example, a robot may pick the item104from a first inventory location114(1) and move the item104to a second inventory location114(2).

One or more sensors120may be configured to acquire information in the facility102. The sensors120may include, but are not limited to, imaging sensors, weight sensors, proximity sensors, radio frequency (RF) receivers, microphones, temperature sensors, humidity sensors, vibration sensors, and so forth. The sensors120may be stationary or mobile, relative to the facility102. For example, the inventory locations114, the totes118, or other devices such as user devices may contain sensors120to acquire sensor data. The sensors120are discussed in more detail below with regard toFIG. 2. Some sensors may have a field of view (FOV)122that encompasses a region at the facility102, such as a volume in an aisle112, a portion of an inventory location114, and so forth.

One or more of the sensors120may be supported by, coupled to, or incorporated within, a sensor device124. The sensor device124may comprise a computing device, one or more sensors120, and so forth. One or more sensors120associated with the sensor device124, electronics or other components associated with the sensor device124itself, and so forth, may fail.

In some implementations, the sensor device124may process “raw” sensor data obtained from the sensor120and provide as output the sensor data132. In some implementations, the sensor data132may have a smaller file size than the “raw” sensor data. For example, the raw image data from an imaging sensor120(1) may comprise approximately 40 megabits (Mb), while the sensor data132produced by the sensor device124may be processed, compressed, and so forth, to approximately 1.2 Mb. The sensor device124is discussed in more detail below with regard toFIG. 4.

While the storage area108is depicted as having one or more aisles112, inventory locations114storing the items104, sensors120, and so forth, it is understood that the receiving area106, the transition area110, or other areas of the facility102may be similarly equipped. Furthermore, the arrangement of the various areas within the facility102is depicted functionally rather than schematically. In some implementations, multiple different receiving areas106, storage areas108, and transition areas110may be interspersed rather than segregated.

The facility102may include, or be coupled to, an inventory management system126. The inventory management system126is configured to interact with users116or devices such as sensors120, robots, material handling equipment, computing devices, and so forth, in one or more of the receiving area106, the storage area108, or the transition area110.

The facility102may be configured to receive different kinds of items104from various suppliers, and to store them until a customer orders or retrieves one or more of the items104. A general flow of items104through the facility102is indicated by the arrows ofFIG. 1. Specifically, as illustrated in this example, items104may be received from one or more suppliers, such as manufacturers, distributors, wholesalers, and so forth, at the receiving area106. In various implementations, the items104may include merchandise, commodities, perishables, or any suitable type of item104, depending on the nature of the enterprise that operates the facility102.

Upon being received from a supplier at the receiving area106, the items104may be prepared for storage. For example, items104may be unpacked or otherwise rearranged. The inventory management system126may include one or more software applications executing on a computer system to provide inventory management functions. These inventory management functions may include maintaining information indicative of the type, quantity, condition, cost, location, weight, or any other suitable parameters with respect to the items104. The items104may be stocked, managed, or dispensed in terms of countable, individual units or multiples, such as packages, cartons, crates, pallets, or other suitable aggregations. Alternatively, some items104, such as bulk products, commodities, and so forth, may be stored in continuous or arbitrarily divisible amounts that may not be inherently organized into countable units. Such items104may be managed in terms of measurable quantity such as units of length, area, volume, weight, time, duration, or other dimensional properties characterized by units of measurement. Generally speaking, a quantity of an item104may refer to either a countable number of individual or aggregate units of an item104or a measurable amount of an item104, as appropriate.

After arriving through the receiving area106, items104may be stored within the storage area108. In some implementations, like items104may be stored or displayed together in the inventory locations114such as in bins, on shelves, hanging from pegboards, and so forth. In this implementation, all items104of a given kind are stored in one inventory location114. In other implementations, like items104may be stored in different inventory locations114. For example, to optimize retrieval of certain items104having frequent turnover within a large physical facility102, those items104may be stored in several different inventory locations114to reduce congestion that might occur at a single inventory location114.

When a customer order specifying one or more items104is received, or as a user116progresses through the facility102, the corresponding items104may be selected or “picked” from the inventory locations114containing those items104. In various implementations, item picking may range from manual to completely automated picking. For example, in one implementation, a user116may have a list of items104they desire and may progress through the facility102picking items104from inventory locations114within the storage area108and placing those items104into a tote118. In other implementations, employees of the facility102may pick items104using written or electronic pick lists derived from customer orders. These picked items104may be placed into the tote118as the employee progresses through the facility102.

After items104have been picked, they may be processed at a transition area110. The transition area110may be any designated area within the facility102where items104are transitioned from one location to another or from one entity to another. For example, the transition area110may be a packing station within the facility102. When the item104arrives at the transition area110, the item104may be transitioned from the storage area108to the packing station. Information about the transition may be maintained by the inventory management system126.

In another example, if the items104are departing the facility102, a list of the items104may be obtained and used by the inventory management system126to transition responsibility for, or custody of, the items104from the facility102to another entity. For example, a carrier may accept the items104for transport with that carrier accepting responsibility for the items104indicated in the list. In another example, a customer may purchase or rent the items104and remove the items104from the facility102.

The inventory management system126may include a failure management system128. The failure management system128may be configured to determine which sensor devices124, or sensors120therein, have failed, are operational, or both. Failure of the sensor device124may result in a sensor blackout region. The sensor blackout region may comprise an area or volume at the facility102at which phenomena that would otherwise be detected by an operational sensor120go undetected or erroneously detected. For example, the sensor blackout region may comprise a portion of the facility102for which an inoperable imaging sensor120(1) is unable to obtain an image.

In some implementations, failure may be defined as operation or performance of the sensor device124, or components thereof such as the sensor120, that is beyond the threshold value. For example, failure may comprise the delivery of images acquired by imaging sensors at less than the desired frame rate. In another example, failure may comprise the sensor device124, or sensor120coupled thereto, being completely inoperable. For example, a sensor device124that has experienced complete loss of power would be deemed to be in a failed state. Failures may be the result of malfunctioning software, malicious attack, physical component failure, human error, outside agency, and so forth. For example, a sensor device124may fail due to a tote118bumping or brushing against the sensor device124and changing the orientation so that the FOV122is pointing in a direction other than that intended for operation of the inventory management system126.

The failure management system128may also initiate one or more actions to mitigate failures. For example, the failure management system128may determine a region of the facility102that is impacted by the failure and may decrease the confidence value associated with that region.

During operation, the failure management system128may access one or more of physical layout data130, sensor data132, sensor status data134, or threshold data136. The physical layout data130comprises information about the physical configuration of the facility102or portions thereof. For example, the physical layout data130may include electronic representations of the physical structures in the facility102, such as computer aided design (CAD) data of the aisle112configurations, inventory locations114, information about which items104are in what inventory locations114, real coordinates of the sensor devices124, orientation of the FOV122of the sensor devices124, and so forth. The physical layout data130may include information about the presence of walls; heating, ventilation, and air conditioning (HVAC) equipment; location of doors and windows; and so forth.

In some implementations, the physical layout data130may indicate a location or position of a sensor device124with respect to one or more of another sensor device124or the facility102. For example, the physical layout data130may indicate that the sensor device124(1) is adjacent to the sensor device124(3). Sensor devices124may be deemed to be adjacent when they are within a threshold distance of one another, have FOVs122that share at least a portion of the same scene, and so forth.

The sensor data132may comprise information acquired from, or based on, the one or more sensors120. For example, the sensor data132may comprise 3D information about an object in the facility102as acquired by the depth sensors120(2) or weight data as acquired by the weight sensors120(6). In some implementations, sensor data132may include metadata such as information about imaging sensor settings used to acquire image data, filename, and so forth. The metadata associated with the sensor data132may be particularly associated with the acquisition of that sensor data132. For example, the metadata may include a timestamp comprising time information indicative of the acquisition of the sensor data132. The timestamp may be generated using time data from an internal clock of the sensor device124. For example, the timestamp may comprise date and time information as obtained from the clock of the sensor device124.

In comparison, sensor status data134is indicative of operation of the sensor device124, or the components thereof, as distinct from the output of the sensor120itself that generates sensor data132. The sensor device124may include onboard circuitry, software, and so forth, to generate the sensor status data134. For example, the sensor status data134may indicate one or more of temperature of one or more components such as the electronics of the sensor device124, relative humidity within the sensor device124, processor load, memory usage, direction or orientation of the FOV122, available electrical power, status of a network connection used by the sensor, and so forth. For example, the status of the network connection may indicate maximum throughput, number of dropped packets, packet retry count, connection speed, and so forth. The sensor status data134may include an explicit indicator of a failure as determined by the sensor device124. For example, the sensor status data134may include a fault code or trouble report generated by the sensor device124.

In some implementations, the sensor status data134may be generated based on information received by the inventory management system126, or by another system. For example, the sensor status data134may be generated at least in part by monitoring traffic on the network associated with the sensor device124. For example, the sensor status data134may comprise a time series indicating a count of packets transmitted by the sensor device124to the network within a particular period of time. In another example, the sensor status data134may be based on receiving a heartbeat packet or other indicator transmitted by the sensor device124via the network. In yet another example, the sensor status data124may be indicative of data transfer of data on the network involving the sensor device124. Continuing the example, data indicative of the sensor device124communicating with a router may be used to generate the sensor status data134.

The failure management system128may access threshold data136to determine the operational status of a sensor device124. Operational status may comprise information indicative of whether the sensor device124, or portion thereof, is deemed to be in a failed state, operational state, or other intermediate state. For example, the operational status for a particular sensor device124may be total failure, partial failure, operational, operational but degraded, and so forth. The threshold data136specifies one or more thresholds associated with operation of the facility102. The threshold data136may specify thresholds in terms of one or more of boundaries, minima, maxima, percentages, functions, conditions, and so forth. For example, the threshold data136may specify a timeout interval after which failure to receive a heartbeat signal from the sensor device124would result in that sensor device124being designated as a failure. In some implementations the threshold136may specify a binary value, such as “true” or “false”. The threshold data136may specify fixed thresholds, or thresholds that dynamically adjust.

The failure management system128may determine operational status of the sensor device124based at least in part on the sensor data132. The sensor data132may be analyzed to determine if the sensor data132met, exceeded, or otherwise satisfied one or more threshold values specified by the threshold data136. For example, where the sensor data132comprises image data, a histogram of colors appearing within the image data may be compared with one or more values specified in threshold data136. If the image data obtained by the sensor device124exhibits a color histogram indicating more red values than specified by the threshold data136, the sensor device124may be deemed to have a failed operational status.

The failure management system128may determine operational status of the sensor device124based at least in part on the sensor status data134. The sensor status data134may be compared to one or more thresholds specified in threshold data136. For example, the threshold data136may specify that the sensor device124(1) is to deliver sensor data132comprising image data at a rate of no less than 30 frames per second (FPS). The sensor status data134generated by the sensor device124(1) may indicate that for the last 30 seconds, the sensor device124(1) has delivered image data at a rate of 13 FPS. The failure management system128may determine that the delivered rate is less than that specified by the threshold data136. Based on this determination, the failure management system128may designate the sensor device124(1) as having an operational status of failed. In another example, the threshold data136may specify a maximum period that may elapse without receipt of a heartbeat packet before the sensor device124associated with the heart packet is declared to have an operational status of failed.

In some implementations, the failure management system128may determine operational status of the sensor device124based at least in part on the sensor data132and the sensor status data134. For example, the analysis of the sensor data132may indicate that the sensor data132has been relatively unchanging over a period of time while the sensor status data134may indicate that the processor on board the sensor device124is operating at 99% capacity. This combination may be deemed to be indicative of a failure such as a “stuck” sensor device124in which the sensor data132is deemed to be unreliable.

The inventory management system126is configured to generate output data138based at least in part on the sensor data132. The output data138may comprise information about occurrences or activities within the facility102, status of objects within the facility102, and so forth. For example, the inventory management system126may generate output data138indicative of a location of an object such as the user116, the tote118, an item104, and so forth, based at least in part on the sensor data132. In another example, the output data138may comprise information indicative of a quantity of items104picked or placed from the inventory locations114, quantity of items104present at the inventory location114at a particular point in time, and so forth.

Based on the operational status determined by the failure management system128, one or more mitigating actions may be taken. For example, the inventory management system126may be configured to utilize sensor data132obtained from other sensor devices124that are deemed to be operational or at least partially operational. Conversely, the inventory management system126may be configured to disregard or reduce the confidence value associated with output data138generated from sensor data132provided by the sensors120or sensor devices124having particular operational statuses, such as failed.

In the event of a failure, the failure management system128may mitigate the impact of the failure by accessing sensor data132acquired from other sensors120. In some implementations, these other sensors120may have different capabilities. For example, the inventory management system126may be configured to generate data indicative of the location of an object such as the user116within the facility102based on information from sensor devices124employing three-dimensional (3D) depth sensors120(2). The depth sensor120(2) may provide three-dimensional point cloud data that provides information such as a relative distance between the depth sensor120(2) and a point or area on a detected object. The point cloud data may have relatively low resolution, such as 640×480 pixels with distance indicated as an 8 bit value. Using this information, the three-dimensional point cloud provides information as to the size and shape of the object. The inventory management system126may use the point cloud data from one or more sensor devices124to determine a location of an object at the facility102.

In comparison, an imaging sensor120(1) may acquire color (red-green-blue or “RGB”) image data with a resolution of 1920×1080 and a color depth of 12 bits. The inventory management system126may process the image data from one or more imaging sensors120(1) to determine the location of an object at the facility102. However, compared to the processing of the sensor data132from the depth sensor120(2), this processing may be more computationally intensive such as requiring the use of various image processing techniques including classifiers, neural networks, and so forth, to recognize the object, transform a position in two-dimensional space as represented by the object in the image data into a three-dimensional location, and so forth. As a result, the determination of the output data138such as location information may be more efficiently done using fewer resources from the point cloud data compared to the image data.

The failure management system128may use the sensor data132from other sensor devices124or other sensors120therein to generate the output data138. For example, when the depth sensor120(2) is determined to have an operational status of “failed”, the inventory management system126may process the sensor data132from other sensors120, such as imaging sensors120(1), that sense data from the same region as the failed sensor120. As a result, even with the failure of the depth sensor120(2) in that region, the inventory management system126is still able to provide the output data138indicative of the location of the object at the facility102.

By using the failure management system128, the inventory management system126may acquire and utilize the sensor data132to generate output data138and maintain normal operation of the facility102even while some sensor devices124exhibit failures. As a result, the facility102may more reliably provide a variety of services to the user116as well as the operator, improving the overall user experience.

FIG. 2is a block diagram200illustrating additional details of the facility102, according to some implementations. The facility102may be connected to one or more networks202, which in turn may connect to one or more servers204. The network202may include private networks, public networks such as the Internet, or a combination thereof. The network202may utilize wired technologies (e.g., wires, fiber optic cable, and so forth), wireless technologies (e.g., radio frequency, infrared, acoustic, optical, and so forth), or other connection technologies. The network202is representative of any type of communication network, including one or more of data networks or voice networks.

The servers204may be configured to execute one or more modules or software applications associated with the inventory management system126. While the servers204are illustrated as being in a location outside of the facility102, in other implementations, at least a portion of the servers204may be located at the facility102. The servers204are discussed in more detail below with regard toFIG. 3.

The facility102may include one or more sensor devices124. The sensor devices124may communicate to the servers204using the network202.

The users116, the totes118, or other objects in the facility102may be equipped with one or more tags206. The tags206are configured to emit a signal208. In one implementation, the tag206may be a radio frequency identification (RFID) tag206configured to emit an RF signal208upon activation by an external signal. For example, the external signal may comprise a radio frequency signal or a magnetic field configured to energize or activate the RFID tag206. In another implementation, the tag206may comprise a transmitter and a power source configured to power the transmitter. For example, the tag206may comprise a Bluetooth® Low Energy (BLE) transmitter and battery. In other implementations, the tag206may use other techniques to indicate presence to a corresponding sensor or detector. For example, the tag206may be configured to generate an ultrasonic signal208that is detected by corresponding acoustic receivers. In yet another implementation, the tag206may be configured to emit an optical signal208.

The inventory management system126may be configured to use the tags206for one or more of identification of the object, determining a location of the object, and so forth. For example, the users116may wear tags206, the totes118may have tags206affixed, and so forth, that may be read and used to determine identity and location.

Generally, the inventory management system126or other systems associated with the facility102may include any number and combination of input components, output components, and servers204.

The one or more sensors120or the one or more sensor devices124may be arranged at one or more locations within the facility102. For example, the sensors120may be mounted on or within a floor, wall, or ceiling, at an inventory location114, on the tote(s)118, may be carried or worn by the user(s)116, and so forth.

The sensors120may include one or more imaging sensors120(1). These imaging sensors120(1) may include cameras configured to acquire images of a scene. The imaging sensors120(1) are configured to detect light in one or more wavelengths including, but not limited to, terahertz, infrared, visible, ultraviolet, and so forth. The inventory management system126may use image data acquired by the imaging sensors120(1) during operation of the facility102. For example, the inventory management system126may identify items104, users116, totes118, and so forth, based at least in part on their appearance within the image data.

One or more depth sensors120(2) may also be included in the sensors120. The depth sensors120(2) are configured to acquire spatial or three-dimensional data, such as depth information, about objects within a sensor FOV122. The depth sensors120(2) may include range cameras, lidar systems, sonar systems, radar systems, structured light systems, stereo vision systems, optical interferometry systems, coded aperture systems, and so forth. For example, the depth sensor120(2) may include a light source, a sensor, and circuitry to generate light pulses using the light source. For example, the light source may comprise one or more light emitting diodes (LEDs) and the sensor may comprise a charge coupled device (CCD), complementary metal oxide semiconductor (CMOS), photodetector, and so forth. The depth sensor120(2) may use the processor to execute instructions to initiate emission of a light pulse from the light source and detect a return of the pulse with the sensor. Based on a time for the light pulse to be emitted and return to the sensor and given a known velocity of light, the processor may determine a distance to at least a portion of an object in the region.

The inventory management system126may use the three-dimensional data acquired to identify objects, determine one or more of a location, orientation, or position of an object, and so forth. For example, the output data138may be based on the point cloud data provided by the depth sensor120(2) and may include one or more of a location, orientation, position, or pose of the user116in three-dimensional space within the facility102. The location may be described as where in space within the facility102an object is. For example, the location may be specified as X and Y coordinates relative to an origin, where X and Y are mutually orthogonal. In comparison, orientation may be indicative of a direction the object (or a portion thereof) is facing. For example, the orientation may be that the user116is facing south. Position may provide information indicative of a physical configuration or pose of the object, such as the arms of the user116are stretched out to either side. Pose may provide information on a relative configuration of one or more elements of an object. For example, the pose of the user's116hand may indicate whether the hand is open or closed. In another example, the pose of the user116may include how the user116is holding an item104.

One or more buttons120(3) may be configured to accept input from the user116. The buttons120(3) may comprise mechanical, capacitive, optical, or other mechanisms. For example, the buttons120(3) may comprise mechanical switches configured to accept an applied force from a touch of the user116to generate an input signal. The inventory management system126may use data from the buttons120(3) to receive information from the user116. For example, the buttons120(3) may be used to accept input from a user116such as a username and password associated with an account.

The sensors120may include one or more touch sensors120(4). The touch sensors120(4) may use resistive, capacitive, surface capacitance, projected capacitance, mutual capacitance, optical, Interpolating Force-Sensitive Resistance (IFSR), or other mechanisms to determine the point of a touch or near-touch. For example, the IFSR may comprise a material configured to change electrical resistance responsive to an applied force. The point of that change in electrical resistance within the material may indicate the point of the touch. The inventory management system126may use data from the touch sensors120(4) to receive information from the user116. For example, the touch sensor120(4) may be integrated with the tote118to provide a touchscreen with which the user116may select from a menu one or more particular items104for picking.

One or more microphones120(5) may be configured to acquire audio data indicative of sound present in the environment. In some implementations, arrays of microphones120(5) may be used. These arrays may implement beamforming or other techniques to provide for directionality of gain. The inventory management system126may use the one or more microphones120(5) to accept voice input from the users116, and so forth.

One or more weight sensors120(6) may be configured to measure the weight of a load, such as the item104, the tote118, and so forth. The weight sensors120(6) may be configured to measure the weight of the load at one or more of the inventory locations114, the tote118, or elsewhere. The weight sensors120(6) may include one or more sensing mechanisms to determine the weight of a load. These sensing mechanisms may include piezoresistive devices, piezoelectric devices, capacitive devices, electromagnetic devices, optical devices, potentiometric devices, microelectromechanical devices, and so forth. The sensing mechanisms may operate as transducers that generate one or more signals based on an applied force, such as that of the load due to gravity. The inventory management system126may use the data acquired by the weight sensors120(6) to identify an object, determine a location of an object, maintain shipping records, and so forth. For example, the weight sensors120(6) at a particular location in the facility102may report a weight of the tote118, indicating the tote118is present at that location.

The sensors120may include one or more light sensors120(7). The light sensors120(7) may be configured to provide information associated with ambient lighting conditions such as a level of illumination. Information acquired by the light sensors120(7) may be used by the inventory management system126to adjust a level, intensity, or configuration of the output device210.

One more radio frequency identification (RFID) readers120(8), near field communication (NFC) systems, and so forth, may also be provided as sensors120. For example, the RFID readers120(8) may be configured to read the RF tags206. Information acquired by the RFID reader120(8) may be used by the inventory management system126to identify an object associated with the RF tag206such as the item104, the tote118, and so forth.

One or more RF receivers120(9) may also be provided. In some implementations, the RF receivers120(9) may be part of transceiver assemblies. The RF receivers120(9) may be configured to acquire RF signals208associated with Wi-Fi™ Bluetooth®, ZigBee®, 3G, 4G, LTE, or other wireless data transmission technologies. The RF receivers120(9) may provide information associated with data transmitted via radio frequencies, signal strength of RF signals208, and so forth. For example, information from the RF receivers120(9) may be used by the inventory management system126to determine a location of an RF source such as a device carried by the user116.

The sensors120may include one or more accelerometers120(10), which may be worn or carried by the user116, mounted to the tote118, and so forth. The accelerometers120(10) may provide information such as the direction and magnitude of an imposed acceleration. Data such as rate of acceleration, determination of changes in direction, speed, and so forth, may be determined using the accelerometers120(10).

A gyroscope120(11) may provide information indicative of rotation of an object affixed thereto. For example, the tote118or other objects or devices may be equipped with a gyroscope120(11) to provide data indicative of a change in orientation.

A magnetometer120(12) may be used to determine a heading by measuring ambient magnetic fields, such as the terrestrial magnetic field. The magnetometer120(12) may be worn or carried by the user116, mounted to the tote118, and so forth. For example, the magnetometer120(12) as worn by the user116may act as a compass and provide information indicative of which way the user116is facing.

A proximity sensor120(13) may be used to determine presence of an object, such as the user116, the tote118, and so forth. The proximity sensors120(13) may use optical, electrical, ultrasonic, electromagnetic, or other techniques to determine a presence of an object. In some implementations, the proximity sensors120(13) may use an optical emitter and an optical detector to determine proximity. For example, an optical emitter may emit light, a portion of which may then be reflected by the object back to the optical detector to provide an indication that the object is proximate to the proximity sensor120(13). In other implementations, the proximity sensors120(13) may comprise a capacitive proximity sensor120(13) configured to provide an electrical field and determine a change in electrical capacitance due to presence or absence of an object within the electrical field.

The proximity sensors120(13) may be configured to provide sensor data132indicative of one or more of a presence or absence of an object, a distance to the object, or characteristics of the object. An optical proximity sensor120(13) may use time-of-flight (ToF), structured light, interferometry, or other techniques to generate the distance data. For example, ToF determines a propagation time (or “round-trip” time) of a pulse of emitted light from an optical emitter or illuminator that is reflected or otherwise returned to an optical detector. By dividing the propagation time in half and multiplying the result by the speed of light in air, the distance to an object may be determined. In another implementation, a structured light pattern may be provided by the optical emitter. A portion of the structured light pattern may then be detected on the object using an imaging sensor120(1) such as a camera. Based on an apparent distance between the features of the structured light pattern, the distance to the object may be calculated. Other techniques may also be used to determine distance to the object. In another example, the color of the reflected light may be used to characterize the object, such clothing, tote118, and so forth. In some implementations, a proximity sensor120(13) may be installed at the inventory location114.

The sensors120may include other sensors120(S) as well. For example, the other sensors120(S) may include instrumented auto-facing units, thermometers, barometric sensors, hygrometers, and so forth. Instrumented auto-facing units may be configured to maintain items104in an orderly fashion and provide information indicative of addition or removal of items104from the instrumented auto-facing unit. For example, the instrumented auto-facing unit may include a position sensor that moves responsive to a pick or place of an item104therefrom.

The facility102may include one or more access points212configured to establish one or more wireless networks. The access points212may use Wi-Fi™, NFC, Bluetooth®, or other technologies to establish wireless communications between a device and the network202. The wireless networks allow the devices to communicate with one or more of the inventory management system126, the sensors120, the sensor devices124, the tag206, a communication device of the tote118, or other devices. In other implementations, a wired networking infrastructure may be implemented. For example, cabling may be used to provide Ethernet local area network connectivity.

The output devices210may also be provided in the facility102. The output devices210may be configured to generate signals that may be perceived by the user116.

Haptic output devices210(1) may be configured to provide a signal that results in a tactile sensation to the user116. The haptic output devices210(1) may use one or more mechanisms such as electrical stimulation or mechanical displacement to provide the signal. For example, the haptic output devices210(1) may be configured to generate a modulated electrical signal that produces an apparent tactile sensation in one or more fingers of the user116. In another example, the haptic output devices210(1) may comprise piezoelectric or rotary motor devices configured to provide a vibration that may be felt by the user116.

One or more audio output devices210(2) are configured to provide acoustic output. The acoustic output includes one or more of infrasonic sound, audible sound, or ultrasonic sound. The audio output devices210(2) may use one or more mechanisms to generate the sound. These mechanisms may include, but are not limited to, the following: voice coils, piezoelectric elements, magnetostrictive elements, electrostatic elements, and so forth. For example, a piezoelectric buzzer or a speaker may be used to provide acoustic output.

The display output devices210(3) may be configured to provide output that may be seen by the user116or detected by a light-sensitive detector such as an imaging sensor120(1) or light sensor120(7). The output from the display output devices210(3) may be monochrome or color. The display output devices210(3) may be emissive, reflective, or both emissive and reflective. An emissive display output device210(3) is configured to emit light during operation. For example, an LED is an emissive visual display output device210(3). In comparison, a reflective display output device210(3) relies on ambient light to present an image. For example, an electrophoretic display is a reflective display output device210(3). Backlights or front lights may be used to illuminate the reflective visual display output device210(3) to provide visibility of the information in conditions where the ambient light levels are low.

Mechanisms of the display output devices210(3) may include liquid crystal displays (LCDs), transparent organic LEDs, electrophoretic displays, image projectors, or other display mechanisms. The other display mechanisms may include, but are not limited to, micro-electromechanical systems (MEMS), spatial light modulators, electroluminescent displays, quantum dot displays, liquid crystal on silicon (LCOS) displays, cholesteric displays, interferometric displays, and so forth. These mechanisms are configured to emit light, modulate incident light emitted from another source, or both.

The display output devices210(3) may be configured to present images. For example, the display output devices210(3) may comprise a pixel-addressable display. The image may comprise at least a two-dimensional array of pixels or a vector representation of an at least two-dimensional image.

In some implementations, the display output devices210(3) may be configured to provide non-image data, such as text characters, colors, and so forth. For example, a segmented electrophoretic display, segmented LED, and so forth, may be used to present information such as a stock keeping unit (SKU) number. The display output devices210(3) may also be configurable to vary the color of the text, such as using multicolor LED segments.

In some implementations, display output devices210(3) may be configurable to provide image or non-image output. For example, an electrophoretic display output device210(3) with addressable pixels may be used to present images of text information, or all of the pixels may be set to a solid color to provide a colored panel.

The output devices210may include hardware processors, memory, and other elements configured to present a user interface. In one implementation, the display output devices210(3) may be arranged along the edges of inventory locations114.

Other output devices210(T) may also be present at the facility102. The other output devices210(T) may include lights, scent/odor dispensers, document printers, three-dimensional printers or fabrication equipment, and so forth. For example, the other output devices210(T) may include lights that are located on the inventory locations114, the totes118, and so forth.

FIG. 3illustrates a block diagram300of a server204configured to support operation of the facility102, according to some implementations. The server204may be physically present at the facility102, may be accessible by the network202, or a combination of both. The server204does not require end-user knowledge of the physical location and configuration of the system that delivers the services. Common expressions associated with the server204may include “on-demand computing”, “software as a service (SaaS)”, “platform computing”, “network-accessible platform”, “cloud services”, “data centers”, and so forth. Services provided by the server204may be distributed across one or more physical or virtual devices.

One or more power supplies302are configured to provide electrical power suitable for operating the components in the server204. The server204may include one or more hardware processors304(processors) configured to execute one or more stored instructions. The processors304may comprise one or more cores. The cores may be of one or more types. For example, the processors304may include application processor units, graphic processing units, and so forth. One or more clocks306may provide information indicative of date, time, ticks, and so forth. For example, the processor304may use data from the clock306to generate timestamps, trigger a preprogrammed action, and so forth.

The server204may include one or more communication interfaces308, such as input/output (I/O) interfaces310, network interfaces312, and so forth. The communication interfaces308enable the server204, or components thereof, to communicate with other devices or components. The communication interfaces308may include one or more I/O interfaces310. The I/O interfaces310may comprise Inter-Integrated Circuit (I2C), Serial Peripheral Interface (SPI), Universal Serial Bus (USB) as promulgated by the USB Implementers Forum, RS-232, and so forth.

The I/O interface(s)310may couple to one or more I/O devices314. The I/O devices314may include input devices such as one or more of a sensor120, keyboard, mouse, scanner, and so forth. The I/O devices314may also include output devices210such as one or more of a display output device210(3), printer, audio speakers, and so forth. In some embodiments, the I/O devices314may be physically incorporated with the server204or may be externally placed.

The network interfaces312are configured to provide communications between the server204and other devices, such as the totes118, routers, access points212, and so forth. The network interfaces312may include devices configured to couple to personal area networks (PANs), local area networks (LANs), wide area networks (WANs), and so forth. For example, the network interfaces312may include devices compatible with Ethernet, Wi-Fi™, Bluetooth®, ZigBee®, and so forth.

The server204may also include one or more busses or other internal communications hardware or software that allow for the transfer of data between the various modules and components of the server204.

As shown inFIG. 3, the server204includes one or more memories316. The memory316comprises one or more non-transitory computer-readable storage media (CRSM). The CRSM may be any one or more of an electronic storage medium, a magnetic storage medium, an optical storage medium, a quantum storage medium, a mechanical computer storage medium, and so forth. The memory316may provide storage of computer-readable instructions, data structures, program modules, and other data for the operation of the server204. A few example functional modules are shown stored in the memory316, although the same functionality may alternatively be implemented in hardware, firmware, or as a system on a chip (SoC).

The memory316may include at least one operating system (OS) module318. The OS module318is configured to manage hardware resource devices such as the I/O interfaces310, the I/O devices314, the communication interfaces308, and provide various services to applications or modules executing on the processors304. The OS module318may implement a variant of the FreeBSD™ operating system as promulgated by the FreeBSD Project; other UNIX™ or UNIX-like variants; a variation of the Linux™ operating system as promulgated by Linus Torvalds; the Windows® operating system from Microsoft Corporation of Redmond, Wash., USA; and so forth.

Also stored in the memory316may be a data store320and one or more of the following modules. These modules may be executed as foreground applications, background tasks, daemons, and so forth. The data store320may use a flat file, database, linked list, tree, executable code, script, or other data structure to store information. In some implementations, the data store320or a portion of the data store320may be distributed across one or more other devices including the servers204, network attached storage devices, and so forth.

A communication module322may be configured to establish communications with one or more of the totes118, the sensors120, the display output devices212(3), other servers204, or other devices. The communications may be authenticated, encrypted, and so forth.

The memory316may store an inventory management module324. The inventory management module324may be configured to provide the inventory functions as described herein with regard to the inventory management system126. For example, the inventory management module324may track items104between different inventory locations114, to and from the totes118, and so forth. Tracking of an object may include the determination of successive locations within the facility and information indicative of a persistence of an object, such as continuity that the same object has moved from one location to another.

The inventory management module324may include one or more of a data acquisition module326, a failure management module328, or a data processing module330. The data acquisition module326may be configured to acquire and access information associated with operation of the facility102. For example, the data acquisition module326may acquire information from the sensors120, the sensor devices124, and so forth.

The inventory management module324may be configured to track objects in the facility102or provide other services using the physical layout data130, the sensor data132, the sensor status data134, and so forth, which may be stored in the data store320.

The physical layout data130comprises information about the physical configuration of the facility102or portions thereof. For example, the physical layout data130may include electronic representations of the physical structures in the facility102, such as CAD data of the aisle112configurations, inventory locations114, information about which items104are in what inventory locations114, real coordinates of the sensor devices124, and so forth. The physical layout data130may include information about the presence of walls; HVAC equipment; location of doors and windows; and so forth.

The sensor data132may comprise information acquired from, or based on, the one or more sensors120. For example, the sensor data132may comprise 3D information about an object in the facility102as acquired by the depth sensors120(2) or weight data as acquired by the weight sensors120(6).

The failure management system128may be implemented at least in part by the failure management module328. The failure management module328may be configured to use one or more of the physical layout data130, the sensor data132, the sensor status data134, and so forth, to determine the operational status of one or more of the sensor devices124in the facility102, initiate mitigating actions, and so forth. For example, the failure management module328may compare the values of sensor data132with one or more values of threshold data136to determine if the sensor data132from a particular sensor device124is indicative of a failure. In another example, the failure management module328may compare the sensor status data134with one or more values of the threshold data136to determine if a particular sensor device124or sensor120has failed. As described above, in some implementations, the failure management module328may analyze the sensor data132and the sensor status data134to determine operational status of the sensor device124or the sensors120therein.

The data processing module330is configured to process at least a portion of the sensor data132. In some implementations, the data processing module330executed by the server204may implement different functions from those performed by the data processing module330executed on the sensor device124. For example, the data processing module330executed by the server204may use functions that require computational or memory resources that exceed those available on the sensor device124. For example, the data processing module330may perform object recognition using more comprehensive classifiers.

The inventory management module324may use the sensor data132to generate output data138. The output data138may include information about one or more objects, movement of those objects, and so forth. For example, the output data138may comprise identification data about an object, such as the user identity of the user116. The identity of the user may be determined by processing at least a portion of the sensor data132received from one or more sensor devices124. In another example, the output data138may include tracking data indicative of location data or motion data of an object within the facility102based on sensor data132received from one or more sensor devices124. The identification data may identify a particular user116, a particular item104, tote118, or other object. For example, the identification data may comprise an account name, a real name, a temporary identifier issued to a previously unregistered user116, and so forth.

The tracking data provides information as to the location of an object within the facility102, movement of the object, and so forth. For example, the tracking data may comprise data indicative of the user116being in aisle112(13) in front of the inventory location114(37).

Operation of the data processing module330may be affected based on the operational status determined by the failure management module328. For example, sensor data132obtained from or associated with sensor devices124that are deemed to be in a failed state may be disregarded, or the sensor data132may be associated with output having a lesser confidence value than output resulting from sensor data132obtained from sensor devices124that are deemed to be in an operational state. In another example, upon determination of a failure of a particular sensor device124, the data processing module330may use sensor data132from other sensor devices124or sensors120to maintain production of output data138. Continuing this example, upon failure of a depth sensor120(2), image data acquired from an imaging sensor120(1) that covers at least a portion of the same region in the facility102may be processed using one or more of the techniques described below to generate the output data138.

In some implementations, the data processing module330may utilize one or more services involving human operators. For example, a human operator may be presented with information and provide input indicative of operational status. The operator may determine the operational status based on one or more of the sensor data132, the sensor status data134, and so forth. In another example, in the event of a failure, a human operator may be presented with the sensor data132and asked to provide input that may then be used to generate the output data138.

The data processing module330may be configured to use one or more data processing functions to identify an object depicted in the sensor data132, a position of the object in the sensor data132, determine motion of objects in an image, and so forth. The objects may include, but are not limited to, the items104, users116, totes118, and so forth.

The data processing functions described in this disclosure may be performed at least in part using one or more of the following tools or techniques. In one implementation, the object recognition or other image processing described in this disclosure may be performed, at least in part, using one or more tools available in the OpenCV library as developed by Intel Corporation of Santa Clara, Calif., USA; Willow Garage of Menlo Park, Calif., USA; and Itseez of Nizhny Novgorod, Russia, with information available at www.opencv.org. In another implementation, functions available in the OKAO machine vision library as promulgated by Omron Corporation of Kyoto, Japan, may be used to process the images. In another implementation, the EyeFace SDK as promulgated by Eyedea Recognition Ltd. of Prague, Czech Republic, may be used to process the image data. The OpenBR library and tools as originated by MITRE Corporation of Bedford, Mass., USA, and McLean, Va., USA, and promulgated by the OpenBR group at openbiometrics.org may also be used in some implementations for image processing.

Clothing recognition analyzes images to determine what articles of clothing, ornamentation, and so forth, the user116is wearing or carrying in the facility102. Skin and hair detection algorithms may be used to classify portions of the image that are associated with the user's116skin or hair. Items that are not skin and hair may be classified into various types of articles of clothing such as shirts, hats, pants, bags, and so forth. The articles of clothing may be classified according to function, position, manufacturer, and so forth. Classification may be based on clothing color, texture, shape, position on the user116, and so forth. For example, classification may designate an article of clothing worn on the torso as a “blouse” while color or pattern information may be used to determine a particular designer or manufacturer. The determination of the article of clothing may use a comparison of information from the images with previously stored data. Continuing the example, the pattern of the blouse may have been previously stored along with information indicative of the designer or manufacturer.

In some implementations, identification of the user116may be based on the particular combination of classified articles of clothing. The clothing may be used to identify the user116or to distinguish one user116from another. For example, the user116(1) may be distinguished from the user116(2) based at least in part on the user116(1) wearing a hat and a red shirt while the user116(2) is not wearing a hat and is wearing a blue shirt.

Gait recognition techniques analyze one or more of images, three-dimensional data, or other data, to assess how a user116moves over time. Gait comprises a recognizable pattern of movement of the user's116body that is affected by height, age, and other factors. Gait recognition may analyze the relative position and motion of limbs of the user116. Limbs may include one or more arms, legs, and in some implementations, the head. In one implementation, edge detection techniques may be used to extract a position of one or more limbs of the user116in the series of images. For example, a main leg angle of a user's116leg may be determined, and based on the measurement of this main leg angle over time and from different points-of-view, a three-dimensional model of the leg motion may be generated. The change in position over time of the limbs may be determined and compared with previously stored information to determine an identity of the user116or to distinguish one user116from another.

In some implementations, identity of the object may be based on a combination of these or other recognition techniques. For example, the user116may be identified based on clothing recognition, gait recognition, object recognition, and so forth. The different recognition techniques may be used in different situations or in succession. For example, clothing recognition and gait recognition may be used at greater distances between the user116and the imaging sensors120(1) or when sensor data is inadequate for accurate object recognition. Once identified, such as by way of object recognition, one or more of gait recognition or clothing recognition may be used to track the user116within the facility102.

Other techniques such as artificial neural networks (ANN), active appearance models (AAM), active shape models (ASM), cascade classifiers, support vector machines, active shape models (ASM), Haar detectors, local binary pattern (LBP) classifiers, and so forth, may also be used to process images. For example, the ANN may be a trained using a supervised learning algorithm such that object identifiers are associated with images of particular objects within training images provided to the ANN. In another example, the ANN may be trained to recognize that a particular shape is associated with the user116. Once trained, the ANN may be provided with the images and may provide, as output, the object identifier.

Other modules332may also be present in the memory316, as well as other data334in the data store320. For example, the other modules332may include an accounting module. The accounting module may be configured to add or remove charges to an account based on movement of the items104determined from the sensor data132. The other data334may comprise information such as costs of the items104for use by the accounting module.

FIG. 4illustrates a block diagram400of a sensor device124, according to some implementations. In some implementations, the following components described below may include diagnostic software, circuitry, and so forth, to provide information indicative of operation.

One or more power supplies402are configured to provide electrical power suitable for operating the components in the sensor device124. The power supply402may provide power conditioning such as direct current (DC) to DC conversion, alternating current (AC) to DC conversion, voltage adjustment, frequency adjustment, and so forth. In one implementation, the power supply402may be configured to acquire electrical energy from a communication interface, such as a wired Ethernet connection providing power over one or more of the wires. In another implementation, the power supply402may comprise a wireless power receiver configured to receive transmitted electrical energy. For example, the power supply402may include an inductive loop or capacitive plate configured to receive electrical energy from another inductive loop or capacitive plate external to the sensor device124.

The sensor device124may include one or more hardware processors404(processors) configured to execute one or more stored instructions. The processors404may comprise one or more cores of different types, functions, and so forth. For example, the processors404may include application processor units, graphic processing units, and so forth. In one implementation, the processor404may comprise the Tegra K1 processor from Nvidia Corporation of Santa Clara, Calif. In another implementation, the processor404may comprise an Etron eSP870 processor from Etron Technology America, Inc. of Santa Clara, Calif., configured to process image data from two imaging sensors120(1) and generate depth data.

One or more clocks406may provide information indicative of date, time, ticks, and so forth. For example, the processor404may use data from the clock406to generate timestamps, trigger a preprogrammed action, and so forth.

The sensor device124may include one or more communication interfaces408, such as I/O interfaces410, network interfaces412, and so forth. The communication interfaces408enable the sensor device124, or components thereof, to communicate with other devices or components. The communication interfaces408may include one or more I/O interfaces410. The I/O interfaces410may comprise I2C, SPI, USB, RS-232, and so forth.

The I/O interface(s)410may couple to one or more input devices414. In some implementations, the sensor device124may be a “headless” device, in that it may not have any output devices210onboard for use by a human. The input devices414may include sensors120, such as the imaging sensor120(1), depth sensor120(2), and so forth. A single sensor device124may include, support, or otherwise be associated with one or more sensors120. For example, the sensor device124may include a depth sensor120(2) comprising a pair of stereoscopic infrared imaging sensors120(1) to provide stereoscopic infrared imaging and a visible light (red-green-blue or “RGB”) imaging sensor120(1). For example, the stereoscopic infrared imaging sensors120(1) may comprise an Aptina AR0134 from Aptina Imaging Corporation of San Jose, Calif. In one implementation, the pair of imaging sensors in the depth sensor120(2) may be arranged along a 200 mm baseline relative to one another to provide for a disparity of FOV122suitable for stereoscopic vision.

The input devices414may be at least partially physically incorporated within the sensor device124. In one implementation, the depth sensor120(2) may be incorporated within an enclosure of the sensor device124. The enclosure may comprise a case, which may have an opening for light to enter to reach the sensors120, such as the imaging sensor(s)120(1). In another implementation, at least a portion of the sensor120may be configured to pan, tilt, or rotate, relative to the enclosure. In yet another implementation, the sensor device124may comprise a main enclosure attached to one or more secondary enclosures containing one or more of the input devices414. For example, the imaging sensors120(1) may be located within a separate enclosure and coupled by way of a cable to the main enclosure of the sensor device124.

The network interfaces412are configured to provide communications between the sensor device124and other devices, routers, servers204, access points212, and so forth. The network interfaces412may include devices configured to couple to PANs, LANs, WANs, and so forth. For example, the network interfaces412may include devices compatible with Ethernet, Wi-Fi™, Bluetooth®, ZigBee®, and so forth.

The sensor device124may also include one or more busses or other internal communications hardware or software that allow for the transfer of data between the various modules and components of the sensor device124.

As shown inFIG. 4, the sensor device124includes one or more memories416. The memory416comprises one or more non-transitory CRSM, as described above. The memory416provides storage of computer-readable instructions, data structures, program modules, and other data for the operation of the sensor device124. A few example functional modules are shown stored in the memory416, although the same functionality may alternatively be implemented in hardware, firmware, or as a SoC.

The memory416may include at least one OS module418. The OS module418is configured to manage hardware resource devices such as the communication interfaces408, the input devices414, and so forth, and provide various services to applications or modules executing on the processors404. The OS module418may implement a variant of the FreeBSD™ operating system as promulgated by the FreeBSD Project; other UNIX™ or UNIX-like variants; a variation of the Linux™ operating system as promulgated by Linus Torvalds; the Windows® operating system from Microsoft Corporation of Redmond, Wash., USA; and so forth.

Also stored in the memory416may be a data store420and one or more of the following modules. These modules may be executed as foreground applications, background tasks, daemons, and so forth. The data store420may use a flat file, database, linked list, tree, executable code, script, or other data structure to store information. In some implementations, the data store420or a portion of the data store420may be distributed across one or more other devices including other sensor devices124, network attached storage devices, and so forth.

A communication module422may be configured to establish communications with one or more of the other sensor devices124, servers204, or other devices. The communications may be authenticated, encrypted, and so forth.

The memory416may store a data processing module330, such as described above. The data processing module330may provide one or more data functions. For example, the data processing module330may implement one or more image processing functions424, such as a threshold function424(1), a motion vector function424(2), an object recognition function424(3), and so forth.

The threshold function424(1) is configured to apply a thresholding operation to sensor data132, such as image data obtained from an imaging sensor120(1), point cloud data from a depth sensor120(2), and so forth. For example, the threshold function424(1) may be configured to generate a binary image from image data. Continuing the example, implementations utilizing the OpenCV library may execute the function “threshold” to generate sensor data132comprising a binary image. In one implementation, the binary image comprises a digital image in which each pixel is expressed as a single bit value, such as a “1” or a “0”. For example, the binary image may comprise a version of the image data that been reduced to a 1-bit depth for each pixel.

The motion vector function424(2) may be configured to provide a motion vector associated with a portion of the image data. For example, the user116walking through the FOV122may manifest as a group of pixels changing from one frame of image data to another. The motion vector function424(2) may generate sensor data132comprising information indicative of an optical flow between a plurality of images. Continuing the example, implementations utilizing the OpenCV library, the functions “calcOpticalFlowPyrLKd” or “DenseOpticalFlow” may be used to generate sensor data132comprising motion data.

The object recognition function424(3) may be configured to provide information that identifies an object. In one implementation, the object recognition function424(3) may be configured to identify a human face within image data. Continuing the example, implementations utilizing the OpenCV library, the function “FaceRecognizer” may be used to generate sensor data132comprising a subset of the image data that includes the image of the face.

The data processing module330may include other functions424(F) (not shown) as well. For example, a depth data generation module may be configured to generate depth data from image data acquired by two or more imaging sensors120(1) separated by a distance, from the depth sensor120(2), and so forth. Continuing the example, output from the two or more imaging sensors120(1) may be used to generate sparse stereo depth data, dense stereo depth data, and so forth. Other modules may provide registration data for alignment between depth data and image data, construction of three-dimensional point clouds, point cloud projection, application of one or more filter functions, and so forth.

The other functions424(F) may also include data compression. For example, the sensor data132may comprise image data for which the background has been removed, and the image of the remaining foreground object(s) has been processed with one or more filter functions.

A local failure management module426may also be stored in the memory416. The local failure management module426may be configured to determine the operational status of the components onboard or otherwise coupled to or associated with the sensor device124. For example, the local failure management module426may gather data from various hardware sensors and software systems to generate the sensor status data132. The local failure management module426may access threshold data136stored in the data store420, from one or more of the servers204, or a combination thereof.

The local failure management module426may pass the sensor status data134to the data processing module330onboard the sensor device124, the data processing module330at the server204, or both. Operation of the data processing module330may then change based at least in part on one or more of the analysis of the sensor data132, analysis of the sensor status data134, and so forth.

Other modules428may also be present in the memory416, as well as other data430in the data store420. For example, the other modules428may include a power management module. The power management module may be configured to transition the sensor device124from a low power mode to a higher power operating mode. For example, the data processing module330may determine that no motion has occurred within subsequent image data for a threshold amount of time. Responsive to this determination, the power management module may place the sensor device124into a lower power mode, to reduce power consumption, heat generation, and so forth. The other data430may comprise information such as timing thresholds for use by the power management module.

The sensor device124may also include an infrared illuminator432to illuminate at least a portion of the FOV122. For example, the infrared illuminator432may comprise a vertical-cavity surface-emitting laser (VCSEL). The infrared illuminator432may project a pattern from a mask onto at least a portion of the FOV122using the VCSEL and a projection lens. In some implementations, the infrared illumination may be at a center wavelength of about 850 nanometers (nm) to 870 nm.

The sensor device124may include one or more modular connectors434. The modular connectors434may provide one or more of mechanical, electrical, optical, or other types of connectivity. In one implementation, the modular connectors434may comprise an Ethernet connection such as an RJ-45 jack. In another implementation, the modular connector434may comprise a mechanical coupling such as a slot, tab, or other mechanical engagement feature suitable for affixing the sensor device124to another structure, such as an inventory location114, wall, light fixture, and so forth.

FIG. 5illustrates a user interface500for presenting information based on sensor data132acquired from one or more of the sensors120, according to some implementations. As described above, the inventory management module324may process the sensor data132and generate output data138. The output data138may include information such as location and identification of one or more objects within the facility102. The output data138may be presented to a human operator using the user interface500depicted here to facilitate administration the facility102such as a fulfillment center. As described above, the output data138may provide information indicative of a location502of an object within the facility102. In this illustration, visual indicia are provided in the form of dotted circles surrounding the object. The output data138may also include identification information, such as a user identifier504. In this illustration, the user identifier504is presented as a dotted box attached by a dotted line to the visual indicia of location502that includes information such as the role of the user116. For example, user116(2) is a “Supervisor” while user116(5) is a “Trainee”.

By utilizing the sensor data132from a plurality of sensor devices124within the facility102, the inventory management system126may be capable of providing one or more services. The services may be accessible to the user116, an administrator, or a third party. For example, the user116in the facility102who is picking an order for a customer may ask for directions to find a particular item104within the facility102. Based on the location502determined by the inventory management system126, and given previously stored data indicative of where items104are stowed, the user116may be directed to the item104sought.

FIG. 6illustrates an overhead view600of the facility102during failure of a sensor device124and use of other sensor devices124to maintain operation of the facility102, according to some implementations.

In this illustration, a plurality of inventory locations114is depicted on either side of an aisle112. The user116is standing in front of the inventory location114(1). Phenomena proximate to the inventory location114(1) may be detected by one of a plurality of sensors120and their corresponding sensor devices124. For the purposes of this illustration, the sensors120are depicted, while the sensor devices124that support them are omitted for clarity.

Depicted are the FOVs122for some of the sensors120that cover the region at least partially occupied by the user116. The imaging sensor120(1)(1) has a FOV122(1) which includes the same region as occupied by the user116, while the imaging sensor120(1)(2) has a FOV122(2) that also includes the same region as occupied by the user116. Overhead imaging sensors120(1)(3) and120(1)(4) have FOVs122(3) and122(4), respectively, that also include the same region as occupied by the user116. The depth sensor120(2)(5) has a FOV122(5) and the depth sensor120(2)(6) has a FOV122(6) that each include the same region as occupied by the user116. The region that includes the user116is also within the FOV122of other sensors120, which are not depicted here.

In this illustration, the depth sensor120(2)(5) has failed and is designated as a failed sensor602producing a sensor blackout region604corresponding to the FOV122(5). However, sensor data132continues to be gathered from that region by other sensors120. In some implementations, the sensors120may be divided into different groups to provide for some measure of redundancy. For example, a first group may include the depth sensor120(2)(5) while a second group may include the depth sensor120(2)(6). In some implementations the groups may include different types of sensors120. The first group may be powered by a first bus while the second group may be powered by second bus, each bus supplied by an independent power supply. Similarly, sensor devices124in the first group may be attached to a first network202(1) while sensor devices124in the second group may be attached to a second network202(2). In this way, failure of the infrastructure supporting the first group still leaves the second group operational. The sensors120associated with the different groups may have some degree of overlap. For example, as shown in this disclosure, the FOV122(2) overlaps at least partially the FOV122(5).

The sensor blackout region604may be determined based on the physical layout data130or other information. For example, given a known direction of the FOV122(5), data about the shape of the FOV122(5), and the location of the depth sensor120(2)(5) within the facility102, the sensor blackout region604may be determined.

The failure management module328may send information indicative of the failed sensor602, the sensor blackout region604, and so forth, to the data processing module330. The data processing module330may mitigate the failure by removing from consideration the sensor data132associated with the failed sensor602, designating the sensor data132received from a failed sensor602as being less reliable, changing a confidence value associated with the output data138, and so forth. For example, given the failed sensor602, information such as the location or identity of the user116within the region may be determined to have a lower confidence value then when all of the sensors120covering that region are operational.

As described above, the failure management module328may be configured to utilize sensor data132from other sensors120to maintain services such as object tracking, object identification, and so forth. For example, the failure management module328may determine the sensor blackout region604and may initiate mitigation actions that utilize the overhead imaging sensors120(1)(3) and120(1)(4) to determine the location of the user116in conjunction with point cloud data from the depth sensor120(2)(6).

FIG. 7illustrates a side view700of the facility102during the failure of the sensor device124as described inFIG. 6. As illustrated, the failure management module328may mitigate the effect of the failed sensor602on the operation of the inventory management module324. From the side view700, the FOV122(6) of the depth sensor120(2)(6) encompasses a region that includes approximately an upper half of the body of the user116. In comparison, the FOV122(5) of the depth sensor120(2)(5), which has been determined to be the failed sensor602, has a sensor blackout region604that encompasses a region of the user116from about the knees up. The employee badge or identification number of “171” on a jacket of the user116is within the FOV122(2) of the imaging sensor120(1)(2). As a result of the failed sensor602, the failure management module328may provide operational status data to the data processing module330indicative of the failure. The data processing module330may then process the available sensor data132using different techniques. For example, rather than looking for an overall shape of the entire user116, the data processing module330may switch to classifiers or more computationally intensive algorithms that are able to determine the shape of the arms, torso, head, or other portion of the user116and maintain tracking with only a partial view of the user116.

Illustrative Processes

FIG. 8depicts a flow diagram800of a process of determining failure of a sensor device124based on sensor status data134, according to some implementations. In some implementations, the process may be performed at least in part by the server204.

Block802accesses sensor status data134indicative of one or more operational parameters of at least a portion of a sensor device124having one or more sensors120covering a region at the facility102. For example, the sensor status data134may be indicative of a number of frame delivery failures by the sensor120over a period of time. In another implementation, instead of or in addition to the sensor status data134, the sensor data132may be accessed.

Block804accesses threshold data136. For example, the threshold data136may specify a maximum number of frame delivery failures that are permitted for a sensor120to be deemed operational.

Block806determines one or more of the sensor status data134or the sensor data132exceeds the threshold data136. For example, the number of missed frames indicated by the sensor status data134may exceed the value specified in the threshold data136.

As described above, in other implementations, instead of or in addition to the analysis of the sensor status data134, the sensor data132may be analyzed to determine the operational state of the sensor device124or one or more sensors120associated with the sensor device124.

Block808determines the sensor device124or one or more of the sensors120associated with the sensor device124has failed. For example, based at least in part on the determination that a value of the sensor status data134exceeds a value specified by the threshold data136, the operational status of the sensor device124may be determined.

Block810initiates one or more mitigation actions. One or more of the following mitigation actions may be taken independently of one another, in series, in parallel, and so forth.

Block810(1) may disregard sensor data132generated by the sensor device124or the sensor120that has been determined to have failed. For example, the disregarded sensor data132may be omitted from consideration by the data processing module330.

Block810(2) may restart, initialize, cycle the power, or initiate a reset of the sensor device124or the sensor120that has been determined to have failed. For example, the sensor device124may be instructed to shut down and restart.

Block810(3) deactivates the sensor device124or the sensor120. For example, the sensor device124may be instructed to power down the sensor120that is determined to have failed.

Block810(4) may access sensor data132from another sensor120that covers at least a portion of the region affected by the failure. Returning to the example ofFIG. 6, the sensor data132may be obtained from the overhead imaging sensors120(1)(3) and120(1)(4).

Block810(5) may generate an alarm. For example, the alarm may comprise a dispatch sent to one or more maintenance personnel or robots to repair or replace the failed sensor602.

Block810(6) may decrease the confidence value of output data138. In one implementation, the confidence value of output data138that is based at least in part on sensor data132obtained from a sensor120or sensor device124that is deemed to have failed may be decreased. For example, the sensor data132from a sensor device124that is deemed to be partially failed may be deemed to be less reliable and therefore be associated with a lower confidence value. In another implementation, the confidence value of output data138that is associated with the region within which one or more sensors120or sensor devices124have failed may be decreased. For example, sensor data132from the failed sensor602may be disregarded and the output data138may be determined based on sensor data132from other operational sensors120sensing at least a portion of the region. The confidence value of this output data138may be decreased given that less sensor data132was available to generate the output data138.

Block810(7) may utilize human input to mitigate the failure of the sensor120or the sensor device124. In one of implementation, data indicative of the failure and at least a portion of sensor data132may be provided to a manual processing system. The manual processing system may include the user interface500as described above. The user interface may receive input such as the results of analysis that have been obtained from the manual processing system. Output data138based on the input may then be generated. Returning to the example ofFIG. 6, image data obtained from the overhead imaging sensors120(1)(3) and120(1)(4) may be provided via the user interface500to a human operator. The human operator may then associate a particular user identifier504with a particular person depicted in the image data. This association may then be used to generate the output data138.

Block810(8) utilizes different processing techniques, or modifies existing processing techniques, to process the at least a portion of the available sensor data132. For example, the failed sensor602may be experiencing a delay in the delivery of sensor data132, such as providing image data at 15 frames per second instead of 30 frames per second. The data processing module330may “slow down” by dropping some of the frames of image data that are acquired by other sensors120that sense the same region. As part of this “slow down” to match the delay in delivery, the refresh rate of location data of the user116may be decreased to 15 times per second rather than 30 times per second.

The inventory management module324may utilize one or more of the techniques to synchronize sensor data132. Different sensors may deliver frames at different rates. For example, an imaging sensor120(1) may deliver 30 frames of image data per second, a failed sensor602(1) may deliver 2 frames of image data per second, and another failed sensor602(2) may deliver 7 frames of image data per second. Furthermore, the delivery of the frames to a processing device such as a server may be irregular. For example, traffic congestion on a local area network may introduce latency in the delivery of packets carrying the frames, the sensor120may be delayed in sending the frame due to other computational tasks, and so forth.

Several frames may be deemed to be synchronized when timestamps (or other indicia of time) associated with the frames occur within a particular time window or interval of time. For example, the interval of time may be about 70 milliseconds (ms) in width. Frames that occur within the same 70 ms window may thus be deemed to be synchronized. Thus, synchronized data may comprise a set of frames that have occurred contemporaneously or nearly-so with respect to one another.

Each of the sensors120may generate a feed of frames. For example, the feed may comprise one or more frames sent from a sensor120to another device such as a server. In one implementation, the feed may comprise a single connection that is maintained between the sensor120and the receiving device. In another implementation, the feed may comprise data sent over many connections between the sensor120and the receiving device. The many connections may be established in series, in parallel, or a combination thereof.

The frames in the feed may be sent at various times. For example, the frames in the feed may be sent at regular intervals. In another example, the frames may be sent at irregular intervals.

The frames may include header information such as a timestamp, sensor identifier, and so forth. For example, the timestamp may indicate the time, as reported by a clock internal to the sensor120, at which the data in the frame was acquired. The frames may include payload information, such as image data, weight data, or other information. For example, the payload for a frame generated by an imaging sensor120(1) may comprise a series of images captured over some interval of time.

The inventory management system may include a synchronization module. The synchronization module processes the feeds of frames from several sensors120and determines sets of frames that are within a common time window. The set of frames that are determined to be within a common interval of time or time window may then be designated as synchronized data. The synchronized data may be used for subsequent processes of the system100.

In one implementation, the synchronization module may generate synchronized data by using the following process. The frames from each feed may be stored in a separate buffer. For example, the facility102may include three cameras, each camera generating a feed of frames. The frames received from a first camera may be stored in a first buffer, frames from a second camera may be stored in a second buffer, and frames from a third camera may be stored in a third buffer. Each of the frames has a timestamp indicating a time the frame was generated by the respective sensor.

Continuing the implementation, the process may determine a first set of frames. The first set of frames includes the oldest frame from each of the buffers. Oldest, newest, and so forth, may be determined by comparing the timestamp of one frame to another, comparing the timestamp of one frame to a current time, and so forth.

Among the first set of frames, a timestamp value of a newest frame is determined. For example, the frame in the second buffer may be the newest or most recently generated of the three frames in the first set of frames.

A time window may be designated, with a newest point of the time window being set to the timestamp value of the newest frame in the first set of frames. For example, an end point of the time window may be set to the time indicated by the time value of the newest frame. The time window has a duration that extends from a start point to an end point, with the start point being at an earlier or older time than the later or newer end point. Successive time windows may occur at irregular intervals with respect to one another, and may even overlap with one another in some implementations. For example, where the time window is 70 ms in duration, a first time window may start at time=10 and end at time=80, while the second time window may start at time=17 and end at time=97.

The frames having timestamps before the start point of the time window may be discarded or otherwise disregarded from further consideration. For example, these frames may be removed from the buffer.

For each buffer, the frame within the time window having a newest timestamp is determined. For example, within the time window the third buffer may have two frames. The frame that has a later timestamp that is closer to current time may be determined. Continuing the example, the first buffer may have only one frame within the time window, and the second buffer may have only one frame within the time window. These frames may be designated as the newest within the time window for their respective buffers.

The frames for each of the buffers that occur within the time window and have the newest timestamp may be designated as synchronized data. Continuing the example, the three frames may be designated as synchronized data. The synchronized data may be sent to other systems or modules for further processing. Once sent, the frames in the synchronized data may be discarded from the buffer, and the process may continue, selecting another set of synchronized data. Frames that were not included in the synchronized data may remain in the buffer.

In some situations, frames may be absent from the buffer, frames may be distributed in time such that they do not fall within the time window, and so forth. The inventory management system126may specify service level guarantees, such as providing synchronized data within 1 second or less. A maximum delay value may be specified that indicates a maximum deviation from current time that frames may have to be considered.

In some implementations, frames that are older than this maximum delay value may be discarded from the buffers. In other implementations, a last known frame in the buffer may be retained and used in the synchronized data, even if that last known frame has a timestamp outside of the time window.

In other implementations, other techniques may be used to mitigate the failure. For example, the other sensors120having FOVs122that cover the region affected by the failure may be configured to increase the rate of data acquisition, such as increasing frame rate of the imaging sensor120(1).

FIG. 9depicts a flow diagram900of a process of determining and coping with failure of sensor devices124or sensors120, according to some implementations. In some implementations, the process may be performed at least in part by the server204.

Block902determines, at a first time, a first location of the user116at the facility102using point cloud data. For example, the point cloud data may be acquired by one or more of the depth sensors120(2). Data indicative of the first location may be used to track the user116, or another object, within the facility102.

Block904accesses sensor status data134indicative of operation of the depth sensor120(2). For example, the sensor status data134may be indicative of a number of frame delivery failures by the depth sensor120(2) over a period of time.

Block906accesses threshold data136specifying one or more thresholds for the sensor status data134. Continuing the example, the one or more thresholds for the sensor status data134may specify a maximum number of frame delivery failures.

Block908determines one or more values of the sensor status data134exceed the values specified by the threshold data136. Continuing the example, the sensor status data134may indicate 13 frame delivery failures in the past 1 second and the threshold may specify a maximum of 5 frame delivery failures per second.

Block910designates the depth sensor120(2) as failed. For example, the failure management module328may write data to the data store320that is indicative of the failure.

Responsive to the determination of the failure, block912accesses the image data obtained from one or more imaging sensors120(1) having a FOV122that includes at least a portion of a region associated with the depth sensor120(2) that failed.

Block914processes the image data to determine a portion of the user116or other object is in the image data. For example, the data processing module330may utilize one or more of the processing techniques described above such as ANN, cascade classifiers, and so forth, to determine a portion of the user116, such as a head, arm, leg, foot, and so forth. A transformation matrix may be used to associate the position in the image (such as row and column of associated pixels) of the portion to a particular location within the facility102. For example, the pixel having coordinates of (1217,773) may be associated with a location in aisle112(17) in front of inventory location114(3). Based on this association, a second location of the user116or other object may be determined. The second location may be the same as the first, or the object may have moved. Data indicative of the second location may be used to track the user116or other object at the facility102.

In some situations the second location may be confirmed or cross-checked using another process. Block916may determine, based at least in part on the point cloud data a predicted location of the user116or other object. Based at least in part on the point cloud data, a direction of motion of the user116and a speed of the user may be determined. For example, a time series of point cloud data may be used to determine one or more of a direction or velocity of the user116, such as the user116is heading north at 1 meter per second. An elapsed time since the point cloud data was acquired may be determined, such as 2 seconds. A distance traveled is determined by multiplying the speed of the user116by the elapsed time. Continuing the example, 1 m/s×2 s=2 meters traveled. Coordinates of the first location are summed with a vector comprising the distance traveled and the direction of motion. If the first location was at coordinates (1,1) then the predicted location may be at (1, 2). A threshold distance may be specified. When a distance between the predicted location and the second location are within the threshold distance, the second location may be deemed to be valid. When the distance between the predicted location and the second location are greater than the threshold distance, the system may take one or more actions such as re-processing the image data using another algorithm, acquiring new image data, generating an error message, and so forth.

Block918determines, at a second time, a second location of the user116in the facility102based on a position within the image data of the portion of the user116. For example, the data processing module330may use a transform function to associate a particular row and column of a pixel or group of pixels in the image data with a particular location or set of locations within the facility102. For example, a calibration target may be placed at a known location within the FOV122of the imaging sensor120(1), and the correspondence between the known location and actual position within the image may be used to generate the transform function. By recognizing the failure of the sensor120, the failure management module328may thus direct the data processing module330to determine the output data138, such as the location of the user116, using the image data.

In some implementations, the determination of the second location may be based at least in part on the first location and the one or more of direction of motion or speed of the object, such as determined at block916. For example, based on the direction of motion and the speed of the user116, the location at the second time may be estimated. This estimation may be compared with the location based on the image data. Where the estimation and location based on the image data are within a threshold value of one another, the second location may be assigned a confidence value is relatively high.

Responsive to the data indicative of the depth sensor as having failed, block920initiates one or more mitigation actions to attempt to restore the sensor120to operation. For example, the failed depth sensor may be rebooted. One or more of the mitigation actions as described herein may be taken independently of one another, in series, in parallel, and so forth.

In some situations, the mitigation actions may restore the sensor120or sensor device124to an operational status. A block may determine the sensor120is operational, subsequent to the one or more mitigation actions. Another block may acquire third sensor data132(3) from the sensor120that is now operational. Third output data may be generated using the third sensor data132(3).

FIG. 10depicts a flow diagram1000of a process of determining output data138during failure of a group of sensor devices124, according to some implementations. In some implementations, the process may be performed at least in part by the server204.

Block1002acquires first sensor data132(1) from a first sensor120(1) sensing a region. Block1004acquires second sensor data132(2) from a second sensor120(2) sensing at least a portion of the region. The first sensor120(1) and the second sensor120(2) may have FOV122that encompass the same volume of the facility102.

Block1006generates first output data138(1) using the sensor data132(1). The first output data138(1) may have a first confidence value indicative of predicted accuracy of the first output data138(1). In some implementations, the first output data138(1) may be generated using a first processing module or a first algorithm. For example, the first algorithm may be designed to process point cloud data to track a location of an object over time. The first algorithm may be optimized to use fewer computational resources such as processor cycles, memory, and so forth, compared to the second algorithm. Continuing the example, the second algorithm may be designed to recognize an object in image data, utilize a transform to determine a location of an object at the facility102, and so forth.

Block1008determines failure of a particular sensor120or sensor device124of the plurality of sensors120or sensor devices124. The block may determine the failure based at least in part on the sensor status data134. The block may access sensor status data134received from or associated with at least a portion of the plurality of sensors120, sensor devices124, and so forth. For example, the sensor status data134may be indicative of a rate of delivery of data from the particular sensor120. The failure may be determined based at least in part on the rate of delivery of data being less than a threshold specified by the threshold data136.

In another implementation, failure may be determined based at least in part on the sensor data132, such as described above. For example, a block may determine third output data138(3) based on sensor data132acquired from a third sensor120(3).

Another block may then determine difference data that is indicative of a difference between the third output data138(3) and the first output data138(1) exceeds a threshold amount. For example, the different may be calculated by subtracting values of the first output data138(1) from the third output data138(3). In another example, the difference may be based on analysis of the images, such as utilizing edge detection algorithms to determine edges in the respective output data, and comparing the size, shape, relative position, and so forth of those edges.

The failure may be determined based at least in part on the difference exceeding the threshold amount. With this implementation, two or more different sensors120or sensor devices124may be used to test one another. Should a discrepancy between them exceed a threshold amount, the failure may be determined.

In still another implementation, the failure may be determined based at least in part on the sensor data132. A block may determine difference data that is indicative of a difference between one or more values of the sensor data132from the particular sensor120and one or more values of threshold data136. The failure may be determined based at least in part on the difference exceeding the threshold amount. For example, the sensor data132may include data indicating a row of pixels has a measured average red channel value of 255. The threshold data136may specify a maximum average red channel value variance of 250. The difference may be calculated as 255−250=5. The threshold data136may also specify that a difference that is greater than 0 is indicative of a failure.

The sensor120associated with a failure may be determined using the sensor data132obtained by the particular sensor120or sensor device124. For example, metadata tags within the sensor data132may be read to determine a sensor identifier that is indicative of the first sensor120or sensor device124.

Block1010, based at least in part on the data indicative of failure, generates second output data138(2) using at least a portion of the second sensor data132(2). The second output data138(2) having a second confidence value indicative of predicted accuracy of the second output data138(2). In some implementations, the second confidence value is less than the first confidence value. For example, the second confidence value may be indicative of the fact that, based on the lesser sensor data132available, the accuracy of the output data138is decreased relative to when all sensors120servicing that region are operable.

As described above, in some implementations, the failure may comprise a slowdown in the delivery of sensor data132to the server204. In this situation, a block may determine first frames provided by the plurality of sensors120that have not been determined to have failed. The frames may comprise image data, point cloud data, or other sensor data132. Another block may then determine second frames provided by the particular sensor120determined to have failed. Synchronized data may be generated using at least a subset of the first frames and the second frames. For example, some of the delayed frames may be used in conjunction with the timely frames to generate synchronized data that may then be used by the data processing module330to generate the output data138.

The generation by block1010of the second output data138(2) may utilize a second processing module or algorithm. The second algorithm may consume more computational resources than the first algorithm while in operation. For example, the second algorithm may utilize additional cascade classifiers, ANNs, pattern recognition algorithms, and so forth to process image data to recognize the object to be tracked, determine motion of the object, and so forth.

The various sensors120and sensor devices124within the facility102may have different capabilities, such as described above. In one implementation, the plurality of sensors120may include a first set of imaging sensors120(1)(1) and a second set of imaging sensors120(1)(2). The second set of imaging sensors120(1)(2) may generate images with a higher resolution than the first set of imaging sensors120(1)(1). The sensor data132used to generate the first output data138(1) may be obtained from the first set of imaging sensors120(1)(1), and the sensor data132used to generate the second output data138(2) may be obtained from the second set of imaging sensors120(1)(2).

Block1012initiates one or more mitigation actions. For example, one of the mitigation actions described above with regard toFIG. 8may be initiated. In some implementations, the data processing module330may slow down the rate at which output data138is generated. For example, the inventory management module324may generate the first output data138(1) at a first rate of output per unit of time. As the sensors120or sensor devices124fail, the inventory management module324may generate the second output data138(2) at a second rate of output per unit of time, wherein the second rate is less than the first rate of output per unit of time. As a result, the output data138may be delivered less frequently, but still is being delivered in spite of the failure, allowing at least partial functionality of the inventory management module324.

In some situations, one or more of the mitigation actions may be successful in restoring the previously failed sensor120or sensor device124to an operational state. In such a situation, a block may determine the particular sensor120or sensor device124is operational. Based at least in part on this determination, the sensor data132acquired by the particular sensor120or sensor device124may be used to generate third output data138(3).

FIG. 11depicts a flow diagram1100of a process of determining output data138during failure of a group of sensor devices124, according to some implementations. In some implementations, the process may be performed at least in part by the server204.

Block1102determines a region of a facility102that is impacted by failure of a first group of sensor devices124that are sensing at least a portion of the region. The determination as to which sensor devices124have failed may be made as described above. For example, the determination of the failure may be based at least in part on a comparison between the threshold data136and one or more of the sensor data132or the sensor status data134.

Block1104identifies an object in the region. The identification may be based at least in part on sensor data132such as point cloud data, image data, weight data, data obtained from an RFID tag, from an optical tag, and so forth. For example, the data processing module330may use one or more data processing techniques such as ANNs or cascade classifiers to recognize that the object is a user116.

Block1106disregards first sensor data132(1) acquired from the first group of sensors devices124. For example, the data processing module330may not use the first sensor data132(1) for the generation of output data138.

Block1108determines, using physical layout data130of the facility102, a second group of sensor devices124sensing at least a portion of the region. Returning to the example ofFIG. 6, it may be determined that the depth sensor120(2)(6) has some overlap in the region that is within the sensor blackout region604produced by the failure of the depth sensor120(2)(5).

Block1110accesses second sensor data132(2) acquired from the second group of sensor devices124. The second group of sensor devices124may include the same types of sensors120as the first group of sensor devices124or may include different types of sensors120. For example, the first group of sensor devices124may be imaging sensors120(1) while the second group of sensor devices124may be depth sensors120(2).

Block1112determines, using the second sensor data138(2), motion of the object with respect to the region. For example, optical flow techniques may be used to determine the motion of the user116between two points in time.

Block1114determines output data138based on the second sensor data138(2), the identity of the object, and the motion.

For example, block1104may determine the object is a user116, such as the user116. Block1112may determine the motion indicates the user116entered the region. Block1114may receive third sensor data132(3) from an instrumented auto-facing unit in the region indicating a pick or place of one or more items104. Block1112may determine the motion indicates the user116exited the region. Block1114may then determine the output data138based at least in part on the third sensor data132(3). In this example, no other users116were detected within the region. The motion of the user116was into the region before the pick was detected by the instrumented auto-facing unit, and the motion of the user116out of the region was detected after the pick was detected. Based on this data, the output data138may indicate that the user116picked an item104from the inventory location114associated with the instrumented auto-facing unit.

In some implementations, block1114may access, based on the identity of the object, data indicative of one or more of physical characteristics or behavior of the object. For example, where the object is identified as user116, the data may include a maximum speed the user116is expected to move, details about shape and size typically associated with the user116, and so forth. The output data138may be determined based at least in part on the data indicative of one or more of physical characteristics or behavior of the object.