Entry portal identification system

An entry portal to a facility may include radio receivers with directional antennas and cameras at known positions in the facility. The receivers acquire device identifier and device data from a mobile device carried by a user passing through the portal. Portal data is generated that is indicative of the coverage volume of a particular antenna that received a signal carrying the device identifier and signal strength of that signal. The cameras acquire images of the user. Other sensors may acquire other information included in the portal data. The portal data, device data, device identifier, and the images are processed to associate a particular account with a person passing through the entry portal. Subsequently, that person may then be tracked using images from other cameras in the facility.

BACKGROUND

Retailers, wholesalers, and other product distributors typically maintain an inventory of various items that may be ordered, purchased, leased, borrowed, rented, viewed, and so forth, by clients or customers. For example, an e-commerce website may maintain inventory in a fulfillment center. When a customer orders an item, the item is picked from inventory, routed to a packing station, packed, and shipped to the customer. Likewise, physical stores maintain inventory in customer accessible areas (e.g., shopping area), and customers can pick items from inventory and take them to a cashier for purchase, rental, and so forth. Many of those physical stores also maintain inventory in a storage area, fulfillment center, or other facility that can be used to replenish inventory located in the shopping area or to satisfy orders for items that are placed through other channels (e.g., e-commerce). Other examples of entities that maintain facilities holding inventory include libraries, museums, rental centers, and so forth. In each instance, for an item to be moved from one location to another, it is picked from its current location and transitioned to a new location. It is often desirable to monitor the entry of users into the facility, movement of inventory, users, and other objects within the facility, and so forth.

DETAILED DESCRIPTION

This disclosure describes systems and techniques for associating an account with a person at a portal. In one implementation, one or more portals may be deployed at a materials handling facility (facility).

The facility may include, or have access to, an inventory management system. The inventory management system may be configured to maintain information about items, users, condition of the facility, and so forth. For example, the inventory management system may maintain data indicative of what items a particular user is ordered to pick, location of the particular user, availability of a user providing support services to others, requests for assistance, environmental status of the facility, and so forth. The inventory management system, or another system, may generate this data based on sensor data, such as images acquired from imaging sensors like cameras, weight sensors in shelves, and so forth. For example, the images may be used to identify an object such as a user or item, track the object, and so forth.

The portal allows one or more users to enter the facility without impediment. From the perspective of a person passing through the portal, they pass without having to take any action. After passage through one or more portals, an account associated with a particular user may be associated with the person. Once the account has been associated, further tracking may maintain that association. For example, once identified using the portal, image data from cameras in the facility may be processed to track that identified user as they move about in the facility.

The portal comprises one or more antennas coupled to a radio. The antennas may be directional in that they provide gain in a particular direction at a particular frequency. The antennas may have fixed directionality or may be electronically steerable. The antennas are arranged to provide coverage within an area through which one or more users may pass.

Users may have in their possession one or more mobile devices. For example, the mobile device may comprise a smartphone, wearable computer, fitness tracker, and so forth. The mobile device may have a device identifier, such as a media access control (MAC) address, integrated circuit card identifier (ICCID), serial number, and so forth.

The mobile device may include one or more sensors, such as accelerometers, gyroscopes, magnetometers, and so forth. The mobile device acquires data from these one or more sensors and generates device data. The device data include accelerometer data, gyroscope data, compass heading obtained from a magnetometer, and so forth.

As the mobile device passes through the portal with the user, the radio in the portal antennas may emit a portal signal. The mobile device, responsive to the portal signal, transmits a device signal. For example, the portal signal may be sent as a Bluetooth broadcast. The device signal sent by the mobile device may also be sent as a Bluetooth broadcast. In this implementation, the portal signal and the device signal may be provided without the communications overhead associated with establishing a connection between the devices.

The device signal may be used to transfer data. In one implementation, the payload of the Bluetooth broadcast sent by the mobile device may include the device identifier and at least a portion of the device data. For example, the Bluetooth broadcast may include the MAC address and accelerometer data for the 5 seconds prior to transmission.

The portal is configured to generate portal data. In one implementation, the portal data may include data indicative of received signal strength of the device signal and information indicative of the specified volume associated with the antenna. For example, the specified volume may comprise a volume through which a primary lobe of the gain associated with the antenna passes. In another implementation, the portal data may comprise a distance and a relative bearing indicative of a direction to the transmitter on the mobile device. In yet another implementation, the portal data may include coordinates relative to the facility that indicates a determined location of the transmitter on the mobile device.

The accuracy of a determined location of a transmitter may be limited. For example, the location of a transmitter may be determined to within 50 centimeters (cm). However, during operation, individual persons in the portal may be so close that individuals cannot be distinguished using portal data.

While the accuracy of the transmitter may be limited, other information may also be determined from the portal data. An estimated motion value may be generated from the portal data that provides information about movement of the mobile device at the portal. For example, the portal data acquired over a period of time may be used to determine a first location in space at a first time and a second location in space at a second time. Based on the difference in time, distance between the first location and the second location, the estimated motion value may comprise a vector indicative of a speed and direction of change in location.

The device data received from the mobile device provides information about the movement of the mobile device. The device data may be processed to generate an inertial motion value. The inertial motion value provides information indicative of movement of the sensor associated with the mobile device. For example, the accelerometer data obtained over a period of time may be integrated to determine an overall vector indicative of change in speed and direction of change in location, with magnetometer data providing information such as a compass heading.

The estimated motion value and the inertial motion value may be compared to determine if the two are within a threshold value of one another. For example, the estimated motion value may indicate that the transmitter of the mobile device was moving with a heading of 170 at 0.89 meters/second (m/s). The inertial motion value may indicate that the mobile device was moving with a heading of 181° at 0.76 m/s. The threshold value may indicate that headings within 20° of one another are to be deemed similar, and that speeds within ±0.15 m/s of one another are deemed to be similar. Continuing the example, the estimated motion value and the inertial motion value are determined to be within a threshold value of one another.

In some implementations, the portal may be configured with one or more features or obstructions designed to elicit a particular set of accelerations or changes in direction of the mobile device. For example, the portal may include decorative features such as posts that the user carrying the mobile device may navigate around. Given a known location of these obstructions, or using information from testing during set up of the portal, a profile of particular accelerations may be determined. Additionally, the antennas may be configured to have gain specific to one side of the obstruction or another. As a result, the portal data and the corresponding estimated motion value and inertial motion value may indicate a particular displacement, such as a turn to the left. Combined with other information such as received signal strength, the presence and location of the mobile device may be determined.

Account data may have been previously stored that associates a particular device identifier with a particular account. Based on the determination that the estimated motion value and the inertial motion value correspond, presence data may be generated that indicates a mobile device associated with the particular account is present at the portal.

The presence data may be utilized by the inventory management system to facilitate operation of the facility. For example, image data may be obtained using a camera with a field-of-view including a least a portion of the portal. As the particular account for a particular person who is a user is determined at the portal, the image of the person may be associated with the particular account for further tracking. Instead of, or in addition to, the use of portals throughout the facility, cameras may be used to provide for optical tracking of the user as they move about the facility.

In some implementations, the image data obtained from the camera covering the portal may be used to assist in the association of the person at the portal with a particular account. For example, the image data may be processed to generate one or more of an image gait value or an image motion value. The image gait value may provide information associated with a gait of the person within the portal. For example, the image gait value may indicate a gait cadence of the person within the portal based on image data of the feet of the person coming into contact with the floor of the portal. In another example, the image motion value may describe a trajectory of the person based on changes in apparent motion of at least a part of the person, changes in apparent position within the image data relative to other objects depicted in the image data, and so forth. In some implementations, depth data may also be obtained, such as using a depth sensor. By processing depth data obtained at different times, information such as speed, direction, and so forth, of an object such as the user may be determined.

One or more of the image gait value or the image motion value may also be used to disambiguate between different persons depicted in the image data obtained at the portal. For example, a group of people may pass through the portal at the same time. Image data alone may be insufficient to provide an accurate count of the persons at the portal. For example, a group of people may be walking closely together, carrying children, and so forth, which may result in a count based on the image data of those people entering the facility that is incorrect. The portal data may be processed to determine a count of distinct device identifiers at the portal. This count may be used to determine the number of people carrying mobile devices passing through the portal.

Image data may also be insufficient to accurately identify persons based on their appearance. For example, sunglasses, attire, jewelry, changes in hair color, and so forth, may limit the accuracy of identification based only on the image data. By utilizing the other information acquired at the portal, a person may be associated with a particular account. Furthermore, that person may be associated with a particular portion of the image data.

In one implementation, the correspondence between the inertial motion value and the image gait value may be used to disambiguate one person from another that appear within the same frames of image data. For example, the image data may include images of three people moving through the portal. A confidence value of identification based on image data alone may be below a threshold value indicating that such image-based identification is unreliable. The image data may be used to determine information such as the image gait value for each of the persons appearing in the image. By comparing the inertial motion value to the image gait value, the account may be associated with a particular person appearing in the image data. Continuing the example, the first person may have a gait cadence of 60 steps per minute, the second person may have a gait cadence of 35 steps per minute, and the third person may have a gait cadence of 50 steps per minute. The comparison between the image gait value and the inertial motion value may be used to distinguish between these three people.

By combining the information from different sources such as the receivers at the portal, the sensors on board the computing device, and cameras, the presence of a user associated with a particular account may be determined at the portal. Furthermore, a particular account may be associated with a particular image of a person at the portal, facilitating further tracking of the user with image data. This information may then be used by an inventory management system to associate particular interactions such as the pick or place of items from an inventory location with a particular account. As a result, the overall operation of the facility and the user experience may be improved.

Illustrative System

FIG. 1illustrates a system100to provide a portal at a facility through which people may travel unimpeded and be identified as users, according to some implementations. A user102having a mobile device104in their possession may approach and pass through a portal106. The portal106may include one or more structures such as walls, posts, pillars, decorative elements, and so forth. These structures may be used to direct users102through the portal106without necessarily impeding their ability to pass through the portal106. For example, the portal106may comprise a curving corridor, entryway to the facility, decorative plants around which the user's102must move to enter, and so forth.

The mobile device104may comprise one or more of a smartphone, wearable computer, fitness tracker, and so forth. The mobile device104may include a radio108. The radio108may comprise one or more of a transmitter or receiver suitable for sending and receiving signals. The signal may be used to convey information. For example, the radio108may be part of a wireless network communication interface that is compatible with one or more of Bluetooth, Wi-Fi, and so forth. In another example, the radio108may comprise a Bluetooth Low Energy (BLE) transmitter.

The mobile device104may include one or more sensors110. For example, the sensors110may include an accelerometer, a gyroscope, a magnetometer, a navigation system such as a Global Positioning System (GPS) receiver, acoustic transducer or microphone, and so forth. Output from one or more the sensors110may be used to generate device data112. The device data112provides information about movement of the mobile device104. For example, device data112may comprise a set of measurements at particular times indicated by timestamps. The measurements may be obtained by the accelerometer, compass heading information obtained from the magnetometer, velocity information obtained from the navigation system, and so forth. In some implementations where the portal106includes magnets to generate magnetic fields, the magnetometer data may indicate compass headings or field strengths that result from these magnets.

The portal106may include other types of emitters or sources that generate signals detectable by the mobile device104. For example, the portal106may include ultrasonic transducers or speakers that generate ultrasonic sound. The sound may then be detected by microphone of the mobile device104. Information about the detected ultrasonic signal may then be included in the device data112.

The mobile device104may have a device identifier114. The device identifier114may comprise information such as a media access control (MAC) address, integrated circuit card identifier (ICCID), serial number, and so forth.

The mobile device104may use the radio108to send at least a portion of the device data112and the device identifier114to one or more radios108associated with the portal106. For example, the portal106may have a radio108coupled to an antenna116. The antenna116may be configured to provide gain to a particular volume within the portal106. In some implementations, the antenna116may comprise sub elements or sub antennas arranged into a steerable array. For example, the antenna116may comprise a phased array. The phased array may include a plurality of individual antenna elements that are arranged in a particular physical configuration and which transfer radio frequency energy to a transmission line with predetermined electrical characteristics to produce particular electromagnetic coupling effects between one or more of the antenna elements. As a result of these particular electromagnetic coupling effects, gain may be realized in a particular direction. The phased array may be fixed and configured such that gain is provided in a particular direction, or may be steerable such that a direction of the gain may be changed. In another example, the antenna116may comprise a dipole placed in front of a radio frequency reflector, a Yagi-Uda array having a driven element and one or more director elements or reflector elements, a parabolic antenna, and so forth. In some implementations, a plurality of antennas116may be coupled to a single radio108.

As illustrated here, the portal106comprises a passageway lined with the antennas116. The antennas116may be located to one or more sides of the passageway, above, below, or any combination thereof. Signals acquired by the antennas116and detected by the radio108may be processed to generate portal data118. For example, the radio108may detect a signal received from the mobile device104that then may be demodulated to produce the portal data118.

In some implementations, one or more other sensors110may be present at the portal106. For example, weight sensors may detect the force of one or more of the user's102feet during passage through the portal106.

The portal data118comprises information about a signal received by the antennas116of the portal106. The mobile device104may be configured to respond to a portal signal sent by the radio108of the portal106. For example, the radio108of the portal106may broadcast a portal signal using the Bluetooth protocol. In response to receiving the portal signal, the radio108of the mobile device104may send the device signal. The device signal may include the device data112and the device identifier114of the mobile device104. The portal data118may comprise received signal strength values indicative of signal strength of the device signal as acquired by one or more of the antennas116, timestamp indicative of when the signal is received, and the device identifier114associated with the device signal.

In some implementations, the portal data118may comprise other information that is based on or derived from at least a portion of the device signal. For example, one or more of the systems associated with the portal106may be configured to determine a relative bearing and a distance to the mobile device104based on the device signal. Continuing the example, the radio108and antenna116may comprise a system with a hardware processor to generate the relative bearing and distance information.

The portal106may be configured with one or more magnets or electromagnetics that generate a magnetic field. These magnetic fields may detectable by a magnetometer of the mobile device104. For example, magnets may be arranged to produce a particular magnetic field. Based on the movement through the magnetic field, changes in the magnetometer data included in the device data112may be used to determine information about the position of the mobile device104at a particular time.

In another implementation, the radio108of the mobile device104may be configured to generate device data112that may be used in place of, or in addition to, the portal data118. For example, the radios108of the portal106may transmit signals that are then received by the radio108of the mobile device104. The device data112obtained by the mobile device104may comprise the received signal strength at the mobile device104along with one or more of a timestamp or data indicative of a particular transmission. For example, each transmission made by the portal106may include transmission identification number. The device data112may then be processed to determine a location of the mobile device104at the time the transmissions were received.

The device data112, device identifier114, and the portal data118may be provided to a server120. In one implementation, the device data112and the device identifier114may be received by the radio108at the portal106and relayed to the server120. In another implementation, the mobile device104may establish a connection with the server120and send the device data112by way of the connection.

The server120may also be configured to receive image data122obtained from one or more imaging sensors or cameras110(1). The camera110(1) may be configured to generate image data122comprising still images, video, or both of at least a portion of a volume of the portal106through which the users102may pass. For example, a first camera110(1)(1) may be configured to acquire images of the faces of the users102, while a second camera110(1)(2) may be configured to acquire images of the legs and feet of the users102.

The server120may execute at least a portion of an analysis module124. The analysis module124is configured to use one or more of the device data112, a device identifier114, the portal data118, or the image data122during operation. In one implementation, the analysis module124may determine the movements of the mobile device104associated with the user102(as indicated by the device data112) correspond to information about those movements that are determined from the signals received by the radio108at the portal106, the image data122acquired by the camera110(1) at the portal106, and so forth. This determination is described in more detail in the following figures.

Account data126may be accessed that associates particular device identifier values with particular accounts. For example, the account data126may comprise a table or other data structure that relates a MAC address with a particular account number. The device identifier114that is received from the mobile device104may be used to look up the particular account in the account data126.

Based at least in part on the information obtained at the portal106and the correspondence between the device identifier114and the information in the account data126, presence data128may be generated. The presence data128may comprise information indicative of the presence of a person associated with a particular account at the portal106at a particular date and time.

The presence data128may be provided to an inventory management system130. The inventory management system130may be configured to maintain information about items, users102, condition of the facility, and so forth. For example, the inventory management system130may maintain data indicative of a number of items at a particular inventory location, what items a particular user102is ordered to pick, how many items have been picked or placed at the inventory location by the particular user102, requests for assistance, environmental status of the facility, and so forth.

While this disclosure describes the use of radio frequency (RF) signals, it is understood that in other implementations optical, acoustic, magnetic, or other signals may be used instead of, or in conjunction with, RF signals.

FIG. 2depicts an overhead view200of a portion of the facility depicting the location of various portals106, according to some implementations. In some implementations, the facility may include one or more inventory locations202. The inventory locations202may include one or more of a shelf, a rack, a case, a cabinet, a bin, a floor location, or other suitable storage mechanisms for holding, supporting, or storing items204.

The inventory locations202may be arranged in various physical configurations, such as aisles206. The inventory locations202may be affixed to the floor or another portion of the structure of the facility. The inventory locations202may also be movable such that the arrangements of aisles206may be reconfigurable. In some implementations, the inventory locations202may be configured to move independently of an outside operator. For example, the inventory locations202may comprise a rack with a power source and a motor, operable by a computing device to allow the rack to move from one location within the facility to another.

In some implementations, the end of an aisle206may include an endcap208. The endcap208may comprise inventory locations202, promotional displays, and so forth. For example, the endcap208may include another inventory location202such as shelving or refrigerated case to hold items204.

One or more portals106may be arranged within the facility as shown here. In some implementations, the portals106may be configured to constrain movement of the users102, such as providing a path along which the user102may travel. In some situations, obstructions210may be used to elicit a particular direction of travel with respect to a portal106. For example, a particular obstruction210may result in a direction of travel that brings the user102closer to an antenna116of the portal106. In another example, the obstruction210may result in a particular pattern of motion that may be detected when analyzing the device data112obtained from the mobile device104.

The use of multiple portals106within the facility may facilitate the association of a particular person with a particular account. For example, a group of users102may enter the facility passing through a first portal106(1). Members of the group of users102may subsequently split off and go about their own individual tasks, passing through other portals106along the way. By separating from one another, a particular person may be associated with a particular account. A user102may utilize a tote212to carry one or more items204while at the facility. The tote212is discussed in more detail below.

FIG. 2also illustrates an example of presence data128. For example, a group that includes users102(1),102(2), and102(3) is shown appearing at an entry portal106at a particular time. Later presence data128indicates individual users102present at different portals106within the facility after they have separated from the group.

FIG. 3is a block diagram300of an analysis module124that can determine presence data128that identifies the user102at the portal106, according to some implementations. In some implementations, at least a portion of the analysis module124may execute on the server120. The analysis module124may include, or operate in conjunction with, one or more of the following modules.

An estimated motion module302is configured to process the portal data118and generate an estimated motion value304. The estimated motion value304comprises information indicative of movement of the mobile device104based on one or more signals transmitted by the radio108of the mobile device104. In some implementations, the estimated motion value304may comprise one or more vectors that are indicative of direction and speed. In some implementations, the estimated motion value304may include a temporal component. For example, the estimated motion value304may provide information of changes in the motion over time.

The estimated motion module302may use portal data118obtained at different times to generate the estimated motion value304. For example, the estimated motion module302may determine a first location of the mobile device104at a first time and a second location of the mobile device104at a second time. A distance between the first location and the second location may be calculated, as well as total time between the first time and second time. Based on a direction of travel extending from the first location to the second location, the distance, and the total time, the estimated motion value304may be calculated.

The location of the mobile device104may be determined using a variety of techniques. In one implementation, the location may be determined based on received signal strength and the known location of one or more antennas116. The estimated motion module302may access physical layout data306.

The estimated motion value304may also include information indicative of a gait of a user102. For example, weight sensors110(6) in the floor of the portal106may provide portal data118that is processed by the estimated motion module302to determine force of one or both of the user's102feet, spacing, cadence, and so forth, associated with the gait of the user102.

The physical layout data306may provide information indicative of where antennas116are located, location of sensors110such as the cameras110(1) or weight sensors110(6), position of the inventory locations202, and so forth. For example, the physical layout data306may comprise information representative of a map or floor plan of the facility with relative positions of antennas116, portals106, obstructions210, walls and floors, magnetic field sources, ultrasonic transducers, data indicative of how items204are to be arranged at the inventory locations202, and so forth. The physical layout data306may associate a particular inventory location ID with other information such as physical location data, sensor position data, sensor direction data, sensor identifiers, and so forth. In some implementations, the physical location data may be relative to another object. For example, the physical location data may indicate that a particular antenna116or magnetic field source is located at a particular side of the portal106.

Returning to the estimated motion module302, information about the relative or absolute location of the antennas116at the portal106may be obtained from the physical layout data306. Given this information, and based on received signal strength obtained from the antennas116at the portal106, an estimate of the location of the mobile device104may be obtained.

In another implementation, the antenna116may be steerable such that gain may be directed in a particular direction. The steering of the antenna116may be done manually, electronically, fluidically, and so forth. For example, the antenna116may comprise an electronically steerable phased array having a plurality of fixed antenna elements for which electrical phasing may be varied to provide for directionality in an antenna pattern without moving any physical elements of the fixed antenna elements. During operation, data about an angular bearing at which gain of the antenna pattern is directed may be determined. The angular bearing may be used to determine location of the transmitter. For example, two electronically steerable phased array antennas116(1) and116(2) may be separated by known distance. A location of the mobile device104may be triangulated by using angular bearings from the first antenna116(1) and the second antenna116(2).

In yet another implementation, an angular bearing provided by a steerable antenna116may be combined with an estimated distance to the transmitter to determine location. The estimated distance to the transmitter may be based at least in part on the value of the received signal strength.

In other implementations, other techniques may be used to determine an estimated location of the mobile device104based on signals transmitted by the mobile device104. The estimated motion value304may be provided to a comparison module308. The comparison module308is described in more detail below.

An inertial motion module310may be configured to process the device data112to generate an inertial motion value312. The inertial motion value312provides information indicative of movement of the sensor110associated with the mobile device104. In some implementations, the inertial motion value312may comprise one or more vectors that are indicative of direction and speed. In some implementations, the inertial motion value312may include a temporal component, such as information indicative of changes in motion over time. The inertial motion value312may be provided to the comparison module308. The comparison module308may use the physical layout data306to compare the inertial motion value312with the physical layout data306. Information about the physical layout of structures in the facility may be known to elicit a particular pattern of motion. For example, the obstruction210may result in a change of direction in the path of the mobile device104that may be determined from the inertial motion value312. Based on this determination, the presence of the mobile device104near the obstruction210may be determined.

Information about the biomechanics of the user102such as the general anatomical shape and expected dimensions of the user102, likely locations for the mobile device104relative to the user102, construction details about the mobile device104, position of the sensors110with respect to the mobile device104, and so forth, may be used as inputs to the inertial motion module310. In one implementation, the inertial motion module310may utilize a linear quadratic estimation (LQE) or Kalman filtering to generate the inertial motion value312based on the device data112.

An image processing module314may be configured to process the image data122to generate one or more of image gait value316or image motion value318. The image gait value316may be indicative of one or more aspects associated with movement of one or more limbs of the user102. For example, the image gait value316may indicate a cadence such as a number of steps per minute made by the user102, information about time intervals between steps, and so forth. In another example, the image gait value316may include information indicative of particular times at which the foot of the user102touches the floor, leaves the floor, and so forth.

Gait may be affected by height, age, and other factors associated with the user102. Gait recognition techniques may be used to analyze the relative position and motion of limbs of the user102in the image data122. Limbs may include one or more arms, legs, and in some implementations, the head. In one implementation, edge detection techniques may be used to extract a position of one or more limbs of the user102in the series of images. For example, a main leg angle of a user's102leg may be determined, and based on the measurement of this main leg angle over time and from different points-of-view, a three-dimensional model of the leg motion may be generated. The change in position over time of the limbs may be determined and used to generate the image gait value316.

The image motion value318comprises information indicative of movement of at least a portion of the user102based on one or more of the images in the image data122. In some implementations, the image motion value318may comprise one or more vectors that are indicative of direction and speed of at least a portion of the user102. For example, the vector value may indicate an apparent or actual motion of a point that is based on the head and shoulders of the user102. In some implementations, the image motion value318may be determined using an optical flow function, such as found in the OpenCV library. In another implementation, stadiametric ranging, such as based on an assumed average distance between the eyes of a user102, average size of head, and so forth, may be used to determine a distance from the camera110(1) to the user102. Based on this information, the location of the user102may be determined. A mapping matrix may associate particular locations within a frame of an image to particular locations in the facility. By using the stadiametric ranging and the mapping matrix, a location within the facility may be determined in a given image. By using information from locations at successive times, direction, speed, acceleration, and so forth, may be calculated.

In some implementations, the camera110(1) may comprise a depth sensor110(2) such as described below. The depth sensor110(2) may produce data indicative of the location, speed, direction, acceleration, and other information about an object in view of the depth sensor110(2). The image motion value318may be based at least in part on this information.

The image data122may be acquired from cameras110(1) at different locations relative to the portal106. For example, a pair of cameras110(1) may provide for stereoscopic image acquisition. In other implementations, the image motion value318may be determined based at least in part on three-dimensional data obtained from a depth sensor110(2).

The comparison module308is configured to determine similarity between one or more input values. While various combinations of comparisons are described, it is understood that any combination of inputs based on one or more of the device data112, the portal data118, or other data may be used. In some implementations, comparison module308may utilize one or more threshold values320. The threshold values320may specify one or more of the minima, maxima, range, percentage, and so forth. The comparison module308provides comparison data322as output. The comparison data322may indicate that the values being compared are determined to represent the same user102.

The comparison module308may determine if the estimated motion value304is within a threshold value320of an inertial motion value312. For example, the estimated motion value304may indicate a vector having a heading of 170° at 0.89 m/s. The inertial motion value312may indicate a vector having a heading of 181° at 0.76 m/s. The threshold value320may indicate that headings within 20° of one another are to be deemed similar, and that speeds within ±0.15 m/s of one another are deemed to be similar. Continuing the example, the estimated motion value304and the inertial motion value312are determined to be within a threshold value320of one another, and comparison data322indicative of this may be generated.

The comparison module308may also determine if the estimated motion value304is within a threshold value320of the image motion value318. For example, the estimated motion value304may indicate a vector having a heading of 170° at 0.89 m/s. The image motion value318may indicate a vector having a heading of 167° at 0.81 m/s. The threshold value320may indicate that headings within 10° of one another are to be deemed similar, and that speeds within ±0.10 m/s of one another are deemed to be similar. Continuing the example, the estimated motion value304and the image motion value318are determined to be within a threshold value320of one another, and comparison data322indicative of this may be generated.

The comparison module308may also determine if the estimated motion value304is within a threshold value320of the image gait value316. For example, the estimated motion value304may provide information that the user102is walking with a first step cadence of 17 steps per minute. The image gait value316may indicate the user102is walking with a second step cadence of 15 steps per minute. The threshold value320may indicate that cadences within 3 steps per minute of one another are to be deemed similar. Continuing the example, the estimated motion value304and the image gait value316are determined to be within a threshold value320of one another, and comparison data322indicative of this may be generated.

The comparison module308may determine if the inertial motion value312is within a threshold value320of one or more of the estimated motion value304indicative of gait, the image gait value316, or the image motion value318. The estimated motion value304may comprise information indicative of the motion of the user102, such as the gait based on the portal data118. The inertial motion value312may comprise information indicative of the motion of the user102walking through at least a portion of the portal106, such as timing of the steps of the user102. The inertial motion value312may be compared with image gait value316or the estimated motion value304that also indicate timing of the steps of the user102. If the timing of the steps between the two corresponds within the threshold value320, the two values may be deemed to be indicative of the same user102and comparison data322may be generated.

The comparison module308may determine if the inertial motion value312is within a threshold value320of one or more of the image motion value318. The inertial motion value312may be indicative of a first trajectory of the user102based on the device data112. The image motion value318may be indicative of a second trajectory of the user102based on the image data122. If the first trajectory and the second trajectory are within a threshold value320of each other, comparison data322may be generated indicating that they are representative of the same user102.

The comparison module308may use different threshold values320for different types of comparisons. For example, comparisons between estimated motion values304and inertial motion values312may use a first threshold value320(1) while comparisons between the estimated motion value304and the image motion value318may use a second threshold value320(2).

The comparison module308may generate comparison data322based on one or more comparisons, such as those described above. For example, the comparison module308may generate comparison data322based on comparisons between the estimated motion value304, the inertial motion value312, the image gait value316, and the image motion value318. The comparison data322may include the device identifier114associate with the comparisons. For example, the device identifier114included in the portal data118may be included in the comparison data322involving the estimated motion value304. In another example, the device identifier114associated with the device data112may be included in the comparison data322involving the inertial motion value312.

The comparison data322is accessed by an identification module324. The identification module324accesses account data126. The account data126provides information that associates a particular device identifier114with a particular account. For example, the MAC address 74:72:6f:70:69:63 of the mobile device104that sent the device data112may be used to query a data store of the account data126. The query may return a result of the account identifier 3032674905 that identifies the account of user John Smith.

The identification module324may generate corresponding presence data128that indicates the presence of a mobile device104associated with the particular account has been detected at a particular portal106. In some implementations, it may be assumed that the mobile device104is being carried by the user102associated with particular account.

FIG. 4is a block diagram400illustrating a materials handling facility (facility)402using the system100, according to some implementations. A facility402comprises one or more physical structures or areas within which one or more items204(1),204(2), . . . ,204(Q) may be held. As used in this disclosure, letters in parenthesis such as “(Q)” indicate an integer value greater than or equal to zero. The items204may comprise physical goods, such as books, pharmaceuticals, repair parts, electronic gear, and so forth.

The facility402may include one or more areas designated for different functions with regard to inventory handling. In this illustration, the facility402includes a receiving area404, a storage area406, and a transition area408.

The receiving area404may be configured to accept items204, such as from suppliers, for intake into the facility402. For example, the receiving area404may include a loading dock at which trucks or other freight conveyances unload the items204. In some implementations, the items204may be processed, such as at the receiving area404, to generate at least a portion of item data. For example, an item204may be imaged or otherwise scanned to develop reference images or representations of the item204at the receiving area404.

The storage area406is configured to store the items204. The storage area406may be arranged in various physical configurations. In one implementation, the storage area406may include one or more aisles206. The aisle206may be configured with, or defined by, inventory locations202on one or both sides of the aisle206. In some implementations, one or more portals106may be deployed within the facility402. For example, a portal106may be deployed within an aisle206by placing one or more antennas116overhead, within the inventory locations202, on either side of the aisle206, and so forth.

One or more sensors110may be configured to acquire information in the facility402. The sensors110may include, but are not limited to, cameras110(1), depth sensors110(2), weight sensors110(6), optical sensor arrays110(13), proximity sensors110(14), and so forth. The sensors110may be stationary or mobile, relative to the facility402. For example, the inventory locations202may contain weight sensors110(6) to acquire weight sensor data of items204stowed therein, cameras110(1) to acquire images of picking or placement of items204on shelves, optical sensor arrays110(13) to detect shadows of the user's hands at the inventory locations202, and so forth. In another example, the facility402may include cameras110(1) to obtain images of the user102or other objects in the facility402. The sensors110are discussed in more detail below with regard toFIG. 5.

While the storage area406is depicted as having a single aisle206, inventory locations202storing the items204, sensors110, and so forth, it is understood that the receiving area404, the transition area408, or other areas of the facility402may be similarly equipped. Furthermore, the arrangement of the various areas within the facility402is depicted functionally rather than schematically. For example, in some implementations, multiple different receiving areas404, storage areas406, and transition areas408may be interspersed rather than segregated in the facility402.

The facility402may include, or be coupled to, an inventory management system130. The inventory management system130is configured to interact with users102or devices such as sensors110, robots, material handling equipment, computing devices, and so forth, in one or more of the receiving area404, the storage area406, or the transition area408.

During operation of the facility402, the sensors110may be configured to provide sensor data, or information based on the sensor data, to the inventory management system130. The sensor data may include image data122, non-image data, weight sensor data obtained from weight sensors110(6), and so forth. The sensors110are described in more detail below with regard toFIG. 5.

The inventory management system130or other systems may use the sensor data to track the location of objects within the facility402, movement of the objects, or provide other functionality. Objects may include, but are not limited to, items204, users102, totes212, and so forth. For example, a series of images acquired by the camera110(1) may indicate removal by the user102of an item204from a particular location on the inventory location202and placement of the item204on or at least partially within the tote212.

The facility402may be configured to receive different kinds of items204from various suppliers and to store them until a customer orders or retrieves one or more of the items204. A general flow of items204through the facility402is indicated by the arrows ofFIG. 4. Specifically, as illustrated in this example, items204may be received from one or more suppliers, such as manufacturers, distributors, wholesalers, and so forth, at the receiving area404. In various implementations, the items204may include merchandise, commodities, perishables, or any suitable type of item204, depending on the nature of the enterprise that operates the facility402.

Upon being received from a supplier at the receiving area404, the items204may be prepared for storage in the storage area406. For example, in some implementations, items204may be unpacked or otherwise rearranged. The inventory management system130may include one or more software applications executing on a computer system to provide inventory management functions. These inventory management functions may include maintaining information indicative of the type, quantity, condition, cost, location, weight, or any other suitable parameters with respect to the items204. The items204may be stocked, managed, or dispensed in terms of countable units, individual units, or multiple units, such as packages, cartons, crates, pallets, or other suitable aggregations. Alternatively, some items204, such as bulk products, commodities, and so forth, may be stored in continuous or arbitrarily divisible amounts that may not be inherently organized into countable units. Such items204may be managed in terms of a measurable quantity such as units of length, area, volume, weight, time, duration, or other dimensional properties characterized by units of measurement. Generally speaking, a quantity of an item204may refer to either a countable number of individual or aggregate units of an item204or a measurable amount of an item204, as appropriate.

After arriving through the receiving area404, items204may be stored within the storage area406. In some implementations, like items204may be stored or displayed together in the inventory locations202such as in bins, on shelves, hanging from pegboards, and so forth. In this implementation, all items204of a given kind are stored in one inventory location202. In other implementations, like items204may be stored in different inventory locations202. For example, to optimize retrieval of certain items204having frequent turnover within a large physical facility402, those items204may be stored in several different inventory locations202to reduce congestion that might occur at a single inventory location202.

When a customer order specifying one or more items204is received, or as a user102progresses through the facility402, the corresponding items204may be selected or “picked” from the inventory locations202containing those items204. In various implementations, item picking may range from manual to completely automated picking. For example, in one implementation, a user102may have a list of items204they desire and may progress through the facility402picking items204from inventory locations202within the storage area406and placing those items204into a tote212. In other implementations, employees of the facility402may pick items204using written or electronic pick lists derived from customer orders. These picked items204may be placed into the tote212as the employee progresses through the facility402.

After items204have been picked, the items204may be processed at a transition area408. The transition area408may be any designated area within the facility402where items204are transitioned from one location to another or from one entity to another. For example, the transition area408may be a packing station within the facility402. When the item204arrives at the transition area408, the items204may be transitioned from the storage area406to the packing station. Information about the transition may be maintained by the inventory management system130.

In another example, if the items204are departing the facility402, a list of the items204may be obtained and used by the inventory management system130to transition responsibility for, or custody of, the items204from the facility402to another entity. For example, a carrier may accept the items204for transport with that carrier accepting responsibility for the items204indicated in the list. In another example, a user102may purchase or rent the items204and remove the items204from the facility402. During use of the facility402, the user102may move about the facility402to perform various tasks, such as picking or placing the items204in the inventory locations202.

To facilitate operation of the facility402, the inventory management system130is configured to use the sensor data including the image data122and other information such as the item data, the physical layout data306, the presence data128, and so forth, to generate interaction data410. For example, the presence data128may be used to associate a particular user identity with a particular person that is tracked using image data122acquired by one or more of the cameras110(1) in the facility402.

The interaction data410may provide information about an interaction, such as a pick of an item204from the inventory location202, a place of an item204to the inventory location202, a touch made to an item204at the inventory location202, a gesture associated with an item204at the inventory location202, and so forth. The interaction data410may include one or more of the type of interaction, interaction location identifier indicative of where from the inventory location202the interaction took place, item identifier, quantity change to the item204, user identifier, and so forth. The interaction data410may then be used to further update the item data. For example, the quantity of items204on hand at a particular lane on the shelf may be changed based on an interaction that picks or places one or more items204.

The inventory management system130may combine or otherwise utilize data from different sensors110of different types to generate the interaction data410. For example, weight data obtained from weight sensors110(6) at the inventory location202may be used instead of, or in conjunction with, the image data122to determine the interaction data410.

FIG. 5is a block diagram500illustrating additional details of the facility402, according to some implementations. The facility402may be connected to one or more networks502, which in turn connect to one or more servers120. The network502may include private networks such as an institutional or personal intranet, public networks such as the Internet, or a combination thereof. The network502may utilize wired technologies (e.g., wires, fiber optic cables, and so forth), wireless technologies (e.g., radio frequency, infrared, acoustic, optical, and so forth), or other connection technologies. The network502is representative of any type of communication network, including one or more of data networks or voice networks. The network502may be implemented using wired infrastructure (e.g., copper cable, fiber optic cable, and so forth), a wireless infrastructure (e.g., cellular, microwave, satellite, and so forth), or other connection technologies.

The servers120may be configured to execute one or more modules or software applications associated with the inventory management system130or other systems. While the servers120are illustrated as being in a location outside of the facility402, in other implementations, at least a portion of the servers120may be located at the facility402. The servers120are discussed in more detail below with regard toFIG. 6.

The users102, the totes212, or other objects in the facility402may be equipped with one or more tags504. The tags504may be configured to emit a signal. In one implementation, the tag504may be a radio frequency identification (RFID) tag504configured to emit a RF signal upon activation by an external signal. For example, the external signal may comprise a radio frequency signal or a magnetic field configured to energize or activate the RFID tag504. In another implementation, the tag504may comprise a transmitter and a power source configured to power the transmitter. For example, the tag504may comprise a Bluetooth Low Energy (BLE) transmitter and battery. In other implementations, the tag504may use other techniques to indicate presence of the tag504. For example, an acoustic tag504may be configured to generate an ultrasonic signal, which is detected by corresponding acoustic receivers. In yet another implementation, the tag504may be configured to emit an optical signal.

The inventory management system130may be configured to use the tags504for one or more of identification of the object, determining a location of the object, and so forth. For example, the users102may wear tags504, the totes212may have tags504affixed, and so forth, which may be read and, based at least in part on signal strength, used to determine identity and location.

Generally, the inventory management system130or other systems associated with the facility402may include any number and combination of input components, output components, and servers120.

The one or more sensors110may be arranged at one or more locations within the facility402. For example, the sensors110may be mounted on or within a floor, wall, at a ceiling, at an inventory location202, on a tote212, may be carried or worn by a user102, and so forth.

The sensors110may include one or more cameras110(1) or other imaging sensors. The one or more cameras110(1) may include imaging sensors configured to acquire images of a scene. The cameras110(1) are configured to detect light in one or more wavelengths including, but not limited to, terahertz, infrared, visible, ultraviolet, and so forth. The cameras110(1) may comprise charge coupled devices (CCD), complementary metal oxide semiconductor (CMOS) devices, microbolometers, and so forth. The inventory management system130may use image data122acquired by the cameras110(1) during operation of the facility402. For example, the inventory management system130may identify items204, users102, totes212, and so forth, based at least in part on their appearance within the image data122acquired by the cameras110(1). The cameras110(1) may be mounted in various locations within the facility402. For example, cameras110(1) may be mounted overhead, on inventory locations202, may be worn or carried by users102, may be affixed to totes212, and so forth.

One or more depth sensors110(2) may also be included in the sensors110. The depth sensors110(2) are configured to acquire spatial or three-dimensional (3D) data, such as depth information, about objects within a field-of-view. The depth sensors110(2) may include range cameras, lidar systems, sonar systems, radar systems, structured light systems, stereo vision systems, optical interferometry systems, and so forth. The inventory management system130may use the 3D data acquired by the depth sensors110(2) to identify objects, determine a location of an object in 3D real space, and so forth.

One or more buttons110(3) may be configured to accept input from the user102. The buttons110(3) may comprise mechanical, capacitive, optical, or other mechanisms. For example, the buttons110(3) may comprise mechanical switches configured to accept an applied force from a touch of the user102to generate an input signal. The inventory management system130may use data from the buttons110(3) to receive information from the user102. For example, the tote212may be configured with a button110(3) to accept input from the user102and send information indicative of the input to the inventory management system130.

The sensors110may include one or more touch sensors110(4). The touch sensors110(4) may use resistive, capacitive, surface capacitance, projected capacitance, mutual capacitance, optical, Interpolating Force-Sensitive Resistance (IFSR), or other mechanisms to determine the position of a touch or near-touch. For example, the IFSR may comprise a material configured to change electrical resistance responsive to an applied force. The location within the material of that change in electrical resistance may indicate the position of the touch. The inventory management system130may use data from the touch sensors110(4) to receive information from the user102. For example, the touch sensor110(4) may be integrated with the tote212to provide a touchscreen with which the user102may select from a menu one or more particular items204for picking, enter a manual count of items204at an inventory location202, and so forth.

One or more microphones110(5) or other acoustic transducers may be configured to acquire information indicative of sound present in the environment. In some implementations, arrays of microphones110(5) may be used. These arrays may implement beamforming techniques to provide for directionality of gain. The analysis module124may use information obtained by a microphone110(5) or an array of microphones110(5) to gather portal data118. For example, the mobile device104may emit an ultrasonic signal while the microphones110(5) positioned at the portal106(or elsewhere in the facility402) generate portal data118about the ultrasonic signal. The inventory management system130may use the one or more microphones110(5) to acquire information from acoustic tags504, accept voice input from the users102, determine ambient noise level, and so forth.

One or more weight sensors110(6) are configured to measure the weight of a load, such as the item204, the tote212, or other objects. The weight sensors110(6) may be configured to measure the weight of the load at one or more of the inventory locations202, the tote212, on the floor of the facility402, and so forth. For example, the shelf may include a plurality of lanes or platforms, with one or more weight sensors110(6) beneath each one to provide weight sensor data about an individual lane or platform. The weight sensors110(6) may include one or more sensing mechanisms to determine the weight of a load. These sensing mechanisms may include piezoresistive devices, piezoelectric devices, capacitive devices, electromagnetic devices, optical devices, potentiometric devices, microelectromechanical devices, and so forth. The sensing mechanisms of weight sensors110(6) may operate as transducers that generate one or more signals based on an applied force, such as that of the load due to gravity. For example, the weight sensor110(6) may comprise a load cell having a strain gauge and a structural member that deforms slightly when weight is applied. By measuring a change in the electrical characteristic of the strain gauge, such as capacitance or resistance, the weight may be determined. In another example, the weight sensor110(6) may comprise a force sensing resistor (FSR). The FSR may comprise a resilient material that changes one or more electrical characteristics when compressed. For example, the electrical resistance of a particular portion of the FSR may decrease as the particular portion is compressed. The inventory management system130may use the data acquired by the weight sensors110(6) to identify an object, determine a change in the quantity of objects, determine a location of an object, maintain shipping records, and so forth.

The sensors110may include one or more optical sensors110(7). The optical sensors110(7) may be configured to provide data indicative of one or more of color or intensity of light impinging thereupon. For example, the optical sensor110(7) may comprise a photodiode and associated circuitry configured to generate a signal or data indicative of an incident flux of photons. As described below, the optical sensor array110(13) may comprise a plurality of the optical sensors110(7). For example, the optical sensor110(7) may comprise an array of ambient light sensors such as the ISL76683 as provided by Intersil Corporation of Milpitas, Calif., USA, or the MAX44009 as provided by Maxim Integrated of San Jose, Calif., USA. In other implementations, other optical sensors110(7) may be used. The optical sensors110(7) may be sensitive to one or more of infrared light, visible light, or ultraviolet light. The optical sensors110(7) may include photodiodes, photoresistors, photovoltaic cells, quantum dot photoconductors, bolometers, pyroelectric infrared detectors, and so forth. For example, the optical sensor110(7) may use germanium photodiodes to detect infrared light.

One or more radio frequency identification (RFID) readers110(8), near field communication (NFC) systems, and so forth, may be included as sensors110. For example, the RFID readers110(8) may be configured to read the RF tags504. Information acquired by the RFID reader110(8) may be used by the inventory management system130to identify an object associated with the RF tag504such as the item204, the user102, the tote212, and so forth. For example, based on information from the RFID readers110(8) detecting the RF tag504at different times and RFID readers110(8) having different locations in the facility402, a velocity of the RF tag504may be determined.

One or more RF receivers110(9) may also be included as sensors110. In some implementations, the RF receivers110(9) may be part of transceiver assemblies. The RF receivers110(9) may be configured to acquire RF signals associated with Wi-Fi, Bluetooth, ZigBee, 5G, 4G, 3G, LTE, or other wireless data transmission technologies. The RF receivers110(9) may provide information associated with data transmitted via radio frequencies, signal strength of RF signals, and so forth. For example, information from the RF receivers110(9) may be used by the inventory management system130to determine a location of an RF source, such as a communication interface onboard the tote212.

The sensors110may include one or more accelerometers110(10), which may be worn or carried by the user102, mounted to the tote212, and so forth. The accelerometers110(10) may provide information such as the direction and magnitude of an imposed acceleration. Data such as rate of acceleration, determination of changes in direction, speed, and so forth, may be determined using the accelerometers110(10).

A gyroscope110(11) may provide information indicative of rotation of an object affixed thereto. For example, the tote212or other objects may be equipped with a gyroscope110(11) to provide data indicative of a change in orientation of the object.

A magnetometer110(12) may be used to determine an orientation by measuring ambient magnetic fields, such as the terrestrial magnetic field. The magnetometer110(12) may be worn or carried by the user102, mounted to the tote212, and so forth. For example, the magnetometer110(12) mounted to the tote212may act as a compass and provide information indicative of which direction the tote212is oriented.

An optical sensor array110(13) may comprise one or more optical sensors110(7). The optical sensors110(7) may be arranged in a regular, repeating, or periodic two-dimensional arrangement such as a grid. The optical sensor array110(13) may generate image data122. For example, the optical sensor array110(13) may be arranged within or below an inventory location202and obtain information about shadows of items204, hand of the user102, and so forth.

The sensors110may include proximity sensors110(14) used to determine presence of an object, such as the user102, the tote212, and so forth. The proximity sensors110(14) may use optical, electrical, ultrasonic, electromagnetic, or other techniques to determine a presence of an object. In some implementations, the proximity sensors110(14) may use an optical emitter and an optical detector to determine proximity. For example, an optical emitter may emit light, a portion of which may then be reflected by the object back to the optical detector to provide an indication that the object is proximate to the proximity sensor110(14). In other implementations, the proximity sensors110(14) may comprise a capacitive proximity sensor110(14) configured to provide an electrical field and determine a change in electrical capacitance due to presence or absence of an object within the electrical field.

The proximity sensors110(14) may be configured to provide sensor data indicative of one or more of a presence or absence of an object, a distance to the object, or characteristics of the object. An optical proximity sensor110(14) may use time-of-flight (ToF), structured light, interferometry, or other techniques to generate the distance data. For example, ToF determines a propagation time (or “round-trip” time) of a pulse of emitted light from an optical emitter or illuminator that is reflected or otherwise returned to an optical detector. By dividing the propagation time in half and multiplying the result by the speed of light in air, the distance to an object may be determined. In another implementation, a structured light pattern may be provided by the optical emitter. A portion of the structured light pattern may then be detected on the object using a sensor110such as a camera110(1). Based on an apparent distance between the features of the structured light pattern, the distance to the object may be calculated. Other techniques may also be used to determine distance to the object. In another example, the color of the reflected light may be used to characterize the object, such as skin, clothing, tote212, and so forth.

The sensors110may include a light curtain110(15) that utilizes a linear array of light emitters and a corresponding linear array of light detectors. For example, the light emitters may comprise a line of infrared light emitting diodes (LEDs) or vertical cavity surface emitting lasers (VCSELs) that are arranged above a top shelf in front of the inventory location202, while the light detectors comprise a line of photodiodes sensitive to infrared light arranged below the light emitters. The light emitters produce a “lightplane” or sheet of infrared light that is then detected by the light detectors. An object passing through the lightplane may decrease the amount of light falling upon the light detectors. For example, the user's102hand would prevent at least some of the light from light emitters from reaching a corresponding light detector. As a result, a position along the linear array of the object may be determined that is indicative of a touchpoint. This position may be expressed as touchpoint data, with the touchpoint being indicative of the intersection between the hand of the user102and the sheet of infrared light. In some implementations, a pair of light curtains110(15) may be arranged at right angles relative to one another to provide two-dimensional touchpoint data indicative of a position of touch in a plane. Input from the light curtain110(15), such as indicating occlusion from a hand of a user102may be used to trigger acquisition or selection of image data122for processing by the inventory management system130.

The sensors110may include an instrumented auto-facing unit (IAFU)110(16). The IAFU110(16) may comprise a position sensor or encoder configured to provide data indicative of displacement of a pusher. As an item204is removed from the IAFU110(16), the pusher moves, such as under the influence of a spring, and pushes the remaining items204in the IAFU110(16) to the front of the inventory location202. By using data from the position sensor, and given item data such as a depth of an individual item204, a count may be determined, based on a change in position data. For example, if each item204is 1 inch deep, and the position data indicates a change of 5 inches, the quantity held by the IAFU110(16) may have changed by 5 items204. This count information may be used to confirm or provide a cross check for a count obtained by other means, such as analysis of the image data122.

A location sensor110(17) may be configured to provide information such as geographic coordinates, speed, heading, and so forth. The location sensor110(17) may comprise a radio navigation-based system, such as a terrestrial or satellite-based navigational system. Satellite-based navigational systems may include a GPS receiver, a Global Navigation Satellite System (GLONASS) receiver, a Galileo receiver, a BeiDou Navigation Satellite System (BDS) receiver, an Indian Regional Navigational Satellite System, and so forth.

The sensors110may include other sensors110(S) as well. For example, the other sensors110(S) may include ultrasonic rangefinders, thermometers, barometric sensors, hygrometers, and so forth. For example, the inventory management system130may use information acquired from thermometers and hygrometers in the facility402to direct the user102to check on delicate items204stored in a particular inventory location202, which is overheating, too dry, too damp, and so forth.

In some implementations, the camera110(1) or other sensors110(S) may include hardware processors, memory, and other elements configured to perform various functions. For example, the cameras110(1) may be configured to generate image data122, send the image data122to another device such as the server120, and so forth.

The facility402may include one or more access points506configured to establish one or more wireless networks. The access points506may use Wi-Fi, NFC, Bluetooth, or other technologies to establish wireless communications between a device and the network502. The wireless networks allow the devices to communicate with one or more of the sensors110, the inventory management system130, the optical sensor arrays110(13), the tag504, a communication device of the tote212, or other devices.

Output devices508may also be provided in the facility402. The output devices508are configured to generate signals, which may be perceived by the user102or detected by the sensors110. In some implementations, the output devices508may be used to provide illumination of the optical sensor array110(13), light curtain110(15), and so forth.

Haptic output devices508(1) are configured to provide a signal that results in a tactile sensation to the user102. The haptic output devices508(1) may use one or more mechanisms such as electrical stimulation or mechanical displacement to provide the signal. For example, the haptic output devices508(1) may be configured to generate a modulated electrical signal, which produces an apparent tactile sensation in one or more fingers of the user102. In another example, the haptic output devices508(1) may comprise piezoelectric or rotary motor devices configured to provide a vibration, which may be felt by the user102.

One or more audio output devices508(2) may be configured to provide acoustic output. The acoustic output includes one or more of infrasonic sound, audible sound, or ultrasonic sound. The audio output devices508(2) may use one or more mechanisms to generate the acoustic output. These mechanisms may include, but are not limited to, the following: voice coils, piezoelectric elements, magnetorestrictive elements, electrostatic elements, and so forth. For example, a piezoelectric buzzer or a speaker may be used to provide acoustic output. In another example, a location of the mobile device104in the facility402may be determined based on device data112indicative of ultrasonic sound emitted by audio output devices508(2) positioned within the facility402.

The display devices508(3) may be configured to provide output, which may be seen by the user102or detected by a light-sensitive sensor such as a camera110(1) or an optical sensor110(7). In some implementations, the display devices508(3) may be configured to produce output in one or more of infrared, visible, or ultraviolet light. The output may be monochrome or in color. The display devices508(3) may be one or more of emissive, reflective, microelectromechanical, and so forth. An emissive display device508(3), such as using LEDs, is configured to emit light during operation. In comparison, a reflective display device508(3), such as using an electrophoretic element, relies on ambient light to present an image. Backlights or front lights may be used to illuminate non-emissive display devices508(3) to provide visibility of the output in conditions where the ambient light levels are low.

The display devices508(3) may be located at various points within the facility402. For example, the addressable displays may be located on inventory locations202, totes212, on the floor of the facility402, and so forth.

Other output devices508(P) may also be present. For example, the other output devices508(P) may include scent/odor dispensers, document printers, 3D printers or fabrication equipment, and so forth.

FIG. 6illustrates a block diagram600of a server120configured to support operation of the facility402, according to some implementations. The server120may be physically present at the facility402, may be accessible by the network502, or a combination of both. The server120does not require end-user knowledge of the physical location and configuration of the system that delivers the services. Common expressions associated with the server120may include “on-demand computing”, “software as a service (SaaS)”, “platform computing”, “network-accessible platform”, “cloud services”, “data centers”, and so forth. Services provided by the server120may be distributed across one or more physical or virtual devices.

One or more power supplies602may be configured to provide electrical power suitable for operating the components in the server120. The one or more power supplies602may comprise batteries, capacitors, fuel cells, photovoltaic cells, wireless power receivers, conductive couplings suitable for attachment to an external power source such as provided by an electric utility, and so forth. The server120may include one or more hardware processors604(processors) configured to execute one or more stored instructions. The processors604may comprise one or more cores. One or more clocks606may provide information indicative of date, time, ticks, and so forth. For example, the processor604may use data from the clock606to associate a particular interaction with a particular point in time.

The server120may include one or more communication interfaces608such as input/output (I/O) interfaces610, network interfaces612, and so forth. The communication interfaces608enable the server120, or components thereof, to communicate with other devices or components. The communication interfaces608may include one or more I/O interfaces610. The I/O interfaces610may comprise Inter-Integrated Circuit (I2C), Serial Peripheral Interface bus (SPI), Universal Serial Bus (USB) as promulgated by the USB Implementers Forum, RS-232, and so forth.

The I/O interface(s)610may couple to one or more I/O devices614. The I/O devices614may include input devices such as one or more of a sensor110, keyboard, mouse, scanner, and so forth. The I/O devices614may also include output devices508such as one or more of a display device508(3), printer, audio speakers, and so forth. In some embodiments, the I/O devices614may be physically incorporated with the server120or may be externally placed.

The network interfaces612may be configured to provide communications between the server120and other devices, such as the totes212, routers, access points506, and so forth. The network interfaces612may include devices configured to couple to personal area networks (PANs), local area networks (LANs), wireless local area networks (WLANS), wide area networks (WANs), and so forth. For example, the network interfaces612may include devices compatible with Ethernet, Wi-Fi, Bluetooth, ZigBee, and so forth.

The server120may also include one or more busses or other internal communications hardware or software that allow for the transfer of data between the various modules and components of the server120.

As shown inFIG. 6, the server120includes one or more memories616. The memory616may comprise one or more non-transitory computer-readable storage media (CRSM). The CRSM may be any one or more of an electronic storage medium, a magnetic storage medium, an optical storage medium, a quantum storage medium, a mechanical computer storage medium, and so forth. The memory616provides storage of computer-readable instructions, data structures, program modules, and other data for the operation of the server120. A few example functional modules are shown stored in the memory616, although the same functionality may alternatively be implemented in hardware, firmware, or as a system on a chip (SoC).

The memory616may include at least one operating system (OS) module618. The OS module618is configured to manage hardware resource devices such as the I/O interfaces610, the I/O devices614, the communication interfaces608, and provide various services to applications or modules executing on the processors604. The OS module618may implement a variant of the FreeBSD operating system as promulgated by the FreeBSD Project; other UNIX or UNIX-like variants; a variation of the Linux operating system as promulgated by Linus Torvalds; the Windows operating system from Microsoft Corporation of Redmond, Wash., USA; and so forth.

Also stored in the memory616may be a data store620and one or more of the following modules. These modules may be executed as foreground applications, background tasks, daemons, and so forth. The data store620may use a flat file, database, linked list, tree, executable code, script, or other data structure to store information. In some implementations, the data store620or a portion of the data store620may be distributed across one or more other devices including the servers120, network attached storage devices, and so forth.

A communication module622may be configured to establish communications with one or more of the totes212, sensors110, display devices508(3), other servers120, or other devices. The communications may be authenticated, encrypted, and so forth.

The memory616may store the analysis module124. The functions of the analysis module124are described above with regard toFIG. 3. The analysis module124may store information described above with regard toFIG. 3at least in part in the data store620.

The memory616may also store an inventory management module624. The inventory management module624is configured to provide the inventory functions as described herein with regard to the inventory management system130. For example, the inventory management module624may track items204between different inventory locations202, to and from the totes212, and so forth. The inventory management module624may include a data acquisition module626, object tracking module628and accounting module630.

The analysis module124and the inventory management module624may access sensor data632. The sensor data632may be stored at least in part in the data store620. The sensor data632comprises information acquired by one or more of the sensors110. For example, the sensor data632may include one or more of device data112, portal data118, image data122, or data acquired by other sensors110.

The data acquisition module626may be configured to acquire and access information associated with operation of the facility402. For example, the data acquisition module626may be configured to acquire sensor data632, such as the image data122, from one or more of the sensors110.

Processing of sensor data632, such as the image data122, may be performed by a module implementing, at least in part, one or more of the following tools or techniques. In one implementation, processing of the image data122may be performed, at least in part, using one or more tools available in the OpenCV library as developed by Intel Corporation of Santa Clara, Calif., USA; Willow Garage of Menlo Park, Calif., USA; and Itseez of Nizhny Novgorod, Russia, with information available at www.opencv.org. In another implementation, functions available in the OKAO machine vision library as promulgated by Omron Corporation of Kyoto, Japan, may be used to process the sensor data632. In still another implementation, functions such as those in the Machine Vision Toolbox for Matlab (MVTB) available using MATLAB as developed by MathWorks, Inc. of Natick, Mass., USA, may be utilized.

Techniques such as artificial neural networks (ANNs), active appearance models (AAMs), active shape models (ASMs), principal component analysis (PCA), cascade classifiers, and so forth, may also be used to process the sensor data632or other data. For example, the ANN may be a trained using a supervised learning algorithm such that object identifiers are associated with images of particular objects within training images provided to the ANN. Once trained, the ANN may be provided with the sensor data632such as the image data122to generate identification data of an object.

The object tracking module628may be used to track one or more of the users102, the items204, and so forth. The object tracking module628may use image data122acquired from cameras110(1) throughout the facility402to determine a location of the user102at a particular time. As described above, the presence data128may be used to determine presence of the user102at a particular portal106. The object tracking module628may use the image data122or other sensor data632to track movement of the items204. For example, item data634comprising previously trained ANNs, dimensional information, barcodes, text, or other information about the items204may be accessed and used to identify and track movement of the items item204at the facility402.

The accounting module630may be configured to assess charges to accounts associated with particular users102or other entities. For example, the interaction data410may indicate that the user102has removed a particular item204from an inventory location202. Based on the interaction data410, the accounting module630may assess the charge to a payment instrument associated with the account.

Other modules636may also be present in the memory616as well as other data638in the data store620.

FIG. 7illustrates a block diagram700of a mobile device104configured to participate with operation of the portal106, according to some implementations.

Similar to those described above, the mobile device104may include or more power supplies702. The mobile device104may include one or more hardware processors704(processors) configured to execute one or more stored instructions. The processors704may comprise one or more cores. One or more clocks706may provide information indicative of date, time, ticks, and so forth. For example, the processor704may use data from the clock706to associate a particular interaction with a particular point in time.

The mobile device104may include one or more communication interfaces708such as I/O interfaces710, network interfaces712, and so forth. The communication interfaces708enable the mobile device104, or components thereof, to communicate with other devices or components. The communication interfaces708may include one or more I/O interfaces710. The I/O interfaces710may comprise I2C, SPI, USB, RS-232, and so forth.

The I/O interface(s)710may couple to one or more I/O devices714. The I/O devices714may include input devices such as one or more of a sensor110, keyboard, mouse, scanner, and so forth. The I/O devices714may also include output devices508such as one or more of a display device508(3), printer, audio speakers, and so forth. In some embodiments, the I/O devices714may be physically incorporated with the mobile device104or may be externally placed.

The network interfaces712may be configured to provide communications between the mobile device104, the portal106, the totes212, routers, access points506, and so forth. The network interfaces712may include devices configured to couple to PANs, LANs, WLANS, WANs, and so forth. For example, the network interfaces712may include devices compatible with Ethernet, Wi-Fi, Bluetooth, ZigBee, and so forth. A wireless network interface712may include one or more radios108.

The mobile device104may also include one or more busses or other internal communications hardware or software that allow for the transfer of data between the various modules and components of the mobile device104.

As shown inFIG. 7, the mobile device104includes one or more memories716. The memory716may comprise one or more non-transitory CRSM. The memory716provides storage of computer-readable instructions, data structures, program modules, and other data for the operation of the mobile device104. A few example functional modules are shown stored in the memory716, although the same functionality may alternatively be implemented in hardware, firmware, or as a SoC.

The memory716may include at least one OS module718. The OS module718is configured to manage hardware resource devices such as the I/O interfaces710, the I/O devices714, the communication interfaces708, and provide various services to applications or modules executing on the processors704. The OS module718may implement a variant of the FreeBSD operating system as promulgated by the FreeBSD Project; other UNIX or UNIX-like variants; a variation of the Linux operating system as promulgated by Linus Torvalds; the Windows operating system from Microsoft Corporation of Redmond, Wash., USA; and so forth.

Also stored in the memory716may be a data store720and one or more of the following modules. These modules may be executed as foreground applications, background tasks, daemons, and so forth. The data store720may use a flat file, database, linked list, tree, executable code, script, or other data structure to store information. In some implementations, the data store720or a portion of the data store720may be distributed across one or more other devices including the servers120, network attached storage devices, and so forth.

A communication module722may be configured to establish communications with one or more of the totes212, sensors110, display devices508(3), other servers120, or other devices. The communications may be authenticated, encrypted, and so forth.

The memory716may store a device data acquisition module724. The device data acquisition module724may be configured to generate device data112. The device data acquisition module724may access sensor data726provided by the sensors110on board the mobile device104, or associated with the mobile device104. For example, device data acquisition module724may access accelerometer data from a fitness tracking device in communication with the mobile device104and compass data acquired by the magnetometer110(12) on the mobile device104.

The device data acquisition module724may be configured to send one or more of the device data112or the device identifier114. For example, the device data acquisition module724may receive data indicating a portal signal has been received by the radio108of the mobile device104. The device data acquisition module724may be configured to send the device data112stored in the data store720to the communication module722for transmission by the radio108. The packet transmitted by the mobile device104may include a header that indicates the device identifier114while the payload comprises the device data112. This transmission may be the device signal that is received by the radio108of the portal106. In some implementations, the transmission may be made as a broadcast. For example, the device data112may be sent as a Bluetooth broadcast.

In some implementations, the device data acquisition module724may send the device data112to the server120. For example, the device data acquisition module724may establish a connection with the server120and send the device data112using the connection.

The device data acquisition module724may send the device data112responsive to input of one or more other sensors110, data received by way of a communication interface708, and so forth. In one implementation, the location sensor110(17) may provide location information that indicates the mobile device104has traversed a predefined geographic location. For example, location data provided by the location sensor110(17) may indicate that the mobile device104is entered the facility402. Responsive to this determination, device data112may be transmitted to the server120.

Other modules728may also be present in the memory716as well as other data730in the data store720. For example, the other modules728may include an application to present a pick list to the user102for picking at the facility402.

Illustrative Processes

FIG. 8depicts a flow diagram of a process800of a portal106interacting with a mobile device104, according to some implementations. In some implementations, the process800may be implemented at least in part by one or more of the mobile device104, the portal106, or the server120. In this illustration, operations performed by the portal106are depicted to the left of the page, while operations performed by the mobile device104are depicted to the right of the page.

At802, a radio108of the portal106transmits a portal signal. For example, the transmitter of the radio108may generate the portal signal which is then emitted from one or more of the antennas116. The portal signal may be used to transfer information to a mobile device104that is within range. For example, the portal signal may be used to transmit a digital certificate or other cryptographic credentials that may be used to authenticate or identify the portal106. In some implementations, the portal signal may be transmitted as part of a device pairing process, a connection, a connectionless broadcast, and so forth. In other implementations, other devices may generate the portal signal. For example, a magnet may generate a magnetic field, an ultrasonic transducer may emit ultrasonic sound, an infrared emitter may generate an infrared signal, and so forth. In other implementations, the portal106may not transmit a portal signal802.

The portal signal802may include beacons or other signals generated by devices associated with the facility402but separate from the portal106. For example, the facility402may have devices to generate BLE beacons or Wi-Fi broadcasts that are transmitted within the facility402, outside of an entrance to the portal106, and so forth.

At804, the device data acquisition module724of the mobile device104acquires device data112based on information from one or more of the sensors110associated with the mobile device104. For example, the device data acquisition module724may acquire one or more of accelerometer data from the accelerometer110(10), rotational data from gyroscope110(11), or magnetometer data from the magnetometer110(12) on board the mobile device104. In some implementations, acquisition of the device data112may be responsive to the receipt of the portal signal described next. In other implementations, acquisition of the device data112may be responsive to traversing a particular geographic location, such as entering a geo-fenced area.

At806, the mobile device104receives the portal signal. For example, a receiver of the radio108in the mobile device104may be used to receive the portal signal.

At808, the mobile device104sends a device signal. The sending of the device signal may be responsive to receipt of the portal signal. In one implementation, the device signal may comprise the device data112and the device identifier114. In another implementation, the device signal may comprise the device identifier114. In some implementations, the device data112may be sent to the server120using a connection between the mobile device104and the server120. For example, the mobile device104may establish a connection with the server120that is independent of the systems at the portal106.

In other implementations, the device signal may be sent by the mobile device104based on one or more other factors. As described above, the device signal may be sent in response to receipt of a portal signal. In another example, the mobile device104may send the device signal periodically. In yet another example, the mobile device104may send the device signal after determining a location that is within or past a specified geolocation, such as a geofence boundary.

At810, the portal106receives the device signal from the mobile device104using a receiver of the radio108at the portal106. As described above, the device signal may include one or more of the device data112or the device identifier114.

At812, the portal106generates the portal data118. The portal data118may be indicative of one or more of received signal strength of the device signal or information indicative of the specified volume covered by the antenna116. For example, this may comprise coordinates associated with the volume, an angular bearing and distance, and so forth. As described above, an antenna116may particular gain pattern that is designed to cover specified three-dimensional volume within the portal106. In some implementations, the information indicative of the specified volume may comprise an antenna identifier indicative of the antenna116that was used to acquire the device signal. The antenna identifier may then be used to retrieve information such as coordinates describing the specified volume, specifications that describe the gain, and so forth.

At814, one or more of the device data112, the device identifier114, or the portal data118may be sent to a server120. For example, the network interface of the portal106may be used to transfer this information to the server120using the network502.

FIG. 9depicts a flow diagram of a process900of a server120generating data indicative of presence of a particular user102at a portal106, according to some implementations. In the implementation depicted here, the operations are performed at least in part by the server120. In some implementations, the process may be implemented at least in part by one or more of the mobile device104, the portal106, or the server120.

At902, one or more of the device data112, device identifier114, or portal data118are accessed. For example, this information may be received from one or more of the portal106or the mobile device104as part of the process of sending the portal signal and receiving a device signal. The antennas116of the portal106may cover a specified volume of the portal106.

In some implementations, at904, image data122may be accessed. For example, the camera110(1) having a field-of-view that encompasses at least a portion of the portal106may generate and send the image data122to the server120. The portion within the field-of-view may include at least a part of the specified volume. The image data122may include an image of the user102in the portal106.

At906, an estimated motion value304of the mobile device104based on the portal data118is determined. For example, the estimated motion value304may comprise a vector or trajectory based on the information obtained by the portal106about the device signal. In one implementation, the estimated motion value304comprises one or more vector values, where individual ones of the vector values are indicative of a direction of motion and a speed during a particular interval of time of the mobile device104.

In some implementations, the antennas116that acquire the portal data118may be directional such that they provide gain in a particular direction. The portal data118may be obtained from a first directional antenna116(1) and a second directional antenna116(2) at a first time and a second time. In some implementations, the portal data118may include information indicative of the particular antenna116that was used to receive the device signal or other signals from the mobile device104. Antenna coverage data may be accessed that indicates the specified volume in the facility402covered by the first directional antenna116(1) and the second directional antenna116(2). For example, antenna identifier data indicative of the first directional antenna116(1) may be associated with the left side of portal106while the antenna identifier indicative of the second directional antenna116(2) is associated with the right side of portal106.

A first estimated RF location may be determined based on the antenna coverage data and the received signal strength of the device signal as measured at the first directional antenna116(1) and the second directional antenna116(2) at the first time. A second estimated RF location may be determined based on the antenna coverage data and the received signal strength of the device signal as measured at the first directional antenna116(1) and the second directional antenna116(2) at the second time. The estimated motion value304may be determined based on the first estimated RF location at the first time and the second estimated RF location at the second time.

In one implementation, the portal data118may include a first received signal strength of a first device signal as received using a first directional antenna116(1) at a first time. The portal data118may also include a second received signal strength of a second device signal as received using the first directional antenna116(1) at a second time. A first estimated distance between the mobile device104and the first directional antenna116(1) may be determined using the first received signal strength. For example, a particular received signal strength may be associated with a particular linear distance. A second estimated distance may be determined between the mobile device104and the first directional antenna116(1) using the second received signal strength. Antenna coverage data may be accessed that is indicative of the specified volume in the facility402that is covered by the first directional antenna116(1). The antenna coverage data may also include the location and orientation of the antenna116in the facility402. For example, the antenna coverage data may indicate that the first directional antenna116(1) is located on a right side of the portal106and exhibits primary gain along a right side or lane of the portal106. A first estimated location of the mobile device104may be determined at the first time based on the antenna coverage data and the first estimated distance. For example, given the known gain pattern of the antenna116and the estimated distance, a location within the facility402may be determined. A second estimated location of the mobile device104at the second time may also be determined based on the antenna coverage data and the second estimated distance. A direction of motion of the mobile device104may be determined based on a comparison between the first estimated location and the second estimated location. A movement distance may be determined that indicates a distance between the first estimated location of the mobile device104and the second estimated location of the mobile device104. An interval of time between the first and second time may be determined. For example, a value indicative of the first time may be subtracted from a value indicative of the second time. A speed of the mobile device104may be determined by dividing the movement distance by the interval of time. The estimated motion value may comprise the direction of motion and the speed.

In another implementation, antennas116may provide information about an angular bearing to the source of a signal. For example, an electronically steerable phased array antenna116may have the phasing between a plurality of elements adjusted to electronically “sweep” or move the direction of gain. An angular bearing at which the received signal strength is greatest may be deemed to be indicative of the direction at which the transmitter of that signal is located. By using the angular bearing from two or more physically separate steerable phased array antennas116at known locations that detect the signal at approximately the same time, the location may be determined at the point where those two angular bearings intersect. In further implementations, the signal strength data from each of the steerable phased array antennas116may also be used to determine approximate distance.

At908, an inertial motion value312of the mobile device104is determined from the device data112. For example, the inertial motion value312may comprise a vector or trajectory based on the accelerometer data, gyroscope data, and magnetometer data. In one implementation, the inertial motion value312may comprise one or more vector values, individual ones of the vector values indicative of a direction of motion and a speed during a particular interval of time of the mobile device104.

At910, the estimated motion value304and the inertial motion value312are determined to be within a threshold value320of one another. For example, the two values may be compared and deemed to represent the same trajectory.

At912, one or more of an image gait value316or image motion value318may be determined from the image data122. For example, the image gait value316may include information descriptive of a walk of the user102as they pass through the portal106. Continuing the example, the image processing module314may determine a first estimated optical location of the user102from the image data122acquired at a first time. The image processing module314may also determine a second estimated optical location of the user102from the image data122acquired at a second time. One or more of the image gait value316or the image motion value318of the user102may be determined using the first estimated optical location and the second estimated optical location. For example, the image motion value318may comprise a vector value indicative of movement from the first estimated optical location to the second estimated optical location over time corresponding to the difference between the first time and the second time.

At914, one or more of the image gait value316or the image motion value318are determined to be within a threshold value320of one or more of the estimated motion value304or the inertial motion value312.

By utilizing the comparison module308, information from the different sensors110associated with the mobile device104and the portal106may be combined. For example, the device identifier114associated with the portal data118provides information about the physical presence of the mobile device104at the portal106. The device data112provides information about the motion of the user102, but may not contain information that can be directly tied to a particular location in the facility402. Likewise the image data122provides some information about the movement or relative positioning of the users102at the portal106or elsewhere in the facility402, but may not be reliable enough to use to identify a particular user102based on their appearance.

At916, account data126is accessed. As described above, the account data126associates identification data of a mobile device104with a particular account. In some implementations, the identification module324may perform the operations of916through920.

At918, the device identifier114, such as obtained with the portal data118by the portal106, is used to determine the particular account associated with the mobile device104, based on the account data126.

At920, presence data128indicative of presence of a person associated with the particular account at the specified volume is generated. For example, the identification module324may generate the presence data128.

FIG. 10depicts a flow diagram of a process1000of associating a person appearing in image data122at a portal106with a particular account, according to some implementations.

At1002, first portal data118(1) is accessed. The first portal data118(1) may be obtained using a radio108having a first receiver at a first location in the facility402. In some implementations, the first portal data118(1) is associated with a signal transmitted by a first mobile device104(1). For example, the first portal data118(1) may comprise information based on a device signal transmitted by the radio108of the first mobile device104(1).

In some implementations, the portal data118may include data identifying a particular phased antenna array. For example, the portal data118may include coordinates indicative of a location of the particular phased antenna array, an antenna identification number, and so forth. The portal data118may include an angular bearing associated with reception of the signal transmitted by the first mobile device104(1) by the particular phased antenna array. For example, the angular bearing may indicate that the device signal from the mobile device104was received an angle of 30° relative to a plane of the antenna array. The portal data118may also include signal strength data of the signal as acquired by the particular phased antenna array. For example, signal strength data may be expressed as a unitless value, decibel-milliwatts (dBm), and so forth.

At1004, a first estimated motion value304(1) of the first mobile device104(1) is determined based on the first portal data118(1). For example, the estimated motion module302may process the portal data118. In some implementations, the portal data118may be used to generate other information. A count of different device identifiers114present in the first portal data118(1) may be made. Using the count of different device identifiers114, data indicative of a count of users102at the first location in the facility402such as the portal106may be determined. For example, a group of users102may enter the facility402walking closely to one another. The image processing module314may not be able to generate data indicative of a count of users102. The count based on the device identifiers114may be used in the situation. In some implementations, a difference between a count generated by the image processing module314and the portal data118may be used to affect a confidence level in the output of the analysis module124. For example, a disagreement between the count based on the image data122to a count based on the portal data118may result in the use of a human operator to determine the count.

At1006, first device data112(1) received from the first mobile device104(1) is accessed. The first device data112(1) may be transmitted responsive to one or more of the portal signal, based at least in part on a geolocation of the first mobile device104(1), and so forth.

As described above, the first device data112(1) is indicative of information gathered by one or more sensors110associated with the first mobile device104(1). The first device data112(1) may comprise information about one or more of acceleration of the first mobile device104(1), compass heading of the first mobile device104(1), rotation of the first mobile device104(1), velocity of the first mobile device104(1), geolocation of the first mobile device104(1), or other information. For example, the velocity or the geolocation may be provided by the location sensor110(17). In one implementation, the first device data112(1) may be transmitted as a broadcast packet. This broadcast packet may be received by the one or more the radios108of the portal106, or by another radio108.

In another implementation, the first device data112(1) may be sent directly to the server120. For example, the first mobile device104(1) may establish a connection with the server120. The server120may then receive, from the first mobile device104(1) via the connection, the first device data112(1). The server120may determine first timestamp data indicative of a time of acquisition of the portal data118and second timestamp data indicative of a time of acquisition of the first device data112(1). The server120may select first timestamp data and second timestamp data that are within a threshold value320of each other. For example, the timestamp data may differ due to clock drift, timing differences, and so forth.

In some implementations, the device data112may be encrypted by the mobile device104. The server120may determine decryption credentials and then decrypt the first device data112(1) using the decryption credentials. For example, a public key/private key cryptographic system may be used with the mobile device104encrypting the first device data112(1) using a public key associate with the server120.

At1008, a first inertial motion value312(1) of the first mobile device104(1) based on the first device data112(1) is determined. For example, the inertial motion module310may process the device data112to generate the inertial motion value312.

At1010, a first correspondence is determined that indicates the first estimated motion value304(1) is within a threshold value320of the first inertial motion value312(1). For example, the comparison module308may determine the estimated motion value304(1) and the first inertial motion value312(1) are within a threshold value320of one another.

At1012, image data122is accessed. For example, the image data122may be obtained by the cameras110(1) having a field-of-view that includes at least a portion of the first location in the facility402and an image of at least a portion of a person.

At1014, one or more of an image gait value316or an image motion value318is determined. For example, the image processing module314may make this determination. As described above, the image gait value316may be indicative of a gait of the user102at the portal106that is based on the image data122.

At1016, one or more of the image gait value316or the image motion value318are determined to be within a threshold value320of one or more of the estimated motion value304or the inertial motion value312.

At1018, account data126is accessed. For example, the identification module324may access a data store that stores the account data126. The account data126may include identification data indicative of the first mobile device104(1) at the facility402, such as a previously stored device identifier114that may have been obtained during a registration process.

At1020, an account associated with the identification data is determined based at least in part on the device identifier114.

At1022, presence data128indicative of presence of a mobile device104associated with the account is generated. In some implementations, the presence data128may be indicative of a presence of a person in possession of the mobile device104that is associated with the account.

At1024, the person appearing in the image data122is designated as associated with the account. For example, the person in the image data122obtained at the portal106during entry to the facility402may be determined to be user102(3).

At1026, a confidence level of the determination of the particular account associated with the mobile device104or person is determined. The confidence level may comprise a value that indicates reliability of the presence data128. For example, the confidence level may be high when only a single user102passes through the portal106. In comparison, the confidence level may be low when a group of users102passes through the portal106in close proximity to one another. In another example, the confidence level may be low when a group of users102passes through the portal106but one or more of those users102do not have a mobile device104that participates by sending a device signal.

When the confidence level of the presence data128is below a threshold value320, further portals106throughout the facility402may be used to generate presence data128with a greater confidence level. For example, a group of users102may eventually split apart and go to different portions the facility402. As a result, those individual users102may pass through additional portals106and subsequently have presence data128generated with the confidence level increasing with each pass through a portal106.

In some implementations, a count may be made of the number of different locations at which the presence data128has been determined within a threshold time interval. Determination of confidence level may be generated based at least in part on this count. For example, the initial presence data128may have a relatively low confidence level due to the arrival of the user102within a large group. However, as the user102splits off from the group and travels through three portals106within five minutes, with consistent presence data128across those three portals106, the confidence level of the association of the mobile device104or person with the account may be increased. Continuing the example, second portal data118(2) may be obtained by a second receiver at a second location at the facility402. The second portal data118(2) is associated with operation of the first mobile device104(1), such as by having the same device identifier114in a device signal. A second estimated motion value304(2) of the first mobile device104(1) may be determined based on the second portal data118(2). Second device data112(2) may be received from the first mobile device104(1) and accessed. The second device data112(2) is indicative of information gathered by one or more sensors110associated with the first mobile device104(1) at a time corresponding to the second portal data118(2). A second inertial motion value312(2) of the first mobile device104(1) may be determined based on the second device data112(2). A second correspondence within a threshold value320between the second estimated motion value304(2) and the second inertial motion value312(2) may be determined. For example, the second estimated motion value304(2) and the second inertial motion value312(2) may be within a threshold distance of one another. Based on the first correspondence and the second correspondence, data indicative of an increased confidence level in the presence data128may be determined. For example, the confidence level may be higher when the account associated with the first mobile device104(1) is at a particular location at the facility402.

By using the devices and techniques described in this disclosure, users102may be easily and accurately associated with particular accounts, allowing for identification of the users102. Data from different sensors110is processed, and the results may be compared and used in concert with one another to determine the presence of a person associated with the particular account at the facility402or a particular portion thereof.