Patent ID: 12211023

DETAILED DESCRIPTION

This disclosure describes, in part, systems for enabling physical retail stores and other facilities to implement automated-checkout techniques for customers of the stores and/or facilities. In particular, this disclosure describes systems for retrofitting an existing retail location with a hardware and software package to enable automated-checkout (AC) using a proven and tested AC system. For example, the described systems may enable a retail store to implement technology where users are able to pick items from shelves and other inventory locations and exit the store without performing manual checkout of the items, as well as technology to allow users to pay for their items using point-of-sale (POS) and/or other manual-checkout techniques. The systems described herein thus enable retail locations of all sizes and configurations to leverage AC systems without individual development and by relying on a third party package to provide the system and operations thereof while maintaining inventory and security protocols.

The AC systems described herein may be managed and operated by a separate service provider to ensure stability, security, and operability of the system, thereby enabling facilities of all sizes and sophistication to implement such systems. Operation of the AC system may include providing functionality of the AC system, such as the operation of AC devices to track items and users within the facility to identify items selected by the user. The operation of the AC system may be performed primarily by a service provider separate from the facility service provider. In this manner, the operation of the AC system is a third-party implementation to enable the techniques described herein. Additionally, the AC systems described herein provide for secure transactions by eliminating, in some examples, the need for a user to carry or present a payment card such as a credit card. Instead, a user account secured by user credentials may store the payment information and such user payment information only accessed when needed to process payments, but not transmitted across a network, thereby further improving network security by eliminating the need to transmit such data, which often involves layers of encryption and security.

In some instances, such a facility includes inventory locations housing one or more items that may be picked and/or returned by users. These inventory locations may include AC enabled inventory locations. The AC-enabled checkout locations may be associated with one or more sensors configured to generate sensor data indicative of events that occur with respect to the items housed thereupon. For example, these sensors may generate sensor data indicative of a user picking an item from the AC-enabled inventory location, returning the item to the inventory location, and/or the like. These sensors may include overhead cameras, in-shelf cameras, weight sensors, and/or any other type of sensor configured to generate sensor data indicative of user interactions with the items.

In addition to including different inventory locations, the facility may include, in some instances, one or more entry locations for entering the facility and one or more exit locations for exiting the facility. For example, the facility may include a first entry location at which an entering user provides information for identifying an account of the user, and a second entry location at which users enter without providing identifying information and instead provides only payment information. In some examples, the identifying information may be provided as payment information, such as a credit card. In some examples, the identifying information may be accessed when the user provides information, the identifying information mapped to payment information previously that may be accessed. In some examples, the identifying information may include a palm scan or other biometric information scan. For example, the first entry location may include a scanner or other imaging device at which an entering user scans or otherwise provides a unique code associated with the account of the user, such as a code displayed on a mobile device of the user. Or, the first entry location may include a microphone, camera, or other sensor that generates sensor information at the request of the user for use in identifying the account of the user. In some examples, the second entry location may include an input device for reading information from a payment card of a user, such as a credit card, debit card, prepaid card, etc. After entering the payment information, a user account may be generated and associated with the payment card that may be used to aggregate data about the user across different shopping sessions. For example, the second entry location may include a scanner or camera that scans or captures an image of a payment card, a card reader that receives information from a payment card via a swipe, dip, tap, or the like, or may include any other type of input device configured to receive payment or account information.

In some instances, the account of the user may be associated with a payment instrument of the user such that the payment instrument is able to be charged for items procured by the user, with the charge occurring automatically upon exit of the facility by the user and without the user needing to engage in a manual checkout process of the items. The third entry location, meanwhile, may resemble a traditional retail facility entry location to allow users to shop or otherwise interact with items at the retail facility without needing to provide information for identifying user accounts.

Similarly, the facility may include a first exit location where an exiting user provides information for identifying an account of the exiting user, and a second exit location at which users exit without requiring a stop at a point-of-sale system for typical checkout processes. The first exit location may include, similar to the first entry location, a scanner or other imaging device at which the exiting user scans or otherwise provides a unique code associated with the account of the user, such as the code displayed on the mobile device of the user. Or, the first exit location may include a microphone, camera, or other sensor that generates sensor data at the request of the user for use in identifying the account of the exiting user. In some examples, the first exit location may not include a device to receive identifying information but may instead rely on communication from a user device indicative of when a user has exited a facility, for example based on location data from the user device. The second exit location, meanwhile, may resemble a traditional exit location at a retail facility, including an associate of the facility operating a POS device to manually checkout the exiting user.

Within this example facility, a user may choose to enter the facility using the first, AC-enabled entry location, the second AC-enabled entry location, or may choose to enter the facility using the third, non-AC-enabled entry location. If the user enters through the first entry location or the second entry location and provides information identifying an account of the user, then a system associated with the facility may generate a record indicating the presence of the user at the facility. The record may store an indication of the identity of the user as well as items selected by the user within the facility. In some examples, the first or second entry locations may enable the user to walk out without stopping at a point-of-sale location to check out with an associate of the facility and will instead be billed based on the account of the user or a payment associated with the account of the user.

In addition, sensor data generated from sensors at the facility may be used to determine current positioning of the user, which may also be stored in association with the record of the user. For example, overhead cameras, floor weight sensors, and/or the like may be used to maintain, in the record and at the knowledge and request/consent of the user, the current location of the user within the facility. Further, this sensor data (e.g., image data) may be used to locate an identifier of the user as the user navigates through the store. That is, the sensor data may be used to store an ongoing location of the identifier of the user, with the identifier comprising an anonymized alphanumeric string that does not include any personal identifiable information (PII). Further, if the user interacts within one or more items housed at an AC-enabled inventory location, the system may generate data indicative of the event. This data may be used to identify items selected by the user for checkout. In some instances, this data indicates an action taken (e.g., a pick of an item, a return of an item, etc.), an identity of the item acted upon (e.g., a bottle of ketchup, a pair of jeans, etc.), a quantity of the item involved, a location of the item (e.g., aisle, shelf, lane, etc.), and/or the like.

In some instances, the system may also generate a confidence level indicating a confidence associated with the data. If the confidence level associated with the data is greater than a threshold confidence level, then the data may be associated with an account of the user (e.g., a virtual cart of the user). For example, if the system determines, with a confidence level that is greater than the threshold, that the user picked one bottle of ketchup then one bottle of ketchup may be added to the virtual cart of the user. The cart of the user may store item information identifying each of the items selected by the user

When checking out, using the automated checkout system, the cart of the user may be used to generate an order list or order data identifying the identity and quantity of items selected by the user. The order data is then conveyed from the AC system to an inventory and/or management system of the facility to determine an order cost. The management system may determine the cost by accounting for the price of the items, tax on the items, loyalty card and discount information associated with the user account, coupons, or other user- and/or location-specific cost details. In this manner, the AC system may maintain flexibility for a retailer to adjust prices, implement discounts, add coupons, and otherwise track prices for items and inventory without having to turn such information over to the service provider handling the AC system. In addition, the payment information may be kept secure as only a unique identifier associated with the user account need be transmitted between the facility system and the AC system provider. Avoiding transmitting sensitive information such as credit card details provides additional layers of security to user information.

Upon finishing his or her shopping, the user may approach the AC exit location and, in some instances, scan or otherwise provide identifying information to enable the system to identify the exiting user. After scanning his or her unique code at the AC exit gate, for instance, the user may exit the facility. The system, meanwhile, may thereafter charge an account of the identified exiting user for a cost of the items procured by the user within the facility. Of course, while the above example describes the user scanning a unique code (e.g., via a mobile device of the user), it is to be appreciated that the exiting user may be identified based on other sensor data, such as image data, voice data, location data, or the like.

While some of the examples below are described with reference to a materials handling facility (e.g., a brick-and-mortar retail store, a fulfillment center, etc.), the systems and techniques may be implemented for detecting events and interactions in any type of facility, an airport, a classroom, an outdoor environment, or any other location. Certain implementations and embodiments of the disclosure will now be described more fully below with reference to the accompanying figures, in which various aspects are shown. However, the various aspects may be implemented in many different forms and should not be construed as limited to the implementations set forth herein. The disclosure encompasses variations of the embodiments, as described herein. Like numbers refer to like elements throughout.

The present description provides an overall understanding of the principles of the structure, function, manufacture, and use of the systems and methods disclosed herein. One or more examples are illustrated in the accompanying drawings. Those of ordinary skill in the art will understand that the systems and methods specifically described herein and illustrated in the accompanying drawings are non-limiting embodiments. The features illustrated or described in connection with one embodiment may be combined with the features of other embodiments, including as between systems and methods. Such modifications and variations are intended to be included within the scope of the appended claims.

Additional details are described below with reference to several example embodiments.

Turning now to the figures,FIG.1illustrates an example system for automated AC from a facility100, according to at least some examples. The system enables users, such as user102, to enter the facility100, pick one or more items, and exit the facility without performing a manual checkout of the items. The system includes an AC service provider116and a retail service provider118. The AC service provider116may control operations of an AC entry location104and a AC exit location to identify an entry of a user into the facility100where an AC system may be implemented to enable the user102to walk out of the facility without having to stop to manually check out. The AC service provider116may be implemented as a third party service provider that identifies users as they enter the facility, identifies one or more items the user102selects within the facility100, and generates a final cart as the user102exits the facility. The retail service provider118may include an inventory management system or other existing retail system used to track inventory, prices, and one or more user accounts associated with the facility100. In the system ofFIG.1, the AC service provider116may perform various operations to generate a virtual cart of items selected by the user102and convey the virtual cart to the retail service provider118for cost determination, payment processing, and receipt generation. In this manner, the retail service provider118may implement the AC service provider116, and associated system elements to identify items and place them in a virtual cart that may be checked out using traditional processes at the retail service provider118. In this system, the AC service provider116may be inserted and retrofitted to numerous environments to provide a “just walk out” checkout experience and benefit without requiring re-tooling of the retail service provider118system.

To do so, a sensor system coupled to the facility100may identify a user or receive a user input and charge an account associated with the user102for the cost of the picked items upon exit of the user102from the facility100. The AC system may be operated by a first service provider while the facility100is operated by a second service provider, also referred to as a retailer. In order to enable such retrofitting of facilities100with an AC system without requiring individual development and custom implementation at each location, which may be cost prohibitive to design and implement, the AC system described herein may be provided and operated separately to ensure smooth and accurate operation while allowing the retailer to maintain existing inventory systems and typical operations.

As illustrated, the facility100includes an AC entry location104having an gate114and a kiosk106for the user to interact with at the AC entry location104. The AC entry location104enables the user to enter the store and activate or identify their user account associated with the facility100. The gate114may request that entering users provide identifying information prior to entering the gate114. In some examples, the gate114may be a door of the facility100or may be within an outer door of the facility100. The identifying information may be provided to the gate114in a number of ways. In the illustrated example. the user102enters through the AC entry location104by scanning a unique code presented on a user device108of the user102at a scanning device at the gate114. In some examples, the unique code may be generated by a mobile application hosted by the facility100, or otherwise associated with the facility100. The gate114may also identify the identifying information from the user based on a user account associated with the user device108, based on one or more additional sensors, such as cameras, scanners, and other such devices to access unique identifying information from the user device108. In an example, the user102may enter a grocery store location and open a mobile application on their user device108to access identifying information, such as a QR code, barcode, alphanumeric code, picture, or other identifying data that may be used to connect or associate the user device108with an account the user102has established with the grocery store. Of course, while this example describes identifying the user based on the user scanning an unique code presented on the user device108, the system may additionally, or alternatively, identify the user based on voice data (e.g., the user stating his or her name), image data (e.g., image data of the user), password data (e.g., an alphanumeric string), and/or any other type of data. For example, in some instances the facility100may include a single entry location, with user identification occurring based on image data of users entering the facility100. In these examples, those users that have consented/requested to take part in the automatic-checkout techniques may be identified, while the system may refrain from identifying other users entering the facility100.

In some examples, the gate114may provide the identifying information to a service provider system, such as a management system or inventory system of the facility100. In some examples, the gate114may store the identifying information in a storage system of a service provider running the AC system without communicating the identifying information to the management system until exit and checkout of the user102.

In some examples, the user102may not have an account associated with the facility100, may not have brought their user device108, or may prefer to shop separate from their personal account. In such examples, the AC entry location104includes a kiosk106where a user may enter payment information, such as credit card information, to establish a temporary or anonymous profile for the shopping session. The profile may be associated with a unique identifier that may be used to track information regarding the user's purchases across multiple sessions by identifying the payment information and locating the corresponding account information. In some examples, the gate114may be equipped to read data from a credit card and may therefore be used in place of the kiosk106. In any event, the user102may present their payment information to the AC entry location104to gain access to the facility and enable use of the AC system. In some examples, the AC system may use the identifying information received through the swipe, scan, or input of the payment information and/or user account to verify the payment method is valid. This AC system may use this identifying information to determine whether the payment instrument is valid (potentially along with whether the payment instrument is a prepaid card, such as a gift card as well as whether it has a limit of at least a pre-authorization amount applied by the AC system) and may store an indication that the user is eligible to exit the facility without performing manual checkout of the items (assuming the total of the items is less than the limit). In these instances, the system may or may not identify the user102but may instead simply associate the user in the facility100with the identified payment instrument.

In some examples, the AC system may perform a risk analysis, sometimes referred to as a fraud and abuse analysis. The risk analysis may determine a risk score for the user102, for example indicative of a risk that the user may attempt to perpetuate a fraud or abuse the AC system. The risk analysis may be based on identifying information and/or other data gathered by the sensors of the AC system, the facility, accessible via the user account information, or entered payment information. In some examples, the AC system may perform the risk analysis and pre-authorize a predetermined amount on the payment method provided by the user in the user account. In some examples, the AC system may pre-authorize variable amounts based on the risk score determined. In some further examples, the AC system may perform a risk analysis in addition to a risk analysis performed by the facility management system. In some extreme examples, the AC system may block the gate114or prevent a user102from using the AC system in response to the risk score exceeding a predetermined threshold. For example, a user102may be asked to use a point-of-sale system and to interact with an associate of the facility100to checkout.

Returning to the example, upon the user102entering the facility100via the AC entry location104, the AC system generates a record indicating the presence of the user102within the facility. In some examples, the record may be continuously or periodically updated by a location system of the AC system to generate current location data for the user102within the facility100, at the prior consent of the user102. The facility100may include sensors, such as overhead cameras or the like that may be used to determine the current location of the user102.

The user102enters the facility and proceeds to a location110within the facility. The location may include one or more sensors to record events at the location110and thereby identify an action, such as a selection of an item from the location. For example, the location110may be within the field-of-view of one or more overhead, cameras, in-shelf cameras, or the like, image data from which may be used to determine the results of events that occur at the location110. In addition, or in the alternative, the location110may include one or more other sensors, such as in-shelf weight sensors or the like, which may generate sensor data for identifying the results of the events.

The user102removes an item110(1) from the location110. In response, the system may use sensor data indicative of this event to determine result data associated with this event. In an example, the AC system may determine the identity of the item110(1) selected by the user102. In some other examples, other events may be identified to include picking, returning, moving, or other interactions. The identify may be determined by the AC system based on one or more techniques including optical recognition techniques, location-based identification, and other sensor settings used to identify an item type and quantity selected by the user102.

After selecting the item110(1) the user102may exit the facility100at the AC exit location112. The AC exit location112may include an exit gate at which the user102provides identifying information for indicating to the system that the corresponding user is exiting the facility. For example, similar to the gate114, the AC exit location112may enable users to scan unique codes from their mobile phones or provide any other type of identifying information. In still other instances, users may walk out and the system may identify these users via biometric-recognition techniques, in instances where the users have so requested.

In response, the system may end a shopping session of the corresponding user. In some examples, as with gate114, the AC exit location112may end the shopping session based on sensor information or data indicative of the user102exiting the facility100.

After ending the shopping session of the user102, the AC system may identify items within a virtual shopping cart based on the items selected by the user102during the shopping session, such as item110(1). The AC system may identify the items within the cart and convey the cart information, including the quantity and types of items to a facility system. The facility system may include the inventory management system, a payment system, or other system to manage generation of costs for customer orders.

The facility system may then process the order by first accounting for the price of the items in the order, then performing any and all adjustments and calculations necessary to produce a final total for the order. In an example, the system may generate a subtotal based on the cost of the items in the cart. The subtotal may be adjusted by accounting for taxes and discounts available. In some examples, the system may access loyalty information for the user102to access a loyalty program that provides discounts and apply such discounts to the subtotal. Other adjustments and changes to the price may be made by the system to produce the final total for the order to be billed to user102.

In some examples, the facility100may directly bill the user based on the payment information stored in the user account hosted by the facility100. In some examples, such as when the user102enters the gate114using a credit card but without a user account, the AC system or the facility system may perform the final payment processing to bill the user102.

In a first example, where an AC system is fully delegated to handle interactions on behalf of the facility100, the identifying information provided by the user102at the gate114is received by the AC system. The items selected by the user102, such as item110(1) are tracked and sent as cart items to the facility system upon exit from the facility100. The cart information includes the items and quantities of items selected and also identifies the identifying information provided at the gate114. The facility100may then use the identifying information to access the user account, identify the payment information entered into the user account by the user, identify any discounts such as loyalty and coupon discounts, and then bills the payment method provided in the user account.

In a second example where an AC system is partially delegated to handle transaction on behalf of the facility100, the AC system may perform additional tasks. For example, the user102may provide the identifying information at the gate114. The AC system may perform the risk analysis discussed above and may charge a pre-authorization amount to the payment device provided by the user102. The AC system may also convey the information to the facility100which may perform a second risk analysis. As a result of the one or both risk analyses, the user102may be permitted to enter through gate114and shop in the facility100. At the end of the shopping session, the AC system sends the cart to the facility system. In contrast with the first example, the facility system may then send a request to the AC system to process payment for the final total. In such examples, the payment device may not be transmitted to the facility100but instead the final cart total price is conveyed back to the AC system which communicates with a payment processor, or processes the payment directly, for the amount indicated by the facility system. In some examples, the pre-authorization amount may be used to accomplish the payment processing, for example by charging the order total and releasing the remainder of the pre-authorization. In an additional step, the AC system or payment processor then settles with the facility100such that the payment is received at the facility system from the user either directly or via the AC system.

FIG.2illustrates a system architecture200for an AC system operated by a service provider separate from, but in operation with, the facility operator, according to at least some examples. In the system architecture200, a facility service provider210communicates with an AC service provider212to operate the AC system within a facility, such as facility100ofFIG.1. The system architecture200also includes item data202stored in a database, AC device(s)204, and user device206. The AC service provider212may be an example system to carry out the actions of the AC system described inFIG.1. The facility service provider210may be part of an inventory management system of a facility, such as facility100. The user device206may be an example of the user device108. The AC device(s)204may include the sensor systems associated with the AC system ofFIG.1, some of which are shown and described with respect toFIG.3below. The user device206may be used by the user102to interact with the AC service provider, for example at gate114to provide payment information or identifying information. The user device206may also communicate with the AC device(s)204as well as the AC service provider212to enable the system to identify the items selected by the user102within the facility100.

InFIG.2, the elements of the system architecture200are shown in communication over network208. The network208may be a communication network that enables the various elements of the system architecture200to communicate. In some examples, the network208may include one or more different networks or communication pathways. The communication network may include any one or a combination of many different types of networks, such as cable networks, the Internet, wireless networks, cellular networks, and other private and/or public networks. In addition, the communication network may comprise multiple different networks. For example, the user device206may utilize a wireless local area network (WLAN) to communicate with a wireless router, which may then route the communication over a public network (e.g., the Internet) to the remote computing device AC service provider and/or the facility service provider210.

The item data202may include the items stored in the facility100and may include an inventory management system, or an inventory tracking system. In some examples, the item data202includes identities and identifying information for the items in the facility100to enable the AC system to identify the items as the user102interacts with them. In some examples, the item data202may also include price information or other such data that the facility service provider210may use to generate the order price for a cart of items selected by the user102.

The facility service provider210includes one or more components and may include one or more computing devices. The facility service provider210includes a memory232, one or more processor(s)234, additional storage238, communication connections240, and input/output devices236. The processor(s)234may be implemented as appropriate in hardware, computer-executable instructions, firmware or combinations thereof. Computer-executable instruction or firmware implementations of the processor(s)234may include computer-executable or machine executable instructions written in any suitable programming language to perform the various functions described.

The memory232may store program instructions that are loadable and executable on the processor(s)234, as well as data generated during the execution of these programs. The memory232may be volatile (such as random access memory (RAM)) and/or non-volatile (such as read-only memory (ROM), flash memory, etc.). The facility service provider210may also include additional storage238, such as either removable storage or non-removable storage including, but not limited to, magnetic storage, optical disks, and/or tape storage. The disk drives and their associated computer-readable media may provide non-volatile storage of computer-readable instructions, data structures, program modules, and other data for the computing devices. In some implementations, the memory232may include multiple different types of memory, such as static random access memory (SRAM), dynamic random access memory (DRAM) or ROM. Turning to the contents of the memory232in more detail, the memory232may include an operating system242and one or more application programs or services for implementing the features disclosed herein including at least a user application module244, an account management module246, a risk module248, and a payment module250.

In accordance with an embodiment, the user application module244may provide or host a mobile application for users to interact with. The mobile application may be used to generate the identifying information described above and display the same on the user device206. The user application module244may provide different functionality for the mobile application, such as to include coupons, account information, receive entered payment information, and other such interactions.

The account management module246may provide for management of user accounts associated with the facility100. In an example, the user accounts may include loyalty accounts, discount codes, corporate accounts, family accounts and groups, and other such user accounts. The account management module246may store and access user account information such as the identifying information requested by the user application module244to display at the user device206and may also store and access the payment information entered by users.

The risk module248may be used by the facility service provider210to perform and fraud and abuse analysis for users entering the facility discussed above. In some examples, the risk module may implement machine learning techniques and methods to generate a risk score. In some examples, the risk module248may operate a risk analysis and output a binary output that may be used to assess whether to enable the use of the AC system by a particular user.

The payment module250may include information and instructions for processing and instructing payment for cart items. In some examples, the payment module250may calculate order totals to generate a payment amount to bill a user at the end of a shopping session. The payment module250may also interact with a payment processing interface or other service to perform the payment processing.

The AC service provider212includes components similar to those described above with respect to the facility service provider, including the communication connections222, storage220, input/output devices218, and processor216. The communications connections222allow the AC service provider212to communicate with the other components of the system. The input/output (I/O) device(s)218may be used to enable connection with a keyboard, a mouse, a pen, a voice input device, a touch input device, a display, speakers, a printer, etc.

The memory214of the AC service provider212may include an operating system224, a payment module230similar to the payment module250, a user application module228similar to the user application module244, a risk module270similar to the risk module248, and may also include an order module226. The order module226may be used to communicate with the AC device(s)204to determine one or more items selected by a user to comprise a cart or order. The items may be identified by accessing information from the item data202.

The user device206may be any suitable electronic device that is capable of providing at least a portion of the capabilities described herein. In particular, the user device206may be any electronic device capable of communicating with the AC service provider212, and the other elements of the system, receiving data including payment information, and displaying information such as identifying information. The user device206may also communicate with a remote computing device, e.g., the facility service provider210and/or the AC service provider212to access information and/or to operate a mobile application installed on the user device206. In an embodiment, the user device206may be capable of establishing a communication session with another electronic device and transmitting/receiving data from that electronic device. The user device206may include the ability to download and/or execute mobile applications. User device206may include mobile communication devices as well as personal computers and thin-client devices. In some embodiments, a user device206may include any portable electronic device that has a primary function related to communication. For example, the user device206may be a smart phone, a personal data assistant (PDA), or any other suitable handheld device. The user device206can be implemented as a self-contained unit with various components (e.g., input sensors, one or more processors, memory, etc.) integrated into the user device206.

In an embodiment, the user device206may include a memory252and one or more processing units (or processor(s))262. The processor(s)262may be implemented as appropriate in hardware, computer-executable instructions, firmware or combinations thereof. Computer-executable instruction or firmware implementations of the processor(s)262may include computer-executable or machine executable instructions written in any suitable programming language to perform the various functions described. The user device206may also include one or more input sensors260for receiving user and/or environmental input. There may be a variety of input sensors260capable of detecting user or environmental input, such as an accelerometer, a camera device, a depth sensor, a microphone, a global positioning system (e.g., GPS) receiver, etc.

The memory252may store program instructions that are loadable and executable on the processor(s)262, as well as data generated during the execution of these programs. Depending on the configuration and type of user device206, the memory252may be volatile (such as random access memory (RAM)) and/or non-volatile (such as read-only memory (ROM), flash memory, etc.). The user device206may also include additional storage264, such as either removable storage or non-removable storage including, but not limited to, magnetic storage, optical disks, and/or tape storage. The disk drives and their associated computer-readable media may provide non-volatile storage of computer-readable instructions, data structures, program modules, and other data for the computing devices. In some implementations, the memory252may include multiple different types of memory, such as static random access memory (SRAM), dynamic random access memory (DRAM) or ROM. Turning to the contents of the memory252in more detail, the memory252may include an operating system254and one or more application programs or services for implementing the features disclosed herein including at least a mobile application256. The memory252may also include application data258, which provides information to be generated by and/or consumed by the mobile application256. In some embodiments, the application data258may be stored in a database.

For the purposes of this disclosure, a mobile application may be any set of computer executable instructions installed upon, and executed from, a user device206. In some embodiments, the mobile application256may cause a user device206to establish a communication session with the AC service provider212and/or the facility service provider210that provides backend support for the mobile application256. The remote computing device of the AC service provider212and/or the facility service provider210may maintain account information associated with a particular user device and/or user. In some embodiments, a user may be required to log into the mobile application256in order to access functionality provided by the mobile application256.

In accordance with an embodiment, the mobile application256may be configured to interact with the AC service provider212, for example by outputting a display of a unique identifier for scanning at the gate114, or otherwise communicating identifying information to the AC service provider212. The mobile application256may receive output from the input sensors260and provide interactions to view a cart, access payment or account information and settings, and otherwise interact with the AC service provider212. In some embodiments, the mobile application256may cause the user device206to transmit the output obtained from the input sensors260to the remote computing devices of the AC service provider212and/or the facility service provider210, which may then perform one or more operations.

The user device206may also contain communications interfaces266that enable the user device206to communicate with any other suitable electronic devices. In some embodiments, the communication interface266may enable the user device206to communicate with other electronic devices on a network208(e.g., on a private network). For example, the user device206may include a Bluetooth wireless communication module, which allows it to communicate with another electronic device. The user device206may also include input/output (I/O) device(s) and/or ports268, such as for enabling connection with a keyboard, a mouse, a pen, a voice input device, a touch input device, a display, speakers, a printer, etc.

In an example, the user device206may communicate with the AC service provider212as the user device206is brought into the facility100by the user102. The user device206may provide identifying information via the mobile application256to the AC service provider212to initialize a shopping session. The AC service provider212may then interact with the AC device(s)204, the user device206, and the item data202to identify items selected by the user102within the facility and added to a cart. The AC service provider may generate, via the order module226, a virtual cart of the items. The AC service provider212may then communicate the identifying information of the user and the virtual cart to the facility service provider210for determination of a cost for the virtual cart. The facility service provider210or the AC service provider212may process payment for the order after the user device206is carried out of the facility100as described above.

FIG.3illustrates a block diagram of an example materials handling facility that includes sensors and an inventory management system configured to identify items selected by a user within the facility as part of an AC system, according to at least some examples. Though a particular facility is described herein, the following description is merely one illustrative example of an industry and environment in which the techniques described herein may be utilized. The facility302comprises one or more physical structures or areas within which one or more items304(1),304(2), . . . ,304(Q) (generally denoted as304) may be held. As used in this disclosure, letters in parenthesis such as “(Q)” indicate an integer result. The items304comprise physical goods, such as books, pharmaceuticals, repair parts, electronic gear, groceries, and so forth.

In some examples, the system described and shown with respect toFIG.3may represent a retrofit system including an inventory management system322that may be used at a facility as an AC system to track items and users and provide automated checkout as users exit the facility or move items within the facility. The system may be incorporated with an inventory system of an existing facility as well as payment systems as described above.

The facility302may include one or more areas designated for different functions with regard to inventory handling. In this illustration, the facility302includes a receiving area306, a storage area308, and a transition area310. The receiving area306may be configured to accept items304, such as from suppliers, for intake into the facility302. For example, the receiving area306may include a loading dock at which trucks or other freight conveyances unload the items304.

The storage area308is configured to store the items304. The storage area308may be arranged in various physical configurations. In one implementation, the storage area308may include one or more aisles312. The aisle312may be configured with, or defined by, inventory locations314on one or both sides of the aisle312. The inventory locations314may include one or more of shelves, racks, cases, cabinets, bins, floor locations, or other suitable storage mechanisms for holding or storing the items304. The inventory locations314may be affixed to the floor or another portion of the facility's structure, or may be movable such that the arrangements of aisles312may be reconfigurable. In some implementations, the inventory locations314may be configured to move independently of an outside operator. For example, the inventory locations314may comprise a rack with a power source and a motor, operable by a computing device to allow the rack to move from one location within the facility302to another.

One or more users316(1),316(2), . . . ,316(U), totes318(1),318(2), . . . ,318(T) (generally denoted as318) or other material handling apparatus may move within the facility302. For example, the users316may move about within the facility302to pick or place the items304in various inventory locations314, placing them on the totes318for ease of transport. An individual tote318is configured to carry or otherwise transport one or more items604. For example, a tote318may include a basket, a cart, a bag, and so forth. In other implementations, other agencies such as robots, forklifts, cranes, aerial drones, and so forth, may move about the facility302picking, placing, or otherwise moving the items304.

One or more sensors320may be configured to acquire information in the facility302. The sensors320may be examples of the AC device(s)204ofFIG.2above. The sensors320in the facility302may include sensors fixed in the environment (e.g., ceiling-mounted cameras) or otherwise, such as sensors in the possession of users (e.g., mobile phones, tablets, etc.). The sensors320may include, but are not limited to, cameras320(1), weight sensors, radio frequency (RF) receivers, temperature sensors, humidity sensors, vibration sensors, and so forth. The sensors320may be stationary or mobile, relative to the facility302. For example, the inventory locations314may contain cameras320(1) configured to acquire images of pick or placement of items304on shelves, of the users316(1) and316(2) in the facility302, and so forth. In another example, the floor of the facility302may include weight sensors configured to determine a weight of the users316or other object thereupon.

During operation of the facility302, the sensors320may be configured to provide information suitable for tracking how objects move or are selected by the users within the facility302. For example, a series of images acquired by a camera320(1) may indicate removal of an item304from a particular inventory location314by one of the users.

While the storage area308is depicted as having one or more aisles312, inventory locations314storing the items304, sensors320, and so forth, it is understood that the receiving area306, the transition area310, or other areas of the facility302may be similarly equipped. Furthermore, the arrangement of the various areas within the facility302is depicted functionally rather than schematically. For example, multiple different receiving areas306, storage areas308, and transition areas310may be interspersed rather than segregated in the facility302.

The facility302may include, or be coupled to, an inventory management system322, which may perform some or all of the techniques described herein. For example, the inventory management system322may include the AC system including the AC service provider212and may maintain a virtual cart of each user within the facility. The inventory management system322may also store a record associated with each user indicating the identity of the user, and the location of the user.

As illustrated, the inventory management system322may reside at the facility302(e.g., as part of on-premises servers), on the servers332that are remote from the facility302, or a combination thereof.

By determining the interaction of users with the items304, the inventory management system322may generate output data326. The output data326comprises information about the event324. For example, where the event324comprises an item304being removed from an inventory location314, the output data326may comprise an item identifier indicative of the particular item304that was removed from the inventory location314and a user identifier of a user that removed the item.

The inventory management system322may use one or more automated systems to generate the output data326. For example, an artificial neural network, one or more classifiers, or other automated machine learning techniques may be used to process the sensor data from the one or more sensors320to generate output data326. For example, the inventory management system322may perform some or all of the techniques for generating and utilizing a classifier for identifying user activity in image data, as described in detail above. The automated systems may operate using probabilistic or non-probabilistic techniques. For example, the automated systems may use a Bayesian network. In another example, the automated systems may use support vector machines to generate the output data326or the tentative results. The automated systems may generate confidence level data that provides information indicative of the accuracy or confidence that the output data326or the tentative data corresponds to the physical world. The inventory management system322may perform the actions described above, but may also be separate from an inventory tracking system of a retailer that tracks item quantities for ordering by the retailer.

The inventory management system322may access or generate sensor data about the facility302and the contents therein including the items304, the users316, the totes318, and so forth. The sensor data may be acquired by one or more of the sensors320, data provided by other systems, and so forth. For example, the sensors320may include cameras320(1) configured to acquire image data of scenes in the facility302. The image data may comprise still images, video, or a combination thereof. The image data may be processed by the inventory management system322to determine a location of the user316, the tote318, the identity of the user316, and so forth. As used herein, the identity of the user may represent a unique identifier of the user (e.g., name, number associated with user, username, etc.), an identifier that distinguishes the user amongst other users being located with the environment, or the like.

The inventory management system322, or systems coupled thereto, may be configured to identify the user316, as well as to determine other candidate users. In one implementation, this determination may comprise comparing sensor data with previously stored identity data. For example, the user316may be identified presenting biometric or other PII data to an identity recognition system and/or a credential system, by presenting a token carrying authentication credentials, providing a fingerprint, scanning a barcode or other type of unique identifier upon entering the facility, and so forth. Identity of the user316may be determined before, during, or after entry to the facility302. Determination of the user's316identity may comprise comparing sensor data associated with the user316in the facility302to previously stored user data.

In some instances, the inventory management system group users within the facility into respective sessions. That is, the inventory management system322may utilize the sensor data to determine groups of users that are effectively “together” (e.g., shopping together). In some instances, a session may include multiple users that entered the facility302together and, potentially, that navigate the facility together. For example, when a family of two adults and two children enter the facility together, the inventory management system may associate each user with a session. Locating sessions in addition to individual users may help in determining the outcome of individual events, given that users within a session may not only individually pick or return or otherwise interact with items, but may also pass the items back and forth amongst each other. For instance, a child in the above example may pick the box of cereal before handing the box to her mother, who may place it in her tote318. Noting the child and the mother as belonging to the same session may increase the chances of successfully adding the box of cereal to the virtual shopping cart of the mother. After the box of cereal is added to the virtual shopping cart, along with any other items304, the chopping cart may be processed for payment as described herein, to provide for seamless checkout processes enabling the family to walk directly out the exit while paying for all the items selected by the group.

FIGS.4-6illustrate flow diagrams of example processes400-600for delegating checkout operations of a retail facility to an AC system, according to at least some examples. The processes described herein are illustrated as collections of blocks in logical flow diagrams, which represent a sequence of operations, some or all of which may be implemented in hardware, software or a combination thereof. In the context of software, the blocks may represent computer-executable instructions stored on one or more computer-readable media that, when executed by one or more processors, program the processors to perform the recited operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures and the like that perform functions or implement data types. The order in which the blocks are described should not be construed as a limitation, unless specifically noted. Any number of the described blocks may be combined in any order and/or in parallel to implement the process, or alternative processes, and not all the blocks need be executed. For discussion purposes, the processes are described with reference to the environments, architectures and systems described in the examples herein, such as, for example those described with respect toFIGS.1-3, although the processes may be implemented in a wide variety of other environments, architectures and systems. The order in which the operations or steps are described is not intended to be construed as a limitation, and any number of the described operations may be combined in any order and/or in parallel to implement processes400-600.

FIG.4illustrates an example process400for full delegation of operating an AC system within a facility to enable retail transactions between a retailer and a user, according to at least some examples. The process400includes information related to processes carried out by a retailer402as well as an AC service provider404. The retailer402may be an example of the facility service provider210ofFIG.2. The AC service provider404may be an example of the AC service provider212ofFIG.2.

At406, the process400includes generating a user identifier with a service provider application406. The user identifier may be generated by a mobile application on a user device in communication with the retailer402, specifically with a service provider system of the retailer402. As described above, the user identifier may include visual markers or identifiers that may be used to identify the user account as the user enters the facility. The user identifier may include other non-visual markers such as audible markers in some examples. In some examples, the user identifier may not be generated by the facility302but may be generated at the AC service provider404and communicated to the retailer402. As illustrated, the identifying information is communicated to the AC service provider404as the user device communicates the information.

At408, the process400includes the service provider404determining whether a user entering the facility scans an identifier. The identifier may be one generated at406as described above. The identifier may not be visible in some examples, and may not be scanned, but may otherwise be communicated to the AC service provider404, for example with location data tagged with user information communicated to the AC service provider404as the user enters the facility. In some examples, the user may be identified as entering a location based on a tracklet assigned to the user when viewed by one or more imaging devices. In such examples, the tracklet may be identified entering, exiting, or passing through a volume of interest (“VOI”), such as an entry or exit of the store, and may use the tracklet to associate with a user identifier to identify items selected by the user. In some examples, the user may approach or enter a gate of the retailer402. When the user passes through the VOI, they may scan their identifier, which is associated with the tracklet assigned to the user. In such examples, the user may be identified as exiting and entering different regions of the retail location based on the tracklet.

When the user scans or otherwise presents the identifying information to the AC service provider404, the process400proceeds to410, where sensor data, such as from AC device(s)204ofFIG.2or the sensors ofFIG.3is communicated to the AC service provider404, which may be an embodiment of the inventory management system322. In response to the data, the AC service provider404may identify an item selected by a user and may store the item in a virtual car at412. The virtual cart may include a listing of each item the user has removed and taken with them. In some examples, the virtual cart may include a log of all interactions or events involving the user and items in the environment, such as picking, returning, or moving the items. The virtual cart may determine items kept by the user from the log to determine a final set of items.

At414, the process400includes determining whether additional sensor data is indicative of an event, such as picking items. In the event additional sensor data is available, the process400returns to410and repeats to identify all items interacted with. When no additional sensor data is indicative of events where the user interacts with an item, the process400proceeds to416.

Returning to408, when the user does not scan an identifier, or otherwise provide identifying information to the AC service provider404, the process400includes directing the user to a kiosk at420. The kiosk may be part of the entry to the facility as described with respect toFIG.1above. The kiosk may be included as part of a service location or a point-of-sale location of the retailer402. In some examples, the kiosk may provide a portal for the user to enter contact information, such as an e-mail address, where a receipt from their shopping experience may be sent after completion. The kiosk may include an interface where a user may select a user account for use in the shopping session, or otherwise input payment or account information. In some examples, the kiosk may provide an option for a user to use cash with the AC system, for example by depositing an amount of cash at the kiosk to a virtual account, the virtual account either tied to the user's personal account that may be debited, or to the user identifier. Upon reaching the exit, the user may be directed to a point-of-sale kiosk to retrieve change leftover or otherwise settle the balance.

At422, the process includes determining when the user has an account in the system. When the user has an account and merely needs to select or identify the account, for example if they left their mobile device behind, then once the account is selected by entering user credentials, the process400proceeds to410as described above.

When the user is unable to access or identify an exiting user account, the process400includes directing the user424to enter payment information that may be used to generate an account for use with the AC service provider system404. The user may enter payment information to generate an account. In some examples, the user may enter payment information without generating an account. The AC service provider404may generate an identifier to associate with the payment information, such that the identifier may be communicated to the retailer402without transmitting the payment information directly. In some examples, the kiosk input enables the user to provide the payment information directly to the retailer, as other users have with user accounts, and then the AC service provider uses the identifier but not the payment information for the processes described herein to identify the user.

At416, the process400includes determining that the user exits the facility. The determination may be made in response to the user presenting their identifying information at the exit of the facility. In some examples, the location of the user device may be used to determine when the user has exited. In some examples, sensor data, such as cameras and machine learning or object recognition techniques, may be used to identify when users exit the facility. In some examples, prior to exiting the facility, the tracklet of the user, or the user identifier, may be used to identify that the user is at or near the exit location of the retailer402. The user may have selected one or more items that require identity verification, such as age-restricted items. In such examples, the AC service provider404may direct, via a gate at the exit or display, the user to discuss with a sales associate to confirm their age or identity before proceeding. In such examples, the AC service provider404extrapolates actions to be performed before the user exits or their order is processed based on items selected by the user. The user may not be able to exit through a gate until the conditions are met, such as age verification.

After the user exits the facility, or otherwise terminates their shopping session, the AC service provider communicates the cart items as order data and a user identifier indicating a user associated with the order data to the retailer402for payment processing. Because the AC service provider404does not have access to the user accounts in full, and the full prices and tax or other cost information, the retailer402performs the final processing to determine the payment amount due from the user.

At426, the retailer creates a subtotal of the items in the cart from the order data sent from the AC service provider404. The subtotal is then adjusted based on tax and loyalty and/or discount information at428and430before generating a final payment amount due from the user. At432, the retailer402bills the user (client) based on the final payment amount determined at426-430. The retailer402can charge the payment directly to the user based on the payment information accessible from the user account.

FIG.5illustrates an example process for partial delegation of operating an AC system within a facility to enable retail transactions between a retailer and a user, according to at least some examples. The process500includes information related to processes carried out by a retailer502as well as an AC service provider504. The retailer502may be an example of the facility service provider210ofFIG.2. The AC service provider504may be an example of the AC service provider212ofFIG.2.

At506, the process500includes the AC service provider504receiving payment information from a user. The payment information may be received by accessing a user account based on a user identifier received at an entrance to the facility or may be received directly, for example when a user presents their credit card to enter the gate to the facility. In any event, the payment information is accessible to the AC service provider504but is not available to the retailer502. In some examples, the payment information may be used as a proxy for the user identifier, and a truncated portion of the payment information may be used as the user identifier.

After receiving the payment information, the process500includes generating a user risk score at508. The risk score may be generated based on a risk analysis of the user and/or the payment information entered. The risk analysis may determine a risk score for the user, for example indicative of a risk that the user may attempt to perpetuate a fraud or abuse the AC system. The risk analysis may be based on identifying information and/or other data gathered by the sensors of the AC system, the facility, accessible via the user account information, or entered payment information. In some examples, the AC system may access a user account to identify an existing debt account with the AC service provider504that may be weighed as part of the risk score. In some examples, the AC system may perform the risk analysis and pre-authorize a predetermined amount on the payment method provided by the user in the user account. In some examples, the AC system may pre-authorize variable amounts based on the risk score determined. In some further examples, the AC system may perform a risk analysis in addition to a risk analysis performed by the facility management system. In some extreme examples, the AC system may block the gate or prevent a user from using the AC system in response to the risk score exceeding a predetermined threshold. For example, a user may be asked to use a point-of-sale system and to interact with an associate of the facility to checkout.

At510, the unique identifier of the user associated with the payment information and the risk score determined by the AC service provider are conveyed to the retailer502. The retailer may choose to perform a second risk analysis at512. In some examples, the second risk analysis may be like the risk analysis carried out at508. In some examples, the retailer502may choose to perform a risk analysis based on different criteria. In some examples, the retailer502may receive the risk score from the AC service provider504and generate a score based on that input. In some cases, the received score maybe output as the final score, in some examples it may be one input, among others, into a risk analysis to determine a risk score.

At514, the retailer makes a determination of whether to allow the user to use the AC system. In some examples, the risk score may exceed a threshold or there may be insufficient confidence in a user account or payment method. In such examples, the process500may cause the AC service provider504or the retailer502to direct the user to use a traditional point-of-sale system or otherwise interact with an associate to resolve the risk score or complete their transaction in an alternative manner.

In the event the AC service provider or the retailer does not allow the user to use the AC system at step514, the user id directed to a point of sale system at516. The point of sale may include a kiosk or service counter where the user may be able to settle an account or otherwise resolve any issues preventing their use of the AC system.

When the retailer determines that the risk score is acceptable, such as below a threshold, or decides to disregard the risk score, then the retailer conveys a gate decision to the AC service provider504to open the gate at step518. The gate decision instructs the gate, such as gate114to open and allow the user to enter the facility and shop using the AC system.

At520, the process500includes the AC service provider504communicating with sensors of the AC system to receive sensor data and item data. In response to the data, the AC service provider504may identify an item selected by a user and may store the item in a virtual car at522. The virtual cart may include a listing of each item the user has removed and taken with them. In some examples, the virtual cart may include a log of all interactions or events involving the user and items in the environment, such as picking, returning, or moving the items. The virtual cart may determine items kept by the user from the log to determine a final set of items.

At524, the process500includes determining whether additional sensor data is indicative of an event, such as picking items. In the event additional sensor data is available, the process500returns to520and repeats to identify all items interacted with. When no additional sensor data is indicative of events where the user interacts with an item, the process500proceeds to526.

At526, the process500includes determining that the user exits the facility. The determination may be made in response to the user presenting their identifying information at the exit of the facility. In some examples, the location of the user device may be used to determine when the user has exited. In some examples, sensor data, such as cameras and machine learning or object recognition techniques, may be used to identify when users exit the facility.

After the user exits the facility, or otherwise terminates their shopping session, the AC service provider504communicates the cart items as order data and a user identifier indicating a user associated with the order data to the retailer402for cost processing at528.

At530, the retailer502creates a subtotal of the items in the cart from the order data sent from the AC service provider504. The subtotal is then adjusted based on tax and loyalty and/or discount information at532and534before generating a final order total payment amount due from the user at536. The order total is then conveyed from the retailer502to the AC service provider504at536so the AC service provider, which has access to the payment information, is able to process the payment at538.

At538, the AC service provider504bills the user (client) based on the final payment amount determined at536. The step538is shown on both retailer502and the AC service provider504sides because it may be performed by one or both. In some examples, the AC service provider504triggers payment to a payment acquirer who settles with the retailer. In this example, the AC service provider504does not process the payment, but only provides a trigger to initiate payment processing by a payment application. In some examples, the AC service provider504may process the payment and act as a payment processor.

FIG.6illustrates an example process600for processing an AC system transaction on behalf of a retailer, according to at least some examples. The process600begins at602with an AC system receiving an indication that a user is located at a facility equipped with an AC system including sensors and techniques for determining which items in the facility the user interacts with. In some examples, the indication is received as the user enters and scans identifying information at an entrance of the facility. In some examples the indication may be based on a user device and a location of the user derived from the user device being within the facility.

At604, the process600includes the AC system conveying a user identifier to a facility managing system. The user identifier may identify a user account or otherwise uniquely identify a user and payment method without including the payment method or payment information in the conveyed data. The user identifier may be used by the facility managing system to access a user account and identify payment information in some examples. In some cases, the user identifier may be used to track individual shopping behaviors between shopping sessions to provide a shopping history for a user that may be used as an input for a risk analysis described herein or for otherwise improving a shopping experience for the user.

At606, the process600includes receiving item data indicative of items selected by the user. The item data may be received from the AC system as items are identified based on sensor data, image data, and other data from within the facility. The items are identified and associated with the user identifier and/or the user account accessed above. The items may be stored in a virtual cart that is used ultimately to determine the cost for the items at the facility management side.

At608, the process600includes storing the order data at the AC system. The order data may be stored when the user completes their shopping session, such as by exiting the facility. The order data may include item identities and quantities of each item as indicated in the virtual cart for the user.

At610, the process600includes conveying the order data from the AC system to the facility management system. The order data may be conveyed in its entirety to the facility management system for the system to determine a price for the items in the order and appropriately charge the user. The facility management system may determine the price based on prices stored at the system for the items, any discounts or coupons for the user, location-based taxes, and any other adjustments to price.

At612, the process600includes instructing payment for the order. Instructing the payment for the order may include the facility management system directly billing the user at the payment information stored in the user account. In some examples, the AC system may be delegated to handle payment processing and payment information. In such examples, the AC system may be built with robust security features to protect the user account information and the payment information and avoid conveying such information across network communications. Instructing the payment processing may include receiving the final order total from the retailer and subsequently processing the payment via the AC system based on payment information stored thereon associated with the user.

In some examples, the process600may include additional steps, for example to receive contact information, such as an email address from the user. The contact information may be entered by the user at the entrance, may be accessed from a user account associated with a user identifier scanned at the entrance, or otherwise available based on the user identifier. The contact information may be used to provide receipts or order summary information for the shopping trip.

Additionally, the process600may include one or more risk assessment steps, such as described above with respect toFIG.5. The AC service provider may perform a risk assessment in response to the indication that the user is at the facility and may condition entrance to the facility on the risk score. In some examples, the risk score may be conveyed with the user identifier to the retailer, who may perform a separate risk analysis of the user, as described above.

While the foregoing is described with respect to the specific examples, it is to be understood that the scope of the description is not limited to these specific examples. Since other modifications and changes varied to fit particular operating requirements and environments will be apparent to those skilled in the art, the invention is not considered limited to the example chosen for purposes of disclosure, and covers all changes and modifications which do not constitute departures from the true spirit and scope of this invention.

Although the application describes embodiments having specific structural features and/or methodological acts, it is to be understood that the claims are not necessarily limited to the specific features or acts described. Rather, the specific features and acts are merely illustrative some embodiments that fall within the scope of the claims.