TRANSACTION SESSION FOR WEARABLE PROCESSING DEVICE

A transaction session is established directly or indirectly between a wearable processing device and a cloud-based server of a store. During the session, items are recognized by placing the items in a field-of-view of a front-facing camera of the device. Item recognition does not require item barcode identification. A depth sensor associated with the camera creates a three-dimensional mapping of a given item. The mapping and image features are processed to uniquely identify the item even when the item is associated with a same category of items. Customer input during the session can be achieved through gestures (hand, eyes, head, fingers, etc.) and/or voice commands. The customer input is translated and mapped into transaction interface commands/options and processed during the session to select items, delete items, view a transaction receipt, identify a quantity of items, obtain item details for a given item, etc.

BACKGROUND

As technology advances and consumers embrace it in all aspects of their lives, many retailers have taken advantage of this phenomenon with technology offerings that make it easier for their customers to interact with the retailers and transaction with the retailers.

For example, most retail stores now have Self-Service Checkouts (SCOs) where customers of the stores can self-checkout. Customers utilizing SCOs typically have to pick items from the store shelves, carry them to the SCOs, scan the item barcodes at the SCOs, and pay for the goods. The problem with this approach is that the customers have to handle the items multiple times before checking out (pick from shelves, place in cart, remove from cart, scan at the SCOs, bag the items, etc.).

Consequently, many retailers now offer mobile applications accessible from their customer phones that permit their customers to scan item barcodes as they shop in the stores and place scanned items in bags of a cart or a basket. Scan as you shop applications have streamlined the customer experience within the stores. However, these applications still have a number of problems, which have limited customer adoption of this technology.

The scan as you shop applications require the user to actively operate their mobile phones as they shop. This creates a usability issue because one customer hand has to hold a picked item while the other customer hand has to operate the phone and interact with the mobile application during shopping. Customers struggle to carry their personal belongings, deal with small children and/or push a cart (or carry a basket) while operating scan as you shop applications on their phones. Many customers find this experience too cumbersome and difficult.

Additionally, most scan as you shop applications require the customers to properly orient a held item so that its barcode is placed in the field of view of the phone's camera for properly identifying and recording an item identifier for the item.

As a result, there is a need for improved scan as you shop applications, workflows, and interfaces.

SUMMARY

In various embodiments, a system and methods for transaction sessions with wearable processing devices are presented.

According to an embodiment, a method for managing a transaction session with a wearable processing device is provided. A connection to a cloud-based store server is requested during a shopping trip of a customer to a store. A wireless transaction session is established with the cloud-based store server based on the requested connection. Item images placed within a field of view of a front-facing camera of a wearable processing device worn by the customer are captured during the transaction session. Item identifiers and item information are obtained for items associated with the item images based at least on the item images. Gestures of the customer are translated during the transaction session into customer selections, customer options, and customer-initiated commands associated with a virtual shopping cart maintained by the cloud-based store server during the transaction session; the virtual shopping cart comprises the item identifiers and item information.

DETAILED DESCRIPTION

FIG.1is a diagram of a system/platform100for managing transaction sessions with wearable processing devices, according to an example embodiment. It is to be noted that the components are shown schematically in greatly simplified form, with only those components relevant to understanding of the embodiments being illustrated.

Furthermore, the various components (that are identified in system/platform100) are illustrated and the arrangement of the components are presented for purposes of illustration only. It is to be noted that other arrangements with more or less components are possible without departing from the teachings of conducting, operating, and managing transaction sessions via a wearable processing device, presented herein and below.

System/platform100(herein after just “system100”) provides a processing environment by which a customer engages in a transaction session with a retail store's server via an improved and seamless interface associated with a wearable processing device (such as glasses/goggles/headsets) during a shopping trip at the store. The wearable processing device permits the customer to view their surroundings unobstructed through the lenses of the wearable processing device. A camera of the device captures what the customer is looking at within the physical surroundings during the session and permits seamless and handsfree identification of items being handled by the customer. Within a portion of one or both lenses, superimposed item and transaction information is rendered for viewing, selection, acceptance, and/or modification by the customer using Augmented Reality (AR) techniques.

The wearable processing device may have its own independent wireless network connection with the store's server or may utilize a wireless connection to the store's server via a second device in possession of or in proximity to the customer. Item recognition of items captured within the field-of-view of the lenses do not have to include images of the items' barcodes. Moreover, machine-learning models may be continuously trained to identify hard to recognize items, such that the item recognition is continuously improving on item recognition accuracy.

The customer may complete the session and pay for items of the customer's virtually maintained cart (with the store's server) without the customer having to operate any additional device during the shopping trip or during the transaction session, such as the customer's phone, a Self-Service Terminal (SST) during self-checkouts, and/or a Point-Of-Sale (POS) terminal during cashier-assisted checkouts.

A variety of embodiments and operational features of system100are now discussed with reference toFIG.1.

As used herein, the terms “user,” “consumer,” and/or “customer,” may be used interchangeably and synonymously herein and below. This refers to an individual who is wearing a wearable processing device and is engaged in a transaction session with a store's server during a customer's shopping trip with a store.

The system100comprises a cloud/server110, a wearable processing device120, and a user-mobile devices130(optional configuration of system100).

Cloud/server110comprises at least one processor111and a non-transitory computer-readable storage medium112. Medium112comprises executable instructions for a transaction manager113and an item recognizer114. The executable instructions when provided to processor111from medium112cause the processor111to perform operations discussed herein and below with respect to113-114.

Wearable processing device120comprises at least one processor121, at least one camera122, an accelerometer123, a wireless transceiver124, zero or more sensors125, and a non-transitory computer-readable storage medium126. Medium126comprises executable instructions for a Human Input Device (HID) manager127, a session agent128, and a transaction manager129. The executable instructions when provided to processor121from medium126cause the processor121to perform operations discussed herein and below with respect to127-129.

Mobile device130(optional) comprises at least one processor131, one or more sensors132, wireless transceiver133, and a non-transitory computer-readable storage medium134. Medium134comprises executable instructions for a transaction manager135and a session manager136. The executable instructions when provided to processor131from medium134cause the processor131to perform operations discussed herein and below with respect to135and136.

System100initially detects a connection request to cloud/server110for a transaction session associated with a customer's shopping trip to a store of a given retailer. Detection can occur or can be requested in a variety of manners.

For example, a Global Positioning Satellite (GPS) sensor125may report a current physical location of wearable processing device120, which transaction manager129maps to a given store's location. The lenses (AR display) of device120superimposes a question overlaid on the customer's view of their surroundings as a notification stating, “Start a Shopping Trip with Store X?” HID manager125monitors a front facing camera122, a rear facing camera122, the accelerometer123, and/or the sensors125(such as a microphone, pressure sensors on the rims of glasses120, and/or a button located on the rims of the glasses) for customer-provided input. An affirmative response causes session agent128to activate wireless transceiver124and establish a wireless connection140between transaction manager129and transaction manager113of the store's cloud/server110.

In another example, glasses120maintains a continuous wireless connection160using wireless transceiver124with mobile device130via wireless transceiver133of mobile device130. (Note mobile device130can be in a pocket, purse, of bag of the customer and does not have to be actively operated by the customer for this continuous wireless connection160.) Mobile device130uses a GPS sensor132to detect a mobile device's physical location being associated with a physical location of the store. Session manager136establishes a wireless connection150(cellular, Wi-Fi, etc.) between transaction manager113of the cloud/server110to transaction manager135of mobile device130. Based on an existing connection160, session manager136sends the notification to session agent128for presenting the question as to whether the customer wants to start a shopping trip (as discussed above in the previous example) on the AR lenses/display of glasses120to the customer. An affirmative response from the customer as detected by HID manager126causes session agent128to interact with transaction manager135during the transaction session being conducted between transaction manager135and transaction manager113over wireless connection150. In this embodiment, session manager136acts as a proxy between a user-facing interface of transaction manager135of mobile device130and HID manager125via session agent128over wireless connection160.

In still another embodiment, a Quick Response (QR) code placed on a display at the front of the store or placed on a label of a surface proximate to the store's entrance is captured by the customer via a front-facing camera122of glasses120or via a camera (not illustrated) of mobile device130. The session agent128or session manager136establishes connection140or connection150between glasses120or device130based on reading the QR code and the notification presented on the AR lenses/display of glasses120for an affirmative response by the customer in manners similar to what were discussed above in the previous two examples (first example, glasses120directly interacts with transaction manager113of cloud/server110for connection140during the session using transaction manager129; second example, glasses120indirectly interacts with transaction manager113of cloud/server110over connection160via transaction manager135of device130and device130directly interacts with transaction manager113via transaction manager135).

Once an affirmative response to the shopping trip notification is received from the customer via HID manager127over connection140or over both connections160and150, a transaction session for a customer's shopping trip with a store is established between transaction manager129and transaction manager113(direct connected session) or between transaction manager135and transaction manager113(indirect connected session).

Session agent128begins monitoring HID manager127events for user-directed input and begins monitoring images appearing within a field of view of front facing camera122for item images of items after the transaction session is established. Interaction for a direct connected session may cause transaction manager129to download and to process cloud/server-hosted Application Programming Interfaces (APIs) to transaction manager113. The APIs may already be available and in use by transaction manager135for an indirect connected session.

During the transaction session, when a customer desires to purchase an item, the customer picks the item off a shelf and places it within the field of view of front facing camera122.

In a directly connected session, session agent128samples features of the item image and sends the features over connection140to item recognizer114. Item recognizer114scores the features and matches to candidate items, when a score of a candidate item is above a predefined threshold, item recognizer114returns the candidate item's code to transaction manager113and transaction manager113provides to transaction manager129. Transaction manager129displays the item name and image within the AR lenses/display of glasses120to the customer. When none of the candidate items for the features of the item image are above the predefined threshold, a top N number of item identifiers for the candidate items are provided by recognizer114to manager113. Manager113provides to manager129and the candidate item images and descriptions are rendered within the AR lenses/display for selection by the customer. The customer may select one of the candidate item images through input identified by HID manager127. For example, a thumbs up placed in front of a displayed candidate item image is interpreted as a user selection for the item code associated with the candidate item image. Each candidate item image may be presented sequentially to the customer within the AR lenses/display and browsed by the customer swiping a hand to the right or left within the field of view of front facing camera122to see a previously presented candidate item image (swipe right) or see a next candidate item image (swipe left).

Some items may not be able to be completely recognized based on the captured item image. For example, different flavors of a brand-name drink, in such cases item recognizer114provides the different subcategories (flavors) for manager113and the user is presented the choices via the AR lenses/display for selection via HID manager127.

When an item is recorded for the transaction session through automatic recognition or through user-selection via HID manager127, the item code for the selected item is stored in a virtual cart being maintained for the shopping trip and the transaction session by transaction manager113for the customer. Additionally, when an item code is added a corner of the AR lenses/display is updated to show a current running price total and a total quantity of items associated with the virtual cart, such that the customer is always aware of a current price of the items recorded in their virtual shopping cart during the transaction session.

In an indirectly connected session, the above-described workflow of item recognition, selection, and updating of the virtual shopping cart is similar; however, the workflow is processed over connection160(glasses120to mobile device130) and connection150(mobile device130to cloud/server110). Here, session agent128interacts with session manager136to proxy a user-facing interface to transaction manager135on the glasses120using HID manager127to capture and process user inputs.

In an embodiment, glasses120includes a depth sensor125, which allows the features of an item image taken to be taken in three dimensions (3D) to capture an item's depth, height, and width. A 3D mapping produced by camera122can be derived using the depth sensor120with the 3D mapping provided as the item features to item recognizer114. This allows item recognizer114to distinguish between different sized items of the same type, such as an 8-ounce drink versus a 12-ounce drink; a smaller sized piece of fruit versus a larger sized piece of the same fruit. Moreover, this provides fine grain item recognition which has heretofore not been possible in the industry.

In an embodiment, when a candidate item's image is captured by front facing camera122and that image includes a completed item barcode, item recognizer114can quickly identify the item code, item image, and item pricing/details and return to manager113, which is then used to update the virtual cart causing the update to be reflected in the cart summary presented to the customer in a corner of the AR lenses/display of glasses120.

HID manager127can map a variety of user gestures and other user provided inputs during the transaction session to options associated with a user-facing interface of transaction manager129and transaction manager135. For example, when camera122detects a customer hand gesture to the side of one of the AR lenses, HID manager127maps this gesture to a user-facing interface option associated with a request by the customer to view the total list of items scanned and recorded (current transaction receipt) in the virtual cart for the session. A hand gesture that moves from a center of the two lenses to the side is mapped to a user-interface request/option to minimize the transaction receipt back to a transaction total and total quantity of items in a corner of the AR lenses/display of glasses120. To scroll the transaction items in the receipt, the customer moves their hands upward or downward. The displayed items are numbered or labeled when presented within the AR lenses/display, a microphone sensor125may capture the customer speech that references a specific number in the list and the customer and see more details or delete the item associated with the specific number audibly (which HID manager127maps the audible speech into a text command recognized by the user-facing interface of manager129and/or manager135). A quantity of any given item can be increased by the customer placing a same item back in the field of view of the AR lenses/display for each number of the items or the customer can place the item in the field of view and audibly state quantity is N.

In an embodiment, HID manager127tracks events from sensors125and accelerometer123of glasses120and maps predefined gestures and/or audible spoken words into commands recognized by the user-facing interface of manager129and manager135. The gestures can be a head nod up, a head node down, head swipes side to side, a head tilt up down or side to side, which are detected by accelerometer123; eye movements of the customer detected in images of a front-facing camera122of glasses120while eyes of the customer are being tracked during the transaction session; spoken words detect recorded as audio by a microphone sensor125; finger taps on a surface of the rims of the glasses120detected by a pressure sensor125; pressed buttons manufactured in the rims of the glasses120and associated with a button sensor125, etc.

The customer can finish their session and corresponding their shopping trip through payment for the virtual shopping cart. Payment workflows can occur in multiple manners. Payment processing can be initiated by the customer capturing the field of view of front facing camera122a QR code being displayed on an SST, this causes the QR code to be decoded and an SST identifier for the SST communicated to transaction manager113. Manager113identifies which device that communicated the SST identifier, glasses120or device130and identifies the virtual cart. Using the SST identifier, the SST is placed in payment mode for the transaction details associated with the customer's virtual cart and payment can be received in a number of manners. The total transaction receipt is displayed on a display of the SST and displayed within the AR lenses/display of glasses120. The customer may nod or may audibly say yes and a registered payment method for the customer is processed for the payment. In another case, the customer may simply say ready for payment audibly, which HID manager127translated into a pay now option associated with the user-facing interface of manager129and135. If the customer does not have a registered payment method in their profile or if the customer wants to use a different payment method from an existing registered payment method, the customer may place a payment card in the field of view of the AR lenses/display and the image of the payment card is processed using Optical Character Recognition (OCR) and provided to transaction manager113as a new payment method.

In an embodiment of an indirect connected session, session manager136keeps the user-facing interface of transaction manager135in synchronization with actions taken by the user during the session and the current state of the session. This allows the customer/user to switch back and forth between providing gestures or speech as input via glasses120and touch-based input for the user-facing interface of transaction manager135on device130. The customer/user can perform or provide commands that the user may be unfamiliar with through glasses120using the touch-based input for the user-facing interface of manager135on device130. It is noted that any available command to the user-facing interface of manager135is available to the user through the gestures or audio inputs on glasses120, such that this option is available as an alternative input mechanism to the user but does not have to be used by the user.

In an embodiment, item recognizer114utilizes a trained machine-learning model for item recognition. During a training session, each item presented to the front facing camera122of glasses120is labeled with its item code. The model configures itself during the training session to predict item codes when presented images without the corresponding labels. Furthermore, when options for candidate items are presented to customers during shopping sessions, the actual item codes selected by the customers are noted with the corresponding features of the item images and used in subsequent training sessions of the model to continuously improve item recognition accuracy of the model. In cases where the item barcode is visible in the images, the item barcode may be provided with the item image for improving the accuracy of the model.

In an embodiment, the actual item images captured by camera122for items are provided to item recognizer114for item recognition rather than select features. In this embodiment, the actual item images may be modified for resolution to reduce the size of the item images before being provided to item recognizer114.

In an embodiment, item recognizer114may be processed on glasses120and/or mobile device130.

In an embodiment, glasses120include a high-resolution camera122of 12 megapixels or more and glasses120include a depth sensor125associated with camera122. Features of the item images captured by camera122and 3D mapping of the item image captured by depth sensor125are processed on glasses120and/or mobile device130to derive a set of features (image and dimensions), which are provided to item recognizer114for item recognition.

In an embodiment, an AR generated mapping of the store in connection with a planogram are used to provide location information for where each item image is captured within the store and the current location as well as the planogram are provided as input with the item images to the item recognizer114for item recognition.

It is noted that although the wearable processing device120was discussed as glasses120, the wearable processing device120may be goggles120or an AR headset120.

The above-referenced embodiments and other embodiments are now discussed withinFIGS.2-3.

FIG.2is a diagram of a method200for managing a transaction session via a wearable processing device, according to an example embodiment. The software module(s) that implements the method200is referred to as a “transaction session assistant.” The transaction session assistant is implemented as executable instructions programmed and residing within memory and/or a non-transitory computer-readable (processor-readable) storage medium and executed by one or more processors of one or more devices. The processor(s) of the device that executes the transaction session assistant are specifically configured and programmed to process the transaction session assistant. The transaction session assistant may have access to one or more network connections during its processing. The network connections can be wired, wireless, or a combination of wired and wireless.

In an embodiment, the device that executes the transaction session assistant is wearable processing device110. In an embodiment, the devices that execute the transaction session assistant are wearable processing device120and mobile device130.

In an embodiment, the transaction session assistant is all or some combination of127,128,129,135,136, and/or114(implemented on device120or device130).

At210, the transaction session assistant requests a connection to a cloud-based store server110during a shopping trip of a customer to a store.

In an embodiment, at211, the transaction session assistant makes the connection based on a current location of a wearable processing device120or a current location of a mobile device130paired to the wearable processing device120when the current location corresponds to a known store location for the store.

At220, the transaction session assistant establishes a wireless transaction session to the cloud-based server110based on the request of210.

In an embodiment, at221, the transaction session assistant establishes the transaction session as a direct session between the wearable processing device120and the cloud-based server110(using wireless connection140).

In an embodiment, at222, the transaction session assistant establishes the transaction session as an indirect session between the wearable processing device120and the cloud-based server110that utilizes a second connection between the wearable processing device110and the mobile device130(using wireless connection160with the mobile device130using wireless connection150to the cloud-based store server110).

At230, the transaction session assistant captures item images placed within a field-of-view of a front-facing camera122of the wearable processing device120during the transaction session.

At240, the transaction session assistant obtains item identifiers and item information for the items associated with the item images based on at least the item images.

In an embodiment, at241, the transaction session assistant extract features for the item images, provides the features to the cloud-based store server110and receives the item identifiers back from the cloud-based store server110.

In an embodiment of241and at242, the transaction session assistant obtains a 3D mapping from a depth sensor125associated with the front-facing camera122for each item image, provides the 3D mapping with the item features to the cloud-based store server110, and receives the item identifiers back from the cloud-based store server110.

At250, the transaction session assistant translates gestures of the customer during the transaction session into customer selections, customer options, and customer-initiated commands associated with a virtual shopping cart maintained by the cloud-based store server110during the transaction session. The virtual shopping cart comprises the item identifiers and item information for each of the items. The gestures can be based on hand gestures made in front of the front-facing camera122, eye gestures made to a rear-facing camera122, or head tilts/nods/swings detected by an accelerometer123of the wearable processing device120.

In an embodiment, at251, the transaction session assistant maps the gestures to interface selections, interface options, and interface commands of a transaction user interface associated with the cloud-based store server110that the cloud-based store server110processes to update and to maintain the virtual shopping cart during the transaction session.

In an embodiment, at252, the transaction session assistant further translates audio spoken by the customer (captured by a microphone sensor125of the wearable processing device120) into text selections, text options, and text commands associated with the virtual shopping cart and processed by the cloud-based store server110.

In an embodiment, at260, the transaction session assistant is processed only on the wearable processing device120or the transaction session assistant is processed on a combination of the wearable processing device120and a mobile device130in possession of or in proximity to the customer during the transaction session.

In an embodiment, at270, the transaction session assistant identifies an image of a code captured within the field-of-view of the front-facing camera122. In an embodiment, the code is a QR code displayed on a display of a payment terminal (SST). In response to the captured code, payment processing for the items of the virtual shopping cart is initiated with the virtual shopping cart.

In an embodiment of270and at271, the transaction session assistant identifies an authorization gesture or an audible command indicating the customer authorizes applying a registered payment method to complete the payment processing and the transaction session assistant provides an authorization to the cloud-based store server110to perform the payment processing with the registered payment method and to end the transaction session completing the shopping trip of the customer with the store.

In an embodiment of270and at272, the transaction session assistant identifies an authorization gesture or an audible command indicating the customer is ready for payment processing. The transaction session assistant captures an image of a payment card placed in the field-of-view of the front-facing camera122and transaction the image into payment details. The transaction session assistant provides the payment details to the cloud-based store server110to perform the payment processing using the payment details. In another case, the image of the payment card is provided by the transaction session assistant to the cloud-based store server110and the cloud-based store server translates the image into the payment details and performs the payment processing with the payment details.

FIG.3is a diagram of another method300for operating, conducting, and managing a transaction session with a wearable processing device, according to an example embodiment. The software module(s) that implements the method300is referred to as a “glasses shopping manager.” The glasses shopping manager is implemented as executable instructions programmed and residing within memory and/or a non-transitory computer-readable (processor-readable) storage medium and executed by one or more processors of a device. The processors that execute the glasses shopping manager are specifically configured and programmed for processing the glasses shopping manager. The glasses shopping manager may have access to one or more network connections during its processing. The network connections can be wired, wireless, or a combination of wired and wireless.

In an embodiment, the device that executes the glasses shopping manager is device110. In an embodiment, the device that executes the glasses shopping manager is a combination of devices comprising110,120, and optionally130. In an embodiment device120is glasses, goggles, or an AR headset.

In an embodiment, the glasses shopping manager is some combination or all of113,114,127,128,129,135,136, and/or method200.

The glasses shopping manager presents another and, in some ways, an enhanced processing perspective from that which was shown above for system100and/or method200.

At310, the glasses shopping manager establishes a shopping session with a wearable processing device120(hereinafter AR-enabled glasses120). The AR-enabled glasses120are worn by the customer in a store during a shopping trip. Lenses or a display of the AR-enabled glasses120permit the customer to view their physical surroundings in the store along with AR information rendered as an overlay of the physical surroundings.

At320, the glasses shopping manager maintains a virtual shopping cart for the shopping session.

At330, the glasses shopping manager receives item features for item images captured by a front-facing camera122of the AR-enabled glasses120during the shopping session.

In an embodiment, at341, the glasses shopping manager receives 3D measurements for each item with the corresponding item features; the 3D measurements captured by a depth sensor125associated with the front-facing camera122of the AR-enabled glasses120.

At340, the glasses shopping manager resolves item identifiers for the items based on at least the item features during the shopping session.

In an embodiment of331and340, at341, the glasses shopping manager scores the 3D measurements and the item features, and the glasses shopping manager matches scores produced against candidate scores for candidate item identifiers to determine the item identifiers.

In an embodiment, at342, the glasses shopping manager provides the item features to a trained machine-learning model (such as item recognizer114) and receives the item identifiers as output from the machine-learning model.

In an embodiment, at343, the glasses shopping manager identifies a candidate list of item identifiers for at least one of the item images based on the corresponding item features. The glasses shopping manager provides the candidate item images and candidate item information for the candidate list to the AR-enabled glasses120and receives a gesture-selected option made by the customer to resolve particular item identifiers for each of the corresponding item images, which were unable to be initially determined by the glasses shopping manager.

At350, the glasses shopping manager adds the item identifiers and corresponding item details/information to the virtual shopping cart during the shopping session.

In an embodiment, at351, the glasses shopping manager provides a summary, a running price total, and a running item quantity total for the virtual shopping cart to the AR-enabled glasses120for presentation in an AR-enabled display or AR-enabled lenses of the AR-enabled glasses120during the shopping session.

At360, the glasses shopping manager modifies the item identifiers or a quantity total for a given item identifier within the virtual shopping cart based on gesture-based or audio-based input received from the customer through the AR-enabled glasses120during the shopping session.

At370, the glasses shopping manager processes a payment for the items of the virtual shopping cart to conclude the shopping session based on a payment option communicated by the customer through the AR-enabled glasses120.