Motion based account recognition

A user establishes an account with an account management system and downloads a service application on a user computing device associated with the user. The user enters a service provider location and signs into a service application. A service camera device captures a video feed of one or more users within a visual field of the service camera device and a service computing device compares movement data received from user computing devices at the service provider location against movement of detected objects in the video feed to identify one or more users in the video feed. After identifying the one or more users within the video feed of the service camera device, in response to a particular user initiating a service request at the service computing device, the service computing device identifies the particular user at the service computing device as being within a service area of the video feed.

TECHNICAL FIELD

The present disclosure relates to identifying user accounts for use in service requests based on facial recognition of users and correlating motion data received from user computing devices to a video feed.

BACKGROUND

When users are provided with services at a service provider location, many methods of conducting service requests are available. Consumers may use many different cards or accounts. The user account identifiers and other data represented by the cards may be communicated to the service provider system via magnetic stripes, near field communication technologies involving user computing devices, and other suitable mechanisms.

Current applications for conducting service requests at a service provider system location do not provide the opportunity for the user to make a hands-free service request and instead require the user to interact with a user computing device. Additionally, current applications require the user to perform actions to identify himself or herself by providing user account identifiers or other data to the service provider system.

SUMMARY

Techniques herein provide computer-implemented methods to conduct a hands-free service request with facial recognition and movement recognition. In an example, a service provider system registers with an account management system. A user establishes an account with the account management system and transmits an image of himself to the account management system to establish a facial template associated with the user account. A user signs into the service application via the user computing device and enters the service provider system location. The user computing device receives a service beacon device identifier from the service beacon device and transmits the identifier to the account management system. The account management system transmits to the service computing device facial templates corresponding to other users whose user computing devices are within network range of the service beacon device and who are signed in to the service application. The service camera device captures a facial image of the user and the service computing device identifies the user by comparing the facial image captured by the service camera device against the received facial templates. The service computing device also compares movement data received from the account management system for user computing devices at the service provider system location against movement of users recorded by the service camera device to identify one or more users in the visual field of the service camera device. After identifying the one or more users within the visual field of the service camera device, in response to a particular user initiating a service request at the service computing device, the service computing device identifies the particular user based on a proximity to a designated service area in the video feed. The service computing device operator selects an account of the identified user for use in a service request from one or more displayed accounts of the user. The service provider device transmits service request details to the account management system, which generates a service request authorization request.

In certain other example aspects described herein, systems and computer program products to conduct a hands-free service request with facial recognition and movement recognition are provided.

These and other aspects, objects, features, and advantages of the examples will become apparent to those having ordinary skill in the art upon consideration of the following detailed description of illustrated examples.

DETAILED DESCRIPTION OF EXAMPLES

Overview

The examples described herein provide computer-implemented techniques for conducting a hands-free service request with facial recognition and movement recognition.

In an example, a service provider system registers with an account management system. The service provider system installs one or more service beacon devices, one or more service computing devices, and one or more service camera devices at a service provider system location. A user establishes an account with the account management system and downloads a service application on a user computing device associated with the user. In an example, the user transmits an image of himself to the account management system to establish a facial template associated with the user account. The user enters a service provider system location and signs into the service application via the user computing device. The user computing device receives a service beacon device identifier from the service beacon device and transmits the identifier to the account management system. The account management system transmits to the service computing device facial templates corresponding to users whose user computing devices are within network range of the service beacon device and who are signed in to the service application. The service camera device captures facial images of one or more users within a visual field of the service camera device. The service computing device identifies the one or more users based on comparing the facial images captured by the service camera device against the received facial templates. The service camera device also compares movement data received from the account management system for user computing devices at the service provider system location against movement of users recorded by the service camera device to identify one or more users in the visual field of the service camera device. After identifying the one or more users within the visual field of the service camera device and in response to a particular user initiating a service request at the service computing device, the service computing device identifies the particular user at the service computing device. In an example, the service computing device displays one or more accounts of the identified particular user. The service computing device operator selects an account of the identified particular user for use in a service request. The service computing device transmits service request details to the account management system, which generates a service request authorization request to transmit to an issuer system associated with the user account selected for use in the service request. The account management system receives an approval of the service request authorization request and transmits a receipt to the service computing device.

In an example, the service computing device displays one or more accounts of the identified particular user. The service computing device operator selects an account of the identified particular user for use in a service request. The service computing device transmits service request details to the account management system, which generates a service request authorization request to transmit to an issuer system associated with the user account selected for use in the service request. The account management system receives an approval of the service request authorization request and transmits a receipt to the service computing device.

In an example, a service provider system registers with an account management system. A service computing device operator installs a service application on a service computing device. In another example, the service computing device operator installs the service application on a plurality of service computing devices at a service provider system location. A service beacon device receives a beacon identifier code from an account management system. The service beacon device broadcasts the beacon identifier code via wireless communication at the service provider system location. The service computing device operator installs a service camera device at the service provider system location to correspond to the service computing device. The account management system receives a service computing device identifier corresponding to a service computing device and associates it with a corresponding beacon identifier of the service beacon device.

In an example, the user registers with an account management system. For example, the user accesses an account management system website via a user computing device associated with the user. The user registers with the account management system and downloads a service application onto the user computing device. In an example, the account management system establishes a facial template associated with the user account. The account management system creates a facial template associated with the user account based on the received facial image. The account management system deletes the received facial image.

In an example, the user signs in to a service application on the user computing device. The user carries the user computing device within a threshold distance of a service beacon device at the service provider system location. The user computing device receives a service beacon device identifier broadcast by the service beacon device and transmits the received service beacon device identifier and a user account identifier to the account management system. The account management system receives the service beacon device identifier and the user account identifier. The account management system extracts a facial template associated with the user account identifier and identifies a service camera device associated with the service beacon device identifier. The account management system transmits a facial template of the identified user to the service computing device associated with the service beacon device identifier. For example, a facial template associated with the identified user's account is transmitted to the service computing device. The service computing device receives the facial template of the user. The service computing device adds the facial template of the user to a current customer log. The service computing device periodically updates the current customer log based on updates received from the account management system. The service computing device captures video of the user via the service camera device. The service computing device receives the video feed of the service camera device and extracts a facial image of the user from the captured video and generates a facial template from the captured facial image. The service computing device deletes the received video feed and extracted facial image. The service computing device retrieves facial templates from the current customer log. The service camera device compares the generated facial template from the extracted facial image to facial templates from the current customer log. The service camera device is able to identify the user if there is a match between a facial template from the current customer log and the generated facial template. The service camera device is unable to identify the user if there is no match between a facial template from the current customer log and the generated facial template. In an example, if the service camera device is able to identify the user, the service camera device notifies the account management system of the identity of the user. In another example, if the service camera device is able to identify the user, the service camera device notifies the service computing device of the identity of the user. In an example, the service computing device identifies one or more users from the video feed based on facial recognition.

In an example, instead of or in addition to using facial recognition, the service computing device compares the received video feed with movement data of user computing devices received from the account management system to identify users in the video feed. In other examples, the account management system compares the video feed with the movement data to identify users in the video feed. For example, in addition to transmitting a beacon device identifier to the account management system, each user computing device at the service system location transmits movement data logged by the user computing device. In an example, each user computing device continuously or periodically transmits logged movement data to the service computing device via a wireless communication channel if the user is signed in to the service application. In an example, if the user is not signed in to the service application, the user computing device does not transmit logged movement data to the service computing device via the wireless communication channel. In other examples, the user computing device continuously or periodically transmits logged movement data to the account management system via the network when the user is signed in to the service application on the user computing device. Example movement data comprises accelerometer data and/or gyroscopic data of the user computing device, a user computing device identifier, and timestamp data. For example, movement data comprises a velocity of the user computing device, an acceleration of the user computing device, an orientation of the user computing device, a rotation rate of the user computing device, or other data that describes or indicates an orientation, position, or a movement of the user computing device. For example, timestamp data comprises a current time comprising a year, month, date, hour of the day, minute of the hour, second of the minute, millisecond of the second, and/or other appropriate indication or indications of the time at which the timestamp is generated. In an example, the service computing device receives the movement data from user computing devices at the service provider location via the wireless communication channel. The service computing device compares a received video feed with received movement data of user computing devices to identify users in the video feed. In an example, the service computing device receives a continuous video feed from the service computing device comprising video captured of an area of the service provider system location. Example video feed data comprises timestamp data. In an example, the service computing device defines a location of the area captured by the video feed associated with a service request area and determines an identity of the user computing device within the service request area based on a received user computing device identifier. In an example, the video feed comprises multiple service request areas, each service request area associated with a particular service computing device.

In an example, the account management system receives the movement data from each user computing device at the service provider system location via the network. In this example, the account management system receives a video feed from the service computing device and compares the received video feed with the movement data of user computing devices to identify users in the video feed. In this example, the account management system transmits identities of the users identified in the video feed to the service computing device. In another example, the account management system transmits the received movement data to the service computing device at the service provider location. In this other example, the service computing device compares the received video feed with movement data of user computing devices received from the account management system to identify users in the video feed. In an example, the service computing device defines a location of the area captured by the video feed associated with a service request area and determines an identity of the user within the service request area. In an example, the video feed comprises multiple service request areas, each service request area associated with a particular service computing device. In this example, the service computing device or account management system determines an identity of a particular user within each service request area associated with each particular corresponding service computing device.

In an example, the service computing device or the account management system may be trained to identify users in a video feed based on movement data received from user computing devices. For example, a service application on each user computing device at the service provider system location logs data from accelerometers and/or gyroscopes and sends that data via a wireless communication channel to a processing unit of the service computing device. The processing simultaneously records a video feed from a service camera device communicatively coupled to the service computing device. In other examples, the service computing device comprises the service camera device. In an example, the processing unit uses a machine learning model to identify similarities between gyroscopic and accelerometer data and visual data from the video feed received from the service camera device. In an example, the machine learning model identifies similarities between gyroscopic and accelerometer data and visual data from the video feed that corresponds to the same time period as the received gyroscopic and accelerometer data. In this example, the time period may be determined based on time stamp data associated with the received video feed data and the received gyroscopic and accelerometer data for each user computing device. For example, if a user were to pull his or her user computing device out of his or her pocket at a time period A, the processing unit would identify a signature both in the video feed as well as a corresponding signature from the received movement data comprising the gyroscopic and the accelerometer data during the same time period A. The processing unit would associate the changes in the movement data with changes in the video feed data during the time period and would associate a user computing device identifier with a service request area in the video feed for the time period. In an example, the time period comprises a time period that terminates at the current time. In an example, the service request area corresponds to a location in front of the service computing device. The processing unit would identify the user based on the user computing device identifier or receive, from the account management system, an identity of the user in response to transmitting a request for the identity of the user to the account management system along with the user computing device identifier in the service request area.

For example, the machine learning system is structured as follows: received gyroscopic data and accelerometer data for each user computing device at the location is entered into a Long Short Term Memory (“LSTM”) Model and a Deep Neural Network (“DNN”) classifier is used to compare data from the video feed to the gyroscopic data and accelerometer data to detect users within the video feed. In an example, the DNN classifier is able to detect potential users within the video feed based on object recognition. For example, the DNN classifier executes a person detection algorithm on the service computing device to determine where each person in each frame of the video feed is located. In an example, the DNN Classifier determines whether a region of the video feed that corresponds to a potential user is associated with a particular user computing device represented by the received user computing device movement data. In an example, the LSTM Model would be running on the service application on each user computing device at the service provider system location to generate a serializable encoding of gyroscopic signatures and accelerometer signatures that the DNN Classifier can compare with the visual data for each detected person in the video feed to determine a similarity score. In an example, if the similarity score for a particular user computing device identifier is greater than a predetermined threshold, the DNN classifier determines that the user computing device identifier corresponds to the detected person in the video feed.

In other examples, instead of determining the user computing device identifier that corresponds to the detected person in the video feed based on a similarity score being greater than a predetermined threshold, the DNN classifier uses a tiered threshold matching system. In an example tiered threshold matching system, the DNN determines a strong match and a weak match for each detected person in the video feed. The example strong match comprises a higher threshold similarity score than the example weak match so as to maximize precision. The example weak match comprises a threshold similarity score low enough so that an actual present match would be a weak match. In an example, the DNN classifier generates strong and weak matches for all detected objects detected to be persons in the video feed and tracks a unique strong match for a detected object or, if a strong match is not present, a unique weak match for the detected object for all frames of the video feed. Once all detected objects resembling potential persons are tracked, the DNN classifier determines if multiple users are determined to match a single tracked object in any frame of the video feed. Any frames of the video feed for the single tracked object for which the tracked object is assigned to more than one user computing device are excluded from consideration when determining movement of the object in the video feed. In an example, the LSTM Model would be running on the service application on each user computing device at the service provider system location to generate a serializable encoding of gyroscopic signatures and accelerometer signatures that the DNN Classifier can compare with the visual data for each tracked object in the video feed, excluding any frames of the video feed for which the tracked object was assigned to multiple user computing devices, to determine a similarity score. In an example, if the similarity score for a particular user computing device identifier is greater than a predetermined threshold, the DNN classifier determines that the user computing device identifier corresponds to the detected person in the video feed.

The user approaches a service computing device. The service computing device operator totals items of the user for purchase. The service computing device operator asks the user to select a processing option option. The user directs the service computing device operator to initiate a service request via the service application. The service computing device operator selects an option on the service computing device to initiate a service request using the service application. The service computing device identifies the user at the service computing device that is initiating the service request. For example, the service computing device identifies a user within the service request area associated with the service computing device captured by the video feed based on movement data and/or facial recognition. The service computing device transmits a request for account information and an identity of the identified user within the service request area of the video feed to the account management system via the network. The account management system receives the request for account information and the identity of the identified user within the service request area of the video feed. In other examples, the service computing device already comprises account information of the identified user.

The service computing device displays accounts of the identified user. The service computing device operator selects a user account for the service request and confirms the service request with permission of the user. The service computing device transmits service request details to the account management system. For example, the service request details comprise a total amount, a selected user account for use in the service request, an account of the service provider for use in the service request, and other useful or relevant information. The account management system transmits a service request authorization request to an issuer system. For example, the issuer system is associated with the user account selected for use in the service request. The issuer system approves the service request authorization request and transmits a service request authorization approval to the account management system. The account management system transmits a service request receipt to the service computing device.

By using and relying on the methods and systems described herein, the account management system, the service camera device, the service beacon device, the user computing device, and the service computing device enable the user to conduct a service request with the service provider system without the user having to interact with the user computing device or produce identity documents or physical account cards, as required in some current technology. As such, the systems and methods described herein may reduce the inputs required by the user via the user computing device and the inputs required by the service computing device operator to identify the user. Further, by using and relying on the methods and systems described herein, the account management system, the service camera device, the service beacon device, the user computing device, and the service computing device enable the user to conduct a service request with the service provider system without the service computing device having to rely entirely on facial recognition to recognize users. By correlating movement data received from a user computing devices to an object in a video feed identified as users, a service computing device may identify a user in cases where the video feed captured images of the user over a period of time but was unable to capture a facial image of the user that would enable identification via facial recognition.

Example System Architecture

Turning now to the drawings, in which like numerals indicate like (but not necessarily identical) elements throughout the figures, examples are described in detail.

FIG. 1is a block diagram depicting a system100for conducting a hands-free transaction with facial recognition and movement recognition of users101, in accordance with certain examples. As depicted inFIG. 1, the system100includes network computing devices110,130,140, and150that are configured to communicate with one another via one or more networks120. In some embodiments, a user associated with a device must install an application and/or make a feature selection to obtain the benefits of the techniques described herein.

In examples, the network140can include a local area network (“LAN”), a wide area network (“WAN”), an intranet, an Internet, storage area network (“SAN”), personal area network (“PAN”), a metropolitan area network (“MAN”), a wireless local area network (“WLAN”), a virtual private network (“VPN”), a cellular or other mobile communication network, Bluetooth, Bluetooth low energy, NFC, or any combination thereof or any other appropriate architecture or system that facilitates the communication of signals, data, and/or messages. Throughout the discussion of examples, it should be understood that the terms “data” and “information” are used interchangeably herein to refer to text, images, audio, video, or any other form of information that can exist in a computer-based environment.

Each network computing device110,130,140, and150includes a device having a communication module capable of transmitting and receiving data over the network140. For example, each network computing device110,130,140, and150can include a server, desktop computer, laptop computer, tablet computer, a television with one or more processors embedded therein and/or coupled thereto, smart phone, handheld computer, personal digital assistant (“PDA”), or any other wired or wireless, processor-driven device. In the example depicted inFIG. 1, the network computing devices110,130,140,150, and160are operated by users101, service beacon device120operators, service provider point of sale (“POS”) device130operators, service camera device135operators and account management system150, respectively.

An example user computing device110comprises a user interface111, a service application112, an antenna113, a Wi-Fi controller114, a data storage unit115, a camera component116, a web browser117a, a communication application117b, an accelerometer118, and a global positioning system (“GPS”) module119.

In an example, the user interface111enables the user101to interact with the service application112and/or web browser117a. For example, the user interface111may be a touch screen, a voice-based interface, or any other interface that allows the user101to provide input and receive output from an application or module on the user computing device110. In an example, the user101interacts via the user interface111with the service application112and/or web browser117ato configure user101accounts on the account management system150. In another example, the user101interacts via the user interface111with the service application112and/or the web browser117ato enable hands-free payments, if needed.

In an example, the service application112is a program, function, routine, applet, or similar entity that exists on and performs its operations on the user computing device110. In certain examples, the user101must install the service application112and/or make a feature selection on the user computing device110to obtain the benefits of the techniques described herein. In an example, the user101may access the service application112on the user computing device110via the user interface111. In an example, the service application112may be associated with the account management system150. In another example, the service application112may be associated with a service provider system associated with the service beacon device120, the service computing device130and the service camera device135. In yet another example, two service applications112exist, one associated with the service provider system and another associated with the account management system150.

In an example, the antenna113is a means of communication between the user computing device110and a service beacon device120. In an example, a Wi-Fi controller114outputs through the antenna113a radio signal, or listens for radio signals from the service beacon device120. In another example a Bluetooth controller or a near field communication (“NFC”) controller is used. In an example, the Wi-Fi controller114outputs through the antenna113a radio signal, or listens for radio signals from the service beacon device120.

In an example, the Wi-Fi controller114is capable of sending and receiving data, performing authentication and ciphering functions, and directing how the user computing device110will listen for transmissions from the service beacon device120or configuring the user computing device110into various power-save modes according to Wi-Fi-specified procedures. In another example, the user computing device110comprises a Bluetooth controller or an NFC controller capable of performing similar functions. An example Wi-Fi controller114communicates with the service application112and is capable of sending and receiving data over a wireless, Wi-Fi communication channel. In another example, a Bluetooth controller112or NFC controller112performs similar functions as the Wi-Fi controller114using Bluetooth or NFC protocols. In an example, the Wi-Fi controller114activates the antenna113to create a wireless communication channel between the user computing device110and the service beacon device120. The user computing device110communicates with the service beacon device120via the antenna113. In an example, when the user computing device110has been activated, the Wi-Fi controller114polls through the antenna113a radio signal, or listens for radio signals from the service beacon device120.

In an example, the data storage unit115comprises a local or remote data storage structure accessible to the user computing device110suitable for storing information. In an example, the data storage unit115stores encrypted information, such as HTML5 local storage.

In an example, the camera component116may be any module or function of the user computing device110that obtains a digital image. The camera component116may be resident on the user computing device110or in any manner logically connected to the user computing device110. For example, the camera component116may be connected to the user computing device110via the network140. The camera component116may be capable of obtaining individual images or a video scan. Any other suitable image capturing device may be represented by the camera component116.

In an example, the user101can use a communication application117b, such as a web browser117aapplication or a stand-alone application, to view, download, upload, or otherwise access documents or web pages via a distributed network140.

In an example, the web browser117acan enable the user101to interact with web pages using the user computing device110. In an example, the user101may access the user's101account maintained by the account management system150via the web browser117a. In another example, the user101may access the service provider system website via the web browser117a. In certain examples described herein, one or more functions performed by the service application112may also be performed by a web browser117aapplication associated with the account management system150.

In an example, the communication application117bcan interact with web servers or other computing devices connected to the network140, including the user computing device110and a web server of a service provider system.

In certain examples, one or more functions herein described as performed by the service application112may also be performed by a web browser117aapplication, for example, a web browser117aapplication associated with a service provider system website or associated with the account management system150. In certain examples, one or more functions herein described as performed by the service application112may also be performed by the user computing device110operating system. In certain examples, one or more functions herein described as performed via the web browser117amay also be performed via the service application112.

In an example, the accelerometer118is able to detect a magnitude and direction of acceleration or magnitude and direction of velocity of the user computing device110. In an example, the accelerometer118communicates with the service application112or the service application112is otherwise able to receive data from the accelerometer118. In an example, the accelerometer comprises a gyroscope or the user computing device110comprises both an accelerometer118and a gyroscope. In an example, the gyroscope is able to detect an orientation of the user computing device110, a rotation rate of the user computing device110, or other data that describes or indicates an orientation, position, or a movement of the user computing device110. In an example, the gyroscope communicates with the service application112or the service application112is otherwise able to receive data from the gyroscope.

In an example, the GPS module119is able to detect a current location of the user computing device110comprising a longitude and a latitude or other descriptive data describing a location of the user computing device110. In an example, the GPS module119communicates with the service application112or the service application112is otherwise able to receive data from the GPS module119.

An example service beacon device120comprises an antenna121and a Wi-Fi controller122. In an example, a service provider system location comprises one or more service beacon devices120installed at the service provider system location. In an example, each installed service beacon device120is associated by an account management system150with a particular service computing device140installed at the service provider location. For example, the account management system150may comprise a database that correlates service beacon device120identifiers with service computing device140identifiers for associated service camera devices140. For example, a service computing device140identifier may comprise hardware identifier specific to the device such as a serial number or a MAC ID associated with the service computing device140. In another example, a service beacon device120identifier may comprise a hardware identifier specific to the beacon device or an identifier generated by the account management system150and stored in the service beacon device120. An example service beacon device120is programmed to broadcast, emit, or otherwise transmit a particular service beacon device120identifier over a wireless network140to any user computing devices110within a threshold distance required to maintain the wireless network140. For example, the wireless network may comprise a Wi-Fi network140, a Bluetooth network140, an NFC network140, or any other appropriate wireless network140.

In an example, the antenna121is a means of communication between the user computing device110and a service beacon device120. In an example, a Wi-Fi controller122outputs through the antenna121a radio signal, or listens for radio signals from the user computing device110. In another example a Bluetooth controller or a near field communication (“NFC”) controller is used. In an example, the Wi-Fi controller122outputs through the antenna121a radio signal, or listens for radio signals from the payment card device120.

In an example, the Wi-Fi controller122is capable of sending and receiving data, performing authentication and ciphering functions, and directing how service beacon device120will listen for transmissions from the user computing device110or configuring the service beacon device120into various power-save modes according to Wi-Fi-specified procedures. In another example, the service beacon device120comprises a Bluetooth controller or an NFC controller capable of performing similar functions. An example Wi-Fi controller122communicates with the service application112and is capable of sending and receiving data over a wireless, Wi-Fi communication channel. In another example, a Bluetooth controller122or NFC controller122performs similar functions as the Wi-Fi controller122using Bluetooth or NFC protocols. In an example, the Wi-Fi controller122activates the antenna121to create a wireless communication channel between the user computing device110and the service beacon device120. The service beacon device120communicates with the user computing device110via the antenna121. In an example, when the service beacon device120has been activated, the Wi-Fi controller122polls through the antenna121a radio signal, or listens for radio signals from the user computing device110.

An example service computing device130comprises a user interface131, a service application132, a communication application133, a data storage unit134, and a service camera device135, which comprises a camera component136, a processor137, and a data storage unit139.

In an example, the user interface131enables the service provider POS device operator102to interact with the service computing device130. For example, the user interface131may be a touch screen, a voice-based interface, or any other interface that allows the service provider POS device operator102to provide input and receive output from an application or module on the service computing device130. In an example, the service provider POS device operator102interacts via the user interface131with the service application132.

In an example, the service application132is a program, function, routine, applet, or similar entity that exists on and performs its operations on the service computing device130. In certain examples, the service provider point of sale (“POS”) device operator102or other service provider system operator must install the service application132and/or make a feature selection on the service computing device130to obtain the benefits of the techniques described herein. In an example, the service provider POS device operator102may access the service application132on the service computing device130via the user interface131. In an example, the service application132may be associated with the account management system150. In another example, the service application132may be associated with a service provider system associated with the service beacon device120and the service camera device135. In yet another example, two service applications133exist, one associated with the service provider system and another associated with the account management system150.

In an example, the communication application133, such as a web browser application or a stand-alone application, enables an operator of the service computing device130to view, download, upload, or otherwise access documents or web pages via a distributed network140. For example, the communication application133may enable communication over the network140with the account management system150. In an example, the communication application133enables the service computing device130and/or service camera device135to communicate with a server of the account management system150.

In an example, the data storage unit134comprises a local or remote data storage structure accessible to the service computing device130suitable for storing information. In an example, the data storage unit134stores encrypted information, such as HTML5 local storage.

An example service camera device135comprises a camera component136, a processor137, and a data storage unit139.

In an example, the camera component136may be any module or function of the service camera device135that captures a video input or captures a digital image of an external environment of the service camera device135. The camera component136may be resident on the service camera device135or in any manner logically connected to the service camera device135. For example, the camera component136may be connected to the service camera device135via the network140. The camera component136may be capable of obtaining individual images or a video scan. Any other suitable image capturing device may be represented by the camera component136.

In an example, the processor137performs one or more functions described herein as being performed by the service camera device135.

In an example, the data storage unit139comprises a local or remote data storage structure accessible to the service camera device135suitable for storing information. In an example, the data storage unit139stores encrypted information, such as HTML5 local storage.

An example issuer system (not depicted) approves or denies a payment authorization received from the account management system150. For example, a service request may comprise a request to process a payment transaction. In an example, the issuer system communicates with the account management system150over the network140. In an example, the issuer system communicates with an acquirer system to approve a credit authorization and to make payment to the account management system150and/or service provider system. For example, the acquirer system is a third party account management company.

An example account management system150comprises an account management module151, a movement recognition module152, a facial recognition module153, a service processing module154, a data storage unit155, a server157, and a website159.

In an example, the account management module151manages one or more user101accounts. In an example, a user101account may comprise a digital wallet account, an email account, a social networking account, or any other appropriate account associated with the account management system150. In an example, the account management system161communicates with a service application112operating on a user computing device110associated with a user101having a user101account with the account management system150. In an example, the user101enters payment account information into the user101account via the service application112and the account management module151receives the payment account information over the network140and associates the received payment account information with the user101account.

In an example, the movement recognition module152receives the movement data from each user computing device110at the service provider system location via the network140. In this example, the movement recognition module152receives a video feed from the service computing device130and compares the received video feed with the movement data of user computing devices110to identify users in the video feed. In this example, the movement recognition module152transmits identities of the users identified in the video feed to the service computing device130. In another example, the movement recognition module152transmits the received movement data to the service computing device130at the service provider location. In this other example, the service computing device130compares the received video feed with movement data of user computing devices110received from the movement recognition module152to identify users in the video feed. In an example, the service computing device130defines a location of the area captured by the video feed associated with a service request area and determines an identity of the user within the service request area. In an example, the video feed comprises multiple service request areas, each service request area associated with a particular service computing device130. In this example, the service computing device130or movement recognition module152an identity of a particular user within each service request area associated with each particular corresponding service computing device130.

In an example, the facial recognition module153receives a facial image of a user101associated with a user101account submitted by the user101via the user computing device110over the network140. For example, the user101submits the facial image at the time the user101establishes the user101account with the account management system150. In an example, the facial recognition module153generates a facial template based on a received facial image.

In an example, the service processing module154receives transaction details from a service computing device130and a request to initiate a transaction. Example transaction details comprise service provider system account information, a total amount of the transaction, and a user101selection of a user101payment account associated with the user's101account with the account management system150. For example, the user's101account is a digital wallet account comprising one or more payment account information corresponding to one or more respective payment accounts of the user101. In an example, the service processing module154extracts payment account information from the user101account corresponding to the user101selection of the user101payment account received in the transaction details from the service computing device130. In an example, the service processing module154transmits a payment authorization request to an issuer system or other appropriate financial institution associated with the payment account selected by the user101for use in the transaction. An example payment authorization request may comprise service provider system payment account information, user101payment account information, and a total amount of the transaction. In an example, after the issuer system processes the payment authorization request, the service processing module154receives an approval or denial of the payment authorization request from the issuer system over the network140. In an example, the service processing module154transmits a receipt to the service computing device130and/or the user computing device110comprising a summary of the transaction.

In an example, the data storage unit155comprises a local or remote data storage structure accessible to the account management system150suitable for storing information. In an example, the data storage unit155stores encrypted information, such as HTML5 local storage.

In an example, the server157provides content accessible by the user101through the web browser117aand/or service application112on the user computing device110, including but not limited to html documents, images, style sheets, and scripts. In an example embodiment, the server157supports the account management system website159.

In an example embodiment, the website159provides the service application112for download by the user computing device110and the service application132for download by the service computing device130. In an example embodiment, the account management system website159is accessed by the user101via the web browser117a. In another example embodiment, the user101accesses the website159via a service application112resident on the user computing device110.

It will be appreciated that the network connections shown are example and other means of establishing a communications link between the computers and devices can be used. Moreover, those having ordinary skill in the art having the benefit of the present disclosure will appreciate that the user computing device110, the service beacon device120, the service computing device130, the service camera device135, the issuer system, and the account management system150illustrated inFIG. 1can have any of several other suitable computer system configurations. For example, a user computing device110embodied as a mobile phone or handheld computer may or may not include all the components described above.

In examples, the network computing devices and any other computing machines associated with the technology presented herein may be any type of computing machine such as, but not limited to, those discussed in more detail with respect toFIG. 13. Furthermore, any modules associated with any of these computing machines, such as modules described herein or any other modules (scripts, web content, software, firmware, or hardware) associated with the technology presented herein may by any of the modules discussed in more detail with respect toFIG. 13. The computing machines discussed herein may communicate with one another as well as other computer machines or communication systems over one or more networks, such as network140. The network140may include any type of data or communications network, including any of the network technology discussed with respect toFIG. 13.

Example Processes

The example methods illustrated inFIGS. 2-12are described hereinafter with respect to the components of the example operating environment100. The example methods ofFIGS. 2-12may also be performed with other systems and in other environments.

FIG. 2is a block diagram depicting a method200for conducting a hands-free transaction with facial recognition and movement recognition of users101, in accordance with certain examples. The method200is described with reference to the components illustrated inFIG. 1.

In block210, the service provider system registers with the account management system150and installs hardware in a service provider location. The method for registering, by a service provider system, with a account management system150and installing hardware at a service provider system location is described in more detail hereinafter with reference to the method described inFIG. 3.

FIG. 3is a block diagram depicting a method210for registering, by a service provider system, with an account management system150and installing hardware at a service provider system location, in accordance with certain examples. The method210is described with reference to the components illustrated inFIG. 1.

In the examples described herein, the service provider system does not need to install hardware at the example service provider system location in any particular order. The method210describes one example method of installing hardware at the service provider location. However, the service provider system or other system installing the service provider hardware does not need to install the service computing device130, the service camera device135, or the service beacon device120in the order described herein. Further, in certain examples, the service computing device130the service camera device135and, in certain examples, the service computing device130comprises the service beacon device120. In yet other examples, the service computing device130comprises the service camera device135and the service beacon device120.

In block310, a service provider system registers with the account management system150. In an example, an agent of the service provider system accesses a account management system150website159and registers for a service provider account with the account management system150via the website159. In an example, the service provider system adds account information associated with a service provider account to the service provider account managed by the account management system150. For example, the account information comprises payment account information associated with a service provider system payment account. In an example, the service provider system comprises one or more service provider system locations. For example, the service provider system may comprise one or more physical store locations. An example service provider location comprises one or more service computing devices130. In an example, one or more service system operators102operate the one or more service computing devices130at the service provider system location.

In block320, a service provider system operator102installs the service application132on the service computing device130. In another example, the service provider system operator purchases a service computing device130from the account management system150with the service application132pre-installed on the service computing device130. In an example, the service computing device130is able to communicate with the account management system150over a network140. In an example, the service computing device130communicates with the account management system150via the service application132. For example, the service computing device130may be able to transmit transaction details to the account management system150via the service application132over the network140to enable the account management system150to process a transaction. In another example, the service computing device130may be able to receive a receipt from the account management system150that notifies a service provider POS device operator102as to whether a transaction was successful or not.

In block330, the service beacon device120receives a beacon identifier code from the account management system150. In an example, the service provider system receives a beacon identifier from the account management system150and installs or otherwise saves the beacon identifier on the service beacon device120. In an example, a service provider system operator installs the service beacon device120in proximity to a service computing device130. In an example, the service provider system operator installs a plurality of service beacon devices120, each service beacon device120in proximity to one or more associated service computing devices130. In an example, the service beacon device120is able to broadcast a service beacon device120identifier over a wireless medium, wherein one or more user computing devices110located within a threshold proximity to the service beacon device120are able to receive the service beacon device120identifier over the wireless medium. In another example, the service beacon device120is able to establish a local network140connection to one or more user computing devices110located within a threshold proximity to the service beacon device120and the service beacon device120transmits the service beacon device120identifier to the one or more user computing devices110over the established local network140connection. For example, the threshold proximity depends on the network140communication protocol utilized by the service beacon device120.

In block340, the service beacon device120broadcasts the beacon identifier code via wireless communication at the location of the service provider system. For example, the service beacon device120may broadcast, emit, or otherwise transmit data comprising the beacon identifier via Wi-Fi, Bluetooth, Bluetooth low energy (“BLE”), near field communication (“NFC”), or other appropriate communication protocol to one or more user computing devices110located at the service provider system location within a threshold proximity to the service beacon device120. In some examples, the service beacon device120, at a time before transmitting the service beacon device120identifier, is operable to establish a network140connection between the service beacon device120and one or more user computing devices110located at the service provider system location within a threshold proximity to the service beacon device120.

In block350, a service provider system operator102installs the service camera device135at the service provider system location to correspond to the service beacon device120. In an example, both a service camera device135and a service beacon device120are installed in proximity to a particular service computing device130. In another example, a service camera device135and a service beacon device120are installed in proximity to two or more particular service computing devices130. In an example, the service camera device135is oriented to be able to capture video and/or images of a face of a user101standing in front of one or more service computing devices130during the process of checkout. In an example, the service provider system installs a service camera device135that is oriented to capture video and/or images of the face of a user standing in front of a particular service computing device130. In another example, the service provider system installs a service camera device135that is oriented to capture video and/or images of the faces of one or more users101standing within a proximity of a particular plurality of service computing devices130within a range of a field of vision of the camera component136of the service camera device135.

In block360, the account management system150receives a service camera device135identifier and associates it with the corresponding beacon identifier code of the service beacon device120. In an example, the service provider system and/or the account management system150configures the service camera device135so that the service camera device135is able to communicate with the account management system150over the network140. An example service camera device135identifier comprises a hardware identifier, a MAC address, or other useful or relevant identifier associated with the service camera device135. In an example, the account management system150comprises a database comprising service camera device135identifiers, service computing device130identifiers, and associated beacon identifiers for service beacon device120identifiers for a particular service provider system location. In an example, the service camera device135transmits the service beacon device120identifier in addition to the service camera device135identifier to the account management system150. In an example, the service camera device135, during the setup and installation process, may receive the service beacon device120identifier over an appropriate wireless communication channel from the service beacon device120. In another example, the service camera device135, during the setup and installation process, may establish a network140connection with the service beacon device120and receive the service beacon device120identifier over the network140. In another example, the account management system150receives the service camera device135identifier, extracts one or more service beacon device120identifiers from the database, and associates the service camera device135identifier with one or more of the one or more extracted service beacon device120identifiers. In yet another example, the service provider system operator installs the one or more service beacon devices120after installing the one or more service camera devices140. In this example, the account management system150generates a service beacon device identifier to associate with a service camera device135identifier and transmits the generated service beacon device identifier to the service provider system. In this example, the service provider system operator manually configures the service beacon device120to broadcast, emit, or otherwise transmit the service beacon device identifier assigned by the account management system150over a network140.

In certain examples, one or both of the service camera device135and the service beacon device120are components of the service computing device130or are wirelessly or physically connected to the service computing device130and controlled by one or more processors of the service computing device130. In certain examples, certain functions described herein as performed by the service camera device135and/or the service beacon device120may also be performed by the service computing device130.

From block360, the method210proceeds to block220ofFIG. 2.

Returning toFIG. 2, in block220, the user101registers with the account management system150. The method for registering, by a user101, for an account with a account management system150is described in more detail hereinafter with reference to the method220described inFIG. 4.

FIG. 4is a block diagram depicting a method220for registering, by a user101, for an account with an account management system150, in accordance with certain examples. The method220is described with reference to the components illustrated inFIG. 1.

In block410, the user101accesses the account management system website159. For example, the user101accesses the account management system150via the web browser117aof the user computing device110. In another example, the user101may otherwise contact the account management system150to register for a user101account.

In block420, the user101registers with the account management system150. The user101may obtain a user account number, receive the appropriate applications and software to install on the user computing device110, request authorization to participate in hands-free account management, or perform any action required by the account management system150. The user101may utilize the functions of the user computing device110, such as the user interface111and the web browser117a, to register and configure a user101account. In an example, the user101may enter payment account information associated with one or more user101accounts, for example, one or more credit accounts, one or more bank accounts, one or more stored value accounts, and/or other appropriate accounts into the user101account maintained by the account management system150.

In block430, the user101downloads the service application112onto the user computing device110. In an example, the service application112operating on the user computing device110is able to communicate with the account management system150over the network140. In an example, the user101may configure user101account settings or add, delete, or edit payment account information via the service application112. In an example, the user101may select an option to enable or disable the permission of the account management system150to process hands free transactions. For example, a hands free transaction comprises a transaction wherein the user101does not need to interact with the user computing device110or requires minimal user101interaction with the user computing device110to initiate a transaction with the service provider system.

In block440, the account management system150establishes a facial template associated with the user101account. The method for establishing a facial template associated with a user101account is described in more detail hereinafter with reference to the method440described inFIG. 5.

FIG. 5is a block diagram depicting a method440for establishing a facial template associated with a user101account, in accordance with certain examples. The method440is described with reference to the components illustrated inFIG. 1.

In block510, the service application112displays a request for the user101to capture a facial image via the user computing device110. In an example, the service application112displays the request via the user interface111. In an example, the user interface111may display a request that reads, “to enable hands free service requests, we need an image of your face. Would you like submit a facial image now?” In this example, the user101may select an option to take a current picture or may otherwise select a picture stored on the user computing device110.

In block520, the user101selects an option to capture a facial image. For example, the user101actuates an object on the user interface111that reads, “yes, I would like to take a picture now.”

In block530, the service application112activates the camera component116on the user computing device110and the user101captures a facial image of himself or of herself. In an example, the user computing device user interface111may display a live camera feed of the user101to aid the user101in aligning the user's101face to take the facial image. In an example, the service application112may display on the user computing device110a box or other perimeter on the user interface111within which the user101should align his face to take a picture of a required size predetermined by the account management system150. In an example, the user101may actuate an object on the user interface111to capture the image. In this example, in response to the user actuating the object on the user interface111, the camera component116receives a command from the service application112to capture an image of the user101. In another example, the camera component116receives a command from the service application112to capture a plurality of images of the user101as the user101moves the camera around the user's101face. For example, each of the plurality of images of the user101may correspond to a particular pose of the user's101face. An example facial image may comprise a digital image of the face of a user101. In an example, the account management system150may establish guidelines for users101in submitting facial images. For example, the service application112may direct the user101to remove any hats, head coverings, glasses, or other objects or accessories that may occlude regions of the user's101face so that service application160may receive a complete depiction of the user's101face.

In an example, the user computing device110determines if the captured facial image is a valid facial image or an invalid facial image. For example, a valid facial image complies with guidelines predetermined by the account management system150and an invalid facial image does not comply with one or more of the guidelines. For example, if the user computing device110captures a facial image that comprises incorrect dimensions, if part or all of the user's101face is occluded, or if the image is too dark or too bright, the user computing device110rejects the invalid facial image and displays a request directing the user101to capture a subsequent facial image. In this example, the user101captures a subsequent facial image via the user computing device110, and the user computing device110transmits the subsequent facial image to the account management system150via the network140.

In block540, the account management system150receives the facial image. In another example, the account management system150receives a plurality of facial images of the user101. For example, the service application112transmits the one or more facial images of the user101to the account management system150via the network140. In an example, the account management system150associates the received one or more facial images with the user101account. For example, the account management system150is able to identify the user101account to associate with the received one or more images because the user101is currently logged in to the service application112on the user computing device110at the time the one or more facial images are transmitted to the account management system150. In certain examples, the account management system150determines if the received facial image is a valid facial image or an invalid facial image. For example, a valid facial image complies with all guidelines predetermined by the account management system150and an invalid facial image does not comply with one or more of the guidelines. For example, if a user101submits a facial image that comprises incorrect dimensions, if part or all of the user's101face is occluded, or if the image is too dark or too bright, the account management system150rejects the invalid facial image and transmits a request to the user computing device110directing the user101to capture a subsequent facial image to transmit to the account management system150. In this example, the user computing device110receives and displays the request, the user101captures a subsequent facial image via the user computing device110, and the user computing device110transmits the subsequent facial image to the account management system150via the network140.

In block550, the account management system150creates a facial template associated with the user101account based on the received facial image. In another example, the account management system150generates a corresponding facial template for each of a plurality of received facial images associated with the user101account. In an example, the facial template is of a predetermined size, for example, a 128-byte facial template. In an example, the account management system150generates a facial template comprising a computer code representation of the digital facial image. For example, the facial template may describe key features of the facial image of the user101, such as shape, color, line, value, space, form, texture, or other useful or relevant feature of the image or of particular regions of the image. In an example, the facial template is generated by processing the facial image through a convolutional neural network. In an example, the account management system150stores the generated facial template associated with the user101in a data storage unit155associated with the account management system150. For example, the account management system150database may comprise a table or other means by which it correlates each user101account identifier with an associated facial template of the user101.

In another example, after the user computing device110captures one or more facial images of the user101, the user computing device110generates one or more facial templates corresponding to one or more of the one or more captured facial images of the user101. In this example, the user computing device110transmits the one or more generated facial templates to the account management system150over the network140.

In block560, the account management system150deletes the received facial image. For example, the account management system150only uses a facial template comprising a computer code representation of the facial image of the user101. In another example, the account management system150saves the received facial image for future processing. For example, the account management system150, at a later time, updates a facial template generation algorithm and generates an updated facial template corresponding to the saved facial image.

From block560, the method440proceeds to block230inFIG. 2.

Returning to block230, inFIG. 2, the user101enters the service provider system location and signs into the service application112on the user computing device110. In another example the user101signs into the service application112at a time before entering the service provider system location and enters the service provider location carrying the user computing device110signed into the service application112.

In block240, the user device receives a service beacon device120identifier. The method for receiving, by a user computing device110, a service beacon device120identifier broadcast by a service beacon device120is described in more detail hereinafter with reference to the method240described inFIG. 6.

FIG. 6is a block diagram depicting a method240for receiving, by a user computing device110, a service beacon device120identifier broadcast by a service beacon device120, in accordance with certain examples. The method240is described with reference to the components illustrated inFIG. 1.

In block610, the user101signs in to the service application112on the user computing device110. In an example, the user101may have a username and password associated with the user101account maintained by the account management system150. In an example, the user101opens the service application112on the user computing device110and enters a username and/or password via the user interface111to sign in to the service application112. In an example, when the user101is signed in to the service application112, the service application is able to communicate with the account management system150over the network140. In this example, when the user101is not signed in to the service application112, the service application does not communicate with the account management system150even if the a network140connection is available. In an example, the user101may sign out of the service application112at any time by actuating one or more objects on the user interface111of the user computing device110. In an example, after signing in to the service application112, the user101configure one or more user101account settings, add, edit, or delete user101payment account information, and/or change user101preferences. In certain examples, a user101may be required to make a feature selection to obtain the benefits of the techniques described herein. For example, the user101may have to enable one or more user101account settings to enable hands free transactions according to the methods described herein.

In an example, service application112may provide options, data, configurable alerts, and other suitable features to the user101. For example, the service application112may comprise a listing of service provider systems and service provider locations that participate in hands free payment transactions according to one or more of the methods described herein. The listing may be updated periodically from the account management system150. The service application112may notify the user101when the user101is within a configured vicinity of a participating service provider system. The service application112may provide the user101with options to update payment preferences. The service application112may provide the user101with a listing of recent transactions. The service application112may provide any other suitable information to the user101.

In block620, the user101carries the user computing device110within a threshold distance of the service beacon device120at the service provider system location. In an example, the user101enters a location of the service provider system. The user101may enter the service provider location carrying the user computing device110in a pocket or a bag, in the hands of the user101, or in any suitable manner. The location of the service provider system may be a store location, a kiosk location, or any suitable physical location of a service provider system. In another example, a service provider POS operator102may be mobile and arrive at a location of the user101. For example, the service provider system may be a restaurant and the service provider POS device operator102may be a delivery person possessing a portable service computing device130.

In certain examples, the service application112may alert the user101when the user101is in the vicinity of a service provider system that accepts hands-free payments. The alert may be provided via a message on the user computing device110, via an email or a text, or in any suitable manner. In an example, the alert may be based on the location of the user101as determined by a GPS module119resident on the user computing device110. For example, the service application112accesses the GPS data from the GPS module119and compare the GPS location to a list of locations of service provider systems that accept hands free payments. For example, the service application112comprises a list or accesses a list maintained by the account management system150of service provider system locations that accept hands free payments. If a match results from the comparison, then an alert is generated and provided to the user101. The match may result if the user101is within a configured distance of a qualified service provider system location. In an example, the alerts may be configured to alert in any suitable manner. In an example, the alerts may be combined in commercially dense environments or the alerts may be presented individually. In another example, the alerts may be configured to only alert the user101a configured number of times. For example, the alert may be presented three times, but upon a fourth instance, the alert is not presented. The alerts may be presented as a notification with an audible alert, a vibration, a popup alert on the user interface111of the user computing device110, or other suitable alert.

In block630, the user computing device110receives a service beacon device120identifier broadcast by the service beacon device120. The user computing device110recognizes a service beacon device120via wireless communication at the location of the service provider system. The user computing device110may be configured to search for beacons or other wireless signals. In an example, the user computing device110and the service beacon device120establish a Wi-Fi wireless network140connection. In other examples, the user computing device110and the service beacon device120establish a Bluetooth, BLE, NFC, or other appropriate network140connection. Upon entering the range of the signal of the service beacon device120, the user computing device110receives the service beacon device120identifier.

In block640, the user computing device110transmits the received service beacon device120identifier and a user101account identifier to the account management system150. In an example, the user computing device110transmits the data received in the service beacon device120identifier along with a user101account identifier to the account management system150over the network140.

In block650, the account management system150receives the service beacon device120identifier and the user101account identifier. For example, the account management system150receives the service beacon device120identifier and the user101account identifier over the network140. The user computing device110may compare the data from the service beacon device120identifier to a database of service beacon device120identifier data and service camera device identifier data to determine an identity of the service provider system and service camera device135associated with the service beacon device120identifier and/or to verify the authenticity of the beacon.

From block750, the method240proceeds to block250inFIG. 2.

Returning toFIG. 2, in block250, the service computing device130receives a facial template for each user101in range of the service beacon device120. The method for receiving, by a service computing device130, a facial template for each user101in range of the service beacon device120is described in more detail hereinafter with reference to the method250described inFIG. 7.

FIG. 7is a block diagram depicting a method250for receiving, by a service computing device130, a facial template for each user101in range of the service beacon device120, in accordance with certain examples. The method250is described with reference to the components illustrated inFIG. 1.

In block710, the account management system150extracts a facial template associated with the user101account identifier. In an example, the account management system150accesses a database comprising stored facial templates of a plurality of users101with corresponding user101account identifiers for each user101. For example, this database is stored in the data storage unit155.

In block720, the account management system150identifies a service computing device130associated with the service beacon device120identifier. In an example, the account management system150recognizes that the service beacon device120identifier is associated with the account management system150and a particular service computing device130at the service provider system location. In an example, the account management system150recognizes that the service beacon device120identifier is associated with a plurality of service camera devices140installed at a particular service provider location.

In block730, the account management system150transmits the facial template of the identified user101to the service computing device130associated with the service beacon device120identifier. In another example, the account management system150transmits the facial template of the identified user101to a plurality of service computing devices130associated with the service beacon device120identifier. In certain examples, the account management system150receives, in real time, a plurality of transmissions from user computing devices101corresponding to a plurality of users101present at the service provider system location, each transmission comprising a user101account identifier and a retransmitted service beacon device120identifier. In these examples, the account management system150retrieves, in response to receiving each such transmission, a facial template associated with the received user101account identifier and transmits a facial template to one or more service camera devices140at the service provider location associated with the service beacon device120identifier.

In block740, the service computing device130receives the facial template of the user101. In another example, a plurality of service computing devices130receive the facial template of the user101. In yet another example, the service computing device130and/or the plurality of service computing devices130receive one or more additional facial templates from the account management system150corresponding to one or more users101other than the instant user101having user computing devices110in network140connection to a service beacon device according to the method previously described herein. For example, the one or more additional facial templates are received in real time from the account management system150as additional users101other than the instant user101receive the service beacon device120identifier over a wireless communication network140or otherwise establish a network140connection between their user computing devices110and one or more service beacon devices120. For example, the one or more service camera devices140may receive one or more additional facial templates corresponding to one or more additional users101at a time before, at the same time, or after the time at which the service computing device130receives the facial template of the instant user101.

In block750, the service camera computing device130adds the facial template of the user101to a current customer log. In an example, the current customer log is accessible by the service computing device130and by the account management system150. In another example, the current customer log is additionally accessible by the service computing device130and/or an applicable computing device of the service provider system. In an example, the account management system150comprises and maintains the current customer log, which the service computing device130may access via the network140. In another example, the service computing device130comprises and maintains the current customer log, which is accessible to the account management system150by communicating with the user computing device110via the network140.

In block760, the service computing device130periodically updates the current customer log. The method for receiving, by a service camera device135, notification from a account management system150as users101enter or leave a network range of a service beacon device120is described in more detail hereinafter with reference to the method760described inFIG. 8.

FIG. 8is a block diagram depicting a method760for receiving, by a service camera computing device130, notification from a account management system150as users101enter or leave a network range of a service beacon device120, in accordance with certain examples. The method760is described with reference to the components illustrated inFIG. 1.

In block810, the service computing device130is notified by the account management system150as users101signed into a service account enter or leave a network range of the service beacon device120. For example, as previously discussed, when a user101carrying a user computing device110enters a threshold distance from a service beacon device120, the service beacon device120or the user computing device110of the user101are able to detect the other device and establish a wireless network140connection between the two devices at the service provider system location. In this example, the service beacon device120transmits the service beacon device120identifier corresponding to the service beacon device120over the wireless network140to the user computing device110. For example, the service beacon device120transmits the service beacon device120identifier to the user computing device110via a Wi-Fi, Bluetooth, BLE, or NFC wireless communication network140. In this example, the user computing device110retransmits the received service beacon device120identifier to the account management system150along with a user101account identifier identifying the user101.

In block820, the service computing device130determines whether a new user101is in range of the service beacon device120. For example, if the account management system150receives a new user101account identifier in addition to the same service beacon device120identifier, the account management system150may determine that a new user101is in range of the service beacon device120. In this example, the account management system150may infer that the new user101has entered the service provider location based on receipt of the new user101account identifier. In another example, if the account management system150does not receive any new user101account identifiers along with the same service beacon device120identifier within a threshold length of time, the account management system150may determine that no new users101have entered the network140range of the service beacon device120.

If a new user101is in range of the service beacon device120, the method860proceeds to block830inFIG. 8. For example, the account management system150receives a new user101account identifier in addition to the same service beacon device120identifier. In this example, the account management system150infers that the new user101has entered the service provider location based on receipt of the new user101account identifier and the same service beacon device120identifier as previously received from the first user101.

Returning toFIG. 8, in block810, the account management system150extracts a facial template associated with the new user101account identifier. In an example, the account management system150transmits the facial template to the appropriate one or more service computing devices130and the one or more service computing devices130add the new user's101facial template to the current customer log according to the example method previously described in method250inFIG. 7.

Returning toFIG. 8, in block820, if there is a not a new user101in range of the service beacon device120, the method760proceeds to block830. For example, the account management system150does not receive any new user101account identifiers along with the same service beacon device120identifier within a threshold length of time and determines that no new users101have entered the network140range of the service beacon device120.

In block830, the service camera device135determines whether a current user101has moved out of range of the service beacon device120. In an example, the user computing device110continues to receive from the service beacon device120identifier from the service beacon device120and retransmit the service beacon device120identifier along with the user101account identifier to the account management system150. In this example, the user computing device110may periodically transmit information comprising the service beacon device120identifier and user101account identifier to the account management system150as long as the user computing device110continues to detect the service beacon device120and receive the service beacon device120identifier via periodic scans. For example, the user computing device scans for the service beacon device120every five seconds. In another example, the user computing device110may periodically transmit information comprising the service beacon device120identifier and user101account identifier to the account management system150as long as the user computing device110maintains a wireless network140connection with the service beacon device120. For example, the user computing device110may transmit this information to the account management system150at every five seconds. In this example, if the account management system150ceases to receive the information from the user computing device for a predefined number of intervals, the account management system150may determine that the corresponding user101has moved out of range of the service beacon device. In this example, if the account management system150continues to receive the information transmitted by the user computing device110at the expected intervals, the account management system150determines that the user101is still in network140range of the service beacon device120.

If no current user101has moved out of range of the service beacon device120, the method860proceeds to block260inFIG. 2. For example, the account management system150continues to receive the service beacon device120identifier and user101account identifier transmitted by the user computing device110at the expected intervals and determines that the user101is still in network140range of the service beacon device120.

Returning to block260, inFIG. 2, the service computing device130identifies the user101via facial recognition.

Returning toFIG. 8, in block830, if a current user101has moved out of range of the service beacon device120, the method760proceeds to block840.

In block840, the service computing device130receives a notification from the account management system150that a current user101is out of network range of the service beacon device120. In another example, the service computing device130receives a notification from the account management system150that the user computing device110associated with the current user101has stopped sending notifications to the account management system150comprising the service beacon device120identifier. For example, the service computing device130receives the user101account identifier associated with the current user101associated with a user computing device110that is either out of network range or has stopped transmitting notifications comprising the service beacon device120identifier to the account management system150, accesses the current customer log, and finds an entry corresponding to the current user101. For example, the current customer log is maintained by the account management system150and the service computing device130accesses the current customer log over the network140by communicating with the account management system150. In another example, the account management system150does not transmit a notification to the service computing device130that the current user101is out of network range. In this example, the account management system150accesses the current customer log and deletes the facial template of the current user101from the current customer log.

In block850, the service computing device130deletes the facial template of the current user101from the current customer log. For example, the current customer log comprises a table and the service computing device130deletes or requests the deletion of an entry or row corresponding to data associated with the current user101for which the service camera device135received the notification. In another example, the account management system150accesses the current customer log and deletes the facial template of the current user101from the current customer log.

Returning toFIG. 2, in block260, the service computing device130identifies the user101via facial recognition. The method for identifying, via a service computing device130, a user101via facial recognition is described in more detail hereinafter with reference to the method260described inFIG. 9.

FIG. 9is a block diagram depicting a method260for identifying, via a service computing device130, a user101via facial recognition, in accordance with certain examples. The method260is described with reference to the components illustrated inFIG. 1.

In block910, the service computing device130captures video feed of the user101. In an example, in response to receiving a request to identify the user101, the service computing device130activates the camera component136of a service camera device135to begin to capture a video of the surroundings of the service camera device135. In an example, the service camera device135captures a video feed. In another example, the service camera device135continuously captures, but does not record, a video feed of its surroundings. In this example, when the service camera device135receives a request to identify the user101from the account management system150, the service camera device135beings to record the video feed for a threshold amount of time. In an example, the user101may be moving during the period in which the service camera device135records the video feed. In an example, a processor137of the service camera device135extracts a facial image by determining a particular frame of the video feed and area of the instance of the video feed corresponding to the face of the user101.

In block920, the service computing device130extracts a facial image of the user101from the captured video feed. In an example, the service computing device130determines a frame of the captured video to provide an image of the user's101face and extracts the frame of the captured video comprising the facial image of the user101.

In certain other examples, the service computing device130determines a frame of the captured video to provide an image of the faces of a plurality of users101. For example, the frame comprises an image of the face of a first user101, a second user101, and a third user101at different locations in the image. In this example, one service camera device135may capture video of an environment corresponding to an area in the proximity of multiple service computing devices130. In this example, the service computing device130may determine to which particular service computing device130each of the plurality of faces of the corresponding plurality of users101in the extracted image.

In an example, in which a service camera device135is mounted at an angle parallel to a plurality of adjacent service computing devices130, the field of view of the service camera device135may be divided into bounding boxes such that each bounding box is assigned to a particular service computing device130. In this example, since one or more users101may be situated between any two adjacent service computing devices130, the bounding boxes may overlap. In this example, each bounding box is assigned to at most two adjacent service computing devices130. In this example, in which the service camera device135is mounted parallel to a plurality of adjacent service computing devices130, the size of a detected facial image of a user101is directly proportional to the distance the user101is from a service computing device130detected in the image. In this example, the service computing device130assigns each detected facial image to one or two particular service computing devices130based on the position of the facial image relative to the bounding boxes.

In another example, the service camera device135is mounted at an angle that is not parallel to a plurality of adjacent service computing devices130. In this example, the service camera device135may be able to monitor an environment comprising a greater number of adjacent service computing device130than the example in which the service camera device135is mounted parallel to adjacent service computing devices130. In this example, however, the size of a detected facial image is not proportional to the distance between a service computing device130detected in the image and the detected facial image. In this example, the service computing device130assigns each detected facial image to one or two particular service computing devices130based on the position of the facial image relative to the bounding boxes.

In block930, the service computing device130generates a facial template from the captured facial image. In an example, the facial template is of a predetermined size, for example, a 128-byte facial template. In an example, the account management system150generates a facial template comprising a computer code representation of the digital facial image. For example, the facial template may describe key features of the facial image of the user101, such as shape, color, line, value, space, form, texture, or other useful or relevant feature of the image or of particular regions of the image. In another example, the facial template is generated by processing the facial image through a convolutional neural network. In an example, the service computing device130stores the generated facial template in a data storage unit139associated with the service camera device135and the service computing device130. For example, the service computing device130database may comprise a log of facial templates of current customers wherein the service computing device130stores the generated facial template.

In certain other examples, the service camera device135continuously captures a video feed of its surroundings as users101enter and leave the vicinity of one or more service computing devices130over the course of a certain time period. In this example, the service camera device135processor137is able to continuously monitor the incoming video feed to detect faces from extracted frames of the video feed. In this example, the service camera device135, each time the processor137detects the presence of one or more faces in the video feed, the service computing device130extracts a frame of the video feed comprising one or more facial images of one or more corresponding detected faces and creates facial templates based on the extracted one or more facial images. In this example, the service computing device130stores facial templates in the log of facial templates of current customers as they are generated. In this example, as the service computing device130generates a subsequent facial templates, the service computing device130determines whether the generated subsequent facial template is similar to within a threshold compared to any of the facial templates already stored in the log of facial templates of current customers. If the generated subsequent facial template is similar to within a threshold to any of the facial templates already stored in the log, the service computing device130, after associating the facial template to one or two particular service computing devices130based on the position of the associated facial images in the extracted frame of the captured video, adds the facial template to the log of facial templates of current customers. If the generated subsequent facial template is not similar to within a threshold to any facial templates already stored in the log of facial templates of current customers, the service computing device130deletes or otherwise ignores and/or does nothing with the generated facial template. In this example, if the service computing device130determines that certain facial image is no longer in the field of the video feed, the corresponding facial template is deleted from the log of facial templates of current customers.

In block940, the service computing device130deletes the captured video and the extracted facial image. For example, the service computing device130does not store captured images or video. In this example, facial templates generated by the service computing device130comprise computer code representations of facial images of users101. In this example, after generating a facial template or after a threshold time has passed after capturing video or images or extracting an image from a video, the service computing device130deletes any captured or extracted video or images.

In block950, the service computing device130retrieves facial templates from the current customer log. For example, the current customer log comprises facial templates received from the account management system150corresponding to all current users101whose associated user computing devices110are located within a network distance of a service beacon device120.

In block960, the service computing device130compares the generated facial template from captured facial image to facial templates from the current customer log.

In block970, the service computing device130determines whether there is a match between the generated facial template and one of the facial templates from the current customer log. In this example, when the service computing device130determines a match between a first facial template in the log of facial templates of current customers and a second facial template the log of current customers, the service computing device130determines with which service provider one or two service computing devices130the second facial template is associated. In this example, the service computing device130updates the entry of the second facial template in the log of current customers with an annotation assigning the second facial template to a particular one or two service computing devices130. For example, as previously discussed, the facial image upon which the facial template was located within a bounding box of an area of the video or image corresponding to one or two particular service computing devices130. In this example, when the service computing device130generates a facial template for the facial image, the service computing device130associates the generated facial template with the particular one or two service computing devices130. For example, if the facial image was in a bounding box in front of a first service computing device130, the resulting facial template may be associated with the first service computing device130. In another example, if the facial image was in a bounding box between the first service computing device130and a second, adjacent service computing device130, the resulting facial template may be associated with both the first service computing device130and the second service computing device130.

If a facial template from the current customer log matches the generated facial template, the method260proceeds to block980. In another example, if a facial template from the current customer log matches a facial template from the log of facial templates of current customers, the method260proceeds to block980.

In block980, the service computing device130transmits the facial template from the current customer log corresponding to the identified user to the account management system150. For example, the transmitted facial template comprises an annotation assigning the facial template to a particular one or two service computing devices130.

From block980, the method260proceeds to block270inFIG. 2. In this example, the account management system150identifies the user101requesting a service request and a particular service computing device130to which the user101is assigned and may proceed with processing the transaction.

Returning to block970, if none of the facial templates from the current customer log matches the generated facial template, the method260proceeds to block270inFIG. 2. In another example, if none of the facial templates from the current customer log matches any of the facial templates from the log of facial templates of current customers, the method270proceeds to block270inFIG. 2.

Returning toFIG. 2, in block270, the service computing device130identifies the user101via movement recognition. The method for identifying, via a service computing device130, a user101via movement recognition is described in more detail hereinafter with reference to the method270described inFIG. 10. In an example, the method270for identifying, by a service computing device130, users101in a video feed based on movement data received from user computing devices110may be performed instead of or in addition to the method260for identifying users101in a video feed based on facial recognition. In other examples, the account management system150compares the video feed with the movement data to identify users101in the video feed.

FIG. 10is a block diagram depicting a method270for identifying, via a service computing device130, a user101via movement recognition, in accordance with certain examples. The method270is described with reference to the components illustrated inFIG. 1.

In block1010, the user computing device110logs movement data and location data. Example movement data comprises accelerometer118data and/or gyroscopic data of the user computing device110, a user computing device110identifier, and timestamp data. For example, movement data comprises a velocity of the user computing device110, an acceleration of the user computing device110, an orientation of the user computing device110, a rotation rate of the user computing device110, or other data that describes or indicates an orientation, position, or a movement of the user computing device110. For example, timestamp data comprises a current time comprising a year, month, date, hour of the day, minute of the hour, second of the minute, millisecond of the second, and/or other appropriate indication or indications of the time at which the timestamp is generated. Example location data comprises a longitude and latitude logged by the GPS module119of the user computing device110.

In block1020, the user computing device110transmits the logged movement and the location data to the service computing device130when the user101is signed into the service application112. In an example, each user computing device110continuously or periodically transmits logged movement data to the service computing device130via a wireless communication channel if the user is signed in to the service application112. In an example, if the user101is not signed in to the service application112, the user computing device110does not transmit logged movement data to the service computing device130via the wireless communication channel. In other examples, the user computing device110continuously or periodically transmits logged movement data to the account management system150via the network140when the user101is signed in to the service application112on the user computing device110. For example, in addition to transmitting a beacon device identifier to the account management system150, each user computing device110at the service system location transmits movement data logged by the user computing device110to the account management system150.

In block1030, the service computing device130receives the movement data and the location data from the user computing device110. In an example, the service computing device130receives the movement data from user computing devices110at the service provider location via the wireless communication channel. In another example, the service computing device130receives the movement data and the location data from the account management system150, which received the movement data and the location data from each user computing device110.

In block1040, the service computing device130compares the received movement data and the received location data to the current video feed of the service computing device130. For example, the service computing device130compares a received video feed with received movement data of user computing devices110to identify users in the video feed. In an example, the service computing device130receives a continuous video feed from the service camera device135comprising video captured of an area of the service provider system location. Example video feed data comprises timestamp data. In an example, the service computing device130defines a location of the area captured by the video feed associated with a service request area and determines an identity of the user computing device110within the service request area based on a received user computing device110identifier. In an example, the video feed comprises multiple service request areas, each service request area associated with a particular service computing device130. For example, the account management system150receives the movement data from each user computing device110at the service provider system location via the network140. In this example, the account management system150receives a video feed from the service computing device130and compares the received video feed with the movement data of user computing devices110to identify users101in the video feed. In this example, the account management system transmits identities of the users identified in the video feed to the service computing device130. In another example, the account management system150transmits the received movement data to the service computing device130at the service provider location. In this other example, the service computing device130compares the received video feed with movement data of user computing devices110received from the account management system150to identify users101in the video feed. In an example, the service computing device130defines a location of the area captured by the video feed associated with a service request area and determines an identity of the user101within the service request area. In an example, the video feed comprises multiple service request areas, each service request area associated with a particular service computing device130. In this example, the service computing device130or account management system150determines an identity of a particular user101within each service request area associated with each particular corresponding service computing device130.

In an example, the service computing device130or the account management system150may be trained to identify users101in a video feed based on movement data received from user computing devices110. For example, a service application130on each user computing device110at the service provider system location logs data from accelerometers118and/or gyroscopes and sends that data via a wireless communication channel to a processing unit of the service computing device130. The processor simultaneously records a video feed from a service camera device135communicatively coupled to the service computing device130. In other examples, the service computing device130comprises the service camera device135. In an example, the processing unit137uses a machine learning model to identify similarities between gyroscopic and accelerometer118data and visual data from the video feed received from the service camera device135. In an example, the service computing device110uses previous identifications of user computing device110identifiers corresponding to objects video feeds based on correlation of movement data to changes in position of an object in the video feed identified as a potential user101to train the machine learning model to identify similarities between gyroscopic and accelerometer118data and visual data from the video feed. In an example, the machine learning model identifies similarities between gyroscopic and accelerometer118data comprising and visual data from the video feed that corresponds to the same time period as the received gyroscopic and accelerometer118data. In this example, the time period may be determined based on time stamp data associated with the received video feed data and the received gyroscopic and accelerometer118data for each user computing device110. For example, if a user101were to pull his or her user computing device110out of his or her pocket at a time period A, the service computing device130would identify a signature both in the video feed as well as a corresponding signature from the received movement data comprising the gyroscopic and the accelerometer118data during the same time period A. The processing unit would associate the changes in the movement data with changes in the video feed data during the time period and would associate a user computing device110identifier with a service request area in the video feed for the time period. In example, the time period comprises a time period that terminates at the current time. In an example, the service request area corresponds to a location in front of the service computing device130. The processing unit would identify the user101based on the user computing device110identifier or receive, from the account management system150, an identity of the user101in response to transmitting a request for the identity of the user101to the account management system along with the user computing device110identifier in the service request area.

For example, the machine learning system is structured as follows: the service computing device130or account management system150enters received gyroscopic data and accelerometer118data for each user computing device110at the location into a Long Short Term Memory (“LSTM”) Model and a Deep Neural Network (“DNN”) Classifier to compare data from the video feed to the gyroscopic data and accelerometer118data and detect users101within the video feed. In an example, the service computing device130or the account management system150uses the DNN classifier to detect potential users101within the video feed based on object recognition. For example, the DNN classifier executes a person recognition algorithm on the service computing device for each frame of the video feed to determine where each user101is located in the frame. In an example, the service computing device130or the account management system150uses the DNN Classifier to determine whether a region of the video feed that corresponds to a potential user101is associated with a particular user computing device110represented by the received user computing device110movement data. In an example, the LSTM Model operates on the service application112on each user computing device110at the service provider system location to generate a serializable encoding of gyroscopic signatures and accelerometer signatures. The service computing device130or the account management system150uses the DNN Classifier to compare a serializable encoding of gyroscopic signatures and accelerometer118signatures with the visual data for each detected person in the video feed to determine a similarity score. In an example, if the similarity score for a particular user computing device110identifier is greater than a predetermined threshold, the service computing device130or the account management system150uses the DNN classifier to determine that the user computing device110identifier corresponds to the detected person in the video feed.

In other examples, instead of determining the user computing device110identifier that corresponds to the detected person in the video feed based on a similarity score being greater than a predetermined threshold, the DNN classifier uses a tiered threshold matching system. In an example tiered threshold matching system, the DNN determines a strong match and a weak match for each detected person in the video feed. The example strong match comprises a higher threshold similarity score than the example weak match so as to maximize precision. For example, a similarity score is determined based on comparing a signature from the video feed as well as a corresponding signature from the received movement data comprising the gyroscopic and the accelerometer118data during a time period. The example weak match comprises a threshold similarity score low enough so that an actual present match would be a weak match. In an example, the DNN classifier generates strong and weak matches for all detected objects detected to be persons in the video feed and tracks a unique strong match for a detected object. If a strong match is not present, the DNN classifier tracks a unique weak match for the detected object for all frames of the video feed. Once all detected objects resembling potential persons are tracked, the DNN classifier determines if multiple user computing devices110match a single tracked object in any frame of the video feed. Any frames of the video feed for the single tracked object for which the tracked object is assigned to more than one user computing device110are excluded from consideration when determining movement of the object in the video feed.

In block1050, the service computing device130determines if a user101can be identified in the video feed based on comparing the received movement data and the received location data to the current video feed of the service computing device130. In an example, determining if a user101can be identified in the video feed comprises determining whether the user101can be identified in the video feed within a service area of the video feed.

If the service computing device130does not identify a user101in the video feed, the method270returns to block1030. In block1030, the service computing device130receives subsequent movement data and subsequent location data from the user computing device110.

Returning to block1050, if the service computing device130identifies a user101in the video feed, the method270proceeds to block1060. In an example, the service computing device130uses the DNN Classifier to determine whether a region of the video feed that corresponds to a potential user101is associated with the user computing device110represented by the received user computing device110movement data. In an example, the LSTM Model operates on the service application112on the user computing device110at the service provider system location to generate movement data comprising a serializable encoding of gyroscopic signatures and accelerometer signatures. The service computing device130uses the DNN Classifier to compare a serializable encoding of gyroscopic signatures and accelerometer118signatures with the visual data for the object in the video feed comprising the potential user101to determine a similarity score. In an example, if the similarity score for a particular user computing device110identifier is greater than a predetermined threshold, the service computing device130uses the DNN classifier to determine that the user computing device110identifier corresponds to the detected user101in the video feed

In block1060, the service computing device130designates a user's101specific location at the service provider system location based on the correspondence between the video feed and the movement data and the location data. In an example, the user's101specific location at the service provider location is designated as being in a service area if the correspondence of gyroscopic and/or accelerometer118data of the user computing device110with the video feed indicates that the user101is in a service area of the service provider system location. For example, the service area comprises an area within a threshold proximity to the service computing device130within which the user101may initiate a service request.

From block1060, the method270proceeds to block280inFIG. 2.

Returning toFIG. 2, in block280, the user101initiates a service request at the service computing device130. For example, a service request comprises a payment transaction. The method for initiating, by a user101, a service request at a service computing device130is described in more detail hereinafter with reference to the method280described inFIG. 11. In the examples described herein, the user101initiates a service request comprising a “hands free transaction” at the service computing device130. An example hands free transaction does not require any interaction with the user computing device110on the part of the user101. In another example, a hands free transaction requires only minimal interaction with the user computing device110by the user101.

FIG. 11is a block diagram depicting a method280for initiating, by a user101, a transaction at a service computing device130, in accordance with certain examples. The method280is described with reference to the components illustrated inFIG. 1.

In block1110, the user101approaches the service computing device130. In an example, at a time prior to approaching the service computing device130, the user101browses the service provider system location and selects one or more items to purchase. In this example, the user101may collect the one or more items and carry, or otherwise transport via physical basket or shopping cart, the one or more items to the service computing device130.

In block1120, the service computing device130operator102totals the items of the user101for purchase. In an example, the service computing device operator102scans barcodes attached to the one or more items or otherwise enters descriptions and prices associated with the one or more items into the service computing device130. In an example, after scanning or manually entering the items into the service computing device130, the service computing device operator102actuates an object on the user interface131of the service computing device130to order the service computing device130to total the items. In an example, the service computing device130displays, via the user interface131, the total to the user101.

In block1130, the service computing device130operator asks the user101to select a payment option. In an example, the service computing device130displays one or more payment options that the user101may select to use in a transaction. Example payment options may comprise payment via a service application112associated with the account management system150, payment by cash, payment by check, payment by credit card, payment by debit card, and/or any other means of payment that the service provider system can or is willing to accept for payment from the user101. In an example, the one or more payment options are displayed as objects on the user interface131and are selectable by the service system operator102in response to the user101directing the service system operator102to make a selection.

In another example, the account management system150or service camera device135transmits a notification to the service computing device130that the user101is able to initiate a transaction via the service application112. In this example, the service computing device130notifies the service system operator102that the user101is able to participate in a service request using the service application112. In this example, the service system operator102asks the user101to confirm whether the user101wants to initiate a transaction using the service application112.

In block1040, the user101directs the service system operator102to initiate a transaction via the service application112. In an example, in response to receiving a verbal request from the user101to select the service application112as a payment option, the service system operator102actuates an object on the user interface131of the service computing device130corresponding to the service application112payment option.

In block1150, the service system operator102selects an option on the service computing device130to initiate a transaction using the service application112. In an example, the service computing device130displays a confirmation screen after the service system operator102selects an option to initiate a transaction using the service application112. An example confirmation screen may display information summarizing the potential transaction and comprising one or more of a transaction total, a description of the one or more items being purchased by the user101, and a indication that the user101selected the service application112as the method of payment for the transaction. An example confirmation screen may further display options to confirm the transaction or cancel the transaction. In an example, the user101reviews the confirmation screen, determines that the information displayed on the confirmation screen is correct, determines to continue with the transaction, and directs the service system operator102to select the option to confirm the transaction via the user interface131.

In another example, the user101decides to abandon the transaction because the information is incorrect, or because the user101changed his mind and decided not to purchase the items. In yet another example, the confirmation screen further comprises an option to edit the transaction details. In this example, the service system operator102, upon direction of the user101, may select the option to edit the transaction details and may then edit, add, or delete one or more of the items in the transaction or edit payment details or payment methods for the transaction.

In block1160, the service computing device130identifies the user101from the current customer log. For example, at this point, the service computing device130is aware that one or more users101are physically present in the service provider system location based on the current customer log maintained by the account management system150and accessible and editable by the service computing device130. Additionally, the service computing device130is aware that a user101has initiated a transaction with the service application112payment method based on the request to process the transaction received from the service computing device130. However, the service computing device130may not be aware of exactly which user101requested to process the hands free transaction involving the service application112. In an example, the service computing device130continuously monitors a video feed received from the service camera device135and determines, in response to initiating the service request, that the user101is within the service area of the video feed based on comparing the movement data and the accelerometer118data to the video feed.

In block1170, the service computing device130transmits a user computing device110identifier and service request details to an account management system150. For example, in response to receiving an initiation of the service request and in response to identifying the user computing device110in the video feed as being located within the service area of the video feed based on correlating movement data and accelerometer118data received from the user computing device110to data from the video feed, the service computing device130transmits a user computing device110identifier to the account management system150corresponding to the user computing device110detected as being within the service area of the video feed. For example, the service computing device130transmits a user computing device110identifier and service request details to an account management system150via the network140.

In block1180, the account management system150receives the user computing device110identifier and the service request details. For example, the account management system150receives the user computing device110identifier and the service request details via the network110.

In block1190, the account management system150identifies a user101account identifier associated with the user computing device110identifier received from the service computing device130. For example, the account management system150manages the user101account associated the user computing device110.

From block1190, the method280proceeds to block290inFIG. 2.

Returning toFIG. 2, in block290, an account management system150processes a service request. The method for processing, by an account management system150, a service request is described in more detail hereinafter with reference to the method290described inFIG. 12.

FIG. 12is a block diagram depicting a method290for processing, via an account management system150, a service request, in accordance with certain examples. The method290is described with reference to the components illustrated inFIG. 1.

In block1210, the account management system150transmits account information of the identified user101to the service computing device130. For example, the user101account with the account management system150comprises payment account information associated with one or more payment accounts of the user101. An example payment account of the user101comprises a credit account, bank account, stored value account, gift card, service provider system charge account, or other relevant user101account. In an example, the service computing device130does not receive complete payment information for one or more of the user101payment accounts. For example, the payment information may be obfuscated or truncated. For example, the service computing device130may only receive abbreviated payment information sufficient for a user101to identify a payment account via the user interface131.

In block1220, the service computing device130displays accounts of the identified user101. In an example, the service computing device130does not display complete account information of the user101on the user interface131. For example, for a credit card account, the service computing device user interface131may display only the name associated with the credit card, last four digits of the credit card number, and the expiration date.

In block1230, the service computing device operator102selects a user101account for transaction and confirms the transaction with permission of the user101. For example, the user101views the account options directly displayed on the user interface131or listens to the service computing device operator102read the user101payment account options to the user101. In an example, the user101selects a payment account option to use in the transaction. For example, the user101decides to use a credit card account. In an example, the user101communicates a selection of a payment account for use in the transaction to the service computing device operator102. In an example, in response to receiving the user's101indication of the payment account, the service computing device operator102selects the payment account indicated by the user101on the service computing device user interface131.

In block1240, the service computing device130transmits transaction details to the account management system150. For example, the service computing device130transmits transaction details comprising a transaction total, an account identifier of the service provider system, and the selection of the user101account payment option. In an example, the account management system150receives the transaction details over the network140. In an example, the account management system150extracts payment account information from the user101account corresponding to the received selection of the user101payment account.

In block1250, the account management system150transmits a transaction authorization request to an issuer system. In an example, the account management system150generates a transaction authorization request comprising user101payment account information, service provider system payment account information, and a total amount of the transaction. In an example, the account management system150receives the transaction authorization request from the account management system150via the network140.

In block1260, the issuer system approves the transaction authorization request and transmits a transaction authorization approval to the account management system150. In an example, the issuer system subtracts the user's101current account balance from the user's101credit limit associated with the user's101credit account. In an example, if the different between the credit limit and the credit balance is a positive number greater than the total amount of the transaction, the issuer system approves the transaction authorization. For example, the user101will not exceed his credit limit if the transaction were to be approved, therefore the issuer system approves the transaction authorization request. In an example, the account management system150receives the transaction authorization approval from the issuer system. In another example, the account management system150receives a denial of the transaction authorization request in response to the issuer system declining the transaction authorization request. In an example, if the issuer system approves the transaction authorization request, the issuer system credits the service provider system account and adds a charge to the user101credit account statement, bank account statement, or other appropriate statement.

In block1270, the account management system150transmits a transaction receipt to the service computing device130. For example, the account management system150transmits a confirmation of the approval of the transaction or a notice of a declined transaction to the service computing device130over the network140. In an example, the service computing device130prints a receipt comprising a summary of a successful or unsuccessful transaction via a printer device that is a component of or otherwise communicatively coupled to the service computing device130. In another example, the account management system150may transmit a transaction receipt to the user computing device110to be displayed on the user computing device. In yet another example, the service computing device130transmits a transaction receipt to the user computing device110to be displayed on the user computing device110.

Additional Examples

In certain examples, the service computing device130processes a service request to determine and/or verify the identity of a user101for purposes of authentication, granting access, or personalizing experiences with the user101at the service provider system location. In an example, gaining access to a location or authorization to initiate a service request is restricted to a certain type of user101based on user101attributes or based on the identity of the user. An example user attribute is the age of the user. In this example, a user101signs into the service application112via the user computing device110and enters the service provider system location. The user computing device110receives a service beacon device identifier120from the service beacon device and transmits the identifier to the account management system140. The account management system140transmits facial templates to the service computing device130corresponding to the user101and any other users101whose user computing devices110are in network range of the service computing device, who are signed in to the service application associated with the account management system. The service camera device130captures a facial image of the user101and identifies the user101by comparing the captured facial image against the received facial templates to find a match. In addition to or instead of facial recognition, the service computing device130identifies the user101in a video feed based on comparing movement data received from the user computing device110to data from a video feed captured by the service camera device135. In an example, the user101requests to purchase a restricted item, for example, an alcoholic beverage. For example, the service provider system130location does not process service requests for users that are less than twenty one years old. In this example, the service computing device130approves or denies service request requested by the user101based on the user101identity and/or user101attributes.

In another example, only users101who are employees of the service provider system are authorized to access a door to a restricted room at the service provider system location. In this example, a user101attribute comprises a user's101name or identity. In this example, the user101requests to access the restricted area of the service provider system location or otherwise unlock a restricted access door. For example, the service computing device130comprises an electronic lock or a device communicatively coupled to an electronic lock. In an example, the service computing device130retrieves data comprising a set of user101identities allowed access to a restricted area. In an example, the service computing device130unlocks the electronic lock or otherwise allows access to the restricted area if the identity of the user101matches the user101identity of one of the users101of a set of users101allowed access to the restricted area. In another example, based on the identity of the user101not corresponding to an identify of one of the set of users101allowed access to the restricted area, the service computing device130does not allow the user101access to the restricted area.

In another example, the service provider system location provides a personalized one-on-one experience to a user101based on the user's101service request history and habits. The process involves notifying service computing device130operators of the entry of a user101at the service provider location, the identity of the user101, and any associated current or past service request history that will allow the service provider system130to streamline the experience and provide a satisfactory experience to the user101at the service provider system location. In an example, once the user101is identified via movement data recognition and/or facial recognition, the service computing device130displays details about the user101to the operator of the service computing device130. For example, the user's101name, nickname, interests, and/or other relevant data is displayed to the operator so that the service computing device130operator can provide a personalized experience to the user at the service provider system location. For example, the operator views the name of the user101, “Charlotte Thomas,” displayed on the service computing device and says, “How may I help you today, Ms. Thomas?” In this example, the service computing device130may display current or past service request history of the user. For example, the service computing device130operator views the name of the user101, “Charlotte Thomas,” and that the user101requested a particular service request in the past. In this example, the service computing device130operator may greet the user101as follows: “How may I help you today, Ms. Thomas?” and then follow up with “Are you satisfied with your previous service request?”

In yet another example, a service provider system may identify a user101to provide delivery to a user101. In this example, the user101signs into the service application via the user computing device110and enters the service provider system location. The user computing device110receives a service beacon device120identifier from the service beacon device120and transmits the identifier to the account management system140. The account management system140transmits facial templates to the service computing device130corresponding to the user101and any other users101whose user computing devices110are in network range of the service beacon device120and who are signed in to the service application112. The service camera device135captures a facial image of the user101and identifies the user101by comparing the captured facial image against the received facial templates to find a match. In another example, in addition to or instead of identifying the user101via facial recognition, the service computing device130identifies the user101in a video feed based on comparing movement data received from the user computing device110to data from a video feed captured by the service camera device135. In this example, the identified user101requests to pickup and item that was previously ordered. In this example, based on the identity of the identified user101matching a user101identity associated with an item for pickup, the operator of the service computing device130delivers the item to the user101. For example, the service computing device130operator determines the item for pickup associated with the identified user101.

Other Examples

FIG. 13depicts a computing machine2000and a module2050in accordance with certain examples. The computing machine2000may correspond to any of the various computers, servers, mobile devices, embedded systems, or computing systems presented herein. The module2050may comprise one or more hardware or software elements configured to facilitate the computing machine2000in performing the various methods and processing functions presented herein. The computing machine2000may include various internal or attached components such as a processor2010, system bus2020, system memory2030, storage media2040, input/output interface2060, and a network interface2070for communicating with a network2080.

The computing machine2000may be implemented as a conventional computer system, an embedded controller, a laptop, a server, a mobile device, a smartphone, a set-top box, a kiosk, a vehicular information system, one more processors associated with a television, a customized machine, any other hardware platform, or any combination or multiplicity thereof. The computing machine2000may be a distributed system configured to function using multiple computing machines interconnected via a data network or bus system.

The processor2010may be configured to execute code or instructions to perform the operations and functionality described herein, manage request flow and address mappings, and to perform calculations and generate commands. The processor2010may be configured to monitor and control the operation of the components in the computing machine2000. The processor2010may be a general purpose processor, a processor core, a multiprocessor, a reconfigurable processor, a microcontroller, a digital signal processor (“DSP”), an application specific integrated circuit (“ASIC”), a graphics processing unit (“GPU”), a field programmable gate array (“FPGA”), a programmable logic device (“PLD”), a controller, a state machine, gated logic, discrete hardware components, any other processing unit, or any combination or multiplicity thereof. The processor2010may be a single processing unit, multiple processing units, a single processing core, multiple processing cores, special purpose processing cores, co-processors, or any combination thereof. According to certain embodiments, the processor2010along with other components of the computing machine2000may be a virtualized computing machine executing within one or more other computing machines.

The system memory2030may include non-volatile memories such as read-only memory (“ROM”), programmable read-only memory (“PROM”), erasable programmable read-only memory (“EPROM”), flash memory, or any other device capable of storing program instructions or data with or without applied power. The system memory2030may also include volatile memories such as random access memory (“RAM”), static random access memory (“SRAM”), dynamic random access memory (“DRAM”), and synchronous dynamic random access memory (“SDRAM”). Other types of RAM also may be used to implement the system memory2030. The system memory2030may be implemented using a single memory module or multiple memory modules. While the system memory2030is depicted as being part of the computing machine2000, one skilled in the art will recognize that the system memory2030may be separate from the computing machine2000without departing from the scope of the subject technology. It should also be appreciated that the system memory2030may include, or operate in conjunction with, a non-volatile storage device such as the storage media2040.

The storage media2040may include a hard disk, a floppy disk, a compact disc read only memory (“CD-ROM”), a digital versatile disc (“DVD”), a Blu-ray disc, a magnetic tape, a flash memory, other non-volatile memory device, a solid state drive (“SSD”), any magnetic storage device, any optical storage device, any electrical storage device, any semiconductor storage device, any physical-based storage device, any other data storage device, or any combination or multiplicity thereof. The storage media2040may store one or more operating systems, application programs and program modules such as module2050, data, or any other information. The storage media2040may be part of, or connected to, the computing machine2000. The storage media2040may also be part of one or more other computing machines that are in communication with the computing machine2000such as servers, database servers, cloud storage, network attached storage, and so forth.

The computing machine2000may operate in a networked environment using logical connections through the network interface2070to one or more other systems or computing machines across the network2080. The network2080may include wide area networks (WAN), local area networks (LAN), intranets, the Internet, wireless access networks, wired networks, mobile networks, telephone networks, optical networks, or combinations thereof. The network2080may be packet switched, circuit switched, of any topology, and may use any communication protocol. Communication links within the network2080may involve various digital or an analog communication media such as fiber optic cables, free-space optics, waveguides, electrical conductors, wireless links, antennas, radio-frequency communications, and so forth.

The processor2010may be connected to the other elements of the computing machine2000or the various peripherals discussed herein through the system bus2020. It should be appreciated that the system bus2020may be within the processor2010, outside the processor2010, or both. According to some embodiments, any of the processor2010, the other elements of the computing machine2000, or the various peripherals discussed herein may be integrated into a single device such as a system on chip (“SOC”), system on package (“SOP”), or ASIC device.

The example systems, methods, and acts described in the embodiments presented previously are illustrative, and, in alternative embodiments, certain acts can be performed in a different order, in parallel with one another, omitted entirely, and/or combined between different examples, and/or certain additional acts can be performed, without departing from the scope and spirit of various embodiments. Accordingly, such alternative embodiments are included in the scope of the following claims, which are to be accorded the broadest interpretation so as to encompass such alternate embodiments.