Systems and Methods for Use in Determining Consumer Interest in Products Based on Intensities of Facial Expressions

Exemplary systems and methods are provided for directing offer content to consumers at communication devices, while the consumers are at merchants. An exemplary method includes capturing an image of a consumer when the consumer is in the vicinity of a product at a merchant, where the image depicts a facial expression of the consumer, and determining an intensity associated with the facial expression of the consumer, as captured in the image. The method also includes determining, by a computing device, a location at the merchant of a communication device associated with the consumer and selecting, by the computing device, an offer associated with the product for the consumer, based on the intensity of the facial expression and the determined location of the communication device, thereby relying at least in part on a consumer reaction to select the offer.

FIELD

The present disclosure generally relates to systems and methods for use in determining consumer interest in products, and in particular, for use in determining consumer interest in products based on intensities of facial expressions, as to particular emotional factors, for consumers when viewing and/or in the presence of the products.

BACKGROUND

This section provides background information related to the present disclosure that is not necessarily prior art.

Products (e.g., goods, services, etc.) are known to be offered for sale, and to be sold, by merchants. Consumers are also known to purchase the products from the merchants. The consumers may purchase the products to fulfill needs for the products (e.g., groceries, etc.), or based on desires for the products. In any event, the consumers often have choices between different products, whereby sale of the products is one indicator of the consumers' like or dislike of the products, of the product costs, of advertising relating to the products, etc. Separately, manufacturers of products and/or merchants involved with offering the products for sale and selling the products are known to interact with consumers to elicit verbal and/or written feedback from the consumers regarding various ones of the products. The interactions with the consumers and the feedback provided by the consumers may, from time to time, lead to changes in the products offered for sale, in general or specifically, to certain consumers or groups of consumers.

DETAILED DESCRIPTION

Merchants and/or manufactures attempt to sell products to consumers, and further attempt to understand the likes and dislikes of consumers in connection with selecting products to be offered for sale. To do so, the merchants and/or manufactures often rely on historical purchase data for the products, related products, and, sometimes, unrelated products, as well as market research and surveys through which consumers provide feedback related to different products. As such, the merchants and/or manufactures are limited in the data upon which decisions related to product offerings may be based. Uniquely, the systems and methods herein are suited to determine consumer interest in products offered for sale at merchants based on facial expressions of consumers. In particular, for example, cameras are disposed at merchant locations to capture, with the consumer's consent, images of a consumer's face when the consumer is in the vicinity and/or presence of one or more particular products offered for sale by a merchant. The images are analyzed to determine an intensity of facial expressions of the consumer in the images (in connection with one or more emotional factors (e.g., likeness, surprise, confusion, focus, exhaustion, etc.). By merging the intensities derived from the images and a location of the consumer when the images are captured (determined based on a location of a communication device associated with the consumer, for example), an evaluation engine may determine whether the consumer is interested in the particular product(s) or not. The evaluation engine then may select one or more offers for the consumer for the product(s) in which the consumer's facial expressions reveal a “like” while omitting offers for the product(s) in which the consumer's facial expressions reveal a “dislike.” Apart from selecting the offer(s) for the consumer, the merged data and/or the facial expression intensity data may also be provided to entities involved in selecting the product(s) (or features of products) to be offered for sale, etc. (e.g., product manufacturers, etc.), whereby the consumer's various emotional responses may be factored into such decisions in an efficient and accurate manner.

FIG. 1illustrates an exemplary system100, in which one or more aspects of the present disclosure may be implemented. Although, in the described embodiment, the system100is presented in one arrangement, other embodiments may include the system100arranged otherwise, depending, for example, on merchant locations and/or configurations, types of products offered for sale by merchants, manners of capturing consumer expressions, etc.

Referring toFIG. 1, the system100generally includes a merchant102, an acquirer104, a payment network106, and an issuer108, each coupled to (and in communication with) network110. The network110may include, without limitation, a wired and/or wireless network, a local area network (LAN), a wide area network (WAN) (e.g., the Internet, etc.), a mobile network, and/or another suitable public and/or private network capable of supporting communication among two or more of the illustrated parts of the system100, or any combination thereof. In one example, the network110includes multiple networks, where different ones of the multiple networks are accessible to different ones of the illustrated parts inFIG. 1. In this example, the network110may include a private payment transaction network made accessible by the payment network106to the acquirer104and the issuer108and, separately, a public network (e.g., the Internet, etc.) through which the merchant102and the acquirer104may communicate (e.g., via a website or via various network-based applications, etc.).

In the system100, the merchant102offers a variety of products (e.g., goods and/or services, etc.) for sale.FIG. 1illustrates four exemplary products as blocks, referenced112a-d. The products may include any different products, for example, automobiles, books, electronics, power tools, home care items, toys, etc. The products112a-dare often situated at the merchant102, so that consumers are able to view the products112a-d(and potentially interact with the products112a-d(e.g., pick up, hold, feel, touch, interaction with, etc.)). However, in some instances, rather than the physical products112a-d, the merchant102may include a presentation related to the products112a-doffered for sale (e.g., images, summaries, demonstration, videos, advertising kiosks, etc.). Here, when consumers view the presentations related to the products112a-d, the consumers should still be understood to be viewing and/or in the vicinity of the products112a-d. For example, a consumer, such as exemplary consumer114, viewing and/or holding a smartphone at the merchant102(i.e., in the vicinity of the physical smartphone) would be the same as the consumer114viewing a description of the smartphone including subscription fees for streaming services and/or images of the same smartphone, etc. (i.e., the consumer114would still be considered in the vicinity of the smartphone herein, even though not in the vicinity of the physical smartphone).

In addition in the system100, multiple cameras116a-dare disposed at the merchant102. Each of the cameras116a-dis generally assigned to a region of the merchant102and/or to one or more of the products112a-dat the merchant102. The cameras116a-dare generally disposed to capture images of the consumers with their consent (e.g., the consumer114, etc.) when in the presence and/or vicinity of the products112a-d. In connection therewith, the cameras116a-dmay be, for example, suitable cameras to be mounted on shelves of the merchant102, so that the cameras are installed at a height to capture images of consumers' faces. In particular, the cameras116a-dmay be disposed to capture images of the consumers while the consumers are interacting with and/or in the vicinity of the specific products112a-d, but not other (or all) products at the merchant102. Further, the cameras116a-dmay be configured to capture the images of the consumers and transmit image records associated therewith to a desired computing device. The image records may include the raw images captured by the cameras116a-d, and/or analysis of the images as described in more detail below. In at least one embodiment, the camera116c, for example, is configured to capture an image of a consumer viewing product112c, to analyze the captured image, and to transmit the raw image (as captured) and an analysis thereof in an image record to a computing device, for example, all with the consumer's consent.

In one implementation of the system100, the cameras116a-dmay each include micro cameras, being of sufficient size to operate as described herein, but small enough to reduce the impact on a product display and/or shelving at the merchant102. For example, a micro camera may measure at about one inch square or less. However, cameras having other sizes, dimensions, and/or shapes (e.g., dimensions greater than one inch, etc.; shapes other than square; etc.) may be used in other implementations of the system100. Further, cameras other than micro cameras may be used in various implementations of the system100(e.g., any camera suitable to capture facial expressions of consumers, etc.).

FIG. 2illustrates an example installation of a camera216(e.g., a micro camera, etc.) at a merchant in accordance with the present disclosure. In this example, the camera216is installed at a display230associated with various products (each indicated at212) offered for sale by the merchant. In particular, the camera216is disposed along an upper portion of the display230within a sign232. In this position, the camera216is capable of capturing images of consumers' faces when viewing the products212(without impacting the product display230and/or shelving234, and without interfering with consumers views of the products212). In other embodiments, the camera216may be positioned differently in the display230, for example, on one of the shelves234, etc.

With reference again toFIG. 1, in an example interaction in the system100, the consumer114is able to move within the merchant102to view and/or interact with the products112a-d, which the consumer114may or may not decide to purchase (during a shopping session). In doing so, the consumer114establishes himself/herself in the vicinity of the products112a-d, as the consumer114moves through the merchant102. Being within the vicinity and/or presence of the products112a-dmay include, for example, the consumer114being within 5 meters, 2 meters, 1 meter, a half meter, or less, etc., of the products112a-d, or in front of the products112a-dwithin an aisle of the merchant102, etc. In addition, being within the vicinity and/or presence of the products112a-dmay include, for example, the consumer114moving in a direction toward the products112a-d, facing the products112a-d, etc.

The consumer114is associated with a communication device118, which may include, for example, a smartphone, a tablet, etc. The communication device118generally moves with the consumer114, as the consumer114moves from location to location at/within the merchant102, etc. The communication device118, as shown, is coupled to and is in communication with the network110.

In this exemplary embodiment, the consumer's communication device118is configured, via a network-based application (not shown) (e.g., installed on the communication device118or otherwise accessible by the communication device118, etc.), to perform one or more of the operations described herein. Specifically, the communication device118is configured, via the application, to cooperate with the merchant102(and specifically, an evaluation engine120associated with the merchant102(described below)) to provide its location to the merchant102(which is generally understood to be an approximation and/or indication of the location of the consumer114). In connection therewith, the communication device118may determine its location through GPS, or based on one or more network interactions (e.g., interactions with routers at the merchant102, etc.), or by transmitting certain information from which the merchant102is able to determine the communication device's location. For example, router signals may be employed consistent with the description in Applicant's co-owned U.S. patent application Ser. Nos. 14/978,686, 14/978,706, and 14/978,735, each of which is incorporated herein by reference in its entirety. Additionally, the communication device118is also configured to transmit to the merchant102(and specifically, to the evaluation engine120) an identifier (ID) for the communication device118, such as, for example, a unique application ID (APP ID), a media access control (MAC) address, or other ID associated with and/or unique or at least partially unique to the communication device118and/or the consumer114.

With that said, and as an example, the evaluation engine120may be configured to determine if the consumer114is within the vicinity and/or presence of one of the products112a-dbased on location data received from the consumer's communication device118combined with image data received from the cameras116a-d. For example, the evaluation engine120may receive multiple location coordinates from the consumer's communication device118as a function of time, for example, XY(t), when the consumer enters the merchant102. Based thereon, a current position of the consumer114at the merchant102can be estimated (based on location data received for the current time), as well as a distance of the consumer114from each of the products112a-din the merchant (based on location coordinates of the products112a-din the merchant102). From the received location coordinates for the most recent times (e.g., the last five seconds, the last ten seconds, the last thirty seconds, the last minute, the last five minutes, etc.), a direction of consumer movement within the merchant102can be determined, for example, XY(t-1) . . . XY(t-k). Then, based on information received from the cameras116a-d(e.g., face direction of the consumer114, etc.), the particular one of the products112a-dactually being viewed by the consumer114can be determined (as well as consumer interest in the particular one of the products, based on facial expressions, etc.). This will be described in more detail hereinafter.

With that said, reference to XY coordinates within the merchant102is generally based on a coordinate system or grid system defined by the merchant102at the merchant's location (with the coordinates (0, 0) generally representing where an X-axis and a Y-axis intersect, as overlaid on the merchant's location; etc.). However, it should be appreciated that other coordinate systems may be used herein without departing from the scope of the present disclosure, for example. Longitude and latitude coordinate systems, etc.

It should be appreciated that although only one merchant102, one acquirer104, one payment network106, one issuer108, one consumer114, and one communication device118are illustrated inFIG. 1, other system embodiments, may, and often will, include a different number of one or more of these components. Further, the merchant102may, and likely will, offer more or fewer than the four different products112a-dfor sale, potentially causing more or fewer cameras116a-dto be disposed at locations of the merchant102. With that said, the number of the components illustrated in the system100herein is intended to facilitate clear description and not to limit the scope of the present disclosure.

It should also be appreciated that upon selecting one or more of the products112a-dfor purchase, the consumer114may interact with the merchant102to transact for the purchased product(s). In connection therewith, to facilitate the transaction in the illustrated embodiment, the consumer114is associated with a payment account, issued to the consumer114by the issuer108, which is used to fund the transaction (as described next).

In an exemplary transaction, the consumer114, after selecting and presenting the product112afor purchase, for example, presents a payment device associated with the payment account (e.g., a credit card, a debit card, a fob, a smartcard, a virtual application (included in communication device118, etc.), etc.) to the merchant102. In turn, the merchant102generates an authorization request for the transaction (e.g., including a payment account number and an amount of the purchase, etc.) and communicates the authorization request to the acquirer104. The authorization request is transmitted generally along path A in the system100, as shown inFIG. 1. The acquirer104communicates the authorization request with the issuer108through the payment network106, such as, for example, through MasterCard®, VISA®, Discover®, American Express®, etc., to determine whether the payment account is in good standing and whether there is sufficient funds and/or credit to cover the transaction. If the transaction is approved, an authorization reply (indicating the approval of the transaction) is transmitted back from the issuer108to the merchant102, along path A, thereby permitting the merchant102to complete the transaction. The transaction is later cleared and/or settled by and between the merchant102, the acquirer104, and the issuer108. If declined, however, the authorization reply (indicating a decline of the transaction) is provided back to the merchant102, along the path A, thereby permitting the merchant102to end the transaction or request alternative forms of payment.

Transaction data is generated, collected, and stored as part of the above exemplary interactions among the merchant102, the acquirer104, the payment network106, the issuer108, and the consumer114. The transaction data includes a plurality of transaction records, one for each transaction, or attempted transaction. The transaction records, in this exemplary embodiment, are stored at least by the payment network106(e.g., in a data structure associated with the payment network106, etc.). Additionally, or alternatively, the merchant102, the acquirer104, and/or the issuer108may store the transaction records in corresponding data structures, or transaction records may be transmitted between parts of system100. The transaction records may include, for example, payment account numbers, amounts of the transactions, merchant IDs, and dates/times of the transactions. It should be appreciated that more or less information related to transactions, as part of either authorization or clearing and/or settling, may be included in transaction records and stored (and/or transmitted) within the system100, at (or by) the merchant102, the acquirer104, the payment network106and/or the issuer108.

In the embodiments herein, consumers (e.g., consumer114, etc.) involved in the different transactions and/or interactions are prompted to agree to legal terms associated with their payment accounts and/or network-based applications, for example, during enrollment in their accounts and/or installation of such applications, etc. In so doing, the consumers voluntarily agree, for example, to allow merchants, issuers, payment networks, etc., to use transaction and/or location data generated and/or collected during enrollment, or later, and/or in connection with processing transactions, for subsequent use in general, and as described herein.

FIG. 3illustrates an exemplary computing device300that can be used in the system100. The computing device300may include, for example, one or more servers, workstations, routers, personal computers, tablets, laptops, smartphones, PDAs, point of sale (POS) devices, etc. In addition, the computing device300may include a single computing device, or it may include multiple computing devices located in close proximity or distributed over a geographic region, so long as the computing devices are specifically configured to function as described herein.

In the exemplary system100ofFIG. 1, each of the merchant102, the acquirer104, the payment network106, and the issuer108are illustrated as including, or being implemented in, computing device300, coupled to (and in communication with) the network110. In addition, the communication device118associated with the consumer114and the cameras116a-dassociated with the merchant102(as well as the camera216, and the cameras516a-c) can also each be considered a computing device (potentially coupled to and in communication with the network110) consistent with computing device300for purposes of the description herein. However, the system100should not be considered to be limited to the computing device300, as described below, as different computing devices and/or arrangements of computing devices may be used. In addition, different components and/or arrangements of components may be used in other computing devices.

As shown inFIG. 3, the exemplary computing device300includes a processor302and a memory304coupled to (and in communication with) the processor302. The processor302may include one or more processing units (e.g., in a multi-core configuration, etc.). For example, the processor302may include, without limitation, a central processing unit (CPU), a microcontroller, a reduced instruction set computer (RISC) processor, an application specific integrated circuit (ASIC), a programmable logic device (PLD), a gate array, and/or any other circuit or processor capable of the functions described herein.

The memory304, as described herein, is one or more devices that permit data, instructions, etc., to be stored therein and retrieved therefrom. The memory304may include one or more computer-readable storage media, such as, without limitation, dynamic random access memory (DRAM), static random access memory (SRAM), read only memory (ROM), erasable programmable read only memory (EPROM), solid state devices, flash drives, CD-ROMs, thumb drives, floppy disks, tapes, hard disks, and/or any other type of volatile or nonvolatile physical or tangible computer-readable media. The memory304may be configured to store, without limitation, a variety of data structures, product records, consumer location records, image records, intensity thresholds, and/or other types of data suitable for use as described herein. Furthermore, in various embodiments, computer-executable instructions may be stored in the memory304for execution by the processor302to cause the processor302to perform one or more of the functions described herein, such that the memory304is a physical, tangible, and non-transitory computer readable storage media. Such instructions often improve the efficiencies and/or performance of the processor302that is performing one or more of the various operations herein. It should be appreciated that the memory304may include a variety of different memories, each implemented in one or more of the functions or processes described herein.

In the exemplary embodiment, the computing device300includes a presentation unit306that is coupled to (and is in communication with) the processor302(however, it should be appreciated that the computing device300could include output devices other than the presentation unit306, etc.). The presentation unit306outputs information, either visually or audibly to a user of the computing device300, such as, for example, to the consumer114at the communication device118(e.g., product incentives, product offers, etc.), to a user associated with the merchant102(e.g., image data from cameras116a-d, etc.), etc. It should be further appreciated that various interfaces (e.g., as defined by network-based applications, etc.) may be displayed at computing device300, and in particular at presentation unit306, to display, for example, an offer for the consumer114, etc. The presentation unit306may include, without limitation, a liquid crystal display (LCD), a light-emitting diode (LED) display, an organic LED (OLED) display, an “electronic ink” display, speakers, etc. In some embodiments, presentation unit306includes multiple devices.

The computing device300also includes an input device308that receives inputs from the user (i.e., user inputs) such as, for example, selection of an offer for redemption from the consumer114, etc., or otherwise (e.g., image inputs of the consumer114via the cameras116a-d, etc.). The input device308is coupled to (and is in communication with) the processor302and may include, for example, a keyboard, a pointing device, a mouse, a button, a stylus, a touch sensitive panel (e.g., a touch pad or a touch screen, etc.), another computing device, a camera, and/or an audio input device. Further, in various exemplary embodiments, a touch screen, such as that included in a tablet, a smartphone, or similar device, behaves as both a presentation unit and an input device.

In addition, the illustrated computing device300also includes a network interface310coupled to (and in communication with) the processor302and the memory304. The network interface310may include, without limitation, a wired network adapter, a wireless network adapter, a mobile network adapter, a GPS transmitter, a GPS receiver, combinations thereof (e.g., a GPS transceiver, etc.), or other device capable of communicating to/with one or more different networks, including the network110. Further, in some exemplary embodiments, the computing device300includes the processor302and one or more network interfaces310incorporated into or with the processor302.

Referring again toFIG. 1, the evaluation engine120of the system100is specifically configured, often by computer-executable instructions, to perform one or more of the operations described herein. In the exemplary embodiment, the evaluation engine120is provided at the standalone computing device300at the merchant102, and is generally dedicated to the operations described herein. It should be appreciated, however, that the evaluation engine120may be incorporated within one or more other computing devices and/or computing operations at the merchant102. For example, the evaluation engine120may be incorporated into a server computing device, which also hosts advertising, marketing, financial, and/or security operations associated with the merchant102, or not. Further, in certain other embodiments, the evaluation engine120may be incorporated into computing devices apart or remote from the merchant102(e.g., with the evaluation engine120located external of the merchant102(e.g., as a standalone part of the system100, as part of the payment network106, etc.), etc.).

The system100also includes multiple data structures122-126. As shown, the data structures122-126are associated with the merchant102and coupled to the evaluation engine120. It should be appreciated that the data structures122-126may be separate from the evaluation engine120, as shown, or incorporated therein. Further, in various embodiments, one or more of the data structures122-126may be separate from the merchant102, with the evaluation engine120then having access thereto (e.g., via network110, etc.).

The data structure122is a product data structure, which includes product records for various products, including products112a-d, at the merchant102. Each product record includes a location, represented by XY coordinates within the merchant location (e.g., based on a coordinate system or grid system defined by the merchant102at the merchant location, etc.), and a unique product code (or UPC) or other indicator of the corresponding ones of the products112a-d, which is at least partially unique to the product. As such, in this embodiment, each product record is expressed as, for example {XY, UPC}. However, in other embodiments, product records may include further and/or other information about the corresponding products112a-d, the merchant102, and/or the particular location of the products112a-din the merchant102(e.g., price, latitude and longitude coordinates, etc.).

The data structure124is an image data structure, which is configured to store image records from the cameras116a-d. As described above, the image records may include raw image data received from the cameras116a-d, including time stamps, whereupon the evaluation engine120may then provide analysis upon receipt. Or, the image records may further include expression intensities (of the consumers whose images have been captured) and locations, based on the cameras116a-dperforming an analysis on the images captured thereby. In such examples, the image records, received from the cameras116a-d, may include, for each record, the location of the captured image (e.g., the location of the one or more of the cameras116a-dthat captured the image, etc.), the time captured, and an intensity of an expression of a consumer in the captured image, together provided in the form {XY, t, EI}, where XY is again the location, t is the time, and EI is the expression intensity (e.g., likeness-surprise intensity (LS), etc.).

Finally, the data structure126is a consumer location data structure, which is configured to store consumer location records, which are received as described below (e.g., based on location records of communication devices associated with the consumers at the merchant102, etc.). The consumer location records are generally in the form of {XY, t, MAC}, where, for each record, XY is again a location, t is a time, and MAC is an address/identifier associated with the particular communication device associated with a consumer (e.g., communication device118, etc.) (e.g., a MAC address, etc.). It should be appreciated that the consumers may register to the evaluation engine120, whereby the MAC addresses (or potentially APP IDs or other identifiers) for the consumers' communication devices are registered to the evaluation engine120, and stored in the data structure126, to thereby enable the evaluation engine120to operate as described herein.

In operation in the system100, the evaluation engine120is configured to interact with the communication device118, when the consumer114enters and/or moves through the merchant102(broadly, during a shopping session by the consumer114). In doing so, the evaluation engine120is configured to receive an identifier associated with the communication device118(e.g., a MAC address, APP ID, etc.), and to track the location of the communication device118within the merchant102, in general or at various times (and potentially to store the data in the data structure126). The evaluation engine120is also coupled to (and is in communication with) the cameras116a-dand configured to receive image records from the cameras116a-d(and potentially to store the records in the data structure124). As previously described, the cameras116a-dare assigned to specific locations in the merchant102. In connection therewith, the image records provided by each of the cameras116a-didentify the specific locations of the particular ones of the cameras116a-dproviding the records. In connection therewith, the evaluation engine120is configured to merge the received image records from the cameras116a-dwith the consumer location records, based on the location(s) of the communication device118during the shopping session for the consumer114and product records for the products112a-dat the merchant102(from the data structure122), to thereby identify the received image records to the communication device118and to particular ones of the products112a-dfor which the consumer114was in the vicinity. Further, upon receiving the image records, the evaluation engine120is configured (or, alternatively, prior to transmitting the image records, the cameras116a-dare configured to) determine an emotional intensity of the consumer114, and in particular of the consumer's facial expressions, in connection with the various products being viewed by the consumer114. In turn, the evaluation engine120is configured to select an offer for the consumer114, based on intensity of the facial expressions of the consumer114in the various images when in the vicinity of one or more of the various products112a-d, and to transmit the offer to the consumer114at the communication device118(e.g., while the consumer is still at the merchant102, later, etc.).

It should be appreciated that the evaluation engine120may be further configured to compile one or more models, either specific to the consumer114or specific to one of the products112a-d, or more generically (e.g., to a demographic including the consumer114, etc.). The model(s) may then be transmitted, by the evaluation engine120, to the merchant102and/or other entities associated with the products112a-d(e.g., competitors, manufacturers, product researchers, etc.), for example, for use in modifying products, product offers, etc.

As described herein, the evaluation engine120(and/or the cameras116a-d) is (are) configured to determine an emotional intensity of the consumer114, and in particular of the consumer's facial expression(s), based on the image record(s) associated therewith. In so doing, the evaluation engine120(and/or the cameras116a-d) may be configured to use/apply any suitable operations or methodologies to determine the emotional intensity including, for example, one or more of those described in the following references, which are incorporated herein by reference in their entirety: “Identifying Emotional Expressions, Intensities and Sentence level Emotion Tags using a Supervised Framework,” Dipankar Das and Sivaji Bandyopadhyay, Department of Computer Science and Engineering, Jadavpur University, PACLIC 24 Proceedings (2010), pages 95-104; and “Measuring the intensity of spontaneous facial action units with dynamic Bayesian network,” Yongqiang Li et al., Pattern Recognition 48 (2015) 3417-3427.

FIG. 4illustrates an exemplary method400for use in determining interest of a consumer in a product based on intensities of facial expressions of the consumer for one or more emotional factors. The method400is described herein with reference to the system100, and in particular, as operations of the portable communication device118and the evaluation engine120. It should be appreciated, however, that the methods described herein are not limited to the system100. And, conversely, it should be appreciated that the systems described herein are not limited to the exemplary method400.

In the method400, the consumer114initiates a shopping session by entering the location of the merchant102along with the communication device118and beginning to shop for products, including products112a-d. As the consumer moves through the location of the merchant102, the consumer114comes within the vicinity of one or more of the products112a-d, occasionally.

At402in the method400, the evaluation engine120initially identifies the communication device118. The evaluation engine120may identify the communication device118based on, for example, a MAC address associated with the communication device118, or an interaction with a network-based application installed and active on the communication device118(e.g., via an APP ID, etc.). In at least one embodiment, the consumer114may activate the network-based application, at the communication device118(e.g., manually, automatically, etc.), to signal to the evaluation engine120that the consumer114is at the merchant location. In connection therewith, and as described above, the consumer114may be registered to the evaluation engine120, whereby the MAC address (or the APP ID or other identifier) is registered to the evaluation engine120(and stored in the data structure122) to thereby enable the evaluation engine120to identify the communication device118and/or communicate therewith. Either by registering, or through some other method, the consumer114consents to the tracking of their communication device118and to the capture of their facial expression for analysis as described in the present disclosure.

Once identified, the evaluation engine120cooperates with the communication device118to determine, at404, the location of the communication device118at the merchant location. The evaluation engine120may determine the location at multiple different times, or continuously track the location of the communication device118as the consumer114moves about the merchant location. In so doing, the evaluation engine120may determine the location of the communication device118based on GPS data received from the communication device118and/or based on router data revived from the communication device118, and/or based on one or more other interactions with the communication device118, as described above. Then, when the location is determined, the evaluation engine120generates a consumer location record, at406, and stores the record in the data structure126, at408. Here, the consumer location record includes the location of the communication device118at the merchant102, or XY coordinate, the time (t) at which the communication device118is at the location, and the MAC address associated with the communication device118, thereby providing a consumer location record in the format: {XY, t, MAC}. As indicated by the dotted line inFIG. 4, the evaluation engine120repeats operations404-408as the consumer114(and the communication device118) moves through the merchant location, during the shopping session (continuously, periodically, etc.). As such, multiple consumer location records are generated, at406, and stored, at408. It should be appreciated that consumer location records for the communication device118, or for other devices, may be expressed in various other formats (e.g., other than {XY, t, MAC}, comprising other combinations of data (other than XY, t, and MAC), etc.), yet still be consistent with the description herein.

With continued reference toFIG. 4, separately in the method400, as the consumer114continues to move at the merchant location, the consumer114may stop in front of the product112b, for example, to examine the product112band potentially pick it up (depending on the product112b, its packaging, etc.). In response, the camera116bcaptures, at410, an image of the consumer114, and in particular, the consumer's face. The captured image is time stamped with the date and/or time of the captured image. The camera116bthen transmits, at412, the image to the evaluation engine120. Operations410-412may be repeated each time the consumer112stops at (or is in the vicinity of) one of the products112a-dand/or one of the cameras116a-d.

Upon receipt of the image(s) (and upon identifying the particular product in which the consumer114is interested, or prior thereto as in the illustrated method400), the evaluation engine120performs one or more facial analysis operations on the image(s), at414. Specifically, for example, the evaluation engine120determines an emotional intensity from the image(s). In so doing, the evaluation engine120may initially determine, for each of various different emotional dimensions for the consumer114, a level of intensity of the consumer in connection with a particular product being viewed (e.g., on a scale of one to ten for each emotional dimension, with one representing a weakest value of the particular emotional dimension and ten representing a strongest value of the particular emotional dimension; on another scale; etc.). Then, the intensity levels for each of the different emotional dimensions may be combined (e.g., summed, averaged, etc.) and compared to a threshold to make an inference, for example, regarding the consumer's probable inner thoughts about the product being viewed. Such emotional dimensions may include, without limitation, likeness, surprise, confusion, focus, exhaustion, sadness, interest, happiness, amazement, openness, understanding, anger etc. And, a level of intensity of such emotional dimensions may be determined (e.g., on the scale of one to ten, etc.) based on various facial characteristics known to be associated with the particular emotional dimensions.

In one implementation, the facial analysis, by the evaluation engine120, may be specific to the likeness (e.g., as generally indicated by crow's feet wrinkles, raised cheeks, etc.) and surprise (e.g., as generally indicated by raised eyebrows, widened eyes, open mount, etc.) emotion factors of facial expression (thereby generating a likeness-surprise (LS) emotional intensity). That is, the intensity is generally indicative of a positive facial expression. Here, for example, LS emotional intensity from a scale of −k to +k can be evaluated at individual levels (e.g., for the consumer114, for other consumers, etc.) for the likeness and surprise emotion factors based on historical facial responses. As can be appreciated, LS emotional intensity can be a complex function of shape and relaxing of facial muscles, mouth shape, eye shape, and so on (e.g., taking into account the various facial features identified above, taking into account other facial features or combinations of facial features, etc.). Thus, from the recorded historical shapes, the evaluation engine120can determine a normal facial expression for the consumer114(e.g., resulting in an LS score of 0, etc.). Then, facial expressions by the consumer consistent with the likeness and surprise factors (suggesting a positive purchase) will have a relatively high, positive LS score on the scale of −k to +k (based on the particular scale used to represent the LS score). While facial expressions by the consumer inconsistent with the likeness and surprise factors will have a relatively low, negative LS score on the scale of −k to +k.

It should be appreciated that while the evaluation engine120performs the facial analysis in the method400to determine the intensity of the facial expression of the consumer114, as to one or more emotional factors, the cameras116a-d, or ones of the cameras116a-d, may be employed to perform the analysis in other embodiments in one or more of the manners described above. In such embodiments, the camera116b, for example, transmits, at412, image data (broadly, an image record) to the evaluation engine120, which includes an intensity of the facial expression and a time of the captured image, etc. The image data may additionally, or optionally, include a location of the camera116b(or a location of the camera's view) and the raw image captured by the camera116b, etc.

In either case, once the intensity of the facial expression is determined (whether at the evaluation engine120, at the camera116b, or elsewhere), the evaluation engine120generates an image record for the image, at406, and stores the image record to the image data structure124, at408. Generating the image record may simply include arranging image data consistent with the format included in the image data structure124, or other operations may be performed. In the method400, for example, generating the image record may include arranging the image data in the format: {XY, t, LS }, where XY is the location within the merchant102at which the image is captured, t is the time (and/or date) the image is captured, and LS is the likeness-surprise emotional intensity. It should be appreciated that image records may be expressed in various other formats (e.g., other than { XY, t, LS}, comprising other combinations of data (other than XY, t, and LS), etc.), yet still be consistent with the description herein.

It should also be appreciated that, in the method400, the cameras116a-dare disposed in substantially fixed locations in the merchant102(although this is not required in all embodiments). As such, when the image is received by the evaluation engine120from the camera116b, the evaluation engine120understands that the image is associated with a specific location at the merchant102. The location may, therefore, be stored with the image record (at408) and/or maintained in a separate data structure, which is accessible to the evaluation engine120. In one or more other embodiments, the camera116band/or the evaluation engine120may determine the location of the camera116b, for example, at the time that the image is captured and append the location to the image and to the image record (when generating the image record at406, for example). Similar to the communication device118, the location of the cameras116a-dmay be determined based on an XY coordinate system of the merchant102, GPS data, router data, etc., as described above.

Next in the method400, the evaluation engine120accesses, at416, the product data structure122, which includes product records for the product112b(as well as for multiple other products). In the method400, the product records are in the format: {XY, UPC}, where XY is the location of the particular product at the merchant102and UPC is the unique product code for the product. The product data structure122(or records therein) may further include an indication of the particular camera for (or associated with) each of the products. It should be appreciated that product records may be expressed in various other formats (e.g., other than {XY, UPC}, comprising other combinations of data (other than XY and UPC), etc.), yet still be consistent with the description herein.

Then, the evaluation engine120merges, at418, the determined consumer location records and image records (from408), and the product records (from416), based on at least time and location. Specifically, for example, the evaluation engine120may merge the records based on time and location being within respective thresholds (e.g., based on time in the records being within 3 seconds, 5 second, 10 seconds, etc. of each other; based on location in the records being within the same zone in a grid, being no more than one zone apart, being no more than five zones apart, etc.; combinations thereof; etc.). In turn, based on the captured images from the different cameras116a-dand the location records associated with the consumer114at the time the images are captured, as included in the merged records, the evaluation engine120can identify a particular one of the products112a-don which the consumer114is focused (e.g., based on a direction the consumer114is moving, a direction the consumer114is facing, an angle of the consumer's head, etc.). In connection therewith, the merged records may have a format: {MAC, t, UPC, LS}, thereby linking the intensity of facial expressions determined from the captured images (e.g., the likeness-surprise intensities, etc.) to the product(s) and the consumer114(via the communication device118) in the vicinity of the product(s).

Once the records are merged, the evaluation engine120selects, at420, at least one offer to be provided to the consumer114based on the merged records. The selection, by the evaluation engine120, may include an offer corresponding to a product (by UPC) at the merchant102for which the consumer, when in the vicinity of the product, expressed a particular likeness-surprise intensity during the shopping session (e.g., a highest likeness-surprise intensity, a median likeness-surprise intensity, etc.). Additionally, or alternatively, the evaluation engine120may select the offer when the intensity in one or more merged records exceeds a predefined threshold.

The evaluation engine120generally selects the offer during the current shopping session for the consumer114, to thereby encourage the consumer114to purchase a product (e.g., one of the products112a-d, etc.) in which the consumer114has interest (or appears to have expressed interest). It should be appreciated that selection may be based on the merged records for the current shopping session, and further on one or more prior shopping sessions, whereby the selection by the evaluation engine120accounts for interest over time (i.e., not only in the current shopping session but in prior shopping sessions as well). For example, the consumer114may linger at product112cin multiple different shopping sessions, where the intensity of the facial expressions for the consumer114is always just below a threshold for selecting an offer. Based on the consumer114repeating this behavior in multiple shopping sessions, the evaluation engine120may detected the duplicate merged records for the product112cand adjust the threshold and/or select an offer for the product. As another example, the evaluation engine120may identify that multiple different consumers continually linger at product112c. As such, when the consumer114is also identified as lingering at the product112c, but the intensity of his/her facial expressions is just below a threshold for selecting an offer, the evaluation engine120may adjust the threshold or select an offer, for the consumer114, based on/taking into account the common interest among the multiple consumers (including the consumer114).

With that said, the offer selected by the evaluation engine120(at420) may include any desired offer such as, for example, a coupon for a percentage off the purchase price of a particular product, a rebate for a particular product, or any other offer to reduce the price of a product, reduce the price of related products (or competing products), or otherwise encourage the consumer114to purchase a particular product or like product. Further, in some embodiments, the offer may be based on a value of the intensity, determined by the evaluation engine120(where different offers may be available to the consumer114when the intensity exceeds a predefined threshold, based on an amount by which the threshold is exceeded).

In some implementations of the method400, the offer selection, by the evaluation engine120, may be based on the merged records in combination with historical transaction data (e.g., transaction data for one or more payment accounts, etc.), consumer demographics (e.g., age, gender, location, etc.), etc. Here, the transaction data may be compiled as described above with reference toFIG. 1for the payment account associated with the consumer114, for example, or for other consumers.

Finally in the method400, once an offer is selected, the evaluation engine120transmits, at422, the offer to the consumer114, and specifically, to the communication device118. The evaluation engine120may transmit the offer via short-message service (SMS), e-mail, and/or to a network-based application in the communication device118, any of which may result in a visual and/or audible notification to the consumer114. In response, the consumer114, if interested in the product112b, for example (as indicated by the intensity of the facial expression), is further encouraged to purchase the product112bfrom the merchant102.

In another aspect of the present disclosure, the evaluation engine120may further be able to compile the merged records (e.g., from418in the method400, etc.), which include intensity of facial expressions and product identifiers (with or without the communication device identifiers). The merged records may be provided to one or more entities, including, for example, the merchant102, to provide feedback as to interest in the products by the consumers, including the consumer114. Based on such feedback, the entities can alter products, product offers, etc., as appropriate to potentially improve salability of the products.

FIG. 5schematically illustrates an example grouping/installation of cameras516a-cthat may be employed in the method400(and/or in the system100) to capture images of the consumer114at the merchant102, when the consumer114is in the vicinity of products512a-eat the merchant102. The grouping/installation of the cameras516a-cmay be within a single product display at the merchant102(with the single product display comprising the products512a-e), or the grouping/installation may be within an isle or other common location of the merchant102(potentially having multiple product displays for the different products512a-e).

In particular in this example, as the consumer114moves through the merchant102(as the consumer initiates a shopping session), consumer location records are generated for the consumer114with the consumer's consent. In connection therewith, Table 1 illustrates multiple exemplary consumer location records that may be generated for the consumer114and stored in the data structure126, during the shopping session by the consumer114at the merchant102. In this example, each consumer location record is in the format: {XY, t, MAC}, where XY is representative of a grid system associated with the merchant102, t is representative of a time at which the consumer114is at the corresponding location, and MAC is representative of the consumer's communication device118.

The cameras516a-cmay be pre-installed in the merchant102at specific locations and facing specific directions. For instance, in this example, the cameras516a-cmay be located in the merchant102as indicated in Table 2. And, when the consumer114is within view of the cameras516a-c(with the cameras516a-ccontinually collecting data, for example), the cameras516a-ceach capture an image (or multiple images) of the consumer114and transmit the image(s) to the evaluation engine120.

In turn, the evaluation engine120analyzes the received image(s) to determine an emotional intensity of the consumer114in the image(s) (if not done by the cameras516a-c). In this example, the analysis is specific to a likeness-surprise (LS) emotional intensity, and includes evaluation of the image(s) relative to a likeness emotional factor (e.g., as generally indicated by crow's feet wrinkles, raised cheeks, etc.) and a surprise emotional factor (e.g., as generally indicated by raised eyebrows, widened eyes, open mount, etc.). In particular, based on the facial expression(s) of the consumer114in the received image(s), the evaluation engine120rates the consumer114on a scale of 1-10 for each factor, taking into account the general facial features known to indicate each factor. The evaluation engine120then combines the ratings (e.g., sums the ratings in this example, etc.) to determine the LS emotional intensity for the consumer114.

The evaluation engine120next generates an image record for the image (or for each of the images) received and analyzed. Table 3 illustrates multiple exemplary image records (stored in the data structure124), as generated during the shopping session by the consumer114at the merchant102. In this example, each image record is in the format: {XY, t, LS }. Also in this example, a normal LS emotional intensity score has been shifted to LS=10, on a scale of 0-20.

In addition in this example, the products512a-eare located in the merchant102at specific positions. In particular, the products512a-emay be positioned in the merchant102at the specific locations indicated in Table 4. In connection therewith, each product may be associated with a product record, as shown in Table 5, which is stored in the data structure122. In this example, each of the product records is in the format: {XY, UPC}, where XY is the location of the particular product at the merchant102, and UPC is the unique product code for the product.

Then in this example, the evaluation engine120merges the various consumer location records, the image records, and the product records based on time and location. Specifically, for example, the evaluation engine120merges the records based on time records being within ten seconds of each other, and location records being within plus/minus five zones of the same XY grid zone. Table 6 illustrates the merged records, for the example records presented in Tables 1, 3, and 5, in the format: {MAC, t, UPC, LS}. The merged records thereby link the intensity of the consumer's facial expressions, as determined from the captured images at the cameras516a-c, to the various product(s) that were viewed by the consumer114and the consumer114himself/herself (via the communication device118), when the consumer114is in the vicinity of the product(s).

Based on the above, in general, when the consumer114is located in the merchant102at XY=(11, 32) (as illustrated inFIG. 5), the evaluation engine120can determine a particular product being viewed by the consumer114. For instance, based on a shortest distance match from the position of the consumer114to the various products512a-ein the merchant102(e.g., based on distances D1-D5, etc.), the evaluation engine120can determine that the closest ones of the products512a-eare products512a-c(with UPCs=123, 124, and 125). In addition, based on a shortest distance match from the consumer114to the various cameras516a-c, the closest are cameras516a,516b(with camera IDs1036and1236), facing in a direction of −90 degrees so as to be capable of capturing images of the consumer114. In turn, from the relative angle between the consumer114and the cameras516a,516b, the evaluation engine120can determine that the consumer114is looking at product512b(e.g., taking into account a direction the consumer114is moving, a direction the consumer114is facing, an angle of the consumer's head, etc.).

Finally in this example, the evaluation engine120selects at least one offer to be provided to the consumer114based on the merged records (e.g., as shown in Table 6, etc.). The selection, by the evaluation engine120, may include an offer corresponding to a product (by UPC) at the merchant102for which the consumer114, when in the vicinity of the product, expressed a particular likeness-surprise intensity during the shopping session (e.g., a highest likeness-surprise intensity, a median likeness-surprise intensity, etc.). For example, with reference to the merged records in Table 6, and based on a predefined threshold of 12 (which, for example, may be empirically determined as a sufficient level of LS intensity), the evaluation engine120may identify an offer for the product with the UPC 124, and provide the offer to the consumer114(via the communication device118, etc.). As another example, again with reference to the merged records in Table 6, and based on predefined thresholds of 8 and 12, the evaluation engine may identify one offer for the product with the UPC 123 and another offer for the product with the UPC 124 (and provide the offers to the consumer114via his/her communication device118, etc.). In other examples, the offer engine may further take into account merchant inventory and cost in determining which offers, if any, to provide to the consumer114.

In view of the above, the systems and methods herein permit reactions of consumers, as determined from facial expressions, to provide indications of the consumers' like and/or dislike of products offered for sale by merchants. While written and/or verbal consumer feedback (i.e., active feedback) in various forms is often available and useful, for the merchants, to determine what products to continue selling, or not, the facial expressions of consumers, as addressed herein, in the presence and/or vicinity of products, provides an unbiased, pure passive feedback mechanism, which alone, or in combine with other feedback, may provide a more accurate and efficient indication of the consumers' response, thoughts and/or feelings about the products. The methods and systems herein therefore may be able to provide a passive feedback mechanism, which avoids consumer overthinking and/or bias in connection with active feedback.

The systems, devices, and methods described herein may be partially or fully implemented by a special purpose computer created by configuring a general purpose computer to execute one or more particular functions embodied in computer executable instructions. In addition, the functional blocks and flowchart elements described above serve as software specifications, which can be translated into the computer executable instructions by the routine work of a skilled technician or programmer.

As will be appreciated based on the foregoing specification, the above-described embodiments of the disclosure may be implemented using computer programming or engineering techniques including computer software, firmware, hardware or any combination or subset thereof, wherein the technical effect may be achieved by performing at least one of the following operations: (a) capturing an image of a consumer when the consumer is in the vicinity of a product at a merchant, the image depicting a facial expression of the consumer; (b) determining an intensity associated with the facial expression of the consumer, as captured in the image; (c) determining a location at the merchant of a communication device associated with the consumer; (d) selecting an offer associated with the product for the consumer, based on the intensity of the facial expression and the determined location of the communication device, thereby relying at least in part on a consumer reaction to select the offer; (e) merging an intensity record associated with the intensity of the facial expression of the consumer and a consumer location record indicative of the location at the merchant of the communication device associated with the consumer; (f) identifying a product record associated with the identified location; and (g) transmitting the offer to the communication device, thereby permitting the consumer to redeem the offer.

In addition, as used herein, the term product may include a good and/or a service, or further summaries, depictions, descriptions, or other data pertaining to the good and/or service.

Further, the methods and systems described herein are intended to be carried out in full compliance with all applicable consumer data privacy and data usage laws.