Determining quality of images for user identification

In an approach to determining quality score of images for user identification, one or more processors receive a set of input images for a first user. One or more processors may determine a quality score for each image in the set of input images with respect to an ability of each respective image to identify the first user, wherein the quality score pertains to both: (i) a quality the image, and (ii) a quality of a subject of the image. One or more processors may additionally determine a first set of output images based on the quality score for each image in the set of input images. One or more processors may further identify the first user based on the first set of output images.

BACKGROUND OF THE INVENTION

The present invention relates generally to the field of user identification, and more particularly to selection of images for user identification based on quality scores.

Computer vision is an interdisciplinary field that aims for computers to analyze and extract, and interpret contents of images and videos. In other words, computer vision attempts to describe the reality (e.g., the identity of a person) based on analysis of one or more images. Computer vision is used widely for tasks such as optical character recognition (OCR), object recognition in retail industry, medical imaging, motion capture, and security (e.g., monitoring pools, highway traffic, etc.). Computer vision is further used for applications such as face detection and visual authentication for computer users.

SUMMARY

Embodiments of the present invention disclose a method, a computer program product, and a system for establishing personal identity using real-time contextual data. The method may include one or more processors determining a quality score for each image in the set of input images with respect to an ability of each respective image to identify the first user, wherein the quality score pertains to both: (i) a quality the image, and (ii) a quality of a subject of the image. The method may also include one or more processors determining a first set of output images based on the quality score for each image in the set of input images. The method may further include one or more processors identifying the first user based on the first set of output images.

DETAILED DESCRIPTION

Identifying users walking into locations (e.g., stores, restaurants, or other similar locations) by comparing user biometric data (e.g., data captured with video cameras) with a plurality of social media profiles may be inefficient based on the large number social media profiles available for inspection. In general, passive management of customer loyalty programs using biometric data has been unfeasible due to the difficulty of identifying users by capturing biometric data.

Some embodiments of the present invention provide improved techniques for user identification by: (1) building subsets of social profiles to match contextual, real-time social media data, and static social media data; (2) applying local filtering to a set of input images to improve selection of images for user identification; and (3) matching multiple sub-optimal images of the user to multiple photos in a social media profile.

Some embodiments of the present invention further disclose a passively managed customer loyalty program through the use of social media profiles. In some embodiments, the loyalty program includes the following characteristics: (1) using a shadow profile for customers for which no social media profile is found; (2) using social media profile data at the point of sale (POS); (3) managing customer data online and/or on premise at the retailer using data isolation; (4) performing analytics on the social media profiles; and (5) generating target advertisements.

Implementation of embodiments of the present invention may take a variety of forms, and exemplary implementation details are discussed subsequently with reference to the Figures.

Distributed data processing environment100includes server computer104, mobile device110, site sensors116, and social media sources118, all interconnected over network102.

In general, network102can be any combination of connections and protocols that will support communications between server computer104, mobile device110, site sensors116, and social media sources118, and other computing devices (not shown) within distributed data processing environment100. Network102can be, for example, a telecommunications network, a local area network (LAN), a wide area network (WAN), such as the Internet, or a combination of the three, and can include wired, wireless, or fiber optic connections. Network102can include one or more wired and/or wireless networks that can receive and transmit data, voice, and/or video signals, including multimedia signals that include voice, data, and video information.

Server computer104can be a standalone computing device, a management server, a content service, a mobile computing device, or any other electronic device or computing system capable of receiving, sending, and processing data. In other embodiments, server computer104can represent a server computing system utilizing multiple computers as a server system, such as in a cloud computing environment. In another embodiment, server computer104can be a laptop computer, a tablet computer, a netbook computer, a personal computer (PC), a desktop computer, a personal digital assistant (PDA), a smart phone, or any other programmable electronic device capable of communicating with mobile device110, site sensors116, social media sources118, and other computing devices (not shown) within distributed data processing environment100via network102. In another embodiment, server computer104represents a computing system utilizing clustered computers and components (e.g., database server computers, application server computers, etc.) that act as a single pool of seamless resources when accessed within distributed data processing environment100. Server computer104may include internal and external hardware components, as depicted and described in further detail with respect toFIG. 25.

Mobile device110enables a user to access social media sources118and other websites and applications (not shown). For instance, mobile device110can be a laptop computer, a tablet computer, a smart phone, or any programmable electronic mobile device capable of communicating with various components and devices within distributed data processing environment100, via network102. In general, mobile device110represents any programmable electronic mobile device or combination of programmable electronic mobile devices capable of executing machine readable program instructions and communicating with other computing devices (not shown) within distributed data processing environment100via a network, such as network102. Mobile device110includes an instance of user interface112. Mobile device110may include internal and external hardware components, as depicted and described in further detail with respect toFIG. 25.

User interface112provides an interface to social media sources118for a user of mobile device110. In some embodiments of the present invention, user interface112may be a graphical user interface (GUI) or a web user interface (WUI) and can display text, documents, web browser windows, user options, application interfaces, instructions for operation, and include the information (such as graphic, text, and sound) that a program presents to a user and the control sequences the user employs to control the program. In other embodiments, user interface112may also be mobile application software that provides a connection between data generated by a user of mobile device110and server computer104. Mobile application software, or an “app,” is a computer program designed to run on smart phones, tablet computers and other mobile devices. In other embodiments, user interface112may be an external device operatively connected to mobile device110via near field communication or other types of wired and/or wireless technologies.

Mobile device110may also provide mobile sensors114useful to identify users. Mobile sensors114may include cameras, global position system (GPS) sensors, and near-field communication sensors, among others.

Site sensors116may include any device from which an identification of a user may be obtained. For example, site sensors116may include biometric devices such as video cameras, depth sensors, and other similar devices. Site sensors116may also include near-field communication sensors to identify a user via mobile device110.

Social media sources118may include one or a combination of social media websites, focused chat sites or applications, blogs, and wikis, or other content sites. Social media sources118may include text, graphics, pictures, and other content posted by users.

Database108is a repository for data used by user identification program106. Data used by user identification program106may include user data such as images, identification information, user behaviors, social network data, profile data, and other information useful to determine user identification or behavior. In the depicted embodiment, database108resides on server computer104. In another embodiment, database108may reside elsewhere within distributed data processing environment100provided user identification program106has access to database108.

Server computer104includes user identification program106for real-time identification of users. Some embodiments of the present invention use real-time contextual data (e.g., user location) as well as real-time generated insights for a user (e.g., user behavior with respect to products or services), to compare with stored social profiles based on real-time dynamic social data (e.g., user activity in social media) and static social data (e.g., user profile data) for purposes of user identification. Other embodiments of the present invention build subsets of social profiles to match contextual, real-time social media data, and static social media data in order to improve user identification. These aspects of the disclosure are discussed subsequently with reference toFIGS. 2-5.

User identification program106is depicted and described in further detail with respect toFIG. 2. Referring to flowchart200, user identification program106receives identification for a user and analyzes user behavior captured via site sensors116in order to identify the user in the future.

Processing begins at operation255, where user identification program106receives a registration information for a user. In some embodiments of the present invention, the registration information may be received via a registration process where the user provides biometric data that allows high confidence identification of the user by the system. In some embodiments, biometric data includes images received from a photographic or video camera, data received from a depth sensing camera, data received from temperature sensors, or any other data that may be used to identify the user and the user behavior in a physical location such as a retail store. In some embodiments, the biometric data may be obtained via site sensors116. In other embodiments, the user may provide biometric data as part of the registration process. In an example embodiment, a department store runs user identification program106to provide marketing services for the users. In this exemplary embodiment, a user named Ben registers in the user identification program106by providing one or more pictures coupled with personal information (e.g., address, telephone, and other similar information).

Processing continues at operation260, where user identification program106monitors a user by receiving biometric data corresponding to the user. In some embodiments of the present invention, biometric data may be received from site sensors116placed at a specific location (e.g., a store or restaurant). In an example embodiment, Ben walks into the department store a week after registering in user identification program106. Site sensors116begin to monitor Ben by obtaining his biometric data.

Processing proceeds at operation265, where user identification program106determines user behavior based on the biometric data for the user. In some embodiments of the present invention, user identification program106performs sensor based methods (e.g., cameras, depth sensors, temperature sensors, among others) for identifying the users and detecting interactions between the user and the site location using one or more site sensors116. In some embodiments, user behavior is determined by factors such as the time spent looking at a specific product, product characteristics (e.g., clearance or sales products) or product categories (e.g., clothes for 3-5 years old babies), time spent at a specific aisle, identification of products for which the user has performed a price check, path taken in the store, among others.

In some embodiments of the present invention, user identification program106continuously learns and stores the user behavior. In some embodiments, user information and behaviors are added or modified in the user profile as they are discovered. In some embodiments, user information and behaviors are removed or weighted relatively lower for identification purpose as the user information is found to be no longer relevant. In some embodiments, changes to the user information and behaviors (e.g., adding, modifying, removing, or weighting user information and behaviors) may be determined based using existing methods, such as aging of data, and trends, among other similar factors. In some embodiments, user information and behaviors may be determined by probability over a period of time. For example, if a user looked for clothes for a 3-year old baby 2 years ago, user identification program106may identify the user by associating behavior based on clothes for a 5-year old baby.

In some embodiments of the present invention, user identification program106associates the user behavior captured at different times. The association can be done using a path tracking method, where the user is tracked from location A to location B. This path tracking method will ensure it is the same user at location A and location B, and therefore the behavior belongs to the same user.

In our exemplary embodiment, Ben looks for products and services inside the department store while user identification program106analyzes his behavior in order to identify Ben in a future visit to the store.

Processing continues at operation270, where user identification program106stores the user information for identification of the user. In some embodiments of the present invention, the user information, comprising the high confidence identification and the user behavior, is stored in database108for future identification of the user. Continuing our exemplary embodiment, after Ben walks out of the department store, user identification program106stops collecting user information for Ben, creates a user profile for Ben, and stores the collected information in the user profile.

Another aspect of user identification program106is depicted and described in further detail with respect toFIG. 3. Referring to flowchart300, user identification program106receives identification for a user, including social media profiles, and analyzes online user behavior in order to identify the user in the future.

Processing begins at operation355, where user identification program106receives registration information for a user. In some embodiments of the present invention, the registration information for the user may be received via a registration process where the user provides biometric data for high confidence identification by the system. In an example embodiment, Ben registers in the user identification program106by providing one or more pictures coupled with personal information (e.g., address, telephone, and other similar information).

Processing continues at operation360, where user identification program106receives social network information for the user. In some embodiments of the present invention, the user provides links for his/her social media profiles. In other embodiments, user identification program106obtains authorization to access data from the social media profiles provided by the user. In our exemplary embodiment, Ben provides a link to his social media profiles and provides user identification program106with the pertinent authorization to access the social media profiles.

Processing proceeds at operation365, where user identification program106receives online shopping history for the user. In some embodiments of the present invention, user identification program106traverses the social media profiles of the user to determine products that the user has recently bought, expressed sentiment (e.g., liked or followed products), browsed, discussed, or searched for. In other embodiments, user identification program106traverses the social media profiles of the user to determine the shopping preferences of the user (e.g., favorite brands). Continuing our exemplary embodiment, user identification program106receives information about the shopping history and preferences for Ben.

Processing proceeds at operation370, where user identification program106receives additional personal data for the user. In some embodiments of the present invention, user identification program106further receives additional personal data for the user such as calendar information. Continuing our exemplary embodiment, Ben provides a link to his cloud-based calendar and provides user identification program106with the pertinent authorization to access the social media profiles.

Processing continues at operation375, where user identification program106creates a profile for the user and stores the user information for identification. In an exemplary embodiment, user identification program106stores the identification and collected information for Ben in database108.

A further aspect of user identification program106is depicted and described in further detail with respect toFIG. 4. Referring to flowchart400, user identification program106receives a low confidence identification for a user. User identification program106further receives biometric data from site sensors116and identifies user behavior based on the received biometric data. User identification compares the biometric data and the user behavior with stored user profiles in database108.

Processing begins at operation455, where user identification program106receives a biometric data for an unknown user. In some embodiments of the present invention, user identification program106receives a set of images for an unknown user. In this context, unknown user means a user that has not yet been identified either because there is no profile associated with the user or because there is not enough information to identify the user yet. In some embodiments, user identification program106captures biometric data for a user from site sensors116. In an exemplary embodiment, User A walks into a department store where site sensors116begin to capture his biometric data while User A is inside the store. In this exemplary embodiment, user identification program106cannot determine the identity of User A based on the information available.

Processing proceeds at operation460, where user identification program106determines a behavior of the unknown user based on the biometric data. In some embodiments of the present invention, user behavior is determined by factors such as the time spent looking at a specific product, time spent at a specific aisle, identification of products for which the user has performed a price check, among others. In our exemplary embodiment, User A visits the children toy section and later the sports clothing section.

Processing continues at operation465, where user identification program106searches for a matching user in a database based on the biometric data of the user and the behavior of the user. In some embodiments of the present invention, user identification program106searches for a matching user profile in database108. In some embodiments of the present invention, a subset of the user behavior can be used to calculate the confidence score when matching against a user profile. In other embodiments, the percentages or probability from multiple user behaviors can be combined using in a linear combination or any other suitable formulation. In these embodiments, the formulation and appropriateness of the user behavior can be determined through continuous verification of whether a user was correctly identified or not. In some embodiments, the percentages or probability of a behavior can be measured, projected, and/or determined by other existing methods. In other embodiments, the relationship between each behavior from multiple users (both online and in store behavior) can be determined using existing analytics methods including clustering, simulation, and/or other existing methods.

In our exemplary embodiment, the user profile for Ben states 5 different categories that Ben is interested in as follows: (i) Ben stops by the food court 95% of the time, (ii) Ben visits 2-3 categories each visit, (iii) Ben visits the children toy section 40% of the time, (iv) Ben visits the sports clothing section 70% of the time, and (v) Ben visits both the children toy section and the sport clothing section 80% of the time. In our exemplary embodiment, user identification program106identifies the user profile for Ben based on the biometric data and user behavior for User A.

An additional aspect of user identification program106is depicted and described in further detail with respect toFIG. 5. Referring to flowchart500, user identification program106receives contextual data about a user. User identification program106additionally determines one or more subsets of user profiles and compares the contextual data and the user insights with the subsets of user profiles.

Processing begins at operation555, where user identification program106receives user contextual data including biometric data and associated metadata for a user. Examples of user contextual data include GPS position, location name, date, and others. In an exemplary embodiment, User A walks into a department store. User identification program106identifies User A as an unknown user. In our exemplary embodiment, User A looks for products and services inside the department store while user identification program106analyzes his behavior in order to identify User A.

Processing proceeds at operation560, where user identification program106determines a first subset of social media profiles by comparison of the user contextual data and real-time social media updates. In some embodiments of the present invention, user identification program106determines a first subset of social media profiles by comparison of the contextual data with the social media updates, such as geotagging of digital pictures, videos, and postings on social media profiles. Other examples of real-time social updates include mention of locations (e.g., mentioning the name of a store, restaurant, and/or shopping mall), check-in to location in specialized mobile applications, among others. In some embodiments, a list of selection criteria (e.g., geotags, posts about particular products, and others) are assigned a weight based on their relative importance to match users. For example, a list of possible matches for user profiles may be ranked in accordance to each of the selection criteria (e.g., ranking=criteria_1*weight_1+criteria_2*weight_2+ . . . +criteria_n & weight_n). In these and some other embodiments, the user with the highest overall ranking above a predetermined threshold will be selected. In our exemplary embodiment, User A checks-in to the location using a mobile application.

Processing continues at operation565, where user identification program106compares the biometric data with the first subset of social media profiles. In some embodiments of the present invention, user identification program106searches for a matching user profile in database108.

If a match is not found (operation570, “no” branch), processing continues at operation575, where user identification program106determines a second subset of social media profiles by comparison of the user contextual data with static social media data. In some embodiments of the present invention, user identification program106determines a second subset of social media profiles by comparison of the user contextual data with static social media, such as the pages that a user follows in social media profiles. In some embodiments, operation560and operation575may be combined by determining rankings of profiles based on predetermined criteria and relative weights. In these and some other embodiments, the list of selection criteria may comprise real-time and static social network data. In our exemplary embodiment, User A follows the page of the department store where he is shopping.

Processing proceeds at operation580, where user identification program106compares the biometric data with the second subset of social media profiles. If a match is not found (operation585, “no” branch), processing continues at operation590, where user identification program106sends a notification that no match was found for the user.

Conversely, if a match is found (operations570and585, “yes” branch), processing continues at operation595, where user identification program106sends the matched identity of the user for the matching social profile. In our exemplary embodiment, user identification program106identifies the user profile for Ben based on his contextual data and user identification program106sends the user profile.

Server computer104includes user identification program106for real-time identification of users. Some embodiments of the present invention apply local filtering to a set of input images to improve selection of images for user identification. These aspects of the disclosure are discussed subsequently with reference toFIGS. 6-8.

User identification program106is depicted and described in further detail with respect toFIG. 6. Referring to flowchart600, user identification program106receives a set of input images and selects the best images for user identification based on a quality score.

Processing begins at operation655, where user identification program106receives a set of input images. In some embodiments of the present invention, user identification program106captures a set of input images for a user from site sensors116. In our exemplary embodiment, user identification program106receives a set of input images (shown inFIG. 8) for Ben. The set of input images includes a plurality of images of varying quality.

Processing continues at operation660, where user identification program106determines a quality score for each image in the set of input images. The attributes affecting the score may include: (a) resolution; (b) luminance and chroma image noise; (c) focus; (d) user identity; (e) angle; and (f) obstruction. Continuing our exemplary embodiment, user identification program106analyzes each image as follows: (i) the quality score for image802is 50 based on face angle and poor resolution of the image; (ii) the quality score for image804is 65 based on high image noise; (iii) the quality score for image806is 50 based on the focus of the image; (iv) the quality score for image808is 85 based on the face angle of the image; (v) the quality score for image810is 0 based on the subject of the image; (vi) the quality score for image812is 70 based on an obstruction of the face of the user; (vii) the quality score for image814is 50 based on the face angle of the image; and (viii) the quality score for image816is 70 based on the face angle of the image.

Processing proceeds at operation665, where user identification program106determines a set of output images based on the quality score for each image in the set of input images. In some embodiments of the present invention, if there are a plurality of images having similar quality scores and characteristics, user identification program106may include only a single representative image to the set of output images in order to improve storage and bandwidth utilization. In our exemplary embodiment, user identification program106determines a set of user images based on quality score threshold of 70. In this example, user identification program106selects images808,814, and816. User identification program106excludes image812based on an obstruction in the image. Furthermore, user identification program106excludes image808based on its similarity with image814.

Processing continues at operation670, where user identification program106sends the set of output images. In some embodiments of the present invention, user identification program106sends the set of output images to another computer (not shown) through network102. In other embodiments, user identification program106stores the set of output images in database108. In our exemplary embodiment, user identification program106sends the set of output images to a server of a department store and stores the set of output images in database108for future access.

User identification program106is depicted and described in further detail with respect toFIG. 7. Referring to flowchart700, user identification program106receives a set of input images and selects the best images for user identification based on a set of attributes.

Processing begins at operation755, where user identification program106receives the set of input images. In our exemplary embodiment, user identification program106receives a set of input images (shown inFIG. 8) for Ben. The set of input images includes a plurality of images of varying attributes.

Processing proceeds at operation760, where user identification program106receives a set of image attributes for filtering the set of input images. The set of attributes may include: (a) whether the image includes a body; (b) whether the image includes a face; (c) whether the image includes hair; (d) whether the image includes a face in a predetermined angle (e.g., facing front, facing left, etc.); and (e) whether the image includes a person interacting with an object, among others. The attributes may further include static rules (e.g., face at an angle of 30 degrees or all images with a quality score higher than a predetermined threshold). The attributes may also be determined dynamically by user identification program106(e.g., determining a scoring mechanism).

In some embodiments of the present invention, the set of image attributes may be determined by server computer104and received by site sensors116. In these and some other embodiments, site sensors116perform the analysis of input images based on the image attributes determined by server computer104. In an exemplary embodiment, user identification program106receives a set of image attributes including face angle (facing at an angle higher than 0) and interaction with objects. In another exemplary embodiment, server computer104determine image attributes based on the captured images for a user. For example, images for unknown User A is captured. Server computer104identified two known users, Alice and Ann potentially matching this user based on existing image attributes (e.g. long hair, blood hair, 5′10″ height). Based on Alice and Ann, server computer104determines that Alice wears a necklace 95% of time, while Ann never wears a necklace. In this exemplary embodiment, server computer104instructs site sensor116to send only images of User A exposes the neck. In a further exemplary embodiment, other similar user attributes can be identified based on information available at the server computer or using computing resources available at the server computer. In these embodiments, the server computer can determine images attributes (e.g., images of users without necklaces) that does not require significant processing resources and in-depth knowledge of the user. In some embodiments, image attributes may also be selected based on the computing resources available at one or more site sensors116. In these and other embodiments, each site sensor of site sensors116may perform different types of pattern recognition. For example, a first site sensor may be configured to recognize body parts based on relative position from the head. In this example, a second site sensor may recognize the shape of hands or feet in relation to a body and a third site sensor may recognize clothing or clothing accessories. In other embodiments, server computer104dynamically communicates the image attributes to site sensors116.

Processing continues at operation765, where user identification program106determines a set of output images by analyzing which images in the set of input images matches one or more attributes in the set of image attributes. Continuing our exemplary embodiment, user identification program106analyzes each image and determines a set of output images as follows: (i) image808is selected based on a face angle of 15 degrees; (ii) image812is selected based on additional object in the image; (iii) image714is selected based on face angle of 45 degrees; and (iv) image816is selected based on face angle of 90 degrees.

Processing continues at operation770, where user identification program106sends the set of output images. In some embodiments of the present invention, user identification program106sends the set of output images to another computer (not shown) through network102. In other embodiments, user identification program106stores the set of output images in database108. In our exemplary embodiment, user identification program106sends the set of output images to a server of a department store and stores the set of output images in database108for future access.

Some embodiments of the present invention match multiple sub-optimal images of the user to multiple photos in a social media profile or improve confidence score of identifying an individual by using additional sub-optimal images. These aspects of the disclosure are discussed subsequently with reference toFIGS. 9-12.

User identification program106is depicted and described in further detail with respect toFIG. 9. Referring to flowchart900, user identification program106creates and manages temporary user profiles for users in a specific location (e.g., a store or a restaurant).

Processing begins at operation955, where user identification program106receives a set of input images. In some embodiments of the present invention, user identification program106captures a set of input images for a user from site sensors116. In an exemplary embodiment, User A enters a department store and user identification program106receives a set of input images (shown inFIG. 12) corresponding to User A.

Processing continues at operation960, where user identification program106determines whether the user is entering or exiting the premises. If the user is entering the premises (operation965, “enter” branch”), processing proceeds at operation970, where user identification program106determines an identity pattern for a user based on the set of input images. Examples of identity patterns include: (a) body shape; (b) clothing color; (c) clothing patterns; (d) physical appearance; (e) gait; and (f) accessories (e.g., necklace, watch, and similar accessories), among other identity patterns. In our exemplary embodiment, User A enters the department store and user identification program106identifies a pinstriped shirt (seeFIG. 12, profile images1202,1204,1206,1208, and1210).

In some embodiments of the present invention, the identity patterns are unique for each user currently within the premises (i.e., users that have entered the premises but have not yet exited). In these embodiments, user identification program106determines a set of identify patterns for the user when a user enters the premises. These identity patterns are compared with identity patterns of other users within the premises, such that unique identity patterns can be determined. For example, User X entered the premises with a red t-shirt. User identification program106determined that the red t-shirt is a unique identity pattern for User X because there are no other users with red t-shirts currently in the premises. Later, User Y enters the premises with a red t-shirt. In response, user identification program106determines that User X and User Y have an overlapped identity pattern (i.e., the identity pattern of red t-shirt) and re-evaluates the identity patterns for User X and User Y, such that the chosen identity patterns allow unique identification of User X and User Y within all the users within the premises. For example, if User X has black hair and User Y has blonde hair, user identification program106may determine an identity pattern of black hair with red t-shirt for User X and an identity pattern of blonde hair with red t-shirt for User Y. In some embodiments, users are advised of the identity activities performed before entering the location to minimize privacy concerns. In other embodiments, users consent to the identity activities before entering the location, for example, as part of a registration process.

In some embodiments of the present invention, the identity patterns for a user are selected based on the usability of the identity pattern. For example, a unique shoe might not be usable all the time because the shoe might be blocked by obstacle between the camera and the user. Instead, patterns from the upper body might be preferred. In an alternative embodiment, multiple identity patterns might be used for a user, while a subset of the pattern is required for uniquely identifying a user. Continuing the example above, User X might have shoes in orange color, which is unique among all users in the store. In this example, the identity pattern for User X could have three components: orange shoe, black hair, and red t-shirt, where User X can be identified as long as one of the components of the identity pattern matches. In other embodiments, user identification program106might select an identity pattern to use based on needs, efficiency, or any other criteria. For example, if the upper body for a user is blocked, user identification program106can use lower body components of the identity pattern for identification of the user (e.g., shoes in orange color). In the case of a full body image, user identification program106can use the component which is easier to process (e.g., shoes in orange color) to identify the user instead of comparing “black hair and red t-shirt,” which involves multiple components or more complicated pattern matching.

Processing proceeds at operation975, where user identification program106determines an unknown user behavior based on the set of input images. Examples of user behavior include: (a) interaction with objects, and (b) time at specific locations. Continuing our exemplary embodiment, user identification program106tracks the user behavior of User A throughout the store in order to identify his user profile in database108. User A spends twenty minutes in the electronics department looking for flat-screen television sets.

Processing continues at operation980, where user identification program106creates a temporary profile for the user based on the identity pattern and the user behavior. Continuing our exemplary embodiment, user identification program106user identification program106creates a temporary profile including the images and user behavior for User A.

If the user is exiting the premises (operation960, “exit” branch”), processing proceeds at operation985, where user identification program106stores the temporary user profile in database108and deletes the local temporary profile. In our exemplary embodiment, User A exits the store and user identification program106stores the temporary user profile for User A in database108and deletes the local temporary file.

User identification program106is depicted and described in further detail with respect toFIG. 10. Referring to flowchart1000, user identification program106matches temporary user profiles for users in a specific location (e.g., a store or a restaurant) with a stored user profile.

Processing begins at operation1055, where user identification program106receives a set of input images. In some embodiments of the present invention, user identification program106captures a set of input images for a user from site sensors116. In an exemplary embodiment, User A enters a department store and user identification program106receives a set of input images (shown inFIG. 12) for Ben.

Processing continues at operation1060, where user identification program106determines an identity pattern for a user based on the set of input images. Examples of identity patterns include: (a) body shape; (b) clothing color; (c) clothing patterns; (d) physical appearance; (e) gait; and (f) accessories (e.g., necklace, watch, and similar accessories), among other identity patterns. In our exemplary embodiment, User A enters the department store and user identification program106identifies a pinstriped shirt (seeFIG. 12, profile images1202,1204,1206,1208, and1210).

Processing proceeds at operation1065, where user identification program106determines a user behavior based on the set of input images. Examples of user behavior include: (a) interaction with objects, and (b) time at specific locations. Continuing our exemplary embodiment, user identification program106tracks the user behavior of User A throughout the store in order to identify his user profile in database108. User A spends twenty minutes in the electronics department looking for flat-screen television sets.

Processing continues at operation1070, where user identification program106matches the set of input images and the user behavior to a user profile. In our exemplary embodiment, user identification program106matches each profile image1202,1204,1206,1208, and1210in the set of input images with the profile images1212,1214, and1216associated with a user profile for Ben. Profile images1212,1214, and1216associated with the user profile may have been chosen in accordance to one or more embodiments of the present invention based on the quality score and/or the attributes of the image. The results of the image comparisons are as follows: (i) profile image1212matches the profile images1202,1204,1206,1208, and1210with confidence of 0, 0.6, 0.4, 0.2, and 0.7, respectively; (ii) profile image1214matches the profile images1202,1204,1206,1208, and1210with confidence of 0, 0.2, 0.5, 0.3, and 0.6, respectively; and (iii) profile image1212matches the profile images1202,1204,1206,1208, and1210with confidence of 0, 0.3, 0.4, 0.2, and 0.6, respectively. Based on these results, user identification program106identifies Ben as User A. Ben also matches User A's behavior with high confidence (0.8) based on a user profile with an average fifteen minutes spent in the electronics department per visit. Thus, user identification program106identifies matches the temporary user profile of User A to the user profile of Ben.

If a match is not found (operation1075, “no” branch), execution continues at operation1085, where user identification program106may generate a temporary user profile in accordance with the embodiment described inFIG. 9. If a match is found (operation1075, “yes” branch), processing proceeds at operation1080, where user identification program106stores the set of input images and the user behavior under the user profile. In our exemplary embodiment, the set of input images and the user behavior for User A is stored in the user profile for User A in database108.

User identification program106is depicted and described in further detail with respect toFIG. 11. Referring to flowchart1100, user identification program106matches a temporary user profile for users in a specific location (e.g., a store or a restaurant) with a stored user profile.

Processing begins at operation1155, where user identification program106receives a set of input images. In some embodiments of the present invention, user identification program106captures a set of input images for a user from site sensors116. In an exemplary embodiment, User A enters a department store and user identification program106receives a set of input images (shown inFIG. 12) for User A.

Processing continues at operation1160, where user identification program106determines a user behavior based on the set of input images. Examples of user behavior include: (a) interaction with objects, and (b) time at specific locations. Continuing our exemplary embodiment, user identification program106tracks the user behavior of User A throughout the store in order to identify his user profile in database108. User A spends twenty minutes in the electronics department looking for flat-screen television sets.

Processing continues at operation1165, where user identification program106selects a new user profile for matching. In some embodiments of the present invention, selection of the user profile for matching may be based on one or more embodiments described in this disclosure (e.g.,FIG. 2-5).

Processing continues at operation1170, where user identification program106compares the set of input images and the user behavior to the user profile. In our exemplary embodiment, user identification program106matches each profile image1202,1204,1206,1208, and1210in the set of input images with the profile images1212,1214, and1216stored in a user profile for User A. Profile images1212,1214, and1216stored in the user profile may have been chosen in accordance to one or more embodiments of the present invention based on the quality score and/or the attributes of the image. The results of the image comparisons are as follows: (i) profile image1212matches the profile images1202,1204,1206,1208, and1210with confidence of 0, 0.6, 0.4, 0.2, and 0.7, respectively; (ii) profile image1214matches the profile images1202,1204,1206,1208, and1210with confidence of 0, 0.2, 0.5, 0.3, and 0.6, respectively; and (iii) profile image1212matches the profile images1202,1204,1206,1208, and1210with confidence of 0, 0.3, 0.4, 0.2, and 0.6, respectively. User A also matches the user behavior with high confidence (0.8) based on a user profile with an average fifteen minutes spent in the electronics department per visit.

Processing continues at operation1175, where user identification combines multiple confident scores from all profile images and user behavior. In some embodiments of the present invention, the confidence for each profile photo can be combined by any suitable function (e.g., maximum confidence for each image weighted average of the confidence for each image, weighted average of highest score matching for each profile image, or dynamic weight that is learned over time). In our exemplary embodiment, user identification program106combines the confidence scores for the images using a max function. In this case, the maximum confidence is 0.7. In the exemplary embodiment, user identification program106combines the confidence score for the user behavior (0.8) and the confidence scores for the image (0.7) by a weighted average function. In this case, the combined confidence score is 0.7.

In some embodiments of the present invention, combining the confidence scores for each profile photo and behavior can improve over time and a profile photo can be added or removed based on its effectiveness of providing accurate matching based on machine learning. For example, a 90% match with frontal facial image typically means a very good match of captured image and profile photo for User A. If images with 45 degrees of the frontal face are also captured for User A, these images can be compared with all the profile images belonging to the user and profile images belonging to different users. If these captured images match the profile image of User A with 95% confidence while the highest confidence for profile images for other users is 60%, then these captured images will be assigned to User A with a very high weight. In another example, if the captured image for the user has a very low confidence score when compared with profile images for other users, then this captured image can be added as a profile image for the user because no one else matches it. In some embodiments, profile images that are added to a user profile will have a high weight only if the confidence score is high with respect to the matching user. In some embodiments, if the confidence score of matching is low (e.g., less than 60% confident), the profile image will not affect the combined confidence score. In another example, a profile image can be compared with profile images of other users. In some embodiments, the weight for a profile image may be decreased, removed, or remove when weight is below a threshold, if the profile image provides relatively high confidence score of matching other users (e.g., 70% or more).

Processing continues at operation1180, where user identification determines the combined confidence score is higher than or equal to a threshold. In some embodiments of the present invention, the threshold may be a predetermined threshold or an adjustable threshold based on machine learning patterns. If the combined threshold is higher than or equal to the threshold (operation1180, “yes” branch), processing continues at operation1185, where user identification program106receives the matched user profile. In some embodiments, user identification program106receives a user profile ID for the matched user profile. In our exemplary embodiment, the predetermined threshold is 0.7 and the combined confidence score is equal to the predetermined threshold (0.7). Therefore, user identification program106receives the user profile for Ben.

Processing proceeds at operation1190, where user identification program106stores the set of input images and the user behavior under the matched user profile. In our exemplary embodiment, the set of input images and the user behavior for User A is stored in the user profile for Ben in database108.

If the combined threshold is below the threshold (operation1180, “no” branch), processing continues at operation1165, where user identification program106selects a new user profile for matching.

Some embodiments of the present invention provide a passively managed loyalty program based on user attributes and behaviors. Some embodiments of the present invention may also provide additional functionality to traditional loyalty programs and/or actively managed loyalty programs. These aspects of the disclosure are discussed subsequently with reference toFIGS. 13-24.

User identification program106is depicted and described in further detail with respect toFIG. 13. Referring to flowchart1300, user identification program106provides a passively managed loyalty program for users in a specific location (e.g., a store or a restaurant) based on a database of user profiles (e.g., social network profiles, profiles within the brick and mortar retail store database, profiles from the online retailer database, profiles from a distributed and/or shared ledger, among others).

Processing begins at operation1355, where user identification program106receives user attributes and user behaviors. In some embodiments of the present invention, user identification program106captures a set of input images for a user from site sensors116and determines user attributes and user behaviors from the set of input images. In some embodiments of the present invention, receiving the user attributes (e.g., identity patterns) and the user behaviors for a user may be based on one or more embodiments described in this disclosure.

Processing continues at operation1360, where user identification program106matches a user profile based on the user attributes and the user behaviors. In some embodiments of the present invention, user identification program106queries one or more user profiles successively to find a matching user. In some embodiments, querying for a user profile may be based on one or more embodiments described in this disclosure.

Processing proceeds at operation1365, where user identification program106returns a user profile ID. In some embodiments of the present invention, the user profile ID is a numeric or alphanumeric unique identifier (UID) guaranteed to be unique for the matched user profile within a specific client (e.g., a database for a specific retail store). In other embodiments, the user profile ID is a numeric or alphanumeric universally unique identifier (UUID) guaranteed to be unique for the matched user profile for all stored user profiles.

Processing continues at operation1370, where user identification program106stores the user attributes and user behaviors in the user profile. In some embodiments of the present invention, user identification program106uses the user profile ID to access and store the user attributes and user behaviors in the matched user profile.

FIG. 14shows an embodiment of a passively managed loyalty program and a social network managed loyalty program for users in a specific location (e.g., a store or a restaurant) based on social network profiles. Referring to diagram1400, processing begins at operation1455, where retail store1402provides customer characteristics (i.e., user attributes, images of the customer, etc.) and customer behaviors (i.e., user behaviors) for processing by retailer backend office1404. In some embodiments of the present invention, one or more site sensors116at retail store1402capture a set of input images and determine user attributes and user behaviors for one or more users. In some embodiments of the present invention, receiving the user attributes (e.g., identity patterns) and the user behaviors for a user may be based on one or more embodiments described in this disclosure.

Processing continues at operation1460, where retailer backend office1404authenticates to social network1406using a client ID. In some embodiments of the present invention, a second retailer backend office authenticates to social network1406using a second client ID. In some embodiments, the retailer backend office1404authenticates to social network1406with a client ID via an application programming interface (API). In some embodiments, social network1406includes virtual multi-tenant profiles corresponding to different clients (e.g., retail store1402) each virtual multi-tenant profile providing a separate client configuration and data access configuration. These virtual multi-tenant profiles may map to physical storage located on premise, in the cloud, or as part of social network storage (e.g., social network database1408). Data stored in this way can be provided with different access control mechanisms, such as crowd-based anonymization, and other mechanisms.

In an exemplary embodiment shown inFIG. 18, diagram1800, social network database1408may include virtual multi-tenant profiles1802,1804, and1806grouped and isolated from each other in1808and1810. Virtual multi-tenant profiles1802,1804, and1806may include user1812named Patrick and user1814named Mark. In this exemplary embodiment, user1812represents a social network profile while user1814represents a shadow (i.e., temporary) profile.

Processing proceeds at operation1465, where retailer backend office1404provides the customer characteristics (i.e., user attributes) and customer behaviors (i.e., user behaviors) for processing by social network1406. In some embodiments of the present invention, the retailer backend office1404sends the customer characteristics and customer behaviors to social network1406with a client ID via an application programming interface (API). In some embodiments, retailer backend office1404sends the customer characteristics and customer behaviors to social network1406to determine whether the there is a social network profile associated to the user.

Processing continues at operation1470, where social network1406queries social network database1408to match a user based on the customer characteristics (i.e., user attributes) and the customer behaviors (i.e., user behaviors). In some embodiments of the present invention, social network1406performs operation1470at the request of the retailer backend office1404by providing its client ID via the application programming interface (API). In some embodiments, social network database1408resides on the same computer system as social network1406. In some embodiments, social network database1408includes social network profiles for users. Social network profiles for users include images, videos, text, URL links, and user behaviors, among other types of data. Social network1406may query social network1408based on the customer characteristics and the customer behaviors in accordance with one or more embodiments of the present invention.

Processing proceeds at operation1475, where social network1406receives a user profile ID. In some embodiments of the present invention, the user profile ID is a numeric or alphanumeric identification used to identify the user profile in social network1406and social network database1408.

Processing continues at operation1480, where social network1406queries for a unique ID specific to the client ID based on the user profile ID. In some embodiments of the present invention, social network1406performs operation1470at the request of the retailer backend office1404by providing its client ID via the application programming interface (API). In some embodiments, the user profile ID is a numeric or alphanumeric universally unique identifier (UUID) guaranteed to be unique for the matched user profile for all stored user profiles. In these and other embodiments, the social network profile for a user may be accessed via an API call using the unique ID after authenticating to social network1406using a corresponding client ID. Continuing the exemplary embodiment (seeFIG. 18), user1812named Patrick has UUID A for virtual multi-tenant profile1802, UUID B for virtual multi-tenant profile1804, and UUID C for virtual multi-tenant profile1806. Similarly, user1814named Mark has UUID X for virtual multi-tenant profile1802, UUID Y for virtual multi-tenant profile1804, and UUID Z for virtual multi-tenant profile1806.

In some embodiments of the present invention, social network database1408and social network1406may be responsible to store and manages user data belong to the retailer store1402. In these and other embodiments, social network1406may as part of or on behalf of retailer store1402. In some embodiments, retailer store1402may be owned or controlled to a chain of stores corresponding to a parent company. In some embodiments, social network database1408may merge the user data collected from each store belonging to a chain of stores, or track the user data from each store independently and provide grouping for the chain of stores. For example, using customizable access control, both Retail Store X and Retail Store Y belonging to the same parent company may have access to the same data corresponding to User A.

In some embodiments of the present invention, the data belonging to a client in social network profile1812can be used by a different client for selected purposes. For example, Client 1 submits images and behavior corresponding to User A captured by site sensor116to social network1406. In this exemplary embodiment, social network1406can use the data and identity patterns from Client 2 and Client 3 to identify User A. However, Client 1 may not be allowed to retrieve specific customer behavior or purchases from Client 2 or Client 3.

Processing proceeds at operation1485, where social network1406receives the unique ID and sends the unique ID to retailer backend office1404. In some embodiments of the present invention, retailer backend office1404and social network1406may access the social network profile for a user via an API call using the user profile ID or the unique ID.

FIG. 15shows an alternative embodiment of the embodiment shown inFIG. 14for a passively managed loyalty program and a social network managed loyalty program for users in a specific location (e.g., a store or a restaurant) based on social network profiles. Referring to diagram1500, upon completion of operation1485(seeFIG. 14), processing continues at operation1555, where retailer backend office1404receives the unique ID and stores the customer characteristics and the customer behaviors in retailer database1502. In some embodiments of the present invention, the customer characteristics and customer behaviors are stored locally for improved querying of users that have already been identified at retail store1402. In some embodiments, the customer characteristics and customer behaviors are protected to ensure they are not available to third parties for security reasons. In some embodiments, retailer database1502may be any database within any server computer for storing user profiles such as social network profiles, profiles within the brick and mortar retail store database, profiles from the online retailer database, profiles from a distributed or shared ledger, among others. In some embodiments, the customer characteristics and customer behaviors are stored in a shadow (i.e., temporary) profile database1504temporarily while retailer backend office1404has not yet received a unique ID for a user (seeFIG. 19). In these embodiments, the shadow profile corresponding to a user is merged with the user profile to retailer database1502when a unique ID is received. In some embodiments of the present invention, the user profile may be stored in single-tenant storage based on a single retail store1402or multi-tenant fashion based on one or more retail stores, where each retail store1402is treated as a different client (seeFIG. 18).

FIG. 16shows yet another alternative embodiment of the embodiment shown inFIG. 14for a passively managed loyalty program and a social network managed loyalty program for users in a specific location (e.g., a store or a restaurant) based on social network profiles and local customer profiles. Referring to diagram1600, upon completion of operation1485(seeFIG. 14), processing continues at operation1655, where retailer backend office1404creates a local customer profile at retailer database1502after receiving the unique ID for a user. In some embodiments of the present invention, the local customer profile is used for local identification of users in retail store1402(seeFIG. 17).

Processing proceeds at operation1660, where retailer backend office1404stores data associated to the unique ID for a user in retailer database1502. In some embodiments of the present invention, retailer backend office1404accesses social network1406via an API call and downloads the user profile associated to the unique ID.

Processing continues at operation1665, where retailer backend office1404performs a periodic sync up of the images and user attributes from social network database1408. In some embodiments of the present invention, retailer backend office1404periodically accesses social network1406via an API call and updates the user profile associated to the unique ID.

FIG. 17shows still another embodiment of the embodiment shown inFIG. 16for a passively managed loyalty program for users in a specific location (e.g., a store or a restaurant) based on social network profiles stored in a retailer database. Referring to diagram1700, processing begins at operation1755, where retail store1402provides customer characteristics (i.e., user attributes) and customer behaviors (i.e., user behaviors) for processing by retailer backend office1404. In some embodiments of the present invention, one or more site sensors116at retail store1402capture a set on input images and determine user attributes and user behaviors for one or more users. In some embodiments of the present invention, receiving the user attributes (e.g., identity patterns) and the user behaviors for a user may be based on one or more embodiments described in this disclosure.

Processing continues at operation1760, where retailer backend office1404queries a retailer database1502for a user based on customer characteristics and customer behaviors. Retailer backend office1404may query retailer database1502based on the customer characteristics and the customer behaviors in accordance with one or more embodiments of the present invention.

Processing proceeds at operation1765, where retailer backend office1404receives a user ID from retailer database1502. In some embodiments of the present invention, the user profile ID is a numeric or alphanumeric identification used to identify the user profile in retailer backend office1404and retailer database1502.

Processing continues at operation1770, where retailer backend office1404stores customer behaviors and customer characteristics on retailer database1502. In some embodiments of the present invention, the customer characteristics and customer behaviors are stored locally for improved querying of users that have already been identified at retail store1402. In some embodiments, the customer characteristics and customer behaviors are protected to ensure they are not available to third parties for security reasons. In some embodiments, the customer characteristics and customer behaviors are stored locally for improved querying of users that have already been identified at retail store1402. In some embodiments, retailer database1502may be any database within any server computer for storing user profiles such as social network profiles, profiles within the brick and mortar retail store database, profiles from the online retailer database, profiles from a distributed digital ledger, among others. In some embodiments, the customer characteristics and customer behaviors are stored in a shadow (i.e., temporary) profile database1504temporarily while retailer backend office1404has not yet received a unique ID for a user (seeFIG. 19). In these embodiments, the shadow profile corresponding to a user is merged with the user profile to retailer database1502when a unique ID is received (i.e., the shadow profile collected sufficient information and the identity of the user can be determined). In some embodiments of the present invention, the user profile may be stored in single-tenant storage based on a single retail store1402or multi-tenant fashion based on one or more retail stores, where each retail store1402is treated as a different client (seeFIG. 18).

User identification program106is depicted and described in further detail with respect toFIG. 19. Referring to flowchart1900, user identification program106provides a passively managed loyalty program for users in a specific location (e.g., a store or a restaurant) based on shadow (i.e., temporary) profiles while existing user profiles are identified in a social network. In other embodiments, the customer characteristics and customer behaviors are stored in a shadow (i.e., temporary) profile while the corresponding customer has not been identified or associated with a user profile. In these embodiments, the shadow profile corresponding to a user is merged with the user profile when the customer is identified.

Processing begins at operation1360, where user identification program106queries for a user profile based on the user attributes and the user behaviors. In some embodiments of the present invention, user identification program106queries one or more user profiles successively to find a matching user. In some embodiments of the present invention, querying for a user profile may be based on one or more embodiments described in this disclosure.

If a match is found (operation1955, “yes” branch), processing proceeds at operation1365, where user identification program106returns the user profile ID matching the user attributes and the user behavior. If a match is not found (operation1955, “no” branch), processing continues at operation1960, where user identification program106creates a shadow profile for the user. In some embodiments of the present invention, the shadow profile is a temporary profile for a user that does not have a stored user profile in database108or social media sources118. In other embodiments, the shadow profile is a temporary profile for a user that has not yet been matched with a stored user profile based on the user attributes and the user behaviors.

Processing proceeds at operation1965, where user identification program106receives user attributes and user behaviors. In some embodiments of the present invention, user identification program106continues receiving user attributes and user behaviors corresponding to the shadow profile. In some embodiments of the present invention, receiving the user attributes (e.g., identity patterns) and the user behaviors for a user may be based on one or more embodiments described in this disclosure.

Processing continues at operation1970, where user identification program106determines if the shadow profile matches an existing user profile. In some embodiments of the present invention, determining if the shadow profile matches an existing user profile may be performed in accordance to one or more embodiments of the present disclosure. If a match is not found (operation1975, “no” branch), processing proceeds at operation1965, where user identification program106further receives user attributes and user behaviors. If a match is found (operation1975, “yes” branch), processing proceeds at operation1980, where user identification program106merges the shadow profile with the existing user profile and returns the profile ID. In some embodiments of the present invention, the user attributes and user behaviors are stored in the existing user profile. In other embodiments, the user attributes and user behaviors in the shadow profile are compared to the user attributes and user behaviors in the existing user profile. In yet other embodiments, selection of the user attributes and user behaviors that will be stored in the existing user profile may be performed in accordance to one or more embodiments described in this disclosure.

FIG. 20shows still another alternative embodiment of the embodiment shown inFIG. 14for a passively managed loyalty program for users in a specific location (e.g., a store or a restaurant) based on shadow profiles. Referring to diagram2000, upon completion of operation1470(seeFIG. 14), processing continues at operation2055, where social network1406creates a shadow profile at social network database1408if no user matches the customer characteristics and the customer behaviors. In some embodiments of the present invention, the customer characteristics and customer behaviors are stored in the shadow (i.e., temporary) profile while the corresponding customer has not been identified or associated with a user profile. In some embodiments, the shadow profile is created for users that have no known identity and/or users without a shadow profile.

Processing proceeds at operation2060, where social network1406receives the shadow profile ID from social network database1408. In some embodiments of the present invention, the shadow profile ID is a numeric or alphanumeric unique identifier (UID) guaranteed to be unique for the matched user profile within a specific client (e.g., a database for a specific retail store) or a numeric or alphanumeric universally unique identifier (UUID) guaranteed to be unique for the matched user profile. In some embodiments, the shadow profile ID corresponds to a new shadow profile created for a user that without a known identity and for which a shadow profile has not yet been created. In other embodiments, the shadow profile ID corresponds to an existing shadow profile for which there is no known user identity.

Processing continues at operation2065, where social network1406stores the customer characteristics and the customer behaviors under the shadow profile ID and the client ID. In some embodiments of the present invention, social network1406stores the customer characteristics and customer behaviors in the shadow profile for future identification of users that do not currently have an associated user profile in social network1406. In these embodiments, the user will continue to be identified based the customer characteristics and customer behaviors.

Processing proceeds at operation2070, where social network1406periodically evaluates all shadow profiles to find a matching social network profile. In some embodiments of the present invention, once the user has been associated with a user profile in social network1406(e.g., the user registers an account in the social network1406or signs up for the loyalty program at retail store1402) all the customer characteristics and customer behaviors are copied to the user profile and the shadow profile is deleted. In other embodiments, the shadow profile and the user profile are compared for selection of customer characteristics and customer behaviors that help identify the user more efficiently. In some embodiments, selecting the customer characteristics (e.g., identity patterns) and the customer behaviors for a user may be based on the user profile or the shadow profile and/or one or more embodiments described in this disclosure.

User identification program106is depicted and described in further detail with respect toFIG. 21. Referring to flowchart2100, user identification program106provides a passively managed loyalty program including storing transactions made at a point of sale (POS) system.

Processing begins at operation2155, where user identification program106receives customer identity for a user. In some embodiments of the present invention, the customer identity may include customer characteristics and customer behaviors. In other embodiments, receiving the customer identity for a user may be based on one or more embodiments described in this disclosure.

Processing continues at operation2160, where user identification program106receives a transaction data for the user from a point of sale (POS) system. In some embodiments of the present invention, any transactions performed at the POS are logged at a backend service for determining award points and/or advertisements for the user. In some embodiments, the transaction data may include the following data, among others: (a) list of items; (b) price of items; (c) coupons or offers applied, if any; (d) total price of sale; (e) method of payment; (f) time and date.

Processing proceeds at operation2165, where user identification program106associates the transaction data to a temporary user ID. In some embodiments of the present invention, a temporary user ID is assigned to the transaction data to log the transaction data for the user while the user is identified in asynchronous form.

Processing continues at operation2170, where user identification program106receives a social network user ID associated to the user. In some embodiments of the present invention, requests and asynchronously receives the social network user ID associated to the user. In these embodiments, the transaction data is associated to a temporary user ID while the social network user ID has not yet been received.

Processing proceeds at operation2175, where user identification program106replaces the temporary user ID with the social network user ID. In some embodiments of the present invention, the temporary user ID is replaced with the social network user ID when it becomes available.

Processing continues at operation2180, where user identification program106stores the transaction data for user ID. In some embodiments of the present invention, once the transaction data is associated to the social network user ID, user identification program106may store the transaction in database108as part of the customer characteristics and the customer behaviors used to identify the user and/or provide analytics for the user.

FIG. 22shows an embodiment for a passively managed loyalty program and a social network managed loyalty program for users in a specific location (e.g., a store or a restaurant) that associates transactions made at a point of sale device with a social network profile. Referring to diagram2200, processing begins at operation2255, where retailer backend office1404receives a customer identity from a retail store camera2202. In some embodiments of the present invention, receiving the customer identity for a user may be based on one or more embodiments described in this disclosure.

Processing continues at operation2260, where retailer backend office1404sends the customer identity to social network1406for identification of the user. In some embodiments of the present invention, social network1406may require authentication of retailer backend office1404via an API call. In these and other embodiments, retailer backend office1404asynchronously requests the social network user ID from social network1406and continues to process the transaction data at POS2204.

Processing proceeds at operation2265, where retailer backend office1404receives POS data, including a transaction ID, from POS2204. In some embodiments of the present invention, the transaction data may include the following data, among others: (a) list of items; (b) price of items; (c) coupons or offers applied, if any; (d) total price of sale; (e) method of payment; (f) time and date.

Processing continues at operation2270, where retailer backend office1404associates the transaction ID to a temporary user ID. In some embodiments of the present invention, a temporary user ID is assigned to the transaction data to log the transaction data for the user while the user is identified in asynchronous form.

Processing proceeds at operation2275, where retailer backend office1404receives a social network user ID for the user from social network1406. In some embodiments of the present invention, requests and asynchronously receives the social network user ID associated to the user. In these embodiments, the transaction data is associated to a temporary user ID while the social network user ID has not yet been received.

Processing continues at operation2280, where retailer backend office1404replaces the temporary user ID with the social network user ID for the transaction ID. In some embodiments of the present invention, the temporary user ID is replaced with the social network user ID when it becomes available. In these embodiments, replacing the temporary user ID with the social network user ID allows identification of the transactions made by the user.

Processing proceeds at operation2285, where retailer backend office1404stores a transaction ID for the user ID in retailer database1502. In some embodiments of the present invention, once the transaction data is associated to the social network user ID, user identification program106may store the transaction in retailer database1502as part of the customer characteristics and the customer behaviors used to identify the user and/or provide analytics for the user.

User identification program106is depicted and described in further detail with respect toFIG. 23. Referring to flowchart2300, user identification program106provides a passively managed loyalty program including analytics of retail data and social network data for users.

Processing begins at operation2355, where user identification program106receives a request for analytics on retail data from a retail store and social network data from a social network for a user. In some embodiments of the present invention, user identification program106performs one or more analytics of the customer characteristics and customer behaviors associated to the users to determine one or more advertisement strategies.

Processing continues at operation2360, where user identification program106receives retail data and social network data. In some embodiments of the present invention, user identification program106receives data stored at database108related to retail data (e.g., customer characteristics and customer behaviors at one or more retail stores) and social network data (e.g., followed brands, photographed items, items placed in shopping carts, among others).

Processing proceeds at operation2365, where user identification program106performs analytics on retail data and social network data to determine an advertisement strategy for a user. In some embodiments of the present invention, analytics may be performed based on: (i) time spent at specific departments; (ii) time spent looking at specific items; (iii) items bought; (iv) items returned; (v) items placed in shopping cart; (vi) items liked or searched for in a social network; (vii) items saved or photographed in a social network; (viii) specific brands followed in social networks; among other factors.

Processing continues at operation2370, where user identification program106sends the advertisement strategy to the retail store and the social network. In some embodiments of the present invention, the advertisement strategy is sent to one or more retail stores, one or more mobile devices for the user, among other devices. In some embodiments, the advertisement strategy may include text, picture, videos, coupons, buy more and save campaigns, subscription for future shipments, installation of retailer mobile apps, additional offerings such as delivery to home, and/or other marketing and advertisement mechanism. In some embodiments, the advertisement strategy may be a real-time strategy (e.g., by text messaging or instant notification on mobile device) or it may be a delayed strategy (e.g., by showing ads on a search engine at a future time).

FIG. 24shows an embodiment of a passively managed loyalty program for users in a specific location (e.g., a store or a restaurant) that provides advertisement strategies based on social network profiles. Processing begins at operation2455, where retailer backend office1404sends a request for analytics of retail data to social network1406. In some embodiments of the present invention, user identification program106performs one or more analytics of the customer characteristics and customer behaviors associated to the users to determine one or more advertisement strategies.

Processing proceeds at operation2465, where social network1406receives retail data and social network data from social network database1408. In some embodiments of the present invention, social network1406may also receive retail data from retailer database1502. In some embodiments, retail data may include, for example, customer characteristics and customer behaviors at one or more retail stores, and social network data may include, for example, followed brands, photographed items, items placed in shopping carts, among others.

Processing continues at operation2470, where social network1406performs analytics to develop an advertising strategy based on the retail data and the social network data. In some embodiments of the present invention, analytics may be performed based on: (i) time spent at specific departments; (ii) time spent looking at specific items; (iii) items bought; (iv) items returned; (v) items placed in shopping cart; (vi) items liked or searched for in a social network; (vii) items saved or photographed in a social network; (viii) specific brands followed in social networks; among other factors.

Processing proceeds at operation2475, where social network1406sends the analytics results to retailer backend office1404. In some embodiments of the present invention, the analytics results are directed to determine an advertising strategy based on the retail data and the social network data.

Processing continues at operation2480, where retailer backend office1404determines an advertising strategy. In some embodiments of the present invention, the advertising strategy comprises items and/or offers that the user may be interested in based on past behavior.

Processing continues at operation2485, where retailer backend office1404requests advertising to the user on social network1406.

Processing continues at operation2490, where retailer backend office1404requests advertising at retail store1402. In some embodiments of the present invention, retailer backend office1404may also request advertising at mobile device110(seeFIG. 1). In some embodiments, advertising at retail store1402may include advertisements or offers in a receipt or similar manner.

Communications fabric2502can be implemented with any architecture designed for passing data and/or control information between processors (such as microprocessors, communications and network processors, etc.), system memory, peripheral devices, and any other hardware components within a system. For example, communications fabric2502can be implemented with one or more buses.

User identification program106is stored in persistent storage2508for execution by one or more of the respective computer processors2504via one or more memories of memory2506. In this embodiment, persistent storage2508includes a magnetic hard disk drive. Alternatively, or in addition to a magnetic hard disk drive, persistent storage2508can include a solid state hard drive, a semiconductor storage device, read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, or any other computer-readable storage media that is capable of storing program instructions or digital information.

The media used by persistent storage2508may also be removable. For example, a removable hard drive may be used for persistent storage2508. Other examples include optical and magnetic disks, thumb drives, and smart cards that are inserted into a drive for transfer onto another computer-readable storage medium that is also part of persistent storage2508.

Communications unit2510, in these examples, provides for communications with other data processing systems or devices, including resources of distributed data processing environment100. In these examples, communications unit2510includes one or more network interface cards. Communications unit2510may provide communications through the use of either or both physical and wireless communications links. User identification program106may be downloaded to persistent storage2508through communications unit2510.

I/O interface(s)2512allows for input and output of data with other devices that may be accessible to computing device2505and server computer104, such as mobile device110, site sensors116, social media sources118, and other computing devices (not shown). For example, I/O interface2512may provide a connection to external devices2518such as a keyboard, keypad, a touch screen, and/or some other suitable input device. External devices2518can also include portable computer-readable storage media such as, for example, thumb drives, portable optical or magnetic disks, and memory cards. Software and data used to practice embodiments of the present invention, e.g., user identification program106can be stored on such portable computer-readable storage media and can be loaded onto persistent storage2508via I/O interface(s)2512. I/O interface(s)2512also connect to a display2520.