Abstract:
A system and method for advertising in augmented reality to a user uses an augmented reality device and a processing device in communication with the augmented reality device. The processing device is also adapted to communicate with a network. Media scrape and advertising databases are in communication with the processing device. The advertising database stores advertising data. The processing device contains software or is programmed to receive an image from the augmented reality device, scrape social media data relating to the user stored on the network, store the scraped social media data on the media serape database, compare the image to the scraped social media data to determine if there is a connection between the user and the image, compare the image to the advertising data to determine if there is a connection between the user and the image, generate an advertisement using advertising data corresponding to the image and transmit the advertisement to the augmented reality device for viewing by the user.

Description:
CLAIM OF PRIORITY 
       [0001]    This application claims priority to U.S. Provisional Patent Application Ser. No. 61/644,573, filed May 9, 2012, the contents of which are hereby incorporated by reference, 
     
    
     FIELD OF THE INVENTION 
       [0002]    The present invention relates generally to augmented reality devices and systems and, in particular, to an advertising system and method for such devices and systems. 
       BACKGROUND 
       [0003]    Augmented reality provides a user with a live view of a physical, real-world environment, augmented with artificial computer-generated sound, video and/or graphic information A device typically displays the live view of the physical, real-world environment on a screen or the like, and the artificial, computer-generated information is overlaid on the user&#39;s live view of the physical, real-world environment. This is in contrast with vitrual reality, where a simulated environment entirely replaces environment. 
         [0004]    Augmented reality is beginning to be incorporated into and used on smart phones and other personal display devices, using GPS and other technology. Furthermore, personal display devices that are especially suited to implement augmented reality technology, such as eyeglasses (GOOGLE GLASS), head-mounted displays and even contact lens and virtual retinal displays, are being developed. 
         [0005]    Augmented reality provides a tremendous opportunity for businesses to advertise to individuals using the technology. A need exists for a system and method that provides advertisements to a user that target the user&#39;s interests and environment. In addition, it would be desirable for such a system and method to use information mined from social networking websites to generate advertising for use as overlays in augmented reality. Furthermore, it would be desirable for such a system and method to target the advertising based on a user&#39;s real-time location and view. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0006]      FIG. 1  is a block diagram an embodiment of the augmented reality advertising system of the invention; 
           [0007]      FIG. 2  is a flowchart illustrating an embodiment of the augmented reality advertising method of the present invention as performed by the system of  FIG. 1 ; 
           [0008]      FIGS. 3A and 3B  are flowcharts illustrating the calculation of the location and viewing angle in an embodiment of the method of  FIG. 2 ; 
           [0009]      FIG. 4  is a block diagram illustrating an embodiment of the architecture of the server of the augmented reality advertising system of  FIG. 1 ; 
           [0010]      FIG. 5  is an illustration of a display viewed by a user in an embodiment of the system and method of the invention. 
       
    
    
     DETAILED DESCRIPTION OF EMBODIMENTS 
       [0011]    Embodiments of the invention use information available from a user&#39;s social networking and the augmented reality viewed by the user to create targeted advertisements and further overlay such advertisements on top of the user&#39;s viewed augmented reality. 
         [0012]    An embodiment of the system of the present invention is illustrated in  FIG. 1 . As illustrated in  FIG. 1 , the system includes an augmented reality display device  8 , which may be any device capable of creating an augmented reality for a user to view. For example, such a device may be, but is not limited to, a computer display, smart phone, head-mounted display or GOOGLE GLASS device. A further example is provided in U.S. Patent Application Publication No. US 2010/0164990 A1. U.S. patent application Ser. No. 12/063,145 to Van Doom, the contents of which are hereby incorporated by reference. The augmented reality device includes a central processing unit (CPU), microcontroller or similar central processing or controller device or circuitry  10 . 
         [0013]    The augmented reality device  8  includes a display  12  and an image capture device  14  (such as a camera, sensor or the like). The image capture device, as the name implies, “views” images and transmits them to the display via the CPU  10  to provide the user with a real-time view of the user&#39;s physical, real-world environment. In addition, the augmented reality display device is capable of receiving and displaying information, that is generated in accordance with embodiments of the invention as described below, and overlaying it on the user&#39;s live view of the physical, real-world environment. 
         [0014]    The augmented reality device is also preferably provided with an accelerometer device  16  and a GPS module  20  that communicate with CPU  10 . These components are used to calculate the coordinates of a vector (V) representing user&#39;s view for use by the system in determining what a user is viewing through the image capture device  14  and display  12 . The use of these components in calculating vector V will now be explained with reference to  FIGS. 2 ,  3 A and  3 B. 
         [0015]    As illustrated in  FIG. 1 . The GPS module  20  of the augmented reality device  8  communicates with a GPS system  22  via wireless communications link  24 . Such GPS systems are well known in the art. As illustrated at block  26  of  FIG. 2 , the GPS system and module determines the user&#39;s location in terms of global position coordinates x, y and z. As indicated by block  28  of  FIG. 3A , this information is provided to the augmented reality device CPU  10  ( FIG. 1 ). As indicated at  30  in  FIG. 3A , any movement of the user is also tracked by the GPS system and module and provided to the augmented reality device CPU. 
         [0016]    As indicated at  31  and  32  in  FIG. 2 , in addition to knowing the location (x, y, z) of the user via the GPS system and module, the system needs to track the movement of the augmented reality image capture device ( 14  of  FIG. 1 ) in terms of the viewing, angles theta (θ) and phi (φ) to determine what the user is viewing via the device display ( 12  of  FIG. 1 ). The direction of viewing by the image capture device is illustrated as the angle theta (θ) and phi (φ) relative to the initial or previous viewing vector represented by N in  FIG. 2  (at  32 ). With reference to  FIG. 1 , this may be accomplished via the accelerometer  16 . More specifically, as illustrated in  FIG. 3A , the CPU of the augmented reality device receives data from the accelerometer at  34 . As a result, the CPU checks for viewing angle movement by checking tier non-zero accelerometer signals at  38 . 
         [0017]    As indicated at  44  in  FIG. 3A , any movement detected by the accelerometer or GPS results in the CPU calling the V calculation subroutine. An acceleration vector (DeltaA)  46  and GPS position vector (DeltaPos)  48  are used to calculate view angles theta and phi ( 32  of  FIG. 2 ) as indicated at block  54  of  FIG. 3B . As indicated by block  56  of  FIG. 3B , the subroutine first removes the components of the UPS position vector in the x, y and z directions (DeltaPosX. DeltaPosY, DeltaPosZ) after calculating the velocity of DeltaPosX, DeltaPosY and DeltaPosZ so that they do not interfere with the accelerometer readings in the x, y and z directions (DeltaA.X, DeltaA.Y, DeltaA.Z). Next as indicated at blocks  58  and  62 , theta and phi are calculated using the a sin function in the equations: 
         [0000]      Theta=a sin (DeltaA.y/|g|) 
         [0000]      Phi=a sin (−DeltaA.X/(|g|*cos (a sin (DeltaA.Y/|g|)))
 
         [0018]    Where |g| is the gravitational field vector from the accelerometer (as is typically provided by accelerometer devices). 
         [0019]    As illustrated in  FIG. 1 , the augmented reality device may optionally or alternatively include a vector magnetometer  64  as a redundancy or so as to serve as a back-up for the accelerometer  16 . As illustrated in  FIGS. 3A and 3B , the CPU of the augmented reality device receives data from the magnetometer at  66 . As a result, the CPU checks (hr viewing angle movement by checking for movement in the magnetometer at  68 . As indicated at  44  in  FIG. 3A , any movement detected by the magnetometer results in the CPU caning the V calculation subroutine. A magnetometer acceleration vector (DeltaG)  72  and position vector  48  (from the GPS) are used to calculate view angles theta and phi ( 32  of  FIG. 2 ) as indicated at block  54  of  FIG. 3B . As yet another option or alternative for determining the viewing angle of the image capture device (and thus the user), prior art GPS systems often include a “heading” feature which can determine the direction of the device incorporating the GPS system. As a result, embodiments of the system could optionally or alternatively use the heading feature of the GPS system to determine the initial viewing angle theta. 
         [0020]    As indicated at  74  in  FIG. 3B , the GPS position vector is used to determine the Correct current values for x, y and z, as indicated in blocks  76 ,  78  and  82  using the equations: 
         [0000]      x+=DeltaPosX 
         [0000]      y+=DeltaPosY 
         [0000]      z+=DeltaPosZ 
         [0000]    More specifically, x, y and z remain the same if they already equal DeltaPosX, DeltaPosY and/or DeltaPos Z, or, if not, the values of DeltaPosX, DeltaPosY and/or DeltaPosZ are added to x, y and/or z. 
         [0021]    As indicated at block  84  of  FIG. 3B , and block  86  of  FIG. 2 , the values determined for x, y, z, theta and phi are assembled to form vector V. 
         [0022]    Additional information and optional additional features relating to the calculation of the vector V (or point of interest/POI) may be found in U.S. Patent Application Publication No. US 2013/0046461 A1, U.S. patent application Ser. No. 13/213,492 to Balloga, the contents of which are hereby incorporated by reference. 
         [0023]    As indicated in  FIG. 1 , the augmented reality device communicates with a processing device, such as server  88 , via a network connection  92 , such as a wireless Internet connection. As an alternative to the Internet, the network could be a private network or a private/public network. The augmented reality device  8  may communicate with the server  88  using an alternative type of wireless connection for any other type of connection). The server includes a processor or microprocessor and memory storage. The software for performing the functions described below is loaded onto the server as are the databases that store the data entered into and processed by the software. In alternative embodiments, the various databases described below may be stored on one or more computer devices separate from the server  88 . 
         [0024]    As illustrated in  FIG. 2 , the location and viewing angle calculated at  94  in the form of vector V and the view received by the user via the image capture device at  96  are passed to the server at  98 . 
         [0025]    As illustrated in  FIG. 4 , the server  88  includes a log-in module  102 , which enables the user to log onto the system. Once the user logs in with his or her user name or password, which is transmitted from the user device, the data (vector V and images from the augmented reality device) are received by a location module  104  of the server. The user&#39;s username and password are stored on a user information database  106 . 
         [0026]    In addition, the user information database  106  includes information about the user, including, but not limited to, personal user data such as address, educational background, marital status, family information, pet information and the names of friends. This information may be entered by the user when he or she registers to use the system. 
         [0027]    The server of  FIG. 4  also includes a social media scrape module  108 . In addition to the user&#39;s personal information and username and password for the server  88 , the user information database  106  includes the user&#39;s usernames and passwords for all of the user&#39;s social media websites. These usernames and passwords are provided to a social media scrape module  108 . The social media scrape module accesses social media websites  112  through network  92  and accesses the user&#39;s social media data after logging on to the social media websites using the user&#39;s social media website usernames and passwords. The social media scrape module  108  then scrapes the users social media data for information that may be relevant and of interests to businesses and advertisers. This information may include, but is not limited to:
       a. Likes/dislikes   b. Mentions in posts   c. Captions in pictures   d. Comments in pictures   e. Pictures at location   f. Friends of friend&#39;s mentions
 
The user&#39;s personal information stored on the user information database  106  may be used by the social media scrape module  108  to assist in identifying relevant data during the social media scraping (but use of the personal information is not mandatory).
       
 
         [0034]    Examples of the social media websites  112  include, but are not limited to, Facebook, Linkedin, MySpace, Pinterest, Tumblr, Twitter, Google+, DeviantArt, LiveJournal, Orkut, Flickr, Sina Weibo, Vkontakte, Renren, Douban, Yelp and Mixi, Qzone. As a result, data regarding the user&#39;s likes, interests, hobbies, travel preferences and the like is collected from the user&#39;s pages on the social media websites  112 . Examples of the social media scrape techniques and systems that may be used by social media scrape module  108  include, but are not limited to, those presented in U.S. Patent Application Publication No. US 2013/0035982 A1, U.S. patent application Ser. No. 13/368,515 to Zhang et al., the contents of which are hereby incorporated by reference and U.S. Patent Application Publication No. US 2013/0073374 A1. U.S. patent application Ser. No. 13/233,352 Heath, the contents of which are also hereby incorporated by reference. 
         [0035]    In addition to scraping the user&#39;s social media web pages, the social media scrape module  108  may scrape data from the social media web pages of friends of the user and store it on the social media scrape database  114 . Such friends may be identified by the social media scrape module using data from the user information database  106  (such as a list of friends) or from the user&#39;s social media web pages (for example, “friends” on Facebook). 
         [0036]    The data obtained by the social media scrape performed by the social media scrape module  108  is stored on social media scrape database  114 , with the user identifier as the key. The social media scrape module  108  regularly scrapes information available on the user&#39;s pages on the social media websites  112  so that current information for the information is stored on the media scrape database  114 . 
         [0037]    The location module  104  also has access to a personal image capture database  110  upon which the location module stores images of locations frequently viewed by the user which. As a result, pattern data in terms of images of locations, businesses, etc. frequently visited by the user are stored on the personal image capture database  110  for access by the location module  104 . Businesses and locations frequently visited by the user, for example, may be considered the same as the user liking such businesses and locations or having an interest in the subject matter of such businesses and locations. For example, if the user frequently visits an Italian restaurant, the pattern may indicate that the user likes Italian cuisine. 
         [0038]    As described previously, with reference to block  98  of  FIG. 2 , the location and viewing angle in the form of vector V and the view received by the user via the image capture device at are passed from the augmented reality device to the location module ( 104  of  FIG. 4 ) of the server. This data is used to determine what the user is looking at as follows. 
         [0039]    As indicated by block  120  of  FIG. 2 . the location module compares the user view (from the image capture device  14  of  FIG. 1 ) with a street level mapping database to determine what the user is viewing. More specifically, with reference to  FIG. 4 , the location module  104  communicates through network  92  with a street level mapping database  122 . The location module  104  compares the user view with images on the street level mapping database  122 . When there is a match, the location of the user, and what the user is looking at, may be determined from the street level mapping database. 
         [0040]    As an alternative, the location module may use the technology of U.S. Patent Application. Publication No. US 2012/0310968 A1, U.S. patent application Ser. No. 13/118,926 to Tseng, the contents of which are hereby incorporated by reference, to identify the location based on the viewed objects. 
         [0041]    Alternatively, or as a backup, the location module uses the OPS location and viewing angle (vector V) to determine the user location and what the user is looking at. If the users location and what the user is looking at cannot be determined using the street level mapping database or the GPS position and vector V, the location module searches for matches in the personal image capture database  110 , the social media scrape database  108  and an the social media websites  112  via the social media scrape module. Alternatively, or in addition to the social media websites  112 , the location module may use the social media scrape module to search the Internet in general for images that match the user&#39;s view so that the user&#39;s location and what the user is viewing, may be determined. 
         [0042]    As another alternative, the location module  104  may identify the business being viewed by the user through use of the technology disclosed in U.S. Pat. No, 8,379,912 to Yadid et al., the contents of which are hereby incorporated by reference. 
         [0043]    Once the location module  104  of  FIG. 4  determines where the user is and what the user is looking at, the social media scrape database and personal image capture database are accessed to determine if there is some connection between the user and the location or business being viewed (block  126  of  FIG. 2 ). For example, a user may have lots of information on her social media websites regarding bowling. The user mentions that she is a member of a bowling league on her social medial website, posts photos on the website, posts howling scores on her website, etc. Such a connection exists if the user is viewing a bowling alley. As another example. a user may be viewing an auto parts store, and posts frequently on his social media websites about his classic muscle car. The system identities such connections, which are opportunities for targeted advertising. As still another example, the user walks past and views a restaurant. A friend of the user has recently mentioned liking it in her social media. The location module  104  identifies a connection and advertising opportunity between that business and the user. 
         [0044]    A user may also view an object or person that has a connection with the social media scrape database. Such objects or persons are identified by comparing the user&#39;s view (from the user device  8 ) with photographs stored on the social media scrape database  114  or the personal image capture database  110 . For example, the user frequently looks at PORSCHE automobiles as they drive by. A pattern is established on the personal image capture database that indicates that the user is interested in PORSCHE automobiles. As a result, the location module  104  identifies a connection and advertising opportunity when the user&#39;s view includes a PORSCHE automobile. As another example, a user views a friend coming out of a business. The location module  104  identifies a connection and advertising opportunity between that business and the user (the friend has gone there and possibly likes it). 
         [0045]    As examples only, the location module may use the technology of U.S. Pat. No. 8,341,145 to Dodson et al., the contents of which are hereby incorporated by reference, to recognize the faces of friends of the user, while the technology disclosed in U.S. Patent Application Publication No. US 2012/0310968 A1, U.S. patent application Ser. No. 13/118,926 to Tseng, may be used to identify viewed objects. 
         [0046]    With reference to  FIG. 4 , if there is a connection between the user and the location, business and/or object being viewed, determined as described above, a targeted advertisement generator engine  130  is used to identify and display relevant advertisements to the user (block  131  of  FIG. 2 ). More specifically, the targeted ad generator  130  communicates with an advertising database  132 , upon which advertisements are stored. The advertisements are indexed by the location, business and/or object names or other identifiers. The name or other identifier of the location, business and/or object being viewed, and having a connection with the user, is used by the targeted ad generator  130  to pull corresponding, relevant ads from the advertising database  132 . For example, a connection has been established between a restaurant that the user is viewing and the user (because, using an example from above, a friend of the user indicates she likes the restaurant on her social media). The targeted advertisement generator  130  would retrieve an advertisement for that restaurant from the advertising database, if any such advertisements are present on the advertising database. Continuing with another example from above, the user is viewing a PORSCHE automobile, where a connection has been made between the user and PORSCHE automobiles. If there are any advertisements for PORSCHE automobiles on the advertising database  132 , such advertisements would be retrieved by the targeted ad generator. 
         [0047]    The advertising database  132  of  FIG. 4  may contain “template” style advertisements where information from the social media scrape database  114  or personal image capture database  110  may be inserted by the targeted ad generator to create more personalized advertisements. As a result, the connection information for a viewed location, business and/or object that caused the advertisement to be retrieved may be used in the advertisement. For example, continuing with an example from above, where a user&#39;s friend likes a restaurant viewed by the user, the targeted ad generator could retrieve a template advertisement from the advertising database and insert the friend&#39;s name to generate an advertisement such as “Suzy really likes (restaurant name)”. 
         [0048]    In an alternative embodiment, the targeted ad generator may display the raw social media data that creates the connection between the user and the location, business and/or object. For example, a friend of the user may tweet “I am at Bill&#39;s Tavern,” which will be displayed on the display of the user&#39;s augmented reality device as described below. Such information may be displayed in addition to any advertisements on the advertising database  132  for the location, business or object. Alternatively, the information may be displayed even if no such advertisements exist on the advertising database for the location, business or object. 
         [0049]    If multiple objects and/or businesses for which there exists connections are being viewed by the user, the targeted ad generator may pull a number of corresponding, relevant advertisements from the advertising database. 
         [0050]    The system is constantly identifying the location and view of the user, whether there are any connections and constantly checking if the system has any corresponding, relevant advertisers or advertisements. 
         [0051]    One the targeted advertisements have been generated by the targeted ad generator, they are displayed as banners or textual overlays on the user&#39;s augmented reality device (see block  134  of  FIG. 2 ). With reference to  FIG. 4 , this is accomplished using an image rendering engine  136  which communicates with the user device  8  through network  92 . The image rendering engine translates the advertisements from two-dimensional coordinates (such as a JPEG format picture) of the image as stored on the advertising database  132  to three-dimensional coordinates for the display ( 12  of  FIG. 1 ) of the augmented reality device  8 . This may be accomplished, as an example only, using the technology of U.S. Patent Application Publication No. US 2009/0237328 A1, U.S. patent application Ser. No. 12/051,969 to Gyorfi et al., the contents of which are hereby incorporated by reference. 
         [0052]    A simplified example of a display presented to a user of the augmented reality device ( 8  of  FIGS. 1 and 4 ) is presented in  FIG. 5 . In this example, the user is viewing a business  142  ( FIGS. 4 and 5 ), which is “Justin&#39;s Chicken, Waffles and Beer.” A connection exists in that the user&#39;s friend, Rachael Olson, has indicated on her social media web pages that she likes the restaurant. The targeted ad generator ( 130  of Fig,  4 ) retrieves as template advertisement from the advertising database ( 132  of  FIG. 4 ) and inserts Rachael&#39;s name. In addition, the advertising database contains an advertisement for the restaurant indicating a special for that day only of 18% off. As a result, in addition to the real-time view of the restaurant  142  on the display ( 12  of  FIG. 1 ) of the augmented reality device, the user sees advertising banner  144  ( FIG. 5 ). The banner  144  is essentially a superimposed image of a large sign which appears over the location, business or object and may explain the connection to the user, and display any advertisements or other information for the location, business or object. 
         [0053]    While the preferred embodiments of the invention have been shown and described, it will be apparent to those skilled in the art that changes and modifications may be made therein without departing from the spirit of the invention, the scope of which is defined by the appended claims.