Focus-object-determined communities for augmented reality users

According to one embodiment, a method, computer system, and computer program product for creating a social network in augmented reality (AR) based on focus objects is provided. The present invention may include monitoring an augmented reality session of a user; detecting focus objects of a user during the augmented reality session; labeling the detected focus objects with indications of interest; associating the labeled detected focus objects with additional users; and establishing, based on the labelled detected focus objects, a social network, where the social network contains focus objects sourced from additional users.

BACKGROUND

The present invention relates, generally, to the field of computing, and more particularly to augmented reality.

Augmented reality (AR) is a modern computing technology that uses software to generate images, sounds, haptic feedback, and other sensations to augment a real world environment. While the creation of this augmented environment can be achieved with general-purpose computing devices, such as cell phones, more specialized equipment is also used, typically in the form of glasses or headsets where computer generated elements are overlaid onto a view of the real world by being projected or mapped onto a lens in front of a user's eyes. With the help of computer augmentation, information about the surrounding world of the user, as well as other digital elements overlaid onto the world, become interactive and digitally manipulable. This technology has the potential to transform countless aspects of human life, from construction to military training to space exploration. One salient field where augmented reality is poised to make a significant impact is that of social networking. With the rampant success of such social media platforms as Facebook® (Facebook® and all Facebook®-based trademarks and logos are trademarks or registered trademarks of Facebook, Inc. and/or its affiliates), Instagram® (Instagram® and all Instagram®-based trademarks and logos are trademarks or registered trademarks of Instagram LLC and/or its affiliates) and Snapchat® (Snapchat® and all Snapchat®-based trademarks and logos are trademarks or registered trademarks of Snap Inc. and/or its affiliates), augmented reality seems likely to be the next direction in social networking.

SUMMARY

According to one embodiment, a method, computer system, and computer program product for creating a social network in augmented reality (AR) based on focus objects is provided. The present invention may include monitoring an augmented reality session of a user; detecting focus objects of a user during the augmented reality session; labeling the detected focus objects with indications of interest; associating the labeled detected focus objects with additional users; and establishing, based on the labelled detected focus objects, a social network, where the social network contains focus objects sourced from additional users.

DETAILED DESCRIPTION

Embodiments of the present invention relate to the field of computing, and more particularly to augmented reality (AR). The following described exemplary embodiments provide a system, method, and program product to, among other things, establish a social network within an AR session by locating, tagging, and aggregating focus objects of a user. Therefore, the present embodiment has the capacity to improve the technical field of augmented reality by generating temporary communities of interest for augmented reality, reducing friction to tap a community of interest, and improving the social networking capabilities of AR technology by allowing users to engage efficiently with others of similar interest.

As previously described, AR is a modern computing technology that uses software to generate images, sounds, haptic feedback, and other sensations to augment a real world environment. While the creation of this augmented environment can be achieved with general-purpose computing devices, such as cell phones, more specialized equipment is also used, typically in the form of glasses or headsets where computer generated elements are overlaid onto a view of the real world by being projected or mapped onto a lens in front of a user's eyes. With the help of computer augmentation, information about the surrounding world of the user, as well as other digital elements overlaid onto the world, become interactive and digitally manipulable. This technology has the potential to transform countless aspects of human life, from construction to military training to space exploration. One salient field where augmented reality is poised to make a significant impact is that of social networking. With the rampant success of such social media platforms as Facebook®, Instagram® and Snapchat®, augmented reality seems likely to be the next direction in social networking.

Many AR wearable vendors, such as Facebook®, Google® (Google® and all Google®-based trademarks and logos are trademarks or registered trademarks of Google Inc. and/or its affiliates), and Samsung® (Samsung® and all Samsung®-based trademarks and logos are trademarks or registered trademarks of Samsung Electronics Co., Ltd. and/or its affiliates), are driving changes in user behavior; AR platforms allow users to, among many other things, present contact information, combine reality with games, and overlay map information with reality. AR's combination of live views with digital flags and markers enables useful information to be presented and acted upon; there are many untapped possibilities to change experiences with AR. However, as of yet, there exists few methods in the art that capitalize on the social networking advantages of focus-object-based labeling, and communities of interest based on focus objects. As such, it may be advantageous to, among other things, implement a system capable of tracking the focus of users, labeling objects that are the target of a user's focus, and creating social networks based on the focus objects in order to allow users to connect and interact based on their interest in real world objects.

According to one embodiment, the invention may be a system that integrates communities of interest into augmented reality sessions, by monitoring the augmented reality session of a user, detecting focus objects of a user during the AR session, capturing and labeling the focus objects with an indication of interest, associating the labeled objects with users within a specified area, and establishing, based on the area and focus object, a geo-fenced social network. The monitoring may occur within a geo-fenced area, for a set number of objects or a maximum amount of time. As an example, where a user, named Alice, is using an AR device equipped with the system, the system may initiate monitoring the augmented reality session of Alice. Alice may use the AR device to determine what type of apples she is seeing. The system may detect focus objects of Alice, and label the objects as follows:

Green apple

Red apple

The system may further capture and label the focus objects with an indication of interest, as follows:

The system may further associate the labeled objects with users within a specified area, as follows:

The system may establish a geo-fenced social network based on the area, which aggregates both Alice's objects listed above, and the focus objects of other users listed above. The system may detect an overlap with Bob regarding the green-apple|red-apple labeling, and may establish a temporary social network with Bob. In such a situation, Alice might begin seeing Bob's updates and comments overlaid onto focus objects of his that appear within her vision. For instance, Alice may see a small car through AR, which Bob commented on earlier, and may see a conversation bubble overlaid on top of it containing Bob's comment, “What a perfect car!—Bob.” Alice may then ask Bob questions regarding details of the car, and engage in a conversation with him on the topic, thereby engaging efficiently with others of similar interest.

The following described exemplary embodiments provide a system, method, and program product to establishing a social network within an AR session by locating, tagging, and aggregating focus objects of a user.

Referring toFIG. 1, an exemplary networked computer environment100is depicted, according to at least one embodiment. The networked computer environment100may include client computing device102and a server112interconnected via a communication network114. According to at least one implementation, the networked computer environment100may include a plurality of client computing devices102and servers112, of which only one of each is shown for illustrative brevity.

Client computing device102may include a processor104and a data storage device106that is enabled to host and run an AR program108and an AR focus-object community creation program110A and communicate with the server112via the communication network114, in accordance with one embodiment of the invention. AR program108may be any program capable of overlaying digital elements onto a direct or indirect view or representation of the real world. Client computing device102may be, for example, a mobile device, an AR headset, a telephone, a personal digital assistant, a netbook, a laptop computer, a tablet computer, a desktop computer, or any type of computing device capable of running a program, capturing a user's view, and accessing a network. As will be discussed with reference toFIG. 3, the client computing device102may include internal components302aand external components304a, respectively. Client computing device102may be integrated with or otherwise in communication with a camera for providing a real-time or near-real-time video feed of a user's vision.

The server computer112may be a laptop computer, netbook computer, personal computer (PC), a desktop computer, or any programmable electronic device or any network of programmable electronic devices capable of hosting and running an AR focus-object community creation program110B and a database116and communicating with the client computing device102via the communication network114, in accordance with embodiments of the invention. As will be discussed with reference toFIG. 3, the server computer112may include internal components302band external components304b, respectively. The server112may also operate in a cloud computing service model, such as Software as a Service (SaaS), Platform as a Service (PaaS), or Infrastructure as a Service (IaaS). The server112may also be located in a cloud computing deployment model, such as a private cloud, community cloud, public cloud, or hybrid cloud. The server computer112may be integrated with or otherwise in communication with a camera for providing a real-time or near-real-time video feed of a user's vision.

According to the present embodiment, the AR focus-object community creation program110A,110B may be a program capable of establishing a social network within an AR session by locating, tagging, and aggregating focus objects of a user. The AR focus-object community creation method is explained in further detail below with respect toFIG. 2. The AR focus-object community creation program110A,110B may be a discrete program or it may be a subroutine or method integrated into AR program108. The AR focus-object community creation program110A,110B may be located on client computing device102or server112or on any other device located within network114. Furthermore, AR focus-object community creation program110A,110B may be distributed in its operation over multiple devices, such as client computing device102and server112. AR focus-object community creation program110A,110B may be activated by AR program108, and may be triggered by a gesture by the user, an audio command from the user, or by the user entering coordinates within a geo-fenced location.

Referring now toFIG. 2, an operational flowchart illustrating an AR community creation process200is depicted according to at least one embodiment. At202, the AR focus-object community creation program110A,110B monitors the augmented reality session of a user. The augmented reality session may be a real-time or near-real-time video feed corresponding to the vision of the user, which may have computer generated elements overlaid onto it by AR program108. The AR focus-object community creation program110A,110B may monitor the augmented reality session by using any type of video processing software capable of processing video in real time. The video processing may entail identifying individual objects within a live video feed. The video processing may also capture all frames of the video where the image processing detects dwell time on a view to exceed a minimum threshold, for example 3 seconds. The threshold may be user-determined or site-determined. The monitoring may be limited in operation to within a geo-fenced area, to a set number of objects, or to a maximum amount of time.

Next, at204, AR focus-object community creation program110A,110B detects focus objects of a user during the augmented reality session. AR focus-object community creation program110A,110B detects focus objects by monitoring which objects are the focus of the user's attention. AR focus-object community creation program110A,110B may determine the user to be focusing on an object if the object is centered in the user's field of view, or if the object is kept within the user's range of vision, or any other method of determining focus. An object may be considered to be a ‘focus object’ if it is the subject of a user's attention for an amount of time that exceeds a threshold, where the threshold may be predetermined by a user or dynamically generated or altered by AR program108or AR focus-object community creation program110A,110B. Focus objects may also be determined through a gesture by a user, for instance a user may point at an object that the user wants to label as a focus object. AR focus-object community creation program110A,110B may limit the number of elements in the photo that are considered focus objects by distance, by limiting object detection to near objects vs. far objects, for instance, or by choosing objects to include based on the focal length of the electronic capture device used to record the video feed. The objects may be detected within the image by any object recognition software, such as OpenCV, or reverse image search software of Google® or Microsoft® (Microsoft® and all Microsoft®-based trademarks and logos are trademarks or registered trademarks of Microsoft Corporation and/or its affiliates). The image search may be performed on the still frames captured from the video feed based on dwell time.

Then, at206, AR focus-object community creation program110A,110B labels the focus objects with an indication of interest. The indication of interest may be the amount of time that the user was focusing on the object, and the total amount of interest may aggregate over an entire session or through a single capture event. AR focus-object community creation program110A,110B may also label the focus objects with the location or area of the object, based on the current position (relative or actual) as determined by triangulation or global positioning systems. AR focus-object community creation program110A,110B may also use relative positions based on, for instance, a store or park layout. The invention may also label focus objects based on emotions and activities, as well as object details such as time, date, user, and location coordinates. For example, [Alice|green apple|25-FEB-2017|4:00 PM|30 seconds interest|x1, x2]. AR focus-object community creation program110A,110B may also label the focus object with a description using adjectives like ‘shiny’ or other correlated data from correlated social streams.

Next, at208, AR focus-object community creation program110A,110B associates the labeled focus objects with users within a specified area. AR focus-object community creation program110A,110B may associate the labeled focus objects with users in a specified area by using the augmented object details in the label (object, date, time, and position) to aggregate all focus objects within a geofenced area. The geofenced area, for instance, may encompass a grocery store, or may encompass a four-block radius. AR focus-object community creation program110A,110B may then determine overlap with other uses. For instance, in the following example, three users are presented:

AR focus-object community creation program110A,110B may find overlap between presented objects based on a statistical similarity or characteristic similarity, such as the fact that the three focus objects in the example are all apples, or the invention may determine the overlap with the aggregated focus objects within a set time window. AR focus-object community creation program110A,110B may also scope the association between the objects to a time window relative to the current observed labeled object, for instance all labeled objects at Binney Street in the last two days. In this step, AR focus-object community creation program110A,110B may incorporate labeled objects that were not sourced from augmented reality sessions, such as photos or other images which are labeled with objects and relative positioning.

Then, at210, AR focus-object community creation program110A,110B establishes, based on the specified area and labeled focus objects, a social network. AR focus-object community creation program110A,110B may create the social network by linking the objects within a community of interest, to determine and highlight items that are the focus objects of users within the community of interest and present users the opportunity to converse with the community of interest and other users. AR focus-object community creation program110A,110B may allow a user to engage with other users within the community of interest regarding a focus object by, for instance, sourcing messages from that user's network in real-time, or contemporaneous with the user's observation of a focus object. For instance, a first user may see comments from a second user regarding a focus object overlaid over or next to that focus object when the first user looks at the focus object. AR focus-object community creation program110A,110B may also add a selective filter based on feedback, and may establish the social network based on specific user characteristics. AR focus-object community creation program110A,110B may also maintain a social network proportionally to the time spent observing the focus object; for instance, if a social network is created aggregating fidget spinners for sale in an area, and fidget spinners fall out of favor and cease to be the focus of viewers, AR focus-object community creation program110A,110B may dissolve the fidget spinner social network.

It may be appreciated thatFIG. 2provides only an illustration of one implementation and does not imply any limitations with regard to how different embodiments may be implemented. Many modifications to the depicted environments may be made based on design and implementation requirements. In one embodiment of the invention, a focus object that is for sale may be labeled with a link to initiate a purchase process, so that a user need not search an online forum to locate the item and purchase it. In another embodiment, the method may further be integrated with a rewards system related to use of the method within a specified store. For example, the AR focus-object community creation program110A,110B may incorporate incentives for the person to participate, such that feedback or focus on objects for sale may yield a coupon or discount. In further embodiments of the invention, AR focus-object community creation program110A,110B may calculate the total number of labeled objects within a given area, and limit the labels to focus objects which are likely to engage the user with conversation.

The data processing system302,304is representative of any electronic device capable of executing machine-readable program instructions. The data processing system302,304may be representative of a smart phone, a computer system, PDA, or other electronic devices. Examples of computing systems, environments, and/or configurations that may represented by the data processing system302,304include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, hand-held or laptop devices, multiprocessor systems, microprocessor-based systems, network PCs, minicomputer systems, and distributed cloud computing environments that include any of the above systems or devices.

The client computing device102and the server112may include respective sets of internal components302a,band external components304a,billustrated inFIG. 3. Each of the sets of internal components302include one or more processors320, one or more computer-readable RAMs322, and one or more computer-readable ROMs324on one or more buses326, and one or more operating systems328and one or more computer-readable tangible storage devices330. The one or more operating systems328, the AR program108and the AR focus-object community creation program110A in the client computing device102, and the AR focus-object community creation program110B in the server112are stored on one or more of the respective computer-readable tangible storage devices330for execution by one or more of the respective processors320via one or more of the respective RAMs322(which typically include cache memory). In the embodiment illustrated inFIG. 3, each of the computer-readable tangible storage devices330is a magnetic disk storage device of an internal hard drive. Alternatively, each of the computer-readable tangible storage devices330is a semiconductor storage device such as ROM324, EPROM, flash memory or any other computer-readable tangible storage device that can store a computer program and digital information.

Each set of internal components302a,balso includes a R/W drive or interface332to read from and write to one or more portable computer-readable tangible storage devices338such as a CD-ROM, DVD, memory stick, magnetic tape, magnetic disk, optical disk or semiconductor storage device. A software program, such as the AR focus-object community creation program110A,110B, can be stored on one or more of the respective portable computer-readable tangible storage devices338, read via the respective R/W drive or interface332, and loaded into the respective hard drive330.

Each set of internal components302a,balso includes network adapters or interfaces336such as a TCP/IP adapter cards, wireless Wi-Fi interface cards, or 3G or 4G wireless interface cards or other wired or wireless communication links. The AR program108and the AR focus-object community creation program110A in the client computing device102and the AR focus-object community creation program110B in the server112can be downloaded to the client computing device102and the server112from an external computer via a network (for example, the Internet, a local area network or other, wide area network) and respective network adapters or interfaces336. From the network adapters or interfaces336, the AR program108and the AR focus-object community creation program110A in the client computing device102and the AR focus-object community creation program110B in the server112are loaded into the respective hard drive330. The network may comprise copper wires, optical fibers, wireless transmission, routers, firewalls, switches, gateway computers and/or edge servers.

Each of the sets of external components304a,bcan include a computer display monitor344, a keyboard342, and a computer mouse334. External components304a,bcan also include touch screens, virtual keyboards, touch pads, pointing devices, and other human interface devices. Each of the sets of internal components302a,balso includes device drivers340to interface to computer display monitor344, keyboard342, and computer mouse334. The device drivers340, R/W drive or interface332, and network adapter or interface336comprise hardware and software (stored in storage device330and/or ROM324).

Characteristics are as follows:

Service Models are as follows:

Deployment Models are as follows:

Workloads layer90provides examples of functionality for which the cloud computing environment may be utilized. Examples of workloads and functions which may be provided from this layer include: mapping and navigation91; software development and lifecycle management92; virtual classroom education delivery93; data analytics processing94; transaction processing95; and AR community creation96. AR focus-object community creation96may relate to establishing a social network within an AR session by locating, tagging, and aggregating focus objects of a user.