SYSTEM AND METHOD FOR CREATING AND USING IMMERSIVE MEDIA

A device includes a processing system including a processor; and a memory that stores executable instructions that, when executed by the processing system, facilitate performance of operations of loading a user profile for a user consuming undigitized, static media content; identifying an area of interest in the undigitized, static media content; analyzing the area of interest; responsive to the user profile, creating immersive content to enhance the area of interest; and providing the immersive content for presentation to the user.

FIELD OF THE DISCLOSURE

The subject disclosure relates to a system and method for creating and using immersive media.

BACKGROUND

Static undigitized media, i.e., printed books, newspapers, recorded shows, etc., do not benefit from hyperlinking to other content. Identifying such works during consumption by a user is necessary before any virtual enhancements can be made.

DETAILED DESCRIPTION

The subject disclosure describes, among other things, illustrative embodiments for identifying one or more parts of static, undigitized media and creating immersive content for a consumer of such media. Other embodiments are described in the subject disclosure.

One or more aspects of the subject disclosure include a device that has a processing system including a processor; and a memory that stores executable instructions that, when executed by the processing system, facilitate performance of operations of loading a user profile for a user consuming undigitized, static media content; identifying an area of interest in the undigitized, static media content; analyzing the area of interest; responsive to the user profile, creating immersive content to enhance the area of interest; and providing the immersive content for presentation to the user.

One or more aspects of the subject disclosure include a non-transitory, machine-readable medium, comprising executable instructions that, when executed by a processing system including a processor, facilitate performance of operations including loading a user profile for a user consuming undigitized, static media content; identifying an area of interest in the undigitized, static media content by user interaction; analyzing the area of interest based on the user profile; creating immersive content to enhance the area of interest; and providing the immersive content for presentation to the user.

One or more aspects of the subject disclosure include a method of loading, by a processing system including a processor, a user profile for a user consuming undigitized, static media content; identifying, by the processing system, an area of interest in the undigitized, static media content by user interaction; analyzing, by the processing system, the area of interest based on the user profile; creating, by the processing system, immersive content that enhances the area of interest; and providing, by the processing system, the immersive content for presentation to the user.

Referring now toFIG.1, a block diagram is shown illustrating an example, non-limiting embodiment of a system100in accordance with various aspects described herein. For example, system100can facilitate in whole or in part loading a user profile for a user consuming undigitized, static media content; identifying an area of interest in the undigitized, static media content by user interaction; analyzing the area of interest based on the user profile; creating immersive content to enhance the area of interest; and providing the immersive content for presentation to the user. In particular, a communications network125is presented for providing broadband access110to a plurality of data terminals114via access terminal112, wireless access120to a plurality of mobile devices124and vehicle126via base station or access point122, voice access130to a plurality of telephony devices134, via switching device132and/or media access140to a plurality of audio/video display devices144via media terminal142. In addition, communication network125is coupled to one or more content sources175of audio, video, graphics, text and/or other media. While broadband access110, wireless access120, voice access130and media access140are shown separately, one or more of these forms of access can be combined to provide multiple access services to a single client device (e.g., mobile devices124can receive media content via media terminal142, data terminal114can be provided voice access via switching device132, and so on).

FIG.2Ais a block diagram illustrating an example, non-limiting embodiment of a system for creating and using immersive media, where the system is functioning within the communication network ofFIG.1in accordance with various aspects described herein. As shown inFIG.2A, system200comprises one or more sensors and/or user interfaces coupled to a processing unit201. In an embodiment depicted inFIG.2A, system200provides an immersive media presentation that is displayed based on a user’s activities observed while consuming static, undigitized media using the sensors. In this example, the user is reading a book215. The user can observe immersive media information via display202and speaker203, each of which is connected to processing unit201. In a preferred embodiment, the user can observe immersive media information via augmented reality (AR) glasses204.

Processing unit201is configured to present information retrieved from database205which stores various kinds of information such as immersive media presentations. Immersive media information stored in database205may include visual enhancements to three dimensional objects and object locations detected within the user environment. Immersive media information can be presented to user via display202and speaker203or augmented reality glasses204. In an embodiment, the image disclosed on display202is enhanced with the immersive media information, for example, with text indicating an action that user may take to further the enjoyment of reading the book215.

Camera206is used to capture images of user as well as objects, such as book215, and the environment in which the user is currently located, or even determine where the user’s gaze is directed. In one example, processing unit201can recognize printed media, such as book215. Processing unit201can identify one or more parts of a whole area, e.g., a page, paragraph, or word within book215. Further, the detected area of a user’s intent can be several sections, e.g., the user may be skimming headlines of a newspaper or the ingredients for a recipe of a food item. Each of these actions should guide processing unit201. In one embodiment, processing unit201presents an immersive presentation synchronized with user activity observed via sensors such as camera206, microphone207, motion sensor208, keyboard209, and mouse210, or other input devices such as a tablet, touch screen, virtual reality (VR) headset, or VR glove (not illustrated), each of which is coupled to processing unit201. In an alternative embodiment, the sensors, display and speaker may be integrated in a three-dimensional VR headset (not illustrated).

After identification of potential targeted sections of book215, processing unit201determines one or more functional modifications to enhance the target, for example, by expanding or linking to additional immersive media components that provide, for example, an in-depth analysis or high-level summarization. These modifications should be interactive, with links to additional immersive content, e.g., videos provided by the original publisher, or as suggested by social media, etc. In an embodiment, processing unit201may determine one or more advanced processing steps - like summarization or expansion of the area of interest, as directed by user input or as ascertained from a user profile.

Processing unit201uses images captured by camera206and other sensors to derive a spatial understanding of where user and other objects are in the environment of user. Microphone207is used to receive ambient sounds including the voice of user and may be used for voice-controlled functions based on user input, such as speaking keywords. Keyboard209and mouse210can be used to receive input from the user while motion sensor208can be used to acquire motion and distance information. Motion sensor208can, for example, detect one or more user gestures or movements as well as the location of objects. Although not shown inFIG.2A, other sensors may be used as well, for example range sensors, geolocation sensors, environmental sensors, infrared, temperature, wind speed, radio frequency and other transducers for converting various parameters into signals suitable for input to processing unit201. The sensors can be used in various combinations depending on factors such as user preferences, cost constraints, etc.

Processing unit201is in communication with database205and can retrieve information for presentation to a user, for example from database205. Processing unit201is also in communication with a network220through which processing unit201can acquire immersive media information from various sources such as individual users, content providers, businesses, as well as additional content available from the Internet.

Finally, with the immersive components, processing unit201should dynamically adapt the applied functions according to the user needs. For example, while the user may have originally been curious about visual images presented by the static, undigitized content, e.g., a picture in a book, a demo in a technical paper, etc., that interest and need may change as the user becomes more deeply immersed in the expansion or move to another topic within the original content. Database205may store a user profile for user, to help tailor, refine, and limit the selection of immersive media to present to the user. For example, if the user is a student, and is saddled with time constraints, processing unit201may assist the student by summarizing the static, undigitized content to help the student save time. In another example, if the user is watching a sports program, and the user’s profile indicates that the user is a “sports junkie,” processing unit201may provide statistics and other player or team info related to the sports program.

FIG.2Bis a flow diagram depicting an illustrative embodiment of a method in accordance with various aspects described herein. As illustrated inFIG.2B, processing unit201comprises several modules including a user display and interaction detection module201A, a printed content analyzer & correlation module201B, a user state and context alignment module201C, an immersion and modification orchestrator module201D, and a notification and sharing module201E. Also shown is database205.

Method230begins in step231with a user engaging in a session with system200while consuming undigitized, static content. In an example, the user opens a book, or starts watching a video, and then starts to interact with module201A. The user has options to choose from for a preferred mode of interaction. In one embodiment the user can select a secondary book, video, or other content from one or more prior assets in the user profile that are related to the static content. In another embodiment, the user can search with explicit keywords or examples or review system-generated recommendations for secondary books, video, or other content from the system. In both embodiments, the user may also set preferences as to what type of media that they wish to consume (e.g., video only, etc.) that may further limit the set of secondary results. System200acts as a search engine that responds to queries from the user (for example: “what are the key points from chapter 1.”) During engagement with the static content, the user may select a camera or icons for supplemental information, or change the mode of presentation (e.g., switch from VR to audio only). If the selection is not available, system200may suggest alternatives (e.g., audio instead of video).

Next in step232, module201A loads the user’s profile from database205. Module201A may include a context of the user’s consumption of the undigitized, static content (e.g., whether the user at work, home, on a train, etc.) and audience (e.g., user is with friends, family). Furthermore, module201A may determine user interaction features from the profile, e.g., triggered by hands-free interaction with something else that links to this item. For example, the system may determine that the screwdriver in the user’s hand links to a certain instruction manual for recently purchased furniture. In another embodiment, module201A can indicate the preferences of the user by experiential qualifications and known demographic information about the user. In one example, an elementary school student would need a different level of information than a college student. The system would suggest age and level appropriate secondary content based on the profile. The user profile also gathers info about the user based on what the user has searched, downloaded, saved, and is accessing. The user profile also links to other systems that have user details. In another example, a user is researching how to rebuild a vintage car, so there are “how-to” manuals, engine diagrams, car model/make details within this user’s profile. The system may also link to commerce and purchase history for the book, video, or other content specific to a recently purchased screwdriver and the recently researched vintage car engine. Completing the example, while the user follows a static step-by-step ‘how to’ guide, the system retrieves various additional content from smart suggestions made by the system. In all these examples, the system may provide these additional content sources and allow the user to view or listen to the suggestions in a hands-free fashion.

Then in step233, module201A identifies an area of user interest in the undigitized, static content. For example, module201A can detect or skip to a user’s position in written instructions for performing a working task. Module201A can ascertain the area of interest by tracking the position of the user while performing the task, or from the user’s gaze while reading the undigitized, static content, etc. Module201A passes along the identified area to module201B. In another embodiment, the identified area is passed to module201B but instead of including other books, videos, or content as additional content, the identified area is stored temporarily to seek additional content from tertiary sources (as in step236below), like social media or live communication channels that may include other users, the content author, or designated experts for the initial content analytics and on-the-spot assistance.

Next, in step234, module201B analyzes the identified area of interest. For example, system200may be driven by a user request to summarize the content, expand the content in more detail, or demonstrate actions described by the content. The user may select which text, with their gaze or by a gesture, such as indicating by a finger, which should be considered for creating immersive content. In an embodiment, system200may query the user for their intent so that immersive content generated can be appropriately matched to the user’s intent. For example, upon a user’s double-click of an ingredient in a recipe, the system can lookup quantity and implications from online usage. In another embodiment, module201B may highlight inconsistencies in an area of interest. For example, if there is a factual mismatch between dates, characters, conflicts with actual historical facts/dates, module201B identifies the mismatch and provides a factual correction. In another embodiment, module201B ties the area of interest to historical events/dates, depending on the context. In another embodiment, module201B identifies key characters/subjects appear in an area of interest and provide summaries of that specific location and ties into larger character/subject summary within the context of the media topic.

In step235, module201B analyzes the user context. Module201B tracks what the user has completed, and tags where the user left off. Module201B stores bookmarks and supplemental materials in database205. Hence, module201B can create an individual/personalized repository of information, as well as highlight information in the undigitized, static content that may be of interest to the user.

In step236, module201C receives social input on the identified area of interest from module201E. In this example, there is a social aspect where one or more other users are reading, watching, or learning together. In one embodiment, the other users may or may not be synchronously engaged with the same content at the exact same moment. In another embodiment, the other users may be engaged in the static or by the additional content that the system has identified. Continuing the example, one user highlights a section of interest, which alerts the other users (who can see the highlighted section). The others can provide feedback or perspective with an interactive feature that encourages group feedback. In yet another embodiment, one of the other users may be the author of the original static content and may offer revisions, additional media, or interactions (like highlights) that are sent to the current user, stored in the current user’s profile, or stored for future users to receive when they begin to engage in the original static content - all of which are akin to a post-release “director’s cut” except that the modifications may be made at any time and are not delivered to a user until she or he is directly engaged by the original or additional content.

In step237, module201D creates immersive media for engagement with the user and passes the immersive media to module201A for presentation to the user. System200provides a visual display and playback of the immersive media. In an embodiment, the immersive media is overlaid as an augmented reality for the user through AR glasses204. For example, the immersive content may replace content in the undigitized, static media (i.e., providing a summary, expansion, or simplification of the content). Special effects may be created for skimming, highlight and “fade out” or diminish other media. This is like how a student (or person analyzing a book, article, or static media) would highlight certain sections so that they can go back and find it quickly. In another embodiment, using an immersive presentation, the system would fade (by adding the appearance of transparency) or dim out (by decreasing the apparent contrast versus the remaining page contents) the remaining items so that a particular section stands out. In both embodiments, the immersive presentation creates visual differences in the media that allows user to identify a highlighted area (or an area to ignore). In yet another embodiment, aural or audio-video hybrids may utilize frequency manipulations to simulate a voice or musical instrument moving to the background. In yet another embodiment, spatialized audio (e.g., one that provides an immersion with 360-degree placement of sounds) and video (e.g., augmented reality) may apply effects to the interactive immersion such that some additional media items are brought into (or out of) focus. The immersive media may augment the existing content with new content or media. Module201A can modify a user interface for advanced usage (e.g., a ten-page book may now have virtual folds, increasing the size of the book to 100 pages). In another example, a map may be overlaid with traffic information, or a historical map may illustrate chronological changes under the control of the user (e.g., swipe, turn, flip), or provide a projection to other locations (e.g., by zooming in or out and/or dragging).

In step238, module201A receives feedback from the user concerning the mode of presentation of the immersive media. The process repeats with step233when the user is consuming the undigitized, static media end to end (i.e., in the same mode). However, the user may change modes during consumption. For example, the user may decide to query for additional information or a high-level summary.

In step239, module201D optionally shares the user’s progress consuming the undigitized, static content with module201E. For example, module201D may share the user’s progress with proximal users, or share the undigitized, static content digitally, i.e., though raw content or summary/highlights, like images for kids, or keywords, or dialogs for adults, etc. In another embodiment, system200may create a virtual bookmark where visual component from written asset can be shared (e.g., “tear out” a graphic, put it virtually on your computer screen; when the graphic is tapped, it points back to a page in the book or instruction manual).

In step240, module201D updates the user’s profile in database205. Information stored in the profile includes user preferences for interactions, enhancements most likely sought by the user, any last known moments in the media (position, offset, state of immersions), and possibly additional questions for other social contributors or the user.

In step241, module201D creates a future engagement opportunity and passes the engagement opportunity along to module201A for presentation to the user. For example, when the user reaches the final page of the book/media, the system may include an auto-generated “more like this” button that recommends other assets. In another example, the book or page may refresh at start of interaction to show previews of what may be available from other books.

Referring now toFIG.3, a block diagram300is shown illustrating an example, non-limiting embodiment of a virtualized communication network in accordance with various aspects described herein. A virtualized communication network is presented that can be used to implement some or all the subsystems and functions of system100, the subsystems and functions of system200, and method230presented inFIGS.1,2A,2B and3. For example, virtualized communication network300can facilitate in whole or in part loading a user profile for a user consuming undigitized, static media content; identifying an area of interest in the undigitized, static media content by user interaction; analyzing the area of interest based on the user profile; creating immersive content to enhance the area of interest; and providing the immersive content for presentation to the user.

Turning now toFIG.4, there is illustrated a block diagram of a computing environment in accordance with various aspects described herein. To provide additional context for various embodiments of the embodiments described herein,FIG.4and the following discussion are intended to provide a brief, general description of a suitable computing environment400in which the various embodiments of the subject disclosure can be implemented. Computing environment400can be used in the implementation of network elements150,152,154,156, access terminal112, base station or access point122, switching device132, media terminal142, and/or VNEs330,332,334, etc. Each of these devices can be implemented via computer-executable instructions that can run on one or more computers, and/or in combination with other program modules and/or as a combination of hardware and software. For example, computing environment400can facilitate in whole or in part loading a user profile for a user consuming undigitized, static media content; identifying an area of interest in the undigitized, static media content by user interaction; analyzing the area of interest based on the user profile; creating immersive content to enhance the area of interest; and providing the immersive content for presentation to the user.

The illustrated embodiments of the embodiments herein can be also practiced in distributed computing environments where certain tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules can be in both local and remote memory storage devices.

Turning now toFIG.5, an embodiment500of a mobile network platform510is shown that is an example of network elements150,152,154,156, and/or VNEs330,332,334, etc. For example, platform510can facilitate in whole or in part loading a user profile for a user consuming undigitized, static media content; identifying an area of interest in the undigitized, static media content by user interaction; analyzing the area of interest based on the user profile; creating immersive content to enhance the area of interest; and providing the immersive content for presentation to the user. In one or more embodiments, the mobile network platform510can generate and receive signals transmitted and received by base stations or access points such as base station or access point122. Generally, mobile network platform510can comprise components, e.g., nodes, gateways, interfaces, servers, or disparate platforms, which facilitate both packet-switched (PS) (e.g., internet protocol (IP), frame relay, asynchronous transfer mode (ATM)) and circuit-switched (CS) traffic (e.g., voice and data), as well as control generation for networked wireless telecommunication. As a non-limiting example, mobile network platform510can be included in telecommunications carrier networks and can be considered carrier-side components as discussed elsewhere herein. Mobile network platform510comprises CS gateway node(s)512which can interface CS traffic received from legacy networks like telephony network(s)540(e.g., public switched telephone network (PSTN), or public land mobile network (PLMN)) or a signaling system #7 (SS7) network560. CS gateway node(s)512can authorize and authenticate traffic (e.g., voice) arising from such networks. Additionally, CS gateway node(s)512can access mobility, or roaming, data generated through SS7 network560; for instance, mobility data stored in a visited location register (VLR), which can reside in memory530. Moreover, CS gateway node(s)512interfaces CS-based traffic and signaling and PS gateway node(s)518. As an example, in a 3GPP UMTS network, CS gateway node(s)512can be realized at least in part in gateway GPRS support node(s) (GGSN). It should be appreciated that functionality and specific operation of CS gateway node(s)512, PS gateway node(s)518, and serving node(s)516, is provided and dictated by radio technology(ies) utilized by mobile network platform510for telecommunication over a radio access network520with other devices, such as a radiotelephone575.

Turning now toFIG.6, an illustrative embodiment of a communication device600is shown. The communication device600can serve as an illustrative embodiment of devices such as data terminals114, mobile devices124, vehicle126, display devices144or other client devices for communication via either communications network125. For example, computing device600can facilitate in whole or in part loading a user profile for a user consuming undigitized, static media content; identifying an area of interest in the undigitized, static media content by user interaction; analyzing the area of interest based on the user profile; creating immersive content to enhance the area of interest; and providing the immersive content for presentation to the user.