Methods and systems for providing an immersive virtual reality experience

An immersive virtual reality experience is provided. A plurality of areas within a virtual environment are determined. The plurality of areas includes at least a first area and a second area. A plurality of videos associated with the virtual environment are presented. The plurality of videos include a first video associated with the first area and a second video associated with the second area. The first video has a first quality, and the second video has a second quality lower than the first quality. A three-dimensional model of a first object is overlaid in the virtual environment.

BACKGROUND

The Coronavirus Disease 2019 (COVID-19) pandemic and the resultant measures to keep the public safe have made it difficult and/or onerous to watch sports, concerts, theater, or other live events. Prevailing health and societal dynamics have created a need for alternative solutions to watching games, concerts, and/or shows. While the technology has improved over the years, television and streaming services fail to replicate the experience of being physically present at the game, concert, and/or show. Likewise, known methods and system for providing a virtual reality (VR) experience are relatively limited. For example, some known VR experiences are provided using a pure video format captured by a single camera with a wide angle lens (e.g., for a 180-degree video) or by two opposing cameras with wide angle lenses (e.g., for a 360-degree video “stitched” together from two 180-degree videos). However, depth perception is generally difficult to convey using a pure video format. Aside from reducing immersion, known VR implementations of pure video format can also cause disorientation or motion sickness in at least some viewers.

To improve depth perception, some known VR systems use stereoscopic video that includes two channels with slightly different perspectives. Stereoscopic videos are typically captured using pairs of cameras that are spaced apart about the interpupillary distance of a typical person (e.g., the distance between a person's pupils). Because each eye receives its own video feed, parallax is created, increasing a sense of space and three dimensionality. However, because the cameras are in a fixed location, the depth perception may skew or decrease when looking anywhere other than directly forward. In addition, if the viewer tilts their head sideways (e.g., rotate about a roll axis), the images will “swim” and/or separate, making it difficult to focus and potentially causing disorientation or motion sickness. These issues preclude many spectators from using the technology and limit long term comfort making it challenging to watch an entire game or concert for example thus making existing solutions unsuitable for these applications.

SUMMARY

Examples described herein enable an immersive virtual reality experience to be provided in a spectrum-efficient manner, while also being comfortable and enjoyable for the user. In one aspect, a method is provided for providing an immersive virtual reality experience. The method includes determining at least a first area and a second area within a virtual environment, presenting a first video associated with the first area and a second video associated with the second area, and overlaying a three-dimensional model of a first object in the virtual environment. The first video has a first quality, and the second video has a second quality lower than the first quality.

In another aspect, an immersive virtual reality device is provided. The immersive virtual reality device includes one or more computer storage media including data associated with a virtual environment, one or more objects, and computer-executable instructions, and one or more processors configured to execute the computer-executable instructions. The processors execute the computer-executable instructions to determine at least a first area and a second area within the virtual environment, present a first video showing the first area, present a second video showing the second area, and overlay a three-dimensional model of a first object in the virtual environment. The first video has a first quality, and the second video has a second quality lower than the first quality.

In yet another aspect, a system is provided. The system includes one or more user sensors that detect one or more parameters associated with a user, and a presentation device that communicates with the one or more user sensors to provide an immersive virtual reality experience. Providing the immersive virtual reality experience includes determining at least a first area and a second area within a virtual environment, presenting at least a first video showing the first area and a second video showing the second area, and overlaying a three-dimensional model of a first object in the virtual environment. The first video has a first quality, and the second video has a second quality lower than the first quality.

This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter. Additional features and advantages of one or more embodiments of the present disclosure will be set forth in the Detailed Description, and in part will be obvious from the Detailed Description or may be learned by the practice of such example embodiments.

Like parts are marked throughout the drawings, as well as throughout the Detailed Disclosure, with the same reference numerals. The drawings are for the purpose of illustration and description only and are not intended to define the scope of the claimed subject matter. The drawings are not necessarily drawn to scale, and certain drawings may be shown in exaggerated or generalized form in the interest of clarity and conciseness. Although specific features may be shown in some of the drawings and not in others, this is for convenience only. In accordance with the examples described herein, any feature of a drawing may be referenced and/or claimed in combination with any feature of any other drawing.

DETAILED DESCRIPTION

The present disclosure relates to user interfaces and, more particularly, to methods and systems for providing an immersive virtual reality (VR) experience. Examples described herein seek to address the issues with existing immersive VR solutions and create a comfortable and enjoyable viewing experience in a spectrum-efficient manner. The methods and systems of the present disclosure may craft an experience that blends real-time rendered computer-graphic environments with full-motion video (e.g., a filmed event, live streaming). For example, one or more three-dimensional (3D) models may be used to present one or more proximate objects, and one or more video feeds may be used to present one or more distant objects. Because the stereoscopic convergence distance of the human eye is about 10 meters from the vantage point, 3D models may be used to present one or more objects within a predetermined space (e.g., a luxury box, a front-row seat, a vehicle cabin) or within 10 meters of the predetermined space, and video feeds may be used to present one or more objects outside the predetermined space or beyond 10 meters of the predetermined space.

Referring now to the drawings, which are provided for purposes of illustrating one or more examples and not for purposes of limiting the same, with reference toFIG.1, an example immersive VR experience system100includes a presentation device110that presents a virtual environment to a user112. In some examples, the presentation device110executes one or more applications (“apps”) to selectively present a virtual environment that approximates a real-world location (e.g., luxury box, front-row seats), providing a first-class ticket experience to sports fans and venue attendees in the comfort and convenience of their own home.

At least a portion of the virtual environment may be created using a location capture node including and/or associated with one or more environment sensors120. Example environment sensors120may include, without limitation, a high-resolution camera, a lower-resolution camera, a lidar (light detection and ranging) depth camera, a range sensor, a position and orientation sensor, a texture sensor, and any other sensor that may be used to generate data for use in creating the virtual environment based on a real-world environment.

In some examples, the immersive VR experience system100includes one or more user sensors130that detect one or more parameters associated with the user112and communicate with the presentation device110to enable the presentation device110to modify the virtual environment based on the detected parameters. User sensors130may enable the user112to provide user input. Example user sensors130may include, without limitation, a position sensor, a motion sensor, an accelerometer, a gyroscope, an inertial measurement unit (IMU) sensor, an eye-tracking sensor, an infrared sensor, an optical tracking sensor, and any other sensor that may be used to generate data for determining a user position, a user movement, a head position, a head orientation, a gaze direction, and the like. In some examples, the user sensors130detect a position and/or movement of the user's head or body (e.g., hands, feet), and adjust the virtual environment in accordance with the position and/or movement. In some examples, light, reflections, shadows, and/or texture are used to increase a fidelity or “realness” of the objects in the virtual environment.

The user112may look at or around the objects in the virtual environment and, in some instances, interact with the objects in the virtual environment. The presentation device110may convey visual feedback to the user112. In some examples, the presentation device110includes or is coupled to one or more feedback devices140configured to convey information to the user112via one or more senses other than sight. For example, feedback devices140may provide tactile cues via electrical stimulation, vibration, and/or pushing forces; thermal cues; and/or auditory cues while the user112handles or interacts with one or more virtual objects or to indicate that the user112has approached or is approaching a virtual wall or other boundary within the virtual environment. Example feedback devices140may include, without limitation, actuators, stimulators, speakers, and any other device that may be used to convey non-visual cues to the user112.

FIGS.2-4show an example virtual environment200that may be presented to the user112using the presentation device110. To efficiently and/or effectively present the virtual environment200using the presentation device110, the virtual environment200may be divided or partitioned into a plurality of areas including at least a first area210and a second area220. In some examples, the areas (e.g., first area210, second area220) include one or more regular or semi-regular polygons that tessellate or substantially tessellate (with nominal overlap or gaps) to form a sphere or spheroid surrounding a vantage point inside the virtual environment200. In this manner, the areas may be efficiently and effectively mapped onto 3D geometry.

The presentation device110may present the virtual environment200using a plurality of videos associated with the virtual environment200, including a first video associated with the first area210and a second video associated with the second area220. As shown inFIG.4, an action area provider222may provide the first video associated with the first area210, and an environment provider224may provide the second video associated with the second area220. In some examples, the presentation device110is configured to control video playback of the first video and/or second video independent of the other. For example, the user112may be allowed to selectively rewind, slow, fast forward, or stop (i.e., pause) the first video and/or second video.

The first area210and/or second area220are determined to facilitate balancing user experience and technological requirements (e.g., response time, rendering quality, sampling rate, frame rate consistency, processing power, memory bandwidth, energy consumption). For example, the first video may have a first quality (e.g., a high-resolution video with a high frame rate) and the second video may have a second quality lower than the first quality (e.g., a lower-resolution video with a lower frame rate) while the second video is displayed in a 360-degree spherical angle and the first video is displayed in an angle that is less than 360 degrees (e.g., 90 degrees, 180 degrees). In some examples, the first area210and/or second area220may be predetermined based on application, venue, event, etc. For example, the first area210may be an action area (e.g., court, field, stage, etc.) or other area of interest, and the second area220may be an environment area (e.g., crowd, stands, bleachers, etc.) at least partially surrounding the action area.

In some examples, the first area210and/or second area220are fixed, given a particular vantage point. In such examples, the user112may move to change a head position, a head orientation, a gaze direction, etc., but the first area210and/or second area220stay the same (e.g., the perspective may change, but the polygons in the first area210do not move to the second area220and vice versa). Alternatively, the first area210and/or second area220may be dynamic. For example, the presentation device110may analyze one or more user parameters to determine an area of interest (e.g., based on gaze direction), and dynamically determine the first area210and/or second area220based on the area of interest (e.g., one or more polygons in the first area210become part of the second area220and/or vice versa as the perspective changes). Notwithstanding the above, if the user112moves to another vantage point (e.g., to a new user location within the virtual environment), the presentation device110may determine another first area210and/or second area220corresponding to the other vantage point and/or select a first video and/or second video for presentation based on the user position and/or gaze direction.

To facilitate creating an immersive VR experience, a high-quality, view-dependent, texture-mapped, three-dimensional (3D) model230of one or more objects are superimposed in the virtual environment200. As shown inFIG.4, an object provider232may overlay the objects in the virtual environment200. The 3D models230may be created based on 3D scans of real-world objects that are or may be in the real-world location (e.g., chairs, stools, sofas, televisions). The 3D scans may be performed using one or more sensors (e.g., environment sensors120) configured to generate one or more signals or sensor data based on one or more stimuli. Sensor data may include any information that enables a computing device to map or understand a real-world environment and/or various objects in the real-world environment. In some examples, one or more models230include complex geometric meshes including image-based representations of a 3D object that are rendered in real time at high frame rates. Alternatively, at least a portion of one model230may include simple geometric meshes including geometric-based representations of a 3D object that are pre-rendered at lower frame rates to facilitate reducing processing power, memory bandwidth, and/or energy consumption.

In some examples, the user112may interact with the objects in the virtual environment200. For example, the presentation device110may analyze one or more user parameters to determine whether a user position and/or movement would affect a position and/or movement of one or more objects in the virtual environment200and changes or manipulates the position and/or movement of the objects accordingly. In some examples, a user movement is determined to indicate a desire to interact with a first object, and a 3D model of the first object is manipulated based on the user movement. If a user position and/or movement is in conflict with a position of one or more objects (e.g., a virtual representation of the user occupies or is projected to occupy the same virtual space as a virtual representation of one or more objects), the presentation device110may communicate with one or more feedback devices140to convey to the user112one or more non-visual sensory cues corresponding to the conflict. For example, when the user112closes their hand such that a virtual representation of the user's hand closes around a virtual representation of an object in the virtual environment200, one or more feedback devices140may provide one or more tactile, thermal, and/or auditory cues conveying that the user112is holding the object. For another example, one or more feedback devices140may provide one or more tactile, thermal, and/or auditory cues conveying that the user112is prevented or restricted from moving beyond one or more viewer bounding areas in the virtual environment200(e.g., virtual luxury box, virtual front-row-seat, virtual vehicle cabin).

FIG.5shows an example system300for providing content to a client device310(e.g., presentation device110) for use in creating an immersive VR experience312. To facilitate creating a virtual environment200, the client device310may receive one or more site-based inputs from one or more environment sensors120(e.g., via location node320). For example, one or more near-field objects (e.g., model230) may be created using object input330from an object provider232(shown inFIG.4). Object input330may be generated based on one or more 3D scans, including object 3D scans332of one or more real-world objects and/or environment 3D scans334of a real-world environment (e.g., luxury box, front-row seat, vehicle cabin), which may be performed using a contact 3D scanner (e.g., a coordinate measuring machine) and/or a non-contact 3D scanner (e.g., 3D laser scanner, lidar depth camera). 3D scanning can be performed beforehand or in real time (e.g., at the time of a live broadcast) depending on setup requirements, real-time scan fidelity, and/or security requirements.

Near-field objects may be combined with high-resolution content for an action area (e.g., first area210) and lower-resolution content for an environment (e.g., second area220) to give the virtual environment200a fully immersive presence. High-resolution content may be created using action area input340from an action area provider222(shown inFIG.4). Action area input340may be generated based on a 90-180 degree video feed342of a real-world area of interest (e.g., court, field, stage, etc.), which may be provided using a high-resolution camera fitted with a wide-angle lens. Lower-resolution content may be created using environment input350from an environment provider224(shown inFIG.4). Environment input350may be generated based on a 360 degree video feed352of a real-world environment (e.g., crowd, stands, bleachers, etc.) at least partially surrounding the real-world area of interest, which may be provided using a lower-resolution camera fitted with a wide-angle lens.

To mitigate the likelihood of seams and/or other inconsistencies in the virtual environment200, the 90-180 degree video feed342and/or 360 degree video feed352may be composited and then blended with object 3D scans332and/or environment 3D scans334. This blended combination of the object 3D scans332, environment 3D scans334, 90-180 degree video feed342, and/or 360 degree video feed352provides enhanced comfort for users112(even for long term sessions) and greatly improves visual fidelity by focusing the first area210without losing full immersion of being completely surrounded by video. In some examples, one or more viewer bounding areas are created to at least partially restrain virtual user movement to one or more predetermined vantage points in a 3D portion of the virtual environment200(e.g., virtual luxury box, virtual front-row-seat, virtual vehicle cabin) and ensure the user112has an opportunity to have an unobstructed line of sight to the 90-180 degree video feed342.

In some examples, the client device310may receive one or more secondary inputs360from a broadcast362, social media364, and/or a sponsor366. This technology opens the doors for increased interaction with broadcasters associated with the broadcast362, other spectators (e.g., via social media364), sponsors366, and more in a way that cannot be achieved by existing immersive video formats.

FIG.6shows an example method400for providing an immersive virtual reality experience312. The method400includes various operations that may be performed or executed by the presentation device110to present a virtual environment200. The method400will be described with reference to various elements shown inFIGS.1-5, though it is to be appreciated that the method400may be used with other systems and/or components to present the virtual environment200. In some examples, a plurality of areas, including at least a first area210(e.g., an action area) and a second area220(e.g., an environment), are determined at operation410. A plurality of video streams associated with the virtual environment200are presented at operation420, including a first video associated with a first area (e.g., based on action area input340) and a second video associated with a second area (e.g., based on environment input350). A 3D model of one or more objects (e.g., based on object 3D scans332and/or environment 3D scans334) is overlaid at operation430.

FIG.7shows an example environment500that allows a plurality of users (e.g., user112) to virtually experience an event with other users. Each user112may use their own user device510to enjoy their own immersive VR experience312. The user devices510include one or more client-side applications that perform one or more operations at the user devices510while one or more operations are performed remotely. For example, the client-side applications may allow the user devices510to communicate with one or more computing systems (e.g., the “cloud”) that perform one or more back-end operations using one or more counterpart applications (e.g., server-side applications) and/or through one or more server-side services. In some examples, the user devices510transmit one or more signals to a system server520via a communication network530for back-end processing.

The system server520provides a shared pool of configurable computing resources to perform one or more back-end operations. The system server520may host or manage one or more server-side applications that include or are associated with VR technology, such as a video streaming application configured to generate the plurality of videos in the first area210and second area220and a 3D rendering application configured to generate one or more 3D models230. The system server520may include or store data, such as object input330, action area input340, environment input350, and/or secondary input360. The data may be used, for example, to provide and/or create content for an immersive VR experience312. The system server520may convert the data into one or more other forms (e.g., an analog signal to a digital form), remove at least some undesired portions (“noise”), and/or recognize or identify one or more objects in an environment (real-world or virtual). In some examples, the system server520includes a 3D module522for use in generating a 3D portion of the virtual environment200(e.g., based on object input330), a video module524for use in generating a video portion of the virtual environment200(e.g., based on action area input340and environment input350), and a presentation module526for use in combining the 3D portion of the virtual environment200with the video portion of the virtual environment, along with any other desirable portions (e.g., based on secondary input360), to present an immersive VR experience312.

In some examples, the system server520uses one or more annotations or identifiers that allow one or more objects, or one or more characteristics of an object, to be readily recognized or identified. The system server520may classify or identify an object as a static object, for example, if a position and/or orientation of the object is the same or substantially similar over time (e.g., based on sensor data associated with a first point in time and sensor data associated with a second point in time). Additionally or alternatively, the system server520may classify or identify an object as a dynamic object, for example, if a position and/or orientation of the object changes over time. The system server520may monitor or track dynamic objects by collecting data indicative of a level of activity or movement (e.g., speed, direction), as well as behavior patterns. In this manner, the system server520may predict or determine a position and/or orientation of dynamic objects.

The user device510and/or system server520may communicate with one or more content providers540to retrieve and/or receive various content, including content from a model database542(e.g., for use in the 3D portion of the virtual environment200), video content from a high-resolution (“hi-res”) video database544(e.g., for use in the action area portion of the virtual environment200), video content from a lower-resolution (“lower-res”) video database546(e.g., for use in the environment portion of the virtual environment200), and/or secondary content from a secondary content database or other source (not shown). In some examples, the system server520includes, is included in, or is coupled to one or more artificial neural networks that “learn” and/or evolve based on information or insights gained through the processing of object input330, action area input340, environment input350, and/or secondary input360. A “module,” as used herein, includes, but is not limited to, non-transitory computer readable medium that stores instructions, instructions in execution on a machine, hardware, firmware, software in execution on a machine, and/or combinations of each to perform one or more functions or actions, and/or to cause one or more functions or actions from another module, method, and/or system. A module may also include logic, a software controlled microprocessor, a discrete logic circuit, an analog circuit, a digital circuit, a programmed logic device, a memory device containing executing instructions, logic gates, a combination of gates, and/or other circuit components. Multiple modules may be combined into one module, and single modules may be distributed among multiple modules.

In some examples, remote users112(i.e., other users in the same virtual environment200) may be selectively presented in the virtual environment200. For example, each user112may opt in or opt out from seeing, being shown to, or interacting with remote users112. To present a remote user112in the virtual environment200, a virtual position of the remote user112is determined, and a real-time rendered, high-quality, view-dependent, texture-mapped 3D model of the remote user112may be presented as an object (e.g., a “second” object) in the virtual environment200(e.g., model230). Additionally or alternatively, one or more people or objects in the 3D portion and/or video portion of the virtual environment200may be filtered or edited out from the virtual environment200.

The environment500includes one or more communication networks530that allow information to be communicated between a plurality of computing systems coupled to the communication networks530(e.g., user devices510, system server520, content provider540). Example communication networks530include, without limitation, a cellular network, the Internet, a personal area network (PAN), a local area network (LAN), and a wide area network (WAN). One or more interfaces (not shown) may facilitate communication within the environment500. The interfaces may include one or more gateways that allow the user devices510, system server520, and/or content providers540to communicate with each other and/or with one or more other computing systems for performing one or more operations. For example, the gateways may format data and/or control one or more data exchanges using an Open Systems Interconnection (OSI) model that enables the computing systems (e.g., user devices510, system server520, content provider540) to communicate using one or more communication protocols. In some examples, the gateways identify and/or locate one or more target computing systems to selectively route data in and/or through the environment500.

FIG.8shows an example computing system600configured to perform one or more computing operations. While some examples of the disclosure are illustrated and described herein with reference to the computing system600being included in a presentation device110(shown inFIG.1), user device510(shown inFIG.7), system server520(shown inFIG.7), and/or content provider540(shown inFIG.7), aspects of the disclosure are operable with any computing system that executes instructions to implement the operations and functionality associated with the computing system600. The computing system600shows only one example of a computing environment for performing one or more computing operations and is not intended to suggest any limitation as to the scope of use or functionality of the disclosure.

The computing system600includes a system memory610(e.g., computer storage media) and a processor620coupled to the system memory610. The system memory610can store an operating system that controls or allocates resources of the computing system600. In some examples, the system memory610and processor620are coupled via a bus that enables data to be transferred therebetween. As used herein, a “memory” can include non-volatile memory and/or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), solid-state drives, and/or disks. Volatile memory can include random-access memory (RAM), static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), and/or double data rate SDRAM (DDR SDRAM).

The system memory610includes one or more computer-readable media that allow information, such as the computer-executable instructions and other data, to be stored and/or retrieved by the processor620. For example, the system memory610may include computer storage media in the form of read-only memory (ROM), random-access memory (RAM), magnetic tape, a floppy disk, a hard disk, a compact disc (CD), a digital versatile disc (DVD), a flash drive, a memory card, or any other medium that may be used to store desired information that may be accessed by the processor620. By way of example, and not limitation, computer-readable media may include computer storage media and communication media. Computer storage media are tangible and mutually exclusive to communication media. Computer storage media are implemented in hardware and exclude carrier waves and propagated signals. That is, computer storage media for purposes of this disclosure are not signals per se.

In some examples, the system memory610stores data associated with one or more virtual environments200and/or immersive VR experiences312(e.g., object input330, action area input340, environment input350, and/or secondary input360), and computer-executable instructions, and the processor620is programmed or configured to execute the computer-executable instructions for implementing and/or managing aspects of the disclosure using, for example, the presentation device110. The processor620may execute the computer-executable instructions to determine a plurality of areas (e.g., first area210, second area220) within the virtual environment200, present a first video stream showing the first area210, present a second video stream showing the second area220, and overlay a 3D model230of one or more objects in the virtual environment200.

As used herein, a “processor” processes signals and performs general computing and arithmetic functions. Signals processed by the processor620can include digital signals, data signals, computer instructions, processor instructions, messages, a bit, a bit stream, or other means that can be received, transmitted and/or detected. The processor620may include one or more processing units (e.g., in a multi-core configuration) to execute various functions. Although the processor620is shown separate from the system memory610, examples of the disclosure contemplate that the system memory610may be onboard the processor620, such as in some embedded systems.

A user or operator may enter commands and other input into the computing system600through one or more input devices630coupled to the processor620. The input devices630are configured to receive information. Example input device630include, without limitation, a pointing device (e.g., mouse, trackball, touch pad, joystick), a keyboard, a game pad, a controller, a microphone, a camera, a gyroscope, an accelerometer, a position detector, and an electronic digitizer (e.g., on a touchscreen). Information, such as text, images, video, audio, and the like, may be presented to a user via one or more output devices640coupled to the processor620. The output devices640are configured to convey information, images, sounds, smells, etc. to the user112. Example output devices640include, without limitation, a monitor, a projector, a printer, a speaker, a vibrating component. In some examples, an output device640is integrated with an input device630(e.g., a capacitive touch-screen panel, a controller including a vibrating component).

One or more network interfaces650may be used to operate the computing system600in a networked environment using one or more logical connections. Logical connections include, for example, local area networks, wide area networks, and the Internet. The network interfaces650allow the processor620, for example, to convey information to and/or receive information from one or more remote devices, such as another computing system or one or more remote computer storage media. Computer communication between computing systems can be a network transfer, a file transfer, an applet transfer, an email, a hypertext transfer protocol (HTTP) transfer, and so on. A computer communication can occur across a wireless system (e.g., IEEE 802.11), an Ethernet system (e.g., IEEE 802.3), a token ring system (e.g., IEEE 802.5), a local area network (LAN), a wide area network (WAN), a point-to-point system, a circuit switching system, a packet switching system, among others. Network interfaces650may include a network adapter, such as a wired or wireless network adapter or a wireless data transceiver.

Examples described herein enable a virtual environment200to show or approximate a real-world event (e.g., sports, concert, theater). In some examples, the virtual environment200is presented using a VR headset. For example, the presentation device110may include wireless or mobile-powered options, like the Oculus Quest headset. (Oculus Quest is a registered trademark of Facebook Technologies, LLC of Menlo Park, Calif.). This disclosure accounts for near-field objects, an action area, and an environment by combining one or more 3D models230with a high-resolution video and a lower-resolution video. This combination allows the presentation device110to present the user112with an immersive VR experience312, giving the user112the ability to move around a designated area or predetermined space, switch between seat locations, and even communicate with others who are viewing the event with them in the virtual environment200.

The videos may be taken using one or more cameras in the real-world location with wide angle (e.g., 170-210 degrees) lenses. The videos may be broadcast live or be prerecorded. With the added perception afforded by the 3D objects in the virtual environment200, the presentation device110may present or direct one set of images (e.g., a single monoscopic video feed) to both eyes. Using one set of images allows the system to use images with higher quality or resolution and/or to occupy less memory or bandwidth than if two sets of images were being used (e.g., for 360-degree views and/or for stereoscopic video feeds). Moreover, using one set of images allows the user112to tilt their head without the video “swimming” or separating during use.

In some examples, video is presented on the hemisphere in front of the user112(e.g., first area210) and/or on the hemisphere above the user112(second area220). The video presented on the hemisphere above the user112may be at a lower resolution than the video presented on the hemisphere in front of the user112. Video played on the hemisphere in front of the user112and/or on the hemisphere above the user112may be used to help light and provide reflections in the virtual environment200to tie the real and the virtual together. This is captured all or in part by a video and computing node which contains at least two cameras along with bespoke positional calibration mechanisms to fine tune the alignment of various elements in the virtual environment200.

Examples described herein allow viewers to experience an event from their preferred seats with a high degree of realism and feel like actually being at the event. In some examples, the system is configured to present other feedback (e.g., sounds, smells, etc.) for further creating immersion to the event being presented in the virtual environment200. A plurality of audio channels, for example, may be presented via a plurality of speakers disposed around the user112(e.g., surround sound) to provide positional audio and enrich the fidelity and depth of noises heard in the virtual environment200. The mixed media nature of the presentation allows for and even encourages audio and video feeds from outside of the real-world location. This could include but not be limited to: communications with friends/family/community, input and feedback from broadcasters or commentators, sponsor content, and the ability to rewind, slow, or stop the video feed. Since the models are an integral part of the realistic representation of the event, manipulation of the models is possible without harming the viewer's sense of immersion.

The solutions described herein create a virtual reality experience that blends real-time rendered computer graphics environments with full motion video to create an environment that that offers six degrees of freedom, reduces or eliminates the possibility of motion sickness, promotes depth perception of near-field objects, allows high-resolution video to be selectively used, and creates an immersive experience of an event. This viewing experience will revolutionize the way people remotely view sports, live shows, and other spectator events. Further use of this technology could be for emergency or first responder training, military training, pilot training, sports training, and more.

Example systems and methods for providing an immersive virtual reality experience are described herein and illustrated in the accompanying drawings. Various examples may be implemented as instructions stored on a non-transitory machine-readable storage medium, such as a volatile or non-volatile memory, which may be read and executed by at least one processor to perform the operations described in detail herein. A machine-readable storage medium may include any mechanism for storing information in a form readable by a machine, such as a personal or laptop computer, a server, or other computing device. Thus, a non-transitory machine-readable storage medium excludes transitory signals but may include both volatile and non-volatile memories.

This written description uses examples to disclose aspects of the disclosure and also to enable a person skilled in the art to practice the aspects, including making or using the above-described systems and executing or performing the above-described methods. Having described aspects of the disclosure in terms of various examples with their associated operations, it will be apparent that modifications and variations are possible without departing from the scope of the disclosure as defined in the appended claims. That is, aspects of the disclosure are not limited to the specific examples described herein, and all matter contained in the above description and shown in the accompanying drawings shall be interpreted as illustrative and not in a limiting sense. For example, the examples described herein may be implemented and utilized in connection with many other applications such as, but not limited to, training for fields including emergency, first responder, military, pilot, sports, and more.

Components of the systems and/or operations of the methods described herein may be utilized independently and separately from other components and/or operations described herein. Moreover, the methods described herein may include additional or fewer operations than those disclosed, and the order of execution or performance of the operations described herein is not essential unless otherwise specified. That is, the operations may be executed or performed in any order, unless otherwise specified, and it is contemplated that executing or performing a particular operation before, contemporaneously with, or after another operation is within the scope of the disclosure. Although specific features of various examples of the disclosure may be shown in some drawings and not in others, this is for convenience only. In accordance with the principles of the disclosure, any feature of a drawing may be referenced and/or claimed in combination with any feature of any other drawing.

When introducing elements of the disclosure or the examples thereof, the articles “a,” “an,” “the,” and “said” are intended to mean that there are one or more of the elements. References to an “embodiment” or an “example” of the present disclosure are not intended to be interpreted as excluding the existence of additional embodiments or examples that also incorporate the recited features. The terms “comprising,” “including,” and “having” are intended to be inclusive and mean that there may be elements other than the listed elements. The phrase “one or more of the following: A, B, and C” means “at least one of A and/or at least one of B and/or at least one of C.”

In view of the above, it will be seen that the several objects of the disclosure are achieved and other advantageous results attained. The patentable scope of the disclosure is defined by the claims, and may include other examples that occur to those skilled in the art. Such other examples are intended to be within the scope of the claims if they have structural elements that do not differ from the literal language of the claims, or if they include equivalent structural elements with insubstantial differences from the literal language of the claims.