Patent Description:
Any background information described herein is intended to introduce the reader to various aspects of art, which may be related to the present embodiments that are described below. Accordingly, it should be understood that these statements are to be read in this light.

Immersive technology refers to technology that blurs the line between the physical world or reality and digital or simulated world or reality, thereby creating a sense of immersion, and includes virtual reality, augmented reality, augmented virtuality, and variants like mixed reality and merged reality. Virtual Reality (VR) has been defined as a realistic and immersive simulation of a three-dimensional environment, created using interactive software and hardware, and experienced or controlled by movement of the body or, simply put, an immersive, interactive experience generated by a computer. A person using VR equipment is typically able to "look around" the artificial world, move about in it and interact with features or items that are depicted on a screen or in goggles. Virtual realities artificially create sensory experiences, which may include sight, touch, hearing, and, less commonly, smell. Augmented Reality (AR) (and variants like mixed reality and merged reality) is a live direct or indirect view of a physical, real-world environment whose elements are augmented (or supplemented) by computer-generated sensory input such as sound, video, graphics or GPS data. It is related to a more general concept called mediated reality in which a view of reality is modified (possibly even diminished rather than augmented) by a computer. As a result, the technology functions by enhancing one's current perception of reality. By contrast, virtual reality replaces the real world with a simulated one. In the following, we will jointly refer to these various technologies as Immersive Reality (IR). Immersive realities may be displayed either on a computer monitor, a projector screen, or with a immersive reality headset or viewer (also called head-mounted display or HMD). HMDs typically take the form of head-mounted goggles or glasses with a screen in front of the eyes. Some simulations may include additional sensory information and provide sounds through speakers or headphones. Additionally, gloves or hand wearable devices fitted with sensors may be utilized.

Recently there has been a growth of available large field-of-view IR content (up to <NUM>°), also called immersive or panoramic. Such content is potentially not fully visible by a user watching the content on immersive display devices such as mounted displays, smart glasses, PC screens, tablets, smartphones and the like. That means that at a given moment, a user may only be viewing a part of the content. However, a user may typically navigate within the content by various means such as head movement, mouse movement, touch screen, voice and the like.

Immersive reality systems are often seen as providing solo experiences. An individual user operates alone in the experience, without interaction with other participating users. In cases where multiple users might share an IR experience simultaneously, each occupies a unique position within the IR. That position is the basis for establishing an individual user's viewpoint, from which that user's view, the image of the IR shown to the user, is rendered. This is similar to many multi-user computer-rendered simulations (e.g., video games) where the view rendered for each user appears to be from a high vantage; or as in massively multi-player online games (MMOGs) where the view of each user is rendered from the point of view of their character, at roughly human scale.

However, some kinds of immersive reality content may not be suited to having multiple users, each with a discrete viewpoint. One example of this is a three degree-of-freedom (3DoF) VR. One variety of 3DoF VR is commonly called "<NUM>° video" or a "<NUM>° image". Typically, 3DoF VR is derived from a cluster of outwardly facing cameras, where the images (or video) from the cameras in the cluster are warped and stitched to form a single, composite image (or composite video) that represents the view in all directions surrounding a viewpoint (or center point), which is the location of the camera cluster. In the case of <NUM>° video, that viewpoint may be moving, e.g., aboard a roller coaster train, in which case the resulting <NUM>° video captures the vantage of a roller coaster rider. The resulting composite image or video may be warped and cropped to give the view in one direction from that vantage, but as the user turns to face a different direction, the warp and crop is updated so that the view from that new direction is presented. When viewed by a user using a VR viewer or HMD, e.g., the Vive by HTC Corporation of Taiwan or Cardboard by Google, Inc. with an appropriately configured smartphone, then the view is updated to a new direction merely by the user turning their head to face a new direction. Alternatively, such <NUM>° media may be viewed on a smartphone or computer workstation where changes in facing are achieved by moving a mouse or dragging on a touchscreen or other user interface, such that the display is updated with a view from the viewpoint in the new direction of facing.

In some cases, where the number and coverage of the cameras is sufficient, such <NUM>° immersive media may be stereoscopic (i.e., 3D). When viewed using a stereoscopic viewer, two views are generated, both based on the same facing, but one apropos to the user's left eye, and one for the user's right eye, resulting in a 3D view from the vantage in the direction of the facing. Many examples of <NUM>° video appear on the Virtual Reality channel provided by the YouTube web site by Google, Inc. of Mountain View, California.

The 3DoF VR is not limited to <NUM>° images or video captured by cameras. Such content may be animated or computer generated. Such immersive media may be interactive. For example, facing a particular direction in the VR, perhaps at a particular time, may change the performance, as in a scene, animated or not, that might remain fairly static until the user faces a direction in the VR offering a view of a path, at which point the viewpoint might advance along that path, perhaps until some kind of decision is needed or until some next particular facing at a choice of paths, or other interaction, is required.

The present invention is defined by the appended independent claims, with embodiments set forth in the appended dependent claims, the following description and the drawings.

The present disclosure may be better understood in accordance with the following exemplary figures briefly described below:.

It should be understood that the elements shown in the figures may be implemented in various forms of hardware, software or combinations thereof. Preferably, these elements are implemented in a combination of hardware and software on one or more appropriately programmed general-purpose devices, which may include a processor, memory and input/output interfaces. Herein, the phrase "coupled" is defined to mean directly connected to or indirectly connected with, through one or more intermediate components. Such intermediate components may include both hardware and software based components.

Thus, for example, it will be appreciated by those skilled in the art that the block diagrams presented herein represent conceptual views of illustrative circuitry embodying the principles of the disclosure. Similarly, it will be appreciated that any flow charts, flow diagrams, state transition diagrams, pseudocode, and the like represent various processes which may be substantially represented in computer readable media and so executed by a computer or processor, whether or not such computer or processor is explicitly shown.

The functions of the various elements shown in the figures may be provided through the use of dedicated hardware as well as hardware capable of executing software in association with appropriate software. Moreover, explicit use of the term "processor" or "controller" should not be construed to refer exclusively to hardware capable of executing software, and may implicitly include, without limitation, digital signal processor (DSP) hardware, read only memory (ROM) for storing software, random access memory (RAM), and nonvolatile storage.

It is to be understood that the figures and descriptions have been simplified to illustrate elements that are relevant for a clear understanding of the present disclosure, while eliminating, for purposes of clarity, many other elements found in typical encoding and/or decoding devices.

It will be understood that, although the terms first and second may be used herein to describe various elements, these elements should not be limited by these terms. Various methods are described above, and each of the methods includes one or more steps or actions for achieving the described method.

In the following sections, the words "image", "picture" and "frame" may be used interchangeably.

At least one embodiment of the present disclosure is directed to immersive reality (IR) systems, hereby including at least one of virtual reality, augmented reality, augmented virtuality, mixed reality, merged reality, etc. At least one embodiment of the present disclosure is directed to social interactions among users sharing an IR experience (e.g., watching the same IR content). In particular, at least one embodiment of the present disclosure is directed to providing and/or receiving audio content among users sharing an IR experience.

When two or more individuals, each having their own IR viewer or HMD, watch synchronized presentations of an immersive media, each could look around independently as the presentation proceeds. However, as a social experience, their interactions may be awkward, perhaps depending on content. For example, in a roller coaster experience, each is likely to remain facing forward, more or less, watching what is in front of the roller coaster vehicle. But this might not always be the case. One or another of the users might glance off to the side and spy something of interest. A comment like, "Look at that!" by one user is likely to confuse the other(s). In the case of the roller coaster, a substitute comment like, "Wow, look to the right!" might resolve the issue, but relies on the speaking user to have experience with the kinds of spatially referential ambiguities and to specifically choose the statement to minimize ambiguity. In this case, the crude relative directions of "to the right" or "to the left" would be intuitively understood to be relative to the roller coaster vehicle, which has a forward facing, and thus a left and right facing, too.

In an immersive media experience lacking such a clear reference point, the user saying "Wow, look to the right!" wouldn't know and shouldn't expect that the other user(s) is(are) looking in any particular direction. For example, if a second user already happened to be looking in the intended direction, the utterance would cause the second user to look away - too far to the right. In the middle of a forest scene, "Wow, look at the rock!" may or may not be clear depending on how many rocks there are in the scene and whether any of them are in the field of view determined by the second user's facing or viewing direction or gaze direction. The second user could be flummoxed by ambiguities: Where to look if there are no rocks visible, or which rock if there are multiple rocks, or studying a rock and finding nothing of interest if the single, visible rock is the wrong one.

Accordingly, immersive media experiences such as provided in 3DoF VR or general <NUM>° IR, when presented to multiple users simultaneously, create a need for a way for those users to be able to communicate more clearly, with less ambiguity, particularly with respect to conversations relating to the environment presented in the VR. The present disclosure provides a technique that attempts to solve some of the shortcomings associated with the prior art. According to the present disclosure, the facing or viewing direction of each user (or viewer) within an IR environment is taken into consideration when the user communicates with other users. In the following, embodiments according to the present disclosure are described using an exemplary <NUM>° IR experience (for which one example is 3DoF VR), but they may similarly apply to any multi user IR system.

<FIG> illustrates a simplified block diagram of an exemplary immersive reality system <NUM> in accordance with an embodiment of the present disclosure. System <NUM> may process immersive reality content, e.g., virtual reality, augmented reality, augmented virtuality, mixed reality, merged reality, etc. System <NUM> may include a server or service provider <NUM> which is capable of receiving and processing user requests from one or more of user devices <NUM>-<NUM> to <NUM>-n. The server <NUM> may be, for example, a content server. The content server, in response to a user request for content, may provide program content including various multimedia assets such as, but not limited to, movies or TV shows for viewing, streaming or downloading by users using the devices <NUM>-<NUM> to <NUM>-n, or coupled to the devices <NUM>-<NUM> to <NUM>-n. The devices <NUM>-<NUM> to <NUM>-n may be any consumer electronic device, e.g., a gateway, a settop box, a television, a computer, a laptop, a tablet, a smart phone, etc. The server or service provider may provide other services besides content delivery.

Various exemplary user devices <NUM>-<NUM> to <NUM>-n may communicate with the exemplary server <NUM> and/or each other (e.g., in a multi-user VR game or AR experience) over a communication network <NUM> such as the Internet, a wide area network (WAN), and/or a local area network (LAN). Server <NUM> may communicate with user devices <NUM>-<NUM> to <NUM>-n in order to provide and/or receive relevant information such as recommendations, user ratings, metadata, web pages, media contents, sales offers, sales requests, etc., to and/or from user devices <NUM>-<NUM> to <NUM>-n thru the network connections. Server <NUM> may also provide additional processing of information and data when the processing is not available and/or capable of being conducted on the local user devices <NUM>-<NUM> to <NUM>-n. As an example, server <NUM> may be a computer having a processor <NUM> such as, e.g., an Intel processor, running an appropriate operating system such as, e.g., Windows <NUM> R2, Windows Server <NUM> R2, Linux operating system, etc. According to the present disclosure, processor <NUM> may execute software to perform and control the various functions and components of server <NUM>.

<FIG> also illustrates further details of server or service provider <NUM>. Processor <NUM> may control the various functions and components of the server <NUM> via a control bus <NUM>. Server <NUM> may also include a memory <NUM> which may represent at least one of a transitory memory such as RAM, and a non-transitory memory such as a ROM, a Hard Disk Drive (HDD), a Compact Disk (CD) drive or Digital Video Disk (DVD) drive, and/or a flash memory, for processing and storing different files and information as necessary, including computer program products and software, webpages, user interface information, user profiles, user recommendations, user ratings, metadata, electronic program listing information, databases, search engine software, etc., as needed. Search engine and recommender software may be stored in the non-transitory memory <NUM> of server <NUM>, as necessary, so that media recommendations may be provided, e.g., in response to a user's profile and rating of disinterest and/or interest in certain media assets, and/or for searching using criteria that a user specifies using textual input (e.g., queries using "sports", "adventure", "Angelina Jolie", etc.).

In addition, a server administrator may interact with and configure server <NUM> to run different applications using different user input/output (I/O) devices <NUM> as well known in the art. The user I/O or interface devices <NUM> of the exemplary server <NUM> may represent e.g., a mouse, touch screen capabilities of a display, a touch and/or a physical keyboard for inputting user data. The user interface devices <NUM> of the exemplary server <NUM> may also include a speaker or speakers, and/or other user indicator devices, for outputting visual and/or audio sound, user data and feedback.

Furthermore, server <NUM> may be connected to network <NUM> through a communication interface <NUM> for communicating with other servers or web sites (not shown) and one or more user devices <NUM>-<NUM> to <NUM>-n, as shown in <FIG>. The communication interface <NUM> may also represent television signal modulator and RF transmitter in the case when the content provider <NUM> represents a television station, cable or satellite television provider, or other wireless content provider. In addition, one skilled in the art would readily appreciate that other well-known server components, such as, e.g., power supplies, cooling fans, etc., may also be needed, but are not shown in <FIG> to simplify the drawing.

User devices <NUM>-<NUM> to <NUM>-n may be immersive reality video rendering devices including one or more displays. The device may employ optics such as lenses in front of each display. The display may also be a part of the immersive display device such as, for example, in the case of smartphones or tablets. In another embodiment, displays and optics may be embedded in a helmet, in glasses, or in a wearable visor which are a part of the device or coupled to the device. The immersive video rendering or user device <NUM>-<NUM> to <NUM>-n may also include one or more sensors and/or external auxiliary devices, as further described below.

User devices <NUM>-<NUM> to <NUM>-n may be one or more of but are not limited to, e.g., a PC, a laptop, a tablet, a smart phone, a smart watch, a video receiver, a smart television (TV), an HMD device or smart glasses (such as, e.g., Oculus Rift (from Oculus VR), PlayStation VR (from Sony), Gear VR (from Samsung), Google Glass (from Google), Moverio BT-<NUM> (from Epson), CastAR, Laster SeeThru, etc.), a set-top box, a gateway, or the like. An example of such devices may be, e.g., a Microsoft Windows <NUM> computer/tablet/laptop, an Android phone/tablet, an Apple IOS phone/tablet, a Sony TV receiver, or the like. A simplified block diagram of an exemplary user device according to the present disclosure is illustrated in block <NUM>-<NUM> of <FIG> as Device <NUM>, and is further described below. Similar components and features may also be present in the other user devices <NUM>-<NUM> to <NUM>-n in <FIG>.

User device <NUM>-<NUM> may be directly coupled to network/Internet <NUM> by wired or wireless means through connection or link <NUM>, or through gateway <NUM> and connections or links <NUM> and <NUM>. User device <NUM>-<NUM> may include a processor <NUM> representing at least one processor for processing various data and signals, and for controlling various functions and components of the device <NUM>-<NUM>, including video encoding/decoding and processing capabilities in order to play, display, and/or transport video content. The processor <NUM> may communicate with and controls the various functions and components of the device <NUM>-<NUM> via a control bus <NUM>.

User device <NUM>-<NUM> may also include a display <NUM> which is driven by a display driver/bus component <NUM> under the control of processor <NUM> via a display bus <NUM>. The display <NUM> may be a touch display. In addition, the type of the display <NUM> may be, e.g., Liquid Crystal Display (LCD), Light Emitting Diode (LED), Organic Light Emitting Diode (OLED), etc. In addition, an exemplary user device <NUM>-<NUM> according to the present disclosure may have its display outside of the user device, or an additional or a different external display may be used to display the content provided by the display driver/bus component <NUM>. This is illustrated, e.g., by an exemplary external display <NUM> which is connected through an external display connection <NUM> of device <NUM>-<NUM>. The connection may be a wired or a wireless connection.

Exemplary user device <NUM>-<NUM> may also include a memory <NUM> which may represent at least one of a transitory memory such as a RAM, and a non-transitory memory such as a ROM, an HDD, a CD drive, a DVD drive, and/or a flash memory, for processing and storing different files and information as necessary, including computer program products and software (e.g., as represented by flowchart diagrams <NUM> of <FIG> and <NUM> of <FIG> to be later described in detail), webpages, user interface information, databases, etc., as needed. In addition, device <NUM>-<NUM> may also include a communication interface <NUM> for coupling and communicating to/from server <NUM> and/or other devices, via, e.g., the network <NUM> using the link <NUM>, Communication interface <NUM> may also couple device <NUM>-<NUM> to gateway <NUM> using the link <NUM>. Links <NUM> and <NUM> may represent a connection through, e.g., an Ethernet network, a cable network, a FIOS network, a Wi-Fi network, and/or a cellphone network (e.g., <NUM>, <NUM>, LTE, <NUM>), etc..

One function of an immersive content rendering or user device <NUM>-<NUM> may be to control a virtual camera which captures at least a part of the content structured as a virtual volume. The system may include one or more pose tracking sensors which totally or partially track the user's pose, for example, the pose of the user's head or the facing, viewing or gaze direction of the user, in order to process the pose of the virtual camera. One or more positioning sensors may be provided to track the displacement of the user, for example, identifying the location or position of the user within the immersive reality. Sensor data that identifies pose, viewing direction and/or location is then processed to associate (or translate) the physical or real space (where the user's actions actually happen) with (to) the virtual or immersive space (where the user's actions are meant or intended to happen). The system may also include other sensors related to the environment for example to measure lighting, temperature or sound conditions. Such sensors may also be related to the body of a user, for instance, to detect or measure sweating or heart rate. Information acquired through these sensors may be used to process the content.

According to the present disclosure, an exemplary device <NUM>-<NUM> may also include a sensor <NUM>. In an exemplary embodiment, sensor <NUM> may be at least an audio sensor such as a microphone, a visual sensor such as a camera (video or picture), a gyroscope, a fluxgate compass, an accelerometer, a compass, a motion detector, a wearable hand/leg/arm/body band, a glove, a Global Positioning System (GPS) sensor, a Wi-Fi location tracking sensor, a Radio Frequency Identification (RFID) tag (or tracking tag), and/or other types of sensors as previously described.

In another non-limiting embodiment according to the present disclosure, an exemplary external sensor <NUM>, <NUM> may be separate from and coupled to the user device <NUM>-<NUM> (e.g., placed in the room walls, ceiling, doors, inside another device, on the user, etc.). The exemplary external sensor(s) <NUM>, <NUM> may have wired or wireless connections <NUM>,<NUM>, respectively, to the device <NUM>-<NUM> via an external device interface <NUM> of the device <NUM>-<NUM>, as shown in <FIG>. External sensor(s) <NUM>, <NUM> may be, e.g., a microphone, a visual sensor such as a camera (video or picture), a gyroscope, an accelerometer, a compass, a motion detector, a wearable hand/leg/arm/body band, a glove, a Global Positioning System (GPS) sensor, a Wi-Fi location tracking sensor, a Radio Frequency Identification (RFID) tag (or tracking tag), etc. In accordance with the present disclosure, sensor data, e.g., from sensor <NUM>, <NUM> and/or <NUM>, may be provided to processor <NUM> of user device <NUM>-<NUM> via processor bus <NUM> for further processing.

The processor <NUM> may process the signals received from the sensor <NUM>, <NUM>, <NUM>. Some of the measurements from the sensors may be used to compute the pose and/or position of the device and to control the virtual camera. Sensors which may be used for pose or position estimation include, for instance, gyroscopes, accelerometers or compasses. In more complex systems, a rig of cameras for example may also be used. The processor <NUM> may perform image processing to estimate the pose of an HMD. Some other measurements may be used to process the content according to environmental conditions or user reactions. Sensors used for detecting environment and user conditions include, for instance, one or more microphones, light sensor or contact sensors. More complex systems may also be used such as, for example, a video camera tracking eyes of a user. In such a case, the at least one processor <NUM> performs image processing to perform the expected measurement.

In addition, exemplary device <NUM>-<NUM> may also include user input/output (I/O) devices <NUM>. The user I/O or interface devices <NUM> of the exemplary device <NUM>-<NUM> may represent e.g., a mouse, a remote control, a joystick, a touch sensitive surface (e.g. a touchpad or a tactile screen), touch screen capabilities of a display (e.g., display <NUM> and/or <NUM>), a touch screen and/or a physical keyboard for inputting user data. The user interface devices <NUM> of the exemplary device <NUM>-<NUM> may also include a speaker or speakers, and/or other user indicator devices, for outputting visual and/or audio sound, user data and feedback. Information from user input devices may be used to process the content, manage user interfaces or to control the pose of the virtual camera.

It is to be understood that sensors <NUM>, <NUM>, <NUM> and user input devices <NUM> communicate with the processor <NUM> within the immersive rendering or user device <NUM>-<NUM> through wired or wireless communication interfaces.

In another non-limiting exemplary embodiment in accordance with the present disclosure, as shown in <FIG>, device <NUM>-<NUM> may be coupled to at least one external or auxiliary device <NUM>, via external device interface <NUM> and link <NUM>. Device <NUM> may be, e.g., a smart phone, a tablet, a remote control, a keyboard device, etc. The external device <NUM> may include a touch sensitive surface (e.g. a touchpad or a tactile screen) to be utilized as a user interface (UI).

In another non-limiting exemplary embodiment in accordance with the present disclosure, as shown in <FIG>, device <NUM>-<NUM> may be coupled to an immersive reality HMD device or smart glasses <NUM> (such as, e.g., Oculus Rift (from Oculus VR), PlayStation VR (from Sony), Gear VR (from Samsung), Google Glass (from Google), Moverio BT-<NUM> (from Epson), CastAR, Laster SeeThru, etc.), via external device interface <NUM> and link <NUM>. Notice that user device <NUM>-<NUM> may itself be an HMD device or smart glasses. Besides the inherent display, in one embodiment, the HMD device may include at least one embedded camera which may be utilized as a sensor, e.g., for localization (when observing the surroundings) or for user recognition when pointed to the user's eye (e.g., iris recognition). In one embodiment, the HMD device may also include an embedded microphone which may be utilized as a sensor or as a voice interface to accept voice commands. In one embodiment, the HMD device may also include a headphone or earbuds for providing audio.

A typical HMD has one or two small displays, with lenses and semi-transparent mirrors embedded in eyeglasses (also termed data glasses), a visor, or a helmet. The display units are miniaturized and may include cathode ray tubes (CRT), liquid crystal displays (LCDs), liquid crystal on silicon (LCos), or organic light-emitting diodes (OLED). Some vendors employ multiple micro-displays to increase total resolution and field of view.

HMDs differ in whether they can display only computer-generated imagery (CGI) or VR content, or only live imagery from the physical world, or a combination. Most HMDs can display only a computer-generated image, sometimes referred to as a virtual image. Some HMDs can allow a CGI to be superimposed on a real-world view (AR, mixed reality, merged reality, etc.) Combining real-world view with CGI can be done by projecting the CGI through a partially reflective mirror and viewing the real world directly. This method is often called optical see-through. Combining real-world view with CGI can also be done electronically by accepting video from a camera and mixing it electronically with CGI. This method is often called video see-through.

Continuing with <FIG>, exemplary user devices <NUM>-<NUM> to <NUM>-n may access different media assets, recommendations, web pages, services or databases provided by server <NUM> using, e.g., Hypertext Transfer Protocol (HTTP). A well-known web server software application which may be run by server <NUM> to service the HTTP protocol is Apache HTTP Server software. Likewise, examples of well-known media server software applications for providing multimedia programs include, e.g., Adobe Media Server and Apple HTTP Live Streaming (HLS) Server. Using media server software as mentioned above and/or other open or proprietary server software, server <NUM> may provide media content services similar to, e.g., Amazon, Netflix, or M-GO as noted before. Server <NUM> may also use a streaming protocol such as e.g., Apple HTTP Live Streaming (HLS) protocol, Adobe Real-Time Messaging Protocol (RTMP), Microsoft Silverlight Smooth Streaming Transport Protocol, etc., to transmit various programs including various multimedia assets such as, e.g., movies, TV shows, software, games, electronic books, electronic magazines, etc., to the end-user device <NUM>-<NUM> for purchase and/or viewing via streaming, downloading, receiving or the like.

In one non-limiting exemplary embodiment of the present disclosure, the sensor or sensors <NUM>, <NUM> and/or <NUM> may also be connected to the server or service provider <NUM> by wired (e.g., Ethernet cable) or wireless (e.g., <NUM> standards or Bluetooth) means (e.g., LAN or WAN network) and processor <NUM> may remotely process some or all of the sensor data.

It is to be understood that the connections or links in <FIG>, including <NUM>, <NUM>, <NUM>, <NUM> and <NUM>-<NUM> may each independently be a wired or a wireless connection.

It is to be understood that the various individual components of system <NUM> in <FIG> may be well-known circuits or mechanical components by a person of ordinary skill in the pertinent art and will not be described in detail. It is to be further understood that the example described in <FIG> is not exhaustive and other arrangements may be chosen without departing from the scope of the present disclosure.

<FIG> illustrates drawing <NUM> of user <NUM> interacting with HMD device or IR glasses <NUM> in accordance with an embodiment of the present disclosure. HMD device <NUM> may be similar to device <NUM> of <FIG> and coupled to device <NUM>-<NUM>, or may be similar to device <NUM>-<NUM> itself including HMD functionality <NUM>, as previously explained. Optionally, user <NUM> may utilize a handheld device <NUM> (e.g., a remote control, joystick, wand, etc.) coupled to and/or controlling the operation of HMD <NUM>. Handheld device <NUM> may be a device similar to device <NUM>, user I/O device <NUM> or sensor <NUM>,<NUM>. Optionally, user <NUM> may also utilize a headphone, an earphone, or ear buds <NUM> and a microphone <NUM>. Moreover, speakers may replace the headphone and be placed somewhere else in the room. Furthermore, the microphone <NUM> may also be placed somewhere else in the room.

The handheld device <NUM> and the HMD <NUM> may be coupled to a server (not shown in <FIG>) which may control what is displayed on HMD <NUM>. The IR content displayed on the HMD <NUM> may depend on the relative position and orientation of the HMD. User <NUM> watches at least a section <NUM> of the IR content <NUM> (or field of view <NUM> in case of augmented, merged or mixed reality content). As the HMD <NUM> is moved, its new location or orientation may be sent to the server and the server may update the displayed content. Line <NUM> represents a viewing or gaze direction leading to virtual point <NUM>, which is at the intersection of the viewing direction <NUM> and viewing section <NUM>. Note that line <NUM> may be seen as a vector pointing to virtual point <NUM>. Virtual point <NUM> may be the center of the viewing section <NUM>, or it may be determined by other means. For example, virtual point <NUM> may be an object on viewing section <NUM> which is being observed by user <NUM>. The determination may be based on data from a sensor focused on the eye movement of user <NUM>.

With respect to <FIG>, many combinations of devices are possible that reflect <FIG>. For example, the server may be device <NUM>-<NUM>, HMD <NUM> may be device <NUM> and handheld device <NUM> may be external device <NUM>. In another example, the server may be gateway <NUM>, HMD <NUM> may be device <NUM>-<NUM> and handheld device <NUM> may be sensor <NUM> or <NUM>. In yet another example, the server may be remote server <NUM>, HMD <NUM> may be device <NUM> or <NUM> and handheld device <NUM> may be device <NUM>-<NUM>. Other combinations are also possible without departing from the scope of the present disclosure.

The server may track the motion of HMD <NUM>. Using the position and orientation or direction (e.g., viewing direction) of the HMD <NUM>, the server may calculate the portion (position and scale) of the IR field of view that needs to be displayed on the HMD <NUM>. It is to be understood that the server may be a separate device, or it may be the HMD <NUM>.

<FIG> includes a field of view <NUM> for HMD <NUM> which represents a section of the IR content <NUM>. User <NUM> may move its head in order to move the field of view "window" <NUM> within the wider field of view <NUM> of the IR content. Moreover, user <NUM> may zoom in/out the content via settings and/or gestures (e.g., using handheld device <NUM>), or moving HMD <NUM> (or user's body) in a desired direction. In one embodiment, zooming in/out may be taken into consideration in the field of view <NUM> determination, in real time. In one embodiment, zooming in/out may not be taken into consideration in the field of view <NUM> determination, and the field of view <NUM> determination for the HMD <NUM> may be made according to a determined amount of zoom. The determined amount of zoom may be established by the user.

It is to be understood that the field of view <NUM> may be detected or determined from images by at least one video camera and/or data from sensors attached to or included in the HMD <NUM>, or by other cameras and/or sensors coupled to the system, e.g., sensors <NUM>, <NUM> and <NUM> in <FIG>. Similarly, the field of view <NUM> may be detected or determined from images by at least one video camera and/or data from sensors attached to or included in the handheld device <NUM>, or by other cameras or sensors coupled to the system, e.g., sensors <NUM>, <NUM> and <NUM> in <FIG>. The images in IR content <NUM> and <NUM> may be 2D or 3D images. The sensors may detect direction, movement and/or position of the respective devices and/or users.

The IR content may be VR content, AR content, mixed content, merged content, etc. HMD device <NUM> may operate differently depending on the type of IR content. In addition, HMD device <NUM> may operate in one of the following modes: optical see-through mode, video see-through mode or mixed optical-video see-through mode. With optical-see-through HMDs, the real world is seen through semi-transparent mirrors placed in front of the user's eyes. These mirrors are also used to reflect the computer-generated images into the user's eyes, thereby combining the real- and virtual-world views. With a video see-through HMD, the real-world view is captured with two miniature video cameras mounted on the head gear, and the computer-generated images are electronically combined with the video representation of the real world.

According to the present disclosure, the facing, viewing or gaze direction of each user (or viewer) within an IR environment is taken into consideration when the user communicates with other users. In one embodiment, when a first user speaks, the voice of the first user may be captured, along with data representing the current facing or viewing direction of the first user within the IR environment. Presentation of that captured voice to a second user may be based on the difference between the facings of the first and second users, so that the captured voice appears to the second user to come from the direction in the IR environment corresponding to where the first user is currently facing. More specifically, the captured voice may appear to the second user to come from the virtual point <NUM> representing the intersection of the first user's viewing direction <NUM> with the section of IR content <NUM> viewed by the first user <NUM>. Note that virtual point <NUM> associated with the first user <NUM> may not be included in the section of IR content viewed by the second user at the moment the first user speaks. If the first user moves or turns his head after he speaks, a new viewing direction <NUM> and a new virtual point <NUM> is determined and any new captured voice will be associated with the new virtual point <NUM>.

<FIG> illustrates a drawing <NUM> of a <NUM>° view suitable for presentation as immersive reality content in accordance with an embodiment of the present disclosure. Immersive media <NUM> may be a still image or a single frame of a <NUM>° video. Many formats for <NUM>° media are known. <FIG>, by way of example and not of limitation illustrates media <NUM> in "equirectangular" format, where each pixel of media <NUM> lies at a particular azimuth angle indicated by horizontal axis label <NUM>, and a particular elevation angle indicated by vertical axis label <NUM>. In media <NUM>, the equirectangular representation gives the portions of the view that get mapped to the high and low extremes of elevation a disproportionately large share of the image, as seen where overhead beam <NUM> becomes fat and stretches across half of the image near the top. Other formats are known, including a cube format (not shown), where the composite view surrounding the viewpoint is mapped to the six faces of a cube centered at the viewpoint, resulting in a more uniform distribution of the view (e.g., in terms of solid angles) to the media (e.g., in terms of pixels). Others formats also include tetrahedral, dodecahedral, etc. But for simplicity and clarity of explanation, rather than representational efficiency, equirectangular was chosen for this explanation.

In the case of media <NUM>, a center of interest <NUM> is a particular celebrity receiving a Hollywood Walk of Fame® star <NUM>. In this example, center of interest <NUM> is located at azimuth <NUM>°, at an elevation of about -<NUM>°. By way of demonstration, media <NUM> represents a <NUM>° environment: a right-hand portion of a first pillar 303B is visible at the left edge of media <NUM> at the split at azimuth of -<NUM>°. The left-hand portion of the first pillar 303A is visible at the right edge of media <NUM>. An additional item of interest, a second pillar <NUM>, is located at an azimuth of about <NUM>°. Another item of interest is a panel <NUM> behind celebrity <NUM>. Note that in this example, the azimuth increases on a clockwise direction and that the range <NUM>° to <NUM>° is equivalent to - <NUM>° to <NUM>°, respectively.

In one embodiment, the location of center of interest <NUM> may be represented in media <NUM> by metadata. In other embodiments, the center of interest may, by convention, be established at azimuth = <NUM>° and elevation <NUM>°, or other predetermined values. In those embodiments that establish a center of interest either by convention or with metadata, the center of interest represents the initial facing or viewing direction of a user within the VR environment. In other embodiments, metadata may establish an absolute orientation for the VR environment, e.g., if a VR-based star map in the sky is to be presented in actual alignment with the celestial sphere overhead. This may also be the case for augmented reality (AR) embodiments as well as mixed and merged reality embodiments according to the present disclosure.

<FIG> illustrates a drawing <NUM> of users interacting in an immersive reality system in accordance with an embodiment of the present disclosure. The two users <NUM> and <NUM> are participating in a shared IR experience <NUM>, including first user experience <NUM> and second user experience <NUM>, both based on media <NUM>. User experiences <NUM> and <NUM> may be in physically close proximity, e.g., in the same room or building, though perhaps not as close as shown in <FIG> (which is done for convenience of illustration). Alternately, the two user experiences <NUM> and <NUM> may be arbitrarily remote, e.g., utilizing a connection over the Internet. User experiences <NUM> and <NUM> may each be similar to the IR experience depicted in <FIG> for user <NUM>. Moreover, <FIG> illustrates user experiences in an IR system similar to the system of <FIG>.

For the sake of simplicity, it is assumed in this example that the elevation is <NUM>° throughout, both in the physical space and in the virtual space, however, the orientation or viewing direction may include azimuth and elevation as described in <FIG>. The azimuth accounts for user's gaze to the left or to the right. The elevation accounts for user's gaze looking up or down. Ultimately, the viewing direction is a vector (e.g., <NUM>) pointing to the virtual point (e.g., <NUM>) and it may have any direction, depending on the type of content. For example, the <NUM>° view (e.g., <NUM>) may wrap around in the elevation or vertical axis (e.g., <NUM>) as well.

In first user experience <NUM>, the first user <NUM> wears an HMD <NUM> similar to HMD <NUM> and <NUM>. An orientation sensor (not shown) associated with or coupled to HMD <NUM> (similar to, e.g., sensors <NUM>, <NUM> or <NUM>) detects the orientation of the first user <NUM> in the real or physical world or space. In the example of <FIG>, the orientation of the first user <NUM> is represented as a real-world orientation detection <NUM>, indicating an azimuth of -<NUM>° from magnetic north. This real-world orientation detection <NUM> is recorded as a first initial physical orientation. In first user experience <NUM>, the initial physical orientation of -<NUM>° azimuth is assigned the initial azimuth <NUM> of <NUM>° in the IR environment <NUM> presented to user <NUM> by HMD <NUM>. In the example of <FIG>, the azimuth of <NUM>° corresponds to initial reference <NUM> in the IR media <NUM>, but could be any predetermined initial facing within the immersive media <NUM>. With HMD <NUM>, user <NUM> sees a view into IR environment <NUM>, including center of interest (celebrity) <NUM> (similar to <NUM>), star <NUM> (similar to <NUM>), and the panel <NUM> (similar to panel <NUM>).

The HMD <NUM> is also associated with a microphone <NUM>, which may be connected to the HMD <NUM>, or positioned elsewhere in the room to capture the voice of user <NUM> when user <NUM> speaks. A headphone, an earphone, or ear buds <NUM> in the left ear of user <NUM> may be coupled to the HMD <NUM> (the right earphone of user <NUM> is not visible in <FIG>). In an alternative embodiment, earphones <NUM> may be provided separately (e.g., as a pair of wired or wireless headphones) having connection to HMD <NUM>. In another alternative embodiment, an array of speakers driven directly or indirectly by HMD <NUM> may be provided to deliver audio to user <NUM>.

In one embodiment, the orientation sensor may include a fluxgate compass. In other embodiments, by way of example, the orientation sensor may include a vibrating structure gyroscope (VSG) (also known as a MEMS gyroscope). Though a VSG does not provide an absolute azimuth, it may provide readings representing changes in azimuth, and the changes represented by the readings may be integrated to produce a serviceable value for orientation. In still other embodiments, the orientation sensor may include one or more accelerometers, also used to detect changes in azimuth. In yet other embodiments, the orientation sensor may be an external component (i.e., not mounted in or on the HMD <NUM>), for example including a camera that observes user <NUM>, wherein the orientation sensor detects the facing of HMD <NUM> relative to the optical axis of the camera.

In second user experience <NUM>, the second user <NUM> wears HMD <NUM>, similar to HMD <NUM>, <NUM> and <NUM>. An orientation sensor (not shown), associated with HMD <NUM>, detects the orientation of the second user in the real world, which in the example of <FIG> is represented as a real-world orientation detection <NUM>, indicating an azimuth of <NUM>° from magnetic north. This real-world orientation detection <NUM> is recorded as a second initial orientation. In second user experience <NUM>, this initial orientation of <NUM>° azimuth is assigned the azimuth <NUM> of <NUM>° in the IR environment <NUM> presented to user <NUM> by HMD <NUM>, which corresponds to the azimuth of <NUM>° in media <NUM>. With HMD <NUM>, user <NUM> sees a view into IR environment <NUM>, including center of interest (celebrity) <NUM>, star <NUM>, and panel <NUM> similar to <NUM>, <NUM> and <NUM>, respectively. Notice that view <NUM> is similar to view <NUM> for the two users (except for different perspectives in <FIG>, where they are shown as mirror images of each other, since the users <NUM> and <NUM> are facing somewhat opposite directions in the real world).

The HMD <NUM> may also be coupled with an earphone <NUM> in the right ear of user <NUM> (another earphone <NUM> in the left ear of user <NUM> is not seen in <FIG>). In an alternative embodiment, earphones <NUM> may be provided separately (e.g., as a pair of wired or wireless headphones) having connection to HMD <NUM>. In another alternative embodiment, an array of speakers driving directly or indirectly by HMD <NUM> may be provided to deliver audio to user <NUM>.

<FIG> illustrates a drawing <NUM> of users interacting in an immersive reality system in accordance with an embodiment of the present disclosure. <FIG> is similar to <FIG>, and numbers <NUM>-<NUM> correspond to numbers <NUM>-<NUM>, respectively. <FIG> may be seen as a continuation of the shared IR experience of <FIG>. Therefore, user <NUM> and <NUM> may be the same user; and users <NUM> and <NUM> may also be the same user. At this later time, experience <NUM> includes first user experience <NUM> and second user experience <NUM>, both still based on media <NUM>.

As in <FIG>, and for the sake of simplicity, it is assumed in this example that the elevation is <NUM>° throughout, both in the physical space and in the virtual space, however, the orientation or viewing direction may include azimuth and elevation as described in <FIG>.

<FIG> illustrates the situation when user <NUM> moves or turns his head <NUM>° to the right, with respect to the original position in <FIG>, producing a real-world orientation that is rotated relative to the real-world orientation of user <NUM> in <FIG>. The orientation sensor (still not shown) associated with HMD <NUM>, detects the current orientation of the first user in the real world, producing a real-world orientation detection <NUM>, in this example indicating an azimuth of -<NUM>° from magnetic north. In first user experience <NUM>, the first initial orientation of -<NUM>° (<NUM>) is subtracted from the first current orientation of -<NUM>° azimuth (<NUM>). The resulting difference is +<NUM>°. This difference is added to the azimuth <NUM> of <NUM>° assigned to the first initial orientation and results in a first current facing <NUM> having an azimuth of <NUM>°, that is, view <NUM> is. a view -<NUM>° to the right of view <NUM>. Accordingly, IR environment <NUM> is presented to user <NUM> by HMD <NUM>, on the basis of first current facing <NUM>.

Second user experience <NUM> is likewise a continuation of second user experience <NUM> at the same later time. By way of example, in second user experience <NUM>, the head of second user <NUM> is not turned relative to second user <NUM> in second user experience <NUM>. The orientation sensor (still not shown), associated with HMD <NUM>, detects the current orientation of the second user in the real world and produces real-world orientation detection <NUM> indicating an azimuth of +<NUM>° from magnetic north. In second user experience <NUM>, the second initial orientation of +<NUM>° (<NUM>) is subtracted from the second current orientation of +<NUM>° azimuth (<NUM>). The resulting difference is <NUM>°, which when added to the azimuth <NUM> of <NUM>° assigned to the second initial orientation, results in a second current facing <NUM> having an azimuth of <NUM>° (which, because the head of user <NUM> has not turned, is the same facing as in user experience <NUM>). The second current facing <NUM> corresponds to the azimuth <NUM>° in media <NUM>, so with HMD <NUM>, user <NUM> still sees a view into IR environment <NUM> directed toward center of interest (celebrity) <NUM>.

In one embodiment, the initial orientation (azimuth and/or elevation) of the view of each user may be physically based, rather than being based on a stored measurement; or may be continuously relative. Many diverse physically based orientation references are known. For example, a flux gate compass may detect magnetic north, and magnetic north (or some other fixed magnetic orientation) may be selected as the azimuth for an a priori initial orientation. Similarly, if an external camera rig were watching a user, the camera(s) could be the physical basis for determining orientation: The user being turned directly toward one of the camera(s) could be the initial orientation, while any turn away from the camera, in either of <NUM> directions (up, down, left or right), would register as a different orientation. In some implementations, the HMD may include, or the user might wear, indicia, e.g., markings or beacons, to facilitate orientation determination through the camera. In other embodiments, the indicia could be located in the physical environment surrounding the user, while a camera is worn by the user (e.g., on the HMD) such that a determination of orientation is based on observation with the camera of the environmental indicia.

Some orientation sensors, rather than reading orientation directly, detect a first or second derivative of orientation. A gyroscope may indicate change in orientation, i.e., a rotation rate. An accelerometer might indicate how quickly that rate is changing. In such cases, some form of integration is used to determine orientation from the orientation sensor readings. By way of example, using this choice, and assuming a gyroscope as an orientation sensor, the initial orientation O(t) where t=<NUM> may be taken to be zero. The next detected movement M(t) where t=<NUM> would produce a deviation from that initial orientation to produce a current orientation O(t) where t=<NUM>, which might be described as O(<NUM>) = O(<NUM>) + b*M(<NUM>), where 'b' is a scale factor depending on the sensor. Subsequently, the current orientation is based on accumulation of movements may be described as O(n+<NUM>) = O(n) + b*M(n+<NUM>). Accumulation of these incremental movements M(t) in this way provides the current orientation. Had the orientation sensor been an accelerometer, double integration would be used. Note that any real-world measurements, such as for changes in orientation, will contain noise and that by integration, this noise accumulates. Thus, the strict relationship between the original initial orientation and the current orientation may diverge. Methods to address such noise, in integration (and double integration) are available and well known by persons skilled in the art, such as squelching any readings representing change or acceleration below a predetermined threshold. Also, with all problems of indefinite integration, an initial constant (the "constant of integration") must be provided. The initial orientation is one such constant. In the case of double integration (as when using accelerometers), a second constant is needed, describing the initial rate of rotation. Zero is an appropriate choice if the user is relatively motionless when the measurement process begins.

In the shared IR experience of <FIG> and <FIG>, when first user <NUM> (or <NUM>) speaks, the voice of first user <NUM> is detected (e.g., by microphone <NUM>) and presented to second user <NUM> (e.g., through earphones <NUM>) in a way that is based upon facings <NUM> and <NUM> within the shared IR experience <NUM>, so as to appear to be coming from a direction <NUM> in IR environment <NUM> that corresponds in the shared IR environment <NUM> to the direction <NUM> in which the first user <NUM> is facing in IR environment <NUM>. More particularly, the voice of first user <NUM> appears to the second user <NUM> to be coming from the virtual point <NUM> in view <NUM> of user <NUM>, which is at the intersection of the viewing direction of user <NUM> and the IR view <NUM>.

The shared IR experience begins when the first (<NUM>, <NUM>) and second (<NUM>, <NUM>) users have established which immersive media (e.g., <NUM>) is the basis for their shared IR experience <NUM>/<NUM>; when the corresponding first (<NUM>, <NUM>) and second (<NUM>, <NUM>) HMDs have access to that media; each of the HMDs has established a corresponding first and second initial orientation in the real-world (e.g., the reading of -<NUM>° and <NUM>° from "N" on real-world facing detections <NUM> and <NUM>, respectively), if needed for subsequently determining a current orientation; when each has associated that initial orientation to correspond to a predetermined facing within the media (in example media <NUM>, an azimuth of <NUM>° corresponding with center of interest <NUM>); and when communication between the two HMDs, including synchronization if needed (e.g., for video), has been established.

After the shared IR experience begins, a first current facing <NUM> in virtual environment <NUM> is determined. In one embodiment, this is the difference between a first current real-world orientation detection <NUM> and the corresponding first initial orientation <NUM>. For example, in the case of a fluxgate compass, commonly provided in smartphones to determine orientation, an azimuth reading of -<NUM>° from magnetic north (shown in <FIG>) is established as the first initial orientation and corresponds to the azimuth of <NUM>° in immersive media <NUM>. Subsequently, a first current orientation reading of -<NUM>° (in <FIG>) produces a first current facing <NUM> azimuth of <NUM>°, which is the difference of the first current orientation <NUM> minus the first initial orientation <NUM>.

Based on the first current facing <NUM>, the first user is presented with the immersive content <NUM>, rendered by first HMD <NUM> as virtual environment <NUM>, such that, in this example, the first current facing <NUM> is directed towards pillar <NUM> and the buildings behind it.

Similarly, a second current facing <NUM> of second user <NUM> in virtual environment <NUM> is determined based on a second current orientation reading <NUM> and the second initial orientation <NUM>. The second HMD <NUM> renders immersive content (e.g., <NUM>) based on this second current facing <NUM> and presents IR environment <NUM> to second user <NUM> such that, is this example, the second current facing <NUM> is toward the center of interest (celebrity) <NUM>.

As the first user <NUM> speaks, audio (voice) content is captured, e.g., with microphone <NUM>. The audio content from the first user is processed and presented to the second user with an imposed directionality <NUM> based on difference <NUM> between the first facing <NUM> minus the second facing <NUM>, which in <FIG> is (<NUM>° - <NUM>°) = <NUM>°. By so doing, the voice of user <NUM> appears to user <NUM> to originate from a direction <NUM> in VR environment <NUM> that corresponds to the first facing <NUM> of first user <NUM> in IR environment <NUM>. The sound of the voice of user <NUM> thereby indicates to second user <NUM> where first user <NUM> is looking at the time of the first user's utterance. In particular, the sound of the voice of user <NUM> indicates to second user <NUM> the virtual point <NUM>. If the user <NUM> continues to speak while in the same position, user <NUM> may move towards the same position and eventually align with user <NUM> in the virtual space. Similarly, the voice of user <NUM> may direct user <NUM> to turn his head so as to align himself with user <NUM> in the virtual space, so that they can simultaneously observe the same section of the IR content. When both users are aligned in the virtual space, as in <FIG> (users <NUM> and <NUM> are facing the same section of the IR content), the voice of a user is heard by the other user as if coming from the virtual point in front of the user (e.g., virtual point <NUM> for user <NUM>), as will be further explained in association with <FIG>.

In one embodiment, data representing the location or position of the first user within the IR environment may also be captured, depending on the type of IR experience. For example, when the users are allowed to move independently within the IR environment, in addition to independently changing their viewing directions, then position or location of each user may be necessary to understand the geographic relationship between users in a 3D environment. Therefore, presentation of the captured voice to a second user may be further based on the locations of the first user and the second user within the IR environment, depending on the type of IR experience. In other words, the viewing directions of the users represent vectors that do not have a common origin as in the examples of <FIG>, <FIG> and <FIG>.

It is to be understood that other types of systems and video contents may be utilized without departing from the scope of the present disclosure. For example, the system may be an immersive system of display devices or panels surrounding at least one user to give a similar experience as the ones described in association with <FIG>. As a result, the at least one user may not need an HMD, but the movements and visual experience of the at least one user are similar. In addition, the system of display devices or panels may not cover <NUM>° but just an angular section, e.g., <NUM>°. Moreover, the video content may include other formats and may not cover <NUM>° but just an angular section, e.g., <NUM>°.

<FIG> illustrates a simplified block diagram of an exemplary immersive reality device <NUM> in accordance with an embodiment of the present disclosure. IR device or user station <NUM> may be similar to devices <NUM>-<NUM>, <NUM>-<NUM>,. , <NUM>-n, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM> or <NUM>. Device <NUM> may be used for sharing an IR experience in accordance with the present disclosure, and having access to an immersive media (e.g., <NUM>). Device <NUM> may include orientation sensor <NUM>, facing conversion module <NUM>, image renderer module <NUM>, display module <NUM>, microphone <NUM>, facing difference module <NUM>, audio renderer module <NUM>, audio output module <NUM>, processor <NUM> and memory <NUM>.

User station <NUM> may communicate with at least one second user station (e.g., <NUM>-<NUM> with <NUM>-<NUM>. <NUM>-n, <NUM> with <NUM>, <NUM> with <NUM>) including at least sending data representative of the audio and of the current facing output <NUM> over current communication channel (e.g., <NUM>, <NUM>, <NUM>). The communication may be by way of Internet (e.g., <NUM>), if needed, for example if the two stations (e.g., <NUM>-<NUM> and <NUM>-<NUM>) are widely separated (e.g., in different households).

User station <NUM> works as follows. Orientation sensor <NUM> may provide orientation updates to facing conversion module <NUM>. As previously discussed, orientation sensor <NUM> may rely on absolute orientation (e.g., magnetic north, facing relative to a camera or the environment, etc.), or relative orientation (e.g., for sensors that detect changes in orientation). Facing conversion module <NUM> may operate based on metadata <NUM> associated with the immersive media <NUM> (e.g., similar to IR content <NUM>). Facing conversion module <NUM> may produce a current facing <NUM>, which is supplied to image renderer module <NUM>. The current facing <NUM> may also be sent (current facing output) over a communication channel to a second user station (e.g., <NUM>-<NUM> to <NUM>-n, <NUM> and <NUM> in system <NUM>) or another device (e.g., <NUM>, <NUM>) for processing. Image renderer <NUM> may take at least a portion or section <NUM> of immersive content <NUM> and, in conjunction with a current facing <NUM> from facing conversion module <NUM>, may provide a first view to display module <NUM>, which presents the first view to the user of device <NUM> (e.g., <NUM>, <NUM>, <NUM>, <NUM>, <NUM>) so as to create VR experience (e.g., <NUM>, <NUM>, <NUM>, <NUM>, <NUM>). Audio detected by microphone <NUM> may also be sent (audio output <NUM>) over the communication channel to a second user station or another device (e.g., <NUM>, <NUM>) for processing. It is to be understood that, in other embodiments, the orientation sensor <NUM> and/or display module <NUM> may be externally coupled to device <NUM> by wired or wireless means, as previously described in association with device <NUM>-<NUM> of system <NUM>.

In one embodiment, the at least one section <NUM> of immersive media <NUM> may be determined on the basis of the current facing <NUM>, for example, where portion <NUM> represents a <NUM>° width of media <NUM> (e.g., from +<NUM>° to +<NUM>°, representing the <NUM>° centered on a current facing of azimuth = +<NUM>°). This example provides that the bandwidth requirement for delivering portion <NUM> to image renderer <NUM> is roughly <NUM>/<NUM> the requirement for delivering the whole media <NUM>, with the expectation that user will not know the difference since the unsent portion of the media <NUM> would not have been rendered into the view and seen anyway.

User station <NUM> may require access to immersive media <NUM>. In one embodiment, this access may be provided as a stream, as would be the case for a sporting event or other live show being broadcast or otherwise distributed in an IR format or other format. In one embodiment, user station <NUM> and at least one second user station (e.g., <NUM>-<NUM> to <NUM>-n) may select distinct copies of the same content <NUM> to be played in synchronization with each other. Though not shown in <FIG>, synchronized playback of independent access to a common content such as media <NUM>, is well known in the art (see, for example, the synchronization network of Scott in <CIT>). In one embodiment, separate copies of immersive content <NUM> may be local to each of a plurality of user stations (e.g., <NUM>-<NUM> to <NUM>-n), for example if immersive content <NUM> is provided as a Blu-ray disk or loaded onto a local hard drive, which would be well suited to high quality, high bandwidth content. In another embodiment, the copies may be remotely located, as when media <NUM> is to be streamed from a server (e.g., <NUM>) over the Internet (e.g., <NUM>).

Data representing current facing <NUM> from a second user station (or second user) is received by facing difference module <NUM> and subtracted from the current facing <NUM> received from facing conversion module <NUM>, resulting in an audio direction information <NUM> that is provided to the audio renderer module <NUM>. Audio renderer module <NUM> uses the audio direction information <NUM> to render the audio data <NUM> received from the second user station to appear as if coming from the audio direction identified by the facing difference module. The resulting rendered audio <NUM> is presented to the second user through audio output module <NUM>, which may drive an earphone, earbuds or speakers.

One example audio renderer module suitable for use with earphones is a binaural renderer. At its simplest, a binaural renderer will take a monophonic sound and present it in stereo to be delivered over two earphones, where a slight delay, less than <NUM>, is provided to one or the other of the channels, to simulate sounds coming from different directions. A sound coming from straight ahead (i.e., the apparent direction the sound is from has an azimuth of <NUM>°) will arrive at both ears simultaneously, so the delay is <NUM>. A sound coming from due right (i.e., from an azimuth of +<NUM>°) will arrive at the right ear first, and because the left ear is farther away from the apparent sound source, it will arrive at the left ear later, by an amount of time or delay equal to roughly the additional distance around the head (about <NUM> inches, though a precise answer depends on individual morphology) divided by the speed of sound (about <NUM> meters/second at sea level, or about <NUM> inches/second). The amount of time or delay is about <NUM>/<NUM> = <NUM> milliseconds (ms). Other delays between <NUM> and <NUM> appear to place the sound at various angles around the listener, with roughly a sine function relationship, i.e., <NUM> * sin(<NUM>°) = <NUM>, whereas <NUM> * sin(<NUM>°) = <NUM>. For a sound coming from the left, <NUM> * sin(-<NUM>°) = -<NUM>, which is a negative delay representing a need to delay right ear by <NUM>, that is, positive delays are applied to audio for the left ear, and negative delays are applied (after taking the absolute value) to the audio for the right ear. The computed delay is always supplied to rendering of the audio for the farther away ear.

In one embodiment, the audio processing may be split among different devices. For example, a second user station may receive the current facing output <NUM> and audio output <NUM> from the first user station and performs the functionalities associated with Facing Difference module <NUM>, Audio Renderer module <NUM> and Audio Output module <NUM>. As an exemplary implementation, the first user, speaking into a microphone, has his voice recorded and encoded into an ambisonic encoding, where the directionality of his voice is set to the first viewing direction relative to the virtual point (e.g., the az=<NUM>° direction in <FIG>, <FIG> and <FIG>). Upon receiving this ambisonic encoded presentation, the second user device performs the rendering of the ambisonic encoding into a binaural presentation, but with a rotation of the ambisonic field by an amount corresponding to the second viewing direction relative to the virtual point.

Note that there is nothing in the simplified rendering described above to differentiate sounds that are in front of the listener as opposed to behind, i.e., an audio direction of <NUM>° and <NUM>° both generate a <NUM> delay; a direction of +<NUM>° (ahead, to the right) and +<NUM>° (behind, to the right) both generate a <NUM> delay for the left ear. More sophisticated binaural rendering is known, based on more detailed head-related transfer functions (HRTF) that account for spectral differences in the sounds heard at each ear, the differences caused by the ear pinnae (also "auricle"), hair, difference in the lengths of various paths around the head to the far ear (e.g., around the front vs. over the top), and other even more subtle sources. An individual's HRTF may be directly measured and supplied to a more sophisticated renderer, but it is also the case that one individual's HRTF may work adequately for someone else. In some cases, the HRTFs of multiple subjects may be combined, to synthesize a generic HRTF that works adequately for the general population (though generally not as well as a customized one). Moreover, for the human brain, height may be generally differentiated by frequency, where a sound having a lower frequency may be perceived as coming from below the listener and a sound of higher frequency may be perceived as coming from above the listener. Furthermore, distance may be differentiated by volume or amplitude, where a loud sound may be perceived as close and a weak sound may be perceived as far. Hence, processing of the audio may also take advantage of these known rules.

In one embodiment, the audio output <NUM> might be an array of speakers (not shown) situated around the user head. For example, if on the headband of HMD (e.g., <NUM>, <NUM>), there may be four speakers, two speakers mounted on each side of user's head (e.g., <NUM>), one a bit in front of the right ear, one a bit behind the right ear, and two similarly disposed on the left side, then audio renderer module <NUM> may direct sound to any of those speakers. Sound directly in front, i.e., with an audio direction having an azimuth of <NUM>°, may be played simultaneously to both speakers in front of their respective ears. Sound from the front-left might be played only on the front-left speaker, although performance might be enhanced by playing sounds to the front-right side, too, but with the simple delay described above. Sounds to the rear may be played using the rear speakers to either side accordingly. In still another embodiment, the array of speakers might be located in the environment around the user, rather than be carried by user. For example, the speakers of a <NUM> sound system (not shown, but such as the one used for a household entertainment system), might be used to present sounds around a user seated among them.

In other embodiments, rather than wearing an HMDs (e.g., <NUM>), a user might be using a non-worn HMD, as might be implemented with a desktop personal computer (PC), laptop, tablet, or smartphone. Such devices are capable of providing all the modules of device <NUM>, though PCs and laptops typically lack an orientation sensor. For those instances where the HMD lacks an orientation sensor, or where the orientation sensor provided is not to be used (e.g., as a matter of user preference), the current facing may be based on a pointing input device, e.g., a mouse, flying mouse, track pad, touch screen, game controller, or the like, which allows the user to command a change in facing. Such a command, is received an acted on by the facing conversion module, which otherwise operates as herein described. A camera may also be used to identify a facing direction of a user. Either or both of users may employ a non-worn HMD to participate in the shared VR experience.

It is to be understood that at least some modules of user station <NUM> may be implemented in software stored in memory <NUM> and executed by processor <NUM>, e.g., facing conversion module <NUM>, image renderer module <NUM>, facing difference module <NUM>, audio renderer module <NUM>. Although not shown in <FIG>, processor <NUM> may control operation of all the other modules of device <NUM>.

<FIG> illustrates a flowchart <NUM> of an exemplary method of providing audio content in accordance with the present disclosure. The method includes at step <NUM>, receiving a first audio content from a first user. Then, at step <NUM>, the method includes processing the first audio content based on a first viewing direction for the first user facing at least one first section of an audiovisual content and on a second viewing direction for a second user facing at least one second section of the audiovisual content. Finally, at step <NUM>, the method includes providing the processed first audio content. The at least one first section of the audiovisual content is being displayed to the first user. And the at least one second section of the audiovisual content is being displayed to the second user. The steps of the method may be performed by a processor, e.g., <NUM>, <NUM>, <NUM>. Alternately, the step of processing may be performed by, e.g., facing difference module <NUM> and audio renderer module <NUM>. The step of providing may be performed by, e.g., audio output module <NUM>. The providing step may provide for audio playback, store in memory (e.g., <NUM>, <NUM>, <NUM>), provide to the second user and/or provide to the first user.

In one claimed embodiment of the method, the first audio content is processed based on an angular difference between the first viewing direction and the second viewing direction. The angular difference may be determined by, e.g., facing difference module <NUM>. The audio processing may be performed by, e.g., audio renderer module <NUM>.

In the claimed embodiment of the method, the processed first audio content is perceived by the second user as originating from a virtual point in the at least one first section of the audiovisual content, the virtual point being an intersection between the first viewing direction and the at least one first section of the audiovisual content.

In one embodiment of the method, a left audio component and a right audio component of the processed first audio content are adjusted according to the angular difference. The audio processing may be performed, e.g., in audio renderer module <NUM>.

In one embodiment of the method, the left audio component is delayed in time with respect to the right audio component when the angular difference between the first viewing direction and the second viewing direction is between zero degree (<NUM>°) and one hundred and eight degrees (<NUM>°). In other words, the virtual point or the first viewing direction is to the right of the second viewing direction.

In one embodiment of the method, the right audio component is delayed with respect to the left audio component amplitude when the angular difference between the first viewing direction and the second viewing direction is between minus one hundred and eighty degrees (-<NUM>°) and zero degree (<NUM>°) or, equivalently, between one hundred and eighty degrees (<NUM>°) and three hundred and sixty degrees (<NUM>°). In other words, the virtual point or the first viewing direction is to the left of the second viewing direction.

In one embodiment of the method, the processed first audio content is provided to the second user for audio playback. The providing may be performed, e.g., in audio output module <NUM>.

In one embodiment of the method, a viewing direction is a function of at least one of a head movement and eye movement of the user.

In one embodiment of the method, the audiovisual content is an immersive reality content.

In one embodiment of the method, the audiovisual content is a <NUM>-degree content.

In one embodiment of the method, the audiovisual content is a 3DoF immersive reality content.

In one embodiment, the method may be performed by one of a first user's device, a second user's device and a third device (e.g., a server).

The method <NUM> may be performed by, e.g., device <NUM>-<NUM>, <NUM>-<NUM>,. <NUM>-n, <NUM>, <NUM>, <NUM>, <NUM>, <NUM> or <NUM>, including any of the embodiments previously described. In particular, the steps of the method may be performed by, e.g., processor <NUM>, <NUM> or <NUM>.

According to one embodiment of the present disclosure, an apparatus <NUM>-<NUM>,<NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM> for providing audio content is provided including a processor <NUM>, <NUM>, <NUM> and at least one memory <NUM>, <NUM>, <NUM> coupled to the processor, the processor being configured to perform any of the embodiments of the method <NUM>.

It is to be understood that the term processor may represent at least one processor performing the functions described above in a joint architecture, serial, parallel or mixed.

<FIG> illustrates a flowchart <NUM> of an exemplary method of receiving audio content in accordance with the present disclosure. The method includes at step <NUM>, receiving a processed first audio content from a first user based on a first viewing direction for the first user facing at least one first section of an audiovisual content and on a second viewing direction for a second user facing at least one second section of the audiovisual. Then, at step <NUM>, the method includes outputting the processed first audio content for audio playback. The at least one first section of the audiovisual content is being displayed to the first user. And the at least one second section of the audiovisual content is being displayed to the second user. The steps of the method may be performed by a processor, e.g., <NUM>, <NUM>, <NUM>. Alternately, the step of providing may be performed by, e.g., audio output module <NUM>.

In a claimed embodiment of the method, the first audio content is processed based on an angular difference between the first viewing direction and the second viewing direction. The angular difference may be determined by, e.g., facing difference module <NUM>. The audio processing may be performed by, e.g., audio renderer module <NUM>.

In this claimed embodiment of the method, the processed first audio content is perceived by the second user as originating from a virtual point in the at least one first section of the audiovisual content, the virtual point being an intersection between the first viewing direction and the at least one first section of the audiovisual content.

It is important to note that one or more of the elements in the process <NUM> may be combined, performed in a different order, or excluded in some embodiments while still implementing the aspects of the present disclosure.

According to one embodiment of the present disclosure, an apparatus <NUM>-<NUM>,<NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM> for receiving audio content is provided including a processor <NUM>, <NUM>, <NUM> and at least one memory <NUM>, <NUM>, <NUM> coupled to the processor, the processor being configured to perform any of the embodiments of the method <NUM>.

Moreover, methods <NUM> and <NUM> may be implemented as computer program products including computer executable instructions which may be executed by a processor. The computer program product having the computer-executable instructions may be stored in the respective non-transitory computer-readable storage media of the respective above-mentioned devices, e.g., <NUM>, <NUM>, <NUM>.

According to one aspect of the present disclosure, a non-transitory computer program product is provided including program code instructions for performing any of the embodiments of the method <NUM> of providing audio content.

According to one aspect of the present disclosure, a non-transitory article of manufacture is provided tangibly embodying computer readable program code instructions which when executed cause a computer to perform any of the embodiments the method <NUM> of providing audio content.

According to one aspect of the present disclosure, a computer program product is provided including code instructions executable by a processor for implementing any of the embodiments the method <NUM> of providing audio content.

According to one aspect of the present disclosure, a non-transitory computer program product is provided including program code instructions for performing any of the embodiments of the method <NUM> of receiving audio content.

According to one aspect of the present disclosure, a non-transitory article of manufacture is provided tangibly embodying computer readable program code instructions which when executed cause a computer to perform any of the embodiments the method <NUM> of receiving audio content.

According to one aspect of the present disclosure, a computer program product is provided including code instructions executable by a processor for implementing any of the embodiments the method <NUM> of receiving audio content.

Furthermore, aspects of the present disclosure can take the form of a computer-readable storage medium. Any combination of one or more computer-readable storage medium(s) may be utilized. A computer-readable storage medium can take the form of a computer-readable program product embodied in one or more computer-readable medium(s) and having computer-readable program code embodied thereon that is executable by a computer. A computer-readable storage medium as used herein is considered a non-transitory storage medium given the inherent capability to store the information therein as well as the inherent capability to provide retrieval of the information therefrom. A computer-readable storage medium can be, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing.

It is to be appreciated that the following, while providing more specific examples of computer-readable storage mediums to which the present disclosure can be applied, is merely an illustrative and not exhaustive listing as is readily appreciated by one of ordinary skill in the art: a portable computer diskette, an HDD, a ROM, an erasable programmable read-only memory (EPROM or Flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

According to an aspect of the present disclosure, a computer-readable storage medium is provided carrying a software program including program code instructions for performing any of the embodiments of the method <NUM> of providing audio content.

According to an aspect of the present disclosure, a computer-readable storage medium is provided carrying a software program including program code instructions for performing any of the embodiments of the method <NUM> of receiving audio content.

It is to be understood that reference to "one embodiment" or "an embodiment" or "one implementation" or "an implementation" of the present disclosure, as well as other variations thereof, mean that a particular feature, structure, characteristic, and so forth described in connection with the embodiment is included in at least one embodiment of the present disclosure.

As noted before, the functions of the various elements shown in the figures may be provided through the use of dedicated hardware as well as hardware capable of executing software in association with appropriate software. Also, when provided by a processor, the functions may be provided by a single dedicated processor, by a single shared processor, or by a plurality of individual processors, some of which may be shared.

It is to be further understood that, because some of the constituent system components and methods depicted in the accompanying drawings are preferably implemented in software, the actual connections between the system components or the process function blocks may differ depending upon the manner in which the present disclosure is programmed. Given the teachings herein, one of ordinary skill in the pertinent art will be able to contemplate these and similar implementations or configurations of the present disclosure.

Claim 1:
A method (<NUM>) comprising:
receiving (<NUM>) a first audio content from a first user;
processing (<NUM>) the first audio content based on a first viewing direction for the first user facing at least one first section of an audiovisual content and on a second viewing direction for a second user facing at least one second section of the audiovisual content; and
providing (<NUM>) the processed first audio content to the second user with a directionality based on an angular difference between the first viewing direction and the second viewing direction, wherein the processed first audio content is perceived by the second user as originating from a virtual point, determined based on an intersection between the first viewing direction and the at least one first section of the audiovisual content.