Patent Description:
VR refers to a particular environment or context that is artificially created using a computer that is similar to a natural environment, or a technology therefor. An environment or context provided to the user through VR content may stimulate the user's five senses to allow the user to feel a spatial or temporal experience similar to reality. The user may be engrossed in VR content and also be able to manipulate or command a real device to interact with objects implemented in the VR content. VR content may be distinct from simulations that are uni-laterally implemented in that it may interact with users and create experiences with users.

VR content may be provided to the user via a VR device such as a head mounted display (HMD) that is put on the user's head, with a display that is positioned before the user's eyes and displays VR content.

The user may move his/her head in various directions while using the VR device, thus changing the direction in which she is viewing. To provide lifelike VR content, VR devices should be able to provide VR content that reflects changes in the direction of the user's view.

<CIT> discloses an apparatus for generating a sequence of stereoscopic images for a head-mounted display depicting a virtual environment including an angular motion sensor that outputs an indication of the orientation of the head mounted display, a texture buffer that is refreshed with left and right textures that define left and right pre-rendered scenes in the virtual environment, a rendering processor which then renders left and right images from perspective render viewpoints determined by the output of the angular motion sensor, by mapping the textures onto respectively left and right spheres of polyhedrons, the left and right rendered images then being provided to a stereoscopic display in the head-mounted display, and the rendering processor renders the left and right images at a higher rate than the left and right textures are refreshed in the texture buffer.

The present disclosure provides a method for providing VR content that may reflect the user changing a direction in which she is viewing (e.g., turning or moving her head), and an apparatus therefor.

According to an aspect of the present disclosure, a VR device may provide VR content reflecting the user changing the direction in which she is viewing (e.g., by turning or moving her head), allowing more lifelike VR experiences for users.

Further, the VR device may identify the area the user is viewing and transmitting VR content from, improving the transmission efficiency of VR content.

The above and other aspects, features, and advantages of the present disclosure will be more apparent from the following detailed description when taken in conjunction with the accompanying drawings, in which:.

According to an embodiment of the present disclosure, a method for processing VR content by a head mounted display, HMD, device includes identifying cartesian coordinates of a first position on the VR content, estimating a movement of a user of the content providing device, identifying cartesian coordinates of a second position by applying a matrix representing the estimated movement of the user to the cartesian coordinates of the first position, converting the cartesian coordinates of the second position into spherical coordinates of the second position, and providing an area corresponding to the spherical coordinates of the second position to the user. The method is defined by claim <NUM>.

According to another embodiment of the present disclosure, a head mounted display, HMD, device according to claim <NUM> is configured to be operated according to the method for processing the VR content.

Embodiments of the present disclosure are described in detail with reference to the accompanying drawings. The same or similar components may be designated by the same or similar reference numerals although they are illustrated in different drawings. Detailed descriptions of constructions or processes known in the art may be omitted to avoid obscuring the subject matter of the present disclosure.

Although the terms "first" and "second" are used to describe various components, the components are not limited by the terms. These terms are provided simply to distinguish one component from another. Accordingly, a first component described herein may be referred to as a second component and vice versa within the technical spirit of the present disclosure.

VR content, projection structure, a user's viewing perspective, and coordinate systems of a VR device, according to an embodiment of the present disclosure, are described with reference to <FIG>.

<FIG> illustrates a projection structure applied to a VR device, according to an embodiment of the present disclosure. <FIG> illustrates a user's viewing perspective in a VR device, according to an embodiment of the present disclosure. <FIG> illustrates, on coordinate axes, a method for representing the rotation of a user's view direction in a VR device, according to an embodiment of the present disclosure.

VR content is assumed to be omnidirectional content that is mapped to <NUM>-degree directions around the user who uses a VR device. However, VR content need not necessarily include all content corresponding to the whole <NUM>-degree area, but may rather include only content corresponding to a part of the area. VR content may include content elements, such as audio content, video content, image content, subtitles, smells, or winds, which may be classified into virtual content, auditory content, olfactory content, or tactile content. When produced, at least some of the elements of the VR content may be sorted in corresponding directions of <NUM>-degree directions.

A projection structure is described with reference to <FIG>.

Visual content may be produced in <NUM>-degree directions, implementing a virtual spherical space <NUM>. It may be assumed that a user <NUM> wearing a VR device <NUM> is positioned at the center of the virtual spherical space <NUM> to observe the inner surface of the virtual spherical space <NUM>. In other words, VR content produced may be projected to the virtual spherical space <NUM>, which is referred to as a projection structure. Herein, the projection structure is assumed to be a unit sphere with a radius (r) of <NUM>.

VR content is provided to the user through the VR device. In this disclosure, the VR device is assumed to be an HMD that is put on the user's head, with a display that is positioned before the user's eyes and displays VR content.

Viewing perspectives of the VR device and the user are described below with reference to <FIG>.

The user <NUM> wearing the VR device <NUM> may move her head in various directions while in use of the VR device <NUM> and the direction in which the user <NUM> is viewing may change accordingly. At this time, the direction in which the user <NUM> is viewing is defined as the viewing perspective <NUM>. The viewing perspective <NUM> of the VR device may mean the viewpoint that is the center point of the viewport which is an area of the VR content being currently provided to the user <NUM>.

Specifically, the viewing perspective <NUM> of the VR device <NUM> is assumed to be a direction in which a straight line passing through the center of the surface of the display screen included in the VR device and perpendicular to the surface of the display area is oriented towards the outside of the user <NUM> wearing the VR device <NUM>.

For example, assuming that the user <NUM> wearing the VR device <NUM> is positioned at the center of the virtual spherical space <NUM>, the direction in which the user <NUM> is viewing from the center of the virtual sphere <NUM> through the inner surface of the sphere to the outside may be assumed to be the viewing perspective <NUM> of the VR device <NUM>. Further, under the assumption that the virtual spherical space <NUM> has a predetermined r, the point where the straight line perpendicular to the display area and passing through the center of the surface of the display screen included in the VR device <NUM> as viewed from the center of the virtual sphere <NUM> to the outside meets the inner surface of the virtual sphere <NUM> may be defined as the viewing perspective <NUM> of the VR device <NUM>. Here, the virtual spherical space <NUM> may correspond to a virtual space that the VR content implements.

The viewing perspective <NUM> of the VR device <NUM> may be the same as the viewing perspective <NUM> of the user <NUM> wearing the VR device <NUM>. At this time, the user <NUM> is assumed to be gazing at the display screen of the VR device <NUM>. The user's viewing perspective <NUM> may be defined as the direction in which a straight line passing through the center of a straight line connecting the user's eyes is directed to the outside of the user.

Although the viewing perspective <NUM> of the VR device <NUM> and the user's viewing perspective <NUM> are defined as above, the viewing perspective of the VR device and the user's viewing perspective may be defined in other various ways, according to embodiments of the present disclosure.

Further, where a sensor tracking the movement of the user's pupils is equipped in the VR device, the direction in which the user's pupils move may be defined as the user's viewing perspective in which case the user's viewing perspective may be different from the viewing perspective of the VR device.

Coordinate systems applied in this disclosure are described below with reference to <FIG>.

The viewing perspective of the VR device, the user's viewing perspective, the user's motion, and VR content are assumed to move on the same coordinate axes. The viewing perspective of the VR device, the user's viewing perspective, the user's motion, and VR content may be represented as spherical coordinates or cartesian coordinates on the same coordinate axes.

As shown in <FIG>, the center point of the VR device <NUM> or the center of the head of the user <NUM> are assumed to be positioned at the center of the coordinate axes. The center of the VR space that the VR content implements may be assumed to be positioned at the center of the coordinate axes, or the position where visual VR content is obtained may be assumed to be positioned at the center of the coordinate axes.

The movement of the user's viewing perspective may be represented with a yaw angle, a pitch angle, and a roll angle. The yaw angle α may mean an angle at which the user's viewing perspective rotates around the y axis. The yaw angle α may be defined in the range from -<NUM> degrees to <NUM> degrees and as rotation counterclockwise as viewed from above the y axis to the center of the coordinate axes. The pitch angle β may mean an angle at which the user's viewing perspective rotates around the x axis. The pitch angle β may be defined in the range from -<NUM> degrees to <NUM> degrees and as rotation counterclockwise as viewed from above the x axis to the center of the coordinate axes. The roll angle γ may mean an angle at which the user's viewing perspective rotates around the z axis. The roll angle γ may be defined in the range from -<NUM> degrees to <NUM> degrees and as rotation counterclockwise as viewed from above the z axis to the center of the coordinate axes.

On the other hand, according to an embodiment of the present disclosure, the yaw angle α may mean an angle at which the user's viewing perspective rotates around the z axis, the pitch angle β may mean an angle at which the user's viewing perspective rotates around the y axis, and the roll angle γ may mean an angle at which the user's viewing perspective rotates around the x axis.

Alternatively, the yaw angle α may mean an angle at which the user's viewing perspective rotates around the x axis, the pitch angle β may mean an angle at which the user's viewing perspective rotates around the z axis, and the roll angle γ may mean an angle at which the user's viewing perspective rotates around the y axis.

In other words, the yaw angle, pitch angle, and roll angle may be defined based on different rotation axes as long as the same reference is used in the internal system of one VR device.

The VR device <NUM> includes a location sensor. When the location sensor detects a variation in the viewing perspective of the VR device <NUM>, the location sensor obtains information about the movement of the viewing perspective of the VR device <NUM> and provides the information to the VR device <NUM>. The location sensor may provide the yaw angle α, pitch angle β, and roll angle γ for the movement of the viewing perspective of the VR device <NUM>. At this time, the order in which the yaw, pitch, and roll components are measured or obtained for the movement of the viewing perspective of the VR device <NUM> may be previously defined. In other words, the order in which the movement of the viewing perspective of the VR device <NUM> is measured or obtained may be previously set in the location sensor or VR device. For example, the location sensor may measure or obtain the yaw angle α, pitch angle β, and roll angle γ in the order thereof for the movement of the viewing perspective of the VR device <NUM>.

The location sensor may include at least one or more various sensors, such as an accelerometer for measuring acceleration, a gyroscope for measuring angular speed, or a magnetometer which is a geo-magnetic sensor. As an example of measuring the movement of the viewing perspective of the VR device, the roll angle or pitch angle may be measured by an accelerometer or a gyroscope, and the yaw angle may be measured by a gyroscope or a magnetometer. However, this is merely an example of a method for obtaining information about the movement of the viewing perspective of the VR device. Embodiments of the present disclosure are not limited thereto and any other various types of location sensors or measuring methods may be adopted as long as they are able to measure the movement of the VR device.

A method for operating a VR device, according to an embodiment of the present disclosure, is described below with reference to <FIG> and <FIG>.

<FIG> illustrates a view direction of a VR device, according to an embodiment of the present disclosure. <FIG> is a flowchart illustrating a method for operating a VR device, according to an embodiment of the present disclosure. <FIG> illustrates a relationship between spherical coordinates and cartesian coordinates in a VR device, according to an embodiment of the present disclosure.

Referring to <FIG>, a user's first viewing perspective <NUM> is assumed as a reference in a virtual spherical space <NUM>. The user's first viewing perspective <NUM> may be represented as spherical coordinates (r1, Φ1, θ1). Here, since the r of the virtual spherical space <NUM> is assumed as <NUM>, the following is assumed: r1=<NUM>. The user's first viewing perspective <NUM> may mean a first viewpoint that is the center point of a first viewport which is an area of the VR content being currently provided to the user <NUM>.

The user <NUM> wearing the VR device <NUM> may move her head in various directions while in use of the VR device <NUM>, and according to the movement, the user's viewing perspective may change from the user's first viewing perspective <NUM> to a user's second viewing perspective <NUM>'. The VR device <NUM> may provide VR content reflecting the change in the user's viewing perspective. At this time, the movement of the user's viewing perspective should be reflected in the VR content provided to the user <NUM>. The user's second viewing perspective <NUM>' may mean a second viewpoint that is the center point of a second viewport which is an area of the VR content is provided to the user <NUM> after the user has moved.

Referring to <FIG> and <FIG>, the user's first position is identified in step <NUM>. Here, the user's first position means the user's first viewing perspective <NUM>. The user's first position may be a reference point and start point of the VR content, and the user's first position may previously be defined and stored in the VR device, or the user's first position may be measured and obtained by a location sensor. Alternatively, the user's first position may be obtained by reflecting a user's movement measured by a location sensor in a prior position to the first position. In this disclosure, the user's first position is assumed to be identified as spherical coordinates, (<NUM>, Φ1, θ1). However, the user's first position may also be identified as cartesian coordinates (x1, x2, x3). In this case, step <NUM>, described below, may be skipped.

Next, the user's movement is measured in step <NUM>. Here, measuring the user's movement may mean measuring the user's movement from the user's first position to the user's second position. In other words, measuring the user's movement may mean measuring the movement of the user's viewing perspective from the user's first viewing perspective <NUM> to the user's second viewing perspective <NUM>'. The movement of the user's viewing perspective may be measured by the location sensor included in the VR device. The method in which the location sensor measures the user's viewing perspective has been described above, and no detailed description thereof is thus presented below. The movement from the user's first position to the user's second position may be provided as yaw angle α, pitch angle β, and roll angle γ by the location sensor.

To apply the yaw angle α, pitch angle β, and roll angle γ to the user's first position, the user's first position represented as spherical coordinates may be converted into cartesian coordinates in step <NUM>.

Referring to <FIG>, the cartesian coordinates (x1,y1,z1) converted from the user's first position (r1, Φ1, θ1) may be obtained using Equations (<NUM>) to (<NUM>) as follows: <MAT>
<MAT><MAT>.

Steps <NUM> to <NUM> need not necessarily follow the order shown in <FIG> or as set forth above, and may be performed in a different order, or some of the steps may simultaneously be performed.

Referring again to <FIG> and <FIG>, the user's first position cartesian coordinates X<NUM>=(x1, y1, z1) are rotated and converted to identify the user's second position at step <NUM>. Here, the user's second position means the user's second viewing perspective <NUM>'. The conversion from the user's first position to the user's second position may be achieved via use of a rotation matrix. As a yaw angle α rotation matrix (Ryaw), Equation (<NUM>) may be used, and this may be multiplied by the user's first position cartesian coordinates (X<NUM>) as represented in Equation (<NUM>), obtaining the yaw angle rotation position (Xyaw). <MAT><MAT>.

As a pitch angle β rotation matrix (Rpitch), Equation (<NUM>) may be used, and this may be multiplied by the user's yaw angle rotation position (Xyaw) as represented in Equation (<NUM>), obtaining the pitch angle rotation position (Xpitch). <MAT><MAT>.

As shown in <FIG>, the roll angle γ may be <NUM>, and the user's movement may be represented with only yaw angle α and pitch angle β. In this case, the user's second position cartesian coordinates may be obtained as follows: X<NUM>=(x2, y2, z2)= Xpitch.

In some cases, it is necesarry to convert the roll angle γ, in which case the yaw angle α rotation matrix (Ryaw) and the pitch angle β rotation matrix (Rpitch) may properly be multiplied to obtain the roll angle γ rotation matrix (Rroll).

The roll angle γ rotation matrix (Rroll) is as represented in Equation (<NUM>), and may be multiplied by the user's pitch angle rotation position (Xpitch) as represented in Equation (<NUM>), to obtain the roll angle rotation position (Xroll). <MAT><MAT>.

In this case, the user's second position cartesian coordinates may be obtained as follows: X<NUM>=Xroll.

The order of multiplying the rotation matrix is not limited to that shown above as long as at least one of an order of multiplying a yaw angle rotation matrix, pitch angle rotation matrix, is defined; or a roll angle rotation matrix for the yaw angle, pitch angle, or roll angle rotation axis reference used in the internal system of one VR device is defined.

As set forth above, an embodiment of the present disclosure is also possible where the yaw angle means an angle at which the user's viewing perspective rotates around the z axis, the pitch angle means an angle at which the user's viewing perspective rotates around the y axis, and the roll angle means an angle at which the user's viewing perspective rotates around the x axis.

In this case, Equation (<NUM>) may be used as yaw angle rotation matrix (Ryaw) for rotation by p around the z axis.

Equation (<NUM>) may be used as pitch angle rotation matrix (Rpitch) for rotation by q around the y axis.

Equation (<NUM>) may be used as roll angle rotation matrix (Rroll) for rotation by v around the x axis.

In this case, the user's second position cartesian coordinates, X<NUM>=(x2, y2, z2), may be obtained as shown in Equation (<NUM>).

Although in the above embodiment of the present disclosure, the user's first position cartesian coordinates, X<NUM>=(x1, y1, z1), are multiplied by the yaw angle rotation matrix (Ryaw(p)), pitch angle rotation matrix (Rpitch(q)), and the roll angle rotation matrix (Rroll(v)) in the order thereof to obtain the user's second cartesian coordinates, X<NUM>=(x2, y2, z2), the order of multiplying the rotation matrices is not limited thereto, as long as an order of multiplying the roll angle rotation matrix, pitch angle rotation matrix, and roll angle rotation matrix for the yaw angle, pitch angle, and roll angle rotation axis references used in the internal system of one VR device is defined.

When the angle at which the yaw angle rotates around the z axis, the pitch angle rotates around the y axis, and the roll angle rotates around the x axis in the internal system of the VR device, the location sensor is set to measure or obtain the user's movement in the order of yaw angle (p), pitch angle (q), and roll angle (v) such that the user's first cartesian coordinates (X<NUM>) may be multiplied by the roll angle rotation matrix (Ryaw(p)), pitch angle rotation matrix (Rpitch(q)), and roll angle rotation matrix (Rroll(v)) in the order thereof, obtaining the user's second position cartesian coordinates. X<NUM>=(x2, y2, z2).

Next, the user's identified second position cartesian coordinates (X<NUM>) are converted into the spherical coordinates (r2, Φ2, θ2) at step <NUM>.

The user's second position cartesian coordinates, X<NUM>=(x2, y2, z2), may be converted into the user's second position spherical coordinates (r2, Φ2, θ2) using Equations (<NUM>) to (<NUM>). Here, it be assumed that r2=<NUM>. <MAT> <MAT><MAT>.

Here, the reason why Equations (<NUM>) and (<NUM>) are multiplied by <NUM>°/π is to match the radian value to the degree value.

Where z2 is <NUM> in Equation (<NUM>), any value may be used as z2.

In the case where calculation is impossible, such as when z2 is represented as in Equations (<NUM>) and (<NUM>), or if it is ambiguous to specify a resultant value given the diverging form of the graph of tan-<NUM>, the rule may be defined to use a particular value. <MAT><MAT>.

Conversion among Equations (<NUM>) to (<NUM>) as per coordinate axes is described below with reference to <FIG>.

<FIG> illustrates coordinate axes, according to an embodiment of the present disclosure.

For the coordinate axes shown in <FIG>, Equations (<NUM>) to (<NUM>) may be represented as Equations (<NUM>) to (<NUM>). <MAT><MAT><MAT>.

Upon obtaining the user's second position spherical coordinates (r2, Φ2, θ2), the VR device may provide the user with the area corresponding to the user's second position spherical coordinates (r2, Φ2, θ2) of the VR content.

The area corresponding to the user's second position spherical coordinates (r2, Φ2, θ2) may be an area corresponding to the field of view (FOV) of the VR device with respect to the point corresponding to the user's second position spherical coordinates (r2, Φ2, θ2) specified on the projection structure of the VR content. Or, the area corresponding to the user's second position spherical coordinates (r2, Φ2, θ2) may be an area corresponding to the area of the viewport of the VR device with respect to the point corresponding to the user's second position on the projection structure. Alternatively, the area corresponding to the user's second position spherical coordinates (r2, Φ2, θ2) may be an area pre-determined in upper-lower and left-right ranges with respect to the point corresponding to the user's second position on the projection structure. On the other hand, the area corresponding to the user's second position spherical coordinates (r2, Φ2, θ2) may be an area specified by the second position of N samples obtained by applying a rotation matrix, as in the method of applying N sample points, around the user's viewing perspective, to the user's viewing perspective.

The area corresponding to the user's second position spherical coordinates (r2, Φ2, θ2) of the VR content may be specified as long as the VR content is provided so that the point corresponding to the user's second position spherical coordinates (r2, Φ2, θ2) of the VR content is positioned at or near the center of the FOV of the user gazing in the second user's viewing perspective.

VR content may be stored in a memory of the VR device, and the area corresponding to the user's second position spherical coordinates of the VR content may be provided to the user. VR content may be stored in a server, and when the VR device transmits the user's first position spherical coordinates information and the user's movement information to the server, the server calculates the second position cartesian coordinates and provides the VR content of the area corresponding to the second position cartesian coordinates to the VR device. When the VR device transmits second position cartesian coordinates information to the server, the server may transmit the VR content of the area corresponding thereto to the VR device.

A method for obtaining the user's second position spherical coordinates when the user's identified second position cartesian coordinates correspond to a polar value is described below with reference to <FIG> and <FIG>, according to an embodiment of the present disclosure.

<FIG> schematically illustrates a method for obtaining spherical coordinates of an identified user's second position using an ambient sample when the cartesian coordinates of the user's second position correspond to a polar value. <FIG> is a flowchart illustrating a method for operating a VR device, according to an embodiment of the present disclosure.

The polar value refers to the direction up the user's head and may be a value on the y axis, which is directed up from the head of the user <NUM> wearing the VR device <NUM> as shown in <FIG>. In other words, when the user's second position cartesian coordinates are (<NUM>, <NUM>, y2) on the y axis, the user may be said to be position at the polar value. However, the polar value is not limited to any value on the y axis, and the polar value may be determined as a different value depending on coordinate axes or the plane where the user is positioned. For example, when the user is positioned on the xy plane and the z axis is directed up the user's head, a value on the z axis may become the polar value.

Obtaining the user's second position spherical coordinates (r2, Φ2, θ2) when the user's second position cartesian coordinates are (<NUM>, <NUM>, y2) on the y axis will be described. r2 may be assumed to be <NUM>.

The step <NUM> of identifying the user's first position, the step <NUM> of measuring the user's movement, the step <NUM> of converting the user's first position represented as spherical coordinates into cartesian coordinates, and the step <NUM> of rotating and converting the user's first position cartesian coordinates to identify the user's second position may be performed as described above with reference to the aforementioned embodiments.

Next, in step <NUM>, it is determined whether the user's second position identified is the polar value. Unless the user's second cartesian coordinates are the polar value, the user's second position cartesian coordinates identified are simply converted into spherical coordinates in <NUM>.

However, where the user's second position cartesian coordinates are P'=(<NUM>, <NUM>, y2), the first positions of N points around the user's first position cartesian coordinates P are identified in step <NUM>. In <FIG>, four points s1, s2, s3, and s4 are selected around the user's first position cartesian coordinates P.

Next, the aforementioned rotation matrix are applied to each of the four points s1, s2, s3, and s4 around the user's first position cartesian coordinates P, identifying the second positions s1', s2', s3', and s4' of the four points in step <NUM>.

Then, the second positions s1', s2', s3', and s4' of the four points are converted into spherical coordinates in step <NUM>. The second positions s1', s2', s3', and s4' of the four points may be converted into spherical coordinates by the method for converting cartesian coordinates into spherical coordinates as set forth above.

Four second position Φ values (Φs1', Φs2', Φs3', Φs4') may be obtained from the spherical coordinates of the second positions of the four points. A mean value is calculated for the four second position Φ values in step <NUM>. The mean value of the four second position Φ values may be obtained using Equations (<NUM>) to (<NUM>). <MAT><MAT> <MAT><MAT>.

Wi denotes the weight and Wi may be set to be inverse-proportional to the distance from the user's first position cartesian coordinates P.

By assuming that ϕavg corresponds to Φ2 of the spherical coordinates (<NUM>, Φ2, θ2) of the user's second position P' and obtaining θ2 using Equation (<NUM>), the spherical coordinates (<NUM>, Φ2, θ2) of the user's second position P' may be obtained.

A method for producing VR content is described below with reference to <FIG>, according to an embodiment of the present disclosure.

<FIG> schematically illustrate a method for producing VR content, according to an embodiment of the present disclosure. <FIG> illustrate a device for capturing images to produce visual content contained in VR content, according to an embodiment of the present disclosure. <FIG> illustrate examples of 2D projection methods and 2D projection images.

First, captured images may be stitched to a virtual spherical projection structure as shown in <FIG>. Stitching may mean arraying captured images to cover the surface of the projection structure. Images may be captured in various directions to project to the spherical projection structure as shown in <FIG>.

Referring to <FIG>, images may be captured by a tetrahedron camera having a camera in each surface as shown in <FIG>. Images may be captured by a cube camera having a camera in each surface as shown in <FIG>. Images may be captured by a dodecahedron camera having a camera in each surface as shown in <FIG>. However, other various devices and methods may also be adopted as long as they may capture images in various directions.

Referring back to <FIG>, when a plurality of captured multi-directional images are stitched onto the surface of the virtual spherical projection structure as shown in <FIG>, the coordinate axes of the virtual spherical projection structure may be rotated as shown in <FIG>. The rotation of the coordinate axes of the spherical projection structure may be carried out when the user's view point is moved. Methods described according to the embodiments in connection with <FIG> may be applied to rotate the coordinate axes of the virtual spherical projection structure.

After rotating the images-stitched virtual spherical projection structure, the images stitched on the virtual spherical projection structure are converted into two-dimensional (2D) projection images (or pictures) as shown in <FIG>.

A method for converting a 2D projection image is described with reference to <FIG>. Images stitched on the virtual spherical projection structure may be attached together on a tetrahedron 2D plane, producing a 2D projection image <NUM>. The images stitched on the virtual spherical projection structure may be captured by a tetrahedron camera.

Images stitched on the virtual spherical projection structure may be attached together on a 2D plane, producing a cube 2D projection image <NUM>. At this time, the images stitched on the virtual spherical projection structure may be captured by a cube camera. <FIG> illustrates an example of the cube 2D projection image <NUM>.

Images stitched on the virtual spherical projection structure may be attached together on a 2D plane, producing an octahedron 2D projection image <NUM>. The images stitched on the virtual spherical projection structure may be captured by an octahedron camera.

Images stitched on the virtual spherical projection structure may be attached together on a 2D plane, producing a dodecahedron 2D projection image <NUM>. The images stitched on the virtual spherical projection structure may be captured by a dodecahedron camera.

Images stitched on the virtual spherical projection structure may be attached together on a 2D plane, producing an icosahedron 2D projection image <NUM>, as shown in <FIG>. The images stitched on the virtual spherical projection structure may be captured by an icosahedron camera.

Any other method for producing a 2D projection image may also be adopted which may capture images in various directions and project the images to a 2D plane.

An example of a system for converting images stitched on the spherical position into a 2D projection image may be one as represented in Table <NUM>.

Referring back to <FIG>, the 2D projection image may be packed, producing a packed image (or picture). Packing is the method of dividing a 2D projection image into regions and remapping them. Packing may include transforming, resizing, and relocating. Packing may be performed to transmit VR content in a motion picture experts group (MPEG) media transport (MMT) method or MPEG dynamic adaptive streaming over hyper text transfer protocol (HTTP) (DASH) method.

Upon packing the 2D projection image, a weight may be assigned per location as shown in <FIG>, which may be related to one obtained by conducting packing with a weight assigned to a second region, but not to a first region and third region. The user's viewpoint may be positioned in the second region. The center of the 2D projection image or packed image may be subject to a smaller loss than the edges when the image is restored.

Various packing methods may be adopted. For example, the overall 2D projection image may be packed, or the area from the shifted user's viewpoint to the viewport or FOV may be packed. Further, other content information, such as auditory content, may be packed together.

The packed data may be VR content, and may be transmitted from the server to the VR device in an MMT method or DASH method.

Upon receiving the packed data, the VR device may perform the operations of producing the packed data in reverse order to render the VR content.

The VR device may unpack the packed data. An example of a system for unpacking the packed data may be one as represented in Table <NUM>, Table <NUM>, and Table <NUM>.

The VR device may receive, from the server, information for unpacking the packed data. An example of the information for unpacking may be shown in Table <NUM>.

The semantic shown in Table <NUM> may be the same as that shown in Table <NUM>, Table <NUM> or Table <NUM>.

The VR device may map the unpacked 2D projection image to the virtual spherical projection structure. An example of a system for mapping the 2D projection image to the virtual spherical projection structure may be one as represented in Table <NUM>.

The VR device may receive, from the server, information for mapping the 2D projection image to the projection structure. An example of the information about the 2D projection image may be the same as that shown in Table <NUM> to Table <NUM>.

The VR device may be obtained by mapping the 2D projection image to the virtual spherical projection structure.

The VR content need not necessarily be received from the server and may be stored in the memory of the VR device. In addition, the operations of 2D-projecting, packing, and transmitting the VR content may be skipped.

According to the embodiment described above, the captured images are stitched onto the spherical projection structure, then the coordinates are rotated, and the 2D projection image is produced, packed, and transmitted. However, the 2D projection image may be produced with the captured images, the positions of the first viewpoint and the second viewpoint to which it has been shifted may be specified on the 2D projection image, and the 2D projection image may be packed and transmitted.

A configuration of a VR device is described below with reference to <FIG>.

<FIG> schematically illustrates a configuration of a VR device, according to an embodiment of the present disclosure.

A VR device <NUM> may include a location sensor <NUM> measuring a movement of the VR device <NUM>, a memory <NUM> storing VR content, a processor <NUM> controlling all the operations of the VR device <NUM>, a display <NUM> including a display screen displaying VR content provided to a user, and a transceiver <NUM> transmitting and receiving signals to/from a server.

All the schemes or methods performed by the VR device <NUM> set forth herein may be appreciated as being performed under the control of the processor <NUM>.

The whole or part of the VR content set forth herein may permanently or temporarily be stored in the memory <NUM>. The whole or part of an algorithm for all the schemes or methods performed by the VR device <NUM> may temporarily or permanently be stored in the memory <NUM>.

The location sensor <NUM>, the memory <NUM>, the processor <NUM>, the display <NUM>, and the transceiver <NUM> need not necessarily be implemented as separate devices, but may be implemented in a single configuration unit in the form of a single chip. Further, each of the location sensor <NUM>, the memory <NUM>, the processor <NUM>, the display <NUM>, and the transceiver <NUM> need not necessarily be included and some may be omitted.

<FIG> schematically illustrates a configuration of a server, according to an embodiment of the present disclosure.

A server <NUM> may include a transceiver <NUM> transmitting and receiving signals to/from a VR device, a memory <NUM> storing VR content, and a processor <NUM> controlling all the operations of the server <NUM>. All the schemes or methods performed by the server <NUM> set forth herein may be appreciated as being performed under the control of the processor <NUM>.

The whole or part of the VR content set forth herein may permanently or temporarily be stored in the memory <NUM>. The whole or part of an algorithm for all the schemes or methods performed by the server set forth herein may temporarily or permanently be stored in the memory <NUM>.

However, the processor <NUM>, the memory <NUM>, and the transceiver <NUM> are not necessarily implemented as separate devices, respectively, but may be implemented in a single configuration unit in the form of a single chip.

All the configurations or operations illustrated in <FIG> may or may not be construed as essential components to practice the present disclosure, and the present disclosure may be implemented with only some of the components.

The above-described operations may be realized by equipping a memory device retaining corresponding program codes in any component of the server or VR device. That is, the processor in the server or VR device may execute the above-described operations by reading and running the program codes stored in the memory device by a processor or central processing unit (CPU).

Various components or modules in the server or VR device may be operated using a hardware circuit, a complementary metal oxide semiconductor-based logic circuit, firmware, software, and/or a hardware circuit such as a combination of hardware, firmware, and/or software embedded in a machine-readable medium. Various electric structures and methods may be executed using electric circuits such as transistors, logic gates, or application specific integrated circuits (ASICs).

Claim 1:
A method for processing virtual reality, VR, content by a head mounted display, HMD, device (<NUM>), comprising:
identifying (<NUM>, <NUM>) first cartesian coordinates of a first position on the VR content;
identifying (<NUM>) rotation information including a yaw rotation component indicating a rotation around a z axis, a pitch rotation component indicating a rotation around a y axis, and a roll rotation component indicating around an x axis;
identifying (<NUM>) second cartesian coordinates of a second position by multiplying a rotation matrix which is defined based on the rotation information to the first cartesian coordinates;
determining (<NUM>) whether the identified second cartesian coordinates is a polar value, wherein the polar value refers to coordinates on the y axis;
if the identified second cartesian coordinates is not the polar value, converting (<NUM>) the identified second cartesian coordinates of the second position into second spherical coordinates of the second position;
if the identified second cartesian coordinates is the polar value,
identifying (<NUM>) a plurality of points around the first position;
identifying (<NUM>) a plurality of rotated cartesian coordinates of the plurality of points, by multiplying the rotation matrix to cartesian coordinates of each of the plurality of points;
converting (<NUM>) the plurality of rotated cartesian coordinates to a plurality of spherical coordinates;
calculating (<NUM>) averaged spherical coordinates of the plurality of spherical coordinates, wherein the averaged spherical coordinates is set as the second spherical coordinates of the second position; and
rendering an area of the VR content based on the second spherical coordinates of the second position,
wherein the rotation matrix is defined by: <MAT>
wherein v is an angle corresponding to the roll rotation component, q is an angle corresponding to the pitch rotation component, and p is an angle corresponding to the yaw rotation component.