Patent ID: 12198279

DETAILED DESCRIPTION

Terms used herein should not be construed as being limited to their usual or dictionary meanings. In view of the fact that the inventor can appropriately define the meanings of terms in order to describe his/her own invention in the best way, the terms should be interpreted as meanings consistent with the technical idea of the present disclosure. In addition, the following description and corresponding drawings merely relate to specific embodiments of the present disclosure and do not represent all the subject matter of the present disclosure. Therefore, it will be understood that there are various equivalents and modifications of the disclosed embodiments at the time of the present application.

Now, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In the drawings, like elements are denoted by the same reference numerals. In addition, detailed descriptions of well-known functions and elements that may obscure the subject matter of the present disclosure will be omitted. For the same reason, some elements are exaggerated, omitted, or schematically illustrated in the drawings, and the size of each element does not fully reflect the actual size.

First, a system for providing a virtual reality (VR) image according to an embodiment of the present disclosure will be described.FIG.1is a diagram illustrating the configuration of a system for providing a virtual reality image according to an embodiment of the present disclosure.

Referring toFIG.1, a system for providing a virtual reality image according to an embodiment of the present disclosure includes an edge server10and a virtual reality device20. The edge server10and the virtual reality device20are connected through wireless communication.

The edge server10may be an edge cloud server located closest to the virtual reality device20or a high-performance PC connected to the virtual reality device20by Wi-Fi or the like.

The virtual reality device20may be any device capable of reproducing virtual reality. Representatively, the virtual reality device20may be a head mounted display (HMD).

According to the present disclosure, the edge server10is capable of rendering virtual reality images and providing them to the virtual reality device20through real-time streaming, and the virtual reality device20is capable of reproducing the virtual reality images received through real-time streaming. Therefore, it is possible to provide high-quality virtual reality images regardless of the performance of the virtual reality device20.

Next, the edge server10for providing a virtual reality image according to an embodiment of the present disclosure will be described.FIG.2is a diagram illustrating the configuration of an edge server for providing a virtual reality image according to an embodiment of the present disclosure. Referring toFIG.2, the edge server10according to an embodiment of the present disclosure includes a communication module11, a storage module12, and a control module13.

The communication module11is for communicating with the virtual reality device20through a network. The communication module11may transmit/receive data to/from the virtual reality device20. The communication module11may include a radio frequency (RF) transmitter (Tx) for up-converting the frequency of a signal to be transmitted and amplifying the signal, and an RF receiver (Rx) for low-noise amplifying a received signal and down-converting the frequency of the signal. Also, in order to transmit/receive data, the communication module11may include a modem for modulating a signal to be transmitted and demodulating a received signal. The communication module11may transmit data received from the control module13to the virtual reality device20. Also, the communication module11may deliver data received from the virtual reality device20to the control module13.

The storage module12stores programs and data necessary for the operation of the edge server10. For example, the storage module12may store virtual reality image contents. In addition, the storage module12may store the visual field and margin of the virtual reality device20. A scale factor of each of a plurality of layers of the margin may also be stored. Various data stored in the storage module121may be registered, deleted, changed, or added according to a manipulation of an administrator of the edge server10.

The control module13may control the overall operation of the edge server10and a signal flow between internal blocks of the edge server10, and may perform a data processing function. The control module13may be a central processing unit, a digital signal processor, or the like. In addition, the control module13may further include an image processor or a graphic processing unit (GPU). The control module13includes a rendering synchronization unit (or a rendering synchronization processor)110, a rendering unit (or a rendering processor)120, an encoding unit (or an encoder)130, and a streaming transmission unit (or a streaming transmitter)140. The operation of the control module13will be described in more detail below.

Next, the virtual reality device20for providing a virtual reality image according to an embodiment of the present disclosure will be described.FIG.3is a diagram illustrating the configuration of a virtual reality device for providing a virtual reality image according to an embodiment of the present disclosure. Referring toFIG.3, the virtual reality device20according to an embodiment of the present disclosure includes a communication unit21, a sensor unit22, an audio unit23, an input unit24, a display unit25, and a storage unit26, and a control unit27.

The communication unit21is for communication with the edge server10. The communication unit21may include a radio frequency (RF) transmitter (Tx) for up-converting the frequency of a signal to be transmitted and amplifying the signal, and an RF receiver (Rx) for low-noise amplifying a received signal and down-converting the frequency of the signal. In addition, the communication unit21may include a modem that modulates a signal to be transmitted and demodulates a received signal.

The sensor unit22is for measuring inertia. The sensor unit12includes an inertial measurement unit (IMU), a Doppler velocity log (DVL), an attitude and heading reference system (AHRS), and the like. The sensor unit22measures inertial information including the position and speed of rotation and movement of the virtual reality device20and provides the measured inertial information of the virtual reality device20to the control unit27.

The audio unit23includes a speaker (SPK) for outputting an audio signal, and a microphone (MIKE) for receiving an audio signal. The audio unit23may output an audio signal through the speaker under the control of the control unit27, or deliver an audio signal inputted through the microphone to the control unit27. In particular, the audio unit23outputs an audio signal of a virtual reality image.

The input unit24receives a user's key manipulation for controlling the virtual reality device20, generates an input signal, and delivers the generated input signal to the control unit27. The input unit24may include various keys for controlling the virtual reality device20. Some of the functions of the input unit24may be formed in a touch screen.

The display unit25visually provides a menu of the virtual reality device20, input data, function setting information, and various other kinds of information to a user. The display unit25performs a function of outputting a booting screen, an idle screen, a menu screen, and the like of the virtual reality device20. In particular, the display unit25performs a function of outputting a virtual reality image according to an embodiment of the present disclosure to the screen. The display unit25may be formed of a liquid crystal display (LCD), an organic light emitting diode (OLED), an active matrix OLED (AMOLED), or the like.

The storage unit26stores programs and data necessary for the operation of the virtual reality device20. In particular, the storage unit26includes a decoding buffer (DB) for temporarily storing a virtual reality image, a split virtual reality image, and the like, and a rendering buffer (RB) for temporarily storing a reproduction area. Also, the storage unit26may store various parameters such as a visual field, a margin, and a scaling factor of a margin area. Various data stored in the storage unit26may be deleted, changed, or added according to a manipulation of a user of the virtual reality device20.

The control unit27may control the overall operation of the virtual reality device20and a signal flow between internal blocks of the virtual reality device20, and perform a data processing function. Also, the control unit27basically controls various functions of the virtual reality device20. The control unit27may include a central processing unit (CPU), a baseband processor (BP), an application processor (AP), a graphic processing unit (GPU), a digital signal processor (DSP), or the like. The operation of the control unit27will be described in more detail below.

Next, a method for providing a split-rendered virtual reality image according to an embodiment of the present disclosure will be described.FIG.4is a flowchart illustrating a method for providing a split-rendered virtual reality image according to an embodiment of the present disclosure.FIGS.5to12are screen examples illustrating a method for providing a split-rendered virtual reality image according to an embodiment of the present disclosure.

Referring toFIG.4, at step S110, the reproduction synchronization unit210of the virtual reality device20derives information related to the movement of the virtual reality device20through the sensor unit22, representatively, inertial information including the rotation center (RC) of the virtual reality device20and the position and speed of the rotation center (RC) being moved, and transmits the derived inertial information to the edge server10through the communication unit21. Providing the inertia information at then step S110is continuously made. Accordingly, at step S120, the edge server10and the virtual reality device20may synchronize a visual field (VF) and a margin (M). To this end, the rendering synchronization unit110of the edge server10derives the visual field (VF) and the margin (M) of the virtual reality device20based on the inertia information. Here, the visual field (VF) of the virtual reality device20or the user is derived from the inertia information. The margin (M) is derived in consideration of the visual field (VF) of the user and the user's viewpoint moving speed derived from the inertial information. Meanwhile, the margin (M) may be changed according to a user's behavior pattern or network environment derived from the inertia information continuously received. Subsequently, the rendering synchronization unit110transmits the derived visual field (VF) and margin (M) to the virtual reality device20through the communication module11. Then, the reproduction synchronization unit210of the virtual reality device20receives the visual field (VF) and the margin (M) through the communication unit21, and stores the received visual field (VF) and margin (M) in the storage unit26. As such, the edge server10and the virtual reality device20can synchronize the visual field (VF) and the margin (M).

Then, at step S130, the rendering unit120of the edge server10generates a rendered image as shown inFIG.5by rendering a visual field area (VA) corresponding to the visual field (VF) and a margin area (MA) corresponding to the margin (M) based on the rotation center (RC) in the entire virtual reality image (EQ), that is, in the 360-degree equirectangular image. As shown inFIG.6, to provide a stereoscopic image, the rendered image includes a left image (LI) and a right image (RI), each of which includes the visual field area (VA) and the margin area (MA).

As shown inFIG.7, the visual field area (VA) derived from the visual field (VF) in the entire virtual reality image (EQ) has a size of Vx×Vy. An area extending outward (up, down, left and right) by the margins (M) Tx and Ty from the edges of the visual field area (VA) is the margin area (MA).

Next, at step S140, the encoding unit130generates a reduced margin area (RMA) by dividing the resolution of the margin area (MA) by a scaling factor, and then encodes the visual field area (VA) and the reduced margin area (RMA) to generate a split virtual reality image including the encoded visual field area (VA) and the encoded reduced margin area (RMA). For example, as shown inFIG.8, the reduced margin area (RMA) may be generated by dividing the resolution of the margin area (MA) by a scale factor n (Tx/n, Ty/n). Accordingly, the image quality is reduced, so that the size of the split virtual reality image can be reduced.

According to another embodiment, when encoding the rendered image, the encoding unit130may apply a gradation to the margin area. A detailed description is as follows. As described above, according to an embodiment, the margin area (MA) is an area extending outward from the edges of the visual field area (VA). According to another embodiment, the margin area (MA) may be divided into a plurality of sections. The scale factor of each of the plurality of sections has a relatively large value as the distance from the visual field area (VA) increases. Therefore, each of the plurality of sections within the margin area (MA) may be reduced using the scale factor having a gradation to generate the reduced margin area (RMA). For example, as shown inFIG.9, the margin area (MA) may be divided into a first section (ma1), a second section (ma2), and a third section (ma3). Also, the scale factors n1, n2, and n3respectively corresponding to the first section (ma1), the second section (ma2), and the third section (ma3) may be set to have relatively large values (n1<n2<n3) as they are away from the visual field area (VA). Therefore, the reduced margin area (RMA) may be generated such as (Tx1/n1, Ty1/n1), (Tx2/n2, Ty2/n2), and (Tx3/n3, Ty3/n3). Through this, it is possible to generate a virtual reality image having a relatively wider area from the same resource. Also, if necessary, it is possible to generate a virtual reality image with a relatively smaller size. In particular, the embodiment with reference toFIG.9can provide an image with no significant degradation in image quality when the user of the virtual reality device20moves relatively small.

Next, at step S150, the streaming transmission unit140muxes a common media application format (CMAF) live profile to the previously encoded split virtual reality image, and transmits the split virtual reality image including the CMAF live profile to the virtual reality device20. At this time, after split-encoding the split virtual reality image in units of chunks according to the CMAF, the streaming transmission unit140may transmit the split virtual reality image in units of chunks.

Upon receiving the split virtual reality image including the CMAF live profile, the streaming reception unit220of the virtual reality device20derives the split virtual reality image by demuxing (or demultiplexing) the CMAF live profile from the received split virtual reality image.

Then, at step S180, the decoding unit230decodes the reduced margin area (RMA) according to the visual field area (VA) corresponding to the visual field (VF) and the scaling factor based on the rotation center (RC), and inversely enlarges the reduced margin area (RMA) to derive the split virtual reality image including the visual field area (VA) and the margin area (MA). Then, the derived split virtual reality image including the visual field area (VA) and the margin area (MA) is stored in the decoding buffer (DB) of the storage unit26.

For example, as shown inFIG.10, when the width and height of the visual field area (VA) are Vx and Vy, and when the width and height of the margin area (MA) are 2Tx and 2Ty, the decoding unit230decodes the visual field area (VA) corresponding to the visual field (VF) and the reduced margin area (RMA), like “Tx/n, Ty/n”, reduced according to the scaling factor n, and inversely enlarges the reduced margin area (RMA) reduced according to the scaling factor to derive the split virtual reality image including the visual field area (VA) and the margin area (MA). Then, the decoding buffer (DB) stores the split virtual reality image which is set to a size of (Vx+2Tx)×(Vy+2Ty) in consideration of both the visual field area (VA) and the margin area (MA).

Similarly, in the case that the reduced margin area (RMA) is generated according to another embodiment of the present disclosure by reducing each of the plurality of sections within the margin area (MA) by using a scale factor having a gradation, it is possible to derive the split virtual reality image by inversely enlarging the reduced margin area according to the scaling factor.

Next, at step S190, the reproduction unit240reproduces the split virtual reality image. Referring toFIGS.11A-11C, the reproduction unit240derives the user's rotation center (RC) at the reproduction time point through the sensor unit22. Then, from the split virtual reality image stored in the decoding buffer (DB), for example, as shown inFIG.11A, the reproduction unit240derives a reproduction area through a view window (VW) corresponding to the visual field (VF) based on the user's rotation center (RC) at the reproduction time point, for example, as shown inFIG.11B. Such a reproduction area may be stored in the rendering buffer (RF). Subsequently, as shown inFIG.11C, the reproduction unit240renders and reproduces the derived reproduction area.

In the case that the user's field of vision moves before the next frame of the split virtual reality image is received, the view window (VW) is moved to render the corresponding area. If the view window (VW) deviates from the visual field area (VA) due to the user's movement as shown inFIG.12A, a portion of the reproduction area may have a relatively low resolution because it is the margin area (MA) as shown inFIG.12B. Therefore, during reproduction as shown inFIG.12C, the user enjoys the reproduction area containing the low resolution portion until the split virtual reality image of the next frame is received and decoded.

The method according to embodiments of the present disclosure may be provided in the form of a non-transitory computer-readable recording medium suitable for storing computer program instructions and data. The computer-readable recording medium may include program instructions, data files, data structures, etc. alone or in combination, and includes all kinds of recording devices in which data that can be read by a computer system is stored. The computer-readable recording medium includes a hardware device specially configured to store and execute program instructions, including magnetic media such as a hard disk, a floppy disk, and a magnetic tape, optical media such as a compact disc read only memory (CD-ROM) and a digital versatile disc (DVD), magneto-optical media such as a floptical disk, and semiconductor memories such as a read only memory (ROM), a random access memory (RAM), and a flash memory. Further, the computer-readable recording medium may be distributed over networked computer systems so that computer-readable code can be stored and executed in a distributed fashion. In addition, functional programs, associated codes, and code segments for implementing the present disclosure may be easily deduced or altered by programmers in the art to which the present disclosure belongs.

Families and caregivers complain of difficulties for constant care of mentally ill people with severe emotional ups and downs or the elderly living alone. However, even in a home environment, the present disclosure can support continuous care through sympathetic dialogue with people with mental disorders, such as panic disorder, depression, and schizophrenia, who require continuous observation and monitoring. Furthermore, according to the present disclosure, a guardian can be notified of an emergency situation of a patient even from a distance, and it is possible to monitor and deliver the patient's condition outside the hospital to medical staff 24 hours a day, so that it is possible to accurately identify the patient's condition.

Although embodiments of the present disclosure are described above, these embodiments are exemplary only and not construed as a limitation. Various changes and modifications to the present disclosure and their equivalents can be made as well understood by those skilled in the art without departing from the technical subject matter of the present disclosure and the scope of appended claims.