Patent ID: 12236919

Throughout the drawings, like reference numerals will be understood to refer to like parts, components, and structures.

DETAILED DESCRIPTION

The following description with reference to the accompanying drawings is provided to assist in a comprehensive understanding of various embodiments of the disclosure as defined by the claims and their equivalents. It includes various specific details to assist in that understanding but these are to be regarded as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the various embodiments described herein can be made without departing from the scope and spirit of the disclosure. In addition, descriptions of well-known functions and constructions may be omitted for clarity and conciseness.

The terms and words used in the following description and claims are not limited to the bibliographical meanings, but, are merely used by the inventor to enable a clear and consistent understanding of the disclosure. Accordingly, it should be apparent to those skilled in the art that the following description of various embodiments of the disclosure is provided for illustration purpose only and not for the purpose of limiting the disclosure as defined by the appended claims and their equivalents.

It is to be understood that the singular forms “a,” “an,” and “the” include plural referents unless the context clearly dictates otherwise. Thus, for example, reference to “a component surface” includes reference to one or more of such surfaces.

Throughout the specification, when a portion “includes” an element, another element may be further included, rather than excluding the existence of the other element, unless otherwise described. In addition, terms such as “ . . . unit”, “ . . . module”, and the like refer to units that perform at least one function or operation, and the units may be implemented as hardware or software or as a combination of hardware and software.

Hereinafter, the disclosure will now be described more fully with reference to the accompanying drawings, in which various embodiments of the disclosure are shown such that one of ordinary skill in the art may easily work the disclosure. This disclosure may, however, be embodied in many different forms and should not be construed as limited to the embodiments of the disclosure set forth herein. In addition, elements not related to description are omitted in the drawings for clear description of the disclosure, and like reference numerals in the drawings denote like elements throughout the specification.

Throughout the disclosure, the expression “at least one of a, b or c” indicates only a, only b, only c, both a and b, both a and c, both b and c, all of a, b, and c, or variations thereof.

The functions according to the disclosure may be performed using a processor and a memory of a head mounted device (HMD). The processor may include one or more processors. The one or more processors may include a general-purpose processor such as a central processing unit (CPU), an application processor (AP), a digital signal processor (DSP), and the like, a processor dedicated to graphics, such as a graphics processing unit (GPU), a vision processing unit (VPU), or a processor dedicated to artificial intelligence, such as a neural processing unit (NPU). The one or more processors control processing of input data according to previously defined operating rules or models stored in a memory. Alternatively, when the one or more processors include a processor dedicated to artificial intelligence, the processor dedicated to artificial intelligence may be designed in a hardware structure specialized for processing of a certain artificial intelligence model.

Head-mounted devices (or head-mounted displays, HMDs) project the full view of an image to a user, even though the user is not able to gaze at (view) the whole range at the same time. It is generally known that the visual field of the human eye is less than 120 degrees. Therefore, loss of data packets that exist in a portion of an image (including video) that a user is not gazing at does not affect the user's viewing experience.

At the current level of application protocols (e.g., Real-time Transport Protocol (RTP) or MPEG Media Transport Protocol (MMTP)), there is currently no method available to distinguish a viewing area that a user is currently viewing, from a non-viewing area. Accordingly, even when a user is not viewing a portion of an image, client devices, including HMDs, request retransmission of all lost data packets in the non-viewing area that the user is not viewing. This leads to an unnecessary increase in network transmission bandwidth requirements. In addition, due to the retransmission of unnecessary data packets, power consumption and delay between endpoints of a network may increase. Waste of resources and the increased load in a network and a client device may even impair the quality of an image that a user is currently viewing.

FIG.1is a diagram illustrating a viewing angle when viewing an image by using an HMD according to an embodiment of the disclosure.

Referring toFIG.1, when a user110of the HMD is looking at an image120projected through the HMD, a viewing angle of the user110is typically restricted to 120 degrees or less. When the user is standing, the user's line of sight is typically at an angle of only about 10 degrees with respect to the normal line of sight, and when the user is sitting, the line of sight of the user is at an angle of about 15 degrees with respect to the normal line of sight. A range of view within which a user is able to distinguish colors is about 60 degrees or less. A maximum viewing angle at which an image is recognizable is only 120 degrees.

Accordingly, when image content projected to the HMD is 360 degree-image content or image content having a viewing angle greater than 120 degrees, the user cannot view the whole image, but there is always an image area that the user cannot see.

FIG.2is a diagram illustrating a relationship between an image and a viewport when staring at the image by using an HMD according to an embodiment of the disclosure.

Referring toFIG.2, a viewport20representing an area that a user is looking at does not cover the entire 360-degree image21but an area corresponding to about 120 degrees.

In panel210, the user is looking at an area corresponding to about 20 degrees to about 140 degrees out of an area of the 360-degree image21from 0 degrees to 360 degrees. Thus, image areas corresponding to 0 degrees to 20 degrees and 140 degrees to 360 degrees in panel210are not viewed by the user.

In panel220, the user is looking at an area corresponding to about 120 degrees to about 240 degrees out of the area of the 360-degree image21from 0 degrees to 360 degrees. Thus, image areas corresponding to 0 degrees to 120 degrees and 240 degrees to 360 degrees in panel220are not viewed by the user.

In panel230, the user is looking at an area corresponding to about 220 degrees to about 340 degrees out of the area of the 360-degree image21from 0 degrees to 360 degrees. Thus, image areas corresponding to 0 degrees to 220 degrees and 340 degrees to 360 degrees in panel230are not viewed by the user.

When an area that the user is not looking at is determined according to the above examples, even when data packet loss occurs in that area, a client device does not have to request retransmission of lost data packets.

There are various methods to determine a position of the viewport20of the user in a panoramic image or a 360-degree image. According to an embodiment of the disclosure, the viewport20of the user may be determined using an eye-tracking camera tracking the line of sight of the user. The viewport20is typically rectangular. In an embodiment of the disclosure, a client device (e.g., an HMD) may track the line of sight of a user by using an eye-tracking camera, and when a focus on an image is determined, the client device may determine, based on the focus, the viewport20having a rectangular shape with a certain width and height.

While the viewport20is moving in a horizontal direction and a horizontal area that the user is not looking at within the 360-degree image21is determined, in the embodiment of the disclosure ofFIG.2, the disclosure is not limited thereto. That is, when a vertical length of an image that a user is looking at is long, the client device may determine a vertical length that is not covered by the viewport20. For example, when an image height is determined to be 0 to 100 as inFIG.2, and a vertical viewport of the user is 30, the remaining length of 70 does not overlap the viewport. Accordingly, the client device according to an embodiment of the disclosure needs not consider data packet loss regarding the area corresponding to the length of 70, which the user is not looking at.

FIG.3is a diagram illustrating a retransmission of data packets performed as packet loss occurs during a data packet transmission via a network according to an embodiment of the disclosure.

Referring toFIG.3, a sender310transmits a data packet to a receiver330via a server320. However, according to an embodiment of the disclosure, the sender310may also send a data packet directly to the receiver330without the data packet passing through the server320.

When a connection between the sender310and the receiver330is initially established, in operation 1, the sender310transmits a data packet through a communication protocol such as RTP. Each data packet may be assigned with a sequence number of 1, 2, 3, 4, or 5, and the data packets may be identified from each other based on the sequence number. As a communication protocol, without being limited to RTP, any type of communication protocols for retransmitting lost data packets may be used.

In operation 2, the receiver330may recognize that data packets identified by the sequence numbers 1, 3, and 5 are received from among the data packets but data packets identified by the sequence numbers 2 and 4 are lost.

Representative factors that cause data packet loss are Layer 2 (L2) error and network congestion. Data packet loss is detected using a reliable protocol such as the Transmission Control Protocol (TCP), and a network manager may use status information or a certain tool to detect such packet loss and diagnose the same.

In an embodiment of the disclosure, the above-described sequence number to identify lost packets may be used as a data packet identifier.

Referring back toFIG.3, in operation 3, the receiver330transmits, to the sender310via the server320, packet information indicating that loss of the data packets identified by the sequence numbers 2 and 4 among the data packets and a non-acknowledgement (NACK) message. In an embodiment of the disclosure, the packet information may include the sequence numbers 2 and 4 identifying the lost data packets and a request for retransmission of the data packets corresponding to the sequence numbers 2 and 4.

In operation 4, the sender310receives the NACK message and the packet information transmitted by the receiver330.

In operation 5, the sender310retransmits, to the receiver330via the server320, the data packets corresponding to the sequence numbers 2 and 4 identified through the packet information.

According to the TCP, whether byte-size data is properly received may be checked using a sequence number identifying the number of bytes received from an end point, which may typically be one of the sender310, the receiver330, a client, or the server320. Every time a payload byte is transmitted, the sequence number increases.

The TCP basically uses a cumulative acknowledgement scheme, and here, a receiver sends to a sender acknowledgement (ACK) indicating that all data before the acknowledged sequence number are received. The sender sets a sequence number field to a sequence number of a first payload byte in a data field of a segment, and the receiver sends acknowledgement designating a sequence number of a next byte to be received. For example, when the sender sends to the receiver a packet including four payload bytes with the sequence number of 150, the sequence numbers of the four payload bytes are 150, 151, 152, and 153. When this packet arrives at the receiver, the reception identification number of 154 is the sequence number of a next byte that is expected to be received in a next packet, and thus, the receiver sends the reception identification number of 154 to the sender.

Examples of the communication protocol for retransmitting lost data packets as shown inFIG.3are not limited to the TCP, and may also include RTP, MMTP or the like. However, neither of the TCP, the RTP, and the MMTP uses a method of selectively transmitting a data packet by determining an area that a user is not looking at.

FIG.4is a diagram illustrating connection between a sender and a receiver, the connection being established via a network according to an embodiment of the disclosure.

Referring toFIG.4, an HMD410, which is a client device and also a data packet receiver device, is connected to a sender device430transmitting a data packet, via a communication network420.

According to an embodiment of the disclosure, the sender device430includes an input/output (I/O) interface4301, a memory4303storing transmitted or received messages and data packets, and a processor4305controlling transmission and reception of data packets and input and output via the I/O interface4301.

Although not illustrated, the sender device430may further include a display displaying images. The communication network420may include a server (not shown) connecting the HMD410, which is a receiver device, to the sender device430.

FIG.5is a block diagram of an HMD as a receiver device according to an embodiment of the disclosure.

Referring toFIG.5, a block diagram of an HMD510as a receiver device receiving a data packet corresponding to image or video data, according to an embodiment of the disclosure, is illustrated.

The HMD510includes an I/O interface5101, a memory5103storing transmitted and received messages and data packets, and a processor5105controlling transmission and reception of data packets and input and output via the I/O interface5101. In addition, according to an embodiment of the disclosure, an eye-tracking camera5107to track a location which a user is looking at on an image may be included.

FIG.6Ais a diagram illustrating division of an image into a plurality of segments according to an embodiment of the disclosure.

Referring toFIG.6A, an image600may be divided into three segments610,620, and630, according to an embodiment of the disclosure. The three segments acquired through the division are referred to as a first segment610, a second segment620, and a third segment630, respectively. For convenience of description, a segment in the specification may also be alternatively referred to as an image segment, an image tile, or a tile. An image throughout the specification includes a static image and a moving image (video). The number of segments acquired through division and each segment identifier information are shared between a sender and a receiver before transmission and reception of data packets is initiated.

For a 360-degree image, a viewing angle of a user is about 120 degrees, and thus, it is appropriate to divide the 360-degree image into three segments. However, the disclosure is not limited thereto, and the number of segments to divide into may vary according to situations. That is, the image600may be divided into two, four, five or six segments according to the processing capability of a processor or other various conditions.

When a viewport of a user is on the first segment610, the user is not looking at the second segment620and the third segment630, and thus, despite data packet loss occurred on any portion of images corresponding to the second segment620and the third segment630, a client device (e.g., the HMD510) needs not request the server320or the sender310to retransmit the lost data packets.

On the other hand, when a viewport of a user is on the first segment610and data packet loss occurs in any portion of an image corresponding to the first segment610, this loss affects the image viewing experience of the user and the client device needs to request the server320or the sender310to retransmit the lost data packet.

According to an embodiment of the disclosure, when the processor5105of the HMD510identifies a segment that the user is looking at, among a plurality of segments on the image600, the processor5105first determines a viewport at which the user is looking, and determines a segment to which the determined viewport belongs, among the plurality of segments.

A method of determining a location of a viewport according to an embodiment of the disclosure is as follows.

The processor5105may determine the viewport based on a viewing angle of a user's gaze, a gaze direction of the user, and segment identifiers respectively allocated to the plurality of segments. However, the method of determining a location of a viewport, by the processor5105, is not limited thereto, and a location of a viewport on the image600may be determined using various methods.

According to an embodiment of the disclosure, a location of a viewport may be determined using a focus sensor of the user, installed in the HMD510. According to another embodiment of the disclosure, a location of a viewport may be determined using the eye-tracking camera5107of the user, installed in the HMD510. That is, the eye-tracking camera of the user may measure a gaze point and the HMD510may determine which part of the image the user is looking at. When the gaze point is determined, the HMD510may virtually form a viewport rectangle in consideration of upper and lower viewing angles with respect to the gaze point as a center point.

Various methods may be used to divide an image into a plurality of segment, which will now be described below.

FIG.6Bis a diagram illustrating division of an image into a plurality of segments according to another embodiment of the disclosure.

Referring toFIG.6B, similarly toFIG.6A, the image600is divided into three segments610A,610B;620A,620B and630A,630B in a horizontal direction with respect to a user's view, but also into two segments (e.g.,610A;610B) in a vertical direction. Thus, the number of segments acquired through the division according toFIG.6Bis six in total.

LikeFIG.6A, when a viewport of the user is within 0 degrees to about 120 degrees in the image600, the HMD510does not consider data packet loss with respect to the segments620A,620B,630A, and630B.

When the viewport of the user is limited only to the segment610B and the user is not looking at the segment610A, the HMD510does neither consider data packet loss occurring in an image corresponding to the segment610A. Before receiving a data packet, the HMD510shares in advance with a data packet sender identifiers of all segments acquired through the division.

FIG.6Cis a diagram illustrating an example of dividing a segment into sub-segments and selectively sending a retransmission request regarding packet loss according to an embodiment of the disclosure.

Referring toFIG.6C, a viewport20of a user is over the second segment620and the third segment630. In this case, as the user's line of sight is not detected from the first segment610, the HMD510will not request retransmission regarding data packet loss with respect to the first segment610.

However, as the viewport20of the user is over the second segment620and the third segment630, unlikeFIG.6A, resource consumption may increase because the HMD510has to request retransmission regarding data packet loss with respect to the two segments in the embodiment of the disclosure ofFIG.6C.

Accordingly, in this case, when a viewport, the location of which is determined on the image600, belongs to two or more segments, the processor5105of the HMD510divides the segments, to which the viewport belongs, one more time. That is, the second and third segments620and630, over which the viewport stretches, are further divided into a plurality of sub-segments6201,6202;6301,6302. The processor5105identifies the sub-segments6202and6301, to which the viewport20belongs, among the plurality of sub-segments. Then the processor5105shares, with a sender, the number of identified sub-segments and identifiers of the sub-segments. When determining loss in at least one data packet in a segment which a user is looking at, among a plurality of segments, the processor5105may determine whether there is loss in at least one data packet in a sub-segment identified as above and may request retransmission of the lost data packet in the sub-segment that the user is looking at, based on the determination. When requesting retransmission of the lost data packet according to an embodiment of the disclosure, an identifier of the lost data packet may be transmitted together.

WhileFIG.6Cillustrates an example of dividing a segment into sub-segments in a horizontal direction, an embodiment in which a segment is further divided into sub-segments in a vertical direction will be applied in the same manner as shown inFIG.6D.

In regard to the embodiment of the disclosure ofFIG.6C, as processing of dividing sub-segments according to a motion of a viewport may consume large resources of the processor5105, dividing an image into a large number of segments from the start may also be considered.

FIG.6Dis a diagram illustrating an example of dividing an image into a large number of segments according to an embodiment of the disclosure.

Referring toFIG.6D, the image600is divided into nine segments in a horizontal direction and into four segments in a vertical direction, thus into a total of 36 segments. According to an embodiment of the disclosure, the number of segments acquired through division and each of segment identifiers thereof may be shared with a sender in advance. The processor5105only has to determine a real-time location of the viewport20when the viewport20is moving and a segment which is obtained through the division and in which the viewport20is located, and to process only data packet loss in the segment where the viewport20is located.

According to an embodiment of the disclosure, during initial data transmission, the sender310and the receiver330may determine in advance the number of a plurality of segments and segment identifier/sub-segment identifier allocated to each segment, and share the number of segments and information of the segment identifier/sub-segment identifier allocated to each segment. All segments/sub-segments in the image600may be independently and separately decoded at an end of the receiver330.

FIG.7is a diagram illustrating a header structure when an RTP is applied according to an embodiment of the disclosure.

Referring toFIG.7, the RTP is used in transmitting a plurality of segments as described above. In an embodiment of the disclosure, a segment identifier associated with each segment may include a predefined number. In addition, a segment identifier may include viewing angle information of a user and thus constitute segment information. As the predefined number corresponding to the segment identifier, a payload type (PT), a synchronization source identifier (SSRC), and/or a data packet identifier may be used in a data packet. According to an embodiment of the disclosure, a receiver may transmit, to a sender, repeated segment identifiers—PT and S SRC—with respect to one segment to thereby perform cross-checking between two end points.

According to an embodiment of the disclosure, a sequence number included in a header may be used as a data packet identifier identifying a data packet.

A header according toFIG.7is merely an embodiment of the disclosure regarding application of the RTP, and also when other protocols are applied, a header may include both a segment identifier and an identifier of a lost data packet to be described with reference toFIG.9A.

FIG.8Ais a diagram illustrating an example of use of a payload type as a segment identifier according to an embodiment of the disclosure.

Referring toFIG.8A, when using the segment division according toFIG.6A, segment identifiers may be used as follows: PT (apt)=118 is used for the first segment610, PT (apt)=119 is used for the second segment620, and PT (apt)=121 is used for the third segment630. A PT is replaced by a payload number inFIG.8A.

FIG.8Bis a diagram illustrating an example of use of a synchronization source identifier (SSRC) as a segment identifier according to an embodiment of the disclosure.

Referring toFIG.8B, when using the segment division according toFIG.6A, segment identifiers may be used as follows: SSRC=1 (ssrc:1) is used for the first segment610, SSRC=2 (ssrc:2) is used for the second segment620, and SSRC=3 (ssrc:3) is used for the third segment630.

FIG.9Ais a diagram illustrating an example of a data packet loss in an image according to an embodiment of the disclosure.

The image600is divided into three segments according toFIG.6A.

Referring toFIG.9A, a data packet identifier of 7623 is allocated to the first segment610, a data packet identifier of 7624 is allocated to the second segment620, and a data packet identifier of 7625 is allocated to the third segment630. That is, according to an embodiment of the disclosure, a data packet identifier may be used as a segment identifier.

In the first segment610, loss of four data packets has occurred. The lost data packets may be identified based on the sequence number in the header according toFIG.7. That is, loss of data packets corresponding to sequence numbers of 10, 15, 100, and 50 occurred. However, the lost data packets corresponding to the sequence numbers of 10, 15, 100, and 50 are not where the viewport20of the user is located, and thus, the HMD510does not request the sender310to retransmit the lost data packets corresponding to the sequence numbers of 10, 15, 100, and 50.

Likewise, in the third segment630, loss of three data packets has occurred. However, as the third segment630is not where the viewport20is located, a request to retransmit lost data packets corresponding to sequence numbers of 70, 80, and 90 is not transmitted to the sender310.

On the other hand, in the second segment620, data packets corresponding to sequence numbers of 5 and 25 are lost. The second segment620is where the viewport20is located, and thus, the HMD510requests the sender310to retransmit the lost data packets. According to an embodiment of the disclosure, when requesting retransmission of a lost data packet, a sequence number which is an identifier of the lost data packet may be transmitted together. In an embodiment of the disclosure, when the HMD510requests retransmission of a lost data packet, the request may further include a segment identifier of a segment, to which the lost data packet belongs.

FIG.9Bis a diagram illustrating an example of a data packet loss in an image according to another embodiment of the disclosure.

Referring toFIG.9B, unlikeFIG.9A, the viewport20is located at a viewing angle between about 160 degrees and about 270 degrees.

UnlikeFIG.9A, inFIG.9B, as the viewport20is located both in the second segment620and the third segment630, a request for lost data packets is made with respect to lost data packets in the second segment620and the third segment630.

FIG.10is a sequence diagram of requesting retransmission of a lost data packet among a plurality of segments according to an embodiment of the disclosure.

The receiver330is a client device and may be the HMD510.

Referring toFIG.10, the receiver330divides the image600into a plurality of segments and determines and detects an identifier of each of the plurality of segments.

In operation1010, the receiver330and the sender310and/or the server320share the number of the plurality of segments and each of the identifiers of the segments with each other.

In operation1020, the sender310transmits a data packet corresponding to the image600to the receiver330.

In operation1030, the receiver330determines a viewport that a user is currently looking at and a segment where the viewport is located. A processor of the receiver330determines the viewport that the user is gazing at, and identifies a segment to which the determined viewport belongs, among the plurality of segments, to thereby determine a segment where the viewport is located. The viewport may be determined based on a viewing angle of a gaze of the user, a gaze direction of the user, and/or segment identifiers respectively allocated to the plurality of segments.

In operation1040, the receiver330determines whether there is a lost data packet in the segment that the user is looking at, that is, in the segment where the viewport is located.

In operation1050, the receiver330requests the sender310and/or the server320to retransmit the data packet loss in the segment that the user is looking at, that is, the segment where the viewport is located. When requesting to retransmit the lost data packet according to an embodiment of the disclosure, the receiver330may also transmit an identifier of the lost data packet and/or an identifier of the segment to which the lost data packet belongs.

In operation1060, the receiver330receives, from the sender310, the lost data packet, for which retransmission is requested. In an embodiment of the disclosure, the receiver330may periodically receive the entire image that the user is looking at.

Table 1 shows comparison between a conventional case where retransmission of lost data packets is performed with respect to all areas of an image and a proposed case where retransmission of lost data packets is not performed for a non-viewport region.

TABLE 1OriginalNumber of LostNumber ofNumberPacketsLostBandwidth =of PacketsPacketrequested forPackets(RetransmittedTransmitted/Loss/retransmission/retransmitted/packets *BandwidthMethodBitrateSecSecSecSecMTU * 8)/1000SavedConventional10 Mbps8331%8896Kbps3%2424288Kbps5%4040480Kbps10%8080960KbpsProposed10 Mbps8331%3336Kbps60Kbps3%8896Kbps192Kbps5%1313156Kbps324Kbps10%2727324Kbps636Kbps

According to Table 1, by applying the method according to the disclosure, 60 kbps bandwidth may be saved when a packet loss rate from among the total data packets transmitted in an HMD is 1%, 192 kpbs bandwidth may be saved at a packet loss rate of 3%, 324 kpbs bandwidth may be saved at a packet loss rate of 5%, and 636 kbps bandwidth may be saved at a packet loss rate of 10%.

Embodiments of the disclosure may be implemented through at least one software program that is executed on at least one hardware device and performs a network management function to control elements of the hardware device.

The methods according to the embodiments of the disclosure may be embodied as program instructions executable by various computer means and may be recorded on a computer-readable recording medium. The computer readable recording medium may include program instructions, a data file, a data structure etc. alone or in combination. The program instructions written to the computer readable recording medium may be specifically designed and configured for the embodiments of the disclosure or may be well-known and available to one of ordinary skill in the art. Examples of the computer readable recording medium include non-transitory media such as magnetic media (e.g., hard disks, floppy disks, magnetic tapes, etc.), optical media (e.g., CD-ROMs, or DVDs), magneto-optical media (e.g., floptical disks), and hardware devices specifically configured to store and execute program instructions (e.g., ROM, RAM, flash memories, etc.). Examples of the program instructions include advanced language codes that can be executed by a computer by using an interpreter or the like as well as machine language codes made by a compiler.

According to the image data packet managing method and the image display apparatus of to the disclosure, network congestion may be reduced and communication and processing resources for data management in an HMD may be reduced by managing loss of only data packets in a viewing angle of a user's gaze.

While the disclosure has been shown and described with reference to various embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the disclosure as defined by the appended claims and their equivalents.