Patent ID: 12219203

DETAILED DESCRIPTION OF THE INVENTION

Hereinafter, example embodiments are described with reference to the accompanying drawings.

A live video playback system according to the example embodiments may be implemented by at least one computer apparatus. A live video playback method according to the example embodiments may be performed through the at least one computer apparatus included in the live video playback system. Here, a computer program according to an example embodiment may be installed and run on the computer apparatus and the computer apparatus may perform the live video playback method according to the example embodiments under the control of the running computer program. The aforementioned computer program may be stored in a computer-readable recording medium to computer-implement the live video playback method in conjunction with the computer apparatus.

FIG.1illustrates an example of a network environment according to an example embodiment. Referring toFIG.1, the network environment may include a plurality of electronic devices110,120,130, and140, a plurality of servers150and160, and a network170.FIG.1is provided as an example only. The number of electronic devices or the number of servers is not limited thereto. Also, the environment applicable to the example embodiments is not limited to the network environment ofFIG.1.

Each of the plurality of electronic devices110,120,130, and140may be a fixed terminal or a mobile terminal that is implemented as a computer apparatus. For example, the plurality of electronic devices110,120,130, and140may be a smartphone, a mobile phone, a navigation device, a computer, a laptop computer, a digital broadcasting terminal, a personal digital assistant (PDA), a portable multimedia player (PMP), a tablet PC, and the like. For example, althoughFIG.1illustrates a shape of a smartphone as an example of the electronic device110, the electronic device110used herein may refer to one of various types of physical computer apparatuses capable of communicating with other electronic devices120,130, and140, and/or the servers150and160over the network170in a wireless or wired communication manner.

The communication scheme is not limited and may include a near field wireless communication scheme between devices as well as a communication scheme using a communication network (e.g., a mobile communication network, wired Internet, wireless Internet, a broadcasting network, etc.) includable in the network170. For example, the network170may include at least one of network topologies that include a personal area network (PAN), a local area network (LAN), a campus area network (CAN), a metropolitan area network (MAN), a wide area network (WAN), a broadband network (BBN), and the Internet. Also, the network170may include at least one of network topologies that include a bus network, a star network, a ring network, a mesh network, a star-bus network, a tree or hierarchical network, and the like. However, they are provided as examples only.

Each of the servers150and160may be configured as a computer apparatus or a plurality of computer apparatuses that provides an instruction, a code, a file, content, a service, etc., through communication with the plurality of electronic devices110,120,130, and140over the network170. For example, the server150may be a system that provides a service (e.g., a live video service, a content providing service, a group call service (or an audio conference service), a messaging service, a mail service, a social network service, a map service, a translation service, a financial service, a payment service, a search service, etc.) to the plurality of electronic devices110,120,130, and140connected over the network170.

FIG.2is a block diagram illustrating an example of a computer apparatus according to an example embodiment. Each of the plurality of electronic devices110,120,130, and140or each of the servers150and160may be implemented in a computer apparatus200ofFIG.2.

Referring toFIG.2, the computer apparatus200may include a memory210, a processor220, a communication interface230, and an input/output (I/O) interface240. The memory210may include a permanent mass storage device, such as a random access memory (RAM), a read only memory (ROM), and a disk drive, as a computer-readable record medium. The permanent mass storage device, such as ROM and disk drive, may be included in the computer apparatus200as a permanent storage device separate from the memory210. Also, an operating system (OS) and at least one program code may be stored in the memory210. Such software components may be loaded to the memory210from another computer-readable recording medium separate from the memory210. The other computer-readable recording medium may include a computer-readable recording medium, for example, a floppy drive, a disk, a tape, a DVD/CD-ROM drive, a memory card, etc. According to other example embodiments, software components may be loaded to the memory210through the communication interface230, instead of the computer-readable recording medium. For example, the software components may be loaded to the memory210based on a computer program installed by files provided over the network170.

The processor220may be configured to process instructions of a computer program by performing basic arithmetic operations, logic operations, and I/O operations. The instructions may be provided from the memory210or the communication interface230to the processor220. For example, the processor220may be configured to execute received instructions in response to the program code stored in a storage device, such as the memory210.

The communication interface230may provide a function for communication between the computer apparatus200and other apparatuses (e.g., the aforementioned storage devices) over the network170. For example, the processor220of the computer apparatus200may transfer a request or an instruction created based on a program code stored in the storage device such as the memory210, data, a file, etc., to the other apparatuses over the network170under the control of the communication interface230. Inversely, a signal, an instruction, data, a file, etc., from another apparatus may be received at the computer apparatus200through the communication interface230of the computer apparatus200. For example, a signal, an instruction, data, etc., received through the communication interface230may be transferred to the processor220or the memory210, and a file, etc., may be stored in a storage medium (e.g., the aforementioned permanent storage device) further includable in the computer apparatus200.

The I/O interface240may be a device used for interfacing with an I/O apparatus250. For example, an input device of the I/O apparatus250may include a device, such as a microphone, a keyboard, a mouse, etc., and an output device of the I/O apparatus250may include a device, such as a display, a speaker, etc. As another example, the I/O interface240may be a device for interfacing with an apparatus in which an input function and an output function are integrated into a single function, such as a touchscreen. The I/O apparatus250may be configured as a single apparatus with the computer apparatus200.

Also, according to other example embodiments, the number of components included in computer apparatus200may be less than or greater than the components ofFIG.2. For example, the computer apparatus200may be configured to include at least a portion of the I/O apparatus250or may further include other components, for example, a transceiver and a database.

FIG.3illustrates an example of a process of providing a live video service according to an example embodiment.FIG.3illustrates each of a streaming server310, an encoder server320, and a player330. Here, the streaming server310and the encoder server320may be software modules installed and running together on the server150or the server160, or separately on the server150and the server160to provide a live video service. Also, the player330may be software modules installed and running on any of the terminal devices110,120,130,140of a user to play back a video.

Referring toFIG.4, the streaming server310refers to a server device that performs a transmission functionality and may include, for example, an open broadcaster software (OBS) module for recording and real-time streaming. The streaming server310may receive each of videos simultaneously captured through a plurality of cameras through each corresponding channel and may perform real-time streaming to the encoder server320as a multi-channel image. As used herein, the term “image” refers to moving visual images and is used interchangeably with “video.”

Regardless of a camera, an audio channel may use the same input. The encoder server320may use an audio of a channel corresponding to a basic channel among a plurality of channels as a basic stream.

The streaming server310may transmit a multi-channel image using an OBS module for synchronization processing between video channels. Therefore, the encoder server320may process synchronization between the video channels by calculating an absolute time of each video frame using a coordinated universal time (UTC) and a predetermined time standard (PTS).

In particular, the encoder server320may provide at least one composite image in which videos of multiple channels are synchronized into a single video, such that the player330may compose a multi-view mode using a minimum number of networks in a multi-stream live transmission environment.

For example, referring toFIG.5, the encoder server320may create a plurality of composite images502using original images (Cam #1 to Cam #6)501captured through six cameras. The composite image502refers to an image in which the original images501are combined into a single image. The encoder server320may create the composite image502by combining the original images501according to a predetermined layout.

For example, the encoder server320may create a number of composite images502corresponding to a number of video channels, and may compose a video of a single channel as a main image and videos of remaining other channels as sub-images. That is, the encoder server320may compose the video (main image) of the single channel with a higher resolution than those of videos (sub-images) of remaining other channels. Here, the encoder server320may compose the videos (sub-images) of the remaining other channels, excluding the video (main image) of the single channel, with the same resolution or may compose the same with a higher resolution according to the priority of each channel.

The scheme of creating the composite image502, the number of composite images502, the layout structure, and the like are provided as an example only and may be readily modified.

The encoder server320may transmit videos of multiple channels as a single stream using the composite image502instead of transmitting each of the plurality of videos coming into the multiple channels as a single stream in the multi-stream live transmission environment.

In the following example embodiment, a composite image in which a video of channel 1 corresponding to a camera (Cam #1) among the composite images502is a main image is referred to as a ‘Channel #1 composite image’ and is also referred to as a ‘Channel #2 composite image,’ a ‘Channel #3 composite image,’ and the like according to each channel of a video composed as a main image.

Unique information capable of identifying a corresponding image may be included in a stream of each composite image502. When a playback image is changed with the Channel #2 composite image during playback of the Channel #1 composite image through unique image information of each stream, the player330may recognize that the changed image is the Channel #2 composite image.

FIG.6is a flowchart illustrating an example of a live video playback method according to an example embodiment. The live video playback method according to the example embodiment may be performed by the computer apparatus200that implements the aforementioned player330. If the player330is implemented on a terminal device of a user, the computer apparatus200may correspond to the terminal devices110,120,130,140. In this case, the processor220of the computer apparatus200may be configured to execute a control instruction according to a code of at least one program or a code of an OS included in the memory210. Here, the processor220may control the computer apparatus200to perform operations S610to S640included in the method ofFIG.6according to a control instruction provided from a code stored in the computer apparatus200.

In operation S610, the computer apparatus200may receive, from the encoder server320implemented in the server150or160, a single composite image that includes a plurality of images from multiple channels in a multi-stream live transmission environment. The computer apparatus200may download a single stream from the encoder server320and may receive an image stream in which live images of multiple channels are synthesized in a single image. The computer apparatus200may selectively receive a composite image corresponding to layout information determined in the player330by referring to layout information of a composite image for each channel (Channel #1 composite image, Channel #2 composite image, etc.). For example, when the player330requests a layout in which a video of channel 1 is provided as a main image, the computer apparatus200may receive the Channel #1 composite image from the encoder server320.

In operation S620, the computer apparatus200may obtain a sprite for each channel by separating a composition image for each channel from the composite image received as a single stream. That is, the computer apparatus200may obtain a sprite for each channel by separating the composite image through post processing. The computer apparatus200may separate the composite image using various graphics engines, for example, an open graphics library (open GL).

In operation S630, the computer apparatus200may render the sprite for each channel according to a layout of the player330. That is, the computer apparatus200may play back live images of multiple channels received as a single stream by rendering a sprite for each camera separated from the composite image to fit a desired layout.

In operation S640, when a layout switching request is received through the player330from the user, the computer apparatus200may receive a composite image corresponding to the layout switching request among composite images providable from the encoder server320. When there is a need to change a layout according to a user input, such as changing a main image or changing to a full screen mode (hereinafter, a full-view mode) for a live image currently being played back on the player330, the computer apparatus200may change the composite image received from the encoder server320to a composite image of the desired layout.

The computer apparatus200may switch between channels, that is, may switch a layout of a live image being played back on the player330by repeating the aforementioned process (S620and S630) in response to a change of the composite image received from the encoder server320.

FIG.7illustrates an example of a basic process of composing a multi-view mode as a single stream according to an example embodiment.

FIG.7illustrates a process of composing a multi-view mode using a Channel #1 composite image in the player330. In the Channel #1 composite image, a video (Cam #1) of channel 1 is composed as a main image having a higher resolution than those of videos (Cam #2 to Cam #6) of other channels.

Referring toFIG.7, the player330may compose a multi-view mode by receiving a Channel #1 composite image frame, by obtaining a sprite for each channel from the Channel #1 composite image frame through post processing, and then rendering the separated sprite for each channel according to a predetermined layout.

Therefore, the player330may compose a multi-view mode including live images of multiple channels by downloading only a single stream in a multi-stream live transmission environment.

FIG.8illustrates an example of an interface for changing a player layout according to an example embodiment.

The player330provides an interface for requesting a change in a display position between videos (Cam #1 to Cam #6) of the respective channels in a multi-view mode or switching to a full-view mode for an image of a specific channel.

In a situation in which live images of multiple channels are being played back in a multi-view mode form on the player330, the user may input a layout switching request for changing a display position between videos (Cam #1 to Cam #6) in the multi-view mode or switching one of the videos (Cam #1 to Cam #6) to the full-view mode.

For example, referring toFIG.8, it is possible to change a display position between the video (Cam #1) of channel 1 and the video (Cam #2) of channel 2 by moving the video (Cam #2) of channel 2 to the position of a main image in which the video (Cam #1) of channel 1 is currently being played back through a drag80on a multi-view screen using a Channel #1 composite image. Alternatively, it is possible to request switching to a full-view mode through a predetermined menu or another gesture by selecting the video (Cam #2) of channel 2 from among the videos (Cam #1 to Cam #6) in the multi-view mode.

Also, it is possible to request switching from a full-view mode to a multi-view mode on the player330.

FIGS.9and10illustrate examples of a process of for switching layouts between multi-view modes according to an example embodiment.

FIGS.9and10illustrate examples of a process of switching from a multi-view mode using a Channel #1 composite image to a multi-view mode using a Channel #2 composite image.

When the user requests moving a video (Cam #2) of channel 2 to a main image position on a multi-view screen using the Channel #1 composite image, the player330may render a sprite for each channel separated from the Channel #1 composite image to fit a layout corresponding to the request as illustrated inFIG.9, for seamless screen switching. The video (Cam #1) of channel 1 separated from the Channel #1 composite image may be downscaled to fit a layout changed into a high-quality image and the video (Cam #2) of channel 2 separated from the Channel #1 composite image may be upscaled in a low-quality state as a low-quality image and displayed at the main image position.

Next, referring toFIG.10, to provide a high-quality image for the video (Cam #2) of channel 2, the player330may receive, from the encoder server320, the Channel #2 composite image in which the video (Cam #2) of channel 2 is composed as a main image having a higher resolution than those of videos (Cam #1 and Cam #3 to Cam #6) of other channels. The player330may compose a multi-view mode using the Channel #2 composite image by receiving a Channel #2 composite image frame, by obtaining sprites for the respective channels through post processing, and by rendering the separated sprites for the respective channels to fit a corresponding layout. Here, the video (Cam #1) of channel 1 and the video (Cam #2) of channel 2 are sprites separated from the Channel #2 composite image. The video (Cam #1) of channel 1 may be changed to a low-quality image and the video (Cam #2) of channel 2 may be changed to a high-quality image.

Therefore, when the user requests moving the video of channel 2 to the main image position on the multi-view screen using the Channel #1 composite image, the player330may perform layout switching between multi-view modes by initially changing a layout using sprites for the respective channels separated from the Channel #1 composite image to prevent a screen disconnection and then receiving the Channel #2 composite image and rendering sprites for the respective channels separated from the Channel #2 composite image to fit the corresponding layout.

FIGS.11and12illustrate examples of a process of switching a layout from a multi-view mode to a full-view mode according to an example embodiment.

FIGS.11and12illustrate examples of a process of switching from a multi-view mode using a Channel #1 composite image to a full-view mode of a video (Cam #2) of channel 2.

When the user requests switching to a full-view mode for the video (Cam #2) of channel 2 on a multi-view screen using the Channel #1 composite image, the player330may render only a sprite corresponding to the video (Cam #2) of channel 2 among sprites for the respective channels separated from the Channel #1 composite image as illustrated inFIG.11, for seamless screen switching. The video (Cam #2) of channel 2 separated from the Channel #1 composite image may be upscaled in a low-quality state as a low-quality image and displayed in a full-view mode. Here, videos (Cam #1 and Cam #3 to Cam #6) of other channels separated from the Channel #1 composite image are not rendered and maintained in a hidden state.

Next, referring toFIG.12, to provide a high-quality image for the video (Cam #2) of channel 2 displayed in the full-view mode, the player330may receive the Channel #2 composite image from the encoder server320. The player330may receive a Channel #2 composite image frame, may obtain sprites for the respective channels through post processing, and may render only a sprite corresponding to the video (Cam #2) of channel 2 among the separated sprites for the respective channels as a full screen. The video (Cam #2) of channel 2 separated from the Channel #2 composite image may be upscaled in a high-quality state as a high-quality image and displayed in the full-view mode. Here, the videos (Cam #1 and Cam #3 to Cam #6) of other channels separated from the Channel #2 composite image are not rendered and maintained in a hidden state

Therefore, when the user requests switching the video of channel 2 to the full-view mode on the multi-view screen using the Channel #1 composite image, the player330may perform layout switching from the multi-view mode to the full-view mode by initially displaying the video (Cam #2) of channel 2 with a low resolution that is a sprite separated from the Channel #1 composite image and then receiving the Channel #2 composite image and displaying the video (Cam #2) of channel 2 with a high resolution that is a sprite separated from the Channel #2 composite image.

FIGS.13and14illustrate examples of a process of switching a layout from a full-view mode to a multi-view mode according to an example embodiment.

FIGS.13and14illustrate examples of a process of switching from a full-view mode of the video (Cam #2) of channel 2 to a multi-view using the Channel #1 composite image.

When the user requests switching from a full-view screen of a video (Cam #2) of channel 2 using a Channel #2 composite image to a multi-view mode of a default layout, for example, a multi-view mode using a Channel #1 composite image, the player330may initially render a sprite for each channel separated from the Channel #2 composite image to fit a layout corresponding to the request as illustrated inFIG.13, for seamless screen switching. Although the video (Cam #2) of channel 2 is displayed in the full-view mode, videos (Cam #1 and Cam #3 to Cam #6) of other channels are included in a hidden state and thus, immediate switching to the multi-view mode may be performed. The video (Cam #1) of channel 1 and the video (Cam #2) of channel 2 in a current multi-view mode correspond to sprites separated from the Channel #2 composite image. The video (Cam #2) of channel 2 may be downscaled to fit a changed layout as a high-quality (high-resolution) image and the video (Cam #1) of channel 1 may be upscaled in a low-quality state as a low-quality (low-resolution) image and may be displayed at a main image position.

Next, referring toFIG.14, to provide a high-quality image for the video (Cam #1) of channel 1, the player330may receive the Channel #1 composite image from the encoder server320. The player330may compose a multi-view mode using the Channel #1 composite image by receiving a Channel #1 composite image frame, by obtaining sprites for the respective channels through post processing and then rendering the separated sprites for the respective channels to fit a corresponding layout. Here, the video (Cam #1) of channel 1 and the video (Cam #2) of channel 2 are sprites separated from the Channel #1 composite image, and the video (Cam #1) of channel 1 may be changed to a high-quality image and the video (Cam #2) of channel 2 may be changed to a low-quality image.

Therefore, when the user requests switching from the full-view screen of the video (Cam #2) of channel 2 using the Channel #2 composite image to the multi-view mode using the Channel #1 composite image, the player330may perform layout switching from the full-view mode to the multi-view mode by initially changing a layout using sprites for the respective channels separated from the Channel #2 composite image to prevent a screen disconnection and then receiving the Channel #1 composite image and rendering sprites for the respective channels separated from the Channel #1 composite image to fit the corresponding layout.

For layout switching between multi-view modes or for layout switching between a multi-view mode and a full-view mode, a composite image of a layout to switch to may be received using a track switching technology.

In a mode in which an audio and a video are interleaved, the player330may seamlessly play back the audio and may softly switch only the video although the audio and the video are interleaved when switching a composite image received from the encoder server320to a changed track.

In an environment in which a source modification or a buffer control of the player330is customizable, at least one of ultra-low-latency (ULL) technology and technology of flushing data in a buffer of the player330and receiving new data may be applied to minimize a delay occurring in layout switching. For example, for faster switching from a view mode using the Channel #1 composite image to a view mode using the Channel #2 composite image, the player330may apply a scheme of emptying previous data (a Channel #1 composite image stream) and filling new data (a Channel #2 composite image stream) in a buffer at a point in time at which a view mode switching request is received.

In an environment in which the player330is non-customizable, a request of the player330may be implemented by catching the request and making a re-request to a segment (TS file) of a switched track in a multi-view interceptor. The multi-view interceptor serves as a server and may be software modules installed and running on a terminal device of the user to play back a video with the player330. When switching from the multi-view mode using the Channel #1 composite image to the full-view mode, the player330may continuously request the multi-view mode and, here, the multi-view interceptor may change the request of the player330to the full-view mode and may request the changed full-view mode. An audio of a TS file of each track may be encoded with the same audio to provide a seamless user experience when switching.

In general, when a stream of the same resolution is repeatedly output, a type of a corresponding stream may be unidentifiable. In a multi-view environment, a type of a stream that is currently being output needs to be known and, based on the known information, videos of the respective channels may be separated according to a corresponding layout.

The player330may determine an image switching point in time between view modes based on meta information for image identification as unique information included in an image stream. The meta information for image identification may be diversified according to a system environment. For example, in an environment using a TS chunk, the image switching point in time may be determined using timed-metadata included in a TS container. The timed-metadata includes time information of a predetermined time standard (PTS) and metadata is output when a corresponding time is reached.

For example, referring toFIG.15, the player330may identify a corresponding image based on metadata15included in a TS chunk of an image stream in a buffer1502. That is, if a timed-metadata value is input at a point in time of switching from a Channel #1 composite image stream to a Channel #2 composite image stream, the player330may recognize a stream input after timed-metadata as a Channel #2 composite image and may separate and use an image of each channel.

In a system in a structure of providing a live video service through a session connection between a client and a server, an image switching point in time may be changed at a server end, however, a session base may have limitations in a number of users to which a service may be provide. Dissimilar thereto, the example embodiments may determine an image switching point in time on a client side using timed-metadata and, in particular, may provide a live streaming video service through an HTTP-based protocol rather than a session base. Therefore, the example embodiments may apply to an existing system environment and may provide a stable service to more users.

As described above, according to example embodiments, it is possible to play back live videos of multiple channels using a minimum number of networks by composing a multi-view mode by receiving a single stream in which live videos of multiple channels are synthesized into a single video in a multi-stream live transmission environment. In addition, according to example embodiments, it is possible to improve quality of service (QoS) and quality of experience (QoE) by implementing a seamless switching mode while maintaining synchronization between an audio and a video when switching between channels in a multi-stream live transmission environment.

The apparatuses described herein may be implemented using hardware components, software components, and/or a combination of the hardware components and the software components. For example, the apparatuses and the components described herein may be implemented using one or more general-purpose or special purpose computers or processing devices, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of responding to and executing instructions in a defined manner. A processing device may run an operating system (OS) and one or more software applications that run on the OS. A processing device also may access, store, manipulate, process, and create data in response to execution of the software. For purpose of simplicity, the description of a processing device is used as singular; however, one skilled in the art will be appreciated that a processing device may include multiple processing elements and/or multiple types of processing elements. For example, a processing device may include multiple processors or a processor and a controller. In addition, different processing configurations are possible, such as parallel processors.

The software may include a computer program, a piece of code, an instruction, or some combinations thereof, for independently or collectively instructing or configuring a processing device to operate as desired. Software and/or data may be embodied in any type of machine, component, physical equipment, computer storage medium or device, to be interpreted by the processing device or to provide an instruction or data to the processing device. The software also may be distributed over network coupled computer systems so that the software is stored and runs in a distributed fashion. The software and data may be stored by one or more computer readable storage media.

The methods according to the above-described example embodiments may be configured in a form of program instructions performed through various computer devices and recorded in computer-readable media. Here, the media may continuously store computer-executable programs or may transitorily store the same for execution or download. Also, the media may be various types of recording devices or storage devices in a form in which one or a plurality of hardware components are combined. Without being limited to media directly connected to a computer system, the media may be distributed over the network. Examples of the media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical media such as CD-ROM and DVDs; magneto-optical media such as floptical disks; and hardware devices that are specially configured to store program instructions, such as read-only memory (ROM), random access memory (RAM), flash memory, and the like. Examples of other media may include record media and storage media managed by an app store that distributes applications or a site that supplies and distributes other various types of software, a server, and the like.

Although the example embodiments are described with reference to some specific example embodiments and accompanying drawings, it will be apparent to one of ordinary skill in the art that various alterations and modifications in form and details may be made in these example embodiments without departing from the spirit and scope of the claims and their equivalents. For example, suitable results may be achieved if the described techniques are performed in different order, and/or if components in a described system, architecture, device, or circuit are combined in a different manner, or replaced or supplemented by other components or their equivalents.

Therefore, other implementations, other example embodiments, and equivalents of the claims are to be construed as being included in the claims.