Patent ID: 12223719

DETAILED DESCRIPTION

Disclosed hereinafter are exemplary embodiments of the present invention. Particular structural or functional descriptions provided for the embodiments hereafter are intended merely to describe embodiments according to the concept of the present invention. The embodiments are not limited as to a particular embodiment.

Terms such as “first” and “second” may be used to describe various parts or elements, but the parts or elements should not be limited by the terms. The terms may be used to distinguish one element from another element. For instance, a first element may be designated as a second element, and vice versa, while not departing from the extent of rights according to the concepts of the present invention.

Unless otherwise clearly stated, when one element is described, for example, as being “connected” or “coupled” to another element, the elements should be construed as being directly or indirectly linked (i.e., there may be an intermediate element between the elements). Similar interpretation should apply to such relational terms as “between”, “neighboring,” and “adjacent to.”

Terms used herein are used to describe a particular exemplary embodiment and should not be intended to limit the present invention. Unless otherwise clearly stated, a singular term denotes and includes a plurality. Terms such as “including” and “having” also should not limit the present invention to the features, numbers, steps, operations, subparts and elements, and combinations thereof, as described; others may exist, be added or modified. Existence and addition as to one or more of features, numbers, steps, etc. should not be precluded.

Unless otherwise clearly stated, all of the terms used herein, including scientific or technical terms, have meanings which are ordinarily understood by a person skilled in the art. Terms, which are found and defined in an ordinary dictionary, should be interpreted in accordance with their usage in the art. Unless otherwise clearly defined herein, the terms are not interpreted in an ideal or overly formal manner.

Example embodiments of the present invention are described with reference to the accompanying drawings. However, the scope of the claims is not limited to or restricted by the example embodiments. Like reference numerals proposed in the respective drawings refer to like elements.

Hereinafter, example embodiments of an apparatus for predicting a video frame will be described with reference toFIGS.1to5.

FIG.1is a block diagram of an apparatus for predicting a video frame according to an embodiment.

As shown inFIG.1, a video frame prediction apparatus100may include a processor110and a storage unit150connected to the processor110. The video frame prediction apparatus100may further include an interface101connected to at least one of the processor110and the storage unit150.

The processor110may receive at least one raw data151for video prediction from at least one of the interface101and the storage150, perform training on a learning model120based on the received at least one raw data151, or input at least one raw data151to a trained learning model120a, and thus obtain at least one piece of prediction data159corresponding to the at least one piece of raw data151. Here, the raw data151may include at least one video frame “x”, and the at least one video frame “x” may be a video frame “x” at one or more specific time points “t” (“t” is 0 or a natural number of 1 or more, but not limited thereto). At least one video frame “xt” may be a part of at least one moving picture, and in this case, a series of sequentially combined video frames “xi:T” may form at least one moving picture. Here, “i” may be an index for the start of a video, and “T” may be an index for a reference time. “i” may include 1, “T” may include the last value of the input video, and “T” may correspond to the length of the input video. The at least one prediction data159may include at least one video frame “{circumflex over ( )}x” predicted from the at least one raw data151, and specifically, may include a predicted video frame “{circumflex over ( )}xt” corresponding to the frame “xt” of a specific time “t”, which is the raw data151. In detail, the prediction data159may include a prediction result (i.e., at least one predicted video frame ({circumflex over ( )}x(T+1):(T+K)) for at least one video frame in the future (e.g., (T+1) time to (T+K) time point, where “K” is a natural number greater than or equal to 1) that will appear sequentially for a series of sequentially combined video frames “xi:T”.

The processor110may include a data acquisition unit111and a learning processing unit115, and may further include a result processing unit119as necessary.

The data acquisition unit111may acquire data, e.g., the raw data151, required by the learning processing unit115from the interface101or the storage unit150and may transmit it to the learning processing unit115.

The learning processing unit115may train the learning model120using the obtained raw data151. In detail, the learning processing unit115may obtain the trained learning model120aby applying the video frame x1:T or x1:(T−1) up to the reference time point “T” or the earlier time point (e.g., time point (T−1)) to the learning model120. Also, the learning processing unit115may obtain the prediction data159corresponding to the raw data151by performing prediction by applying the raw data151to the trained learning model120a. Here, the learning processing unit115may obtain a future video frame x(T+1):(T+K) (hereinafter, referred to as a predicted video frame) corresponding to the input video frame xT or x(T+1) by applying the video frame xT or x(T+1) of the reference time point “T” or a later time point (e.g., time point (T+1)) to the trained learning model120a. At least one prediction data159obtained by the learning processing unit115may be transmitted to the storage unit150and then stored by the storage unit150, or may be transmitted to the interface101and provided to a user. According to an embodiment, the learning processing unit115may perform both a training and a prediction process, or may perform only one of these processes. A detailed description of the learning processing unit115will be described later.

The result processing unit119may generate at least one data to be provided to a user or transmitted to an external information processing device (not shown) by using the prediction data159obtained by the learning processing unit115. For example, the result processing unit119may generate at least one video data (e.g., a moving picture) by sequentially combining the video frames x(T+1):(T+K) predicted by the learning processing unit115, and transmit it to the storage unit150or the interface101.

In addition, the result processing unit119may generate information on how much the processing result (e.g., video frames acquired during training or prediction {circumflex over ( )}x(2:(T+K) of the learning processing unit115differs from the actual data (e.g., the corresponding actual video frame x2:(T+K), and determine the accuracy of the prediction data159based thereon. For example, the result processing unit119may measure the degree of difference between the predicted data159and the actual data x2:(T+K) using at least one loss function “L2”. The loss function “L2” may include a mean square error (MSE) function, and may be given as in Equation 1 below.

L2⁡(G)=12⁢∑i=2T+K⁢⁢xi-x^i2[Equation⁢⁢1]

When the aforementioned loss function “L2” is minimized, the prediction data159identical to or close to the actual data151(i.e., raw data) may be generated. According to the experiment, the learning result of the above-described learning processing unit115was found to minimize such a loss function “L2”. That is, the processor110may generate a consistent video frame without additional loss correction. In addition, the result processing unit119may generate at least one additional piece of various data based on the prediction data159depending on a selection of a designer or a user.

In addition, according to an embodiment, the processor110may further perform calculation, determination, and control operations related to the overall driving of the video frame prediction apparatus100, besides the above. For example, the processor110may control the interface101, and thus the prediction data159obtained by the processor110may be transmitted to an external device or displayed to a user.

According to an embodiment, the processor110may drive a program (which may be referred to as an app, application, or software) stored in the storage unit150to perform an operation, determination, processing and/or control operation necessary for obtaining the aforementioned prediction data159. The processor110may include a graphics processing unit (GPU), a central processing unit (CPU), a micro controller unit (MCU), a microprocessor (Micom), an application processor (AP), an electronic controlling unit (ECU), and/or other electronic devices capable of processing various calculations and generating control signals, and the like. These devices may be implemented using one or more semiconductor chips or the like.

The storage unit150may temporarily or non-temporarily store data151and159required for the operation of the processor110or acquired by the processor110and may provide the stored data151and159according to the control of the processor110or predefined settings to the processor110or the interface101. For example, the storage unit150may store at least one raw data151acquired through the interface101, and may provide the raw data151to the processor110based on a call of the processor110, and/or may store the prediction data159obtained by the processor110. In addition, the storage unit150may further store at least one learning model120, the trained learning model120aand/or at least one program implemented including one of them, to be used by the processor110to obtain the prediction data159. Here, the learning models120and120aor the program may be stored or modified in the storage unit150after the designer writes or inputs them using the interface101, or may be obtained or updated through an electronic software distribution network accessible via a wired or wireless communication network. The trained learning model120amay be trained by another information processing device and stored in the storage unit150through the interface101or the like. The storage unit150may be fixed or detachably mounted to the video frame prediction apparatus100according to an embodiment, and may include, for example, at least one of a main memory device and an auxiliary memory device. The main memory device may be implemented using a semiconductor storage medium such as ROM and/or RAM. The auxiliary storage device may be implemented using at least one storage medium capable of permanently or semi-permanently storing data, such as a flash memory device, a secure digital (SD) card, a solid state drive (SSD), a hard disk drive (HDD), a magnetic drum, a compact disc (CD), DVD or a laser disc.

The interface101may receive an instruction, command, data or program from a user and/or may output a processing result to the outside. For example, the interface101may receive the raw data151or the learning models120and120afrom the user or other information processing device (not shown), or may receive a command related to learning start from the user, and may visually output the prediction data159or the like obtained according to the learning process to the user, or may transmit it to another information processing device. The interface101may include at least one of an input unit and an output unit. The input unit may include, for example, a keyboard, a mouse, a tablet, a touch screen, a touch pad, a scanner, a video capturing module, a microphone, a track ball, and a track pad, a data input/output terminal capable of receiving data from an external device (memory device, etc.) and/or a communication module (e.g., LAN card, short-distance communication module, or mobile communication module, etc.) that is connected to an external device through a wired/wireless communication network. The output unit may include, for example, a display, a printer device, a speaker device, a video output terminal, a data input/output terminal, and/or a communication module, but is not limited thereto. The interface101may be provided integrally with the video frame prediction apparatus100or may be provided to be physically separable.

The above-described video frame prediction apparatus100may be implemented using one information processing device capable of processing data, or may be implemented by combining two or more information processing devices or data storage devices that are the same or different from each other. In the latter case, the two or more information processing apparatuses may be implemented using the same apparatus or may be implemented using different apparatuses. For example, the interface101of the video frame prediction apparatus100may be implemented using at least one terminal device (e.g., a desktop computer or a smart phone), and the processor110may be implemented by using another information processing device (e.g., a server computer, etc.) communicatively connected to at least one terminal device. Here, another information processing device may be provided to include the storage unit150. The one or more information processing devices described above may include at least one device capable of arithmetic processing contemplated by a designer, for example, a desktop computer, a laptop computer, a server computer, a smart phone, a tablet PC, a navigation device, a portable game console, a smart watch, a head mounted Display (HMD) device, a digital television, a set-top box, home appliances (a refrigerator and a robot vacuum, etc.), artificial intelligence sound reproduction device (an artificial intelligence speaker), vehicles, manned aircraft, an unmanned aerial vehicle, a robot, or an industrial machine. However, the information processing apparatus usable as the video frame prediction apparatus100is not limited thereto, and various apparatuses may be used as the above-described video frame prediction apparatus100if necessary.

Hereinafter, the learning processing unit115will be described in more detail.

FIG.2is a block diagram of an embodiment of a learning processing unit, andFIG.3is a view illustrating an example of a level.

As shown inFIG.2, the learning processing unit115may perform training or prediction based on the learning models120and120a, wherein the learning models120and120amay be previously trained or not trained. In an embodiment, the learning models120and120amay include an N-level encoder121, a feature learning unit123connected to the N-level encoder121, and an N-level decoder129connected to feature learning unit123. Here, the N-level encoder121, the feature learning unit123, and the N-level decoder129may be provided to correspond to each other.

The N-level encoder121may extract a latent representation from the input raw data151, that is, at least one video frame(s). The latent representation may be information that is potentially present in the video frame but is difficult for a human to derive manually, may be features that are self-learned through a deep learning neural network and help prediction and may include N levels (“N” is the number of encoder and decoder layers) of hierarchical feature(s). Here, the feature may include at least one piece of information about a subject or background necessary for prediction, such as a shape or movement (change in position, change in size, change in shape and/or change direction or rate of change, etc.) of a subject or background in at least one video frame (e.g., the video frame “xt” at a specific time point). The number of levels or the content of each level may be arbitrarily defined by a user or a designer. For example, a level may include four levels (e.g., N=4), such a as high-level, a mid-high-level, a mid-low-level, and a low-level, and thus, a high-level feature, a middle-high-level feature, a mid-lower-level feature, and a low-level feature may be extracted from the video frame “xt” input by the N-level encoder121. Here, the higher-level features may make the prediction process simpler, and the lower-level features may be defined to make the predicted video frame more realistic. For example, the higher-level feature may be defined to indicate the identity of the object in the video frame “xt”, the position or speed of the object, and the lower-level feature may be defined to represent a shape such as a circle or a boundary, or a detailed shape of a specific object. However, the number of the above-described level encoders121-1to121-4is not limited to 4. The number of these levels or the definition of the features of each level is exemplary, and the number of levels (i.e., the number of level encoders121-1to121-4) may be 3 or less or 5 or more depending on the designer or user's selection, and correspondingly, the features of each level may also be defined in various ways depending on a designer or a user. The N-level encoder121may implement the accuracy of the prediction data159by extracting the latent feature from the video frame “xt” and using it for prediction. The level-specific features obtained by the N-level encoder121may be transmitted to each feature learning unit123.

In detail, the N-level encoder121, as shown inFIG.3, may include at least one level of encoders121-1to121-4, and may include a first-level encoder121-1, a second-level encoder121-2connected to the first-level encoder121-1, a third-level encoder121-3connected to the second-lever encoder121-2, and a fourth-level encoder121-4connected to the third-level encoder121-3. Each of the first to fourth-level encoders121-1to121-4may hierarchically extract features from the video frame “xt” of at least one time point. For example, the first-level encoder121-1may extract the high-level feature, the second-level encoder121-2may extract the mid-high-level feature, and the third-level encoder121-3may extract the mid-low level feature. Also, the fourth-level encoder121-4may extract a low-level feature. Accordingly, one or more data on the layered feature may be obtained. The feature(s) acquired by each of the N-level encoders121-1to121-4may be transmitted to the corresponding feature learning units123-1to123-4. In other words, in the N-level encoders121:121-1to121-4, the video frame “xt” of at least one time point is compressed into a latent representation, and the latent representation is transmitted to the level decoder129through the feature learning unit123.

According to an embodiment, the first-level encoder121-1, the second-level encoder121-2, the third-level encoder121-3, and the fourth-level encoder121-4may all be built on the same learning model, some may be the same and some may be built on a different learning model, or they may all be built on a different learning model. Here, the learning model may include at least one learning model, such as a convolutional neural network (CNN), a deep neural network (DNN), a recurrent neural network (RNN), and/or a deep belief neural network (DBN).

The feature learning unit123may learn temporal dynamics of each feature of at least one level. The feature learning unit123may include at least one feature learning unit123-1to123-4separately provided for each level. The at least one feature learning unit123-1to123-4may be provided corresponding to the number of levels for each feature to be extracted. For example, if features are extracted through four levels as described above, the feature learning units123corresponds to the four feature learning units123-1to123-4may be included. In other words, when the four level encoders121-1to121-4are provided, the feature learning units123may include the first feature learning unit123-1corresponding to the first-level encoder121-1, the second feature learning unit123-2corresponding to the second-level encoder121-2, the third feature learning unit123-3corresponding to the third-level encoder121-3, and the fourth feature learning unit123-4corresponding to the fourth-level encoder121-4. Here, the number of the first to fourth feature learning units123-1to123-4is exemplary, and the feature learning unit123may include three or less feature learning units123-1to124-3according to an embodiment, or five or more feature learning units (not shown).

Based on the operation result of the feature learning unit123, the number of predicted feature data corresponding to the number of levels may be output. In detail, the first feature learning unit123-1may output feature data predicted by estimating the high-level feature extracted by the first-level encoder121-1, and the second feature learning unit123-2may output feature data predicted for the mid-high-level feature extracted by the second-level encoder121-2. Also, in the same way, the third feature learning unit123-3may output feature data predicted for the mid-low-level feature extracted by the third-level encoder121-3, and the fourth feature learning unit123- may output feature data predicted for the low-level feature extracted by the fourth encoder121-4.

According to an embodiment, each feature learning unit123:123-1to123-4may be implemented using at least one learning model such as a long short term memory (LSTM), a deep neural network, a convolutional neural network, a recurrent neural network, a deep trust neural network, and a deep Q-network, a generative adversarial network (GAN), and/or a conditional adversarial network (cGAN). In this case, each of the feature learning units123:123-1to123-4may be all built based on the same learning model, or some or all of them may be implemented based on different learning models. The predicted feature data obtained by the feature learning units123:123-1to123-4may be transmitted to the N-level decoder129. The N-level decoder129may obtain at least one prediction data159(i.e., the predicted video frame “{circumflex over ( )}xt”) based on the at least one layered data (i.e., at least one predicted feature data corresponding to at least one level of feature) obtained by the feature learning unit123. For example, the N-level decoder129may receive a plurality of predicted feature data (e.g., four predicted feature data corresponding to the four feature learning units123-1to123-4), and may obtain at least one predicted video frame “{circumflex over ( )}xt” by combining and processing the received four predicted feature data. Here, the N-level decoder129may be provided to correspond to the N-level encoder121. Accordingly, “N” of the N-level encoder121and the N-level decoder129may be the same value as each other. In detail, for example, referring toFIG.3, the N-level decoder129may include a first-level decoder129-1, a second-level decoder129-2connected to the first-level decoder129-1, a third-level decoder129-3connected to the second-level decoder129-2, and a fourth-level decoder129-4connected to the third-level decoder129-3. Each of the decoders129-1to129-4Each of the decoders129-1to129-4may receive at least one predicted feature data simultaneously or sequentially from each of the feature learning units123-1to123-4corresponding to the respective decoders129-1to129-4. For example, the first-level decoder129-1may receive predicted feature data from the first feature learning unit123-1connected to the first-level encoder121-1, the second level decoder129-2may receive the predicted feature data from the second feature learning unit123-2connected to the second-level encoder121-2, and the third-level decoder129-3may receive the predicted feature data from the third feature learning unit123-3connected to the third-level encoder121-3. Also, the fourth-level decoder129-4may receive predicted feature data from the fourth feature learning unit123-4connected to the fourth-level encoder121-4. Each of the first to fourth-level decoders129-1to129-4may obtain and output analysis and reconstruction results corresponding to the predicted feature data alone or in combination. The analysis and restoration results of each of the first to fourth-level decoders129-1to129-4, depending on the situation or design, may be transmitted to the other level decoders129-1to129-3and/or the output frame generator129-9, and/or may be input back to the N-level encoder121(e.g., the first-level encoder121-1) and/or output to the outside.

According to an embodiment, the first-level decoder129-1, the second-level decoder129-2, the third-level decoder129-3, and the fourth-level decoder129-4may be built using all the same learning model, some may be built using the same and some may be built using different learning models, or they may all be built using different learning models. In addition, each of the N-level decoders129-1to129-4may be constructed using the same learning model or a different learning model as the corresponding N-level encoders121-1to121-4. Here, the learning model may include at least one of various learning models available as the N-level decoder129, such as a convolutional neural network, a deep neural network, a recurrent neural network, and/or a deep trust neural network, according to an embodiment.

According to an embodiment, the N-level decoder129may further include an output frame generator129-9. The output frame generator129-9may maintain the same size of the output video frame “{circumflex over ( )}xt”. For example, when the size of the video frame predicted by the plurality of level decoders129-1to129-4is different from the size of the input original video frame “xt”, the output frame generator129-9may set the size of the predicted video frame “{circumflex over ( )}xt” to be obtained by correcting the size of the predicted video frame to be the same as the size of the original input video frame “xt”. The output frame generator129-9may be designed using at least one learning model or using a hidden layer of at least one learning model, according to an embodiment. For example, the output frame generator129-9may be implemented using a lxi convolutional layer or the like. In addition, according to the designer's selection, the output frame generator129-9may be implemented using one or more learning models or hidden layers alone or in combination. The output result of the output frame generator129-9may be externally output depending on circumstances and/or may be fed back to the N-level encoder121, for example, the first-level encoder121-1.

Operations of the above-described N-level encoder121, the feature learning unit123, and the N-level decoder129may be expressed by Equations 2 to 4 below.
(Ft1,Ft2,Ft4,Ft4)=Enc(xt)  [Equation 2]
(Htm,Ctm)=ConvLSTMm(Ftm,Ct−1m),m=1,2,3,4  [Equation 3]
{circumflex over (x)}t+1=Dec(Ht1,Ht2,Ht3,Ht4)

Here, “F{circumflex over ( )}m_t” refers to the processing result (e.g., when the convolutional neural network is used as the m-th level encoder121-m, the convolutional neural network features from the low-level to the high-level extracted by the m-th level encoder121-m) of the m-th level encoder121-m(m=1, 2, 3, 4, . . . ) with respect to the input video frame “xt” at a specific time point “t”. convLSTM_m refers to the m-th feature learning unit123-mconstructed using long-term memory, and H{circumflex over ( )}m_t and C{circumflex over ( )}m_t mean a hidden state and a cell state of the m-th feature learning unit123-mfor the video frame “xt” at a specific time point “t”, respectively. Enc( ) and Dec( ) correspond to the N-level encoder121and the N-level decoder129, respectively.

FIG.4is a diagram illustrating a training process of a learning processing unit.

According to an embodiment, the learning processing unit115, as shown inFIG.4, may receive the raw data151, e.g., video frames (x1to x(T−1)) of the first to (T−1)th time points, arbitrarily or sequentially and may train the learning model (i.e., the N-level encoder121, the feature learning unit123, and/or the N-level decoder129) based thereon. According to an embodiment, the learning processing unit115may receive past or present predicted data159, for example, a predicted video frame “{circumflex over ( )}xt”. In this case, while training is being performed, the video frames of the first to (T−1)-th time points x1to x(T−1) and at least one video frame x1to x(T−1) or {circumflex over ( )}xt arbitrarily selected from the predicted video frame “{circumflex over ( )}xt” may be input to the learning processing unit115for training. Selection of at least one of the correct answer video frame “xt” and the predicted video frame “{circumflex over ( )}xt” may be performed based on a probability. In detail, for example, initially, by setting the probability to 0, video frames x1to x(T−1) or {circumflex over ( )}xt are selected, and as the probability value is gradually increased depending on a fixed value or an arbitrary value at each repetition of training, at least one video frame x1to x(T−1) or {circumflex over ( )}xt may be selected. Here, the selected video frame(s) x1to x(T−1) or {circumflex over ( )}xt may be sequentially or arbitrarily input to the learning processing unit115, respectively, and the learning processing unit may perform training using the input video frames x1to x(T−1) or {circumflex over ( )}xt. In this way, when the first to (T−1)-th times or the predicted video frames x1to x(T−1) or {circumflex over ( )}xt are selectively input, the learning processing unit115may gradually train the learning model to correct possible errors or mistakes, and thus the learning model has the advantage of being more robust to prediction errors.

When the video frame(s) x1to x(T−1) and/or {circumflex over ( )}xt are input, the learning processing unit115may perform training using the N-level encoder121, the feature learning unit123, and the N-level decoder129. In other words, when the video frames x1to x(T−1) of the first to (T−1) times are input or at least one of the video frames x1to x(T−1) and the predicted video frame “{circumflex over ( )}xt” of the first to (T−1)-th times are selectively input, the training is performed through each level encoder121-1to121-4, each feature learning unit123-1to123-4, and each level decoder129-1to129-4. The learning model120including the level encoders121-1to121-4, the feature learning units123-1to123-4, the level decoders129-1to129-4, the output frame generating unit129-9may be repeatedly trained at least once according to a sequential or arbitrary input of each video frame x1to x(T−1) or {circumflex over ( )}xt. For example, the learning model120is trained by the input of the video frame x1at the first time point, and by receiving the feature values from the first to fourth feature learning units123-1to123-4, training by the input of the video frame x2at the second time is restarted in the same or partly different way. The trained learning model120is obtained based on repetition of such training. That is, the learning model120is trained by repeating the above process by sequential or arbitrary input of the video frames x1to x(T−1) and/or “{circumflex over ( )}xt”. That is, whenever the video frame “xt” is input and passes through the learning model120consisting of the N-level encoder121, the feature learning unit123, the N-level decoder129, and the output frame generator129-9, the model is trained. Here, the first to fourth feature learning units121-1ato123-4atrained through any one training process (e.g., the first training process) may be designed to further utilize the latent representations obtained by the first to fourth feature learning units121-1to123-4in previous training. In other words, the latent representation obtained in training for the video frame x1at the first time point may affect training for the video frame x2at the second time point immediately following or training on video frames x2to x(T−1) at some point in the future. Here, the previous training may include, for example, the training immediately preceding the first training with respect to the second training and include at least one previous training, according to an embodiment.

According to an embodiment, in the training process, the above-described result processing unit119may use the loss function “L2” to calculate and obtain an error between the predicted value “{circumflex over ( )}xt” and the actual value “xt”. The result processing unit119may transmit the calculation and the obtained error to the learning processing unit115, and thus the learning processing unit115may reflect it in training, or may output it to the outside if necessary.

FIG.5is a diagram for explaining a prediction process of a learning processing unit.

According to an embodiment, the learning processing unit115may perform prediction on the predicted video frames {circumflex over ( )}x(T+1) to {circumflex over ( )}x(T+3), as shown inFIG.5. Here, the predicted video frames {circumflex over ( )}x(T+1) to {circumflex over ( )}x(T+3) may include a video frame that will appear at a time point (T+1) to (T+3) after at least one time point “T”. In detail, when the video frame “xT” at time “T” is input to the learning processing unit115, as described above, the first level encoder to the fourth level encoder121-1to121-4may extract features corresponding to each level from the video frame “xT” and may transmit them to the first to fourth feature learning units123-1kto123-4k, respectively. Each of the first to fourth feature learning units123-1kto123-4kmay obtain predicted feature data corresponding to the transmitted features and may transmit the predicted feature data to the corresponding first to fourth level decoders129-1to129-4. The first to fourth level decoders129-1to129-4and the output frame generator129-9obtains a predicted video frame {circumflex over ( )}x(T+1) for the (T+1)-th time point based on the predicted feature data. The obtained predicted video frame {circumflex over ( )}x(T+1) may be transferred to and stored in the storage150. In addition, according to an embodiment, the obtained predicted video frame {circumflex over ( )}x(T+1) may be input to the first level encoder to the fourth level encoder121-1to121-4again, and same first-level encoder to four-level encoder121-1to121-4trained above may extract features corresponding to each level from the input predicted video frame {circumflex over ( )}x(T+1) in the same manner as described above, and may transmit it to the trained first to fourth feature learning units123-11to123-41. As described above, the features obtained by each of the first to fourth-level encoders121-1to121-4may be transmitted to the corresponding first to fourth feature learning units123-11to123-41. The trained first to fourth feature learning units123-11to123-41may obtain predicted feature data in response thereto, respectively, and the predicted feature data may be transmitted to the trained first to fourth-level decoders129-1to129-4. The first to fourth-level decoders129-1to129-4and the output frame generation unit129-9may obtain a predicted video frame {circumflex over ( )}x(T+2) for the (T+2)-th time point based on the predicted feature data. As described above, similarly, the predicted video frame {circumflex over ( )}x(T+2) for the (T+2)-th time point may be stored in the storage150, and may be re-input to the first to fourth level encoders121-1to121-4. Additionally, as performing operating as described above, the first to fourth feature learning units123-1mto123-4m, the first to fourth-level decoders129-1to129-4, and the output frame generating unit129-9may obtain a predicted video frame {circumflex over ( )}x(T+3) for the (T+3) for the (T+3)-th time point. The process may be continuously repeated, and may be terminated with or without additional model training depending on conditions set by a user or a designer. Here, the conditions set by the user or the designer may include whether a video frame {circumflex over ( )}x(T+K) for a specific time point (T+K) is acquired, whether a specific point in time (T+K+1) has arrived, whether to perform a predetermined number of predictions, elapse of predefined time. As a result, a predicted video frame {circumflex over ( )}x(T+1) for a time point (T+1) to a predicted video frame {circumflex over ( )}x(T+K) for a time point (T+K) are generated. Accordingly, it is possible to obtain at least one video frame {circumflex over ( )}x(T+1) to {circumflex over ( )}x(T+K) corresponding to a desired time point or period in the future.

Hereinafter, an embodiment of a method for predicting a video frame will be described with reference toFIGS.6to8.

FIG.6is a flowchart of an embodiment of a method for predicting a video frame.

As shown inFIG.6, first, at least one video may be input to the video frame prediction apparatus through a data input/output terminal, a disk reader, a card reader, and/or a communication module in200. Each of the at least one video may include at least one video frame, for example, first to T-th video frames, and the at least one video frame is sequentially connected to form at least one video. The video frame consists of a combination of a plurality of pixels. Predetermined information (e.g., RGB values) is mapped to each pixel, and accordingly, when the video frame is displayed through a display device, at least one subject or background appears on the screen. The input video may be temporarily or non-temporarily stored in a storage unit, such as a main memory device or an auxiliary memory device, if necessary.

When a video is input, a learning model may be trained based on the first to T-th video frames in the video, immediately or after a certain time has elapsed according to the user or designer's preset, or according to the operation of the user's input unit in300. Accordingly, a trained learning model is obtained. Here, the learning model including N level encoders (i.e., N-level encoders, “N” is a natural number greater than or equal to 1), N feature learning units corresponding to each of the N level encoders, and N level encoders and N feature learning units. The corresponding N level decoders (i.e., N-level decoders, which may further include an output frame generator if necessary) may be provided. Here, the N-level encoder and the N-level decoder may be implemented using the same or different neural networks, where the neural network is, for example, a convolutional neural network, a deep neural network, a recurrent neural network, a long-term memory and/or a deep trust neural network or the like. In addition, the feature learning unit may also be implemented using a neural network that is different from both the N-level encoder and the N-level decoder or the same as at least one of the N-level encoder and the N-level decoder, and may be implemented with, for example, a long-term memory.

Based on the trained learning model, future video frames (i.e., predicted video frames, e.g., (T+1)th to (T+K)th video frames) may be generated and obtained in400. In detail, the last video frame (i.e., the T-th video frame) of the first to T-th video frames used for training may be input to the trained learning model to obtain the (T+1)-th to (T+K)-th video frames subsequent to the T-th video frame. Acquisition of the predicted video frame may be performed by the same information processing device as the information processing device that has performed learning, or may be performed by an information processing device different from the information processing device that has performed learning. Here, the different information processing devices may be connected to the learned information processing device through a wired/wireless communication network.

When the predicted video frame is generated and obtained, the obtained predicted video frame may be stored or may be visually and/or aurally output to the outside in500. If necessary, the predicted video frames may be sequentially combined based on the reproduction order to form at least one video, and then the formed video may be output or stored. The output or storage of the predicted video frame may be performed in real time. That is, each of the (T+1)th to (T+K)th video frames may be stored in the storage unit and/or output to the outside whenever they are predicted and generated.

Hereinafter, each of the training process300and the predicted video frame generating process400of the above-described learning model will be described in more detail.

FIG.7is a flowchart of an embodiment of a process for training a learning model.

Referring toFIG.7, first, a video frame (i.e., a first video frame) at a first time point (t=1) may be input to the learning model in300and302.

The level encoder of the learning model may extract latent representations from the input first video frame, and more specifically, may extract features of at least one different level from each other in304. For example, the level encoder may include first to N-th level encoders, and the first to N-th level encoders may extract features of the first to N-th level, respectively. The extracted first to N-th level features may be hierarchical to each other. The number of level encoders may be provided equal to the number of levels of features to be extracted.

The feature learning unit may include first to N-th feature learning units corresponding to the number of level encoders. Each of the first to N-th feature learning units may receive the extracted features, may model a temporal motion for each level feature, and may obtain predicted feature data in306. In addition, the first to N-th feature learning units may be trained by the features of each level in the process of obtaining the predicted feature data.

The level decoder may receive the predicted feature data from the feature learning unit, and may obtain at least one piece of prediction data (i.e., a predicted video frame) based on the at least one piece of predicted feature data in308. The level decoder may include first to N-th level decoders corresponding to the number of level encoders and/or the number of feature learning units. Each of the first to N-th level decoders may receive the predicted feature data transmitted from each of the first to N-th feature learning units, and may perform processing thereon to obtain predicted data corresponding to each level feature. Data corresponding to each level feature may be sequentially or simultaneously combined by a level decoder, and as a result, a predicted video frame may be obtained. The output frame generator of the level decoder may obtain a final predicted video frame by correcting the size of the predicted video frame depending on the processing results of the first to N-th level decoders.

The above-described processes of302to308may be repeated at least once, for example, may be repeated for (T−1) times set by a designer or a user in310and312. That is, after the first video frame is input and learning and training are performed through the above-described processes of304to308based thereon, if T is 3 or more, the second video frame may be input again in312and302and learning may be performed on the second video frame in the same manner as in the above-described processes of304to308.

If training is performed in304to308using a predetermined video frame, for example, the (T−1)th video frame depending on a preset selection or setting, the training is completed in S314. The trained learning model may be stored in the storage unit, and may be transmitted to another video frame prediction device through a communication module or data input/output terminal if necessary, or may be output to a user through a display.

FIG.8is a flowchart of an embodiment of a process of predicting a video frame based on a learning model.

According to an embodiment of the process of predicting the video frame shown inFIG.8, the video frame at the T-th time, that is, the T-th video frame, may be input to the trained learning model in400and402. The input of the T-th video frame may be performed immediately after the (T−1)-th video frame ofFIG.7is input, or may be performed after a predetermined time has elapsed. The trained learning model may be trained through the processes of300to314described with reference toFIG.7, and may be obtained from a storage unit of the same device or from another information processing device external to the training model.

When the T-th video frame is input, the level encoder may extract a latent representation by obtaining features based on the level for the T-th video frame in404. As mentioned above, the level encoder may include N level encoders, and each level encoder may extract a feature corresponding to each level encoder. Accordingly, features of N levels are extracted.

Sequentially, the feature learning unit obtains predicted feature data based on the feature according to the level in406. The feature learning unit may also include N feature learning units corresponding to the N level encoders, and each feature learning unit may obtain predicted feature data corresponding to features of different levels.

The level decoder may obtain at least one predicted feature data corresponding to each feature of a different level from the feature learning unit, and may obtain a predicted video frame based thereon in408. In detail, the level decoder may include N level decoders corresponding to the N level encoders and/or N feature learning units, and each level decoder may obtain an analysis and restoration result corresponding to the predicted feature data based on the predicted feature data for different levels, and may obtain a (T+1)th predicted video frame based thereon. Also, the level decoder may appropriately correct the (T+1)th predicted video frame through the output frame generator.

The obtained (T+1)th predicted video frame may be temporarily or non-temporarily stored in a main memory or an auxiliary memory in409.

The above-described processes of404to409may be repeated until a predetermined predicted video frame, that is, a (T+K)th predicted video frame, is obtained according to a predefined rule in410to414. Specifically, for example, when “K” is given as 2 or more by a user or a designer, “T” is less than (T+1) (No in410), and thus (T+1)th predicted video frame is again input to the level encoder of the learning model in412. Then, as the iterative execution of the above-described processes of404to409, the (T+2)th predicted video frame is obtained and stored in414,406to409. By repeating those processes, a (T+K)th predicted video frame may be finally obtained and stored, and as a result, a total of K predicted video frames (i.e., (T+1)th to (T+K)th predicted video frames) may be sequentially obtained.

When the predicted video frame up to the desired time point, that is, the (T+K)th predicted video frame is obtained and stored (Yes in410), the prediction process of the video frame may be completed in response thereto in416. The obtained at least one predicted video frame (e.g., (T+1)th to (T+K)th predicted video frames) may be separately stored, processed, and/or transmitted or output as described with reference toFIG.6in500.

The video frame prediction method according to the above-described embodiment may be implemented in the form of a program that is capable of being driven by a computer device. Here, the program may include program instructions, data files, and data structures alone or in combination. The program may be designed and manufactured using machine code or high-level language code. The program may be specially designed to implement the above-described method, or may be implemented using various functions or definitions that are known and available to those skilled in the art of computer software. Also, here, the computer device may be implemented including a processor or memory that enables the function of the program to be realized, and may further include a communication device if necessary.

A program for implementing the above-described video frame prediction method may be recorded in a computer-readable recording medium. The computer-readable recording medium may include, for example, a semiconductor storage device such as solid state drive (SSD), ROM, RAM, or flash memory, a magnetic disk storage medium such as hard disk or floppy disk, an optical recording medium such as compact disc or DVD, a magneto-optical recording medium such as a floppy disk, and at least one type of physical device capable of storing a specific program executed in response to a computer call.

The above-described apparatus for predicting a video frame or a method for predicting the video frame may be applied and used in various fields. For example, the above-described apparatus or method may be used in the field of video streaming technology, video compression or restoration technology, or video production field, and may be used to predict the motion of a subject (human or animal) in the video-based recognition/input device. In addition, the above-described apparatus or method for predicting the video frame may be used for autonomous driving control of a vehicle, for example, to predict a road condition in front of an autonomous driving vehicle or whether there is a collision between vehicles. In addition, the above-described apparatus or method for predicting the video frame may be used for tracking an object of interest, for example, may be used for forecasting the weather, such as predicting the movement direction of fine dust, typhoons or clouds, and be used for analyzing medical videos, predicting the progression of lesions, and predicting changes in a patient's condition, and may be used to predict changes in economic indicators, such as gross national product, or changes in stocks or interest rates. In addition, the above-described apparatus for predicting the video frame or method for predicting the video frame may be used in various technical fields that require estimation and prediction of a future video based on an existing video.

Although various embodiments of the apparatus for predicting the video frame and the method for predicting the video frame have been described above, the apparatus for predicting the video frame and method for predicting the video frame are not limited to the above-described embodiments. Various devices or methods that can be implemented by those skilled in the art by modification and revision based on the above-described embodiments may also be examples of the above-described apparatus and method for predicting the video frame. For example, even if the described techniques are performed in an order different from the described method, and/or the described components of the system, structure, apparatus, and circuit are combined or mixed in a different form than the described method or substituted by other components or equivalents, it may be an embodiment of the above-described apparatus for predicting the video frame and the method for predicting the video frame.

According to the above-described apparatus and method for predicting the video frame, it is possible to obtain the effect of enabling more accurate, clear and realistic video frame prediction.

In addition, in video prediction, the effect that the device may automatically learn about the layer-by-layer representation of the input frame passed through the neural network may be obtained.

In addition, it is possible to obtain the advantage of automatically performing learning of latent features or presentations of the video frames required for prediction through the encoders, decoders and feature learning units of multiple layers.

In addition, it is possible to perform high-quality video prediction even with relatively few learning parameters, and accordingly, it is possible to obtain an effect of reducing used computer resources or prediction time.

In addition, according to the above-described apparatus and method for predicting the video frame, the video prediction may be performed excellently not only on high-level features such as speed changes of objects in the video, but also on low-level features such as values for each pixel of the video.