Patent ID: 12192673

DETAILED DESCRIPTION

FIGS.1through12, discussed below, and the various embodiments used to describe the principles of the present disclosure in this patent document are by way of illustration only and should not be construed in any way to limit the scope of the disclosure. Those skilled in the art will understand that the principles of the present disclosure may be implemented in any suitably-arranged system or device.

As noted above, with the popularity of mobile devices that include digital cameras, almost everyone can take a picture at any time. As the quality of the camera hardware in mobile devices has improved, users have begun to expect high-quality photographs from their devices. With recent developments in smartphone camera technology, one important smartphone camera function is video capture. With increasing demand to capture high-quality and interesting videos, slow motion video functions are gaining a lot of attention because they can capture very fast motion and moments in videos. However, the camera hardware in mobile devices still has significant limitations, such as poor slow-motion video quality of rapid motion.

A digital camera of a smartphone is often capable of capturing image frames using still capture (in the case of photography) or video capture (in the case of videography). A common frame rate for video capture is 24 frames per second (FPS) or 30 FPS. A digital camera may also have a slow-motion video capture mode in which the frame rate is 30 FPS. When a user uses the digital camera to capture rapid motion, such as the motion of a jumping dog or a sprinting human, slow-motion playback of the rapid motion is not smooth and is choppy. For instance, the location of the jumping dog or the sprinting human from one captured image frame to the next captured image frame can be unpredictable.

This disclosure provides various techniques for performing interpolation of video frames in order to support operations such as slow motion video capture. As described in more detail below, multiple video frames can be obtained, and a non-linear curve can be identified based on pixel coordinate values from at least two of the video frames (where the at least two video frames include first and second video frames). Interpolated video frames can be generated by applying non-linear interpolation based on the non-linear curve, and the interpolated video frames can be output for presentation. Among other things, these techniques incorporate the use of bi-directional optical flows that are consistent with one another, which helps to avoid problems associated with forward and backward optical flows being inconsistent (which yields bad video synthesizing results). These techniques also support the use of multi-frame-based intermediate flow interpolation, which provides for improving smoothness of temporal video sequences. In addition, these techniques may support the use of optical flow post-processing with normalized dilation, which may be used to improve the quality of generated interpolated video frames (such as by reducing distortion artifacts and erosion artifacts of foreground and background pixels).

Note that while the functionality of this disclosure is often described with respect to use in mobile devices, such as to support slow motion or ultra-slow motion video playback, this functionality may be used in any other suitable devices and for any other suitable purposes. For instance, this functionality may be used to support high frame rate conversion in electronic devices like smart televisions, such as to convert videos from 60 FPS to 240 FPS. This functionality may also be used to process video content of services like NETFLIX, YOUTUBE, and HULU to enhance videos from lower frame rates to higher frame rates.

FIG.1illustrates an example network configuration100including an electronic device in accordance with this disclosure. The embodiment of the network configuration100shown inFIG.1is for illustration only. Other embodiments of the network configuration100could be used without departing from the scope of this disclosure.

According to embodiments of this disclosure, an electronic device101is included in the network configuration100. The electronic device101can include at least one of a bus110, a processor120, a memory130, an input/output (I/O) interface150, a display160, a communication interface170, or a sensor180. In some embodiments, the electronic device101may exclude at least one of these components or may add at least one other component. The bus110includes a circuit for connecting the components120-180with one another and for transferring communications (such as control messages and/or data) between the components.

The processor120includes one or more of a central processing unit (CPU), an application processor (AP), or a communication processor (CP). The processor120is able to perform control on at least one of the other components of the electronic device101and/or perform an operation or data processing relating to communication. In some embodiments, the processor120can be a graphics processor unit (GPU). In some cases, the processor120can receive video frame data captured by at least one imaging sensor and process the video frame data to perform accurate optical flow interpolation optimizing bi-directional consistency and temporal smoothness as described below.

The memory130can include a volatile and/or non-volatile memory. For example, the memory130can store commands or data related to at least one other component of the electronic device101. According to embodiments of this disclosure, the memory130can store software and/or a program140. The program140includes, for example, a kernel141, middleware143, an application programming interface (API)145, and/or an application program (or “application”)147. At least a portion of the kernel141, middleware143, or API145may be denoted an operating system (OS).

The kernel141can control or manage system resources (such as the bus110, processor120, or memory130) used to perform operations or functions implemented in other programs (such as the middleware143, API145, or application147). The kernel141provides an interface that allows the middleware143, the API145, or the application147to access the individual components of the electronic device101to control or manage the system resources. The application147includes one or more applications for image processing (such as to perform accurate optical flow interpolation optimizing bi-directional consistency and temporal smoothness) as discussed below. These functions can be performed by a single application or by multiple applications that each carries out one or more of these functions. The middleware143can function as a relay to allow the API145or the application147to communicate data with the kernel141, for instance. A plurality of applications147can be provided. The middleware143is able to control work requests received from the applications147, such as by allocating the priority of using the system resources of the electronic device101(like the bus110, the processor120, or the memory130) to at least one of the plurality of applications147. The API145is an interface allowing the application147to control functions provided from the kernel141or the middleware143. For example, the API145includes at least one interface or function (such as a command) for filing control, window control, image processing, or text control.

The I/O interface150serves as an interface that can, for example, transfer commands or data input from a user or other external devices to other component(s) of the electronic device101. The I/O interface150can also output commands or data received from other component(s) of the electronic device101to the user or the other external device.

The display160includes, for example, a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, a quantum-dot light emitting diode (QLED) display, a microelectromechanical systems (MEMS) display, or an electronic paper display. The display160can also be a depth-aware display, such as a multi-focal display. The display160is able to display, for example, various contents (such as text, images, videos, icons, or symbols) to the user. The display160can include a touchscreen and may receive, for example, a touch, gesture, proximity, or hovering input using an electronic pen or a body portion of the user.

The communication interface170, for example, is able to set up communication between the electronic device101and an external electronic device (such as a first electronic device102, a second electronic device104, or a server106). For example, the communication interface170can be connected with a network162or164through wireless or wired communication to communicate with the external electronic device. The communication interface170can be a wired or wireless transceiver or any other component for transmitting and receiving signals, such as images.

The electronic device101further includes one or more sensors180that can meter a physical quantity or detect an activation state of the electronic device101and convert metered or detected information into an electrical signal. For example, one or more sensors180include one or more cameras or other imaging sensors, which may be used to capture images of scenes (such as still image frames and sequences of video image frames). The sensor(s)180can also include one or more buttons for touch input, one or more microphones, a gesture sensor, a gyroscope or gyro sensor, an air pressure sensor, a magnetic sensor or magnetometer, an acceleration sensor or accelerometer, a grip sensor, a proximity sensor, a color sensor (such as an RGB sensor), a bio-physical sensor, a temperature sensor, a humidity sensor, an illumination sensor, an ultraviolet (UV) sensor, an electromyography (EMG) sensor, an electroencephalogram (EEG) sensor, an electrocardiogram (ECG) sensor, an infrared (IR) sensor, an ultrasound sensor, an iris sensor, or a fingerprint sensor. The sensor(s)180can further include an inertial measurement unit, which can include one or more accelerometers, gyroscopes, and other components. In addition, the sensor(s)180can include a control circuit for controlling at least one of the sensors included here. Any of these sensor(s)180can be located within the electronic device101.

The first external electronic device102or the second external electronic device104can be a wearable device or an electronic device-mountable wearable device (such as an HMD). When the electronic device101is mounted in the electronic device102(such as the HMD), the electronic device101can communicate with the electronic device102through the communication interface170. The electronic device101can be directly connected with the electronic device102to communicate with the electronic device102without involving with a separate network. The electronic device101can also be an augmented reality wearable device, such as eyeglasses, that include one or more cameras.

The wireless communication is able to use at least one of, for example, long term evolution (LTE), long term evolution-advanced (LTE-A), 5th generation wireless system (5G), millimeter-wave or 60 GHz wireless communication, Wireless USB, code division multiple access (CDMA), wideband code division multiple access (WCDMA), universal mobile telecommunication system (UMTS), wireless broadband (WiBro), or global system for mobile communication (GSM), as a cellular communication protocol. The wired connection can include, for example, at least one of a universal serial bus (USB), high definition multimedia interface (HDMI), recommended standard 232 (RS-232), or plain old telephone service (POTS). The network162includes at least one communication network, such as a computer network (like a local area network (LAN) or wide area network (WAN)), Internet, or a telephone network.

The first and second external electronic devices102and104and server106each can be a device of the same or a different type from the electronic device101. According to certain embodiments of this disclosure, the server106includes a group of one or more servers. Also, according to certain embodiments of this disclosure, all or some of the operations executed on the electronic device101can be executed on another or multiple other electronic devices (such as the electronic devices102and104or server106). Further, according to certain embodiments of this disclosure, when the electronic device101should perform some function or service automatically or at a request, the electronic device101, instead of executing the function or service on its own or additionally, can request another device (such as electronic devices102and104or server106) to perform at least some functions associated therewith. The other electronic device (such as electronic devices102and104or server106) is able to execute the requested functions or additional functions and transfer a result of the execution to the electronic device101. The electronic device101can provide a requested function or service by processing the received result as it is or additionally. To that end, a cloud computing, distributed computing, or client-server computing technique may be used, for example. WhileFIG.1shows that the electronic device101includes the communication interface170to communicate with the external electronic device104or server106via the network162, the electronic device101may be independently operated without a separate communication function according to some embodiments of this disclosure.

The server106can include the same or similar components as the electronic device101(or a suitable subset thereof). The server106can support to drive the electronic device101by performing at least one of operations (or functions) implemented on the electronic device101. For example, the server106can include a processing module or processor that may support the processor120implemented in the electronic device101. In some cases, the server106can receive video frame data captured by at least one imaging sensor and process the video frame data to perform accurate optical flow interpolation optimizing bi-directional consistency and temporal smoothness as described below.

AlthoughFIG.1illustrates one example of a network configuration100including an electronic device101, various changes may be made toFIG.1. For example, the network configuration100could include any number of each component in any suitable arrangement. In general, computing and communication systems come in a wide variety of configurations, andFIG.1does not limit the scope of this disclosure to any particular configuration. Also, whileFIG.1illustrates one operational environment in which various features disclosed in this patent document can be used, these features could be used in any other suitable system.

FIG.2illustrates an example forward multi-frame-based intermediate flow interpolation200according to this disclosure. For ease of explanation, the flow interpolation200is described as being performed using one or more components of the electronic device101described above. However, this is merely one example, and the flow interpolation200could be performed using any other suitable device(s) and in any other suitable system(s), such as when performed using the server106.

As shown inFIG.2, the flow interpolation200involves the use of a reference video frame202and a subsequent video frame204. For example, the reference frame202and the subsequent frame204may represent video frames that are captured by a sensor180of the electronic device101. The reference frame202may be referred to as “frame N” or “optical flow N,” and the subsequent frame204may be referred to as “frame N+1” or “optical flow N+1” (which indicates that the video frames202and204are consecutive or otherwise separated in time). In the example shown, the reference frame202and the subsequent frame204show a human sprinting in the foreground.

In some embodiments, the electronic device101may detect a request for a slow-motion playback of video frames or other request that involves interpolation of video frames. In response, the electronic device101may determine a playback speed (such as 2×, 4×, or 8× slower) and generate one or more interpolated video frames206a-206g. The one or more interpolated video frames206a-206gmay be referred to as intermediate flows. The number of interpolated video frames206a-206ggenerated may be based on the determined playback speed. For instance, the electronic device101may generate K interpolated video frames206a-206g, where K is one less than the determined playback speed. In the example shown, based on the playback speed being determined as 8× slower, the electronic device101can generate seven interpolated video frames206a-206gby performing forward optical flow interpolation.

Forward optical flow interpolation is a process that estimates pixel locations of image data in a subsequent frame based on the pixel locations of the image data in an earlier frame. For example, a pixel location can be represented by coordinates (x, y) in a frame, where the frame is an array of pixels. In this disclosure, frames can be indexed such that a reference frame is referred to as frame N, the subsequent frame is referred to as frame N+1, and a previous frame is referred to as frame N−1. Thus, inFIG.2, the electronic device101receives the reference frame202and the subsequent frame204as inputs. Based on known pixel locations of image data in the frame202and in the frame204, the electronic device101estimates pixel locations for image data in the interpolated video frames206a-206gmoving forward from the frame202to the frame204. By generating the interpolated video frames206a-206g, playback of the reference frame202, the interpolated video frames206a-206g, and the subsequent frame204provides a slow-motion effect since more frames are being played at a given rate.

FIG.3illustrates an example backward multi-frame-based intermediate flow interpolation300according to this disclosure. For case of explanation, the flow interpolation300is described as being performed using one or more components of the electronic device101described above. However, this is merely one example, and the flow interpolation300could be performed using any other suitable device(s) and in any other suitable system(s), such as when performed using the server106.

As shown inFIG.3, the flow interpolation300again involves the use of the reference video frame202and the subsequent video frame204. In some embodiments, the electronic device101may detect a request for a slow-motion playback of video frames or other request that involves interpolation of video frames. In response, the electronic device101may determine a playback speed (such as 2×, 4×, or 8× slower) and generate one or more interpolated video frames306a-306g. The one or more interpolated video frames306a-306gmay be referred to as intermediate flows. Again, the number of interpolated video frames306a-306ggenerated may be based on the determined playback speed. For instance, the electronic device101may generate K interpolated video frames306a-306g, where K is one less than the determined playback speed.

Backward optical flow interpolation is a process that estimates pixel locations of image data in an earlier frame based on the pixel locations of the image data in a subsequent frame. Thus, inFIG.3, the electronic device101receives the reference frame202and the subsequent frame204as inputs. Based on known pixel locations of image data in the frame202and in the frame204, the electronic device101estimates pixel locations for image data in the interpolated video frames306a-306gmoving backward from the frame204to the frame202. By generating the interpolated video frames306a-306g, playback of the reference frame202, the interpolated video frames306a-306g, and the subsequent frame204provides a slow-motion effect since more frames are being played at a given rate.

A standard interpolation technique is to simply and iteratively copy and paste one frame into timeslots that are intermediate between that frame and a subsequent frame. However, this produces movement that is not smooth. A bi-directional interpolation technique utilizes strong guidance (a strong indicator of knowing where pixels are going) in order to make interpolation easier. Bi-directional interpolation combines forward optical flow interpolation as shown inFIG.2and backward optical flow interpolation as shown inFIG.3. The forward optical flow interpolation process estimates pixel locations in a subsequent frame based on known pixel locations in an earlier frame, and the backward optical flow interpolation process estimates pixel locations in an earlier frame based on known pixel locations in a subsequent frame.

FIG.4illustrates an example bi-directional (forward/backward) flow interpolation400according to this disclosure. As shown inFIG.4, the electronic device101obtains multiple video frames, including frames402and404(which may represent the video frames202and204discussed above). The flow interpolation400here performs forward interpolation using a forward optical flow406and performs backward interpolation using a backward optical flow408. The forward optical flow406is used to estimate pixel locations starting at the frame402and ending at the frame404, and the backward optical flow408is used to estimate pixel locations starting at the frame404and ending at the frame402.

Smoothness and sharpness are important image quality (IQ) criteria for video interpolation. Smoothness and sharpness of slow-motion video can be achieved when bi-directional consistency is met, which means that the forward optical flow406and backward optical flow408are consistent. Simply applying both forward and backward interpolation may not guarantee that the forward and backward optical flows are consistent, which can lead to poor video synthesizing results. The techniques described below help to ensure that the forward and backward optical flows are consistent, which leads to improved video synthesizing results.

Moreover, an optical flow interpolation that focuses on smoothness too much will cause a scene with a rapidly-moving object (such as a jumping dog or sprinting person) to play back in a choppy manner, in which case the rapidly-moving object looks blurry (less sharp). As a result, there can be a tradeoff between smoothness and sharpness. In some embodiments, the electronic device101may determine how much to focus on smoothness and how much to focus on sharpness based on one or more image quality criteria. In some cases, suitable smoothness and sharpness can be achieved when bi-directional consistency is met, which can be expressed as follows:
argminu0→1,u1→0∥g(I0;u0→i)−g(I1;u1→i)∥2(1)
whereu0→i=f(u0→1),u1→i=f(u1→0)
The meanings of the variables shown in Equation (1) are described below with reference toFIG.6.

FIG.5illustrates an example forward/backward multi-frame-based intermediate flow interpolation500according to this disclosure. For ease of explanation, the flow interpolation500is described as being performed using one or more components of the electronic device101described above. However, this is merely one example, and the flow interpolation500could be performed using any other suitable device(s) and in any other suitable system(s), such as when performed using the server106.

In this example, forward and backward optical flows are used to support the interpolation of three interpolated frames510,520, and530between the video frames402and404. This may allow, for instance, the electronic device101to support a 4× slow motion playback. Of course, other numbers of interpolated frames may be produced between the video frames402and404to support other slow motion playback speeds. In this example, the interpolated frames510,520, and530are generated using forward optical flows540,542,544, and546and backward optical flows550,552,554, and556. The forward optical flows540,542,544, and546ofFIG.5can be the same as or similar to the optical flows used inFIG.2, and the backward optical flows550,552,554, and556ofFIG.5can be the same as or similar to the optical flows used inFIG.3. These optical flows can be consistent as described below so that the interpolated frames510,520, and530appear much more natural and avoid many of the issues with prior approaches.

FIG.6illustrates an example determination of whether forward and backward optical flows are consistent or inconsistent according to this disclosure. For ease of explanation, this determination is described as being made using one or more components of the electronic device101described above. However, this is merely one example, and the determination could be made using any other suitable device(s) and in any other suitable system(s), such as when made by the server106.

As shown inFIG.6, the electronic device101is attempting to determine whether a forward optical flow610from the frame402to the interpolated frame520is matched or consistent with a backward optical flow620from the frame404to the interpolated frame520. For example, the electronic device101may determine if the forward optical flow610(which can define changes of pixel locations from the frame402to the interpolated frame520) results in the same or similar results as the backward optical flow620(which can define changes of pixel locations from the frame404to the interpolated frame520). If the forward and backward optical flows610and620do not match, this may be indicated by significant pixel location differences determined using the optical flows610and620.

Note that, in some embodiments, the electronic device101optimizes bi-directional optical flow consistency using a cost function. The cost function may accumulate costs determined for different pixels in the forward and backward directions, and the optical flows610and620can be adjusted to try to minimize the overall cost. This helps to make the optical flows610and620more consistent with one another. Using this or another approach, the electronic device101can help to prevent single-directional bias of flow vectors when identifying intermediate (interpolated) optical flows, prevent choppy transitions of interpolated video frames, and improve the temporal smoothness of interpolated optical flow maps and interpolated video frames.

In some cases, the forward optical flow610may be represented in the following manner:
g(I0;u0→i)  (2)

Here, I0represents an image602of the reference frame, u0→irepresents the forward optical flow610, and i represents a common intermediate coordinate. Similarly, in some cases, the backward optical flow620may be represented in the following manner:
g(I1;u1→i)  (3)
Here, I1represents an image604of the subsequent frame, and u1→irepresents the backward optical flow620. As described above, Equation (1) can be used to minimize the error of computing the bi-directional optical flows610and620at the common intermediate coordinate i.

FIG.7illustrates an example multi-frame-based intermediate flow interpolation system700according to this disclosure. For ease of explanation, the interpolation system700is described as being implemented using one or more components of the electronic device101described above. However, this is merely one example, and the interpolation system700could be implemented using any other suitable device(s) and in any other suitable system(s), such as when performed using the server106.

As shown inFIG.7, the interpolation system700improves slow-motion or other interpolated image quality, such as in areas of frames where non-linear motion occurs between consecutive frames. In this example, the interpolation system700includes a three-flow alignment and post-processing function710, an equation solver720, and a non-linear interpolation function730. The three-flow alignment and post-processing function710generally operates to align optical flows associated with input video frames740and fill in holes or other artifacts in the optical flows. Example operations of the alignment and post-processing function710are described below with reference toFIG.8.

The equation solver720generally operates to identify linear, quadratic, cubic, or higher-order polynomial curves that define how the positions of pixels change between different video frames740. For example, the equation solver720may receive data identifying how coordinates of certain pixels change between the captured video frames740, and the equation solver720can perform a curve-fitting algorithm or other algorithm to generate one or more equations to represent how pixels move between the frames. Example operations of the equation solver720are described below with reference toFIG.9.

The non-linear interpolation function730uses the one or more equations generated by the equation solver720to produce one or more interpolated frames750. For example, the non-linear interpolation function730can use the one or more equations to estimate where various pixels contained in the video frames740would be located at one or more times between the capture of one video frame740and the capture of another video frame740. These locations for the pixels can be used to generate the one or more interpolated frames750. Example operations of the non-linear interpolation function730are also described below with reference toFIG.9.

FIG.8illustrates an example alignment operation800as part of a forward/backward multi-frame-based intermediate flow interpolation according to this disclosure. For ease of explanation, the alignment operation800is described as being performed using one or more components of the electronic device101described above. However, this is merely one example, and the alignment operation800could be performed using any other suitable device(s) and in any other suitable system(s), such as when performed using the server106.

The alignment operation800is generally used to improve slow-motion or other interpolated image quality, such as in areas of frames where non-linear motion (like a jumping dog) occurs between neighboring frames. Examples of non-linear motion include rotation, morphological deformation, or abrupt flow direction changes. Using prior approaches, it is challenging to interpolate fast motion between a frame802and an earlier frame804or between the frame802and a subsequent frame804, such as due to erosion artifacts. Also, when interpolation is applied to a fast-moving object captured in video, the fast-moving object tends to look smaller/shrunken compared to the actual size of the object. As an example of erosion artifacts, holes where pixels are missing can be seen in a visualization of an optical flow816from a previous frame N−1 to a current frame N, a visualization of an optical flow812from the current frame N to a subsequent frame N+1, and a visualization of an optical flow816from the subsequent frame N+1 to a next subsequent frame N+2. Such holes can be positioned in various locations throughout the optical flows812,814, and816or can be concentrated around a rapidly-moving object818.

In the example shown here, the electronic device101performs the alignment operation800on a set801of frames, namely the frames802,804, and806. In some cases, the alignment operation800can be considered a three-flow alignment operation, where alignment per frame is performed. By performing the alignment operation800, the electronic device101is able to reduce erosion artifacts in the optical flows812,814, and816. The optical flows812,814, and816provide examples of forward optical flows, but it should be understood that the alignment operation800is able to reduce erosion artifacts on backward optical flows that are estimated based on the same set of frames upon which the optical flows812,814, and816are estimated. After alignment has been corrected, the electronic device101proceeds to perform one or more post-processing operations820. The post-processing operations820may include hole-filling or smoothing of any remaining erosion artifacts or other artifacts in the optical flows812,814, and816. The post-processing operations820here can occur or be applied to three estimated bi-directional optical flows before operation of the equation solver720is applied to the three estimated bi-directional optical flows.

FIG.9illustrates an example pixel-wise fitting and frame interpolation operation according to this disclosure. For ease of explanation, the pixel-wise fitting and frame interpolation operation is described as being performed using one or more components of the electronic device101described above. However, this is merely one example, and the pixel-wise fitting and frame interpolation operation could be performed using any other suitable device(s) and in any other suitable system(s), such as when performed using the server106.

As shown inFIG.9, the electronic device101may include or use a quadratic equation solver (QES) or other multi-order polynomial solver. As noted above, the pixel location of each pixel of a frame has an x-coordinate and a y-coordinate, which can be expressed as (x, y). The solver may operate to generate a graph900of a flow curve902, where the horizontal axis of the graph900represents a frame number relative to the reference frame and the vertical axis of the graph900represents the value of the x-coordinate or the value of the y-coordinate of pixels. On the horizontal axis, the reference frame is represented by a value of 0, the subsequent frame is represented by a value of +1, the frame after that is represented by a value of +2, and the previous frame is represented by a value of −1. Interpolated frames may be represented using fractional values, such as 0.25, 0.5, and 0.75. Effectively, the graph900represents where different pixels of the reference frame may be located in other frames. Note that while a quadratic equation solver (which can be used to solve a quadratic equation representing optical flow) is described here, other solvers may be used, such as solvers used to solve cubic flow curves or other multi-order polynomial curves representing optical flow. To avoid duplication, the discussion below may focus on the use of y coordinates in the graph900, although analogous operations may be performed for the x coordinates.

To generate the flow curve902for a particular pixel, the pixel location (x, y) of that particular pixel can be identified in the various frames, and the respective values of its y coordinate are plotted (such as at points904,906, and908). The solver here can solve for a quadratic equation f( ) that approximately represents the flow curve902that fits the pixel movements represented by the points904,906, and908. Again, note that the use of a quadratic curve is for illustration only and that other curves may be used.

Using the determined flow curve902, the electronic device101may perform interpolations to generate one or more interpolated frames between two or more input frames. For example, in the case of a playback speed of 4×, the electronic device101generates three interpolated intermediate frames, which may be denoted as frame N+0.25, frame N+0.5, and frame N+0.75. The electronic device101can determine the y coordinate values of the interpolated intermediate frame N+0.25 by plotting a non-linear interpolation point910where the flow curve902intersects the corresponding frame value of 0.25. In a similar manner, non-linear interpolation points912,914, and916can be plotted on the flow curve902to represent the determined y coordinate values of frame N+0.5, frame N+0.75, and frame N−0.5, respectively. In some cases, the electronic device101may perform interpolations using a quadratic flow curve902.

Again, note that the electronic device101is not limited to performing pixel-wise fitting and frame interpolation using a quadratic flow curve. In other embodiments, other non-linear curves can be applied, such as a cubic flow curve or other higher-order polynomial curve. Also, in some embodiments, the electronic device101may utilize linear curves918aand918bthat respectively correspond to linear relationships between frame N and frame N−1 and between frame N and frame N+1. Linear interpolation points920and922represent y coordinate values of frame N−0.5 and frame N+0.5, respectively. Note that a higher-order curve, such as a cubic flow curve, may produce a smoother video than the quadratic flow curve902. A lower-order curve, such as the linear flow curves918aand918b, may produce a video that is less smooth than the quadratic flow curve902but which can be computed more quickly.

FIG.10illustrates an example normalized dilation operation1000according to this disclosure. For ease of explanation, the operation1000is described as being performed using one or more components of the electronic device101described above. However, this is merely one example, and the operation1000could be performed using any other suitable device(s) and in any other suitable system(s), such as when performed using the server106.

Although optical flow interpolation/estimation can yield highly-accurate forward and backward optical flow results, the optical flow is not always perfectly accurate, and motion boundaries of optical flow maps are often not strictly aligned with input frames (such as the frames202and204ofFIGS.2and3, the frames402and404ofFIG.4, or the frames802-806ofFIG.8). Erosion artifacts occur when a flow boundary is inside of motion boundary, which is a condition that often causes noticeable boundary artifacts within a specified object in a frame. The specified object here may refer to a fast-moving object or an object in the foreground. To help compensate for possible imperfect optical flow boundaries for video interpolation, the electronic device101can perform the operation1000.

As shown inFIG.10, in operation1010, the electronic device101detects a main object in received input frames202and204. The main object represents the object where the largest motion is happening between the input frames. In some embodiments, AI/ML-based optical flow maps may be used to more accurately wrap (or enclose a boundary around) the main object. In some cases, the main object may be a fast-moving object. In operation1020, the electronic device101determines absolute mean flows of a main object blob and a background blob. In some embodiments, the electronic device101separates the background of the input images from the main object, such as by using a layer separation technique in combination with calculating a motion difference. The absolute mean flow of the main object blob may represent the general direction of movement of the main object in the frames, and the absolute mean flow of the background blob may represent the general direction of movement of the background in the frames. In operation1030, the electronic device101applies a normalized dilation of pixels so that the flow boundary of the main object will fully wrap the motion boundary of the main object. Dilation generally involves expanding pixels around a fast-moving object (such as a boundary of a car) so that, in interpolated frames, the fast-moving object appears to be normal-sized or have a size that is suitably close to normal (rather than shrunken). In some embodiments, operation1030includes applying the process ofFIG.11.

FIG.11illustrates an example normalized dilation operation1100according to this disclosure. For ease of explanation, the operation1100is described as being performed using one or more components of the electronic device101described above. However, this is merely one example, and the operation1100could be performed using any other suitable device(s) and in any other suitable system(s), such as when performed using the server106.

Incorrect flow vectors for motion boundaries often yield noticeable boundary artifacts for main objects in frames. To compensate for this, the electronic device101can utilize a normalized dilation to process optical flow maps. For example, the electronic device101may process a raw optical flow map and cluster pixels by applying a clustering algorithm (such as a K-means clustering algorithm). After detecting the layer with the largest motion, the electronic device101applies image dilation to its flow vectors so that the layer with the largest motion can fully wrap the motion boundaries of an object. Each (x, y) component of a flow vector can be separately dilated, and the results can be concatenated afterwards. For example, the x axis and the y axis may undergo separate normalized dilation operations.

In operation1110, the electronic device101normalizes a flow vector into a range of zero to one from an original range. For example, the electronic device101may normalize each of a forward flow vector UFand a backward flow vector UBinto the range of zero to one. Here, optical flow may be defined as pixel displacement between neighboring frames, so the entire frame's optical flow can be normalized in this range. In operation1120, the electronic device101performs dilation, such as circular dilation, of flow pixels. For example, for optical flow values around the boundary of a fast-moving object, a circular or other dilation kernel can be applied to expand or increase pixel sizes. In operation1130, the electronic device101de-normalizes the flow vector or converts the flow vector back to its original range. For example, the electronic device101may convert the forward flow vector UFand the backward flow vector UBback to their original flow ranges. De-normalizing enables the electronic device101to use the same dilation kernel size for each frame and scene. This makes the entire post-processing less complex. In some embodiments, because optical flows of the boundaries of fast-moving objects have been increased, the electronic device101may normalizes again (by repeating operations1110-1130) so that the final optical flow range is between one and zero and then de-normalized back to the original range.

FIG.12illustrates an example method1200for accurate optical flow interpolation optimizing bi-directional consistency and temporal smoothness according to this disclosure. For ease of explanation, the method1200is described as being performed using one or more components of the electronic device101described above. However, this is merely one example, and the method1200could be performed using any other suitable device(s) and in any other suitable system(s), such as when performed using the server106.

As shown inFIG.12, in operation1202, the processor120receives inputs that include multiple video frames. In this example, the processor120receives three video frames. Note, however, that the processor120may receive two video frames or more than three video frames. In operation1204, the processor120performs alignment and post-processing of the video frames. This may include, for example, the processor120performing the alignment operation800and the one or more post-processing operations820.

In operation1206, the processor120utilizes a solver (such as a QES) to generate one or more non-linear curves that fit the pixel values of the video frames. In operation1208, the processor120generates one or more interpolated video frames by applying non-linear interpolation using the non-linear curve(s). In operation1210, the processor120may store, output, or use the interpolated video frames. In some embodiments, the interpolated video frames may be provided to a display device, such as the display160.

It should be noted that the functions shown in or described with respect toFIGS.2through12can be implemented in an electronic device101, server106, or other device in any suitable manner. For example, in some embodiments, at least some of the functions shown in or described with respect toFIGS.2through12can be implemented or supported using one or more software applications or other software instructions that are executed by the processor120of the electronic device101, server106, or other device. In other embodiments, at least some of the functions shown in or described with respect toFIGS.2through12can be implemented or supported using dedicated hardware components. In general, the functions shown in or described with respect toFIGS.2through12can be performed using any suitable hardware or any suitable combination of hardware and software/firmware instructions.

AlthoughFIGS.2through12have illustrated various examples of features and functions used to provide accurate optical flow interpolation optimizing bi-directional consistency and temporal smoothness, various changes may be made toFIGS.2through12. For example, various functions described above may be combined, further subdivided, replicated, omitted, or rearranged and additional functions may be added according to particular needs. Also, various images and graphs are shown to illustrate example types of information that may be received or generated. However, images of scenes can vary widely, and the images received or generated and the graphs associated with the images can also vary widely. In addition, whileFIG.12shows a series of steps, various steps inFIG.12could overlap, occur in parallel, occur in a different order, or occur multiple times.

Although this disclosure has been described with reference to various example embodiments, various changes and modifications may be suggested to one skilled in the art. It is intended that this disclosure encompass such changes and modifications as fall within the scope of the appended claims.