Patent Description:
<NPL> proposes to address the two tasks of learning to estimate 3D geometry in a single frame and optical flow from consecutive frames by watching unlabeled videos as a whole, i.e., to jointly understand per-pixel 3D geometry and motion. This eliminates the need of static scene assumption and enforces the inherent geometrical consistency during the learning process, yielding significantly improved results for both tasks.

<NPL> presents GLNet, a self-supervised framework for learning depth, optical flow, camera pose and intrinsic parameters from monocular video - addressing the difficulty of acquiring realistic ground-truth for such tasks.

A depth model may be pre-trained to generate depth images based on monocular images. The pre-trained depth model may subsequently be fine-tuned to generate, based on monocular images of a video, corresponding depth images that are geometrically and temporally consistent with one another. Fine-tuning of the depth model may be facilitated by a scene flow model configured to generate a scene flow that represents movements over time of 3D points in a scene represented by the video. The 3D points may be generated based on corresponding depth values of the depth images. A depth loss function may be configured to quantify, for a given pair of a first image and a second image, a 3D consistency between (i) post-flow 3D points resulting from displacement of 3D points of the first image according to the corresponding scene flow and (ii) corresponding 3D points of the second image. A pixel flow loss function may be configured to quantify a 2D consistency between (i) induced pixel positions associated with the post-flow 3D points and (ii) corresponding pixel positions indicated by an optical flow between the first image and the second image. Parameters of the scene flow model and the depth model may be adjusted until the generated depth images exhibit a desired level of 3D and/or 2D consistency, as measured by the respective loss functions.

In a first example embodiment, a method includes obtaining a first image from a video, a second image from the video, and an optical flow between the first image and the second image. The method may also include determining that a second pixel of the second image corresponds to a displacement of a first pixel of the first image to the second image according to the optical flow. The method may additionally include determining, by a depth model, (i) a first depth image based on the first image and (ii) a second depth image based on the second image, and determining (i), based on the first depth image, a first depth associated with the first pixel and (ii), based on the second depth image, a second depth associated with the second pixel. The method may yet additionally include determining (i) a first three-dimensional (3D) point based on the first depth associated with the first pixel and (ii) a second 3D point based on the second depth associated with the second pixel. The method may further include determining, by providing as input, to a scene flow model that is specific to the video and comprises a machine learning model, (i) the first 3D point and (ii) a time value associated with the first depth image, a scene flow representing a 3D motion of the first 3D point between the first image and the second image, and determining, for the first pixel, an induced pixel position based on a post-flow 3D point that represents the first 3D point after a displacement according to the scene flow. The method may yet further include determining (i) a pixel flow loss value based on a comparison of the induced pixel position to a position of the second pixel and (ii) a depth loss value based on a comparison of the post-flow 3D point to the second 3D point, and adjusting one or more parameters of one or more of the depth model or the scene flow model based on the pixel flow loss value and the depth loss value.

In a second example embodiment, a system includes a processor and a non-transitory computer-readable medium having stored thereon instructions that, when executed by the processor, cause the processor to perform operations in accordance with the first example embodiment.

In a third example embodiments, a non-transitory computer-readable medium has stored thereon instructions that, when executed by a computing device, cause the computing device to perform operations in accordance with the first example embodiment.

These, as well as other embodiments, aspects, advantages, and alternatives, will become apparent to those of ordinary skill in the art by reading the following detailed description, with reference where appropriate to the accompanying drawings. Further, this summary and other descriptions and figures provided herein are intended to illustrate embodiments by way of example only and, as such, that numerous variations are possible. For instance, structural elements and process steps can be rearranged, combined, distributed, eliminated, or otherwise changed, while remaining within the scope of the embodiments as claimed.

Example methods, devices, and systems are described herein. It should be understood that the words "example" and "exemplary" are used herein to mean "serving as an example, instance, or illustration. " Any embodiment or feature described herein as being an "example," "exemplary," and/or "illustrative" is not necessarily to be construed as preferred or advantageous over other embodiments or features unless stated as such. Thus, other embodiments can be utilized and other changes can be made without departing from the scope of the subject matter presented herein.

Accordingly, the example embodiments described herein are not meant to be limiting. It will be readily understood that the aspects of the present disclosure, as generally described herein, and illustrated in the figures, can be arranged, substituted, combined, separated, and designed in a wide variety of different configurations.

Unless otherwise noted, figures are not drawn to scale.

Estimating the geometry of objects in a scene based on monoscopic videos is an under-constrained problem, especially when both the camera and the objects are moving. Specifically, for any moving object, there may be multiple object geometries that satisfy the visual evidence present within the video. For example, an object could be relatively far away from the camera and moving fast, or the object could be relatively close to the camera and moving slowly. Machine learning-based techniques may be configured to solve this problem by learning aspects of the object's motion and/or shape based on training data. For example, a machine learning model may be trained to determine depth images based on monoscopic color images. Such machine learning models may be particularly useful in cases where epipolar geometry constraints are invalid and/or when triangulation-based methods are inapplicable (e.g., for monoscopic videos containing both object and camera motion).

However, some machine learning approaches to depth estimation may be dependent on availability of training data, which might be unavailable for certain types of objects and/or contexts, or which may be inaccurate. Additionally, machine learning models that consider images of a video individually, or consider only a relatively short-range window of images, may generate depth images that include inconsistent depth and/or flickering/jittering over time. Thus, if such depth images are used to apply depth-based effects to the video, the depth inconsistencies and/or depth flickering/jitter may cause commensurate inconsistencies and/or flickering/jitter in the depth-based effects, resulting in these depth-based effects appearing erroneous, inaccurate, and/or otherwise unpleasing to an observer (e.g., a human viewer).

Accordingly, provided herein is a hybrid approach to depth determination in which a pre-trained machine learning model is fine-tuned on a per-video basis to generate geometrically and temporally consistent depth images for a specific video. Specifically, a depth system configured to determine depth images based on images of a video may include a depth model and a scene flow model. The depth model may have been pre-trained to generate depth images based on a training data set that includes pairs of monoscopic images and corresponding ground-truth depth information. The scene flow model may be configured to model a 3D motion of objects in a scene represented by the video, and may thus facilitate depth determination for dynamic (i.e., moving relative to a world reference frame) parts of the scene.

In order to fine-tune the depth model and the scene flow model to a particular video, the depth system may be configured to process and compare different pairs of images from the video. These image pairs may include images separated by various inter-frame distances ranging from, for example, a pair of consecutive images to a pair of images separated by seven intermediate images. By performing the comparison at various different inter-frame distances, the depth system may quantify, and thus improve, depth consistency among various different time scales. For a given pair of images of the video, a pixel correspondence between a first image of the given pair and a second image of the given pair may be determined based on an optical flow between these images.

For a given pair of images of the video, the depth model may be configured to determine a first depth image and a second depth image. Based on these depth images, the depth system may also be configured to unproject at least some of the pixels of each image to generate corresponding 3D points. Specifically, a first set of 3D points may represent, in a world reference frame, the depth associated with pixels of a first image of the given pair, and a second set of 3D points may represent, in the world reference frame, the depth associated with pixels of the second image of the given pair.

The first set of 3D points may be processed by the scene flow model to determine respective displacements of the first set of 3D points from a time associated with the first image to a time associated with the second image. Thus, the scene flow model may provide an explicit 3D representation of motion of features of the scene, may aggregate information over time and space through the parameters of the scene flow model, and may generate plausible scene flows in cases where an analytic solution based on depth and optical flow might be unstable (e.g., nearly parallel rays between two points). Additionally, the scene flow model may be applied recursively to generate a scene flow between non-consecutive time points corresponding to non-consecutive image pairs. The scene flow may be added to the first set of 3D points, thus determining a corresponding set of post-flow 3D points. A depth loss function may be configured to compare each respective 3D point of the post-flow 3D points to a corresponding 3D point of the second set of 3D points, and thereby quantify, by way of a depth loss value, a 3D geometric and temporal consistency between the first depth image and the second depth image.

Additionally, the set of post-flow 3D points may be projected into image space of the second image to determine induced pixel positions associated with the post-flow 3D points (resulting from displacement of the first set of 3D points by the optical flow). A pixel flow loss function may be configured to compare each respective induced pixel position of the induced pixel positions to a corresponding pixel position determined based on the optical flow between the first and second images, and thereby quantify, by way of a pixel flow loss value, a 2D/pixel-space geometric and temporal consistency between the first depth image and the second depth image. Additional loss functions may be used to generate corresponding loss values that quantify performance of the scene flow model and the depth model in other ways. A total loss value for a given training iteration may be based on a sum of the corresponding loss values of a plurality of different image pairs of the video.

Based on the total loss value, the depth system may be configured to adjust parameters of the depth model and/or the scene flow model. For example, the parameters may be adjusted based on a gradient of the loss functions, such that the total loss value of a subsequent training iteration is reduced. Such training iterations may be repeated, for example, until the total loss value is reduced below a threshold loss value. After training, the depth model may be used to generate a plurality of depth images corresponding to images of the video. As a result of the training, these depth images may be relatively more geometrically and temporally consistent with one another and with the scene represented thereby, and such consistency may be present in both the static and moving regions. Accordingly, depth-based effects applied to the video may appear geometrically and temporally consistent, and thus visually coherent and/or pleasing to an observer.

<FIG> illustrates an example computing device <NUM>. Computing device <NUM> is shown in the form factor of a mobile phone. However, computing device <NUM> may be alternatively implemented as a laptop computer, a tablet computer, and/or a wearable computing device, among other possibilities. Computing device <NUM> may include various elements, such as body <NUM>, display <NUM>, and buttons <NUM> and <NUM>. Computing device <NUM> may further include one or more cameras, such as front-facing camera <NUM> and rear-facing camera <NUM>.

Front-facing camera <NUM> may be positioned on a side of body <NUM> typically facing a user while in operation (e.g., on the same side as display <NUM>). Rear-facing camera <NUM> may be positioned on a side of body <NUM> opposite front-facing camera <NUM>. Referring to the cameras as front and rear facing is arbitrary, and computing device <NUM> may include multiple cameras positioned on various sides of body <NUM>.

Display <NUM> could represent a cathode ray tube (CRT) display, a light emitting diode (LED) display, a liquid crystal (LCD) display, a plasma display, an organic light emitting diode (OLED) display, or any other type of display known in the art. In some examples, display <NUM> may display a digital representation of the current image being captured by front-facing camera <NUM> and/or rear-facing camera <NUM>, an image that could be captured by one or more of these cameras, an image that was recently captured by one or more of these cameras, and/or a modified version of one or more of these images. Thus, display <NUM> may serve as a viewfinder for the cameras. Display <NUM> may also support touchscreen functions that may be able to adjust the settings and/or configuration of one or more aspects of computing device <NUM>.

Front-facing camera <NUM> may include an image sensor and associated optical elements such as lenses. Front-facing camera <NUM> may offer zoom capabilities or could have a fixed focal length. In other examples, interchangeable lenses could be used with front-facing camera <NUM>. Front-facing camera <NUM> may have a variable mechanical aperture and a mechanical and/or electronic shutter. Front-facing camera <NUM> also could be configured to capture still images, video images, or both. Further, front-facing camera <NUM> could represent, for example, a monoscopic, stereoscopic, or multiscopic camera. Rear-facing camera <NUM> may be similarly or differently arranged. Additionally, one or more of front-facing camera <NUM> and/or rear-facing camera <NUM> may be an array of one or more cameras.

One or more of front-facing camera <NUM> and/or rear-facing camera <NUM> may include or be associated with an illumination component that provides a light field to illuminate a target object. For instance, an illumination component could provide flash or constant illumination of the target object. An illumination component could also be configured to provide a light field that includes one or more of structured light, polarized light, and light with specific spectral content. Other types of light fields known and used to recover three-dimensional (3D) models from an object are possible within the context of the examples herein.

Computing device <NUM> may also include an ambient light sensor that may continuously or from time to time determine the ambient brightness of a scene that cameras <NUM> and/or <NUM> can capture. In some implementations, the ambient light sensor can be used to adjust the display brightness of display <NUM>. Additionally, the ambient light sensor may be used to determine an exposure length of one or more of cameras <NUM> or <NUM>, or to help in this determination.

Computing device <NUM> could be configured to use display <NUM> and front-facing camera <NUM> and/or rear-facing camera <NUM> to capture images of a target object. The captured images could be a plurality of still images or a video stream. The image capture could be triggered by activating button <NUM>, pressing a softkey on display <NUM>, or by some other mechanism. Depending upon the implementation, the images could be captured automatically at a specific time interval, for example, upon pressing button <NUM>, upon appropriate lighting conditions of the target object, upon moving digital camera device <NUM> a predetermined distance, or according to a predetermined capture schedule.

<FIG> is a simplified block diagram showing some of the components of an example computing system <NUM>. By way of example and without limitation, computing system <NUM> may be a cellular mobile telephone (e.g., a smartphone), a computer (such as a desktop, notebook, tablet, or handheld computer), a home automation component, a digital video recorder (DVR), a digital television, a remote control, a wearable computing device, a gaming console, a robotic device, a vehicle, or some other type of device. Computing system <NUM> may represent, for example, aspects of computing device <NUM>.

As shown in <FIG>, computing system <NUM> may include communication interface <NUM>, user interface <NUM>, processor <NUM>, data storage <NUM>, and camera components <NUM>, all of which may be communicatively linked together by a system bus, network, or other connection mechanism <NUM>. Computing system <NUM> may be equipped with at least some image capture and/or image processing capabilities. It should be understood that computing system <NUM> may represent a physical image processing system, a particular physical hardware platform on which an image sensing and/or processing application operates in software, or other combinations of hardware and software that are configured to carry out image capture and/or processing functions.

Communication interface <NUM> may allow computing system <NUM> to communicate, using analog or digital modulation, with other devices, access networks, and/or transport networks. Thus, communication interface <NUM> may facilitate circuit-switched and/or packet-switched communication, such as plain old telephone service (POTS) communication and/or Internet protocol (IP) or other packetized communication. For instance, communication interface <NUM> may include a chipset and antenna arranged for wireless communication with a radio access network or an access point. Also, communication interface <NUM> may take the form of or include a wireline interface, such as an Ethernet, Universal Serial Bus (USB), or High-Definition Multimedia Interface (HDMI) port. Communication interface <NUM> may also take the form of or include a wireless interface, such as a Wi-Fi, BLUETOOTH®, global positioning system (GPS), or wide-area wireless interface (e.g., WiMAX or 3GPP Long-Term Evolution (LTE)). However, other forms of physical layer interfaces and other types of standard or proprietary communication protocols may be used over communication interface <NUM>. Furthermore, communication interface <NUM> may comprise multiple physical communication interfaces (e.g., a Wi-Fi interface, a BLUETOOTH® interface, and a wide-area wireless interface).

User interface <NUM> may function to allow computing system <NUM> to interact with a human or non-human user, such as to receive input from a user and to provide output to the user. Thus, user interface <NUM> may include input components such as a keypad, keyboard, touch-sensitive panel, computer mouse, trackball, joystick, microphone, and so on. User interface <NUM> may also include one or more output components such as a display screen, which, for example, may be combined with a touch-sensitive panel. The display screen may be based on CRT, LCD, and/or LED technologies, or other technologies now known or later developed. User interface <NUM> may also be configured to generate audible output(s), via a speaker, speaker jack, audio output port, audio output device, earphones, and/or other similar devices. User interface <NUM> may also be configured to receive and/or capture audible utterance(s), noise(s), and/or signal(s) by way of a microphone and/or other similar devices.

In some examples, user interface <NUM> may include a display that serves as a viewfinder for still camera and/or video camera functions supported by computing system <NUM>. Additionally, user interface <NUM> may include one or more buttons, switches, knobs, and/or dials that facilitate the configuration and focusing of a camera function and the capturing of images. It may be possible that some or all of these buttons, switches, knobs, and/or dials are implemented by way of a touch-sensitive panel.

Processor <NUM> may comprise one or more general purpose processors - e.g., microprocessors - and/or one or more special purpose processors - e.g., digital signal processors (DSPs), graphics processing units (GPUs), floating point units (FPUs), network processors, or application-specific integrated circuits (ASICs). In some instances, special purpose processors may be capable of image processing, image alignment, and merging images, among other possibilities. Data storage <NUM> may include one or more volatile and/or non-volatile storage components, such as magnetic, optical, flash, or organic storage, and may be integrated in whole or in part with processor <NUM>. Data storage <NUM> may include removable and/or non-removable components.

Processor <NUM> may be capable of executing program instructions <NUM> (e.g., compiled or non-compiled program logic and/or machine code) stored in data storage <NUM> to carry out the various functions described herein. Therefore, data storage <NUM> may include a non-transitory computer-readable medium, having stored thereon program instructions that, upon execution by computing system <NUM>, cause computing system <NUM> to carry out any of the methods, processes, or operations disclosed in this specification and/or the accompanying drawings. The execution of program instructions <NUM> by processor <NUM> may result in processor <NUM> using data <NUM>.

By way of example, program instructions <NUM> may include an operating system <NUM> (e.g., an operating system kernel, device driver(s), and/or other modules) and one or more application programs <NUM> (e.g., camera functions, address book, email, web browsing, social networking, audio-to-text functions, text translation functions, and/or gaming applications) installed on computing system <NUM>. Similarly, data <NUM> may include operating system data <NUM> and application data <NUM>. Operating system data <NUM> may be accessible primarily to operating system <NUM>, and application data <NUM> may be accessible primarily to one or more of application programs <NUM>. Application data <NUM> may be arranged in a file system that is visible to or hidden from a user of computing system <NUM>.

Application programs <NUM> may communicate with operating system <NUM> through one or more application programming interfaces (APIs). These APIs may facilitate, for instance, application programs <NUM> reading and/or writing application data <NUM>, transmitting or receiving information via communication interface <NUM>, receiving and/or displaying information on user interface <NUM>, and so on.

In some cases, application programs <NUM> may be referred to as "apps" for short. Additionally, application programs <NUM> may be downloadable to computing system <NUM> through one or more online application stores or application markets. However, application programs can also be installed on computing system <NUM> in other ways, such as via a web browser or through a physical interface (e.g., a USB port) on computing system <NUM>.

Camera components <NUM> may include, but are not limited to, an aperture, shutter, recording surface (e.g., photographic film and/or an image sensor), lens, shutter button, infrared projectors, and/or visible-light projectors. Camera components <NUM> may include components configured for capturing of images in the visible-light spectrum (e.g., electromagnetic radiation having a wavelength of <NUM> - <NUM> nanometers) and/or components configured for capturing of images in the infrared light spectrum (e.g., electromagnetic radiation having a wavelength of <NUM> nanometers - <NUM> millimeter), among other possibilities. Camera components <NUM> may be controlled at least in part by software executed by processor <NUM>.

<FIG> illustrates an example system for determining geometrically and temporally consistent depth images based on images of a video. Specifically, depth system <NUM> may include scene flow model <NUM>, 3D point projector <NUM>, depth model <NUM>, pixel unprojector <NUM>, pixel flow loss function <NUM>, depth loss function <NUM>, adder <NUM>, adder <NUM>, and model parameter adjuster <NUM>, among other components. These components of depth system <NUM> may be implemented as hardware, as software, or as a combination thereof.

Depth system <NUM> may be configured to determine, for one or more images of video <NUM>, corresponding one or more depth images that are geometrically and temporally consistent. That is, the depth (as represented by the depth images) associated with a given feature of the images may change smoothly and/or in proportion to actual physical motion of the given feature represented across image frames, rather than changing erratically, inconsistently, and/or disproportionally. Thus, when presented as part of a video, the depth represented by the depth images might not appear to jitter, and may thus appear consistent with the actual physical movements of various features over the course of the video.

Video <NUM> may be, for example, a monoscopic/monocular color video captured by a monoscopic/monocular color camera, and may represent static features and/or moving features. Video <NUM> may include image <NUM>, image <NUM>, image <NUM>, and zero or more intermediate images, as indicated by the ellipses. Images <NUM>, <NUM>, <NUM>, and the intermediate images may collectively be referred to as images <NUM> - <NUM>. Video <NUM> may be alternatively expressed as V, while images <NUM> - <NUM> may be alternatively expressed as Ii ∈ V, where the index i may indicate a frame number, position, and/or time within video <NUM>.

Optical flow <NUM> may represent the optical flow between different pairs of images of video <NUM>. For example, optical flow <NUM> may include optical flow <NUM> (between image <NUM> and image <NUM>) through optical flow <NUM> (between image <NUM> and image <NUM>), among others. Optical flow <NUM> may be determined using, for example, the RAFT model, as detailed in a paper titled "RAFT: Recurrent All-Pairs Field Transforms for Optical Flow," authored by Teed et al. , and published as arXiv:<NUM>. Alternatively or additionally, optical flow <NUM> may be determined using other models and/or algorithms, such as, for example, the Lucas-Kanade method. In some implementations, optical flow <NUM> may be determined using a component (not shown) of depth system <NUM> based on video <NUM>.

In some implementations, optical flow <NUM> may include, for each respective image of a plurality of images of video <NUM>, a corresponding plurality of optical flows between the respective image and a predetermined sequence (e.g., number and/or position) of subsequent images. For example, the respective image Ii may be associated with corresponding optical flows between image Ii and each of images Ii+k, k ∈ [<NUM>, <NUM>, <NUM>, <NUM>, <NUM>]. Optical flow <NUM> may include, for each image pair of Ii and Ii+k k ∈ [<NUM>, <NUM>, <NUM>, <NUM>, <NUM>], a forward optical flow, which may be expressed as vi→i+k, and/or a backward optical flow, which may be expressed as vi+k→i. For example, optical flow <NUM> may include a forward flow from image <NUM> to image <NUM> and/or a backward flow from image <NUM> to image <NUM>.

Video <NUM> may be pre-processed to determine, for each respective image Ii of a plurality of images of video <NUM>, a corresponding set of extrinsic camera parameters Ri and ti, indicative of the rotation and translation, respectively, of the camera relative to a reference frame (e.g., the world frame), and a corresponding set of intrinsic camera parameters Ki. For example, camera parameters Ri and ti may be determined using a simultaneous localization and mapping (SLAM) algorithm/model and/or a structure-from-motion algorithm/model, among other possibilities. In some implementations, a scale of ti may be aligned to approximately match a scale of initial depth estimates associated with video <NUM>. The preprocessing of video <NUM> to determine camera parameters Ri and ti may be performed, for example, by a component (not shown) of depth system <NUM>.

Depth model <NUM> may be configured to generate, based on image <NUM> (representing first image Ii) of video <NUM>, (first) depth image <NUM>, which may be expressed as Di = FθD(Ii), where FθD represents the function implemented by depth model <NUM> and θD represents parameters of depth model <NUM>. Depth model <NUM> may also be configured to generate, based on image <NUM> of video <NUM> (representing second image Ij), (second) depth image <NUM>, which may be expressed as Dj = FθD(Ij). Depth model <NUM> may also be configured to generate other depth images (e.g., Dj =i+k k ∈ [<NUM>, <NUM>, <NUM>, <NUM>, <NUM>]) based on corresponding other images of video <NUM>, and images <NUM> and <NUM> are discussed herein as a representative example. Similarly, other components of depth system <NUM> may be configured to perform operations based on these other images, such that multiple different i,j pairs may be processed. Depth model <NUM> may therefore take as input values corresponding to pixels of an input image and output a corresponding depth image indicating depth values associated with each input value.

Depth model <NUM> may be, for example, a convolutional neural network that has been pre-trained to generate depth images based on monoscopic images. Prior to the training discussed herein, depth images generated by depth model <NUM> may include geometric and/or temporal inaccuracies and/or inconsistencies. Thus, although depth model <NUM> may, due to the pre-training, generate depth images that, when considered individually, are at least partially correct, any inaccuracies and/or inconsistencies in these depth images may become visually apparent when the depth images are viewed in a sequence (e.g., as part of a video of the depth images). Thus, depth system <NUM> may be configured to fine-tune depth model <NUM> by way of additional training to generate depth images that have improved accuracy and/or consistency.

Pixel unprojector <NUM> may be configured to determine 3D points corresponding to and representing the depth values associated with pixels of the images in video <NUM>. Specifically, for a first plurality of pixels of image <NUM>, pixel unprojector <NUM> may be configured to determine corresponding 3D points <NUM> (i.e., first-image 3D points) and, for a second plurality of pixels of image <NUM>, pixel unprojector <NUM> may be configured to determine corresponding 3D points <NUM> (i.e., second-image 3D points). 3D points <NUM> may be represented in a world reference frame, and 3D points <NUM> may be represented in the world reference frame or a camera reference frame associated with image <NUM>. The first plurality of pixels may include all or a subset of the pixels of image <NUM>, and the second plurality of pixels may include all or a subset of the pixels of image <NUM>.

3D points <NUM> may be based on respective depth values of depth image <NUM>, and 3D points <NUM> may be based on respective depth values of depth image <NUM>. 3D points <NUM> may be expressed as <MAT> (in the world reference frame), where x ∈ Ii represents coordinates of a respective pixel of the first plurality of pixels of image <NUM>, Di(x) represents the depth value, as represented by depth image <NUM>, associated with the respective pixel, Ri, ti, and Ki represent the camera parameters associated with image <NUM>, and x' represents a 2D homogeneous augmentation of coordinates of the respective pixel. Similarly, 3D points <NUM> may be expressed as <MAT> (in the world reference frame) or as <MAT> (in the camera reference frame of image <NUM>), with the index j indicating commensurate values associated with image <NUM>.

Scene flow model <NUM> may be configured to determine scene flow <NUM> based on 3D points <NUM>. Scene flow <NUM> may include, for each respective 3D point of 3D points <NUM>, a corresponding 3D vector indicating a displacement of the respective 3D point from a time (represented by index i) associated with image <NUM> to a time (represented by index j) associated with image <NUM>. Thus, scene flow model <NUM> may be configured to model how different parts of a scene represented by video <NUM> move over time in 3D, and may thus explicitly account for dynamic (i.e., non-static) features of the scene. Scene flow model <NUM> may be implemented as an artificial neural network, such as, for example, a multi-layer perceptron (MLP), among other possible architectures. Scene flow model <NUM> may include a positional encoding of the input, and which may be expressed as ξ(c) = [sinπc, cosπc, sin2πc, cos2πc,. , sinNπc, cosNπc], where N = <NUM> for example. The positional encoding ξ(c) may be applied to each element of the input (e.g., the x-coordinate, the y-coordinate, the z-coordinate, and the time i associated with a given 3D point Xi(x) of 3D points <NUM>), resulting in a <NUM>-dimensional vector with <NUM> dimensions per input element.

In cases where images <NUM> and <NUM> are consecutive (i.e., when j = i + <NUM>), scene flow <NUM> may be expressed as Si→i+<NUM>(x) = GθS(Xi(x), i), where GθS represents the function implemented by scene flow model <NUM>, θS represents parameters of scene flow model <NUM>, i represents the time associated with depth image <NUM>, and x and Xi(x) are as defined above. In cases where images <NUM> and <NUM> are non-consecutive, and thus separated by one or more intermediate images (i.e., when j - i > <NUM>), scene flow <NUM> may be determined by applying scene flow model <NUM> recursively j - i times. This recursive application of scene flow model <NUM> may be expressed as <MAT>, where <MAT> for k = i + <NUM>,. ,j - <NUM>. Thus, scene flow model <NUM> may be applied once to 3D points <NUM>, and once to each set of intermediate 3D points that are (i) associated with a corresponding intermediate image of the one or more intermediate images and (ii) representing 3D points <NUM> at the time associated with the corresponding intermediate image.

For example, when image <NUM> and image <NUM> are separated by two intermediate images, scene flow model <NUM> may be configured to determine a first intermediate scene flow between image <NUM> and a first intermediate image (consecutive with image <NUM>) based on 3D points <NUM>. 3D points <NUM> may be displaced according to the first intermediate scene flow, thereby generating first intermediate 3D points corresponding to the first intermediate image. Scene flow model <NUM> may then be configured to determine a second intermediate scene flow between the first intermediate image and a second intermediate image based on the first intermediate 3D points. The first intermediate 3D points may be displaced according to the second intermediate scene flow, thereby generating second intermediate 3D points corresponding to the second intermediate image. Scene flow model <NUM> may be configured to determine a third intermediate scene flow between the second intermediate image and image <NUM> based on the second intermediate 3D points. The second intermediate 3D points may be displaced according to the third intermediate scene flow, thereby generating 3D points (i.e., post-flow 3D points <NUM>) corresponding to the second image. Accordingly, scene flow <NUM> may be based on a sum of the first intermediate scene flow, the second intermediate scene flow, and the third intermediate scene flow, and may thus represent a 3D motion of 3D points <NUM> from time i to time j.

Theoretically, scene flow <NUM> could, instead of being determined by scene flow model <NUM>, be analytically computed based on 2D correspondences (i.e., optical flow) between an image pair and depth images corresponding to the image pair. However, such analytical computation may be noisy and unstable, especially when the camera rays at corresponding points are nearly parallel. Implicitly representing the scene flow using scene flow model <NUM> may act as an auxiliary variable where the relation to the analytically computed scene flow is encouraged through the loss functions used by depth system <NUM>. Additionally, usage of scene flow model <NUM> allows the optimization executed by depth system <NUM> during the course of training to be stable, and thus arrive at a satisfactory depth model <NUM>, which might not be possible using the analytical scene flow computation.

Scene flow <NUM> may be applied to 3D points <NUM> to generate post-flow 3D points <NUM>. Specifically, each respective 3D point of 3D points <NUM> may be displaced according to a corresponding 3D vector of scene flow <NUM> by adding the corresponding 3D vector to the respective 3D point, as indicated by adder <NUM>. Post-flow 3D points <NUM> may be represented in the world reference frame. Post-flow 3D points <NUM> may be expressed as Xi→j(x) = Xi(x) + Si→j(x). Post-flow 3D points <NUM> may thus represent the positions of 3D points <NUM>, which were determined based on image <NUM>, at a time corresponding to image <NUM>. Accordingly, a difference between post-flow 3D points <NUM> and 3D points <NUM> may be indicative of a consistency of depth images <NUM> and <NUM> generated by depth model <NUM>, and may thus be used as a basis for modifying parameters of depth model <NUM>.

In order to quantify a consistency of depth images <NUM> and <NUM>, depth system <NUM> may be configured to identify a pixel correspondence between image <NUM> and image <NUM>. Specifically, adder <NUM> may be configured to add, to each pixel position of image <NUM>, a corresponding optical flow value of optical flow <NUM>, to thereby determine the pixel correspondence between image <NUM> and image <NUM>. Pixel correspondences between other image pairs may be determined by adding another corresponding optical flow to pixel coordinates of a first image of a given pair. The pixel correspondences between image <NUM> and image <NUM> may be represented by pixel positions <NUM>. Specifically, for each respective pixel of the first plurality of pixels of image <NUM>, pixel positions <NUM> may indicate a corresponding position within image <NUM>. Pixel positions <NUM> may thus be expressed as pi→j(x) = x + vi→j(x).

3D point projector <NUM> may be configured to determine induced pixel positions <NUM> based on post-flow 3D points <NUM>. Specifically, 3D point projector <NUM> may be configured to project post-flow 3D points <NUM> from the world reference frame into image space of image <NUM>. Thus, induced pixel positions <NUM> may provide a 2D representation of post-flow 3D points <NUM>, and thus a representation of the displacement of 3D points <NUM> caused by scene flow <NUM>. Induced pixel positions <NUM> may be expressed as <MAT>, where π represents the projection operator π([x, y, w]T) = [x/w,y/w]T.

Since induced pixel positions <NUM> are based on the output of depth model <NUM> and scene flow model <NUM>, a comparison of induced pixel positions <NUM> and pixel positions <NUM> may be performed to evaluate performance of models <NUM> and <NUM>. Specifically, pixel flow loss function <NUM> may be configured to determine pixel flow loss value <NUM> based on a comparison of induced pixel positions <NUM> and pixel positions <NUM>. Because pixel positions <NUM> are based on optical flow, which may have been accurately determined, pixel positions <NUM> may be considered to function as a ground-truth to which induced pixel positions <NUM> are compared. Pixel flow loss value <NUM> may be expressed as <MAT> for a single pixel of image Ii, or as <MAT> for video <NUM> as a whole, where one or more i, j pairs of images may be compared, and where x may be iterated through the respective first plurality of pixels of each image Ii. Pixel flow loss function <NUM> may alternatively be referred to as a 2D consistency loss function, and may thus be expressed as L<NUM>D = LFLOW.

In some implementations, depth system <NUM> may be configured to determine an occlusion mask that corresponds to image <NUM> and indicates a region thereof that is visible in both image <NUM> and image <NUM>. For example, the occlusion mask may be determined by comparing (i) the forward optical flow from image <NUM> to image <NUM> to (ii) the backward optical flow from image <NUM> to image <NUM>. Pixel positions may be considered occluded and/or the optical flow may be considered inaccurate where the forward optical flow and the backward optical flow differ by more than a predetermined amount of pixels. For example, the occlusion mask may be expressed as MOi→i+k(x) = <NUM> if | vi→i+k(x) + vi+k→i(x + vi→i+k(x)) |<NUM> > <NUM>, or <NUM> otherwise. Accordingly, in some cases, the first plurality of pixels of image <NUM> considered by loss functions <NUM> and/or <NUM> may be based on the occlusion mask. Thus, pixel flow loss value <NUM> may alternatively be expressed as <MAT>. That is, pixel flow loss function <NUM> may be configured to compare induced pixel positions <NUM> to pixel positions <NUM> for pixels of image <NUM> that are unoccluded and/or associated with accurate optical flow.

Further, since post-flow 3D points <NUM> and 3D points <NUM> are based on the output of depth model <NUM> and scene flow model <NUM>, a comparison of post-flow 3D points <NUM> and 3D points <NUM> may be performed to evaluate performance of models <NUM> and <NUM>. Specifically, depth loss function <NUM> may be configured to determine depth loss value <NUM> based on a comparison of post-flow 3D points <NUM> and 3D points <NUM>. In some implementations, depth loss function <NUM> may be configured to perform the comparison of post-flow 3D points <NUM> and 3D points <NUM> in the world reference frame. Accordingly, depth loss value <NUM> may be expressed as <MAT> for a single pixel of image Ii, or as <MAT> for video <NUM> as a whole, where one or more i, j pairs of images may be compared, and where x may be iterated through the respective first plurality of pixels of each image Ii.

In other implementations, depth loss function <NUM> may be configured to perform the comparison of post-flow 3D points <NUM> and 3D points <NUM> in the camera reference frame associated with image Ij. Depth loss function may be configured to determine a representation of post-flow 3D points <NUM> in the camera reference frame associated with image Ij, which may be expressed as <MAT>. Accordingly, depth loss value <NUM> may be expressed as <MAT> for a single pixel of image Ii, or as <MAT> for video <NUM> as a whole, where one or more i, j pairs of images may be compared, and where x may be iterated through the respective first plurality of pixels of each image Ii.

In further implementations, depth loss function <NUM> may be configured to compare components of post-flow 3D points <NUM> and 3D points <NUM>. For example, depth loss function may be configured to determine the depth component of post-flow 3D points <NUM> (as represented in the camera reference frame associated with image Ij), which may be expressed as <MAT>, where |·|z denotes taking the depth component of a 3D point, that is, z = | [x,y,z] |z. Accordingly, depth loss value <NUM> may be expressed as <MAT> for a single pixel of image Ii, or as <MAT> for video <NUM> as a whole, where one or more i, j pairs of images may be compared, and where x may be iterated through the respective first plurality of pixels of each image Ii. Accordingly, in cases where depth loss value <NUM> is based on Dj(pi→j(x)), rather than on Xj(pi→j(x)), depth system <NUM> might not explicitly determine all components of 3D points <NUM>, and may instead determine Dj(pi→j(x)) directly from depth image <NUM>.

Further, in some cases, LDEPTH-<NUM>, LDEPTH-<NUM>, and/or LDEPTH-<NUM> may be computed based on pixels of image Ii that are unoccluded and/or associated with accurate optical flow, that is, x ∈ {MOi→j = <NUM>}, as discussed above. Depth loss function <NUM> may alternatively be referred to as a disparity consistency loss function, or a 3D consistency loss function, and may thus be expressed as LDISPARITY = LDEPTH or L<NUM>D = LDEPTH, where LDEPTH may represent any of the variations of depth loss function <NUM> discussed above.

In some implementations, depth system <NUM> may also be configured to determine additional loss values that may be used in training of scene flow model <NUM> and/or depth model <NUM>. In one example, depth system may be configured to determine a scene flow velocity loss value <MAT> using a scene flow velocity loss function, which may be expressed as <MAT> for a single pixel of image Ii, or as <MAT> for video <NUM> as a whole, where i may be iterated through at least a subset of video <NUM>. The scene flow velocity loss function may be configured to compare (i) a first scene flow determined between images Ii and Ii+<NUM> based on the 3D points corresponding to images Ii to (ii) a second scene flow determined between images Ii+<NUM> and Ii+<NUM> based on the post-flow 3D points of image Ii resulting displacement of the 3D points of image Ii according to the first scene flow. Thus, the scene flow velocity loss function may incentivize scene flow model <NUM> to generate scene flows that, for a given point, do not change significantly between consecutive frames, and thus indicate that a corresponding feature moves at an approximately constant velocity.

In another example, depth system may be configured to determine a static region loss value <MAT> using a static region loss function. The static region loss value <MAT> may be based on a static region mask MSi that indicates regions and/or features of image Ii that do not move relative to the world reference frame over time. The static region loss function may be expressed as <MAT> for a single pixel of image Ii, or as <MAT> for video <NUM> as a whole, where i may be iterated through at least a subset of video <NUM>. Thus, the static region loss function may incentivize scene flow model <NUM> to generate scene flows that tend to zero for static regions and/or features, since these regions and/or features are not expected to move relative to the world reference frame.

While the L-<NUM> norm is used for the loss functions discussed herein, it is to be understood that other norms, such as the L-<NUM> norm, may be used in place of the L-<NUM> norm. For some of the loss functions, some norms may result in models <NUM> and/or <NUM> being trained to generate more accurate and/or more consistent (e.g., geometrically and/or temporally) depth images. An appropriate norm may be selected empirically as part of the training process.

Model parameter adjuster <NUM> may be configured to determine updated model parameters <NUM> based on pixel flow loss value <NUM>, depth loss value <NUM>, and any other loss values determined by depth system <NUM>. Model parameter adjuster <NUM> may be configured to determine a total loss value based on a weighted sum of the loss values, which may be expressed as LTOTAL = LFLOW + αLDEPTH + βLSF VELOCITY + γLSTATIC, where LDEPTH may be any one of LDEPTH-<NUM>, LDEPTH-<NUM>, or LDEPTH-<NUM>, and where the values of α, β, and γ represent the relative weight of the corresponding loss values (e.g., α = <NUM>, β = <NUM>, and γ = <NUM>). Updated model parameters <NUM> may include one or more updated parameters of scene flow model <NUM>, which may be expressed as ΔθS, and/or one or more updated parameters of depth model <NUM>, which may be expressed as ΔθD.

Model parameter adjuster <NUM> may be configured to determine updated model parameters <NUM> by, for example, determining a gradient of the total loss function LTOTAL. Based on this gradient and the total loss value, model parameter adjuster <NUM> may be configured to select updated model parameters <NUM> that are expected to reduce the total loss value, and thus improve performance of depth system <NUM>. After applying updated model parameters <NUM> to scene flow model <NUM> and/or depth model <NUM>, the operations discussed above may be repeated to compute another instance of the total loss value and, based thereon, another instance of updated model parameters <NUM> may be determined and applied to models <NUM> and <NUM> to further improve the performance thereof. Such training of models <NUM> and <NUM> may be repeated until, for example, the total loss value is reduced to below a target threshold loss value.

In some implementations, depth model <NUM> may be pre-trained using, for example, a data set that includes pairs of monoscopic images and corresponding ground-truth depth images, while scene flow model <NUM> may be randomly initialized. Thus, depth model <NUM> may be expected to generate outputs that, due to the pre-training, are reasonably accurate (albeit temporally and/or geometrically inconsistent across image frames), while scene flow model <NUM> may be expected to generate outputs that are inaccurate due to the random initialization. Thus, for a predetermined number of initial training iterations, parameters of depth model <NUM> may be kept constant, while parameters of scene flow model <NUM> may be modified. Thus, during the initial training iterations, scene flow model <NUM> may be trained to coordinate with depth model <NUM> without allowing the random initialization of scene flow model <NUM> to cause depth model <NUM> to inadvertently "unlearn" information that is actually useful in depth determination. During subsequent training iterations, parameters of both depth model <NUM> and scene flow model <NUM> may be modified to allow both models to be fine-tuned to video <NUM>.

Once models <NUM> and <NUM> are trained, depth model <NUM> may be used to process images of video <NUM> and generate corresponding depth images. As a result of training, these depth images may have improved geometric and/or temporal consistency, at least relative to depth images that would be generated by depth model <NUM> prior to training by depth system <NUM>. Scene flow model <NUM> may be used during training, but might not be used during the final inference pass following completion of training, since, at least in some cases, the output thereof might not be useful outside of training. Additionally, following training, depth model <NUM> and scene flow model <NUM> may be specific to video <NUM> in that depth model <NUM> may be configured to generate geometrically and temporally consistent depth images based on images of video <NUM>, but might not perform as accurately with respect to other videos. Thus, in order to generate geometrically and temporally accurate depth images for another video, depth system <NUM> may be configured to retrain models <NUM> and <NUM> based on images of this other video using the process discussed above.

The depth images generated for a particular video using the trained depth model may be used to apply one or more depth-based modifications to the video. In one example, the depth images may be used to insert an object into the video in a depth-aware manner and, due to the geometric and temporal consistency, the object may appear at a consistent position in the world reference frame across image frames. In another example, the depth images may be used to alter the lighting of the video and/or apply lighting-based effects to the video and, due to the geometric and temporal consistency, the lighting may appear to change consistently with the object and/or camera motion across image frames. Other depth-based effects are possible.

<FIG> illustrates a geometric representation of aspects of the computation of pixel flow loss value <NUM> and depth loss value <NUM>. Specifically, <FIG> includes image <NUM>, image <NUM>, camera reference frame <NUM> associated with image <NUM>, and camera reference frame <NUM> associated with image <NUM>. Pixel <NUM> is a representative example of the first plurality of pixels considered as part of the computation of the loss values. For example, pixel <NUM> may be located within an unoccluded portion of the occlusion mask associated with image <NUM>.

When a corresponding optical flow value of optical flow <NUM> is added to coordinates of pixel <NUM>, the resulting coordinates may be represented by pixel <NUM> in image <NUM>. Thus, pixel <NUM> corresponds to one of pixel positions <NUM>. Additionally, pixels <NUM> and <NUM> may be expected to represent approximately the same feature of the scene, albeit in different images. Pixel <NUM> is a representative example of the second plurality of pixels considered as part of the computation of the loss values.

Pixel <NUM> may be associated with a corresponding depth value represented by a corresponding pixel in depth image <NUM>. Based on this corresponding depth value, pixel <NUM> may be unprojected (or backprojected) by pixel unprojector <NUM> to generate 3D point <NUM>, which may represent one of 3D points <NUM>. Similarly, pixel <NUM> may be associated with a corresponding depth value represented by a corresponding pixel in depth image <NUM>. Based on this corresponding depth value, pixel <NUM> may be unprojected (or backprojected) by pixel unprojector <NUM> to generate 3D point <NUM>, which may represent one of 3D points <NUM>.

3D point <NUM> may be displaced according to scene flow vector <NUM> to generate 3D point <NUM>. Scene flow vector <NUM> may be one of a plurality of vectors represented by scene flow <NUM>. 3D point <NUM> may be one of post-flow 3D points <NUM>. 3D point <NUM> may be projected into the image space of image <NUM> by 3D point projector <NUM> to determine pixel <NUM>, which may represent one of induced pixel positions <NUM>.

Arrow <NUM> between pixel <NUM> and pixel <NUM> provides a visual representation of the contribution of pixel <NUM> to pixel flow loss value <NUM>. Specifically, the loss contributed by pixel <NUM> to pixel flow loss value <NUM> is based on a distance (e.g., L-<NUM> norm, L-<NUM> norm, etc.) between pixel <NUM> and pixel <NUM>. The distance between pixel <NUM> and pixel <NUM> represents the difference between a pixel position resulting from optical flow <NUM> and a pixel position resulting from scene flow <NUM>.

Arrow <NUM> between the z-axis projection of 3D point <NUM> and the z-axis projection of 3D point <NUM> provides a visual representation of the contribution of pixel <NUM> to depth loss value <NUM>. Specifically, the loss contributed by pixel <NUM> to depth loss value <NUM> is based on a distance (e.g., L-<NUM> norm, L-<NUM> norm, etc., or a component thereof) between 3D point <NUM> and 3D point <NUM>.

<FIG> illustrates a flow chart of operations related to determining geometrically and temporally consistent depth images based on a video. The operations may be carried out by computing device <NUM>, computing system <NUM>, and/or depth system <NUM>, among other possibilities. The embodiments of <FIG> may be simplified by the removal of any one or more of the features shown therein. Further, these embodiments may be combined with features, aspects, and/or implementations of any of the previous figures or otherwise described herein.

Block <NUM> may involve obtaining a first image from a video, a second image from the video, and an optical flow between the first image and the second image.

Block <NUM> may involve determining that a second pixel of the second image corresponds to a displacement of a first pixel of the first image to the second image according to the optical flow.

Block <NUM> may involve determining, by a depth model, (i) a first depth image based on the first image and (ii) a second depth image based on the second image.

Block <NUM> may involve determining (i), based on the first depth image, a first depth associated with the first pixel and (ii), based on the second depth image, a second depth associated with the second pixel.

Block <NUM> may involve determining (i) a first three-dimensional (3D) point based on the first depth associated with the first pixel and (ii) a second 3D point based on the second depth associated with the second pixel.

Block <NUM> may involve determining, by a scene flow model and based on the first 3D point, a scene flow representing a 3D motion of the first 3D point between the first image and the second image.

Block <NUM> may involve determining, for the first pixel, an induced pixel position based on a post-flow 3D point that represents the first 3D point after a displacement according to the scene flow.

Block <NUM> may involve determining (i) a pixel flow loss value based on a comparison of the induced pixel position to a position of the second pixel and (ii) a depth loss value based on a comparison of the post-flow 3D point to the second 3D point.

Block <NUM> may involve adjusting one or more parameters of one or more of the depth model or the scene flow model based on the pixel flow loss value and the depth loss value.

In some embodiments, the pixel flow loss value and the depth loss value may represent an extent of temporal consistency between the first depth image and the second depth image. After adjusting the one or more parameters, the depth model may be configured to generate, based on images in the video, depth images with improved temporal consistency. The depth model and the scene flow model may be specific to the video.

In some embodiments, determining the induced pixel position may include determining a projection of the post-flow 3D point into an image space of the second image.

In some embodiments, determining the depth loss value may include determining a representation of the post-flow 3D point in a reference frame associated with the second image. The second 3D point may be represented in the reference frame associated with the second image. Determining the depth loss value may also include determining (i) a first depth component of the representation of the post-flow 3D point and (ii) a second depth component of the second 3D point, and determining the depth loss value based on a comparison of the first depth component and the second depth component.

In some embodiments, determining the depth loss value may include determining a difference between (i) an inverse of the first depth component and (ii) an inverse of the second depth component.

In some embodiments, the second pixel of the second image may be one of a second plurality of pixels of the second image. Each respective pixel of the second plurality of pixels may correspond to a pixel of a first plurality of pixels of the first image. The second plurality of pixels may be determined based on a displacement of the first plurality of pixels from the first image to the second image according to the optical flow. The first 3D point may be one of a plurality of first-image 3D points corresponding to the first plurality of pixels. Each respective first-image 3D point of the plurality of first-image 3D points may be determined based on a respective depth associated with a corresponding pixel of the first plurality of pixels according to the first depth image. The second 3D point may be one of a plurality of second-image 3D points corresponding to the second plurality of pixels. Each respective second-image 3D point of the plurality of second-image 3D points may be determined based on a respective depth associated with a corresponding pixel of the second plurality of pixels according to the second depth image. The post-flow 3D point may be one of a plurality of post-flow 3D points. Each respective post-flow 3D point of the plurality of post-flow 3D points may represent a corresponding first-image 3D point after a corresponding displacement according to the scene flow.

In some embodiments, determining the depth loss value may include performing, for each respective pixel of the first plurality of pixels, a comparison of the corresponding post-flow 3D point to the corresponding second-image 3D point.

In some embodiments, determining the induced pixel position may include determining, for each respective pixel of the first plurality of pixels, a corresponding induced pixel position based on the corresponding post-flow 3D point. Determining the pixel flow loss value may include performing, for each respective pixel of the first plurality of pixels, a comparison of the corresponding induced pixel position to a position of the corresponding pixel of the second plurality of pixels.

In some embodiments, an occlusion mask corresponding to the first image and indicating an unoccluded region of the first image that is visible in the first image and in the second image may be determined. The first pixel may be selected from the unoccluded region of the first image based on the occlusion mask.

In some embodiments, the depth model may include a convolutional neural network that has been pre-trained to generate depth images using a training data set that includes a plurality of pairs of (i) a respective monoscopic training image and (ii) a corresponding ground-truth depth image.

In some embodiments, the scene flow model may include a multi-layer perceptron that has been randomly initialized.

In some embodiments, the scene flow model may be configured to receive as input (i) 3D coordinates representing a given 3D point in a world reference frame and (ii) a time value associated with a particular depth image on which the given 3D point is based. The scene flow model may be configured to determine, for the given 3D point and the time value, a corresponding vector representing a displacement of the given 3D point from the respective time value to a time value subsequent thereto.

In some embodiments, a plurality of training iterations may be performed. Each respective training iteration of the plurality of training iterations may include determining respective instances of the first depth image, the second depth image, the first depth, the second depth, the scene flow, the first 3D point, the second 3D point, the post-flow 3D point, the induced pixel position, the pixel flow loss value, and the depth loss value. During an initial sequence of training iterations of the plurality of training iterations, one or more parameters of the scene flow model may be adjusted while parameters of the depth model may be kept constant. During a subsequent sequence of training iterations of the plurality of training iterations, one or more parameters of the scene flow model may be adjusted and one or more parameters of the depth model may be adjusted.

In some embodiments, the scene flow may be a first scene flow. The second image may be subsequent to and consecutive with the first image. A third image of the video may be subsequent to and consecutive with the second image. A second scene flow may be determined by the scene flow model and based on the post-flow 3D point. The second scene flow may represent an additional 3D motion of the post-flow 3D point between the second image and the third image. A scene flow velocity loss value may be determined based on a comparison of the first scene flow and the second scene flow. The one or more parameters may be adjusted further based on the scene flow velocity loss value.

In some embodiments, a static region mask may be determined. The static region mask may correspond to the first image and may indicate a static region of the first image that does not move relative to a world reference frame. A static region loss value may be determined based on scene flow values associated with the static region of the first image. The one or more parameters may be adjusted further based on the static region loss value.

In some embodiments, the second image may be subsequent to and consecutive with the first image. A third image may be obtained from the video and an additional optical flow between the first image and the third image may be obtained. The third image may be separated from the first image by one or more intermediate images of the video. It may be determined that a third pixel of the third image corresponds to a displacement of the first pixel from the first image to the third image according to the additional optical flow. A third depth image may be determined by the depth model based on the third image. A third depth associated with the third pixel may be determined based on the third depth image. A third 3D point may be determined based on the third depth associated with the third pixel. An additional scene flow representing a 3D motion of the first 3D point between the first image and the third image may be determined by the scene flow model. An additional induced pixel position may be determined for the first pixel based on an additional post-flow 3D point that represents the first 3D point after an additional displacement according to the additional scene flow. The pixel flow loss value may be determined further based on a comparison of the additional induced pixel position to a position of the third pixel. The depth loss value may be determined further based on a comparison of the additional post-flow 3D point to the third 3D point.

In some embodiments, determining the additional scene flow may include determining, for each respective intermediate image of the one or more intermediate images that separate the third image from the first image, a corresponding intermediate post-flow 3D point that represents the first 3D point after one or more displacements according to one or more preceding scene flows determined for one or more preceding images. Determining the additional scene flow may also include determining, by the scene flow model and for each respective intermediate image, a corresponding scene flow based on the corresponding intermediate post-flow 3D point, and determining the additional scene flow based on a sum of the corresponding scene flow of each respective intermediate image of the one or more intermediate images.

In some embodiments, first camera parameters associated with the first image and second camera parameters associated with the second image may be determined based on the video. The first 3D point may be determined based on the first camera parameters. The second 3D point and the induced pixel position may each be determined based on the second camera parameters.

In some embodiments, after adjusting the one or more parameters, a plurality of depth images based on a corresponding plurality of images of the video may be determined by the depth model. Based on the plurality of depth images, one or more images of the corresponding plurality of images may be modified to insert a depth-based effect into the video.

<FIG> includes a table that shows various performance metrics for a plurality of different depth models tested on a subset of the Sintel data set, which is detailed in a paper titled "A Naturalistic Open Source Movie for Optical Flow Evaluation," authored by Butler et al. , and published at the <NUM> European Conference on Computer Vision. L-<NUM> relative error (L1 REL. ), log root mean square error (LOG RMSE), and RMSE of the determined depth images relative to the ground-truth are shown as measured for (i) the entire area of each of the tested Sintel data set images ("Full"), (ii) dynamic (i.e., moving relative to the world reference frame) regions of these images ("Dynamic"), and (iii) static regions of these images ("Static"). Lower numbers indicate better performance, as indicated by the downward arrows.

Regarding the specific tested models, "MC" in rows <NUM>, <NUM>, <NUM>, <NUM>, and <NUM> indicates a version of depth model <NUM> that has been pre-trained (i.e., initialized) using the techniques and data set described in a paper titled "Learning the Depths of Moving People by Watching Frozen People," authored by Li et al. , and published as arXiv:<NUM>. "MIDAS" in rows <NUM>, <NUM>, <NUM>, <NUM>, and <NUM> indicates a version of depth model <NUM> that has been pre-trained using the techniques and data set described in a paper titled "Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer," authored by Ranftl et al. , and published as arXiv:<NUM>. "CVD" in rows <NUM> and <NUM> indicates a version of depth model <NUM> that has been additionally trained (i.e., fine-tuned) using the techniques described in a paper titled "Consistent Video Depth Estimation," authored by Luo et al. , and published as arXiv:<NUM>.

"Depth system <NUM>" in rows <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, and <NUM> indicates a version of depth model <NUM> that has been additionally trained (i.e., fine-tuned) using the techniques discussed herein. Specifically, "No LSF VELOCITY" in rows <NUM> amd <NUM> indicate that depth model <NUM> has been trained using only pixel flow loss function <NUM> and depth loss function <NUM> (i.e., without using scene flow velocity loss). Rows <NUM> and <NUM> indicate that depth model <NUM> has been trained using pixel flow loss function <NUM>, depth loss function <NUM>, and the scene flow velocity loss function. "W/ LSTATIC" in rows <NUM> and <NUM> indicate that depth model <NUM> was trained using pixel flow loss function <NUM>, depth loss function <NUM>, the scene flow velocity loss function, and the static region loss function.

For a given model initialization, the best result for a given performance metric is indicated by a pattern-filled cell, and the second best result is indicated with an underline. Many of the variations of depth system <NUM>, detailed in rows <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, and <NUM>, improve upon the results produced by the corresponding initial pre-trained depth model, detailed in rows <NUM> and <NUM>. Further, many of the variations of depth system <NUM> outperform CVD (which assumes a static scene) with respect to the full scene and dynamic portions of the full scene, and some perform very closely with CVD in static portions of the full scene. In general, the addition of the scene flow velocity loss function and the static region loss function improve performance of depth model <NUM>.

The present disclosure is not to be limited in terms of the particular embodiments described in this application, which are intended as illustrations of various aspects. Many modifications and variations can be made without departing from its scope, as will be apparent to those skilled in the art. Functionally equivalent methods and apparatuses within the scope of the appended claims, in addition to those described herein, will be apparent to those skilled in the art from the foregoing descriptions. Such modifications and variations are intended to fall within the scope of the appended claims.

The above detailed description describes various features and operations of the disclosed systems, devices, and methods with reference to the accompanying figures. In the figures, similar symbols typically identify similar components, unless context dictates otherwise. The example embodiments described herein and in the figures are not meant to be limiting. Other embodiments can be utilized, and other changes can be made, without departing from the scope of the subject matter presented herein. It will be readily understood that the aspects of the present disclosure, as generally described herein, and illustrated in the figures, can be arranged, substituted, combined, separated, and designed in a wide variety of different configurations.

With respect to any or all of the message flow diagrams, scenarios, and flow charts in the figures and as discussed herein, each step, block, and/or communication can represent a processing of information and/or a transmission of information in accordance with example embodiments. Alternative embodiments are included within the scope of these example embodiments. In these alternative embodiments, for example, operations described as steps, blocks, transmissions, communications, requests, responses, and/or messages can be executed out of order from that shown or discussed, including substantially concurrently or in reverse order, depending on the functionality involved. Further, more or fewer blocks and/or operations can be used with any of the message flow diagrams, scenarios, and flow charts discussed herein, and these message flow diagrams, scenarios, and flow charts can be combined with one another, in part or in whole.

A step or block that represents a processing of information may correspond to circuitry that can be configured to perform the specific logical functions of a herein-described method or technique. Alternatively or additionally, a block that represents a processing of information may correspond to a module, a segment, or a portion of program code (including related data). The program code may include one or more instructions executable by a processor for implementing specific logical operations or actions in the method or technique. The program code and/or related data may be stored on any type of computer readable medium such as a storage device including random access memory (RAM), a disk drive, a solid state drive, or another storage medium.

The computer readable medium may also include non-transitory computer readable media such as computer readable media that store data for short periods of time like register memory, processor cache, and RAM. The computer readable media may also include non-transitory computer readable media that store program code and/or data for longer periods of time. Thus, the computer readable media may include secondary or persistent long term storage, like read only memory (ROM), optical or magnetic disks, solid state drives, compact-disc read only memory (CD-ROM), for example. The computer readable media may also be any other volatile or non-volatile storage systems. A computer readable medium may be considered a computer readable storage medium, for example, or a tangible storage device.

Moreover, a step or block that represents one or more information transmissions may correspond to information transmissions between software and/or hardware modules in the same physical device. However, other information transmissions may be between software modules and/or hardware modules in different physical devices.

Claim 1:
A computer-implemented method comprising:
obtaining a first image from a video, a second image from the video, and an optical flow between the first image and the second image (<NUM>);
determining that a second pixel of the second image corresponds to a displacement of a first pixel of the first image to the second image according to the optical flow (<NUM>);
determining, by a depth model, (i) a first depth image based on the first image and (ii) a second depth image based on the second image (<NUM>);
determining (i), based on the first depth image, a first depth associated with the first pixel and (ii), based on the second depth image, a second depth associated with the second pixel (<NUM>);
determining (i) a first three-dimensional (3D) point based on the first depth associated with the first pixel and (ii) a second 3D point based on the second depth associated with the second pixel (<NUM>);
determining, by a scene flow model and based on the first 3D point, a scene flow representing a 3D motion of the first 3D point between the first image and the second image (<NUM>);
determining, for the first pixel, an induced pixel position based on a post-flow 3D point that represents the first 3D point after a displacement according to the scene flow (<NUM>);
determining (i) a pixel flow loss value based on a comparison of the induced pixel position to a position of the second pixel and (ii) a depth loss value based on a comparison of the post-flow 3D point to the second 3D point (<NUM>); and
adjusting one or more parameters of one or more of the depth model or the scene flow model based on the pixel flow loss value and the depth loss value (<NUM>);
characterized in that the scene flow model is specific to the video and comprises a machine learning model, and in that the scene flow is determined by providing as input to the scene flow model (i) the first 3D point and (ii) a time value associated with the first depth image.