Visual tracking by colorization

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for performing visual tracking. In one aspect, a method comprises receiving: (i) one or more reference video frames, (ii) respective reference labels for each of a plurality of reference pixels in the reference video frames, and (iii) a target video frame. The reference video frames and the target video frame are processed using a colorization machine learning model to generate respective pixel similarity measures between each of (i) a plurality of target pixels in the target video frame, and (ii) the reference pixels in the reference video frames. A respective target label is determined for each target pixel in the target video frame, comprising: combining (i) the reference labels for the reference pixels in the reference video frames, and (ii) the pixel similarity measures.

BACKGROUND

This specification relates to processing data using machine learning models.

Machine learning models receive an input and generate an output, e.g., a predicted output, based on the received input. Some machine learning models are parametric models and generate the output based on the received input and on values of the parameters of the model.

Some machine learning models are deep models that employ multiple layers of models to generate an output for a received input. For example, a deep neural network is a deep machine learning model that includes an output layer and one or more hidden layers that each apply a non-linear transformation to a received input to generate an output.

SUMMARY

This specification describes a system implemented as computer programs on one or more computers in one or more locations that performs visual tracking.

According to a first aspect there is provided a method comprising receiving: (i) one or more reference video frames, (ii) respective reference labels for each of a plurality of reference pixels in the reference video frames, and (iii) a target video frame; processing the reference video frames and the target video frame using a colonization machine learning model to generate respective pixel similarity measures between each of (i) a plurality of target pixels in the target video frame, and (ii) the reference pixels in the reference video frames, wherein the colonization machine learning model is trained to generate pixel similarity measures wherein a respective estimated color of each of target pixel in the target video frame is defined by combining: (i) actual colors of each of the reference pixels in the reference video frames, and (ii) the pixel similarity measures; and determining a respective target label for each target pixel in the target video frame, comprising: combining (i) the reference labels for the reference pixels in the reference video frames, and (ii) the pixel similarity measures.

The reference pixels in the reference video frames may comprise a proper subset of the pixels in the reference video frames.

The reference video frames and the target video frames may be decolorized prior to being processed by the colorization machine learning model.

The colorization machine learning model may include an embedding neural network, and wherein processing the reference video frames and the target video frame using the colorization machine learning model to generate respective pixel similarity measures between each of (i) the target pixels in the target video frame, and (ii) the reference pixels in the reference video frames may comprises: providing the reference video frames and the target video frame as an input to the embedding neural network; processing the input in accordance with current values of embedding neural network parameters to generate a respective embedding of each of (i) the target pixels in the target video frame, and (ii) the reference pixels in the reference video frames; and generating the respective pixel similarity measures between each of (i) the target pixels in the target video frame, and (ii) the reference pixels in the reference video frames using the embeddings.

The embedding neural network may comprise one or more convolutional neural network layers.

Generating the respective pixel similarity measures between each of (i) the target pixels in the target video frame, and (ii) the reference pixels in the reference video frames using the embeddings may comprise: generating the pixel similarity measure between a target pixel and a reference pixel using a similarity measure between the embedding of the target pixel and the embedding of the reference pixel.

A label for a pixel may comprise data indicating, for each of multiple possible categories, a respective likelihood that the pixel corresponds to the category.

A label for a pixel may comprise data indicating, for each of multiple possible key points, a respective likelihood that the pixel corresponds to the key point.

The colorization machine learning model may be trained by a plurality of operations comprising: obtaining a plurality of training examples, wherein each training example comprises: (i) one or more training reference video frames, (ii) a training target video frame, and (iii) actual colors of each pixel in the training reference video frames and the training target video frame; processing the training reference video frames and the training target video frame using the colorization machine learning model to generate respective similarity measures between: (i) each pixel in the training target video frame, and (ii) each pixel in the training reference video frames; determining estimated colors of each pixel in the training target video frame by combining: (i) the actual colors of each pixel in the training reference video frames, and (ii) the similarity measures; and adjusting the current values of the embedding neural network parameters based on a difference between: (i) the actual colors of each pixel in the training target video frame, and (ii) the estimated colors of each pixel in the training target video frame.

Adjusting the current values of the embedding neural network parameters may comprise: determining a gradient of a loss function, wherein the loss function depends on the difference between: (i) the actual colors of each pixel in the training target video frame, and (ii) the estimated colors of each pixel in the training target video frame; and adjusting the current values of the embedding neural network parameters using the gradient.

The reference labels for the reference pixels in one or more of the reference video frames may have been previously determined using the colorization machine learning model.

The target labels may be used to track a position of an object in the reference video frames to the target video frame.

According to a second aspect there is provided a system, comprising: a data processing apparatus; and a memory in data communication with the data processing apparatus and storing instructions that cause the data processing apparatus to perform the operations of the respective method of the first aspect.

According to a third aspect there is provided one or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform the operations of the respective method of the first aspect.

The subject matter described in this specification can be implemented in particular embodiments so as to realize one or more of the following advantages. The system described in this specification can be trained to perform visual tracking using raw, unlabeled color video data. Therefore, the large amount of unlabeled color video data which is readily available (e.g., on video sharing websites) can be exploited to train the system described in this specification. In contrast, some conventional visual tracking systems must be trained using manually labeled video data (e.g., where a human has manually labelled the pixels of the video frames). Manually labeling video data is tedious, time consuming, and difficult (e.g., because many video frame rates exceed 24 frames-per-second, thereby requiring manual labeling of at least 24 video frames to obtain one second of manually labeled data). Therefore the system described in this specification enables more efficient use of computational resources, particularly memory resources, by enabling unlabeled color video data stored in a memory to be directly used as training data (i.e., for training a colorization machine learning model to be used in visual tracking). Moreover, the system described in this specification can perform visual tracking more effectively (e.g., with a higher accuracy) than some conventional systems (e.g., systems based on optical flow methods).

DETAILED DESCRIPTION

This specification describes a system implemented as computer programs on one or more computers in one or more locations that performs visual tracking in sequences of video frames. Visual tracking refers to determining a position of an object (or other point of interest) in a video frame given the position of the object (or other point of interest) in one or more other video frames.

The system can learn to perform visual tracking without manual human supervision using unlabeled videos, i.e., where a person has not manually annotated pixel labels or tracking data on the video frames. In particular, the system automatically learns to track visual regions by learning to colorize a gray-scale “target” video frame in a video by copying colors from one or more “reference” video frames in the video. By learning to perform colorization in this manner, the system learns a “pointing” mechanism that points from pixels in the target video frame to corresponding pixels in the reference video frames in order to copy the right colors. Once the system is trained, the learned pointing mechanism acts as a tracker across time that can be used to perform visual tracking.

These features and other features are described in more detail below.

FIG. 1shows an example visual tracking system100. The visual tracking system100is an example of a system implemented as computer programs on one or more computers in one or more locations in which the systems, components, and techniques described below are implemented.

The visual tracking system100is configured to receive an input including (i) one or more reference video frames102, (ii) respective reference labels104for the pixels from the reference video frames102, and (iii) a target video frame106. The system100processes the input to generate respective target labels108for the pixels from the target video frame106. The reference video frames102and the target video frame106may be consecutive video frames from a video (e.g., a video depicting the natural world or a computer-generated video).

In some implementations, a label for a pixel (e.g., the reference labels104or the target labels108) indicates, for each of multiple possible categories, a respective likelihood that the pixel corresponds to the category. For example, the categories may include multiple different objects (e.g., people, cars, animals, and the like) and a background category. The label may be represented as a vector of numerical values representing the likelihoods that the pixel corresponds to each of the possible categories. In some implementations, a label for a pixel indicates, for each of multiple possible key points, a respective likelihood that the pixel corresponds to the key point. For example, the key points may include human pose key points (e.g., the position of the head, the shoulders, the elbows, and the like). The label may be represented as a vector of numerical values representing the likelihoods that the pixel corresponds to each of the possible key points.

By generating the target labels108for the target video frame106from the reference labels104for the reference video frames102, the system100can perform visual tracking. Visual tracking refers to determining a position of an object (or other point of interest) in the target video frame106given the position of the object (or other point of interest) in the reference video frames102. For example, the system100may perform visual tracking by determining the position of a particular person in the target video frame106(e.g., as defined by the target labels108) given the position of the particular person in the reference video frames102(e.g., as defined by the reference labels104).

The reference labels104may be generated by manual annotation, that is, by a person assigning respective labels to the pixels of the reference video frames102(e.g., using a non-automated or semi-automated annotation procedure). In some cases, the system100receives reference video frames and corresponding reference labels for the initial video frames of a video, and iteratively determines target labels for each subsequent video frame of the video. More specifically, after determining target labels108for a target video frame106, the system may determine the target video frame to be a reference video frame and the corresponding target labels to be reference labels. Thereafter, the system100may use the newly determined reference video frame and reference labels to generate target labels for subsequent video frames. In this manner, the system100may visually track the position of an object (e.g., a particular person, as described earlier) through every video frame of a video.

Generally, the reference video frames102and the target video frame106may be color video frames, that is, video frames where each pixel is associated with data defining a color for the pixel. For example, the reference video frames102and the target video frame106may be RGB video frames, where each pixel is associated with respective intensities of a red color channel, a green color channel, and a blue color channel. Prior to processing the reference video frames102and the target video frame106, the system100partly or fully “decolorizes” the reference video frames102and the target video frame106, e.g., by converting them to a non-color format. For example, prior to processing the reference video frames102and the target video frame106, the system100may convert them to a grayscale format.

To generate the target labels108for the target video frame106, the system100processes the decolorized reference video frames102and the decolorized target video frame106using a colorization machine learning model110to generate respective pixel similarity measures112between the pixels of the target video frame106and the pixels of the reference video frames102. The pixel similarity measure112between a target pixel from the target video frame106and a reference pixel from a reference video frame102may be represented as a number. As will be described in more detail later, the system100includes a colorization training subsystem114which trains the colorization machine learning model110to generate pixel similarity measures112that can be used to “colorize” the decolorized target video frame106using the colors of the pixels from the reference video frames102.

The colorization machine learning model110generates the pixel similarity measures112by providing the decolorized reference video frames102and the decolorized target video frame106as an input to an embedding neural network116. The embedding neural network116is configured to process the input in accordance with current values of embedding neural network parameters to generate a respective embedding of each of the pixels from the reference video frames102(i.e., the reference pixel embeddings118) and of each of the pixels of the target video frame106(i.e., the target pixel embeddings120). An embedding of a pixel refers to a representation of the pixel as an ordered collection of numerical values, for example, as a vector or a matrix of numerical values. The system100generates the pixel similarity measures112using the reference pixel embeddings118and the target pixel embeddings120. For example, for a given target pixel from the target video frame106and a given reference pixel from a reference video frame102, the system100may generate the pixel similarity measure112between the target pixel and the reference pixel based on a similarity measure (e.g., an inner product) between the reference pixel embedding118and the target pixel embedding120.

The system100provides the pixel similarity measures112and the reference labels104as an input to a label propagation engine122which is configured to generate the target labels108using the pixel similarity measures112and the reference labels104. For example, the label propagation engine122may generate the target labels108for the target video frame106by “propagating” the reference labels104from the reference video frames102to the target video frame106in accordance with the pixel similarity measures112. More specifically, the label propagation engine122may generate a target label108for a given target pixel from the target video frame106as a linear combination of the reference labels104where the reference labels104are weighted by the pixel similarity measures112.

An example process for performing visual tracking by determining the target labels108for the target video frame106is described in more detail with reference toFIG. 5.

To enable the system100to effectively generate target labels108for target video frames106, the system100includes a colorization training subsystem114which is configured to train the embedding neural network116. The training subsystem114trains the embedding neural network116over multiple training iterations to determine trained values of the embedding neural network parameters from initial values of the embedding neural network parameters. The training subsystem114can train the embedding neural network116on large amounts of readily available unlabeled color video data without requiring manual human supervision (e.g., without requiring a human to manually annotated pixel labels on the video data).

At each training iteration, the training subsystem114obtains reference video frames102and a target video frame106which are represented in a color format (e.g., as RGB video frames). More specifically, each reference pixel from a reference video frame102is associated with a reference color124, and each target pixel from the target video frame106is associated with a target color130. For example, when the color format is an RGB color format, the reference color124associated with a reference pixel (or the target color130associated with a target pixel) may be represented as a three-dimensional vector, with different dimensions corresponding to the red, green, and blue color channels respectively. The reference video frames102and the target video frame106obtained by the training subsystem114are not necessarily associated with pixel labels (e.g., reference labels104or target labels108).

The training subsystem114decolorizes the reference video frames102and the target video frame106(e.g., by converting them to grayscale) and processes them using the colorization machine learning model110to generate the pixel similarity measures112. The training subsystem114provides the pixel similarity measures112and the reference colors124as an input to a color propagation engine128which is configured to process the input to generate estimated target colors126for the pixels from the target video frame106. The estimated target colors126include an estimated color (e.g., represented in an RGB format, as described earlier) for each target pixel from the target video frame106. The color propagation engine128is configured to operate analogously to the label propagation engine122, that is, by propagating the reference colors124from the reference video frames102to the target video frame106in accordance with the pixel similarity measures112to “colorize” the target video frame106. For example, the color propagation engine128may generate an estimated target color126for a given target pixel from the target video frame106as a linear combination of the reference colors124, where the reference colors124are weighted by the pixel similarity measures112.

After generating the estimated target colors126, the training subsystem114adjusts the current values of the embedding neural network parameters to cause the system100to colorize the target video frame106more accurately. More specifically, the training subsystem114adjusts the current values of the embedding neural network parameters based on a difference between: (i) the (actual) target colors130of the target pixels in the target video frame106, and (ii) the estimated target colors126of the target pixels in the target video frame106. The training subsystem114adjusts the current values of the embedding neural network parameters using a gradient132of a loss function134with respect to the current values of the embedding neural network parameters. The loss function134depends on an error between the actual target colors130of the target pixels and the estimated target colors126of the target pixels.

The pixel similarity measures112can operate as “pointers” from the target pixels of the target video frame106to the reference pixels of the reference video frames102, where the strength of a pointer from a target pixel to a reference pixel is determined by the magnitude of the corresponding pixel similarity measure112. By training the system100to colorize target video frames106from reference video frames102, the training subsystem114causes the colorization machine learning model110to learn to generate pixel similarity measures112which “point” from target pixels in the target video frame106to the right reference pixels in the reference video frames102. Once the colorization machine learning model110is trained, the learned “pointing” mechanism is used to propagate reference labels104from the reference video frames102to the target video frame106, thereby enabling visual tracking.

An example process for training the colorization machine learning model is described in more detail with reference toFIG. 6.

FIG. 2is an illustration of an example data flow200for using the visual tracking system100to track a positon of an object from a reference video frame202to a target video frame204.

The system100starts by decolorizing the reference video frame202and the target video frame204, e.g., by converting them to corresponding grayscale video frames.

The system100processes the reference video frame202and the target video frame204using the embedding neural network206to generate embeddings corresponding to the pixels of the reference video frame202and the target video frame204. The embedding neural network206is a convolutional neural network (CNN), that is, a neural network that includes one or more convolutional neural network layers.

The output of the embedding neural network206after processing an input video frame (e.g., the reference frame202or the target frame204) can be represented as a three-dimensional (3D) matrix of numerical values, with two “spatial” dimensions and one “channel” dimension. The embedding corresponding to a pixel of the input video frame at a particular spatial position (e.g., defined by (x,y) coordinates in the input video frame) is determined by extracting a portion of the embedding neural network output at the corresponding spatial position (i.e., along the channel dimension). In some cases, the spatial dimensionality of the embedding neural network output may be the same as the input video frame, e.g., the input video frame may have a spatial dimensionality of 256×256, and the embedding neural network output may have a spatial dimensionality of 256×256 with 200 channels. In some other cases, the spatial dimensionality of the embedding neural network output may be greater or lesser than the spatial dimensionality of the input video frame, e.g., the input video frame may have a spatial dimensionality of 256×256, and the embedding neural network output may have a spatial dimensionality of 50×50 with 200 channels.

The embedding neural network206can have any appropriate neural network architecture. In one example, the embedding neural network architecture may include a ResNet-18 neural network architecture followed by a five layer 3D convolutional neural network. The spatial locations of each pixel in an input video frame (e.g., represented as respective 2D vectors in the range [−1,1]) may be provided as an intermediate input to the embedding neural network.

To track the position of the object208(i.e., illustrated inFIG. 2as a triangle) from the reference frame202to the target frame204, the system100determines respective similarity measures between embeddings corresponding to pixels from the target frame204and embeddings corresponding to pixels from the reference frame202. In this example, the system100may identify the position of the object208in the target frame204based on the similarity measure between the embedding fiof the object208in the reference frame202and the embedding fjin the target frame204. The system100trains the embedding network206to generate embeddings210that can be used to colorize the target frame204by generating predicted colors212of the target frame204based on the reference colors214of the reference frame202.

FIG. 3andFIG. 4illustrate examples of the performance gains that can be achieved by using the visual tracking system described in this specification to perform a video segmentation task. In a video segmentation task, a semantic segmentation of one or more initial video frames in a video is provided, and the tracking system propagates the semantic segmentation to the remaining video frames in the video. A semantic segmentation of a video frame specifies a label corresponding to each pixel of the video frame.

FIG. 3illustrates a graph300showing the performance of various tracking systems as a function of time (i.e., video frame number) while performing video segmentation. The tracking system302refers to the tracking system described in this specification. The tracking system304uses optic flow techniques to performing tracking. The tracking system306performs tracking by associating each pixel in a video frame with the corresponding pixel at the same spatial position in the preceding video frame. The horizontal axis of the graph300identifies the frame number of the video frame being segmented, and the vertical axis identifies the mean performance of each tracking system at segmenting video frames. The performance of a tracking system at segmenting a video frame can be determined as the overlap between: (i) the actual segmentation of the video frame, and (ii) the segmentation of the video frame determined by the tracking system. It can be appreciated that the tracking system302described in this specification outperforms each baseline tracking method.

FIG. 4shows the performance of various “unsupervised” and “supervised” tracking systems while performing video segmentation. Performance is evaluated in the “Segment” column of the table400based on the overlap between the actual and estimated segmentations of each video frame, and in the “Boundary” column based on the similarity between the borders of respective regions in the actual and estimated segmentation of each video frame. A supervised tracking method refers to a tracking method that is trained with the benefit of labeled images, e.g., images associated with known tracking data or semantic segmentations. An unsupervised tracking method (such as tracking method described in this specification) is trained without using labeled images. In some cases, the performance of supervised tracking methods can be expected to upper-bound the performance of unsupervised tracking methods.

The “Identity” tracking method402performs tracking by associating each pixel in a video frame with the corresponding pixel at the same spatial position in the preceding video frame (as described earlier). The “Single Image Colorization” tracking method404performs tracking in a similar manner as the system described in this specification, except the pixel embeddings are generated as an intermediate output of a neural network trained to perform single image colorization (i.e., by processing a grayscale image to generate a color image). The “Optical Flow (Coarse-to-Fine)” tracking method406uses optical flow techniques to perform tracking and is described with reference to Liu, C., et al.: “Beyond pixels: exploring new representations and applications for motion analysis”, Phi) thesis, Massachusetts Institute of Technology (2009). The “Optical Flow (FlowNet2)” tracking method408uses optical flow techniques to perform tracking and is described with reference to Ilg, E., et al.: “Flownet 2.0: Evolution of optical flow estimation with deep networks”, IEEE Conference no Computer Vision and Pattern Recognition (CVPR), Volume 2 (2017). The “Ours” tracking method410refers to the tracking method described in this specification. The “Fully Supervised” tracking method412refers to the fully supervised tracking method described with reference to, e.g., Yang, L., et al.: “Efficient video object segmentation via network modulation”, arXiv:1802.01218 (2018). It can be appreciated that the tracking method described in this specification achieves a performance that is closer to that of the fully supervised tracking method that any of the other unsupervised tracking methods.

FIG. 5is a flow diagram of an example process500for performing visual tracking. For convenience, the process500will be described as being performed by a system of one or more computers located in one or more locations. For example, a visual tracking system, e.g., the visual tracking system100ofFIG. 1, appropriately programmed in accordance with this specification, can perform the process500.

The system receives: (i) one or more reference video frames, (ii) respective reference labels for each of multiple reference pixels in the reference video frames, and (iii) a target video frame (502). Generally, the reference video frames and the target video frame are drawn from the same video. The reference video frames may precede the target video frame in the video, follow the target video frame in the video, or a combination of both, i.e., some of the reference video frames may precede the target video frame, while the remaining reference video frames may follow the target video frame. The system may receive reference labels for every pixel in the reference video frames, or for only a proper subset of the pixels in the reference video frames (e.g., for only pixels corresponding to a particular object depicted in the reference video frames). The reference labels may have been determined by manual annotation by a person, or may have been previously generated using the process500. Generally, the system decolorizes the reference video frames and the target video frame by removing some or all of the data defining the colors of the pixels, e.g., by converting the video frames to a grayscale format.

The system processes the reference video frames and the target video frames using a colorization machine learning model to generate respective pixel similarity measures between each of (i) multiple target pixels in the target video frame and (ii) multiple reference pixels in the reference video frames (504). In particular, the system provides the reference video frames and the target video frame as respective inputs to an embedding neural network that is configured to process an input video frame to generate an output that defines a respective embedding for each pixel in the input video frame. The system may determine the pixel similarity measure Aijbetween a target pixel j from the target video frame and a reference pixel i from a reference video frame as:

Aij=exp⁡(fiT⁢fjT)∑k⁢exp⁡(fkT⁢fjT)(1)
where fiT∈Dis the transpose of the embedding for reference pixel i, fjis the embedding for target pixel j, T is a temperature parameter (e.g., T=0.5, or T=1), and the sum is over each reference pixel k. The colorization machine learning model (in particular, the embedding neural network) is trained to generate pixel similarity measures having the property that the color of each target pixel can be effectively estimated by combining the: (i) the actual colors of the reference pixels, and (ii) the pixel similarity measures. Training of the colorization machine learning model is described in more detail with reference toFIG. 6.

The system determines a respective target label for each target pixel in the target video frame based on (i) the reference labels for the reference pixels, and (ii) the pixel similarity measures between the reference pixels and the target pixels (506). For example, the system may determine the target label for a target pixel j in the target video frame as:

lj=∑t⁢Aij·li(2)
where represents the pixel similarity measure between target pixel j and reference pixel i (as described with reference to equation (1)), lirepresents the reference label for reference pixel i, and the sum is over each reference pixel i in the reference video frames.

In some implementations, the reference labels indicate whether each reference pixel is included in a particular object, and by determining the target labels, the system “tracks” the object from the reference video frames to the target video frame. In some other implementations, the reference labels define a semantic segmentation of the reference video frames by indicating a respective category (e.g., object category) for each reference pixel, and by determining the target labels, the system propagates the semantic segmentation to the target video frame.

FIG. 6is a flow diagram of an example process600for training a colorization machine learning model. For convenience, the process600will be described as being performed by a system of one or more computers located in one or more locations. For example, a visual tracking system, e.g., the visual tracking system100ofFIG. 1, appropriately programmed in accordance with this specification, can perform the process600.

The system obtains one or more training examples (602). Each training example includes: (i) one or more training reference video frames, (ii) a training target video frame, and (iii) actual colors of each of the pixels in the training reference video frames and the training target video frame. The system may obtain the training examples, e.g., by randomly sampling a predetermined number of training examples from a set of training data that includes multiple training examples.

The colors of the pixels in the training reference video frames and the training target video frame can be represented in any of a variety of ways. In one example, the color of a pixel can be represented “continuously”, e.g., as a 3D vector of red, green, and blue color values that can assume values in a continuous range of possible values. In another example, the color of a pixel can be represented “discretely”, e.g., as a one-hot vector indicating a particular “reference” color value from a predetermined set of possible reference color values. The set of possible reference color values may be determined by clustering a set of vectors representing the colors of pixels in a set of images or videos, e.g., using a k-means or expectation-maximization clustering technique. A pixel may be determined to correspond to the reference color that is closest to its actual color.

The system processes the training reference video frames and the training target video frame using the colorization machine learning model to generate respective pixel similarity measures between: (i) each pixel in the training target video frame, and (ii) each pixel in the training reference video frames (604). An example process for generating such similarity measures using an embedding neural network of the colorization machine learning model is described in more detail with reference to step504ofFIG. 5.

The system determines estimated colors of each pixel in the training target video frame by combining: (i) the actual colors of each pixel in the training reference video frames, and (ii) the pixel similarity measures (606). For example, the system may determine the estimated colorof a target pixel j in the training target video frame as:

cj^=∑i⁢Aij·ci(3)
where Aijrepresents the pixel similarity measure between target pixel j and reference pixel i, represents the color of reference pixel i, and the sum is over each reference pixel i in the training reference video frames.

The system adjusts the current values of the embedding neural network parameters of the colorization machine learning model based on a difference between: (i) the actual colors of each pixel in the training target video frame, and (ii) the estimated colors of each pixel in the training target video frame (608). For example, the system may adjust the current values of the embedding neural network parameters using a gradient of a loss function with respect to the current values of the embedding neural network parameters. The loss function may be, e.g., a cross-entropy loss between the actual colors and the estimated colors of the pixels of the training target video frame. The system may determine the gradient using, e.g., a backpropagation technique. The system may use the gradient to adjust the current values of the embedding neural network parameters using the update rule of any appropriate gradient descent optimization algorithm, e.g., RMSprop or Adam.