Patent Description:
An object localization system is a system having multiple trainable parameters that is configured to process an image to localize (i.e., identify the locations of) objects depicted in the image. An object localization system can be trained using a set of training examples, where each training example includes: (i) a training image, and (ii) training object localization data defining the locations of objects depicted in the training image.

Background prior art can be found in <NPL>, which describes "actionness maps", i.e. maps of generic action given raw frames, each value of an actionness map describing the confidence of (the frame) containing an action instance at that location. The actionness maps can be used for action detection in videos.

This specification describes training systems implemented as computer programs on one or more computers in one or more locations that can be used to train object localization systems.

According to a first aspect there is provided a method performed by one or more data processing apparatus, as recited in claim <NUM>, the method comprising among others: for each video frame of a video comprising a plurality of video frames, obtaining object localization data defining locations of objects depicted in the video frame; processing the video to generate, for each video frame of the plurality of video frames, a corresponding optical flow image characterizing a displacement of each pixel of the video frame between the video frame and a subsequent video frame in the video; and using: (i) the optical flow images, and (ii) the object localization data defining the locations of objects depicted in the plurality of video frames corresponding to the optical flow images, training an optical flow object localization system to process an optical flow image to generate object localization data defining locations of objects depicted in a video frame corresponding to the optical flow image.

Obtaining object localization data defining locations of objects depicted in the video frame comprises processing the video frame using a known object localization system. The known object localization system may be a previously trained object localization system. For example the known object localization system may be configured to process the video frame to generate data defining locations of objects depicted in the video frame that are from a predetermined set of object classes. In some implementations the known object localization system may comprise a neural network.

Additionally, obtaining object localization data defining locations of objects depicted in the video frame may comprise obtaining object localization data defining locations of objects depicted in the video frame which is manually annotated by a person. The object localization data defining locations of objects depicted in the video frame may comprise data defining bounding boxes around locations of objects depicted in the video frame. More generally the object localization data may comprise data defining coordinates of the corners of a shape bounding a located object.

Processing the video to generate, for each video frame of the plurality of video frames, a corresponding optical flow image may comprise processing the video using a direct numerical optimization method to generate, for each video frame of the plurality of video frames, the corresponding optical flow image. The direct numerical optimization method may be a method which fits an optical flow field to pixel data from the video frames, e.g. using local space/time derivatives or by region-matching, in either the time or the frequency domain. Additionally or alternatively the optical flow object localization system may comprise a neural network.

The method may further comprise disregarding object localization data which has been generated for objects which are stationary or which have less than a threshold degree of movement with respect to their background. Thus the method may further comprise, for each object location defined by the object localization data for the video frame corresponding to the optical flow image, determining a respective variance of optical flow data in the optical flow image at (over) the object location. The method may then modify the object localization data used to train the optical flow object localization system by removing data defining object locations where the variance of the optical flow data in the optical flow image at the object location does not satisfy a (minimum) threshold.

The method, as recited in claim <NUM>, comprises processing a video comprising a plurality of video frames to generate, for each video frame of the plurality of video frames, a corresponding optical flow image characterizing a displacement of each pixel of the video frame between the video frame and a subsequent video frame in the video; for each optical flow image, processing the optical flow image using an optical flow object localization system to generate object localization data defining locations of objects depicted in the video frame corresponding to the optical flow image; and using: (i) the plurality of video frames, and (ii) the object localization data generated by the optical flow object localization system by processing the optical flow images corresponding to the plurality of video frames, training a second object localization system, e.g. a novel object localization system, to process a video frame to generate object localization data defining locations of objects depicted in the video frame.

The optical flow object localization system is trained using: (i) optical flow images, and (ii) object localization data defining locations of objects depicted in video frames corresponding to optical flow images. The object localization data defining locations of objects depicted in a video frame corresponding to an optical flow image is obtained by processing the video frame using a known object localization system, as previously described.

Thus the method comprises using a known or trained object localization system, which may fail to detect (localize) certain objects, e.g. certain classes of object, and using this to provide object localization data for training an optical flow object localization system. The optical flow object localization system, which works on a different basis, namely optical flow, may learn to localize objects which the known/trained object localization system missed. Thus the optical flow object localization system may be used to generate training data which can be used to train a second object localization system e.g. to improve the known or trained object localization system e.g. by further training, and/or to train a "novel" object localization system (which is not based on optical flow).

Thus, training the novel object localization system comprises, for one or more of the plurality of video frames: processing the video frame using a known object localization system to generate object localization data defining locations of objects depicted in the video frame, wherein the known object localization system is trained to localize objects that are from a predetermined set of known object classes; identifying locations of novel objects depicted in the video frame, wherein the location of a novel object is: (i) included in the object localization data generated by the optical flow object localization system for the video frame, and (ii) not included in the object localization data generated by the known object localization system for the video frame; and training the novel object localization system to process the video frame to generate object localization data which comprises data defining the locations of the novel objects depicted in the video frame.

The novel object localization system may be trained to process the video frame to generate object localization data which comprises the object localization data generated by the known object localization system by processing the video frame, in addition to the data defining the locations of the novel objects depicted in the video frame.

The method may further comprise processing an image using the trained second object localization system to generate object localization data defining locations of objects depicted in the image, wherein the image is not a video frame. Thus once trained the second object localization system may be applied to any sort of image.

This specification describes a training system that can train an object localization system (referred to in this specification as a "novel object localization system") to localize "novel" objects in images. A novel object in an image refers to an object that is missed (i.e., not localized) by an existing object localization system (referred to in this specification as a "known object localization system"). A novel object localization system can be used to improve the accuracy of the known object localization system by identifying where the known object localization system currently fails. In a particular example, novel objects localized by the novel object localization system can be used as training data to improve the accuracy of the known object localization system in localizing objects in images. This is a technical improvement in the field of image processing.

This specification also describes a training system that can train an object localization system (referred to in this specification as an "optical flow object localization system") to process an optical flow image to localize objects depicted in a video frame corresponding to the optical flow image. The optical flow object localization system trained in this manner can be used to generate training data for training the novel object localization system. Moreover, the training system can efficiently generate training data for training the optical flow object localization system by transferring object localization data from video frames to corresponding optical flow images. For example, the training system can process video frames using the known object localization system to generate object bounding boxes around objects depicted in the video frames, and transfer the generated object bounding boxes to optical flow images corresponding to the video frames. In this manner, the training system can automatically generate training data for the optical flow object localization system rather than requiring object localization data to be manually annotated on optical flow images, which is a time-consuming and difficult process. This is also a technical improvement in the field of image processing.

An object localization system is a system having one or more trainable parameters (e.g., a neural network) that is configured to process an image (e.g., an RGB image) to localize (i.e., identify the locations of) objects depicted in the image. An object localization system can be trained using a set of training examples, where each training example includes: (i) a training image, and (ii) training object localization data defining the locations of objects depicted in the training image. This specification describes a method for training a "novel" object localization system to localize objects that are currently missed (i.e., not localized) by a "known" object localization system. In some cases, the known object localization system may fail to localize an object because the object is rare or unusual (e.g., a space shuttle) and doesn't fit into any of a predetermined set of object categories the known object localization system is trained to localize. In some other cases, the known object localization system may fail to localize an object in an image if the object has a different visual appearance in the image than it does in the training images used to train the known object localization system. For example, the known object localization system may fail to localize a cat which is depicted in an image from a top-down perspective if the training images used to train the known object localization system all depict cats from a head-on perspective. Objects that are missed by the known object localization system are referred to in this specification as "novel objects".

The novel object localization system is trained to localize novel objects using training data generated by an optical flow object localization system. The optical flow object localization system is trained to process optical flow images which correspond to video frames of videos to localize objects depicted in the video frames. An optical flow image corresponding to a video frame describes the movement, but not the visual appearance (e.g., color, texture, and the like), of objects depicted in the video frame. Therefore, the optical flow object localization system localizes objects based on their movement rather than relying on their visual appearance.

The optical flow object localization system generates training data for the novel object localization system by processing optical flow images corresponding to training video frames to localize objects depicted in the training video frames based on their movement. Objects depicted in the training video frames which are: (i) localized by the optical flow object localization system, and (ii) not localized by the known object localization system, can be identified as novel objects. That is, an object may be identified as a novel object if it is localized by the optical flow object localization system based on its movement but is not localized by the known object localization system based on its visual appearance. The novel object localization system can be trained to process the training video frames to localize the identified novel objects depicted in the training video frames. After training, the novel object localization system can process images which include, but are not restricted to, video frames.

The optical flow object localization system is trained using: (i) optical flow images corresponding to video frames, and (ii) training object localization data indicating the locations of objects depicted in the video frames. In some cases, the training object localization data may be generated by a person manually annotating the locations of the objects depicted in the video frames. In some other cases, the training object localization data may be generated by processing the video frames using the known object localization system. The optical flow object localization system is trained to process the optical flow images corresponding to the video frames to localize the objects depicted in the video frames (i.e., as specified by the training object localization data).

These features and other features are described in more detail below.

<FIG> is a block diagram of an example novel object localization training system <NUM> and an example optical flow object localization training system <NUM>. The systems <NUM> and <NUM> are examples of systems implemented as computer programs on one or more computers in one or more locations in which the systems, components, and techniques described below are implemented.

To train the optical flow object localization system <NUM>, the system <NUM> obtains sequences of video frames <NUM> from one or more videos. Each of the video frames <NUM> is an image that can be represented as a two-dimensional (2D) array of pixels, where each pixel is represented as a vector of one or more values. For example, if the video frames are black- and-white images, each pixel can be represented as an integer or floating point number (i.e., a vector with one component) representing the brightness of the pixel. As another example, if the video frames are red-green-blue (RGB) images, each pixel can be represented as a vector with three integer or floating point components, which respectively represent the intensity of the red, green, and blue color of the pixel. In some other examples YUV or HSV color channels may be employed. The pixels of a video frame can be indexed by (x,y) coordinates, where x and y are integer values.

The system <NUM> processes the video frames <NUM> from each video using an optical flow engine <NUM> to generate a respective optical flow image <NUM> corresponding to each video frame <NUM>. The optical flow image corresponding to a video frame in a video characterizes the displacement of each pixel of the video frame between the video frame and a subsequent video frame in the video. The displacement of a pixel from a video frame to a subsequent video frame can be represented as a vector with two numerical (e.g., integer or floating point) components, with the first component representing the x-displacement of the pixel and the second component representing the y-displacement of the pixel. The optical flow image corresponding to a video frame can be represented as a 2D array of displacements which are indexed by the same (x,y) coordinates as the video frame. That is, a displacement at a particular (x,y) coordinate in an optical flow image represents the displacement of the pixel at the same (x,y) coordinate in the corresponding video frame. The optical flow image <NUM> corresponding to a video frame <NUM> from a video describes the movement, but not the visual appearance (e.g., color, texture, and the like) of objects depicted in the video frame <NUM>.

The system <NUM> obtains training object localization data <NUM> for each of the video frames <NUM>. The training object localization data <NUM> for a video frame defines the locations of objects depicted in the video frame. For example, the training object localization data <NUM> for a video frame <NUM> may define the locations of objects depicted in the video frame <NUM> by data defining the coordinates of bounding boxes around objects depicted in the video frame <NUM>. As another example, the training object localization data <NUM> for a video frame <NUM> may define the locations of objects depicted in the video frame <NUM> by data defining segmentations of objects depicted in the video frame <NUM>. The segmentation of an object depicted in a video frame can be represented as a list of the (x,y) coordinates of the pixels in the video frame which are included in the object.

The system <NUM> obtains the training object localization data <NUM> for a video frame <NUM> by processing the video frame <NUM> using the known object localization system <NUM>. As described earlier, the known object localization system <NUM> is trained to process images to localize objects depicted in the images. In some implementations, the system <NUM> additionally obtains the training object localization data <NUM> for a video frame <NUM> by accessing (i.e., from a data store) object localization data that has been manually generated by a person for the video frame <NUM>.

The system <NUM> uses a training engine <NUM> to train the optical flow object localization system <NUM> using: (i) the optical flow images <NUM>, and (ii) the training object localization data <NUM>. The optical flow object localization system <NUM> is configured to process an optical flow image derived from a video frame to generate object localization data which defines the locations of objects depicted in the video frame. The training engine <NUM> trains the optical flow object localization system <NUM> to process each optical flow image <NUM> to generate object localization data which matches the training object localization data <NUM> corresponding to the optical flow image <NUM>. An optical flow image corresponding to a video frame describes the movement, but not the visual appearance (e.g., color, texture, and the like), of objects depicted in the video frame. Therefore, the optical flow object localization system <NUM> localizes objects based on their movement rather than relying on their visual appearance.

The training engine <NUM> may be configured to train the optical flow object localization system <NUM> using any appropriate machine learning training technique (e.g., stochastic gradient descent). For example, the training engine <NUM> may be configured to train the optical flow object localization system <NUM> by iteratively (i.e., at each of multiple training iterations) adjusting the current parameter values of the optical flow object localization system <NUM>. An example process for training an optical flow object localization system <NUM> is described further with reference to <FIG>.

The novel object localization training system <NUM> is configured to train the novel object localization system <NUM> to process images (e.g., video frames) to generate object localization data defining the locations of novel objects depicted in the images (i.e., objects that are missed by the known object localization system <NUM>). As will be described in more detail below, the system <NUM> trains the novel object localization system <NUM> using an optical flow object localization system <NUM>. The system <NUM> may set the parameter values of the optical flow object localization system <NUM> to be the trained parameter values <NUM> of the optical flow object localization system <NUM> trained by the system <NUM>.

To train the novel object localization system <NUM>, the system <NUM> obtains sequences of video frames <NUM> from one or more videos (which may be the same as the video frames <NUM> used in training the optical flow object localization system <NUM>). The system <NUM> processes the video frames <NUM> from each video using the optical flow engine <NUM> to generate a respective optical flow image <NUM> corresponding to each video frame <NUM>. The system <NUM> processes each of the optical flow images <NUM> using the optical flow object localization system <NUM> to generate training object localization data <NUM> which defines the locations of objects depicted in the video frames <NUM>. Since the optical flow object localization system <NUM> localizes objects based on their movement rather than relying on their visual appearance, the object localization training data <NUM> can identify novel objects in the absence of training data from the original source characterizing their visual appearance (e.g., color, texture, and the like).

The system <NUM> uses a training engine <NUM> to train the novel object localization system <NUM> to localize novel objects using: (i) the video frames <NUM>, and (ii) the training object localization data <NUM> generated by the optical flow object localization system <NUM>. For example, to identify novel objects depicted in the video frames <NUM>, the system <NUM> may first process the video frames <NUM> using the known object localization system <NUM>. Then, the system <NUM> may identify an object as a novel object if it is: (i) included in the training object localization data <NUM> generated by the optical flow object localization system <NUM>, and (ii) not included in the object localization data generated by the known object localization system <NUM>. That is, the system <NUM> may identify an object as a novel object if it localized by the optical flow object localization system <NUM> based on its movement but not localized by the known object localization system <NUM> based on its visual appearance. In this example, the training engine <NUM> can train the novel object localization system <NUM> to process each video frame <NUM> to generate object localization data which includes the novel objects identified by the system <NUM> in the video frame <NUM>. In some cases, as will be described in more detail with reference to <FIG>, the system can train the novel object localization system <NUM> to generate object localization data which additionally includes the object localization data generated by the known object localization system <NUM> for the video frame.

The training engine <NUM> may be configured to train the novel object localization system <NUM> using any appropriate machine learning training technique (e.g., stochastic gradient descent). For example, the training engine <NUM> may be configured to train the novel object localization system <NUM> by iteratively (i.e., at each of multiple training iterations) adjusting the current parameter values of the novel object localization system <NUM>. An example process for training a novel object localization system <NUM> is described further with reference to <FIG>.

The novel object localization system <NUM> trained by the system <NUM> is not restricted to processing video frames. In particular, after being trained, the novel object localization system <NUM> can process any images (including, but not restricted to, video frames) to localize novel objects depicted in the images.

<FIG> is an example data flow <NUM> for identifying a novel object <NUM>-A in a video frame <NUM> using a known object localization system <NUM> and an optical flow object localization system <NUM>. The known object localization system <NUM> processes the video frame <NUM> to generate object localization data <NUM> which defines the locations of objects <NUM>-A and <NUM>-B. The optical flow object localization system <NUM> processes the optical flow image <NUM> corresponding to the video frame <NUM> to generate object localization data <NUM> which defines the locations of objects <NUM>-A and <NUM>-B. Object <NUM>-A can be identified as a novel object since it is localized by the optical flow object localization system <NUM> but not the known object localization system <NUM>. Object <NUM>-B is localized by the known object localization system <NUM> but not the optical flow object localization system <NUM>, potentially because object <NUM>-B is a stationary object that cannot be reliably localized from the optical flow image <NUM> (as will be described in more detail below).

<FIG> is a flow diagram of an example process <NUM> for training an optical flow object localization system. For convenience, the process <NUM> will be described as being performed by a system of one or more computers located in one or more locations. For example, an optical flow object localization training system, e.g., the optical flow object localization training system <NUM> of <FIG>, appropriately programmed in accordance with this specification, can perform the process <NUM>.

The system obtains respective sequences of video frames from one or more videos (<NUM>). The videos may be captured by digital video cameras or may be computer-generated. Each of the video frames is an image that can be represented as a 2D array of pixels, where each pixel is represented as a vector of one or more color values (e.g., RGB color values).

The system obtains respective training object localization data for each of the video frames (<NUM>). The training object localization data for a video frame defines the locations of objects depicted in the video frame. For example, the training object localization data for a video frame may define the locations of the objects depicted in the video frame by data defining the coordinates of respective bounding boxes around objects depicted in the video frame. As another example, the training object localization data for a video frame may define the locations of objects depicted in the video frame by data defining respective segmentations of objects depicted in the video frame. In some implementations, the system obtains the training object localization data for a video frame by processing the video frame using the known object localization system. The known object localization system may be configured to process a video frame in accordance with trained values of known object localization system parameters to generate the object localization data for the video frame. In some other implementations, the system can obtain training object localization data for a video frame which is manually annotated by a person.

The system generates a respective optical flow image corresponding to each of the video frames (<NUM>). The system can generate the optical flow image corresponding to a video frame using any appropriate method. For example, the system can process a video frame from a video and a video frame subsequent to the video frame in the video using a direct numerical optimization method (e.g., the Lucas-Kanade method) to generate the optical flow image corresponding to the video frame. Thus a direct numerical optimization method may be a method in which an optical flow defined, for example, by a flow velocity vector, is fitted to flow data defined by the video frames by a direct numerical optimization method. For example in the Lucas-Kanade method a locally constant velocity is assumed. More generally a differential technique may be employed to determine optical flow based on local and/or global first and optionally second derivatives of pixel values with respect to time and/or x- and y-displacement.

As another example, the system can process a video frame from a video and a video frame subsequent to the video frame in the video using a neural network that is configured to generate an output defining the optical flow image corresponding to the video frame.

An optical flow image may comprise, for example a set of x- and y- pixel displacement values for each pixel location. Alternatively, for example, it may comprise x- and y-components of a vector mapping motion of a pixel from one frame to one or more later frames. The optical flow may be unidirectional, either forwards or backwards, or bidirectional. Optionally a global, e.g. mean, motion estimation component may be subtracted to compensate for camera/sensor motion.

The system trains an optical flow object localization system using: (i) the optical flow images, and (ii) the training object localization data defining the locations of objects depicted in the video frames corresponding to the optical flow images (<NUM>). More specifically, the system trains the optical flow object localization system to process each optical flow image to generate object localization data which matches the training object localization data defining the locations of objects depicted in the video frame corresponding to the optical flow image. The system can train the optical flow object localization system by iteratively adjusting the parameter values of the optical flow object localization system over multiple iterations using a machine learning training technique (e.g., stochastic gradient descent). In some cases, the optical flow object localization system can be implemented as a neural network. An example of an object localization neural network, and a method for training an object localization neural network, is described with reference to: S. Girshick, and J. Sun, "Faster R-CNN: towards real-time object localization with region proposal networks", Advances in Neural Information Processing Systems (NIPS), <NUM>. Any other appropriate implementation of an object localization system can also be used.

In some cases, the training object localization data may define the location of an object in a video frame which the optical flow object localization system will be unable to reliably localize from the optical flow image corresponding to the video frame. For example, the optical flow object localization system may be unable to reliably localize an object depicted in a video frame which is stationary (i.e., not moving) relative to the background depicted in the video frame. In this example, the portion of an optical flow image corresponding to a stationary object will not distinctively characterize the stationary object, because the displacement vectors in that portion of the optical flow image may all be approximately zero (or some other constant value). The system may modify the training object localization data used to train the optical flow object localization system by removing data defining the locations of objects which the optical flow object localization system will be unable to reliably localize from the optical flow images.

To determine if the optical flow object localization system will be unable to reliably localize an object specified by the training object localization data, the system can determine the variance of the portion of the optical flow image characterizing the object. If the variance does not satisfy a predetermined (minimum) threshold, the system may determine that the optical flow object localization system will be unable to reliably localize the object, and thereafter refrain from training the optical flow object localization system to localize the object.

<FIG> is a flow diagram of an example process <NUM> for training a novel object localization system. For convenience, the process <NUM> will be described as being performed by a system of one or more computers located in one or more locations. For example, a novel object localization training system, e.g., the novel object localization training system <NUM> of <FIG>, appropriately programmed in accordance with this specification, can perform the process <NUM>.

The system obtains respective sequences of video frames from one or more videos (<NUM>). The videos may be captured by digital video cameras or may be computer-generated. Each of the video frames is an image that can be represented as a 2D array of pixels, where each pixel is represented as a vector of one or more color values (e.g., RGB color values). In some cases, the obtained video frames may be the same video frames used to train the optical flow object localization system (as described with reference to <FIG>).

The system generates a respective optical flow image corresponding to each of the video frames (<NUM>). The system can generate the optical flow image corresponding to a video frame using any appropriate method. A few example methods for generating an optical flow image corresponding to a video frame are described with reference to <NUM>.

The system processes each of the optical flow images using a trained optical flow object localization system to generate respective training object localization data defining the locations of objects depicted in the video frames corresponding to the optical flow images (<NUM>). The optical flow object localization system may be trained, for example, by the process <NUM> described with reference to <FIG>. The training object localization data generated by the optical flow object localization system for an optical flow image may be, for example, data defining respective bounding box coordinates around objects depicted in the video frame, or respective segmentations of objects depicted in the video frame.

The system trains the novel object localization system using: (i) the video frames, and (ii) the training object localization data generated by the optical flow object localization system by processing the optical flow images corresponding to the video frames (<NUM>). For example, to identify novel objects depicted in the video frames, the system may first process the video frames using the known object localization system. Then, the system may identify an object as a novel object if it is: (i) included in the training object localization data generated by the optical flow object localization system, and (ii) not included in the object localization data generated by the known object localization system. That is, the system may identify an object as a novel object if it is localized by the optical flow object localization system based on its movement but not localized by the known object localization system based on its visual appearance. In this example, the system can train the novel object localization system to process each video frame to generate object localization data which includes the novel objects identified by the system in the video frame.

In some implementations, the system may train the novel object localization system to process each video frame to generate object localization data which includes both: (i) the novel objects identified by the system in the video frame, and (ii) the objects localized by the known object localization system in the video frame. The system may further train the novel object localization system to process each video frame to generate respective object class labels for objects depicted in the video frame. In these implementations, the system may initialize the parameter values of the novel object localization system with the parameter values of the known object localization system.

The system may be configured to train the novel object localization system using any appropriate machine learning training technique (e.g., stochastic gradient descent). For example, the system may be configured to train the novel object localization system by iteratively (i.e., at each of multiple training iterations) adjusting the current parameter values of the novel object localization system. For example, the novel object localization system can be implemented as a neural network. An example of an object localization neural network, and a method for training an object localization neural network, is described with reference to: S. Girshick, and J. Sun, "Faster R-CNN: towards real-time object localization with region proposal networks", Advances in Neural Information Processing Systems (NIPS), <NUM>. Any other appropriate implementation of an object localization system can also be used.

While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention. The scope of protection is provided by the appended claims.

Claim 1:
A method performed by one or more data processing apparatus, the method comprising:
processing a video comprising a plurality of video frames (<NUM>) to generate, for each video frame of the plurality of video frames, a corresponding optical flow image (<NUM>) characterizing a displacement of each pixel of the video frame between the video frame and a subsequent video frame in the video;
for each optical flow image, processing the optical flow image using an optical flow object localization system (<NUM>; <NUM>) to generate object localization data (<NUM>) defining locations of objects depicted in the video frame corresponding to the optical flow image; and
using: (i) the plurality of video frames, and (ii) the object localization data generated by the optical flow object localization system by processing the optical flow images corresponding to the plurality of video frames, training a novel object localization system (<NUM>) to process a video frame to generate object localization data defining locations of novel objects depicted in the video frame;
wherein the optical flow object localization system is trained using: (i) optical flow images, and (ii) object localization data defining locations of objects depicted in video frames corresponding to optical flow images;
wherein the object localization data defining locations of objects depicted in a video frame corresponding to an optical flow image is obtained by processing the video frame using a known object localization system (<NUM>), wherein the known object localization system is configured to process the video frame to generate data defining locations of objects depicted in the video frame that are from a predetermined set of object classes; and
wherein a novel object is an object for which the location of the object is included in the object localization data (<NUM>) generated by the optical flow object localization system (<NUM>; <NUM>), and is not included in the object localization data generated by the known object localization system (<NUM>).