Patent Description:
This invention relates to image processing, and more particularly, to conversion of dual-context video data to full color video.

The video data needed for machine learning tasks is often different from what would be suitable for viewing by a human being. For example, in some vehicle safety systems, a machine learning model associated with the system is provided with separate grayscale and color streams from a single imaging sensor, referred to herein as a dual-context camera. Technological background can be found, e.g., in <CIT>.

In accordance with an aspect of the present invention, a system is provided. The system includes a dual-context camera that provides a first series of video frames encoded in accordance with a full color model and a second series of video frames encoded in accordance with an underdetermined color model. The first series of video frames and the second series of video frames are interleaved as to form a series of pairs of frames, each comprising a color video frame and an underdetermined video frame. An image merger generates a composite image for each pair of frames in the series of frames. The composite image includes a set of brightness values from the underdetermined video frame and a set of chrominance values from the color video frame. A color video source replaces the underdetermined image in each of the series of pairs of frames with the composite image generated for the pair of frames to provide a color video stream.

In accordance with another aspect of the present invention, a method is provided. A first series of video frames encoded in accordance with a full color model and a second series of video frames encoded in accordance with an underdetermined color model are provided. The first series of video frames and the second series of video frames are interleaved as to form a series of pairs of frames, with each pair of frames in the series of frames comprising a color video frame and an underdetermined video frame. A composite image is generated for each pair of frames in the series of frames including a set of brightness values from the underdetermined video frame and a set of chrominance values from the color video frame. The underdetermined image in each of the series of pairs of frames is replaced with the composite image generated for the pair of frames to provide a color video stream.

A "chrominance value," as used herein, is a value representing a color content of a pixel, either directly, such as the red, green, and blue values in a Red-Blue-Green (RGB) color model, or in a relative manner, such as the red-difference and blue-difference values in a YCrCb color model.

A "full color model" is a color model in which each pixel within an image or video frame is represented by at least three parameters.

An "underdetermined color model" is a color model in which each pixel within an image or video frame is represented by less than three parameters.

A "dual-context camera," as used herein, is a sensor that collects two series of video frames, with one series of video frames encoded in accordance with a full color model and another series of video frames encoded in accordance with an underdetermined color model. In one example, a dual-context camera can provide a first series of video frames encoded according to an RGB color model and a second series of video frames encoded in a grayscale representation with only a luminance or brightness value.

A grayscale image or video frame is an image or video frame that is encoded in a single parameter color model, such that each pixel is represented by a single value. In general, each pixel value in a grayscale image represents a brightness or luminance value.

"Color video," as used herein, is video in which each frame of the video is encoded in a three-parameter color model. Examples of three-parameter color models include the RGB model and the YCrCb model, which encodes each pixel using a brightness value and two chrominance values, representing, respectively, a difference between a red component of the pixel and a yellow component of the pixel, and a difference between a blue component of the pixel and a yellow component of the pixel.

A "trichromatic color model," as used herein, is a color model in which each pixel is encoded using three chrominance values. The RGB is an example of a trichromatic color model.

A "luma-chroma color model," as used herein, is a color model in which each pixel is encoded using a brightness or luminance value and two chrominance values.

<FIG> illustrates a system <NUM> that produces a color video stream from a dual-context camera <NUM>. The dual-context camera <NUM> that provides a first series of video frames encoded in accordance with a full color model and a second series of video frames encoded in accordance with an underdetermined color model interleaved as to form a series of pairs of frames. In one example, the second series of video frames are provided as grayscale video frames. In the illustrated implementation, the dual-context camera can provide each series of video frame rates at a frame rate less than the twenty-four frames per second (FPS) that are generally believed necessary to provide video with realistic motion for a human viewer. In one implementation, each of the first series of video frames and the second series of video frames are provided at eighteen FPS, with the interleaved video provided at thirty-six FPS.

An image merger <NUM> generates a composite image for each pair of frames in the series of frames using at least a brightness value from an associated frame of the second series of video frames and a set of at least one chrominance value from the color video frame. The image merger <NUM> uses color information from the full color frame before a given underdetermined color frame to provide a full color composite image representing that frame. A color video source <NUM> replaces the grayscale image in each of the series of pairs of frames with the composite image generated for the pair of frames to provide a color video stream. Since each underdetermined image from the dual-context camera <NUM> is replaced with the full color composite image, the color video stream can be provided at a full frame rate of the dual-context camera. It will be appreciated that each of the image merger <NUM> and the color video source <NUM> can be implemented as software instructions executed by an associated processor, dedicated hardware (e.g., an application specific integrated surface or a field programmable gate array), or a combination of software instructions and dedicated hardware.

<FIG> illustrates one example of a system <NUM> that produces a color video stream from a dual-context camera <NUM>. In the illustrated example, the dual-context camera <NUM> provides a series of color video frames encoded in accordance with a full color model and a series of grayscale video frames interleaved as to form a series of pairs of frames. In the illustrated implementation, the dual-context camera provides a total frame rate of thirty-six FPS, with eighteen color frames per second and eighteen grayscale FPS, to a machine learning system <NUM> associated with one or more vehicle safety systems. In one example, the dual-context camera <NUM> is an eight-megapixel imager using a Red-Yellow-Yellow-Cyan (RYYCy) color filter, such that the color video frames are encoded in accordance with a Red-Yellow-Cyan color model. It will be appreciated that the imager merger <NUM>, the color model transformer <NUM>, the motion detector <NUM>, and the color video source <NUM> can each be implemented as dedicated hardware, such as an application specific integrated circuit, as software instructions stored on a non-transitory computer-readable medium, or as a combination of dedicated hardware and software instructions.

An image merger <NUM> generates a composite image for each pair of frames in the series of frames, with each pixel of the composite image using the brightness value of a corresponding pixel from the grayscale frame and two chrominance values from a corresponding pixel in the color video frame. In one example, the color video frames are encoded in a luma-chroma color model, such as YCrCb, such that the two chrominance values associated with each pixel can be readily extracted for use with a brightness value from the grayscale video frame. It will be appreciated, however, that some cameras may provide the color frames in a trichromatic color model, such as the Red-Blue-Green (RBG) color model. In the illustrated example, a color model transformer <NUM> can generate, from a color video frame encoded in a trichromatic color model, a color image encoded in a luma-chroma color model. For example, the color model transformer <NUM> can receive a vector of chromaticity values for each pixel and multiply the chromaticity values by an appropriate color matrix to represent the pixel in the luma-chroma color model. In the example of translating a pixel encoded in the RBG color model to the YCrCb color model, the color matrix translation can be represented as: <MAT>
where R is the red component value in the RGB color model, G is the green component value in the RGB color space, B is the blue component value in the RGB color model, Y is the brightness value in the YCrCb color model, Cr is the red difference value in the YCrCb color space, Cb is the blue difference value in the YCrCb color space, and KR, KG, and KB are constants derived from the definition of the RGB space constrained such that KR + KG + KB = <NUM>.

In one example, the corresponding pixels between the grayscale video frame and the color video frame can be pixels having a same position within the frame. In this example, it is assumed that the two video frames are provided with the same resolution. Where the two video frames do not have the same resolution, a given pixel in the video frame having the lesser resolution (e.g., the grayscale video frame) may provide values for multiple pixels for a composite image having a greater resolution. Alternatively, the composite image can be generated at the lesser resolution, such that values from sets of pixels in the video frame having the greater resolution (e.g., the color video frame) are combined to provide a value for the composite image.

In another example, the corresponding pixels between the grayscale video frame and the color video frame can be selected to account for motion between the grayscale and the color video frame. To this end, the system <NUM> includes a motion detector <NUM> that uses an optical flow algorithm to detect motion between the color video frame and the underdetermined video frame and select the set of chrominance values from the color video frame according to the detected motion. Once the composite image has been generated, a color video source <NUM> replaces the grayscale image in each of the series of pairs of frames with the composite image generated for the pair of frames to provide a color video stream. In the illustrated implementation, the color video stream is provided to a display <NUM> that allows the color video stream to be viewed by a human operator.

In view of the foregoing structural and functional features described above in <FIG>, example methods will be better appreciated with reference to <FIG> and <FIG>. While, for purposes of simplicity of explanation, the methods of <FIG> and <FIG> are shown and described as executing serially, it is to be understood and appreciated that the present invention is not limited by the illustrated order, as some actions could in other examples occur in different orders and/or concurrently from that shown and described herein.

<FIG> illustrates a method <NUM> for providing full color video from a dual context camera. At <NUM>, a first series of video frames encoded in accordance with a full color model and a second series of video frames encoded in accordance with an underdetermined color model are provided. The two series of video frames are interleaved as to form a series of pairs of frames, with each pair of frames in the series of frames comprising a color video frame and an underdetermined video frame. At <NUM>, a composite image is generated for each pair of frames in the series of frames comprising a set of brightness values from the underdetermined video frame and a set of chrominance values from the color video frame.

In one example, each of the first series of video frames is encoded in a luma-chroma color model with each of a plurality of pixels in the color video frame represented as a luminosity value and two chrominance values. In this example, the composite image for each pair of frames is generated such that each pixel of a plurality of pixels in the composite image is represented as the brightness value from a corresponding pixel in the underdetermined image associated with the composite image and the two chrominance values associated with a corresponding pixel in the color video frame associated with the composite image. Where the two video frames have a same resolution, the corresponding pixels can be pixels that are in a same location in each image or can be selected according to motion between the frames as detected by an optical flow algorithm. Where the underdetermined video frame has a resolution less than that of the composite image, a given pixel in the underdetermined video frame can be used to provide a brightness value for a set of multiple pixels in the composite image, which the color information is provided from the corresponding pixels in the color video frame.

In another example, each of the first series of video frames is encoded in a trichromatic color model with each of a plurality of pixels in the color video frame represented as three chrominance values. In this example, a color image encoded in a luma-chroma color model is generated from the color video frame, and the composite image is generated using the brightness value from a corresponding pixel in the underdetermined image associated with the composite image and the two chrominance values associated with a corresponding pixel in the color video frame associated with the composite image. The underdetermined image in each of the series of pairs of frames is then replaced with the composite image generated for the pair of frames at <NUM> to provide a color video stream.

<FIG> illustrates an example implementation of a method <NUM> for providing full color video from a dual context camera. At <NUM>, a first series of video frames encoded in accordance with a full color model and a second series of video frames encoded in accordance with an underdetermined color model are provided. The two series of video frames are interleaved as to form a series of pairs of frames, with each pair of frames in the series of frames comprising a color video frame and an underdetermined video frame. At <NUM>, for each pair of frames of the series of pairs of frames, an optical flow algorithm that detects motion between the color video frame and the underdetermined video frame in the pair of frames is applied. At <NUM>, a set of chrominance values from the color video frame is selected for the composite image according to the detected motion. Essentially, the optical flow analysis is used to match corresponding pixels between the two video frames, such that the brightness values in the underdetermined model can be matched with chrominance values in the color video frame.

At <NUM>, a composite image is generated for each pair of frames in the series of frames comprising a set of brightness values from the underdetermined video frame and the selected chrominance values from the color video frame. The underdetermined image in each of the series of pairs of frames is then replaced with the composite image generated for the pair of frames at <NUM> to provide a color video stream.

<FIG> is a schematic block diagram illustrating an exemplary system <NUM> of hardware components capable of implementing examples of the systems and methods disclosed in <FIG>, such as the image merger <NUM> illustrated in <FIG>. The system <NUM> can include various systems and subsystems. The system <NUM> can be a personal computer, a laptop computer, a workstation, a computer system, an appliance, an application-specific integrated circuit (ASIC), a field programmable gate array, a server, a server blade center, a server farm, etc..

The system <NUM> can includes a system bus <NUM>, a processing unit <NUM>, a system memory <NUM>, memory devices <NUM> and <NUM>, a communication interface <NUM> (e.g., a network interface), a communication link <NUM>, a display <NUM> (e.g., a video screen), and an input device <NUM> (e.g., a keyboard and/or a mouse). The system bus <NUM> can be in communication with the processing unit <NUM> and the system memory <NUM>. The additional memory devices <NUM> and <NUM>, such as a hard disk drive, server, stand-alone database, or other non-volatile memory, can also be in communication with the system bus <NUM>. The system bus <NUM> interconnects the processing unit <NUM>, the memory devices <NUM>-<NUM>, the communication interface <NUM>, the display <NUM>, and the input device <NUM>. In some examples, the system bus <NUM> also interconnects an additional port (not shown), such as a universal serial bus (USB) port.

The processing unit <NUM> can be a computing device and can include an application-specific integrated circuit (ASIC). The processing unit <NUM> executes a set of instructions to implement the operations of examples disclosed herein. The processing unit can include a processing core.

The additional memory devices <NUM>, <NUM>, and <NUM> can store data, programs, instructions, database queries in text or compiled form, and any other information that can be needed to operate a computer. The memories <NUM>, <NUM> and <NUM> can be implemented as computer-readable media (integrated or removable) such as a memory card, disk drive, compact disk (CD), or server accessible over a network. In certain examples, the memories <NUM>, <NUM> and <NUM> can comprise text, images, video, and/or audio, portions of which can be available in formats comprehensible to human beings. Additionally or alternatively, the system <NUM> can access an external data source or query source through the communication interface <NUM>, which can communicate with the system bus <NUM> and the communication link <NUM>.

In operation, the system <NUM> can be used to implement one or more parts of a system in accordance with the present invention. Computer executable logic for implementing the system resides on one or more of the system memory <NUM>, and the memory devices <NUM>, <NUM> in accordance with certain examples. The processing unit <NUM> executes one or more computer executable instructions originating from the system memory <NUM> and the memory devices <NUM> and <NUM>. The term "computer readable medium" as used herein refers to any medium that participates in providing instructions to the processing unit <NUM> for execution, and it will be appreciated that a computer readable medium can include multiple computer readable media each operatively connected to the processing unit.

Also, it is noted that the embodiments can be described as a process which is depicted as a flowchart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. Although a flowchart can describe the operations as a sequential process, many of the operations can be performed in parallel or concurrently. In addition, the order of the operations can be re-arranged. A process is terminated when its operations are completed, but could have additional steps not included in the figure. A process can correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, its termination corresponds to a return of the function to the calling function or the main function.

Furthermore, embodiments can be implemented by hardware, software, scripting languages, firmware, middleware, microcode, hardware description languages, and/or any combination thereof. When implemented in software, firmware, middleware, scripting language, and/or microcode, the program code or code segments to perform the necessary tasks can be stored in a machine readable medium such as a storage medium. A code segment or machine-executable instruction can represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, a software package, a script, a class, or any combination of instructions, data structures, and/or program statements. A code segment can be coupled to another code segment or a hardware circuit by passing and/or receiving information, data, arguments, parameters, and/or memory contents. Information, arguments, parameters, data, etc. can be passed, forwarded, or transmitted via any suitable means including memory sharing, message passing, ticket passing, network transmission, etc..

For a firmware and/or software implementation, the methodologies can be implemented with modules (e.g., procedures, functions, and so on) that perform the functions described herein. Any machine-readable medium tangibly embodying instructions can be used in implementing the methodologies described herein. For example, software codes can be stored in a memory. Memory can be implemented within the processor or external to the processor. As used herein the term "memory" refers to any type of long term, short term, volatile, nonvolatile, or other storage medium and is not to be limited to any particular type of memory or number of memories, or type of media upon which memory is stored.

Claim 1:
A system (<NUM>, <NUM>, <NUM>) comprising:
a dual-context camera (<NUM>, <NUM>) that provides a first series of video frames encoded in accordance with a full color model and a second series of video frames encoded in accordance with an underdetermined color model, the first series of video frames and the second series of video frames being interleaved as to form a series of pairs of frames, each comprising a color video frame and an underdetermined video frame;
an image merger (<NUM>, <NUM>) that generates a composite image for each pair of frames in the series of frames, the composite image comprising a set of brightness values from the underdetermined video frame and a set of chrominance values from the color video frame; and
a color video source (<NUM>, <NUM>) that replaces the underdetermined video frame in each of the series of pairs of frames with the composite image generated for the pair of frames to provide a color video stream.