Patent Description:
<CIT> relates to a method and apparatus for generating local metadata including position information of a similar color mapping region and a color mapping function of the similar color mapping region and a method and apparatus for correcting color components of a pixel in a similar color mapping region based on local metadata.

<CIT> relates to high dynamic range image composition described using multiple images.

<CIT> discloses a detection method for monitoring an abnormal behavior of a target under laser night vision in a video surveillance field.

<CIT> discloses a method for enhancing local contrast of images which involves deriving and applying color mapping function for regions in current frame to generate contrast-enhanced frame.

Implementations improve the visual quality of frames of a video using a quality corrected image captured at substantially the same moment in time as one of the frames of the video. To improve the visual quality of the frames of the video at least one quality parameter difference between the corrected image and a corresponding frame of the video can be determined. Then, the visual quality of the frames of the video can be improved using the at least one quality parameter difference.

A general aspect, includes capturing a plurality of frames associated with a video file, capturing an image corresponding to one of the plurality of frames and generating a color map between the captured image and a corresponding frame of the plurality of frames, where the color map is based on a color and a tone of the captured image and a color and tone of the corresponding frame.

The color map may be used to tone correct one or more selected frames associated with a video file. The color map may be used to color correct the one or more selected frames based on the color map. For example, another general aspect includes receiving a video file including a plurality of frames and a header including a data structure including data representing a color map, selecting a first frame of the plurality of frames, the first frame being associated with the data representing the color map, selecting a second frame of the plurality of frames, the second frame being a target frame for color correction, tone correcting the second frame based on the color map, and color correcting the second frame based on the color map.

The aspect may include generating a data structure including data representing the color map. The data structure may be stored as metadata in a header associated with the video.

For example, the color map may be a color map generated in accordance with the general aspect set out above, or more generally in accordance with the techniques set out in the present application.

Implementation can include one or more of the following features, alone or in any combination with each other. For example, a post-capture process can be performed on the captured image to improve a visual quality of the image.

The color map can include at least one correctable quality parameter difference between the captured image and the corresponding frame.

Tone correcting the one or more selected frames can include matching at least one of a pixel, a block of pixels, a range of pixels, or a partition of the one or more selected frames to at least one of a pixel, a block of pixels, a range of pixels, or a partition of the color map and using a value of the color map to perform the tone correcting of the one or more selected frames, and color correcting the one or more selected frames can include matching at least one of the pixel, the block of pixels, the range of pixels, or the partition of the one or more selected frames to at least one of the pixel, the block of pixels, the range of pixels, or the partition of the color map, and the value of the color map can be used to perform the color correcting of the one or more selected frames.

The color map can be partitioned into an MxN grid having M columns and N rows, where each partition includes data indicating a correction parameter, and the correction parameter can include a value for gamma correction.

Color correcting the one or more selected frames can include performing a smoothing process to minimize color discontinuities in the one or more selected frames.

The image and the frame to which the image corresponds can be captured in a same instance.

The captured image can be partitioned, and the corresponding frame can be partitioned into a partitioned frame, and generating the color map can include: determining a color variance between each partition of the image and a corresponding partition of the partitioned frame and determining a tone variance between each partition of the image and a corresponding partition of the partitioned frame, and at least one of the determined color variance and the determined tone variance can be based on a luminance difference or a chrominance difference.

The color map may be partitioned into an MxN grid having M columns and N rows, where each partition can include data indicating a correction parameter, and the correction parameter can include a value for gamma correction. For example, the partitioning may be performed to generate a data structure including data representing the color map.

The captured image can be partitioned and the frame to which the image corresponds can be partitioned into a partitioned frame, and generating the color map can include: determining a color variance between each partition of the image and a corresponding partition of the partitioned frame and determining a tone variance between each partition of the image and a corresponding partition of the partitioned frame, where at least one of the color variance and the tone variance is based on a luminance difference or a chrominance difference.

Color correcting a frame can include performing a smoothing process to minimize color discontinuities.

Color correcting a frame can include performing a blending process to minimize color discontinuities.

Color correcting a frame can include using a trained convolutional neural network to minimize color variance between a partition of the second frame and a partition of the color map.

Color correcting a frame can include performing a chrominance correction process using a mean value of a color space variable and a variance value of the color space variable.

Another general aspect includes a computer-readable medium storing computer readable instructions configured to cause a computer to perform the method of the general aspect set out above, or more generally to perform the techniques described herein.

Another general aspect includes a computing system comprising one or more processors and one or more computer-readable media storing computer readable instructions configured to cause the one or more processors to perform the method of the general aspect set out above, or more generally to perform the techniques described herein.

Example implementations will become more fully understood from the detailed description given herein below and the accompanying drawings, wherein like elements are represented by like reference numerals, which are given by way of illustration only and thus are not limiting of the example implementations and wherein:.

It should be noted that these Figures are intended to illustrate the general characteristics of methods, structure and/or materials utilized in certain example implementations and to supplement the written description provided below. These drawings are not, however, to scale and may not precisely reflect the precise structural or performance characteristics of any given implementation, and should not be interpreted as defining or limiting the range of values or properties encompassed by example implementations. For example, the relative thicknesses and positioning of molecules, layers, regions and/or structural elements may be reduced or exaggerated for clarity.

Typically, cameras perform at least one type of post-capture processing on an image to improve the visual quality of the image. The post-capture processing that can improve the visual quality can include noise reduction, stabilization, exposure balancing white balancing, color noise reduction, color correction, tone scaling, gamma correction and/or the like. However, not all of these image post-capture processing operations can be performed during video capture because of the processing time restrictions during video capture. In other words, to capture video at a given frame rate, time consuming post-capture processing that can improve visual quality may not be performed.

Accordingly, at least one technical problem addressed by this disclosure is the disparity in visual quality between an image and a corresponding frame of a video resulting from post-capture processing operations that are performed on the image that are not performed on the frame of the video. The disparity in visual quality can result in a less than desirable user experience when a user of a device displays both the image, the corresponding frame of the video and/or other frames of the video on a display of the device at the same time. The disparity in visual quality can result in a less than desirable user experience when the user of the device cycles between displaying the image and the corresponding frame of the video and/or other frames of the video on the display of the device.

The technical solution to the aforementioned technical problem that is disclosed by this disclosure includes generating a color map defining at least one correctable quality parameter difference between the image and the corresponding frame of the video, and correcting at least one frame of the video using the color map. The corrected at least one frame of the video can be stored in memory for future display and/or the corrected at least one frame of the video can be displayed in response to completing the correction.

The technical effect of the aforementioned technical solution is improved visual quality for each frame (or image) associated with the video. The technical improvement can be derived from using the color map to improve visual quality. This is an improvement over a technique that attempts to use post-capture processing on each frame of the video because the disclosed technical improvement reduces processing demand on the capture device (e.g., a device of limited processing capability) and/or as an operation during capture on the capture device because the image improvement operation can be performed independent of the capture process.

<FIG> illustrates a block diagram of a process of generating a video file according to an example implementation. As shown in <FIG>, a capture phase <NUM> includes capturing an image <NUM> including an image <NUM>, capturing a video <NUM> including a video <NUM> having a plurality of frames <NUM>, computing a color map <NUM>, selecting a video frame <NUM>, performing a color correction <NUM>, and generating a video file <NUM>. The capture phase <NUM> can be implemented by a device including a camera (e.g., a digital camera, a digital video device, a mobile phone and the like). The image <NUM> and the video <NUM> can be captured by the camera. One of the plurality of frames <NUM> and the image <NUM> can be captured at substantially (e.g., by a same shutter event) same moment in time.

Typically, images captured with camera movement in environments with uneven lighting or low amounts of light can result in blurry images or include random variations in color, brightness and/or tone from pixel to pixel. Accordingly, in an example implementation, post-capture processing is performed on the image <NUM>. The post-capture processing can be configured to improve the visual quality (e.g., improve color, improve tone, and the like), reduce image noise (e.g., improve color variance, improve tone variance, stabilize for movement, etc.), and the like. The post-capture processing can use optical image stabilization (OIS), artificial intelligence, High Dynamic Range (HDR) techniques, HDR+ techniques, sampling, exposure, multi-image averaging and the like to improve the quality of the image <NUM>.

For example, a HDR+ technique can take a burst of image shots with short exposure times, align the images algorithmically, and replace each pixel in a resultant image (e.g., the image to be stored) with the average color at that position across all the image shots. However, similar post-capture processing may not be performed on the video <NUM>. Accordingly, the corresponding one of the plurality of frames <NUM> and the image <NUM> can have a different visual quality. Furthermore, the image <NUM> can have a relatively higher visual quality than the corresponding one of the plurality of frames <NUM>. As a result, the image <NUM> can have a relatively higher visual quality than the video <NUM>.

In an example implementation, a color map <NUM> is generated (e.g., via computer processing). The color map can be generated based on the corresponding one of the plurality of frames <NUM> and the image <NUM>. The color map can include the color and tone variance between the corresponding one of the plurality of frames <NUM> and the image <NUM>. For example, the corresponding one of the plurality of frames <NUM> can be stored using the YUV color space and the image <NUM> can be stored using the YUV color space. Accordingly, the color map can include the difference between a Y (luminance) and UV (chrominance) of the corresponding one of the plurality of frames <NUM> and the image <NUM>. The color map can include a pixel-by-pixel variance, a value associated with a block of pixels, a value associated with a partition of the corresponding one of the plurality of frames <NUM> and/or the image <NUM> and the like. The plurality of frames <NUM> and/or the image <NUM> can be stored using a color space other than the YUV color space. For example, the plurality of frames <NUM> and/or the image <NUM> can be stored using the RGB, Y'UV, YCbCr, YPbPr, and the like color spaces.

In an example implementation, the image <NUM> can be a color corrected image captured at substantially (e.g., by a same shutter event) same moment in time as a video frame (e.g., one of the plurality of frames <NUM>) is captured. In an example implementation, image <NUM> can be a HDR+ image and video <NUM> can be a short video (e.g., a micro-video) encoded using the MPEG4 standard sometimes called a MPEG4 micro-video. Accordingly, the image <NUM> and the video <NUM> can be stored using the YUV color space. The image <NUM> and/or the video <NUM> can be compressed (e.g., encoded) prior to storing (e.g., as a mp4 file). Although the MPEG4 and mp4 standard is referenced, other image and video file standards can be used in example implementations described herein. Although the YUV color space is referenced, other image color spaces (e.g., RGB, Y'UV, YCbCr, YPbPr, and the like) color spaces can be used in example implementations described herein.

Performing color correction <NUM> (e.g., via computer processing) can include using the color map to perform color and/or tone correction on at least one video frame. For example, color and/or tone correction can be performed on frames of video <NUM>. In an example implementation, the color and/or tone correction can be performed on each of the plurality of frames <NUM>. In another example implementation, the color and/or tone correction can be performed on a selected video frame <NUM> (e.g., selected from the plurality of frames <NUM>). Performing the color correction <NUM> can include matching at least one of a pixel, a block of pixels, a range of pixels, and/or a partition of the selected video frame <NUM> to at least one of a pixel, a block of pixels, a range of pixels, and/or a partition of the color map and then using corresponding values of the color map to perform color and/or tone correction on the video frame (e.g., the selected video frame <NUM>). The video file <NUM> can be generated to include the color and/or tone corrected plurality of frames <NUM>.

The video file <NUM> generated in <FIG> can be color corrected during the capture phase <NUM> at the device capturing the video <NUM> and the image <NUM>. However, computing the color map <NUM>, selecting the video frame <NUM> and performing color correction <NUM> can be performed independent of capturing the video <NUM>. In other words, capturing the video <NUM> can continue while color correction is performed on previously captured frames <NUM>. The (color corrected) video <NUM> can then be rendered using any rendering technique.

In some example implementations, color correction can be performed at a rendering device (e.g., a mobile phone, a laptop computer, a desktop computer, and the like). <FIG> illustrates a block diagram of generating a video file and rendering a video according to an example implementation. As shown in <FIG>, during the capture phase <NUM>, the color map <NUM> is generated (e.g., via computer processing) based on the corresponding one of the plurality of frames <NUM> and the image <NUM>. However, in this implementation generating a video file <NUM> includes using the video <NUM> and the color map.

The video file <NUM> can be encoded and communicated (e.g., via a wired or wireless communication channel) to the rendering device configured to implement the rendering phase <NUM>. The rendering phase <NUM> can include using decoder <NUM> to decode an encoded video file <NUM>, selecting a color map <NUM> from the decoded video file, selecting the video frame <NUM> from the decoded video file, performing color correction <NUM> and displaying <NUM> the corrected video. The decoding the video file <NUM> can be performed by a decoder <NUM> where the color map <NUM> and a reconstructed plurality of frames <NUM> are obtained. Video frame <NUM> can be selected from the reconstructed plurality of frames <NUM>.

Performing the color correction <NUM> (e.g., via computer processing) can use the color map <NUM> to perform color and/or tone correction on a selected video frame <NUM>. The color and/or tone correction can be performed on each of the reconstructed plurality of frames <NUM>. The color correction <NUM> can include matching a pixel, a block of pixels a range of pixels, or a partition of the selected video frame <NUM> to a pixel, a block of pixels a range of pixels, or a partition of the color map and then using a corresponding value of the color map to perform color and/or tone correction on the video frame <NUM>. The color and/or tone corrected video frame can be rendered on display <NUM>.

<FIG> illustrates a data structure for a video file (e.g., a generated video file <NUM>) according to an example implementation. As shown in <FIG>, the data structure <NUM> includes a video <NUM>. The video <NUM> includes metadata <NUM>. The metadata <NUM> includes color map data <NUM>. The color map data <NUM> can include different types of color maps. For example, the color map data <NUM> can include a pixel to pixel map(s), a block to block map(s) and or the like. As shown in <FIG>, the color map data <NUM> includes grid color map data <NUM> having data <NUM>, columns <NUM>, and rows <NUM>.

The video <NUM> can include data associated with a computed color map (e.g., computed color map <NUM>). For example, as discussed above, the color map can be generated based on the corresponding one of the plurality of frames <NUM> and the image <NUM>. The color map can include the color and tone variance between the corresponding one of the plurality of frames <NUM> and the image <NUM>.

Metadata <NUM> can include color map data (e.g., color map <NUM>) as generated using a compute color map process (e.g., by compute color map <NUM>). The columns <NUM> and rows <NUM> indicate a number of rows and a number of columns that the color map is partitioned into. For example, the color map can be partitioned into an MxN grid (e.g., 4x4 grid (see <FIG> below), a 3x4 grid, a 4x3 grid, and the like). Example implementations are not limited to a grid size. However, too few partitions can limit the effectiveness of the color correction and too many partitions can require a relatively large amount of processing resources and/or memory and may create a time delay in rendering due to processing the error correction.

The data <NUM> can include a correction parameter(s). The correction parameter(s) can be a value used to correct for tone and/or color. In an example implementation, Y (luminance) can be corrected using gamma correction Ycorrected = Yγ and UV (chrominance) can be corrected using a mean value of U, V and a variance value of U, V. For example, the mean value of U, V can be subtracted from the data and the result can be scaled based on the variance of U, V. Accordingly, the correction parameter(s) (e.g., for each partition in the MxN grid) can include a gamma correction value, a mean value of U, V and a variance value of U, V.

In another example implementation, a color correction matrix (CCM) can be used to correct for tone and/or color. In this implementation the dot product of a 3x3 CCM and YUV can generate YUV corrected. Accordingly, the correction parameter(s) (e.g., for each partition in the MxN grid) can include a 3x3 CCM. Other techniques can be used to correct for tone and/or color. In addition, color correction can be performed in other color spaces. For example, the aforementioned 3x3 CCM can be used to perform tone and/or color correction in the RGB color space. In an example implementation, the correction parameter(s) can be generated (e.g., via computer processing) in the compute color map <NUM> block based on the corresponding one of the plurality of frames <NUM> and the image <NUM>.

Furthermore, a convolutional neural network (CNN) can be used to correct for tone and/or color. In this implementation, the correction parameter(s) can include a CNN architecture and the weights associated with each neuron of the CNN. In addition, a machine learning (ML) algorithm (e.g., based on the CNN) can be used to train the CNN (e.g., modify the weights) to best correct for tone and/or color. In an example implementation, the correction parameter(s), as weights, can vary the color profile and variance for each partition. Therefore, the CNN can vary the color profile to minimize a variance between the color profile and a pixel (e.g., YUV, RGB and the like) in each partition.

In some implementations, histogram matching and point based transfer can be used in the color and/or tone correcting processes. For example, histogram matching can include modifying the tone and/or color data of the frame to be color corrected until a histogram associated with the frame matches a histogram of the image. In this implementation the histogram can be associated with the matched partition. Point based transfer can include using a scattered point interpolation technique using moving least squares. Scattered point interpolation can include applying a high-order polynomial to the tone and/or color data of the frame in order to interpolate or fit data of the frame to the tone and/or color data of the image. Moving least squares uses a local polynomial such that, in this implementation, the scattered point interpolation can be implemented on the matched partition.

In some implementations, there can be tone and/or color discontinuities on the border between two partitions or blocks in the NxM grid for the tone and/or color corrected frame. Therefore, the color correction process can include a smoothing and/or blending process to remove or minimize the tone and/or color discontinuities. For example, a bilinear interpolation process can be implemented with the color correction <NUM> block. The bilinear interpolation process can be configured to remove or minimize the tone and/or color discontinuities. Bilinear interpolation can be implemented as a resampling and/or filtering operation. Bilinear interpolation can be implemented based on a tile (or fragment) location within a grid. For example, a tile located at a corner of the grid can be mapped to a closes color. A tile located at a boundary of the grid can be linear interpolated based on <NUM> colors (e.g., <NUM> mapped colors). Other tiles can be bi-linear interpolated based on <NUM> colors (e.g., <NUM> mapped colors). Other interpolation techniques can be used as well. For example, nearest neighbor, bicubic, higher order, and the like interpolation techniques can be used.

<FIG> illustrates a block diagram showing processing of a frame of a video file according to an example implementation. As shown in <FIG>, the image <NUM> and frame <NUM> are each partitioned into a 4x4 grid including <NUM> partitions. The partitions can be labeled using the row number and column number. The image <NUM> is shown as being partitioned into <NUM> partitions including H11, H12, H13, H14, H21, H22, H23, H24, H31, H32, H33, H34, H41, H42, H43, and H44. The frame <NUM> is shown as being partitioned into <NUM> partitions including S11, S12, S13, S14, S21, S22, S23, S24, S31, S32, S33, S34, S41, S42, S43, and S44. Each partition (e.g., H11, S11,. , H44, S44) includes at least one pixel. The pixels' values (e.g., YUV, RGB, and the like) of the image <NUM> and the frame <NUM> can be used to generate (e.g., calculate) correction parameter(s) <NUM>.

The grid color map data <NUM> is shown as being partitioned into <NUM> partitions including P11, P12, P13, P14, P21, P22, P23, P24, P31, P32, P33, P34, P41, P42, P43, and P44. Each partition (e.g., P11, P12,. , P44) includes a corresponding correction parameter(s) <NUM>. The video frame <NUM> is shown as being partitioned into <NUM> partitions including K11, K12, K13, K14, K21, K22, K23, K24, K31, K32, K33, K34, K41, K42, K43, and K44. Each partition (e.g., K11, K12,. , K44) includes at least one pixel. The at least one pixel can be an uncorrected pixel. To process (e.g., color and tone correct) the video frame <NUM>, a partition (e.g., the pixels of the partition) can be matched to (or a match can be found in) a partition in the frame <NUM>. As is shown in <FIG>, partition K42 of the video frame <NUM> is matched to partition S43 of the frame <NUM>. Therefore, during color correction, the correction parameter(s) <NUM> for color correcting partition K42 of the video frame <NUM> are selected from partition P43 of the grid color map data <NUM> for the color map.

<FIG> illustrates a flow diagram of rendering a frame of a video file according to an example implementation. As shown in <FIG>, video file <NUM> (or video file <NUM>) is input to the render phase <NUM> where the decoder <NUM> decodes the video file. As discussed above, the video file <NUM> can include video <NUM> including data (e.g., metadata <NUM>) that can be used to color correct frames of video. Flow continues to the uncorrected video frame <NUM> block where a video frame is selected. The selected video frame is a video frame to be color corrected.

Then in the match <NUM> block the video frame is partitioned into NxM partitions or blocks and each block is matched to a block in the one of the plurality of frames <NUM> that corresponds to the image <NUM> (e.g., both captured by a same shutter event). Then, in the parameter fetch <NUM> block the correction parameters associated with the matched blocks are fetched (e.g., requested and retrieved) from the video <NUM>. For example, at least one parameter is selected from grid color map data <NUM> using the column <NUM> and row <NUM> values of the corresponding matched block. Then the correction <NUM> block generates the corrected frame using the uncorrected video frame <NUM> and the at least one parameter as discussed above.

<FIG>, <FIG> and <FIG> are flowcharts of methods according to example implementations. The steps described with regard to <FIG>, <FIG> and <FIG> may be performed due to the execution of software code stored in a memory associated with an apparatus (e.g., a computing device configured to capture a video and/or render the video) and executed by at least one processor associated with the apparatus. However, alternative implementations are contemplated such as a system embodied as a special purpose processor like Application-specific integrated circuit (ASIC). Although the steps described below are described as being executed by a processor, the steps are not necessarily executed by a single processor. In other words, at least one processor may execute the steps described below with regard to <FIG>, <FIG> and <FIG>.

<FIG> illustrates a flow diagram for generating a video file according to an example implementation. As shown in <FIG>, in step S505 a plurality of frames associated with a video are captured. For example, a device including a camera (e.g., a digital camera, a digital video device, a mobile phone and the like) is operated by a user to capture the plurality of frames which can be stored together as a video.

In step S510 an image corresponding to at least one frame of the plurality of frames is captured. For example, the image can be captured by the camera. In an example implementation, the image <NUM> can be captured at substantially (e.g., by a same shutter event) same moment in time as one of the plurality of frames <NUM>.

In step S515 the image is partitioned. For example, the image can be partitioned into NxM grid of partitions or blocks. As discussed above, image <NUM> can be partitioned into a 4x4 grid of <NUM> partitions.

In step S520 the corresponding frame is partitioned. For example, the frame can be partitioned into NxM grid of partitions or blocks. As discussed above, the frame (of the plurality of frames <NUM>) captured at substantially (e.g., by a same shutter event) same moment in time as the image <NUM> can be partitioned into a 4x4 grid of <NUM> partitions.

In step S525 a color map is generated based on the partitioned image and the partitioned corresponding frame. For example, the color map can be generated based on the corresponding one of the plurality of frames <NUM> and the image <NUM>. The color map can include the color and tone variance between the corresponding one of the plurality of frames <NUM> and the image <NUM>. For example, the corresponding one of the plurality of frames <NUM> can be stored using the YUV color space and the image <NUM> can be stored using the YUV color space. Accordingly, the color map can include the difference between a Y (luminance) and UV (chrominance) of the corresponding one of the plurality of frames <NUM> and the image <NUM>. The color map can include a pixel-by-pixel variance, a value associated with a block of pixels, a value associated with a partition of the corresponding one of the plurality of frames <NUM> and/or the image <NUM> and the like. The plurality of frames <NUM> and/or the image <NUM> can be stored using a color space other than the YUV color space. For example, the plurality of frames <NUM> and/or the image <NUM> can be stored using the RGB, Y'UV, YCbCr, YPbPr, and the like color spaces.

In step S530 a frame from the video is selected. For example, each of the plurality of frames associated with the video can be targeted for color correction. Therefore, the selected frame can be one of the plurality of frames. In an example implementation, each of the plurality of frames is selected sequentially (or in some other order) for color correction. Then, in step S535 the selected frame is partitioned. For example, the selected frame can be partitioned into NxM grid of partitions or blocks.

In step S540 a partition of the frame is matched to a partition of the color map. For example, in order to process (e.g., color and tone correct) the selected video frame (e.g., video frame <NUM>), a partition (e.g., the pixels of the partition) can be matched to (or a match can be found in) a partition in frame (of the plurality of frames <NUM>) captured at substantially (e.g., by a same shutter event) same moment in time as the image. As is shown above in <FIG>, partition K42 of the video frame <NUM> is matched to partition S43 of the frame <NUM>. Then, in step S545 at least one parameter value associated with the partition of the color map is fetched. Continuing the example, during color correction, the correction parameter(s) <NUM> for color correcting partition K42 of the video frame <NUM> are selected from partition P43 of the grid color map data <NUM> for the color map.

In step S550 a color correction and a tone correction is performed on the partition of the frame using the at least one parameter value. For example, the at least one parameter value can be used to perform color and/or tone correction on the selected frame. In an example implementation, Y (luminance) can be corrected using gamma correction Ycorrected = Yγ and UV (chrominance) can be corrected using a mean value of U, V and a variance value of U, V. For example, the mean value of U, V can be subtracted from the data and the result can be scaled based on the variance of U, V. Accordingly, the at least one parameter value can include a gamma correction value, a mean value of U, V and a variance value of U, V.

As discussed above, in one or more other example implementation, a color correction matrix (CCM) technique, a trained convolutional neural network (CNN) technique, a histogram matching technique and/or a point based transfer technique can be used to correct for tone and/or color. In addition, as discussed above, color correction process include a smoothing and/or blending process to remove or minimize the tone and/or color discontinuities along a border between two partitions in a grid. Although the YUV color space is referenced, other image color spaces (e.g., RGB, Y'UV, YCbCr, YPbPr, and the like) color spaces can be used in example implementations described herein.

In step S555 a corrected frame is generated using the color corrected and tone corrected partition of the frame. For example, steps S540 to S550 can be repeated for each partition to color correct all of the frame partitions. Accordingly, the frame is color corrected.

In step S560 a video file including a plurality of corrected frames is generated. For example, steps S530 to S555 can be repeated for each of the plurality of frames. Then, the video file (e.g., video file <NUM>) can be generated using the plurality of color corrected frames.

In an example implementation, the color correction can be performed on a rendering device (e.g., a device that is not the capture device). In this implementation, the capture device can perform steps S505 to S525 and not steps S530 to S560. <FIG> illustrates another flow diagram for generating a video file according to an example implementation. As shown in <FIG>, in step S565 a file including the video and the color map is generated. For example, the color map can be included in a data structure corresponding to the video <NUM> as described above. The color map can be inserted and stored, for example, in a header associated with the video. The video <NUM> and/or the metadata <NUM> can be compressed prior to storing the color map as metadata in the header.

<FIG> illustrates a flow diagram for generating a color and tone corrected frame of a video according to an example implementation. As shown in <FIG>, in step S605 a file including a video and a color map is received. For example, the file can be on a memory device (e.g., memory stick, CD-ROM, or the like) and/or downloaded from a cloud device. For example, the color map can be included in a data structure including the metadata <NUM> as described above. The image data can be stored, for example, as metadata in a header associated with the video.

In step S610 the file is decoded. For example, the video file <NUM> and/or the video file <NUM> can be compressed prior to storing the video <NUM> including metadata <NUM> in the header. Therefore, the file can be decoded/ decompressed using a same standard that the video file <NUM> and/or the video file <NUM> was compressed using.

In step S615 a frame from the video is selected. For example, each of the plurality of frames associated with the video can be targeted for color correction. Therefore, the selected frame can be one of the plurality of frames. In an example implementation, each of the plurality of frames is selected sequentially (or in some other order) for color correction. Then, in step S620 the selected frame is partitioned. For example, the selected frame can be partitioned into NxM grid of partitions or blocks.

In step S625 a partition of the frame is matched to a partition of the color map. For example, in order to process (e.g., color and tone correct) the selected video frame (e.g., video frame <NUM>), a partition (e.g., the pixels of the partition) can be matched to (or a match can be found in) a partition in frame (of the plurality of frames <NUM>) captured at substantially (e.g., by a same shutter event) same moment in time as the image. As is shown above in <FIG>, partition K42 of the video frame <NUM> is matched to partition S43 of the frame <NUM>. Then, in step S630 at least one parameter value associated with the partition of the color map is fetched. Continuing the example, during color correction, the correction parameter(s) <NUM> for color correcting partition K42 of the video frame <NUM> are selected from partition P43 of the grid color map data <NUM> for the color map.

In step S635 a color correction and a tone correction is performed on the partition of the frame using the at least one parameter value. For example, the at least one parameter value can be used to perform color and/or tone correction on the selected frame. For example, the at least one parameter value can be used to perform color and/or tone correction on the selected frame. In an example implementation, Y (luminance) can be corrected using gamma correction Ycorrected = Yγ and UV (chrominance) can be corrected using a mean value of U, V and a variance value of U, V. For example, the mean value of U, V can be subtracted from the data and the result can be scaled based on the variance of U, V. Accordingly, the at least one parameter value can include a gamma correction value, a mean value of U, V and a variance value of U, V.

In step S640 a corrected frame is generated using the color corrected and tone corrected partition of the frame. For example, steps S540 to S550 can be repeated for each partition to color correct all of the frame partitions. Accordingly, the frame is color corrected.

<FIG> shows an example of a computer device <NUM> and a mobile computer device <NUM>, which may be used with the techniques described here. Computing device <NUM> is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. Computing device <NUM> is intended to represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smart phones, and other similar computing devices.

In addition, an external interface <NUM> may be provide in communication with processor <NUM>, to enable near area communication of device <NUM> with other devices.

Communication interface <NUM> may provide for communications under various modes or protocols, such as GSM voice calls, SMS, EMS, or MMS messaging, CDMA, TDMA, PDC, WCDMA, CDM72000, or GPRS, among others. In addition, short-range communication may occur, such as using a Bluetooth, Wi-Fi, or other such transceiver (not shown).

Various implementations of the systems and techniques described here can be realized as and/or generally be referred to herein as a circuit, a module, a block, or a system that can combine software and hardware aspects. For example, a module may include the functions/acts/computer program instructions executing on a processor (e.g., a processor formed on a silicon substrate, a GaAs substrate, and the like) or some other programmable data processing apparatus.

Some of the above example implementations are described as processes or methods depicted as flowcharts. Although the flowcharts describe the operations as sequential processes, many of the operations may be performed in parallel, concurrently or simultaneously. In addition, the order of operations may be re-arranged. The processes may be terminated when their operations are completed, but may also have additional steps not included in the figure. The processes may correspond to methods, functions, procedures, subroutines, subprograms, etc..

Methods discussed above, some of which are illustrated by the flow charts, may be implemented by hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware or microcode, the program code or code segments to perform the necessary tasks may be stored in a machine or computer readable medium such as a storage medium.

Specific structural and functional details disclosed herein are merely representative for purposes of describing example implementations. Example implementations, however, be embodied in many alternate forms and should not be construed as limited to only the implementations set forth herein.

For example, a first element could be termed a second element, and, similarly, a second element could be termed a first element, without departing from the scope of example implementations. As used herein, the term and/or includes any and all combinations of one or more of the associated listed items.

It will be understood that when an element is referred to as being connected or coupled to another element, it can be directly connected or coupled to the other element or intervening elements may be present. In contrast, when an element is referred to as being directly connected or directly coupled to another element, there are no intervening elements present. Other words used to describe the relationship between elements should be interpreted in a like fashion (e.g., between versus directly between, adjacent versus directly adjacent, etc.).

The terminology used herein is for the purpose of describing particular implementations only and is not intended to be limiting of example implementations. As used herein, the singular forms a, an, and the are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms comprises, comprising, includes and/or including, when used herein, specify the presence of stated features, integers, steps, operations, elements and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and/or groups thereof.

For example, two figures shown in succession may in fact be executed concurrently or may sometimes be executed in the reverse order, depending upon the functionality/acts involved.

Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which example implementations belong.

Portions of the above example implementations and corresponding detailed description are presented in terms of software, or algorithms and symbolic representations of operation on data bits within a computer memory. These descriptions and representations are the ones by which those of ordinary skill in the art effectively convey the substance of their work to others of ordinary skill in the art. An algorithm, as the term is used here, and as it is used generally, is conceived to be a self-consistent sequence of steps leading to a desired result. The steps are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of optical, electrical, or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated.

In the above illustrative implementations, reference to acts and symbolic representations of operations (e.g., in the form of flowcharts) that may be implemented as program modules or functional processes include routines, programs, objects, components, data structures, etc., that perform particular tasks or implement particular abstract data types and may be described and/or implemented using existing hardware at existing structural elements. Such existing hardware may include one or more Central Processing Units (CPUs), digital signal processors (DSPs), application-specific-integrated-circuits, field programmable gate arrays (FPGAs) computers or the like.

Unless specifically stated otherwise, or as is apparent from the discussion, terms such as processing or computing or calculating or determining of displaying or the like, refer to the action and processes of a computer system, or similar electronic computing device, that manipulates and transforms data represented as physical, electronic quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system memories or registers or other such information storage, transmission or display devices.

Note also that the software implemented aspects of the example implementations are typically encoded on some form of non-transitory program storage medium or implemented over some type of transmission medium. The program storage medium may be magnetic (e.g., a floppy disk or a hard drive) or optical (e.g., a compact disk read only memory, or CD ROM), and may be read only or random access. Similarly, the transmission medium may be twisted wire pairs, coaxial cable, optical fiber, or some other suitable transmission medium known to the art. The example implementations not limited by these aspects of any given implementation.

Claim 1:
A computer-implemented method comprising
capturing a plurality of frames associated with a video file;
capturing an image corresponding to one of the plurality of frames;
performing a post-capture process on the image, the post-capture process is configured to improve a visual quality of the image and the post-capture process is not performed on the plurality of frames;
generating a color map between the captured image and a corresponding frame of the plurality of frames, wherein the color map is based on a color and a tone of the captured image and a color and tone of the corresponding frame;
and either:
tone correcting one or more selected frames associated with the video file based on the color map; and
color correcting the one or more selected frames based on the color map; or
generating a data structure including data representing the color map; and
storing the data structure as metadata in a header associated with the video.