Patent Description:
Conventional Computer Vision (CV) methods are designed to work with standard dynamic range images, typically represented by gamma-corrected <NUM>-bit color values. In other words, the transfer function is usually determined based on such a gamma curve. Moreover, the images are further produced by a camera's Image Signal Processing (ISP) pipeline, which, may reconstruct color from the Bayer pattern, may further reduce noise, enhance contrast and may tone-map the images. In doing so, the images can be encoded in a standard color space (e.g., BT rec. <NUM> (International Telecommunication Union (ITU) Recommendation ITU-R BT. <NUM>)), and can be presented on a display.

However, certain adverse issues arise from the conventional devices and methods As an example, many steps performed in the ISP pipeline may be redundant for some computer vision methods or applications, and may even degrade their performance. Further, tone-mapping may result in clipping of pixel values and contrast distortions that may degrade the performance of many computer vision algorithms.

A conventional camera pipeline for computer vision applications is illustrated in <FIG>.

According to the illustration <NUM> depicted in <FIG>, an image sensor <NUM> comprises at least one photodiode <NUM>, an amplifier <NUM> and an Analog to Digital Convenor (ADC) <NUM>, and provides image data to the image signal processor unit <NUM>. Moreover, several noises (e.g., photon noise, readout noise, ADC noise) are introduced by the image sensor <NUM>. The image signal processor unit <NUM> comprises a High Dynamic Range (HDR) merging unit <NUM>, a demosaicing unit <NUM>, and a transfer function <NUM>. The image signal processor <NUM> generates image data which are provided for the CV application <NUM>.

The conventional methods apply tone mapping or a transfer function intended to the visualization of images before the image is provided for a computer vision algorithm. Examples of such conventional transfer functions are gamma (for low dynamic range sensors and standardized in BT-<NUM>) and Perceptual Quantizer (PQ) (for example, intended for high dynamic range displays and standardized in ST-<NUM>).

However, certain adverse issues arise from the conventional devices and methods. As an example, the transfer functions intended for adapting image data for computer vision applications are not ideally harmonized with the requirements of the (CV) algorithms, and may therefore perform worse, especially in extreme conditions such as low illumination, high noise and high dynamic scene ranges.

<CIT> describes methods, apparatus and systems for inverse tone mapping of a high dynamic range image. <CIT> describes video coding using temporally coherent dynamic range mapping.

<CIT> describes an image processing device that includes a luminance acquiring unit and a local tone-mapping unit.

In view of the above-mentioned adverse issues, the embodiments of the invention aim to improve the conventional devices and methods. An objective is to provide a device and method for pre-processing image data based on a transfer function that provides a more robust signal for computer vision applications. Noise characteristics of the camera should be mitigated (for example, the noise profile of an image sensor of the camera). The device and method should improve the performance of computer vision algorithms that need to operate on the input from high dynamic range sensors.

The objective of the present invention is achieved by the embodiments provided in the enclosed independent claims.

Advantageous implementations of the embodiments are further defined in the dependent claims.

According to a first aspect of the invention, a device for pre-processing image data for a computer vision application is provided. The device is configured to obtain, by means of a channel, image data of an image sensor, wherein the image data comprises an intensity value for each of a plurality of image sensor pixels associated with the channel. The device is further configured to determine a Transfer Function (TF), the TF comprising a first part determined for a first set of the intensity values, and a second logarithmic part determined for a second set of the intensity values, wherein the first part and the second part of the TF are different. The device is further configured to apply the first part of the TF to the first set of the intensity values, and to apply the second logarithmic part of the TF to the second set of the intensity values, in order to obtain transformed image data.

The device may be, or may be incorporated in, a computer vision digital camera, a digital video recorder, a mobile phone, a smartphone, a play station, an augmented reality device, a virtual reality device, a personal computer, a desktop computer, a laptop and a notebook computer, a tablet or a pad device, an image signal processing module of an imaging device, a part of an optical flow generation for an ISP of an imaging device, a camera module of a high dynamic range imaging device, etc..

The first set of intensity values may comprise one continuous intensity range or several continuous intensity ranges. Accordingly, the second set of intensity values may comprise one continuous intensity range or several continuous intensity ranges. According to some embodiments, the first set of intensity values is a first continuous intensity range and/or the second set of intensity values is a second continuous intensity range. According to some embodiments, however, the first set of intensity values may comprise a plurality of discrete intensity values. According to some embodiments, the second set of intensity values may comprise a plurality of discrete intensity values.

The device obtains the image data of the image sensor. Moreover, the device determines the TF. The TF may be a function that maps intensity values to encoded values, i.e. each intensity value to an encoded value. The first part of the TF may have, in a logarithmic-representation, wherein a logarithmic scale is used along the x-axis that indicates the intensity values, a lower slope than the second part of the TF. The first part of the TF may be determined for the intensity range that is strongly affected by noise (e.g., having a low Signal to Noise Ratio (SNR)). The second logarithmic part of the TF may be determined based on a logarithmic mapping that may be desirable in the well-exposed (high SNR) portion of the intensity range as it can discount illuminant in gradient fields (e.g., since many computer vision algorithms rely on gradients).

In some embodiments, the TF may be determined based on an Opto-electrical and Electro-Optical Transfer Functions (OETF). For example, the device may determine the TF (e.g., the first part of the TF and the second part of the TF) based on an OETF curve.

The image sensor may be any image sensor, for example, it may be a Charge-Coupled Device (CCD) image sensor, a Complementary Metal-Oxide Semiconductor (CMOS), etc..

In some embodiments, the determined TF may be adapted to the noise profile of each type of camera sensor, i.e., the device may determine the TF, e.g. once for each type of the camera's image sensor (i.e., not for each specific sensor of the same camera sensor). For example, the TF is determined for the fixed or the preset configuration of the camera's image sensor. In other words, the TF determination may not be based on a dynamic adaptation at a later stage for a specific camera sensor.

Hereinafter, the terms "transform" and "transfer function" are used interchangeably, without limiting the present disclosure. Moreover, the terms "segment" and "part" are also used interchangeably, without limiting the present disclosure.

The device of the first aspect provides the advantage of (consistently) improving the performance of computer vision algorithms, specifically regardless of the dynamic range of the input and capture conditions (noise and illumination levels). This is achieved by the TF, which lets the computer vision algorithms perform better than a conventional TF. In particular, this may be an advantage over conventional devices, in which traditional ISP pipelines are designed for best visual quality, which does not necessarily lead to better performance for computer vision methods.

The device may comprise a circuitry. The circuitry may comprise hardware and software. The hardware may comprise analog or digital circuitry, or both analog and digital circuitry.

In some embodiments, the circuitry comprises one or more processors and a non-volatile memory connected to the one or more processors. The non-volatile memory may carry executable program code which, when executed by the one or more processors, causes the device to perform the operations or methods described herein.

In an implementation form of the first aspect, based on a logarithmic scale, a slope of the TF in the first part is smaller than a slope of the TF in the second logarithmic part.

Using such a TF improves the performance of computer vision algorithms/applications, specifically regardless of the dynamic range.

In particular, the device may determine, for the first part of the TF, a different (type of) function from the second part of the TF. Moreover, the first of the TF may comprise a smaller slope (in logarithmic depiction), which may have the advantage of reducing noisy contributions to computer vision. The logarithmic scale relates to the logarithmic representation of the TF described above. That is, the logarithmic scale may be used for the x-axis indicating the intensity values.

In a further implementation form of the first aspect, the first part of the TF is based on a logarithmic function.

In particular, the determined TF may be a noise-adaptive logarithmic transform which may have two-segment logarithmic transforms.

For example, the TF is based on a divided TF, which may comprise the first logarithmic part and the second logarithmic part, such that the first logarithmic part may comprise a reduced slope. Using such a TF significantly improves the performance of computer vision algorithms/applications.

In a further implementation not covered by the claims, but useful example for understanding the invention, the first part of the TF is based on a noise-variance stabilizing function. Using such a TF, the performance of computer vision algorithms/applications can be optimized.

In particular, the determined TF may be a noise-adaptive logarithmic transform which may have two segments including the first part being a noise-variance stabilizing transform and the second part being a logarithmic part.

In a further implementation form of the first aspect, the first part of the TF is a constant value.

This provides a simple implementation of the TF. In particular, the first part of the TF is zero.

The above-described implementation forms regarding the TF functions improve the performance of computer vision applications.

In a further implementation form of the first aspect, the device is further configured to determine a threshold intensity value dividing the TF into the first part and the second part of the TF based on a SNR threshold.

The SNR threshold may be determined based on a sensor noise profile. The threshold intensity value may be determined such, that the intensity values related to the first part of the TF are intensity values having a SNR below the SNR threshold, and the intensity values related to the second part of the TF are intensity values having a SNR above the SNR threshold.

In a further implementation form of the first aspect, the SNR threshold has a value within the range of -3dB and +3dB.

This SNR range has proven advantageous for improving the performance of computer vision algorithms/applications.

In a further implementation form of the first aspect, the TF is determined based on a noise profile of the image sensor.

In a further implementation form of the first aspect, the SNR threshold is obtained based on a noise profile of the image sensor.

The noise profile of the image sensor may indicate how much noise is produced by the image sensor, in the pixels, at different image sensor settings. The noise profile may be modelled based on a noise model, and/or may be measured, and/or may be provided by a provider/manufacturer of the image sensor. For example, the noise profile of the image sensor may be determined based on testing different SNR values and selecting the SNR value that leads to the best performance of the target CV application. In this way, the TF can be adjusted to the image sensor for achieving improved performance.

In particular, the noise profile of the image sensor may be determined based on (or may include) the photon noise, the read-out noise and the ADC noise, etc..

In a further implementation not covered by the claims, but useful example for understanding the invention, the SNR threshold is further obtained based on a type of the computer vision application and/or a gain parameter of the image sensor.

This allows customizing the TF to the type of computer vision application (algorithm) for achieving improved performance.

In summary, the above-described implementation forms regarding the SNR threshold improve the performance of computer vision applications.

In a further implementation form of the first aspect, the device is further configured to introduce a smoothing function to the TF for extending the threshold intensity value to a threshold range comprising a first lower threshold and a first upper threshold. Hereby, the smoothing function is for substituting the TF in the threshold range. Further, the substitution of the TF in the threshold range by the smoothing function induces the TF to be differentiable for any intensity values.

The smoothing function may comprises any function that is capable of smoothing the TF, in particular of smoothing the transition of the TF from the first part to the second part. As mentioned above, the first part and the second part of the TF are different, in particular may have different slopes, so that an abrupt transition may occur. Applying the smoothing function may make this transition less abrupt, and can thus further improve the performance of the computer vision application.

In a further implementation form of the first aspect, the smoothing function is based on a cubic function.

In a further implementation form of the first aspect, the device is further configured to obtain a ratio of the first part to the second part of the TF, wherein whether the substitution of the TF in the threshold range by the smoothing function occurs depends on the ratio of the first part to the second part of the TF.

In a further implementation form of the first aspect, the device is further configured to apply an inverse logarithmic TF to the transferred image data for obtaining image data or video data in a determined format according to the computer vision application.

In a further implementation form of the first aspect, the device is implemented in one of:.

A second aspect of the invention provides a method for pre-processing image data for a computer vision application. Hereby, the method comprises the step obtaining, by means of a channel, image data of an image sensor, wherein the image data comprises an intensity value for each of a plurality of image sensor pixels associated with the channel. The method further comprises the step determining a Transfer Function (TF), the TF comprising a first part determined for a first set of the intensity values, and a second logarithmic part determined for a second set of the intensity values, wherein the first part and the second logarithmic part of the TF are different. The method further comprises the step applying the first part of the TF to the first set of the intensity values and applying the second logarithmic part of the logarithmic TF to the second set of the intensity values for obtaining transformed image data.

In an implementation form of the second aspect, based on a logarithmic scale, a slope of the TF in the first part is smaller than a slope of the TF in the second logarithmic part.

In a further implementation form of the second aspect, the first part of the TF is based on a logarithmic function.

In a further implementation not covered by the claims, but useful example for understanding the invention, the first part of the TF is based on a noise-variance stabilizing function.

In a further implementation form of the second aspect, the first part of the TF is a constant value.

In a further implementation form of the second aspect, the method further comprises determining a threshold intensity value dividing the TF into the first part and the second part of the TF based on a SNR threshold.

In a further implementation form of the second aspect, the SNR threshold has a value within the range of -3dB and +3dB.

In a further implementation form of the second aspect, the TF is determined based on a noise profile of the image sensor.

In a further implementation form of the second aspect, the SNR threshold is obtained based on a noise profile of the image sensor.

In a further implementation form of the second aspect, the method further comprises introducing a smoothing function to the TF for extending the threshold intensity value to a threshold range comprising a first lower threshold and a first upper threshold, wherein the smoothing function is for substituting the TF in the threshold range, and wherein the substitution of the TF in the threshold range by the smoothing function induces the TF to be differentiable for any intensity values.

In a further implementation form of the second aspect, the smoothing function is based on a cubic function.

In a further implementation form of the second aspect, the method further comprises obtaining a ratio of the first part to the second part of the TF, wherein whether the substitution of the TF in the threshold range by the smoothing function occurs depends on the ratio of the first part to the second part of the TF.

In a further implementation form of the second aspect, the method further comprises applying an inverse logarithmic TF to the transferred image data for obtaining image data or video data in a determined format according to the computer vision application.

In a further implementation form of the second aspect, the method is implemented in one of:.

The method of the second aspect and its implementation forms provide the same advantages and effects as the device of the first aspect and its respective implementation forms.

A third aspect of the invention provides a computer program which, when executed by a computer, causes the method of second aspect (and/or one of the implementation form of the second aspect) to be performed.

The computer program of the third aspect and its implementation forms provides the same advantages and effects as the device of the first aspect and its respective implementation forms.

<FIG> is a schematic view of device <NUM> for pre-processing image data <NUM> for a computer vision application <NUM> according to an embodiment of the present invention.

The device <NUM> may be, or may be incorporated in, a computer vision digital camera, a digital video recorder, a mobile phone, a smartphone, an image signal processing module of an imaging device, a part of an optical flow generation for an ISP of an imaging device, a camera module of a high dynamic range imaging device, etc..

The device <NUM> is configured to obtain, by means of a channel <NUM>, image data <NUM> of an image sensor <NUM>, wherein the image data <NUM> comprises an intensity value for each of a plurality of image sensor pixels associated with the channel <NUM>. As can be deduced from <FIG>, the channel <NUM> may be adapted to interconnect the image sensor <NUM> and the device <NUM>. For obtaining the image data <NUM> of the image sensor <NUM>, the device <NUM> may comprise an image data reception unit <NUM>.

The device <NUM> is further configured to determine a TF <NUM>. For doing so, the device <NUM> may comprise a transfer function determination unit <NUM>. The TF <NUM> determined by the transfer function determination unit <NUM> may comprise a first part <NUM> determined for a first set of the intensity values, and a second logarithmic part <NUM> determined for a second set of the intensity values, wherein the first part <NUM> and the second part <NUM> of the TF <NUM> are different. The determined TF <NUM> may then be provided to a transfer function application unit <NUM>, wherein transfer function application unit <NUM> may be adapted to apply the transfer function to the image data <NUM>. In doing so, transformed image data <NUM> may be obtained.

The TF <NUM> may be a noise-adaptive transfer function determined for a specific computer vision application <NUM>.

The TF <NUM> comprising a first part <NUM> determined for a first set of the intensity values. For example, the first part <NUM> of the TF <NUM> may have, in a logarithmic (log)-representation, wherein the intensity values are plotted using a logarithmic scale on the x-axis and the encoded values are plotted using a linear scale on the y-axis (see e.g., <FIG>), a lower slope (compared to the second part <NUM> of the TF <NUM>) and may be determined for the intensity range of the intensity values that is strongly affected by noise (e.g., having a low SNR).

The second logarithmic part <NUM> of the TF <NUM> determined for the second set of the intensity values having high SNR. Moreover, the first part and the second part of the TF are different.

The device <NUM> is further configured to apply the first part <NUM> of the TF <NUM> to the first set of the intensity values and apply the second logarithmic part <NUM> of the TF <NUM> to the second set of the intensity values, in order to obtain transformed image data. As can be deduced from <FIG>, such way of applying the TF <NUM> to the image data <NUM> may be performed by the transfer function application unit <NUM>. Hereby, applying the TF <NUM> to the image data <NUM> may enable to obtain transformed image data <NUM>, which may be forwarded to an output unit for transformed image data <NUM>. The transformed image data <NUM> forwarded to the output unit for transformed image data <NUM> may then be provided to the computer vision application <NUM>.

For example, the device <NUM> may adapt the transfer function <NUM> to the noise characteristic of the image sensor <NUM> such that it relies on logarithmic mapping for better invariance to illumination.

Furthermore, the device <NUM> may adaptively select the logarithmic mapping in order to make the best use of the available bit-depth of the encoded values. Also, the logarithmic mapping can be combined with noise-variance stabilizing methods, too.

Optionally, the device <NUM> may determine a transfer function <NUM> for encoding color and luma values that may improve the performance of computer vision methods. In the first order, the focus is a transfer function for encoding linear intensity values into integer values, which are suitable input to computer vision methods.

The device <NUM> may comprise a circuitry (not shown in <FIG>). The circuitry may comprise hardware (for example, a camera, an image sensor, a memory, a central processing unit, etc.) and software (for example, an image processing software, a video processing software, a computer vision application, etc.). The hardware may comprise analog or digital circuitry, or both analog and digital circuitry. In some embodiments, the circuitry comprises one or more processors and a non-volatile memory connected to the one or more processors. The non-volatile memory may carry executable program code which, when executed by the one or more processors, causes the device to perform the operations or methods described herein.

Reference is made to <FIG> which is a schematic view of a system <NUM> for determining a noise profile for the device <NUM>. For example, a noise profile for an image sensor <NUM> of the device <NUM> may be determined.

The system <NUM> may be based the device <NUM>, or the device <NUM> may be incorporated in the system <NUM>. Without limiting the present disclosure, in following it is assumed that the device <NUM> is incorporated in the system <NUM>. Moreover, the device <NUM> is exemplarily based on (e.g., comprising) a computer vision digital camera <NUM> (for example, it comprises the CV digital camera <NUM> which comprises the image sensor <NUM>). The system <NUM> further comprises a light box <NUM> with square calibration targets of different light emission (right) used to measure the noise characteristic.

Determining the noise profile of the device <NUM> may be, for example, estimating a noise profile of the image sensor <NUM> of the digital camera <NUM> of the device <NUM>, estimating different parameters of the TF <NUM>, etc..

Different computer vision methods may be robust to the noise (e.g., photon noise, read-out noise and ADC noise, etc.) introduced by the image sensor <NUM>. The system <NUM> (and/or the device <NUM>) is configured to consider the introduced noises when determining the transfer function <NUM> and to simulate different exemplary cameras <NUM> (for several relevant cameras) under different illumination levels. For example, a model (e.g., a noise profile of the image sensor) for the camera noise characteristics may be determined, and the required parameters for the determined model may be estimated. A list of the symbols used for determining the noise model are presented in table I.

It is assumed that the image sensor <NUM> of the camera <NUM> collects Φ electron per second. Moreover, if the gain of the image sensor being g, the image sensor's readout for the color channel of c (c∈ {R,G,B}) over the exposure time t may be estimated according to Eq. (<NUM>): <MAT> where the coefficient kc may account for any digital gain and the white balance correction applied after the charge has been converted into digital units. The symbol n is a normally distributed random variable that models the noises (for example, the photon noise, the read out noise and the ADC noise).

Furthermore, the variance of the noises may be modelled according to Eq. (<NUM>): <MAT>.

In order to measure the parameters for the camera <NUM>, a series of images of a calibration target are captured. The calibration target comprises a light box <NUM> covered by a cardboard with square cutouts, which are covered by a combination of neutral density filters (for example, the recommended densities may be <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, and <NUM>). The camera's exposure time and the gain are adjusted so that the brightest square (ND <NUM>) is registered close to the maximum digital value in the image but isn't saturated. In the image series, the gain is gradually increased and compensated for increased pixel values by shortening the exposure time, in which five images are taken for each exposure/gain pair.

Table II presents the estimated noise parameters fitted for several exemplary cameras (without limiting the present disclosure to a specific camera, a specific image sensor, etc.). All camera RAW images assumed to be <NUM> bit in order to perform an easier comparison.

The device <NUM> (and/or the system <NUM>) may further determine the transfer function <NUM>. In the following, several exemplary procedures for determining the transfer function <NUM> are presented.

A majority of camera sensors, including CCD and CMOS, exhibits a linear response to light, i.e., they count the number of incoming photons and register values that are linearly related to radiometric quantities (irradiance). Furthermore, since the linear color values are strongly perceptually non-uniform and require high bit-depths to store, they are typically tone-mapped into a gamma-corrected representation. The cameras are designed to capture visually pleasant content typically use S-shaped tone-curves, mimicking the response of an analog photographic film. Computer vision cameras more often apply a transfer power function with the exponent less than <NUM>, commonly known as "gamma".

According to exemplary measurements performed by the system <NUM> (including the device <NUM>), it may be stated that the although such gamma encoding is suitable for sensors of moderate dynamic range, and for standard displays, this is not the best choice for the image sensors of higher dynamic range and when the images are intended for computer vision methods rather than for viewing.

In following, several types of transfer functions are described, for example, the device <NUM> may determine the TF <NUM> based on one of the following procedures.

Reference is now made to <FIG>, which is a schematic view of a diagram <NUM> comprising a checkerboard pattern <NUM> used for obtaining a logarithmic transform with illumination-invariant gradients.

The checkerboard pattern <NUM> is directly illuminated on the left side <NUM> and in shadow on the right side <NUM>. Moreover, when computing the gradients (e.g., the partial derivative) by the system <NUM> (and/or the device <NUM>), it is determined that the partial derivatives will be different for both sides when computed in a linear color space.

A large portion of computer vision methods does not operate directly on pixel values but rather on gradients. For example, the classical descriptors, such as the Scale-Invariant Feature Transform (SIFT) or the Histogram of Oriented Gradients (HOG), compute histograms of gradient magnitudes and orientations. The first convolutional layers in deep neural networks often extract features that represent differences between neighboring pixels. Therefore, preserving gradient values across the range of illumination conditions is considered to be a desirable property of color encoding for computer vision.

In diagram <NUM> of <FIG>, in which the checkerboard <NUM> is directly illuminated on one side and in a shadow on the other. Generally, the partial derivatives are different for both sides of the pattern when computed for linear color values. However, if the derivatives are computed for the logarithms of linear values, the partial derivatives are the same. In other words, the gradients stay the same regardless of illumination. Based on this analysis, linear camera color values are supposed to be encoded in the logarithmic domain to improve the performance of gradient-based computer vision methods.

However, the logarithmic transform may strongly enhance the low color values, which are most affected by the noise. This may introduce strong inconsistencies in the level of noise, and may further break another common assumption of computer vision methods, as will be discussed below.

The device <NUM> may determine the TF <NUM> which may be based on a noise-variance stabilizing transform. For example, the TF <NUM> (e.g., the first part <NUM> of the TF <NUM>) may be based on the noise-variance stabilizing transform (the noise-variance TF).

For instance, some computer vision algorithms assume that the camera noise (i.e., the photon noise, the ADC noise, the readout noise) is normally distributed and has a constant variance across the intensity range. This assumption is a reasonable approximation for gamma-corrected <NUM>-bit images, but it may not be used for linear color values coming directly from a camera's image sensor. However, it may be possible to transform the captured linear values into the code space with uniform variance using a noise-variance stabilizing transform.

Moreover, the device <NUM> may determine (e.g., by means of the noise profile estimated according to <FIG>) the noise-variance stabilizing transform (for example, which may be the first part <NUM> of the TF <NUM>) according to Eq. (<NUM>): <MAT> where the Ec(Yc) is the TF <NUM>.

Reference is now made to <FIG>, which show examples of the designed OETF curves for determining a noise-adaptive transfer function.

For example, the device <NUM> may determine a noise-adaptive transfer function <NUM> based on the OETF curves.

A logarithmic TF may keep the gradients under different illumination but it may lead to stronger noise artefacts for low pixel values. The device <NUM> may determine a noise-adaptive transfer functions <NUM> such that it has two parts comprising the first part <NUM> of the TF <NUM> and the second part <NUM> of the TF <NUM> which are different for the low and the high pixel values. The first part <NUM> of the TF <NUM> and the second part <NUM> of the <NUM> may be separated by the threshold value <NUM>. Furthermore, the determined TF <NUM> may be the same for all the three color channels R, G, B if a single noise profile is measured for all R, G, B pixels together. The transfer function (TF <NUM>) may also be separated for R, G, B, if a noise profile is measured for each of the color channel. Examples of all the three variants of the proposed OETFs are shown in diagram 400A of <FIG> and diagram 400B of <FIG>.

In the diagram 400A of <FIG> and the diagram 400B of <FIG>, the curve indicated by "Log-SNR+<NUM>" is a logarithmic curve with threshold of 3dB. The curve indicated by "<NUM>-seg-log SNR+<NUM> SL1/<NUM>" is a two segment logarithmic curve with threshold of 9dB in SNR and the ratio between two slopes is <NUM> to <NUM>. Moreover, the curve indicated by "Log-SNR+25_Var-stablizing" is a combination of logarithmic and variance-stabilizing method with a threshold of 25dB in SNR.

The diagram 400A of <FIG> refers to encoded value vs linear intensity, whereas the diagram 400B of <FIG> refers to encoded value vs logarithmic of the intensity values. They are equivalent. Furthermore, the linear intensity values are pixel values in R or G or B.

For instance, while a logarithmic TF preserves gradients, it may also enhance the noise. The device <NUM> may determine the TF <NUM> by using a logarithmic TF only for the digital values that are above a certain SNR and clamp to zero the values that are too noisy to be useful for a computer vision application. The choice of an SNR threshold may offer a trade-off between using higher precision to represent high digital values and clipping a certain portion of the lower digital range.

The device <NUM> may determine the TF <NUM> according to Eq. (<NUM>), as follows: <MAT>.

Here, the first part <NUM> (<NUM>) of the TF <NUM> (Elog-snr-x(Y)) is determined for Y < T (indicated with reference number <NUM> in <FIG> and <FIG>), and the second part <NUM> ((log(Y) - log(T))·s) of the TF <NUM> (Elog-snr-x(Y)) is determined for Y ≥ T <NUM>.

The symbol s is a scale factor which ensures that the maximum output value of the transfer function is <NUM>: Eq. (<NUM>): <MAT>.

The symbol x stands for SNR threshold and the T is the corresponding threshold for pixel values and their relationship may be obtained according to Eq. (<NUM>): <MAT>.

The SNR threshold x is to be set based on sensor noise profile and the application. Recommended value range is [-3dB, +3dB]. Given x, T <NUM> can be computed using an inverse operation.

Furthermore, considering the logarithmic function, herein, a natural log is written for simplicity. Also, a logarithmic function with any basis value, like log2 or log10 may also work, and the output of the transfer function <NUM> may be the same no matter which logarithmic basis is taken.

Additionally, the input value Y here can either represent the luminance value or the value of one color channel, R or G or B. For simplicity, the subscript of c for color channels is omitted.

Besides, a limitation of the noise-adaptive transfer function determined based on Eq. (<NUM>) may be that the values below the SNR threshold are set to zero and (all) information being lost. Still, some computer vision algorithms can extract information even from such very noisy signal. Therefore, it is desirable to preserve even the low SNR range. However, allocating too large range in the encoded values to the noisy part of the intensity range would reduce contrast in the brighter part of the intensity range, and lower performance of the computer vision algorithms in that range. Therefore, in some embodiments, the device <NUM> may split the TF <NUM> into two logarithmic segments (i.e., the first part <NUM> of the TF <NUM> determined for the low-SNR segment (low intensity values) and the second part <NUM> of the TF <NUM> determined for the high-SNR segment (high intensity values) are both logarithmic functions.

Moreover, the transition between the two parts is selected (e.g., as before) based on the SNR threshold. The slopes of the logarithmic mapping functions may be selected so that the slope of the high-SNR segment is N times of the slope of the low-SNR segment, where N may be any value large than <NUM> (for example, N=<NUM> is recommended). The slope refers to the tangent of the mapping function in the log-linear coordinates. For example, the device <NUM> may determine the TF <NUM> (E<NUM>seg-log(Y)) according to Eq. (<NUM>): <MAT> where, the symbol s (the scale factor) may be estimated according to Eq. (<NUM>) as follow: <MAT> and where r is the preset ratio of the first segment log curve slope and the second segment log curve slope. This value may be adjusted by the device (i.e., the user of device) based on image sensors and the CV applications. For example, the recommended value is <NUM>.

Here, the TF <NUM> (E<NUM>seg-log(Y)) comprises the first part <NUM> (log(Y) ·r·s) that is determined for Y < T (indicated with reference number <NUM> in <FIG> and <FIG>), and the second part ((log(Y) - log(T) · (<NUM> - r)) ·s ) that is determined for Y ≥ T <NUM>.

One potential issue of, determining the TF <NUM> based on the two parts logarithmic curve may be that the transfer function <NUM> is incontinuous at the conjunction point T <NUM>. The slope of the second part (being used for determining the second part <NUM> of the TF <NUM>) is generally a few times higher than the first part (being used for determining the first part <NUM> of the TF <NUM>), and thus the transition at T <NUM> is rather abrupt.

Therefore, the device <NUM> may extend the two parts logarithmic transform (TF) with a smooth transition between two parts. For example, the device <NUM> may introduce a smoothing function <NUM> (e.g. a cubic function) for this purpose and may further fit in with the two segments at a first lower threshold (T<NUM>) <NUM> and a first upper threshold (T<NUM>) <NUM>. The values of the first lower threshold (T<NUM>) <NUM> and the first upper threshold (T<NUM>) <NUM> can be chosen freely, subject to the conditions that T<NUM><T and T<NUM>>T. (for example, it is recommended that T<NUM> <NUM> and T<NUM> <NUM> are close to T <NUM>, e.g., T<NUM>=<NUM>. 8T and T<NUM>=<NUM>. An example of smoothing the two parts of the designed OETF curve is illustrated in <FIG> (for example the device <NUM> may performs the smoothing operation).

The device <NUM> may determine the two-segment logarithmic TF <NUM> (E<NUM>seg-log-smooth(Y)) with smooth transition according to Eq. (<NUM>) as follow: <MAT>.

Furthermore, the device <NUM> may (uniquely) determine the four parameters, i.e., a, b, c and d of the cubic function P(Y), using the four boundary conditions according to Eq. (<NUM>) to Eq. (<NUM>) as follow: <MAT> and, <MAT> and, <MAT> and, <MAT>.

In some implementations not covered by the claims, but useful example for understanding the invention, the device <NUM> may determine the TF <NUM>, for example, based on the two-segment logarithmic transfer function and the noise-variance stabilizing transfer function.

For example, the noise-variance stabilizing transfer function may be combined with the logarithmic transfer function to form a two-segment transfer function. The former one (the noise-variance stabilizing transfer function) is used for low pixel values where noise has stronger impact whereas the latter one (logarithmic transfer function) for higher pixel values. For example, the device <NUM> may determine the TF <NUM> (Elog-nvsta(Y)) based Eq. (<NUM>) as follow: <MAT>.

Similar to the two-segment log transform (i.e., the first logarithmic part of the TF and the second logarithmic part of the TF), a smooth transition can also be applied here and TF (Elog-nvsta(Y)) may be estimated according to Eq. (<NUM>): <MAT>.

The four parameters, i.e., m, n, p and q of the cubic function Q(Y) can be uniquely determined using the four boundary conditions in the following: Eq. (<NUM>) to Eq. (<NUM>): <MAT> and, <MAT> and, <MAT> and, <MAT>.

Reference is now made to <FIG>, which is a schematic view of the device <NUM> comprising a TF <NUM> used in an image signal processing pipeline <NUM> of a camera, according to an embodiment of the present invention.

The device <NUM> of <FIG> is exemplary based on a CV digital camera. Moreover, the TF <NUM> is used (incorporated) in the image signal processing modules of the device <NUM> (camera).

The image signal processing (ISP) herein does not refer to a general concept of processing the image, but the specific systems in cameras, which convert sensor raw data, typical Bayer-pattern but possibly also other formats, to standard RGB or YUV format, which are appropriate for image and video storage in typical formats (for example, known conventional formats such as Joint Photographic Experts Group (JPEG), Portable Network Graphics (PNG), MPEG Layer-<NUM> Audio (MP4), MKV, etc.).

The camera ISP <NUM> is using the transfer function <NUM> as it is depicted in <FIG>. The raw data comes from the image sensor <NUM> (i.e., in <FIG> the image sensor and the ADC are in one unit) and goes into the camera ISP unit <NUM>. Camera ISP unit <NUM> comprises the fixed pattern noise suppression unit <NUM> (may perform a denoising operation), and the demosaicing unit <NUM> and the determined TF <NUM>. This is an exemplary (minimal) ISP setup and there may be more processing steps before and after the TF <NUM>.

The output of the camera ISP <NUM>, image or video either in RGB format or YUV format, is sent to the computer vision application unit <NUM>, e.g., for face detection, face recognition, object detection, etc..

The implementation of the transfer function <NUM> may depend on three parameters. The first parameter is the image sensor noise profile, which is computed offline. The second parameter is the image sensor's gain. This parameter is either determined automatically by the camera based on illumination or other conditions or it may be set by the camera's user. The third parameter is the SNR value, and these values determines the threshold value T <NUM> in the TF <NUM>. The SNR parameter is related to the CV application and is preset for every different CV application.

Reference is now made to <FIG>, which is a schematic view of a flowchart of a procedure <NUM> for determining the OETF component.

At <NUM>, the device <NUM> obtains the gain value (g) from the image sensor <NUM>. For example, it may obtained dynamically.

At <NUM>, the device <NUM> computes the SNR threshold (x) or receive it from the user's settings. For example, it may be computed dynamically.

At <NUM>, the device <NUM> computes the OETF threshold (T) <NUM> using the SNR threshold (x) and the noise profile of the image sensor. For example, the noise profile may be computed offline, and may further be stored in the camera and remains the same.

At <NUM>, the device <NUM> determines the parameters for the two segments of the OETF curve (the first part <NUM> and the second part <NUM> of the TF <NUM>).

Moreover, when the gain value (g) or the SNR threshold (x) becomes a different value, the device <NUM> may compute the OETF again.

Moreover, the OETF curve may be used for determining the TF <NUM>, e.g., the first part <NUM> of the TF <NUM> and/or the second part <NUM> of the TF <NUM>.

Reference is made to <FIG> which is a schematic view of the device <NUM> comprising a TF <NUM> used for generating optical flows for Bayer-pattern raw data, according to an embodiment of the present invention.

The device <NUM> may determine the TF <NUM> that may help generating optical flows for Bayer-pattern raw data. For example, the determined TF <NUM> is used to generate more accurate optical flow to assist the camera's ISP <NUM>. For instance the transfer function <NUM> can also be used in the camera ISP <NUM> herein with a different parameter setting.

In this embodiment, the transfer function <NUM> is applied at an early stage of the device <NUM>, i.e., directly applied to the raw data of the image sensor <NUM>. The optical flow is generated using the output of the transfer function <NUM>. The generated optical flow is forwarded to the camera's ISP <NUM>. Note that, not all camera's ISP <NUM> depends on the transfer function <NUM>, but some of the camera ISP may benefit from optical flow <NUM>. The output of the camera ISP <NUM> is sent to the CV application unit <NUM>.

Moreover, one function of the transfer function <NUM> is a pre-processing step of the raw data for optical flow <NUM>. In this manner, the accuracy of the output of the optical flow unit <NUM> is expected to be improved. Higher accuracy of optical flow may lead to better quality of camera ISP output images. Finally, a higher accuracy of CV applications is expected. Note that the transfer function <NUM> may also be used in the camera ISP, for example, similar to the embodiment of <FIG>, and for the parameter setting, the gain is the same and the SNR threshold may be different.

Reference is now made to <FIG>, which is a schematic view of the device <NUM> comprising a TF <NUM> used on the sensor side and in a camera module <NUM> for performing a bit depth compression according to an embodiment of the present invention.

In the embodiment of <FIG>, the device <NUM> determines the TF <NUM>. Moreover, the transfer function <NUM> is used on the image sensor <NUM> side and in the camera's module <NUM> before compressing or quantizing the HDR raw data to lower bit depths.

The transfer function <NUM> may be used to solve the issue of limited bandwidth between the camera module <NUM> and the camera's ISP <NUM>. The standard connection interface is the Mobile Industry Processor Interface (MIPI) / the Camera Serial Interface (CSI). Although there has been a variant supporting higher bit depth, for legacy or cost reason, the <NUM> bit bandwidth often exists in industry. The HDR camera sensor data, however, are often <NUM> bits or <NUM> bits, and there is a trend for even higher bit depth. Therefore, the HDR image data are quantized to <NUM> bit in this scenario. Direct compression of the image sensor data, however, results in poor performance. And the transfer function <NUM> may be used before quantization.

For example, the HDR sensor raw data, e.g. <NUM> bits, is transformed to nonlinear domain using the determined transfer function <NUM>. Then it may be quantized or compressed to low bit depth, e.g., <NUM> bit to fit in the MIPI/CSI standard. For example, it may be uniform scalar quantization. Furthermore, there may be no predictive coding, no entropy coding, etc..

The camera ISP pipeline <NUM> comprises the inverse of the transfer function <NUM> and the other ISP components <NUM>. At the beginning of the camera ISP pipeline <NUM>, the inverse of the transfer function <NUM> is applied and <NUM> bit HDR raw data in the linear domain are reconstructed. For example, it is converted back to linear domain, because of the ISP processes linear data. These data may be further processed by the ISP pipeline and the output is used in CV applications unit <NUM>. The role of the transfer function <NUM> in this embodiment may be to reduce the distortion between the original <NUM> bit data and the reconstructed <NUM> bit data for CV.

<FIG> shows a method <NUM> according to an embodiment of the invention for pre-processing image data <NUM> for a computer vision application <NUM>. The method <NUM> may be carried out by the device <NUM>, as it is described above.

The method <NUM> comprises a step <NUM> of obtaining, by means of a channel <NUM>, image data <NUM> of an image sensor <NUM>, wherein the image data <NUM> comprises an intensity value for each of a plurality of image sensor pixels associated with the channel <NUM>.

The method <NUM> further comprises a step <NUM> of determining a TF <NUM>, the TF <NUM> comprising a first part <NUM> determined for a first set of the intensity values, and a second logarithmic part <NUM> determined for a second set of the intensity values, wherein the first part <NUM> and the second part <NUM> of the TF <NUM> are different.

The method <NUM> further comprises a step <NUM> of applying the first part <NUM> of the TF <NUM> to the first set of the intensity values and apply the second logarithmic part <NUM> of the TF <NUM> to the second set of the intensity values, in order to obtain transformed image data <NUM>.

Claim 1:
A device (<NUM>) for pre-processing image data (<NUM>) for a computer vision application (<NUM>), the device (<NUM>) being configured to:
obtain, by means of a channel (<NUM>), image data (<NUM>) of an image sensor (<NUM>), wherein the image data (<NUM>) comprises an intensity value for each of a plurality of image sensor pixels associated with the channel (<NUM>),
determine a Transfer Function (<NUM>), TF, the TF (<NUM>) comprising a first part (<NUM>) determined for a first set of the intensity values, and a second logarithmic part (<NUM>) determined for a second set of the intensity values, wherein the first part (<NUM>) and the second part (<NUM>) of the TF (<NUM>) are different, and
apply the first part (<NUM>) of the TF (<NUM>) to the first set of the intensity values and apply the second logarithmic part (<NUM>) of the TF (<NUM>) to the second set of the intensity values, in order to obtain transformed image data (<NUM>);
characterised in that
the device is further configured to determine a threshold intensity value (<NUM>) dividing the TF (<NUM>) into the first part (<NUM>) and the second part (<NUM>) of the TF (<NUM>) based on an SNR threshold.