Exchanging an HDR-combined stream and associated exposures between a camera sensor module and a vision processing system

In an embodiment, a camera sensor module is communicatively coupled to a vision processing system. The camera sensor module captures a plurality of exposures, produces an HDR-combined stream, and sends both the HDR-combined stream and the plurality of exposures to the vision processing system. A first ISP of the vision processing system performs color processing to produce a color-processed HDR-combined stream for neural network processing, presentation to a human, and/or computer vision processing. At least one other ISP of the vision processing system performs color processing to collect exposure-specific statistics on at least one of the plurality of exposures for determining camera calibration data for the camera sensor module.

BACKGROUND

Embodiments relate to exchanging a high-dynamic range (HDR)-combined stream between a camera sensor module and a vision processing system.

Typical camera sensors found in consumer applications such as phones and tablet computers output a single exposure per frame time, typically 10-14 bits per pixel. Camera sensors may be designed and configured to capture multiple exposures per frame time, whereby multiple exposures can be combined/merged together to form a high dynamic range (HDR) image stream. For example, wide or ultra-HDR sensors achieving at least 120 dB may be used for object detection in automotive systems, whereby an HDR image stream (e.g., 20-24 bits or lower, depending on whether companding is used) of at least 120 dB may be achieved via the merge of multiple (e.g., 3 or 4) such exposures.

In certain camera implementations, such as automotive systems, one or more camera sensors may be arranged separately from a vision processing system that performs various processing on the image data captured by the one or more camera sensors. Image signal processors (ISPs) of some vision processing systems do not support HDR-combining, in which case the HDR-combining may be performed by an on-sensor processing system integrated with each respective camera sensor, with the on-sensor processing system providing the HDR-combined stream of images to the vision processing system.

In systems where HDR-combining is performed by on-sensor processing systems integrated with the respective camera sensors, it is difficult to obtain individual exposure-specific statistics (e.g., exposure histograms, etc.) for calibrating the respective camera sensors. In particular, exposure-level information is lost when the exposures are merged to form the HDR-combined stream of images. Alternatively, the exposure-level information may be extracted by the integrated on-sensor processing systems, but this increases the costs and complexity of the camera sensors.

SUMMARY

An embodiment is directed to a method of operating a camera sensor module that is communicatively coupled to a vision processing system, including capturing a plurality of exposures, performing high-dynamic range (HDR) combining on the plurality of exposures to produce an HDR-combined stream, and communicating, over a communications link to the vision processing system, (i) the HDR-combined stream, and (ii) the plurality of exposures.

Another embodiment is directed to a method of operating a vision processing system that is communicatively coupled to a camera sensor module, including receiving, from the camera sensor module over a communications link, (i) a plurality of exposures captured by the camera sensor module, and (ii) a high-dynamic range (HDR)-combined stream of the plurality of exposures based on HDR-combining performed at the camera sensor module, performing, by a first image signal processor (ISP) among a plurality of ISPs, color processing on the HDR-combined stream to produce a color-processed HDR-combined stream for neural network processing, presentation to a human, and/or computer vision processing, and performing, by at least one other ISP among the plurality of ISPs, color processing to collect exposure-specific statistics on at least one of the plurality of exposures for determining camera calibration data for the camera sensor module.

Another embodiment is directed to a camera sensor module that is communicatively coupled to a vision processing system, including means for capturing a plurality of exposures, means for performing high-dynamic range (HDR) combining on the plurality of exposures to produce an HDR-combined stream, and means for communicating, over a communications link to the vision processing system, (i) the HDR-combined stream, and (ii) the plurality of exposures.

Another embodiment is directed to a vision processing system that is communicatively coupled to a camera sensor module, including means for receiving, from the camera sensor module over a communications link, (i) a plurality of exposures captured by the camera sensor module, and (ii) a high-dynamic range (HDR)-combined stream of the plurality of exposures based on HDR-combining performed at the camera sensor module, means for performing, by a first image signal processor (ISP) among a plurality of ISPs, color processing on the HDR-combined stream to produce a color-processed HDR-combined stream for neural network processing, presentation to a human, and/or computer vision processing, and means for performing, by at least one other ISP among the plurality of ISPs, color processing to collect exposure-specific statistics on at least one of the plurality of exposures for determining camera calibration data for the camera sensor module.

Another embodiment is directed to a camera sensor module that is communicatively coupled to a vision processing system, including a camera sensor configured to capture a plurality of exposures, an on-sensor processing system configured to perform high-dynamic range (HDR) combining on the plurality of exposures to produce an HDR-combined stream, and a communications interface configured to communicate, over a communications link to the vision processing system, (i) the HDR-combined stream, and (ii) the plurality of exposures.

Another embodiment is directed to a vision processing system that is communicatively coupled to a camera sensor module, including a communications interface configured to receive, from the camera sensor module over a communications link, (i) a plurality of exposures captured by the camera sensor module, and (ii) a high-dynamic range (HDR)-combined stream of the plurality of exposures based on HDR-combining performed at the camera sensor module, a first image signal processor (ISP) among a plurality of ISPs, the first ISP configured to perform color processing on the HDR-combined stream to produce a color-processed HDR-combined stream for neural network processing, presentation to a human, and/or computer vision processing, and at least one other ISP among the plurality of ISPs, the at least one other ISP configured to perform color processing to collect exposure-specific statistics on at least one of the plurality of exposures for determining camera calibration data for the camera sensor module.

Another embodiment is directed to a non-transitory computer-readable medium containing instructions stored thereon, which, when executed by a camera sensor module that is communicatively coupled to a vision processing system, cause the camera sensor module to perform operations, the operations including at least one instruction to cause the camera sensor module to capture a plurality of exposures, at least one instruction to cause the camera sensor module to perform high-dynamic range (HDR) combining on the plurality of exposures to produce an HDR-combined stream, and at least one instruction to cause the camera sensor module to communicate, over a communications link to the vision processing system, (i) the HDR-combined stream, and (ii) the plurality of exposures.

Another embodiment is directed to a non-transitory computer-readable medium containing instructions stored thereon, which, when executed by a vision processing system that is communicatively coupled to a camera sensor module, cause the vision processing system to perform operations, the operations including at least one instruction to cause the vision processing system to receive, from the camera sensor module over a communications link, (i) a plurality of exposures captured by the camera sensor module, and (ii) a high-dynamic range (HDR)-combined stream of the plurality of exposures based on HDR-combining performed at the camera sensor module, at least one instruction to cause a first image signal processor (ISP) among a plurality of ISPs of the vision processing system to perform color processing on the HDR-combined stream to produce a color-processed HDR-combined stream for neural network processing, presentation to a human, and/or computer vision processing, and at least one instruction to cause at least one other ISP among the plurality of ISPs of the vision processing system to perform color processing to collect exposure-specific statistics on at least one of the plurality of exposures for determining camera calibration data for the camera sensor module.

DETAILED DESCRIPTION

Aspects of the disclosure are disclosed in the following description and related drawings directed to specific example embodiments of the disclosure. Alternate embodiments may be devised without departing from the scope of the disclosure. Additionally, well-known elements of the disclosure will not be described in detail or will be omitted so as not to obscure the relevant details of the disclosure.

The words “exemplary” and/or “example” are used herein to mean “serving as an example, instance, or illustration.” All of the embodiments described herein are intended as exemplary, and no specific embodiment is necessarily intended to be construed as preferred or advantageous over other embodiments. Likewise, the term “embodiments of the disclosure” does not require that all embodiments of the disclosure include the discussed feature, advantage or mode of operation.

FIG. 1illustrates a system control diagram of a vehicle100in accordance with an embodiment of the disclosure. InFIG. 1, the vehicle100includes an infotainment/telematics hub105that is connected via cabling to various peripheral devices, such as camera sensor modules110-125. Other peripheral devices to which the infotainment/telematics hub105is connected include a display gauge, audio devices (e.g., speakers, etc.), a storage device, a network bridge, and so on.

FIG. 2illustrates a system200depicting components of and interconnections between the infotainment/telematics hub105and the camera sensor module110ofFIG. 1in accordance with another embodiment of the disclosure. Referring toFIG. 2, the infotainment/telematics hub105and the camera sensor module110are physically separate from each other and correspond to separate electronic control units (ECUs). The infotainment/telematics hub105includes an application processor system205and a deserializer (DES)210. The camera sensor module110includes a camera sub-system215and a serializer (SER)220. The camera sub-system215includes a camera sensor and an integrated on-sensor processing system. The DES210and SER220are physically wired or cabled to each other. Generally, the camera sensor module110sends camera sensor pixel data (e.g., HDR-combined stream) to the infotainment/telematics hub105, and the infotainment/telematics hub105sends control/set registers back to the camera sensor module110for calibration.

FIG. 3illustrates a system300depicting components of and interconnections between the infotainment/telematics hub105and the camera sensor module110ofFIG. 1in accordance with another embodiment of the disclosure. The system300is a more detailed perspective of the system200ofFIG. 2.

Referring toFIG. 3, the camera sensor module110generates an HDR-combined stream which is sent to the infotainment/telematics hub105and delivered to an image signal processor (ISP)230. The ISP230performs color processing on the HDR-combined stream to produce a color-processed HDR-combined stream for downstream processing (e.g., neural network processing, presentation to a human, and/or computer vision processing, etc.). The color processing performed by the ISP230may include conversion of Bayer RGB to RGB (i.e. demosaicing), bad pixel correction, noise filtering, tone mapping, or any combination thereof. Statistics235are extracted from the color-processed HDR-combined stream. However, these statistics are not exposure-specific statistics, as any exposure-level information is lost due to the merging of the exposures into the HDR-combined stream at the camera sensor module110. Hence, the statistics extracted by the statistics module235may be referred to as “combined stream” statistics. A central processing unit (CPU)240processes the combined stream statistics and generates settings adjustments for the camera sensor module110, such as automatic exposure (AE) control settings and/or automatic white-balance (AWB) control settings, which are sent to the camera sensor module110and/or ISP230for camera sensor calibration. WhileFIG. 3illustrates the statistics235as separate from the ISP230, the statistics235may be produced by the ISP230.

FIG. 4illustrates an example of the camera sub-system215of the camera sensor module110ofFIGS. 2-3in more detail. The camera sub-system215includes a camera sensor400and an on-sensor processing system430. The camera sensor400includes an image array405, an analog to digital converter (ADC)410and a black level correction module415. The image array405may consist of a two dimensional array of photo-diodes to capture “pixels”. The ADC410is used to convert the pixel analog value to a discrete digital valaue. Black level correction module415may be enabled to compensate for dark current or lens flare and ensure black is perceived as black. The on-sensor processing system430includes a defect pixel correction module435, a white balance digital gain module440, an HDR-combine module445, an optional companding module450and a multiplexer455. Defect pixel correction module435may be enabled to replace or substitute an anomalous pixel capture resulting from a sensor manufacturing defect. The white balance digital gain module440can be enabled to apply adjustment on pixel values. The HDR-combine module445enables an algorithm to take multiple exposures within a frame time and produce a single frame output for image processing. The on-sensor processing system430further includes hardware related to statistics collection, such as multiplexer460, a full image histogram module465, user configurable region of interest (ROI) histogram modules470and475, ,and a user configurable ROI Average Pixel module480, which outputs statistics485. Also shown inFIG. 4is a physical layer interface490, or PHY. In particular, the PHY490is a Mobile Industry Processor Interface (MIPI) PHY, such as a MIPI C-PHY or MIPI D-PHY. The camera sub-system215depicted inFIG. 4may be configured to output a20blinear HDR-combined stream along with single exposure specific statistics or combined stream statistics. It will be appreciated that the modules465-480are part of a color filter array specific to green pixels. In other embodiments, the modules465-480may be configured to collect data on other pixel colors, such as red blue, white, clear and so on.

FIG. 5illustrates an image capture system500in accordance with an embodiment of the disclosure. Referring toFIG. 5, the image capture system500includes a camera sensor module505and a vision processing system550. The camera sensor module505includes a camera sub-system510containing a camera sensor515and an on-sensor processing system520, and a communications interface525. One or more optional camera sensor modules530and535may also be included as part of the image capture system500(e.g., for an automotive deployment as shown inFIG. 1). Each of the optional camera sensor modules530-535may be configured similarly to the camera sensor module505.

Referring toFIG. 5, the vision processing system550includes a communications interface555, a memory560, a processor565(e.g., a CPU), an ISP sub-system570. The ISP sub-system570includes a plurality of ISPs570_1. . .570_N, whereby N is greater than or equal to 2. The various components of the vision processing system550communicate with each other over a bus575.

In an example, the image capture system500ofFIG. 5may be deployed with respect to a vehicle as inFIG. 1, whereby the camera sensor module505and the vision processing system550are physically separate from each other. However, it is also possible for the image capture system500ofFIG. 5to be self-contained in a single apparatus, such as a phone or tablet computer. Accordingly, the various embodiments of the disclosure are not limited to automotive deployments.

FIG. 6illustrates an HDR-combining process in accordance with an embodiment of the disclosure. In an example, the process ofFIG. 6is performed by the camera sensor module505of the image capture system500ofFIG. 5.

Referring toFIG. 6, at block600, the camera sensor515of the camera sensor module505captures a plurality of exposures (e.g., a set of 3 or 4 exposures captured within a threshold period of time of each other). At block605, the on-sensor processing system520of the camera sensor module505performs HDR-combining on the plurality of exposures to produce an HDR-combined stream (e.g., at least 120 dB). At block610, the communications interface525of the camera sensor module505communicates, over a communications link to the vision processing system550, the (i) HDR-combined stream and (ii) the plurality of exposures. As will be appreciated, block610is distinct from an implementation whereby the HDR-combined stream is transferred to a vision processing system (with or without exposure-specific statistics) due to the further inclusion of the plurality of exposures being transferred as well. In an example, the communications link between the camera sensor module505and the vision processing system550can correspond to physical cabling (e.g., a MIPI PHY). In another example, the communications link between the camera sensor module505and the vision processing system550can correspond to an internal bus (e.g., for a phone or tablet computer implementation).

FIG. 7illustrates an exposure-specific statistic gathering process in accordance with an embodiment of the disclosure. In an example, the process ofFIG. 7is performed by the vision processing system550of the image capture system500ofFIG. 5.

Referring toFIG. 7, at block700, the communications interface555of the vision processing system550receives, from the camera sensor module505over a communications link, (i) a plurality of exposures captured by the camera sensor module505, and (ii) an HDR-combined stream of the plurality of exposures based on HDR-combining performed at the camera sensor module505. For example, block700may result from block610ofFIG. 6.

Referring toFIG. 7, at block705, a first ISP (e.g., ISP570_1) of the ISP sub-system570performs color processing on the HDR-combined stream to produce a color-processed HDR-combined stream for downstream processing (e.g., neural network processing, presentation to a human, and/or computer vision processing, etc.). The color processing performed by the ISP570_1may include conversion of Bayer RGB to RGB (i.e. demosaicing), bad pixel correction, noise filtering, tone mapping, and so on.

Referring toFIG. 7, at block710, at least one other ISP (e.g., one or more of ISPs570_2. . .570_N) performs color processing to collect exposure-specific statistics on at least one of the plurality of exposures for determining camera calibration data for the camera sensor module. As will be appreciated, the statistics collected at block710are exposure-specific statistics and not combined stream statistics. Of course, combined stream statistics can also be collected based on the color-processed HDR-combined stream produced by the first ISP at block705.

FIG. 8illustrates an example implementation of the processes ofFIGS. 6-7by the image capture system500ofFIG. 5in accordance with an embodiment of the disclosure.

Referring toFIG. 8, at block800(e.g., as in block600ofFIG. 6), the camera sensor515of the camera sensor module505captures a plurality of exposures (e.g., a set of 3 or 4 exposures captured within a threshold period of time of each other). At block805, the camera sensor515sends the captured exposures to the on-sensor processing system520. At block810(e.g., as in block605ofFIG. 6), the on-sensor processing system520of the camera sensor module505performs HDR-combining on the plurality of exposures to produce an HDR-combined stream (e.g., at least 120 dB). At block815(e.g., as in block610ofFIG. 6and block700ofFIG. 7), the camera sensor module505communicates (e.g., via the communications interface525), over a communications link to the vision processing system550, the (i) HDR-combined stream and (ii) the plurality of exposures. In particular, the HDR-combined stream is delivered to ISP570_1, while the exposures are delivered to ISPs570_2. . .570_N (e.g., whereby N may be equal to 2 or greater than 2, as will be described below in more detail).

Referring toFIG. 8, at block820(e.g., as in block705ofFIG. 7), ISP570_1performs color processing on the HDR-combined stream to produce a color-processed HDR-combined stream. At block825, ISP570_1forwards the color-processed HD-combined stream for downstream processing (e.g., neural network processing, presentation to a human, and/or computer vision processing, etc.).

Referring toFIG. 8, at block830(e.g., as in block710ofFIG. 7), one or more of ISPs570_2. . .570_N) performs color processing to collect statistics on at least one of the plurality of exposures for determining camera calibration data for the camera. At block835, the vision processing system550reports the exposure-specific statistics collected at block830back to the camera sensor module505, which then calibrates the camera sensor515based on the exposure-specific statistics at block840. As an alternative to block835, the vision processing system550may instead compute and send the camera sensor calibration parameters (e.g., AE and/or AWB control settings). So, the component that converts the exposure-specific statistics into camera sensor calibration parameters may be flexibly implemented.

FIG. 9illustrates a dual-ISP configuration900of a portion of the image capture system500ofFIG. 5in accordance with another embodiment of the disclosure. In particular,FIG. 9illustrates an example whereby a single ISP is allocated to collect exposure-specific statistics for each individual exposure associated with an HDR-combined stream.

Referring toFIG. 9, the camera sensor module505captures exposures 1-3, merges exposures 1-3 into an HDR-combined stream, and then sends the HDR-combined stream and exposures 1-3 to the vision processing system550. In particular, the HDR-combined stream is delivered to ISP570_1, while exposures 1-3 are delivered to ISP570_2. ISP570_2performs color-processing on exposures 1-3 to produce statistics905(i.e., exposure-specific statistics), which are passed to the processor565. The processor565may convert the exposure-specific statistics into camera sensor calibration parameters (e.g., AWB and/or AE control settings) as calibration feedback to the camera sensor module505, or may simply forward the exposure-specific statistics back to the camera sensor module505. While not shown expressly inFIG. 9, other statistics, such as combined stream statistics (e.g., similar to statistics235), may be produced by ISP570_1as well.

FIG. 10illustrates a quad-ISP configuration1000of a portion of the image capture system500ofFIG. 5in accordance with another embodiment of the disclosure. In particular,FIG. 10illustrates an example whereby a dedicated ISP is allocated to collect exposure-specific statistics for each individual exposure associated with an HDR-combined stream.

Referring toFIG. 10, the camera sensor module505captures exposures 1-3, merges exposures 1-3 into an HDR-combined stream, and then sends the HDR-combined stream and exposures 1-3 to the vision processing system550. In particular, the HDR-combined stream is delivered to ISP570_1, while exposures 1-3 are delivered to ISP570_2, ISP570_3and ISP570_4, respectively. ISPs570_2,570_3and570_4perform color-processing on exposures 1-3, respectively, to produce statistics1005, (i.e., exposure-specific statistics), which are then passed to the processor565. The processor565may convert the exposure-specific statistics into camera sensor calibration parameters (e.g., AWB and/or AE control settings) as calibration feedback to the camera sensor module505, or may simply forward the exposure-specific statistics back to the camera sensor module505. While not shown expressly inFIG. 10, other statistics, such as combined stream statistics (e.g., similar to statistics235), may be produced by ISP570_1as well.

Referring toFIG. 10, in an embodiment, implementing a 1:1 ratio between ISPs and exposures with respect to color processing for exposure-specific statistic collection may result in little to no memory impact (e.g.,3ISPs collect statistics which are output to a DDR memory portion of the memory560, whereas the underlying processed pixel data need not be stored in the DDR memory portion). In other words, in this embodiment, the exposures can be processed to produce relatively small-sized statistic data without requiring the larger exposures to be retained in the DDR memory portion (e.g., by contrast, if real-time processing of the exposures were not possible, the exposures would need to be saved off to the DDR memory portion until processing resources could be allocated for color processing of the exposures for statistics collection).

In a further embodiment, the mapping of ISPs to combined stream color processing and exposure-specific color processing need not be static. Rather, in one embodiment, the ISP mappings may be dynamically allocated for exposure-specific statistic collection as available. In particular, the color processing for the HDR-combined streams may be prioritized over the exposure-specific color processing for statistics collection. So, in the embodiment ofFIG. 10as an example, if two ISPs are required for color processing of HDR-combined streams (e.g., from multiple camera sensor modules, etc.), then only two ISPs may be allocated for exposure-specific statistic collection. Hence, the ISP mappings may change dynamically as the processing load changes. In a high-processing load environment, this may cause some exposures to become expired (e.g., too old to be reliably used for camera sensor calibration), and such exposures can be discarded without statistic collection. In an example, if there are insufficient ISP resources for collecting exposure-specific statistics on all exposures (e.g., a processing resource restriction caused by a high processing requirement for HDR-combined stream processing), the vision processing system550may implement an intelligent sampling algorithm so as to distribute the exposure ‘skips’ (e.g., spread the exposure skips evenly between exposures 1-3 over time, etc.).

FIG. 11illustrates an example configuration1100of the camera sub-system510of the camera sensor module505ofFIG. 5in accordance with an embodiment of the disclosure. The camera sub-system510is arranged similarly to the camera sub-system215depicted inFIG. 4in certain respects. However, the various hardware related to exposure-specific statistic collection which is integrated into the camera sub-system215ofFIG. 4is omitted inFIG. 11. In particular, the multiplexer460, full image histogram module465, a user configurable ROI histogram module470and475, and user configurable ROI Average Pixel module480are omitted, and the exposure-specific statistics485are not output by the camera sub-system510. Once again, this is because the vision processing system550(and not the camera sensor module505) is responsible for collecting the exposure-specific statistics in certain embodiments. Also, the multiplexer455ofFIG. 4is also omitted inFIG. 11, because the exposures and HDR-combined stream are both being sent to the vision processing system550, in contrast toFIG. 4.

As noted above, the camera sub-system215depicted inFIG. 4may be configured to output a20blinear HDR-combined stream along with one-exposure combined stream statistics. By contrast, the camera sub-system510arranged in accordance with the example configuration1100ofFIG. 11can output a20blinear HDR-combined stream as well as three (or four) exposures via a single PHY.

FIG. 12illustrates an example configuration1200of the camera sub-system510of the camera sensor module505ofFIG. 5in accordance with another embodiment of the disclosure. In particular, the camera sub-system configurations1100ofFIG. 11 and 1200ofFIG. 12are identical except for the inclusion of a second PHY1205(e.g., a MIPI PHY such as a MIPI C-PHY or MIPI D-PHY).

As noted above, the camera sub-system215depicted inFIG. 4may be configured to output a20blinear HDR-combined stream along with one-exposure combined stream statistics. By contrast, the camera sub-system510arranged in accordance with the example configuration1100ofFIG. 11can output a20blinear HDR-combined stream as well as three (or four) exposures via multiple PHYs. For example, larger resolution exposures and/or HDR-combined streams may necessitate the second PHY1205in certain implementations.

While various embodiments are described above with respect to automotive camera deployments, it will be appreciated that such embodiments are described for example purposes only, and the embodiments of the disclosure are not limited to automotive camera deployments. For example, the exemplary image capture systems described herein can be implemented in a phone or tablet computer in other embodiments of the disclosure.