Patent ID: 12207002

DETAILED DESCRIPTION

In some examples, different applications have different demosaicing requirements. For example, human vision prioritizes color processed to compensate for perceived deficiencies in image capture and display capabilities, or to smooth brighter and darker regions of a subject image. This can result in displayed images that are more visually appealing or perceived as having greater fidelity to source material, at the cost of some information contained in raw captured image data. Machine vision, such as for object detection and other advanced driver assistance systems (ADAS), prioritizes edge information, preferring minimal loss of information contained in raw image data. Different applications (also referred to herein as modalities) can also have different color format requirements, such as Luma, YUV 4:2:0, YUV 4:2:2, hue-saturation-lightness (HSL), or hue-saturation-value (HSV).

In some examples, a video imaging subsystem (VISS)118(seeFIG.2) includes circuitry to demosaic raw image data and to perform color processing to select one or more output color formats, as well as circuitry for noise filtering and tone mapping. A single VISS118can be used to process image data for human vision, and to process the same image data for machine vision. For example, a VISS118can include circuitry for color processing that receives input, via a synchronizer circuit212(also referred to as a bypass selection circuit), that bypasses some or all processing blocks between demosaicing and color processing. The synchronizer circuit212is used so that a pixel in a line processed for human vision is output by the VISS118at the same time as a corresponding pixel in a line processed for machine vision. This enables a single input image data stream to be processed to produce two horizontally synchronized output streams, while avoiding an increase (or enabling a reduction) of horizontal blanking.

An image is made up of multiple lines. A line contains a series of pixels. In some examples, the number of pixels in a line is the width of the image, and the number of lines in the image is the height of the image. Horizontal blanking refers to an idle or blanking time, such as a number of cycles of a system clock, between lines read into an image processing pipeline. The image processing pipeline includes idle portions corresponding to horizontal blanking periods. Accordingly, reduced horizontal blanking corresponds to improved performance efficiency of the VISS118, and of an image processing system100incorporating the VISS118. Use of the synchronizer circuit212and the circuitry for color processing also avoids duplicating other functional blocks of the pipeline while enabling simultaneous processing that uses a single set of camera data to meet requirements for multiple applications.

FIG.1is a functional block diagram of an example image processing system100. The image processing system100may include an image capture device102, a video preprocessing accelerator (VPAC)104, a shared memory106, an additional processing and memory block108, and a display110. In an example, the shared memory106is a double data rate (DDR) random access memory (RAM). The image capture device102includes a sensor112and a CFA114. The VPAC104includes a hardware task scheduler (HTS)116, a set of processing circuits (e.g., a VISS118, a lens distortion correction (LDC) block120, a noise filter (NF) block122, a multi-scalar (MSC) block124, etc.), a direct memory access (DMA) block126, and a buffer128. In some examples, the buffer128is a circular buffer. In some examples, some or all of the VPAC104, the shared memory106, and/or the additional processing and memory block108is included in an integrated circuit (IC), or in multiple ICs on a chip, or in multiple ICs mounted on a printed circuit board (PCB). In the illustrated example, the VPAC104and the shared memory106are included in an IC130.

In some examples, the LDC block120remaps pixels from a distorted input space to an undistorted input space; and applies perspective transform and/or homography operations. In some examples, the NF block122is configured to perform bilateral noise filtering; perform two-dimensional bilateral filtering; and/or use programmable static weights. In some examples, the MSC block124can simultaneously scale outputs from one or more input planes; perform Pyramid or inter-octave scale generation; and support one or more scaling ratios. In some examples, the additional processing and memory block108includes resources supporting one or more of machine vision, robotics, video surveillance, or ADAS systems.

An output of the image capture device102is connected to a data input of the IC130. Image data received by the IC130can be provided to, for example, the shared memory106or the VPAC104. The VPAC104is connected to communicate with the shared memory106. An output of the IC130is connected to an input of the additional processing and memory block108. The additional processing and memory block108is connected to provide processed image data to the display110for display to a user.

In some examples, the additional processing and memory block108includes image analysis circuitry for object detection or image analysis. In some examples, the additional processing and memory block108is connected to circuitry for vehicle control, industrial equipment control, or safety functionality (not shown). In some examples, the additional processing and memory block108includes circuitry for image processing to prepare for display.

The HTS116is connected to schedule tasks to be performed by, and to receive acknowledgments from, the VISS118, the LDC120, the NF122, the MSC124, and the DMA126. The buffer128is connected to store data and/or instructions received from, and read data and/or instructions out to, the VISS118, the LDC120, the NF122, the MSC124, and the DMA126. Thus, the buffer128can be used to transfer image data between the VISS118, the LDC120, the NF122, the MSC124, and the DMA126. Connections to the HTS116and/or to the buffer128can use, for example, a bus.

In some examples, the VISS118is a line-based image processing component with multiple modes of operation. In an example mode, the VISS118receives image data from the image capture device102, processes the image data, and passes partially processed image data to downstream functional blocks via the buffer128for further processing. In another example mode, the DMA block126reads image data that has been written to the shared memory106, and passes the image data to functional blocks including the VISS118, via the buffer128, for processing. The DMA block126retrieves processed image data from the buffer126and writes the processed image data to the shared memory106. The VISS118is further described with respect toFIGS.2and3.

FIG.2is a functional block diagram of an example VISS118for use in the VPAC104ofFIG.1. The VISS118has a data input (Input), a clock input (Clock), a first output (Output 1), and a second output (Output 2), and includes a set of processing circuits (e.g., a raw front end (FE)202, a noise filter (NSF)204, a global and local brightness contrast enhancement (GLBCE) block206, a first flex color processing (FCP 1) block208, a second flex color processing (FCP 2) block210, etc.), and a synchronizer circuit212. The clock input receives a clock signal from a system clock (not shown). The raw FE202, the NSF204, the GLBCE block206, the FCP 1 block208, and the FCP 2 block210are clocked by the clock signal in streaming fashion, so that circuits that process a pixel or block in the pipeline in one clock cycle process a next pixel or block in the pipeline in a sequentially next clock cycle.

An input of the raw FE202is connected to the data input to receive the image data. An output of the raw FE202is connected to an input of the NSF204and a first input of the synchronizer circuit212. An output of the NSF204is connected to an input of the GLBCE block206, and to a second input of the synchronizer circuit212. An output of the GLBCE block206is connected to an input of the FCP 1 block208and a third input of the synchronizer circuit212. An output of the FCP 1 block208is connected to the first output (Output 1) of the VISS118. An output of the synchronizer circuit212is connected to an input of the FCP 2 block210. An output of the FCP 2 block210is connected to the second output (Output 2) of the VISS118.

In some examples, the raw FE202implements wide dynamic range (WDR) merge, defect pixel correction (DPC), lens shading correction (LSC), decompounding, 3A (auto-focus, auto-exposure, and auto-white balance) statistics, and/or white balance. The NSF204implements a Bayer domain spatial noise filter. The GLBCE block206performs adaptive local tone mapping. The FCP 1 and FCP 2 blocks208and210perform various combinations of demosaicing, color correction, color space conversion, and/or gamma conversion. In an example, the FCP 1 block208and Output 1 provide output corresponding to human vision, and the FCP 2 block210and Output 2 provide output corresponding to machine vision. In some examples, FCP 1208and FCP 2210use a single set of image sensor data to provide outputs corresponding to different applications.

In some examples, each of Output 1 and Output 2 includes multiple output lines. Accordingly, the FCP 1 and FCP 2 blocks208and210each include multiple outputs, and simultaneously output processed pixels in multiple different color formats, so that processed versions of individual pixels or blocks are output by both the FCP 1 and FCP 2 blocks208and210in same clock cycles. The FCP 1 block208can output pixels in color formats that are the same as or different from color formats in which the FCP 2 block210outputs pixels. Output of the FCP 1 and FCP 2 blocks208and210in different color formats is further described with respect toFIG.3. In some examples, the FCP 1 block208and the FCP 2 block210each provides to Output 1 or Output 2, respectively, five color channels of data. In some examples, the FCP 1 block208and/or the FCP 2 block210provide luma 8-bit, chroma 8-bit, luma 12-bit, chroma 12-bit, or saturation data.

Different applications can have different input requirements. For example, as described above, processing for machine vision prefers minimal edge information loss with respect to raw captured image data. This means that, in some applications, it is detrimental for input to the FCP 2 block210to be processed by the NSF204or the GLBCE block206. Accordingly, input to the FCP 2 block210can bypass (or skip) the GLBCE block206, or the NSF204and the GLBCE block206, via the synchronizer circuit212.

FIG.3is a timing diagram300showing example timings of signals at Input302, Output 1312, and Output 2314. Input302corresponds to Input, i.e. the data input of the VISS118as shown inFIG.2. Output 1312and Output 2314respectively correspond to the first and second outputs of the VISS118, i.e. Output 1 and Output 2, as shown inFIG.2.

The periods of Input302with eye shapes correspond to the VISS118receiving image data corresponding to a line304of an image (valid line data). Input302being low (no eye shape) corresponds to a horizontal blanking period306between image data reception periods. A horizontal blanking period306lasts for a time THBP. FCP 1 input308is the input of the FCP 1 block208, and synchronizer input310is the input of the synchronizer circuit212(which provides the input of the FCP 2 block210). FCP 1 input308or synchronizer input310having an eye shape corresponds to the respective functional block receiving a partially processed line of image data304. Output 1312or Output 2314having an eye shape corresponds to output of a processed line of image data304by the FCP 1 block208or the FCP 2 block210, respectively.

A vertical delay316is a time (in an example, measured in a number of lines of image data) from the beginning of receiving a line of image data304at Input302to a beginning of outputting the line, such as at Output 1312. In some examples, vertical delay varies in response to types of processing included in an image processing pipeline such as the VISS118. A horizontal delay318is a time (in an example, measured in clock cycles) to the beginning of outputting a processed line of image data304from the FCP 1 block208or the FCP 2 block210from an immediately previous beginning of receiving a line of image data304at Input302. The received line of image data304and the processed line of image data304used to determine horizontal delay318can be the same line of image data304or different lines of image data304(for example, a third received line of image data304and a seventh received line of image data304). Accordingly, a vertical delay316corresponding to Output 1312can be different from a vertical delay316corresponding to Output 2314, while at the same time, the horizontal delays318corresponding to Output 1312and Output 2314are the same or nearly the same. Herein, horizontal delays corresponding to Output 1312and Output 2314being nearly the same means that an absolute value of a difference between the horizontal delays318corresponding to Output 1312and Output 2314is less than or equal to THBP.

In some examples, the VISS118writes its outputs to the buffer128, and the LDC120, NF122, and MSC124read from, and write their respective outputs to, the buffer128. In some examples, functional blocks require that a preceding pipeline result for all color channels corresponding to a line be written to the buffer128before the functional block will read from the buffer128to continue image processing. With respect to the VISS118, this means that a downstream functional block will wait for both a processed first line of image data304to be provided at Output 1312and a processed second line of image data304to be provided at Output 2314before the downstream function block reads the combined outputs and proceeds with image processing. In some examples, the processed first line of image data304corresponds to a different line of image data304from the processed second line of image data304. In other words, some difference between vertical delays316corresponding to Output 1312and Output 2314is acceptable. In some examples, color channel completeness requirements are set earlier or later with respect to writing to or reading from the buffer128, and similar conditions may apply.

A horizontal delay318corresponding to Output 1312is a first horizontal delay (delayH1). A horizontal delay318corresponding to Output 2314is a second horizontal delay (delayH2). In some examples, if |delayH1−delayH2|>THBP, then downstream functional blocks are not able to read combined outputs to proceed with image processing as often as Input302receives new lines. Processing is delayed by a number of cycles equal to the number of cycles by which |delayH1−delayH2|>THBP. This may lead to, for example, a reduced input rate, or use of a large buffer with a reduced processing rate. This issue can be avoided by horizontally synchronizing Output 1312with Output 2314.

Output 1312is horizontally synchronized with Output 2314if the horizontal delay318of Output 1312is the same or nearly the same as the horizontal delay318of Output 2314. In some examples, this means that where lines of image data304are a number L pixels long, an Nth pixel of the first line of image data304is output at the same time as, or within THBPcycles of, an Nth pixel of the second line of image data304, where N is a number from1to L.

The synchronizer circuit212outputs received pixels with a selected delay, horizontally synchronizing Output 1312with Output 2314, avoiding the described horizontal delay mismatch. For example, the synchronizer circuit212may delay providing pixels to the FCP 2 block210by a number of cycles corresponding to a horizontal delay added by the processing blocks (such as206, or204and206) that are selectively bypassed in the path to Output 2. In an example, the NSF204and the GLBCE block206each add a horizontal delay of 25 cycles. If no processing blocks are bypassed, the synchronizer circuit212delays a pixel by zero cycles before providing the pixel to the FCP 2 block210. If the GLBCE block206is bypassed, the synchronizer circuit212delays a pixel by 25 cycles before providing the pixel to the FCP 2 block210. If both the NSF204and the GLBCE block206are bypassed, the synchronizer circuit212delays a pixel by 50 cycles before providing the pixel to the FCP 2 block210.

In some examples, different functional blocks introduce different amounts of horizontal delay, or different amounts of horizontal delay from those described herein. In some examples, the FCP 2 block210is identical to the FCP 1 block208. This facilitates making the horizontal delays314introduced by the FCP 1 block208and the FCP 2 block210the same or nearly the same, which simplifies determining the amount of delay for the synchronizer circuit212to add. For example, the example given above is provided with the assumption that the horizontal delays314introduced by the FCP 1 block208and the FCP 2 block210are the same or nearly the same.

In some examples, image processing parameters are highly sensitive to image content and are frequently updated by software. Time cost to program software to handle frequent, image-dependent updating can be significant. Accordingly, making the FCP 2 block210identical to the FCP 1 block208can reduce time cost to configure the VISS118, by saving configuration time for common functions.

In other examples, the synchronizer circuit212may apply a delay to ensure that Output 1 produces a first output for a given pixel or block at substantially the same time that Output 2 produces a second output for the pixel or block despite differences in delay between FCP 1 block208and FCP 2 block210.

FIG.4is a functional block diagram of a VISS system400including the example VISS118ofFIG.2with an example implementation of the synchronizer circuit212, and an output multiplexer (MUX)402. The synchronizer circuit212includes a pipeline MUX404, a pipeline balancing memory406, and a control circuit408. First, second, and third inputs of the pipeline MUX404are respectively connected to the first, second, and third inputs of the synchronizer circuit212. This means that the first, second, and third inputs of the pipeline MUX404are respectively connected to the outputs of the raw FE202, the NSF204, and the GLBCE block206. An output of the pipeline MUX404is connected to a data input of the pipeline balancing memory406. An output of the pipeline balancing memory406is connected to the output of the synchronizer circuit212, and accordingly, to the input of the FCP 2 block210. The first output of the VISS118(Output 1) is connected to the first input of the output MUX402, and the second output of the VISS118(Output 2) is connected to the second input of the output MUX402. The output MUX402also includes a control input that receives a color format selection signal. In some examples, the output MUX402is implemented using multiple, cascading multiplexers.

The control circuit408controls the pipeline MUX404to select which input of the synchronizer circuit212to pass to the pipeline balancing memory406. This selects whether a processing pipeline that includes the FCP 2 block210will bypass the GLBCE block206, or the NSF204and the GLBCE block206.

The control circuit408also determines a horizontal delay (such as a number of clock cycles) introduced by respective skippable functional blocks. The control circuit408can determine horizontal delays using, for example, a table or other memory, or using logic that can measure horizontal delay. The control circuit408controls the pipeline balancing memory406to store pixels received via the pipeline multiplexer404for the determined horizontal delay, and then to output the delayed pixels (processed pixel data) to the FCP 2 block210. This enables a pixel received by the VISS118to complete VISS118processing and be output by the FCP 1 block208at the same time (or nearly the same time) that a corresponding pixel from a same line or different line of image data completes VISS118processing and is output by the FCP 2 block210, regardless of which internal pipeline blocks of the VISS118are bypassed. In other words, a pixel from a first line of image data is provided by Output 1 in horizontal synchrony with a pixel from a second line of image data being provided by Output 2, where the first and second lines of image data may be the same or different.

In some examples, the pipeline balancing memory406is a first in first out (FIFO) memory. In an example, pixels output by the raw FE202have 16 bits, and it can take up to 256 cycles for a pixel output by the raw FE202to be received by the input of the FCP 1 block208. In this example, the pipeline balancing memory406is sized to store up to 256×16=4096 bits. In some examples, the pipeline balancing memory406has a different size.

In some examples, Output 1 and Output 2 can each simultaneously provide multiple different color format versions of processed pixels. For example, Output 1 and Output 2 can each output processed pixels in one or more color spaces such as RGB, YUV, or HSV. The output MUX402outputs signals in selected color formats, processed and provided by the FCP 1 or FCP 2 block208or210, in response to the color format selection signal. For example, the output MUX402can output YUV8 (Y 8-bit and UV 8-bit from the FCP 1 block208for human vision processing, and Y12UV8 (Y 12-bit and UV 8-bit) from the FCP 2 block210for machine vision processing. In some examples, a machine vision processing pipeline path uses pixels processed to retain a high dynamic range of pixel intensity, in a Luma or Y color format, for ADAS applications. In some examples, Output 1 includes five color channels, Output 2 includes five color channels, and the output MUX402selects between these ten color channels to provide five color channels as an output of the VISS system400. In some examples, any combination of Output 1 and Output 2 color channels can be selected to produce the output of the VISS system400, such as three color channels from Output 1 and two color channels from Output 2, or one color channel from Output 1 and four color channels from Output 2.

FIG.5is an example process500for operating a video imaging subsystem. In step502, a first image processing circuit receives a first set of image data associated with a first modality and a second modality. In an example, the first modality is human vision processing and the second modality is machine vision processing. In an example, the first imaging processing circuit is the raw FE202. In step504, the first image processing circuit performs a first operation associated with the first modality and the second modality on the first set of image data to produce a second set of image data. In step506, the first imaging processing circuit provides the second set of data to a bypass selection circuit. In step508, a second image processing circuit performs a second operation associated with the first modality on the second set of image data to produce a third set of image data. In an example, the second image processing circuit is the NSF204, or the NSF204and the GLBCE206. In step510, the second imaging processing circuit provides the third set of data to the bypass selection circuit. In step512, a third image processing circuit performs a third operation associated with the first modality on the third set of image data to produce a fourth set of image data. In step514, the bypass selection circuit selects between the second set of image data and the third set of image data to produce a fifth set of image data associated with the second modality. In step516, the bypass selection circuit provides the fifth set of image data to a fourth image processing circuit with a delay responsive to whether the bypass selection circuit selected the second set of image data or the third set of image data. In step518, the fourth image processing circuit performs a fourth operation associated with the second modality on the fifth set of image data to produce a sixth set of image data.

In some examples, a synchronizer circuit212and FCP 2210as described herein enables a single image data input stream to produce dual synchronized, processed output streams. Further, for producing dual/different outputs, some or all of input bandwidth, device area usage, part cost, and power usage can be reduced. In some examples, a VISS118as described can be included in an image processing IC130with little or no rearrangement of circuits in other functional blocks.

FIG.6is an example process600for operating a video imaging subsystem. In step602, an image capture device captures a set of image data and provides the set of image data to a memory of an image processing system. In step604, the image processing system reads the set of image data from its memory once, and provides the set of image data to an image processing pipeline of the image processing system. (In some examples, multiple reads can be used. In some examples, reading from memory once can correspond to multiple reads being performed, so that respective portions of memory are read once. In some examples, some or all respective portions of memory are read, or are each read, multiple times.)

In step606, the image processing pipeline receives the set of image data for processing according to a first application and a second application. In an example, the first application is human vision processing and the second application is machine vision processing. In step608, the image processing pipeline processes the set of image data using functions common and applicable to both the first application and the second application to produce a first-processed set of image data.

In step610, the image processing pipeline processes the first-processed set of image data using functions specific to the first application to produce a second-processed set of image data. In step612, the image processing pipeline processes the first-processed set of image data using functions specific to the second application to produce a third-processed set of image data. In step614, the image processing pipeline outputs the second-processed set of image data and the third-processed set of image data so the horizontal delays with which the second-processed and third-processed sets of data are output are the same or nearly the same. Accordingly, the image processing system is able to read image data once to provide dual, differently-processed outputs while avoiding or minimizing an increase in image processing pipeline delay.

Modifications are possible in the described examples, and other examples are possible within the scope of the claims.

In some examples, the FCP 1 and FCP 2 blocks208and210output processed pixels in other color formats than those described above.

In some examples, circuitry other than a multiplexer can be used to select an output from among multiple inputs.

In some examples, a multiplexer or other circuitry providing one or more selected inputs as output(s) is referred to as switching circuitry or data selection circuitry.

In some examples, the pipeline multiplexer404is referred to as bypass selection circuitry.

In some examples, switching circuitry, data selection circuitry, or bypass selection circuitry includes the pipeline multiplexer404and the control circuit408.

In some examples, a demosaicing and color processing pipeline includes different functionality, or functionality organized in a different order or in different pipeline blocks, than described above with respect to the VISS118.

In some examples, different functional blocks (or portions of functional blocks) are configured to be selectable to be bypassed than those described herein with respect to the VISS118.

In some examples, an image processor includes different functionality, or functionality organized in different functional blocks, than described above with respect to the VPAC104or the IC130.

In some examples, FCP 2210is not identical to and/or performs different functions than FCP 1208.

In some examples, FCP 2210is clock gated.

In some examples, a VISS118includes a second synchronizer circuit, with inputs connected to outputs of the Raw FE202, the NSF204, and the GLBCE206, and an output connected to the input of the FCP 1208. In some examples, the second synchronizer circuit includes a second pipeline MUX, a second pipeline balancing memory, and a second control circuit. Inputs of the second pipeline MUX are connected to inputs of the second synchronizer circuit, and an output of the second pipeline MUX is connected to an input of the second pipeline balancing memory. Outputs of the second control circuit are connected to control inputs of the second pipeline MUX and the second pipeline balancing memory. An output of the second pipeline balancing memory is connected to an output of the second synchronizer circuit. In some examples, a second synchronization circuit allows different paths in a VISS118pipeline to bypass different processing blocks of the pipeline.

In this description, the term “and/or” (when used in a form such as A, B and/or C) refers to any combination or subset of A, B, C, such as: (a) A alone; (b) B alone; (c) C alone; (d) A with B; (e) A with C; (f) B with C; and (g) A with B and with C. Also, as used herein, the phrase “at least one of A or B” (or “at least one of A and B”) refers to implementations including any of: (a) at least one A; (b) at least one B; and (c) at least one A and at least one B.

A device that is “configured to” perform a task or function may be configured (for example, programmed and/or hardwired) at a time of manufacturing by a manufacturer to perform the function and/or may be configurable (or re-configurable) by a user after manufacturing to perform the function and/or other additional or alternative functions. The configuring may be through firmware and/or software programming of the device, through a construction and/or layout of hardware components and interconnections of the device, or a combination thereof.

A circuit or device that is described herein as including certain components may instead be adapted to be coupled to those components to form the described circuitry or device. For example, a structure described as including one or more semiconductor elements (such as transistors), one or more passive elements (such as resistors, capacitors, and/or inductors), and/or one or more sources (such as voltage and/or current sources) may instead include only the semiconductor elements within a single physical device (for example, a semiconductor die and/or integrated circuit (IC) package) and may be adapted to be coupled to at least some of the passive elements and/or the sources to form the described structure either at a time of manufacture or after a time of manufacture, for example, by an end-user and/or a third-party.

Circuits described herein are reconfigurable to include the replaced components to provide functionality at least partially similar to functionality available prior to the component replacement.

The term “couple” is used throughout the specification. The term may cover connections, communications, or signal paths that enable a functional relationship consistent with this description. For example, if device A provides a signal to control device B to perform an action, in a first example device A is coupled to device B, or in a second example device A is coupled to device B through intervening component C if intervening component C does not substantially alter the functional relationship between device A and device B such that device B is controlled by device A via the control signal provided by device A.

While certain elements of the described examples may be included in an IC and other elements are external to the IC, in other examples, additional or fewer features may be incorporated into the IC. In addition, some or all of the features illustrated as being external to the IC may be included in the IC and/or some features illustrated as being internal to the IC may be incorporated outside of the IC. As used herein, the term “IC” means one or more circuits that are: (i) incorporated in/over a semiconductor substrate; (ii) incorporated in a single semiconductor package; (iii) incorporated into the same module; and/or (iv) incorporated in/on the same PCB.

Unless otherwise stated, “about,” “approximately,” or “substantially” preceding a value means+/−10 percent of the stated value, or, if the value is zero, a reasonable range of values around zero.