Patent ID: 12228757

DETAILED DESCRIPTION

Various examples of the present technology are discussed in detail below. While specific implementations are discussed, it should be understood that this is done for illustration purposes only. A person skilled in the relevant art will recognize that other components and configurations may be used without parting from the spirit and scope of the present technology. In some instances, well-known structures and devices are shown in block diagram form in order to facilitate describing one or more aspects. Further, it is to be understood that functionality that is described as being carried out by certain system components may be performed by more or fewer components than shown.

The disclosed technologies address a need in the art for enhancing the low light sensitivity of image sensors while preserving the color accuracy of image data for image signal processing. In some aspects, different color filter arrays (CFAs) can be implemented for small and large photodiodes in an image sensor to increase the quantum efficiency (QE) of the image sensor and thereby enhancing the low light sensitivity of the image sensor, while preserving or improving color separation and fidelity.

For example, to achieve high dynamic range (HDR), an image sensor can implement a split-pixel or sub-pixel design where two physical photodiodes, a large and a small photodiode, are associated with each pixel readout circuitry. To enhance the low light sensitivity of the image sensor while preserving or improving color separation qualities, a specific CFA that can enhance low light sensitivity can be implemented for the large photodiodes and a different CFA that can preserve or enhance color separation capabilities can be implemented for the small photodiodes.

To illustrate, in some examples, an RCCB (Red, Clear, Clear, Blue or 25% Red, 50% Clear, and 25% Blue) CFA can be used for the large photodiodes and an RGGB (Red, Green, Green, Blue or 25% Red, 50% Green, and 25% Blue) CFA can be used for the small photodiodes. Here, the clear portion of the RCCB CFA can increase the low light sensitivity of the large photodiodes while the RGGB CFA can preserve or enhance the color separation qualities of the small photodiodes. In addition, the combination of large photodiodes with higher light sensitivity and the small photodiodes with the lower light sensitivity can enable high dynamic range.

These approaches can be implemented in a variety of image processing applications and use cases. In some illustrative cases, the different CFAs used for the small and large photodiodes in an image sensor as described herein can be implemented in autonomous vehicle (AV) applications to enhance the low light sensitivity of the image sensor while preserving or improving color separation and fidelity to improve the image sensors ability to detect and accurately recognize different traffic lights and other lights in an AV environment. For example, in such implementations, traffic lights with a high signal intensity can be captured by the small photodiodes with a first type of CFA which can allow for better or accurate color separation and accuracy. For low signal intensity areas or regions, the large photodiodes with the second type of CFA containing a clear filter can provide low light sensitivity. The combination can thus provide high sensitivity with high color separation and accuracy.

FIG.1illustrates an example autonomous vehicle environment100, in which the technologies herein can be implemented. The example autonomous vehicle environment100includes an autonomous vehicle102, a remote computing system150, and a ridesharing application172on a computing device170. The autonomous vehicle102, remote computing system150, computing device170(including ridesharing application172) can communicate with each other over one or more networks, such as a public network (e.g., a public cloud, the Internet, etc.), a private network (e.g., a local area network, a private cloud, a virtual private network, etc.), and/or a hybrid network (e.g., a multi-cloud or hybrid cloud network, etc.).

The autonomous vehicle102can navigate about roadways without a human driver based on sensor signals generated by sensors104on the autonomous vehicle102. The sensors104on the autonomous vehicle102can include one or more types of sensors and can be arranged about the autonomous vehicle102. For example, the sensors104can include, without limitation, one or more: inertial measuring units (IMUs), image sensors (e.g., visible light image sensors, infrared image sensors, video camera sensors, etc.), light emitting sensors (e.g., lasers, light detection and ranging sensors (LIDARs), etc.), global positioning system (GPS) devices, radars, sonars, accelerometers, gyroscopes, magnetometers, motion detection sensors, light detectors, audio sensors (e.g., microphones), seat occupancy sensors, ultrasonic sensors, position sensors, steering angle sensors, steering wheel rotation sensors, speedometers, proximity sensors, etc.

In some implementations, the sensors104can include one or more image sensors106A (e.g., camera sensors, visible light sensors, infrared light sensors, HDR image sensors, etc.), one or more radars sensors106B, and one or more LIDAR sensors106N. Other implementations can include any other number and types of sensors.

The autonomous vehicle102can include one or more display devices108for presenting information, such as maps, messages, and interfaces, to passengers in the autonomous vehicle102. The one or more display devices108can be mounted on one or more locations in the autonomous vehicle102. For example, the one or more display devices108can be mounted on one or more seats or headrests in the autonomous vehicle102, a dashboard in the autonomous vehicle102, one or more inner sides or door panels on the autonomous vehicle102, a roof of the autonomous vehicle102, and/or any other interior location of the autonomous vehicle102. The one or more display devices108can include, for example and without limitation, a screen, a television, a projecting device, and/or any other suitable display device for rendering graphical information.

Moreover, the autonomous vehicle102can include several mechanical systems that are used to effectuate motion of the autonomous vehicle102. For instance, the mechanical systems can include, but are not limited to, a vehicle propulsion system130, a braking system132, and a steering system134. The vehicle propulsion system130can include an electric motor, an internal combustion engine, or both. The braking system132can include an engine brake, brake pads, actuators, and/or any other suitable componentry configured to assist in decelerating the autonomous vehicle102. The steering system134includes suitable componentry configured to control the direction of movement of the autonomous vehicle102during navigation.

The autonomous vehicle102can include a safety system136. The safety system136can include lights and signal indicators, a parking brake, airbags, etc. The autonomous vehicle102can also include a cabin system138, which can include cabin temperature control systems, in-cabin entertainment systems, display devices, light-emitting devices, audio systems, etc.

The autonomous vehicle102can include an internal computing system110in communication with the sensors104, the display device(s)108, and the systems130,132,134,136, and138. The internal computing system110can include one or more processors and at least one memory for storing instructions executable by the one or more processors. The computer-executable instructions can make up one or more services for controlling the autonomous vehicle102, communicating with remote computing system150, receiving inputs from passengers or human co-pilots, logging metrics regarding data collected by the sensors104and human co-pilots, etc.

The internal computing system110can include a control service112configured to control operation of the vehicle propulsion system130, the braking system132, the steering system134, the safety system136, and the cabin system138. The control service112can receive sensor signals from the sensors104can communicate with other services of the internal computing system110to effectuate operation of the autonomous vehicle102. In some examples, control service112may carry out operations in concert with one or more other systems of autonomous vehicle102.

The internal computing system110can also include a constraint service114to facilitate safe propulsion of the autonomous vehicle102. The constraint service116includes instructions for activating a constraint based on a rule-based restriction upon operation of the autonomous vehicle102. For example, the constraint may be a restriction on navigation that is activated in accordance with protocols configured to avoid occupying the same space as other objects, abide by traffic laws, circumvent avoidance areas, etc. In some examples, the constraint service114can be part of the control service112.

The internal computing system110can also include a communication service116. The communication service116can include software and/or hardware elements for transmitting and receiving signals to and from the remote computing system150. The communication service116can be configured to transmit information wirelessly over a network, for example, through an antenna array or interface system that provides cellular (long-term evolution (LTE), 3rdGeneration (3G), 5thGeneration (5G), etc.) communication, Bluetooth communication, near-field communication, and/or any other suitable type of wireless communication.

In some examples, one or more services of the internal computing system110are configured to send and receive communications to remote computing system150for reporting data for training and evaluating machine learning algorithms, requesting assistance from remote computing system150or a human operator via remote computing system150, software service updates, ridesharing pickup and drop off instructions, etc.

The internal computing system110can also include a latency service118. The latency service118can utilize timestamps on communications to and from the remote computing system150to determine if a communication has been received from the remote computing system150in time to be useful. For example, when a service of the internal computing system110requests feedback from remote computing system150on a time-sensitive process, the latency service118can determine if a response was timely received from remote computing system150, as information can quickly become too stale to be actionable. When the latency service118determines that a response has not been received within a threshold period of time, the latency service118can enable other systems of autonomous vehicle102or a passenger to make decisions or provide needed feedback.

The internal computing system110can also include a user interface service120that can communicate with cabin system138to provide information or receive information to a human co-pilot or passenger. In some examples, a human co-pilot or passenger can be asked or requested to evaluate and override a constraint from constraint service114. In other examples, the human co-pilot or passenger may wish to provide an instruction to, or receive an instruction from, the autonomous vehicle102regarding destinations, requested routes, drop-off locations, wayfinding tasks, or other requested operations.

As described above, the remote computing system150can be configured to send and receive signals to and from the autonomous vehicle102. The signals can include, for example and without limitation, data reported for training and evaluating services such as machine learning services, data for requesting assistance from remote computing system150or a human operator, software service updates, rideshare pickup and drop off instructions, etc.

The remote computing system150can include an analysis service152configured to receive data from autonomous vehicle102and analyze the data to train or evaluate machine learning algorithms for operating the autonomous vehicle102. The analysis service152can also perform analysis pertaining to data associated with one or more errors or constraints reported by autonomous vehicle102.

The remote computing system150can also include a user interface service154configured to present metrics, video, images, sounds reported from the autonomous vehicle102to an operator of remote computing system150, maps, routes, navigation data, notifications, user data, vehicle data, software data, and/or any other content. User interface service154can receive, from an operator, input instructions for the autonomous vehicle102.

The remote computing system150can also include an instruction service156for sending instructions regarding the operation of the autonomous vehicle102. For example, in response to an output of the analysis service152or user interface service154, instructions service156can prepare instructions to one or more services of the autonomous vehicle102or a co-pilot or passenger of the autonomous vehicle102.

The remote computing system150can also include a rideshare service158configured to interact with ridesharing applications172operating on computing device170. Computing device170can include, for example and without limitation, a tablet computer, a laptop computer, a smartphone, a head-mounted display (HMD), a gaming system, a server, a smart device, a laptop computer, a smart wearable (e.g., smart glasses, smart watch, etc.), a mobile computing device, and/or any other computing device. In some cases, the computing device170can be a mobile computing device of a user or passenger/rider. Moreover, in some cases, the remote computing system150and the autonomous vehicle102can also communicate and interact with other computing devices hosting instances of the ridesharing application172and the delivery service174. For example, the remote computing system150and the autonomous vehicle102can also communicate and interact with other computing devices associated with one or more passengers.

The rideshare service158can receive requests from passenger ridesharing application172, such as user requests to be picked up or dropped off, and can dispatch autonomous vehicle102for a requested trip. The rideshare service158can also act as an intermediary between the ridesharing application172and the autonomous vehicle102. For example, rideshare service158can receive from a passenger instructions for the autonomous vehicle102, such as instructions to go around an obstacle, change routes, select a drop-off location and/or pick-up location, etc. The rideshare service158can provide such instructions to the autonomous vehicle102as requested.

The remote computing system150can also include a package service162configured to interact with the computing device170, the ridesharing application172and/or a delivery service174of the ridesharing application172. A user operating the ridesharing application172can interact with the delivery service174to specify information regarding a package to be delivered using the autonomous vehicle102. The specified information can include, for example and without limitation, package dimensions, a package weight, a destination address, delivery instructions (e.g., a delivery time, a delivery note, a delivery constraint, etc.), and so forth.

The package service162can interact with the delivery service174to provide a package identifier to the user for package labeling and tracking. Package delivery service174can also inform a user of where to bring their labeled package for drop off. In some examples, a user can request the autonomous vehicle102come to a specific location, such as the user's location, to pick up the package. While delivery service174has been shown as part of the ridesharing application172, it will be appreciated by those of ordinary skill in the art that delivery service174can be its own separate application.

One example beneficial aspect of utilizing autonomous vehicle102for both ridesharing and package delivery is increased utilization of the autonomous vehicle102. Instruction service156can continuously keep the autonomous vehicle102engaged in a productive itinerary between rideshare trips by filling what otherwise would have been idle time with productive package delivery trips.

FIG.2is a diagram illustrating an example image processing system200. The image processing system200can perform various image processing tasks and generate image data (e.g., images, videos, etc.) as described herein. In this example, the image processing system200can include an image sensor106A, a storage205, compute components210, and an image processing engine220. The image processing system200can also optionally include one or more other sensors104, such as another image sensor106B, a light detection and ranging (LIDAR) sensing device, an IMU, etc. For example, in dual camera or image sensor applications, the image processing system200can include front and rear image sensors (e.g.,106A,106B).

The image processing system200can be part of a computing device or multiple computing devices. In some examples, the image processing system200can be part of an electronic device (or devices) such as a camera system (e.g., a digital camera, an IP camera, a video camera, a security camera, etc.), a laptop or notebook computer, a tablet computer, a smart television, a display device, a digital media player, a video streaming device, internal computing system110, an IoT (Internet-of-Things) device, a server, or any other suitable electronic device(s).

In some implementations, the image sensor106A, the image sensor106B, the storage205, the compute components210, and the image processing engine220can be part of the same computing device. For example, in some cases, the image sensor106A, the image sensor106B, the storage205, the compute components210, and the image processing engine220can be integrated into a camera system, a computer system (e.g.,110), a smartphone, and/or any other computing device. However, in some implementations, the image sensor106A, the image sensor106B, the storage205, the compute components210, and/or the image processing engine220can be part of two or more separate computing devices.

The image sensors106A and106B can be any image and/or video sensors or capturing devices, such as a digital camera sensor, a video camera sensor, an image/video capture device on an electronic apparatus such as a computer, a camera system, etc. In some cases, the image sensors106A and106B can be part of a camera or computing device such as a digital camera, a video camera, an IP camera, a computing system, etc. In some examples, the image sensors106A and106B can be used to capture image data from an exterior of autonomous vehicle102. In some examples, the image sensors106A and106B can be part of a dual-camera assembly. The image sensors106A and106B can capture image and/or video content (e.g., raw image and/or video data), which can then be processed by the compute components210and image processing engine220as described herein.

The storage205can be any storage device(s) for storing data, such as image or video data for example. Moreover, the storage205can store data from any of the components of the image processing system200. For example, the storage205can store data from any of the sensors106A and/or106B, data from the compute components210(e.g., processing parameters, output images, calculation results, etc.), and/or data from the image processing engine220(e.g., output images/videos, processing results, etc.). In some examples, the storage205can include a buffer for storing data (e.g., image data) for processing by the compute components210.

In some implementations, the compute components210can include a central processing unit (CPU)212, a graphics processing unit (GPU)214, a digital signal processor (DSP)216, and an image signal processor (ISP)218. The compute components210can perform various operations such as image enhancement, object or image segmentation, computer vision, graphics rendering, augmented reality, image/video processing, sensor processing, recognition (e.g., object recognition, feature recognition, tracking or pattern recognition, scene change recognition, etc.), disparity detection, machine learning, filtering, image denoising, image demosaicing, auto white balance, color transforms, tone reproduction, lens shading correction, color interpolation, image scaling, colorspace conversion, and/or any other image processing operations. In some examples, the compute components210can implement the image processing engine220. In other examples, the compute components210can also implement one or more other processing engines.

The operations for the image processing engine220can be implemented by one or more of the compute components210. In one illustrative example, the image processing engine220can be implemented by the CPU212, the DSP216, and/or the ISP218, and the GPU214can implement operations for rendering image data from the image processing engine220. In some cases, the compute components210can include other electronic circuits or hardware, computer software, firmware, or any combination thereof, to perform any of the various operations described herein.

In some cases, the compute components210can receive data (e.g., image/video data, etc.) captured by the image sensor106A and process the data (e.g., via image processing engine220) to generate output images or frames as described herein. For example, the compute components210can process an image signal captured using large photodiodes and a first type of CFA implemented by the image sensor106A and an image signal captured using small photodiodes and a second type of CFA implemented by the image sensor106A, interpolate color channels or information from the image signals, process the interpolated image data and generate an output image. In some examples, an image or frame can be a red-green-blue (RGB) image or frame having red, green, and blue color components per pixel; a luma, chroma-red, chroma-blue (YCbCr) image or frame having a luma component and two chroma (color) components (chroma-red and chroma-blue) per pixel; or any other suitable type of color or monochrome picture.

The compute components210can implement the image processing engine220to perform various image processing operations and generate an output image as described herein. For example, the compute components210can implement the image processing engine220to perform lens shading correction, feature detection, blurring, segmentation, filtering, color correction, noise reduction, scaling, ranking, demosaicing, color interpolation, image signal processing, image enhancement, etc. The compute components210can process image data captured by the image sensors106A and/or106B; image data in storage205; image data received from a remote source, such as a remote camera, a server or a content provider; image data obtained from a combination of sources; etc.

While the image processing system200is shown to include certain components, one of ordinary skill will appreciate that the image processing system200can include more or fewer components than those shown inFIG.2. For example, the image processing system200can also include, in some instances, one or more memory devices (e.g., RAM, ROM, cache, and/or the like), one or more networking interfaces (e.g., wired and/or wireless communications interfaces and the like), one or more display devices, and/or other hardware or processing devices that are not shown inFIG.2. An illustrative example of a computing device and hardware components that can be implemented with the image processing system200is described below with respect toFIG.7.

FIG.3illustrates an example configuration of an image sensor (e.g.,106A) with different color filter arrays (CFAs)300for larger photodiodes314and smaller photodiodes316in the image sensor (e.g.,106A). Each of the color filter arrays300can include respective color filters302,304,306,308, and each color filter302,304,306,308can be configured to respectively cover (e.g., can be applied to, above, or over) a larger photodiode314or a smaller photodiode316. Moreover, the color filters302,306,308can each have a specific spectral sensitivity function, and can filter light by a specific wavelength range such that the separate filtered intensities include information about the color of light.

The color filters302,304,306,308can include, for example, a red (R) filter302, a clear (C) filter304, a green (G) filter306, and a blue (B) filter308. The clear filter304can be a white or transparent filter that does not filter any light or filters a very little amount of light, thus allowing a greater amount of light through than the other color filters302,306, and308and providing a higher pixel sensitivity.

The color filter arrays300can include CFA310which can be configured to cover the large photodiodes314, and CFA312which can be configured to cover the small photodiodes316. The CFA310and the CFA312can be different color filter arrays with different color patterns and/or mosaics. The different color filter arrays with different color patterns and/or mosaics can result in different spectral sensitivities and color separation properties.

For example, the CFA310covering the larger photodiodes314can be designed to increase the low light sensitivity of the larger photodiodes314. The larger surface area or aperture of the larger photodiodes314can allow the larger photodiodes314to collect a greater amount of light which, in combination with the increased low light sensitivity provided by the CFA310, can produce high spectral sensitivity and fidelity. Moreover, the CFA312covering the smaller photodiodes316can be designed to give information about the intensity of light in different color or wavelength regions (e.g., filter different light intensities) and provide higher color separation properties. Together, the CFA310and the CFA312implemented in a split-pixel or sub-pixel configuration (e.g., a configuration of larger photodiodes314and smaller photodiodes316) can thus provide enhanced low light sensitivity and color separation properties, as well as high dynamic range (HDR) imaging capabilities.

InFIG.3, the CFA310includes red filters302, clear filters304, and blue filters308in an RCCB configuration (e.g., 25% Red, 50% Clear, and 25% Blue). However, this example configuration is merely a non-limiting example provided for explanation purposes. One of skill in the art will recognize that, in other cases, the CFA310can include other configurations such as, for example, an RGBC (Red, Green, Blue, Clear) pattern, an RCBE (Red, Clear, Blue, Emerald) pattern, an RCYB (Red, Clear, Yellow, Blue) pattern, an RCCC (Red, Clear, Clear, Clear) pattern, etc.

Moreover, in this example, the CFA312includes red filters302, green filters306, and blue filters308in an RGGB configuration (e.g., 25% Red, 50% Green, and 25% Blue). However, this example configuration is merely a non-limiting example provided for explanation purposes. One of skill in the art will recognize that, in other cases, the CFA312can include other configurations such as, for example, an RCGB (Red, Clear, Green, Blue) pattern, an RGBE (Red, Green, Blue, Emerald) pattern, an RYYB (Red, Yellow, Yellow, Blue) pattern, etc.

In some examples, the CFA310and/or the CFA312can be patterned to include a clear filter304to allow more light to be detected and provide higher spectral sensitivity. For example, one of the CFAs (310or312) can include clear filters304to provide higher spectral sensitivity, while the other CFA (310or312) can include a different configuration of filters that provides better color separation properties. The combination of the different CFAs310and312can thus enhance the sensitivity of the image sensor (e.g.,106A), while also providing high or accurate color separation and fidelity.

FIG.4Aillustrates an example configuration400of a frontside-illuminated image sensor106A implementing the CFAs300shown inFIG.3. In this example configuration400, the image sensor106A can include a substrate region408containing photodiodes314and316configured to convert collected light into an electrical current. The substrate region408can include a substrate such as, for example and without limitation, silicon, germanium, silicon-germanium, a semiconductor compound, etc. Moreover, the photodiodes314and316can include larger photodiodes (314) and smaller photodiodes (316) in a split-pixel or sub-pixel configuration.

A wiring layer406can be positioned on or disposed over (e.g., on the top surface) the substrate region408. The wiring layer406can provide a passage to allow light collected by microlenses402on the image sensor106A to reach the photodiodes314and316. In some examples, the wiring layer406can include transistors and wires connected to the transistors. Moreover, in some examples, the wiring layer406can include conductive layers such as, for example and without limitation, metal and/or polysilicon layers.

A color filter array (CFA) layer404can be positioned on or disposed over (e.g., on the top surface) the wiring layer406and under the microlenses402. The CFA layer404can include CFAs300containing different color filters (e.g.,302A,302B,304A,304B,306,308) that selectively allow different, respective colors of light to pass through to the photodiodes314and316. As previously mentioned, at least one of the different color filters can include a clear filter which can allow light of all the color wavelengths to pass through.

In some examples, the CFAs300in the CFA layer404can include a CFA (e.g.,310) corresponding to the larger photodiodes314and a CFA (e.g.,312) corresponding to the smaller photodiodes316. The CFA corresponding to the larger photodiodes314can cover or overlap over the larger photodiodes314. The color filters in this CFA can filter specific colors of light and help focus the filtered light to corresponding ones of the larger photodiodes314(e.g., respective photodiodes covered or overlapped by corresponding color filters). Moreover, any clear filters in the CFA can allow light of all the color wavelengths to pass through to corresponding ones of the larger photodiodes314.

The CFA corresponding to the smaller photodiodes316can cover or overlap over the smaller photodiodes316. The color filters in this CFA can similarly filter specific colors of light and help focus the filtered light to corresponding ones of the smaller photodiodes316(e.g., respective photodiodes covered or overlapped by corresponding color filters). Moreover, any clear filters in the CFA can allow light of all the color wavelengths to pass through to corresponding ones of the larger photodiodes316.

As previously noted, the CFA layer404can be disposed under the microlenses402. In some cases, each microlens402can be disposed over a respective color filter from the CFAs300in the CFA layer404. The microlenses402can help the photodiodes314and316in the image sensor106A collect more light. For example, the microlenses402can help focus incident light onto the photodiodes314and316. The microlenses402can focus incoming light to the photodiodes314and316. The incoming light can pass through the CFA layer404and the wiring layer406, onto respective photodiodes314and316.

With reference toFIG.4B, which illustrates an example configuration420of a backside-illuminated image sensor106A implementing the CFAs300shown inFIG.3, in some cases the substrate region408containing the photodiodes314and316can instead be configured over the wiring layer406and under the CFA layer404. Here, the CFA layer404can be disposed over the photodiodes314and316, and light can pass through the microlenses402and CFA layer404onto the photodiodes314and316without first traveling through the wiring layer406. The wiring layer406can be positioned under the substrate region408, the CFA layer404and the microlenses402.

FIG.5illustrates an example flow500for processing image data generated by the larger photodiodes314and the smaller photodiodes316based on light signals filtered by CFAs300including a CFA (e.g.,310) corresponding to the larger photodiodes314and a different CFA (e.g.,312) corresponding to the smaller photodiodes316. In this example flow500, the image processing system200can obtain image sensor data502and perform front-end image processing504on the image sensor data502. The image processing system200can obtain the image sensor data502from an image sensor (e.g.,106A) configured with larger photodiodes314and smaller photodiodes316, as well as different CFAs (e.g.,310,312) respectively corresponding to the larger photodiodes314and the smaller photodiodes316.

The image sensor data502can include an image sensor signal502A generated from the larger photodiodes314based on light captured using a CFA (e.g.,310) corresponding to the larger photodiodes314, and an image sensor signal502B generated from the smaller photodiodes316based on light captured using a different CFA (e.g.,312) corresponding to the smaller photodiodes316. The image processing system200can perform the front-end image processing504on the image sensor signals502A and502B and process the output signals through different image signal processing pipelines as described herein.

The front-end image processing504can involve one or more initial or front-end image processing operations. For example, in some cases, the front-end image processing504can include lens shading correction or vignetting performed on the image sensor signals502A and502B. In other examples, the front-end image processing504can include other or additional operations performed on the image sensor signals502A and502B, such as, for example, auto exposure, auto white balance, gain correction, dead pixel detection or correction, etc.

After the front-end image processing504, the image processing system200can perform CFA color interpolation506A (e.g., CFA demosaicing) on the image sensor signal502A. The image processing system200can also (separately) perform CFA color interpolation506B (e.g., CFA demosaicing) on the image sensor signal502B. The CFA color interpolation506A performed on the image sensor signal502A can depend on the CFA implemented with the larger photodiodes314(e.g., the CFA applied to the light collected and processed by the larger photodiodes314) that generated the image sensor signal502A. For example, if the CFA implemented with the larger photodiodes314that generated the image sensor signal502A is an RCCB (Red, Clear, Clear, Blue) CFA, the CFA color interpolation506A can be an RCCB CFA color interpolation operation tailored for the RCCB CFA.

Similarly, the CFA color interpolation506B performed on the image sensor signal502B can depend on the CFA implemented with the smaller photodiodes316that generated the image sensor signal502B. For example, if the CFA implemented with the smaller photodiodes316that generated the image sensor signal502B is an RGGB (Red, Green, Green, Blue) CFA, the CFA color interpolation506B can be an RGGB CFA color interpolation operation tailored for the RGGB CFA.

The goal of the CFA color interpolation506A is to reconstruct a full color image based on the color channels/information in the image sensor signal502A, and the goal of the CFA color interpolation506B is to reconstruct a full color image based on the color channels/information in the image sensor signal502B. In some cases, the CFA color interpolations506A and506B can each implement an algorithm that performs color interpolation based on spatial and/or spectral correlation of pixels within the image sensor signals502A and502B.

In some examples, the CFA color interpolations506A and506B can estimate color channel values based on the color channel values of neighboring color channels or pixels (e.g., bilinear interpolation, bi-cubic interpolation, etc.). For example, in some cases, the CFA color interpolation506A can estimate a specific color channel based on a mean or average of the color channel values of a number of neighboring color channels in the image sensor signal502A. In other examples, the CFA color interpolations506A and506B can estimate color channel values based on edge-aware interpolation.

When completed, the CFA color interpolation506A can produce an image510A created based on the interpolation of color information in the image sensor signal502A. Similarly, when completed, the CFA color interpolation506B can produce an image510B created based on the interpolation of color information in the image sensor signal502B. The image processing system200can then perform post-interpolation image processing508on the images510A and510B. In the post-interpolation image processing508, the image processing system200can perform any of the operations in the image signal processing pipeline. For example, in the post-interpolation image processing508, the image processing system200can perform denoising, downsampling, color transforms, tone reproduction, image scaling, image enhancement, color space conversion, compression, etc.

The output of the post-interpolation image processing508can include image data generated based on the image sensor signals502A-B produced using the different CFAs (e.g.,300) and the photodiodes314-316. In some examples, the image data can include the images510A and/or510B after the post-interpolation image processing508. In other examples, the output of the post-interpolation image processing508can include image data (e.g., an image) generated based on the images510A and510B after the post-interpolation image processing508. In yet other examples, the output of the post-interpolation image processing508can include image data including at least a portion of the image510A and the image510B after the post-interpolation image processing508.

The output image data can be rendered on a display, stored on a storage device (e.g.,205), analyzed for certain image processing tasks (e.g., feature extraction, segmentation, object recognition, feature detection, etc.), sent to a separate or remote computing device, processed to generate one or more image effects, etc. In some examples, the output image data can be used by an autonomous vehicle102to help with self-driving or navigation operations. For example, the output image data can be used by an autonomous vehicle102to detect traffic light signals and/or other environment features (e.g., traffic signals, objects, lights, humans, animals, obstructions, other vehicles, landscape, structures, colors, etc.) captured by the image sensor106A, which the autonomous vehicle102can use to help with self-driving or navigation operations.

Having disclosed some example system components and concepts, the disclosure now turns toFIG.6, which illustrates an example method600for implementing different CFAs on an image sensor (106A) to enhance the image sensor's light sensitivity while preserving or improving color accuracy. The steps outlined herein are examples and can be implemented in any combination thereof, including combinations that exclude, add, or modify certain steps.

At step602, the method600can include receiving, by a first set of photodiodes (314) in an image sensor (106A), light filtered by a first CFA (310) covering the first set of photodiodes (314). The first CFA (310) can include a first set of color filters (e.g.,302A,304A,306). In some examples, some of the first set of color filters can include clear filters (310). The clear filters (310) can capture different colors of light and/or allow different colors of light to pass without being filtered. In some implementations, the first CFA (310) can include an RCCB (Red, Clear, Clear, Blue) color filter pattern. In other implementations, the first CFA (310) can include other color filter patterns, as previously described.

In some examples, the image sensor (106A) can be a high dynamic range (HDR) image sensor in an HDR camera system. Moreover, in some examples, the image sensor (106A) can include microlenses (402) covering the first CFA (310) and the second CFA (312). The microlenses (402) can focus, or aim, any incoming light onto respective color filters in the first CFA (310) and the second CFA (312). The microlenses (402) can also focus, or aim, any incoming light onto respective photodiodes in the first set of photodiodes (314) and the second set of photodiodes (316).

At step604, the method600can include receiving, by a second set of photodiodes (316) in the image sensor (106A), light filtered by a second CFA (312) covering the second set of photodiodes (316). The second CFA (312) can include a second set of color filters (e.g.,302B,304B,308). The second set of color filters in the second CFA (312) can be different than the first set of color filters in the first CFA (310). For example, the second set of color filters can include one or more color filters than the first set of color filters and/or can make up a different color filter pattern than the first set of color filters.

Moreover, each of the second set of photodiodes (316) can have a different size than each of the first set of photodiodes (314). For example, the photodiodes in the first set of photodiodes (314) can be larger than the photodiodes in the second set of photodiodes (316), or vice versa. The first set of photodiodes (314) and the second set of photodiodes (316) can thus have different aperture or surface area sizes. As a result, the first set of photodiodes (314) and the second set of photodiodes (316) can collect different amounts of light and can have different sensitivities.

The different CFAs (310,312) can exploit these characteristics of the first and second set of photodiodes (314,316) to optimize the image sensor's low light sensitivity, color fidelity, and color separation properties. For example, the different CFAs (310,312) can exploit these characteristics of the first and second set of photodiodes (314,316) by implementing a CFA that enhances light sensitivity on one of the sets of photodiodes, such as the set of larger photodiodes which have a higher sensitivity by virtue of their larger size, and implementing on the other set of photodiodes a different CFA that preserves or enhances color separation properties.

At step606, the method600can include generating, by the first set of photodiodes (314), a first image signal (e.g.,502A) that is based on the light filtered by the first CFA (310). At step608, the method600can include generating, by the second set of photodiodes (316), a second image signal (e.g.,502B) that is based on the light filtered by the second CFA (312).

At step610, the method600can include generating a first image (e.g.,510A) based on a first interpolation (e.g.,506A) of color information in the first image signal (e.g.,502A). At step612, the method600can also include generating a second image (e.g.,510B) based on a second interpolation (e.g.,506B) of color information in the second image signal (e.g.,502B).

In some cases, the method600can also include processing the first image (e.g.,510A) and the second image (e.g.,510B) through at least a portion of an image signal processing pipeline. The image signal processing pipeline can include one or more image processing operations such as, for example and without limitation, a denoising operation, a downsampling operation, a color transform, a tone reproduction operation, an image scaling operation, an image enhancement operation, a color space conversion operation, a compression operation, etc.

Moreover, in some cases, the method600can also include sending image data (e.g., the first image, the second image, an image generated based on the first and second images, image data including at least a portion of the first and/or second image, etc.) to one or more computing devices such as, for example, a computing system (e.g.,110), a storage (e.g.,205), a server, a mobile device, a processor, etc.

For example, in some cases, the method600can also include sending, to a computing system (110) configured to control one or more operations of an autonomous vehicle (102), image data including at least a portion of the first image (e.g.,510A) and/or the second image (e.g.,510B); detecting, via the computing system (110) on the autonomous vehicle (102), features in the image data; and generating, via the computing system (110) on the autonomous vehicle (102), navigation instructions based at least partly on the features detected in the image data.

The features detected in the image data can include environment conditions captured in the image data such as, for example and without limitation, a light or light color, an object (e.g., a sign, street/road obstruction, etc.), an animal, a human (e.g., a person crossing a street, a person on a sidewalk, a person on a bicycle, etc.), a traffic signal (e.g., a traffic light signal, a traffic sign signal, a roadside display signal, a break light on a vehicle, a turn signal on a vehicle, etc.), a landscape (e.g., a building, a sidewalk, a hill, a wall, a tree, a pothole, a traffic rail or guard, a dead end, etc.), another vehicle, weather conditions, scene characteristics, etc.

In some examples, the first set of photodiodes (314) and the second set of photodiodes (316) can be contained in a substrate layer (408) on the image sensor (106A). Moreover, in some examples, the image sensor (106A) can include a wiring layer (406). In some cases, the wiring layer (406) can include one or more transistors and one or more wires connected to the one or more transistors. Moreover, in some cases, the wiring layer406can include one or more conductive layers such as, for example and without limitation, a metal and/or polysilicon layer.

As described herein, one aspect of the present technology includes gathering and using data available from various sources to improve quality and experience. The present disclosure contemplates that in some instances, this gathered data may include personal information. The present disclosure contemplates that the entities involved with such personal information respect and value privacy policies and practices.

FIG.7illustrates an example computing system architecture700which can be implemented by, for example, any computing device making up internal computing system110, remote computing system150, computing device170, image processing system200, or any other computing device. InFIG.7, the components of the computing system architecture700are in communication with each other using connection705. Connection705can be a physical connection via a bus, or a direct connection into processor710, such as in a chipset architecture. Connection705can also be a virtual connection, networked connection, or logical connection.

In some implementations, computing system architecture700is a distributed system in which the functions described in this disclosure can be distributed within a datacenter, multiple data centers, a peer network, etc. In some implementations, one or more of the described system components represents many such components each performing some or all of the function for which the component is described. In some implementations, the components can be physical or virtual devices.

The computing system architecture700can include at least one processing unit (CPU or processor)710and connection705that couples various system components including system memory715, such as read-only memory (ROM)720and random access memory (RAM)725to processor710. Computing system architecture700can include a cache712of high-speed memory connected directly with, in close proximity to, or integrated as part of processor710.

Processor710can include any general purpose processor and a hardware service or software service, such as services732,734, and736stored in storage device730, configured to control processor710as well as a special-purpose processor where software instructions are incorporated into the actual processor design. Processor710may essentially be a completely self-contained computing system, containing multiple cores or processors, a bus, memory controller, cache, etc. A multi-core processor may be symmetric or asymmetric.

To enable user interaction, computing system architecture700includes an input device745, which can represent any number of input mechanisms, such as a microphone for speech, a touch-sensitive screen for gesture or graphical input, keyboard, mouse, motion input, speech, etc. Computing system architecture700can also include output device735, which can be one or more of a number of output mechanisms known to those of skill in the art. In some instances, multimodal systems can enable a user to provide multiple types of input/output to communicate with computing system architecture700. Computing system architecture700can include communications interface740, which can generally govern and manage the user input and system output. There is no restriction on operating on any particular hardware arrangement, and therefore the basic features here may easily be substituted for improved hardware or firmware arrangements as they are developed.

Storage device730can be a non-volatile memory device and can be a hard disk or other types of computer readable media which can store data that are accessible by a computer, such as magnetic cassettes, flash memory cards, solid state memory devices, digital versatile disks, cartridges, random access memories (RAMs), read-only memory (ROM), and/or some combination of these devices.

The storage device730can include software services, servers, services, etc., that when the code that defines such software is executed by the processor710, it causes the system to perform a function. In some implementations, a hardware service that performs a particular function can include the software component stored in a computer-readable medium in connection with the necessary hardware components, such as processor710, connection705, output device735, etc., to carry out the function.

For clarity of explanation, in some instances, the present technology may be presented as including individual functional blocks including functional blocks comprising devices, device components, steps or routines in a method embodied in software, or combinations of hardware and software.

Any of the steps, operations, functions, or processes described herein may be performed or implemented by a combination of hardware and software services or services, alone or in combination with other devices. In some embodiments, a service can be software that resides in memory of a client device and/or one or more servers of a content management system and perform one or more functions when a processor executes the software associated with the service. In some embodiments, a service is a program or a collection of programs that carry out a specific function. In some embodiments, a service can be considered a server. The memory can be a non-transitory computer-readable medium.

In some implementations, the computer-readable storage devices, mediums, and memories can include a cable or wireless signal containing a bit stream and the like. However, when mentioned, non-transitory computer-readable storage media expressly exclude media such as energy, carrier signals, electromagnetic waves, and signals per se.

Methods according to the above-described examples can be implemented using computer-executable instructions that are stored or otherwise available from computer-readable media. Such instructions can comprise, for example, instructions and data which cause or otherwise configure a general purpose computer, special purpose computer, or special purpose processing device to perform a certain function or group of functions. Portions of computer resources used can be accessible over a network. The executable computer instructions may be, for example, binaries, intermediate format instructions such as assembly language, firmware, or source code. Examples of computer-readable media that may be used to store instructions, information used, and/or information created during methods according to described examples include magnetic or optical disks, solid-state memory devices, flash memory, USB devices provided with non-volatile memory, networked storage devices, and so on.

Devices implementing methods according to these disclosures can comprise hardware, firmware and/or software, and can take any of a variety of form factors. Typical examples of such form factors include servers, laptops, smartphones, small form factor personal computers, personal digital assistants, and so on. The functionality described herein also can be embodied in peripherals or add-in cards. Such functionality can also be implemented on a circuit board among different chips or different processes executing in a single device, by way of further example.

The instructions, media for conveying such instructions, computing resources for executing them, and other structures for supporting such computing resources are means for providing the functions described in these disclosures.

Although a variety of examples and other information was used to explain aspects within the scope of the appended claims, no limitation of the claims should be implied based on particular features or arrangements in such examples, as one of ordinary skill would be able to use these examples to derive a wide variety of implementations. Further and although some subject matter may have been described in language specific to examples of structural features and/or method steps, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to these described features or acts. For example, such functionality can be distributed differently or performed in components other than those identified herein. Rather, the described features and steps are disclosed as examples of components of systems and methods within the scope of the appended claims.

Claim language reciting “at least one of” a set indicates that one member of the set or multiple members of the set satisfy the claim. For example, claim language reciting “at least one of A and B” means A, B, or A and B.