Patent Publication Number: US-11650360-B2

Title: Color filter array patterns for enhancing a low-light sensitivity while preserving a color accuracy in image signal processing applications

Description:
TECHNICAL FIELD 
     The present disclosure generally relates to color filter array patterns for image signal processing and, more specifically, color filter array patterns for image signal processing. 
     BACKGROUND 
     The sensitivity of an image sensor on a camera device depends on the quantum efficiency (QE) of the image sensor, the conversion gain (CG) of the image sensor, and the pixel size. The QE is the fraction of photon flux that contributes to the photocurrent in a photodetector or pixel. The QE provides the quality of the light or charge transformation in the image sensor, while the CG describes the image sensor&#39;s ability to convert the electrons generated into voltage. Thus, a higher QE can result in a higher sensitivity of an image sensor. To this end, color filter arrays (CFAs) are often implemented in image sensor devices to sense colors. The CFA can affect the QE and sensitivity of the image sensor largely depending on the materials and color pattern arrangement of the CFA. 
     A CFA is a mosaic of color filters placed over pixel/photodiode of an image sensor to capture color information. Unfortunately, while CFAs can improve the QE and sensitivity of an image sensor, current CFA technologies can often have a negative impact on the color separation properties of the image sensor. This drawback can create significant challenges in a variety of image signal processing tasks and applications. For example, in autonomous vehicle applications, which generally implement image sensor to help with various operations such as navigation, such diminished color separation properties can make red, green and yellow traffic light captured by an image sensor more difficult to separate. As a result, current CFA technologies can reduce an autonomous vehicle&#39;s ability to recognize different traffic lights and navigate without a human driver. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       The various advantages and features of the present technology will become apparent by reference to specific implementations illustrated in the appended drawings. A person of ordinary skill in the art will understand that these drawings only show some examples of the present technology and would not limit the scope of the present technology to these examples. Furthermore, the skilled artisan will appreciate the principles of the present technology as described and explained with additional specificity and detail through the use of the accompanying drawings in which: 
         FIG.  1    illustrates an example autonomous vehicle environment including a computing system in communication with an autonomous vehicle, in accordance with some examples; 
         FIG.  2    illustrates an example image processing system, in accordance with some examples; 
         FIG.  3    illustrates an example configuration of an image sensor with different color filter arrays for larger photodiodes and smaller photodiodes in the image sensor, in accordance with some examples; 
         FIG.  4 A  illustrates an example configuration of a frontside-illuminated image sensor implementing different color filter arrays, in accordance with some examples; 
         FIG.  4 B  illustrates an example configuration of a backside-illuminated image sensor implementing different color filter arrays, in accordance with some examples; 
         FIG.  5    illustrates an example flow for processing image data generated by larger photodiodes and smaller photodiodes on an image sensor based on light signals filtered by different color filter arrays corresponding to the larger photodiodes and the smaller photodiodes, in accordance with some examples; 
         FIG.  6    illustrates an example method for implementing different color filter arrays on an image sensor to enhance the image sensor&#39;s light sensitivity while preserving or improving a color accuracy, in accordance with some examples; and 
         FIG.  7    illustrates an example computing system architecture for implementing various aspects of the present technology. 
     
    
    
     DETAILED DESCRIPTION 
     Various examples of the present technology are discussed in detail below. While specific implementations are discussed, it should be understood that this is done for illustration purposes only. A person skilled in the relevant art will recognize that other components and configurations may be used without parting from the spirit and scope of the present technology. In some instances, well-known structures and devices are shown in block diagram form in order to facilitate describing one or more aspects. Further, it is to be understood that functionality that is described as being carried out by certain system components may be performed by more or fewer components than shown. 
     The disclosed technologies address a need in the art for enhancing the low light sensitivity of image sensors while preserving the color accuracy of image data for image signal processing. In some aspects, different color filter arrays (CFAs) can be implemented for small and large photodiodes in an image sensor to increase the quantum efficiency (QE) of the image sensor and thereby enhancing the low light sensitivity of the image sensor, while preserving or improving color separation and fidelity. 
     For example, to achieve high dynamic range (HDR), an image sensor can implement a split-pixel or sub-pixel design where two physical photodiodes, a large and a small photodiode, are associated with each pixel readout circuitry. To enhance the low light sensitivity of the image sensor while preserving or improving color separation qualities, a specific CFA that can enhance low light sensitivity can be implemented for the large photodiodes and a different CFA that can preserve or enhance color separation capabilities can be implemented for the small photodiodes. 
     To illustrate, in some examples, an RCCB (Red, Clear, Clear, Blue or 25% Red, 50% Clear, and 25% Blue) CFA can be used for the large photodiodes and an RGGB (Red, Green, Green, Blue or 25% Red, 50% Green, and 25% Blue) CFA can be used for the small photodiodes. Here, the clear portion of the RCCB CFA can increase the low light sensitivity of the large photodiodes while the RGGB CFA can preserve or enhance the color separation qualities of the small photodiodes. In addition, the combination of large photodiodes with higher light sensitivity and the small photodiodes with the lower light sensitivity can enable high dynamic range. 
     These approaches can be implemented in a variety of image processing applications and use cases. In some illustrative cases, the different CFAs used for the small and large photodiodes in an image sensor as described herein can be implemented in autonomous vehicle (AV) applications to enhance the low light sensitivity of the image sensor while preserving or improving color separation and fidelity to improve the image sensors ability to detect and accurately recognize different traffic lights and other lights in an AV environment. For example, in such implementations, traffic lights with a high signal intensity can be captured by the small photodiodes with a first type of CFA which can allow for better or accurate color separation and accuracy. For low signal intensity areas or regions, the large photodiodes with the second type of CFA containing a clear filter can provide low light sensitivity. The combination can thus provide high sensitivity with high color separation and accuracy. 
       FIG.  1    illustrates an example autonomous vehicle environment  100 , in which the technologies herein can be implemented. The example autonomous vehicle environment  100  includes an autonomous vehicle  102 , a remote computing system  150 , and a ridesharing application  172  on a computing device  170 . The autonomous vehicle  102 , remote computing system  150 , computing device  170  (including ridesharing application  172 ) can communicate with each other over one or more networks, such as a public network (e.g., a public cloud, the Internet, etc.), a private network (e.g., a local area network, a private cloud, a virtual private network, etc.), and/or a hybrid network (e.g., a multi-cloud or hybrid cloud network, etc.). 
     The autonomous vehicle  102  can navigate about roadways without a human driver based on sensor signals generated by sensors  104  on the autonomous vehicle  102 . The sensors  104  on the autonomous vehicle  102  can include one or more types of sensors and can be arranged about the autonomous vehicle  102 . For example, the sensors  104  can include, without limitation, one or more: inertial measuring units (IMUs), image sensors (e.g., visible light image sensors, infrared image sensors, video camera sensors, etc.), light emitting sensors (e.g., lasers, light detection and ranging sensors (LIDARs), etc.), global positioning system (GPS) devices, radars, sonars, accelerometers, gyroscopes, magnetometers, motion detection sensors, light detectors, audio sensors (e.g., microphones), seat occupancy sensors, ultrasonic sensors, position sensors, steering angle sensors, steering wheel rotation sensors, speedometers, proximity sensors, etc. 
     In some implementations, the sensors  104  can include one or more image sensors  106 A (e.g., camera sensors, visible light sensors, infrared light sensors, HDR image sensors, etc.), one or more radars sensors  106 B, and one or more LIDAR sensors  106 N. Other implementations can include any other number and types of sensors. 
     The autonomous vehicle  102  can include one or more display devices  108  for presenting information, such as maps, messages, and interfaces, to passengers in the autonomous vehicle  102 . The one or more display devices  108  can be mounted on one or more locations in the autonomous vehicle  102 . For example, the one or more display devices  108  can be mounted on one or more seats or headrests in the autonomous vehicle  102 , a dashboard in the autonomous vehicle  102 , one or more inner sides or door panels on the autonomous vehicle  102 , a roof of the autonomous vehicle  102 , and/or any other interior location of the autonomous vehicle  102 . The one or more display devices  108  can include, for example and without limitation, a screen, a television, a projecting device, and/or any other suitable display device for rendering graphical information. 
     Moreover, the autonomous vehicle  102  can include several mechanical systems that are used to effectuate motion of the autonomous vehicle  102 . For instance, the mechanical systems can include, but are not limited to, a vehicle propulsion system  130 , a braking system  132 , and a steering system  134 . The vehicle propulsion system  130  can include an electric motor, an internal combustion engine, or both. The braking system  132  can include an engine brake, brake pads, actuators, and/or any other suitable componentry configured to assist in decelerating the autonomous vehicle  102 . The steering system  134  includes suitable componentry configured to control the direction of movement of the autonomous vehicle  102  during navigation. 
     The autonomous vehicle  102  can include a safety system  136 . The safety system  136  can include lights and signal indicators, a parking brake, airbags, etc. The autonomous vehicle  102  can also include a cabin system  138 , which can include cabin temperature control systems, in-cabin entertainment systems, display devices, light-emitting devices, audio systems, etc. 
     The autonomous vehicle  102  can include an internal computing system  110  in communication with the sensors  104 , the display device(s)  108 , and the systems  130 ,  132 ,  134 ,  136 , and  138 . The internal computing system  110  can include one or more processors and at least one memory for storing instructions executable by the one or more processors. The computer-executable instructions can make up one or more services for controlling the autonomous vehicle  102 , communicating with remote computing system  150 , receiving inputs from passengers or human co-pilots, logging metrics regarding data collected by the sensors  104  and human co-pilots, etc. 
     The internal computing system  110  can include a control service  112  configured to control operation of the vehicle propulsion system  130 , the braking system  132 , the steering system  134 , the safety system  136 , and the cabin system  138 . The control service  112  can receive sensor signals from the sensors  104  can communicate with other services of the internal computing system  110  to effectuate operation of the autonomous vehicle  102 . In some examples, control service  112  may carry out operations in concert with one or more other systems of autonomous vehicle  102 . 
     The internal computing system  110  can also include a constraint service  114  to facilitate safe propulsion of the autonomous vehicle  102 . The constraint service  116  includes instructions for activating a constraint based on a rule-based restriction upon operation of the autonomous vehicle  102 . For example, the constraint may be a restriction on navigation that is activated in accordance with protocols configured to avoid occupying the same space as other objects, abide by traffic laws, circumvent avoidance areas, etc. In some examples, the constraint service  114  can be part of the control service  112 . 
     The internal computing system  110  can also include a communication service  116 . The communication service  116  can include software and/or hardware elements for transmitting and receiving signals to and from the remote computing system  150 . The communication service  116  can be configured to transmit information wirelessly over a network, for example, through an antenna array or interface system that provides cellular (long-term evolution (LTE), 3 rd  Generation (3G), 5 th  Generation (5G), etc.) communication, Bluetooth communication, near-field communication, and/or any other suitable type of wireless communication. 
     In some examples, one or more services of the internal computing system  110  are configured to send and receive communications to remote computing system  150  for reporting data for training and evaluating machine learning algorithms, requesting assistance from remote computing system  150  or a human operator via remote computing system  150 , software service updates, ridesharing pickup and drop off instructions, etc. 
     The internal computing system  110  can also include a latency service  118 . The latency service  118  can utilize timestamps on communications to and from the remote computing system  150  to determine if a communication has been received from the remote computing system  150  in time to be useful. For example, when a service of the internal computing system  110  requests feedback from remote computing system  150  on a time-sensitive process, the latency service  118  can determine if a response was timely received from remote computing system  150 , as information can quickly become too stale to be actionable. When the latency service  118  determines that a response has not been received within a threshold period of time, the latency service  118  can enable other systems of autonomous vehicle  102  or a passenger to make decisions or provide needed feedback. 
     The internal computing system  110  can also include a user interface service  120  that can communicate with cabin system  138  to provide information or receive information to a human co-pilot or passenger. In some examples, a human co-pilot or passenger can be asked or requested to evaluate and override a constraint from constraint service  114 . In other examples, the human co-pilot or passenger may wish to provide an instruction to, or receive an instruction from, the autonomous vehicle  102  regarding destinations, requested routes, drop-off locations, wayfinding tasks, or other requested operations. 
     As described above, the remote computing system  150  can be configured to send and receive signals to and from the autonomous vehicle  102 . The signals can include, for example and without limitation, data reported for training and evaluating services such as machine learning services, data for requesting assistance from remote computing system  150  or a human operator, software service updates, rideshare pickup and drop off instructions, etc. 
     The remote computing system  150  can include an analysis service  152  configured to receive data from autonomous vehicle  102  and analyze the data to train or evaluate machine learning algorithms for operating the autonomous vehicle  102 . The analysis service  152  can also perform analysis pertaining to data associated with one or more errors or constraints reported by autonomous vehicle  102 . 
     The remote computing system  150  can also include a user interface service  154  configured to present metrics, video, images, sounds reported from the autonomous vehicle  102  to an operator of remote computing system  150 , maps, routes, navigation data, notifications, user data, vehicle data, software data, and/or any other content. User interface service  154  can receive, from an operator, input instructions for the autonomous vehicle  102 . 
     The remote computing system  150  can also include an instruction service  156  for sending instructions regarding the operation of the autonomous vehicle  102 . For example, in response to an output of the analysis service  152  or user interface service  154 , instructions service  156  can prepare instructions to one or more services of the autonomous vehicle  102  or a co-pilot or passenger of the autonomous vehicle  102 . 
     The remote computing system  150  can also include a rideshare service  158  configured to interact with ridesharing applications  172  operating on computing device  170 . Computing device  170  can include, for example and without limitation, a tablet computer, a laptop computer, a smartphone, a head-mounted display (HMD), a gaming system, a server, a smart device, a laptop computer, a smart wearable (e.g., smart glasses, smart watch, etc.), a mobile computing device, and/or any other computing device. In some cases, the computing device  170  can be a mobile computing device of a user or passenger/rider. Moreover, in some cases, the remote computing system  150  and the autonomous vehicle  102  can also communicate and interact with other computing devices hosting instances of the ridesharing application  172  and the delivery service  174 . For example, the remote computing system  150  and the autonomous vehicle  102  can also communicate and interact with other computing devices associated with one or more passengers. 
     The rideshare service  158  can receive requests from passenger ridesharing application  172 , such as user requests to be picked up or dropped off, and can dispatch autonomous vehicle  102  for a requested trip. The rideshare service  158  can also act as an intermediary between the ridesharing application  172  and the autonomous vehicle  102 . For example, rideshare service  158  can receive from a passenger instructions for the autonomous vehicle  102 , such as instructions to go around an obstacle, change routes, select a drop-off location and/or pick-up location, etc. The rideshare service  158  can provide such instructions to the autonomous vehicle  102  as requested. 
     The remote computing system  150  can also include a package service  162  configured to interact with the computing device  170 , the ridesharing application  172  and/or a delivery service  174  of the ridesharing application  172 . A user operating the ridesharing application  172  can interact with the delivery service  174  to specify information regarding a package to be delivered using the autonomous vehicle  102 . The specified information can include, for example and without limitation, package dimensions, a package weight, a destination address, delivery instructions (e.g., a delivery time, a delivery note, a delivery constraint, etc.), and so forth. 
     The package service  162  can interact with the delivery service  174  to provide a package identifier to the user for package labeling and tracking. Package delivery service  174  can also inform a user of where to bring their labeled package for drop off. In some examples, a user can request the autonomous vehicle  102  come to a specific location, such as the user&#39;s location, to pick up the package. While delivery service  174  has been shown as part of the ridesharing application  172 , it will be appreciated by those of ordinary skill in the art that delivery service  174  can be its own separate application. 
     One example beneficial aspect of utilizing autonomous vehicle  102  for both ridesharing and package delivery is increased utilization of the autonomous vehicle  102 . Instruction service  156  can continuously keep the autonomous vehicle  102  engaged in a productive itinerary between rideshare trips by filling what otherwise would have been idle time with productive package delivery trips. 
       FIG.  2    is a diagram illustrating an example image processing system  200 . The image processing system  200  can perform various image processing tasks and generate image data (e.g., images, videos, etc.) as described herein. In this example, the image processing system  200  can include an image sensor  106 A, a storage  205 , compute components  210 , and an image processing engine  220 . The image processing system  200  can also optionally include one or more other sensors  104 , such as another image sensor  106 B, a light detection and ranging (LIDAR) sensing device, an IMU, etc. For example, in dual camera or image sensor applications, the image processing system  200  can include front and rear image sensors (e.g.,  106 A,  106 B). 
     The image processing system  200  can be part of a computing device or multiple computing devices. In some examples, the image processing system  200  can be part of an electronic device (or devices) such as a camera system (e.g., a digital camera, an IP camera, a video camera, a security camera, etc.), a laptop or notebook computer, a tablet computer, a smart television, a display device, a digital media player, a video streaming device, internal computing system  110 , an IoT (Internet-of-Things) device, a server, or any other suitable electronic device(s). 
     In some implementations, the image sensor  106 A, the image sensor  106 B, the storage  205 , the compute components  210 , and the image processing engine  220  can be part of the same computing device. For example, in some cases, the image sensor  106 A, the image sensor  106 B, the storage  205 , the compute components  210 , and the image processing engine  220  can be integrated into a camera system, a computer system (e.g.,  110 ), a smartphone, and/or any other computing device. However, in some implementations, the image sensor  106 A, the image sensor  106 B, the storage  205 , the compute components  210 , and/or the image processing engine  220  can be part of two or more separate computing devices. 
     The image sensors  106 A and  106 B can be any image and/or video sensors or capturing devices, such as a digital camera sensor, a video camera sensor, an image/video capture device on an electronic apparatus such as a computer, a camera system, etc. In some cases, the image sensors  106 A and  106 B can be part of a camera or computing device such as a digital camera, a video camera, an IP camera, a computing system, etc. In some examples, the image sensors  106 A and  106 B can be used to capture image data from an exterior of autonomous vehicle  102 . In some examples, the image sensors  106 A and  106 B can be part of a dual-camera assembly. The image sensors  106 A and  106 B can capture image and/or video content (e.g., raw image and/or video data), which can then be processed by the compute components  210  and image processing engine  220  as described herein. 
     The storage  205  can be any storage device(s) for storing data, such as image or video data for example. Moreover, the storage  205  can store data from any of the components of the image processing system  200 . For example, the storage  205  can store data from any of the sensors  106 A and/or  106 B, data from the compute components  210  (e.g., processing parameters, output images, calculation results, etc.), and/or data from the image processing engine  220  (e.g., output images/videos, processing results, etc.). In some examples, the storage  205  can include a buffer for storing data (e.g., image data) for processing by the compute components  210 . 
     In some implementations, the compute components  210  can include a central processing unit (CPU)  212 , a graphics processing unit (GPU)  214 , a digital signal processor (DSP)  216 , and an image signal processor (ISP)  218 . The compute components  210  can perform various operations such as image enhancement, object or image segmentation, computer vision, graphics rendering, augmented reality, image/video processing, sensor processing, recognition (e.g., object recognition, feature recognition, tracking or pattern recognition, scene change recognition, etc.), disparity detection, machine learning, filtering, image denoising, image demosaicing, auto white balance, color transforms, tone reproduction, lens shading correction, color interpolation, image scaling, colorspace conversion, and/or any other image processing operations. In some examples, the compute components  210  can implement the image processing engine  220 . In other examples, the compute components  210  can also implement one or more other processing engines. 
     The operations for the image processing engine  220  can be implemented by one or more of the compute components  210 . In one illustrative example, the image processing engine  220  can be implemented by the CPU  212 , the DSP  216 , and/or the ISP  218 , and the GPU  214  can implement operations for rendering image data from the image processing engine  220 . In some cases, the compute components  210  can include other electronic circuits or hardware, computer software, firmware, or any combination thereof, to perform any of the various operations described herein. 
     In some cases, the compute components  210  can receive data (e.g., image/video data, etc.) captured by the image sensor  106 A and process the data (e.g., via image processing engine  220 ) to generate output images or frames as described herein. For example, the compute components  210  can process an image signal captured using large photodiodes and a first type of CFA implemented by the image sensor  106 A and an image signal captured using small photodiodes and a second type of CFA implemented by the image sensor  106 A, interpolate color channels or information from the image signals, process the interpolated image data and generate an output image. In some examples, an image or frame can be a red-green-blue (RGB) image or frame having red, green, and blue color components per pixel; a luma, chroma-red, chroma-blue (YCbCr) image or frame having a luma component and two chroma (color) components (chroma-red and chroma-blue) per pixel; or any other suitable type of color or monochrome picture. 
     The compute components  210  can implement the image processing engine  220  to perform various image processing operations and generate an output image as described herein. For example, the compute components  210  can implement the image processing engine  220  to perform lens shading correction, feature detection, blurring, segmentation, filtering, color correction, noise reduction, scaling, ranking, demosaicing, color interpolation, image signal processing, image enhancement, etc. The compute components  210  can process image data captured by the image sensors  106 A and/or  106 B; image data in storage  205 ; image data received from a remote source, such as a remote camera, a server or a content provider; image data obtained from a combination of sources; etc. 
     While the image processing system  200  is shown to include certain components, one of ordinary skill will appreciate that the image processing system  200  can include more or fewer components than those shown in  FIG.  2   . For example, the image processing system  200  can also include, in some instances, one or more memory devices (e.g., RAM, ROM, cache, and/or the like), one or more networking interfaces (e.g., wired and/or wireless communications interfaces and the like), one or more display devices, and/or other hardware or processing devices that are not shown in  FIG.  2   . An illustrative example of a computing device and hardware components that can be implemented with the image processing system  200  is described below with respect to  FIG.  7   . 
       FIG.  3    illustrates an example configuration of an image sensor (e.g.,  106 A) with different color filter arrays (CFAs)  300  for larger photodiodes  314  and smaller photodiodes  316  in the image sensor (e.g.,  106 A). Each of the color filter arrays  300  can include respective color filters  302 ,  304 ,  306 ,  308 , and each color filter  302 ,  304 ,  306 ,  308  can be configured to respectively cover (e.g., can be applied to, above, or over) a larger photodiode  314  or a smaller photodiode  316 . Moreover, the color filters  302 ,  306 ,  308  can each have a specific spectral sensitivity function, and can filter light by a specific wavelength range such that the separate filtered intensities include information about the color of light. 
     The color filters  302 ,  304 ,  306 ,  308  can include, for example, a red (R) filter  302 , a clear (C) filter  304 , a green (G) filter  306 , and a blue (B) filter  308 . The clear filter  304  can be a white or transparent filter that does not filter any light or filters a very little amount of light, thus allowing a greater amount of light through than the other color filters  302 ,  306 , and  308  and providing a higher pixel sensitivity. 
     The color filter arrays  300  can include CFA  310  which can be configured to cover the large photodiodes  314 , and CFA  312  which can be configured to cover the small photodiodes  316 . The CFA  310  and the CFA  312  can be different color filter arrays with different color patterns and/or mosaics. The different color filter arrays with different color patterns and/or mosaics can result in different spectral sensitivities and color separation properties. 
     For example, the CFA  310  covering the larger photodiodes  314  can be designed to increase the low light sensitivity of the larger photodiodes  314 . The larger surface area or aperture of the larger photodiodes  314  can allow the larger photodiodes  314  to collect a greater amount of light which, in combination with the increased low light sensitivity provided by the CFA  310 , can produce high spectral sensitivity and fidelity. Moreover, the CFA  312  covering the smaller photodiodes  316  can be designed to give information about the intensity of light in different color or wavelength regions (e.g., filter different light intensities) and provide higher color separation properties. Together, the CFA  310  and the CFA  312  implemented in a split-pixel or sub-pixel configuration (e.g., a configuration of larger photodiodes  314  and smaller photodiodes  316 ) can thus provide enhanced low light sensitivity and color separation properties, as well as high dynamic range (HDR) imaging capabilities. 
     In  FIG.  3   , the CFA  310  includes red filters  302 , clear filters  304 , and blue filters  308  in an RCCB configuration (e.g., 25% Red, 50% Clear, and 25% Blue). However, this example configuration is merely a non-limiting example provided for explanation purposes. One of skill in the art will recognize that, in other cases, the CFA  310  can include other configurations such as, for example, an RGBC (Red, Green, Blue, Clear) pattern, an RCBE (Red, Clear, Blue, Emerald) pattern, an RCYB (Red, Clear, Yellow, Blue) pattern, an RCCC (Red, Clear, Clear, Clear) pattern, etc. 
     Moreover, in this example, the CFA  312  includes red filters  302 , green filters  306 , and blue filters  308  in an RGGB configuration (e.g., 25% Red, 50% Green, and 25% Blue). However, this example configuration is merely a non-limiting example provided for explanation purposes. One of skill in the art will recognize that, in other cases, the CFA  312  can include other configurations such as, for example, an RCGB (Red, Clear, Green, Blue) pattern, an RGBE (Red, Green, Blue, Emerald) pattern, an RYYB (Red, Yellow, Yellow, Blue) pattern, etc. 
     In some examples, the CFA  310  and/or the CFA  312  can be patterned to include a clear filter  304  to allow more light to be detected and provide higher spectral sensitivity. For example, one of the CFAs ( 310  or  312 ) can include clear filters  304  to provide higher spectral sensitivity, while the other CFA ( 310  or  312 ) can include a different configuration of filters that provides better color separation properties. The combination of the different CFAs  310  and  312  can thus enhance the sensitivity of the image sensor (e.g.,  106 A), while also providing high or accurate color separation and fidelity. 
       FIG.  4 A  illustrates an example configuration  400  of a frontside-illuminated image sensor  106 A implementing the CFAs  300  shown in  FIG.  3   . In this example configuration  400 , the image sensor  106 A can include a substrate region  408  containing photodiodes  314  and  316  configured to convert collected light into an electrical current. The substrate region  408  can include a substrate such as, for example and without limitation, silicon, germanium, silicon-germanium, a semiconductor compound, etc. Moreover, the photodiodes  314  and  316  can include larger photodiodes ( 314 ) and smaller photodiodes ( 316 ) in a split-pixel or sub-pixel configuration. 
     A wiring layer  406  can be positioned on or disposed over (e.g., on the top surface) the substrate region  408 . The wiring layer  406  can provide a passage to allow light collected by microlenses  402  on the image sensor  106 A to reach the photodiodes  314  and  316 . In some examples, the wiring layer  406  can include transistors and wires connected to the transistors. Moreover, in some examples, the wiring layer  406  can include conductive layers such as, for example and without limitation, metal and/or polysilicon layers. 
     A color filter array (CFA) layer  404  can be positioned on or disposed over (e.g., on the top surface) the wiring layer  406  and under the microlenses  402 . The CFA layer  404  can include CFAs  300  containing different color filters (e.g.,  302 A,  302 B,  304 A,  304 B,  306 ,  308 ) that selectively allow different, respective colors of light to pass through to the photodiodes  314  and  316 . As previously mentioned, at least one of the different color filters can include a clear filter which can allow light of all the color wavelengths to pass through. 
     In some examples, the CFAs  300  in the CFA layer  404  can include a CFA (e.g.,  310 ) corresponding to the larger photodiodes  314  and a CFA (e.g.,  312 ) corresponding to the smaller photodiodes  316 . The CFA corresponding to the larger photodiodes  314  can cover or overlap over the larger photodiodes  314 . The color filters in this CFA can filter specific colors of light and help focus the filtered light to corresponding ones of the larger photodiodes  314  (e.g., respective photodiodes covered or overlapped by corresponding color filters). Moreover, any clear filters in the CFA can allow light of all the color wavelengths to pass through to corresponding ones of the larger photodiodes  314 . 
     The CFA corresponding to the smaller photodiodes  316  can cover or overlap over the smaller photodiodes  316 . The color filters in this CFA can similarly filter specific colors of light and help focus the filtered light to corresponding ones of the smaller photodiodes  316  (e.g., respective photodiodes covered or overlapped by corresponding color filters). Moreover, any clear filters in the CFA can allow light of all the color wavelengths to pass through to corresponding ones of the larger photodiodes  316 . 
     As previously noted, the CFA layer  404  can be disposed under the microlenses  402 . In some cases, each microlens  402  can be disposed over a respective color filter from the CFAs  300  in the CFA layer  404 . The microlenses  402  can help the photodiodes  314  and  316  in the image sensor  106 A collect more light. For example, the microlenses  402  can help focus incident light onto the photodiodes  314  and  316 . The microlenses  402  can focus incoming light to the photodiodes  314  and  316 . The incoming light can pass through the CFA layer  404  and the wiring layer  406 , onto respective photodiodes  314  and  316 . 
     With reference to  FIG.  4 B , which illustrates an example configuration  420  of a backside-illuminated image sensor  106 A implementing the CFAs  300  shown in  FIG.  3   , in some cases the substrate region  408  containing the photodiodes  314  and  316  can instead be configured over the wiring layer  406  and under the CFA layer  404 . Here, the CFA layer  404  can be disposed over the photodiodes  314  and  316 , and light can pass through the microlenses  402  and CFA layer  404  onto the photodiodes  314  and  316  without first traveling through the wiring layer  406 . The wiring layer  406  can be positioned under the substrate region  408 , the CFA layer  404  and the microlenses  402 . 
       FIG.  5    illustrates an example flow  500  for processing image data generated by the larger photodiodes  314  and the smaller photodiodes  316  based on light signals filtered by CFAs  300  including a CFA (e.g.,  310 ) corresponding to the larger photodiodes  314  and a different CFA (e.g.,  312 ) corresponding to the smaller photodiodes  316 . In this example flow  500 , the image processing system  200  can obtain image sensor data  502  and perform front-end image processing  504  on the image sensor data  502 . The image processing system  200  can obtain the image sensor data  502  from an image sensor (e.g.,  106 A) configured with larger photodiodes  314  and smaller photodiodes  316 , as well as different CFAs (e.g.,  310 ,  312 ) respectively corresponding to the larger photodiodes  314  and the smaller photodiodes  316 . 
     The image sensor data  502  can include an image sensor signal  502 A generated from the larger photodiodes  314  based on light captured using a CFA (e.g.,  310 ) corresponding to the larger photodiodes  314 , and an image sensor signal  502 B generated from the smaller photodiodes  316  based on light captured using a different CFA (e.g.,  312 ) corresponding to the smaller photodiodes  316 . The image processing system  200  can perform the front-end image processing  504  on the image sensor signals  502 A and  502 B and process the output signals through different image signal processing pipelines as described herein. 
     The front-end image processing  504  can involve one or more initial or front-end image processing operations. For example, in some cases, the front-end image processing  504  can include lens shading correction or vignetting performed on the image sensor signals  502 A and  502 B. In other examples, the front-end image processing  504  can include other or additional operations performed on the image sensor signals  502 A and  502 B, such as, for example, auto exposure, auto white balance, gain correction, dead pixel detection or correction, etc. 
     After the front-end image processing  504 , the image processing system  200  can perform CFA color interpolation  506 A (e.g., CFA demosaicing) on the image sensor signal  502 A. The image processing system  200  can also (separately) perform CFA color interpolation  506 B (e.g., CFA demosaicing) on the image sensor signal  502 B. The CFA color interpolation  506 A performed on the image sensor signal  502 A can depend on the CFA implemented with the larger photodiodes  314  (e.g., the CFA applied to the light collected and processed by the larger photodiodes  314 ) that generated the image sensor signal  502 A. For example, if the CFA implemented with the larger photodiodes  314  that generated the image sensor signal  502 A is an RCCB (Red, Clear, Clear, Blue) CFA, the CFA color interpolation  506 A can be an RCCB CFA color interpolation operation tailored for the RCCB CFA. 
     Similarly, the CFA color interpolation  506 B performed on the image sensor signal  502 B can depend on the CFA implemented with the smaller photodiodes  316  that generated the image sensor signal  502 B. For example, if the CFA implemented with the smaller photodiodes  316  that generated the image sensor signal  502 B is an RGGB (Red, Green, Green, Blue) CFA, the CFA color interpolation  506 B can be an RGGB CFA color interpolation operation tailored for the RGGB CFA. 
     The goal of the CFA color interpolation  506 A is to reconstruct a full color image based on the color channels/information in the image sensor signal  502 A, and the goal of the CFA color interpolation  506 B is to reconstruct a full color image based on the color channels/information in the image sensor signal  502 B. In some cases, the CFA color interpolations  506 A and  506 B can each implement an algorithm that performs color interpolation based on spatial and/or spectral correlation of pixels within the image sensor signals  502 A and  502 B. 
     In some examples, the CFA color interpolations  506 A and  506 B can estimate color channel values based on the color channel values of neighboring color channels or pixels (e.g., bilinear interpolation, bi-cubic interpolation, etc.). For example, in some cases, the CFA color interpolation  506 A can estimate a specific color channel based on a mean or average of the color channel values of a number of neighboring color channels in the image sensor signal  502 A. In other examples, the CFA color interpolations  506 A and  506 B can estimate color channel values based on edge-aware interpolation. 
     When completed, the CFA color interpolation  506 A can produce an image  510 A created based on the interpolation of color information in the image sensor signal  502 A. Similarly, when completed, the CFA color interpolation  506 B can produce an image  510 B created based on the interpolation of color information in the image sensor signal  502 B. The image processing system  200  can then perform post-interpolation image processing  508  on the images  510 A and  510 B. In the post-interpolation image processing  508 , the image processing system  200  can perform any of the operations in the image signal processing pipeline. For example, in the post-interpolation image processing  508 , the image processing system  200  can perform denoising, downsampling, color transforms, tone reproduction, image scaling, image enhancement, color space conversion, compression, etc. 
     The output of the post-interpolation image processing  508  can include image data generated based on the image sensor signals  502 A-B produced using the different CFAs (e.g.,  300 ) and the photodiodes  314 - 316 . In some examples, the image data can include the images  510 A and/or  510 B after the post-interpolation image processing  508 . In other examples, the output of the post-interpolation image processing  508  can include image data (e.g., an image) generated based on the images  510 A and  510 B after the post-interpolation image processing  508 . In yet other examples, the output of the post-interpolation image processing  508  can include image data including at least a portion of the image  510 A and the image  510 B after the post-interpolation image processing  508 . 
     The output image data can be rendered on a display, stored on a storage device (e.g.,  205 ), analyzed for certain image processing tasks (e.g., feature extraction, segmentation, object recognition, feature detection, etc.), sent to a separate or remote computing device, processed to generate one or more image effects, etc. In some examples, the output image data can be used by an autonomous vehicle  102  to help with self-driving or navigation operations. For example, the output image data can be used by an autonomous vehicle  102  to detect traffic light signals and/or other environment features (e.g., traffic signals, objects, lights, humans, animals, obstructions, other vehicles, landscape, structures, colors, etc.) captured by the image sensor  106 A, which the autonomous vehicle  102  can use to help with self-driving or navigation operations. 
     Having disclosed some example system components and concepts, the disclosure now turns to  FIG.  6   , which illustrates an example method  600  for implementing different CFAs on an image sensor ( 106 A) to enhance the image sensor&#39;s light sensitivity while preserving or improving color accuracy. The steps outlined herein are examples and can be implemented in any combination thereof, including combinations that exclude, add, or modify certain steps. 
     At step  602 , the method  600  can include receiving, by a first set of photodiodes ( 314 ) in an image sensor ( 106 A), light filtered by a first CFA ( 310 ) covering the first set of photodiodes ( 314 ). The first CFA ( 310 ) can include a first set of color filters (e.g.,  302 A,  304 A,  306 ). In some examples, some of the first set of color filters can include clear filters ( 310 ). The clear filters ( 310 ) can capture different colors of light and/or allow different colors of light to pass without being filtered. In some implementations, the first CFA ( 310 ) can include an RCCB (Red, Clear, Clear, Blue) color filter pattern. In other implementations, the first CFA ( 310 ) can include other color filter patterns, as previously described. 
     In some examples, the image sensor ( 106 A) can be a high dynamic range (HDR) image sensor in an HDR camera system. Moreover, in some examples, the image sensor ( 106 A) can include microlenses ( 402 ) covering the first CFA ( 310 ) and the second CFA ( 312 ). The microlenses ( 402 ) can focus, or aim, any incoming light onto respective color filters in the first CFA ( 310 ) and the second CFA ( 312 ). The microlenses ( 402 ) can also focus, or aim, any incoming light onto respective photodiodes in the first set of photodiodes ( 314 ) and the second set of photodiodes ( 316 ). 
     At step  604 , the method  600  can include receiving, by a second set of photodiodes ( 316 ) in the image sensor ( 106 A), light filtered by a second CFA ( 312 ) covering the second set of photodiodes ( 316 ). The second CFA ( 312 ) can include a second set of color filters (e.g.,  302 B,  304 B,  308 ). The second set of color filters in the second CFA ( 312 ) can be different than the first set of color filters in the first CFA ( 310 ). For example, the second set of color filters can include one or more color filters than the first set of color filters and/or can make up a different color filter pattern than the first set of color filters. 
     Moreover, each of the second set of photodiodes ( 316 ) can have a different size than each of the first set of photodiodes ( 314 ). For example, the photodiodes in the first set of photodiodes ( 314 ) can be larger than the photodiodes in the second set of photodiodes ( 316 ), or vice versa. The first set of photodiodes ( 314 ) and the second set of photodiodes ( 316 ) can thus have different aperture or surface area sizes. As a result, the first set of photodiodes ( 314 ) and the second set of photodiodes ( 316 ) can collect different amounts of light and can have different sensitivities. 
     The different CFAs ( 310 ,  312 ) can exploit these characteristics of the first and second set of photodiodes ( 314 ,  316 ) to optimize the image sensor&#39;s low light sensitivity, color fidelity, and color separation properties. For example, the different CFAs ( 310 ,  312 ) can exploit these characteristics of the first and second set of photodiodes ( 314 ,  316 ) by implementing a CFA that enhances light sensitivity on one of the sets of photodiodes, such as the set of larger photodiodes which have a higher sensitivity by virtue of their larger size, and implementing on the other set of photodiodes a different CFA that preserves or enhances color separation properties. 
     At step  606 , the method  600  can include generating, by the first set of photodiodes ( 314 ), a first image signal (e.g.,  502 A) that is based on the light filtered by the first CFA ( 310 ). At step  608 , the method  600  can include generating, by the second set of photodiodes ( 316 ), a second image signal (e.g.,  502 B) that is based on the light filtered by the second CFA ( 312 ). 
     At step  610 , the method  600  can include generating a first image (e.g.,  510 A) based on a first interpolation (e.g.,  506 A) of color information in the first image signal (e.g.,  502 A). At step  612 , the method  600  can also include generating a second image (e.g.,  510 B) based on a second interpolation (e.g.,  506 B) of color information in the second image signal (e.g.,  502 B). 
     In some cases, the method  600  can also include processing the first image (e.g.,  510 A) and the second image (e.g.,  510 B) through at least a portion of an image signal processing pipeline. The image signal processing pipeline can include one or more image processing operations such as, for example and without limitation, a denoising operation, a downsampling operation, a color transform, a tone reproduction operation, an image scaling operation, an image enhancement operation, a color space conversion operation, a compression operation, etc. 
     Moreover, in some cases, the method  600  can also include sending image data (e.g., the first image, the second image, an image generated based on the first and second images, image data including at least a portion of the first and/or second image, etc.) to one or more computing devices such as, for example, a computing system (e.g.,  110 ), a storage (e.g.,  205 ), a server, a mobile device, a processor, etc. 
     For example, in some cases, the method  600  can also include sending, to a computing system ( 110 ) configured to control one or more operations of an autonomous vehicle ( 102 ), image data including at least a portion of the first image (e.g.,  510 A) and/or the second image (e.g.,  510 B); detecting, via the computing system ( 110 ) on the autonomous vehicle ( 102 ), features in the image data; and generating, via the computing system ( 110 ) on the autonomous vehicle ( 102 ), navigation instructions based at least partly on the features detected in the image data. 
     The features detected in the image data can include environment conditions captured in the image data such as, for example and without limitation, a light or light color, an object (e.g., a sign, street/road obstruction, etc.), an animal, a human (e.g., a person crossing a street, a person on a sidewalk, a person on a bicycle, etc.), a traffic signal (e.g., a traffic light signal, a traffic sign signal, a roadside display signal, a break light on a vehicle, a turn signal on a vehicle, etc.), a landscape (e.g., a building, a sidewalk, a hill, a wall, a tree, a pothole, a traffic rail or guard, a dead end, etc.), another vehicle, weather conditions, scene characteristics, etc. 
     In some examples, the first set of photodiodes ( 314 ) and the second set of photodiodes ( 316 ) can be contained in a substrate layer ( 408 ) on the image sensor ( 106 A). Moreover, in some examples, the image sensor ( 106 A) can include a wiring layer ( 406 ). In some cases, the wiring layer ( 406 ) can include one or more transistors and one or more wires connected to the one or more transistors. Moreover, in some cases, the wiring layer  406  can include one or more conductive layers such as, for example and without limitation, a metal and/or polysilicon layer. 
     As described herein, one aspect of the present technology includes gathering and using data available from various sources to improve quality and experience. The present disclosure contemplates that in some instances, this gathered data may include personal information. The present disclosure contemplates that the entities involved with such personal information respect and value privacy policies and practices. 
       FIG.  7    illustrates an example computing system architecture  700  which can be implemented by, for example, any computing device making up internal computing system  110 , remote computing system  150 , computing device  170 , image processing system  200 , or any other computing device. In  FIG.  7   , the components of the computing system architecture  700  are in communication with each other using connection  705 . Connection  705  can be a physical connection via a bus, or a direct connection into processor  710 , such as in a chipset architecture. Connection  705  can also be a virtual connection, networked connection, or logical connection. 
     In some implementations, computing system architecture  700  is a distributed system in which the functions described in this disclosure can be distributed within a datacenter, multiple data centers, a peer network, etc. In some implementations, one or more of the described system components represents many such components each performing some or all of the function for which the component is described. In some implementations, the components can be physical or virtual devices. 
     The computing system architecture  700  can include at least one processing unit (CPU or processor)  710  and connection  705  that couples various system components including system memory  715 , such as read-only memory (ROM)  720  and random access memory (RAM)  725  to processor  710 . Computing system architecture  700  can include a cache  712  of high-speed memory connected directly with, in close proximity to, or integrated as part of processor  710 . 
     Processor  710  can include any general purpose processor and a hardware service or software service, such as services  732 ,  734 , and  736  stored in storage device  730 , configured to control processor  710  as well as a special-purpose processor where software instructions are incorporated into the actual processor design. Processor  710  may essentially be a completely self-contained computing system, containing multiple cores or processors, a bus, memory controller, cache, etc. A multi-core processor may be symmetric or asymmetric. 
     To enable user interaction, computing system architecture  700  includes an input device  745 , which can represent any number of input mechanisms, such as a microphone for speech, a touch-sensitive screen for gesture or graphical input, keyboard, mouse, motion input, speech, etc. Computing system architecture  700  can also include output device  735 , which can be one or more of a number of output mechanisms known to those of skill in the art. In some instances, multimodal systems can enable a user to provide multiple types of input/output to communicate with computing system architecture  700 . Computing system architecture  700  can include communications interface  740 , which can generally govern and manage the user input and system output. There is no restriction on operating on any particular hardware arrangement, and therefore the basic features here may easily be substituted for improved hardware or firmware arrangements as they are developed. 
     Storage device  730  can be a non-volatile memory device and can be a hard disk or other types of computer readable media which can store data that are accessible by a computer, such as magnetic cassettes, flash memory cards, solid state memory devices, digital versatile disks, cartridges, random access memories (RAMs), read-only memory (ROM), and/or some combination of these devices. 
     The storage device  730  can include software services, servers, services, etc., that when the code that defines such software is executed by the processor  710 , it causes the system to perform a function. In some implementations, a hardware service that performs a particular function can include the software component stored in a computer-readable medium in connection with the necessary hardware components, such as processor  710 , connection  705 , output device  735 , etc., to carry out the function. 
     For clarity of explanation, in some instances, the present technology may be presented as including individual functional blocks including functional blocks comprising devices, device components, steps or routines in a method embodied in software, or combinations of hardware and software. 
     Any of the steps, operations, functions, or processes described herein may be performed or implemented by a combination of hardware and software services or services, alone or in combination with other devices. In some embodiments, a service can be software that resides in memory of a client device and/or one or more servers of a content management system and perform one or more functions when a processor executes the software associated with the service. In some embodiments, a service is a program or a collection of programs that carry out a specific function. In some embodiments, a service can be considered a server. The memory can be a non-transitory computer-readable medium. 
     In some implementations, the computer-readable storage devices, mediums, and memories can include a cable or wireless signal containing a bit stream and the like. However, when mentioned, non-transitory computer-readable storage media expressly exclude media such as energy, carrier signals, electromagnetic waves, and signals per se. 
     Methods according to the above-described examples can be implemented using computer-executable instructions that are stored or otherwise available from computer-readable media. Such instructions can comprise, for example, instructions and data which cause or otherwise configure a general purpose computer, special purpose computer, or special purpose processing device to perform a certain function or group of functions. Portions of computer resources used can be accessible over a network. The executable computer instructions may be, for example, binaries, intermediate format instructions such as assembly language, firmware, or source code. Examples of computer-readable media that may be used to store instructions, information used, and/or information created during methods according to described examples include magnetic or optical disks, solid-state memory devices, flash memory, USB devices provided with non-volatile memory, networked storage devices, and so on. 
     Devices implementing methods according to these disclosures can comprise hardware, firmware and/or software, and can take any of a variety of form factors. Typical examples of such form factors include servers, laptops, smartphones, small form factor personal computers, personal digital assistants, and so on. The functionality described herein also can be embodied in peripherals or add-in cards. Such functionality can also be implemented on a circuit board among different chips or different processes executing in a single device, by way of further example. 
     The instructions, media for conveying such instructions, computing resources for executing them, and other structures for supporting such computing resources are means for providing the functions described in these disclosures. 
     Although a variety of examples and other information was used to explain aspects within the scope of the appended claims, no limitation of the claims should be implied based on particular features or arrangements in such examples, as one of ordinary skill would be able to use these examples to derive a wide variety of implementations. Further and although some subject matter may have been described in language specific to examples of structural features and/or method steps, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to these described features or acts. For example, such functionality can be distributed differently or performed in components other than those identified herein. Rather, the described features and steps are disclosed as examples of components of systems and methods within the scope of the appended claims. 
     Claim language reciting “at least one of” a set indicates that one member of the set or multiple members of the set satisfy the claim. For example, claim language reciting “at least one of A and B” means A, B, or A and B.