Patent ID: 12231790

DETAILED DESCRIPTION OF EMBODIMENTS

Described herein are embodiments of systems and methods for optimizing alignment of pixels of two or more different types on an image sensor array, for the performance of two or more camera tasks, taking task priority levels into account.

FIG.1is a flowchart illustrating steps of a method100according to some embodiments of the present invention. At step101, an output of each type of pixel on an image sensor array presented with an input image is obtained. At step102, for a potential pixel alignment pattern on the sensor array, the quality of the outputs is evaluated for each of the N camera tasks. At step103, an optimal pixel alignment pattern is obtained by adjusting the potential pixel alignment pattern with respect to performance of the N camera tasks, subject to a user-determined balance of priorities among the N camera tasks.

In some embodiments of method100, obtaining the optimal pixel alignment pattern comprises evaluating all possible combinations of pixel alignments on the sensor array. In other embodiments, it comprises using one or more rule-based algorithms. In some embodiment of method100, obtaining the optimal pixel alignment pattern comprises using machine learning. The machine learning may utilize a trainable pixel alignment layer, in some cases involving a continuous relaxation technique. Alternatively, the machine learning may utilize a reinforcement learning technique.

In some embodiments of method100, the sensor array comprises a tessellation of identical pixel blocks, where each block comprises a first arrangement of pixels of the X different types; and where obtaining the optimal pixel alignment pattern comprises obtaining an optimal arrangement of pixels of the X different types in one block. In one subset of these embodiments, obtaining the optimal arrangement of pixels in each block comprises evaluating all possible combinations of pixel alignments in the block. In another subset of these embodiments, obtaining the optimal arrangement of pixels in each block comprises using one or more rule-based algorithms. In yet another subset of these embodiments, obtaining the optimal arrangement of pixels in each block comprises using machine learning, where the machine learning may utilize a trainable pixel alignment layer, in some cases involving a continuous relaxation technique, or alternatively, the machine learning may utilize a reinforcement learning technique.

In other embodiments of method100, the sensor array comprises a tessellation of first and second pixel blocks, where each of the first blocks comprises a first arrangement of pixels of the X different types, where each of the second blocks comprising a second arrangement, different from the first arrangement, of pixels of the X different types, and where obtaining the optimal pixel alignment pattern comprises obtaining an optimal arrangement of pixels of the X different types in a first block and in a second block. Subsets of embodiments may readily be envisaged for the various options discussed above regarding different ways of obtaining the optimal pixel alignment pattern (evaluating all possible combinations, using rule-based algorithms etc.).

In some embodiments of method100, each pixel type is characterized by one or more associated tunable parameters, such as phase angle, filter wavelength, etc. In some embodiments of method100, step101, in which an output of each type of pixel on the sensor is obtained, comprises simulating a response of that type of pixel. A description of how this may be done is presented below, with reference to system200, illustrated inFIG.2. In other embodiments, obtaining those outputs comprises measuring a response of each type of pixel, rather than simulating it. This may be done by physically using a camera to capture an image and recording pixel responses directly.

FIG.2illustrates a system200according to some embodiments of the present invention. This system in essence carries out method100using a particular combination of choices for implementing steps101,102and103. Other systems may be envisaged that use other choices, without departing from the spirit and scope of the present invention.

Pixel response simulator202accepts a training image201from a training image database (not shown) and generates a plurality of X sensor images203, one for each type of pixel whose response is simulated, delivering the images to subsampler206. Trainable pixel alignment layer module204generates a pixel alignment pattern205for the mixture of pixel types, delivering that pattern to subsampler206. Subsampler206operates on each one of the received sensor images203in turn, extracting data for pixels of that corresponding pixel type at each location indicated for that pixel type in pattern205for one block, and then tessellating the resulting fractional part of the simulated sensor image over the sensor array. In this way, subsampler206generates X subsampled images207, one image for each of the X types of pixels.

A stack208of N trainable networks receives the subsampled images207, such that each individual network, dedicated to simulating just one camera processing task, receives one or more of the subsampled images as appropriate for the operation of that particular task. So, for example, network208-1, dedicated to simulating camera processing task #1, receives one or more of the subsampled images as appropriate for the operation of task #1, network208-2, for task #1, receives whichever subsampled image or images are relevant to task #2network, and so on. There may, but need not, be some overlap between subsampled images sent to different networks, depending on the nature of the corresponding camera tasks, as will be discussed below in regard toFIGS.3and4.

In some embodiments, each of the N neural network may receive all of the X subsampled images, processing some of them using information that does not directly relate to the specific camera task of that particular network but provides relevant side information. Each network of stack208provides an output value reflective of how well it currently performs its corresponding task, so that208-1provides a performance value209-1for its performance of task #1,208-2provides109-2, and so on with network208-N providing209-N for the Nth task. The individual performance values are input to module210, which calculates a combination performance value1212, reflective of how well all N tasks may be performed for the current version of pixel alignment pattern205and the current internal organization of each network. The calculation involves weighting the individual performance values according to predetermined task priority parameters. In some cases, these parameters may be supplied as another input (not shown) to module210.

The combination performance value212is fed back from the output of module210though inner and outer feedback loops, simultaneously. The inner loop includes path A between module210and stack208, allowing each of the networks208-1through208-N to self-train to optimize performance value212. In some embodiments this means to minimize a corresponding combined loss value. The outer loop includes path B between module210and pixel alignment layer204, allowing that layer204to be trained to optimize performance value212by optimizing pixel alignment pattern205.1Another term in optimization literature for “combination performance value” is “objective function”.

The training of system200is continued according to the process described above for each of a series of training image available from the database.

Modules204and208are tunable, allowing them to be optimized using cues from the combination performance value212.

One possible method for performing this optimization is by using stochastic gradient descent, which computes the gradient of the performance values with respect to the tunable parameters (neural network weights) and applies an update to the tunable parameters the direction of the gradient, thereby increasing the performance value. In the framework of stochastic gradient descent, all modules between the trainable module and the performance value are required to be differentiable.

Given these requirements, it may be appreciated that a trainable alignment layer such as layer204inFIG.2must be defined as a differentiable function. The gradient of the performance value has to somehow be computed with respect to different pixel choices. However, due to the discrete nature of pixel choices the gradient will be undefined.

To address this, the present invention implements layer204using a method that approximates the discrete pixel choices in a way that is differentiable.

The method used in some embodiments of the present invention falls under the umbrella of ‘continuous relaxation’, a technique used to address a general category of problems where a derivative and/or gradient needs to be computed from a non-differentiable discrete or categorical function, by recasting it to an approximation that uses continuous variables.

While continuous relaxation is one way of approximating the non-differentiable function with a differentiable function, there are other methods that could be considered in other embodiments.

By implementing204with a differentiable approximation, the present invention now enables the sensor designer to optimize a sensor where discrete and categorical design choices can be made. This contrasts with prior art, which restricts the optimization space to only be continuous.

FIG.3illustrates a portion of the system of the type shown inFIG.2, for one exemplary embodiment where there are three different pixel types to be arranged on the sensor, and two different camera tasks of interest. In this embodiment, pixel response simulator302accepts training image301and outputs three sensor images,303A,303B and303C, where303A shows what the sensor output would be if the sensor array just had “A” type pixels at every pixel location over a sensor array, and303B and303C do the same for “B” and “C” type pixels respectively.

Pixel alignment layer304generates pattern305in the form of a 4×4 block, which would be repeated (not shown) as a 2D tessellation over the image sensor array surface. Each block happens to have 5 of each of the “A” and “C” type pixels and 6 of the “B” ones, the latter being arranged diagonally, but it should be understood this is just one “candidate” for an optimized arrangement.

Subsampler306operates on sensor images303A,303B, and303C, extracting data for pixels of types A, B and C at each block indicated for that pixel type in pattern305, and then tessellating the resulting fractional part of the simulated sensor image over the sensor array. The resulting output is three subsampled images307A,307B, and307C.

In this exemplary system with two camera tasks of interest, stack308is made up of two corresponding trainable neural networks308-1and308-2, delivering outputs309-1and309-2. In the case shown, the performance of network308-1's camera task is affected solely by how pixels of type A function, not by pixels of type B or C, so only subsampled image307A is fed into that network as an input, The task of network308-1may, for example, be an object recognition task, determined entirely by pixels of type A. At the same time, the performance of the camera task corresponding to network308-2may be affected to a significant extent by how pixels of type B and type C function, so network308-2will require both images307B and307C to be provided as inputs. The task of network308-2may, for example, be an “image quality” task, to provide images with high dynamic range, for which a mixture of pixels of high sensitivity and pixels of high saturation thresholds may be necessary.

In some systems similar to that shown inFIG.3, all three types of pixels may be relevant to the performance of both tasks of interest. In other embodiments, not shown, at least one pixel type may be relevant to both of the camera tasks, and so on. The number of variations will increase, of course, as more pixel types and/or more camera tasks are modeled. A key feature of all embodiments of the current invention is that the X pixel types are distinctly different from each other, and the N camera tasks are distinctly different from each other. In many cases, at least one of the tasks achieves an image quality objective, determined by pixels of one or more types within one subcategory of simple image capture pixels, while another of the tasks achieves an objective that depends on pixels of a distinctly different type, such as phase detection pixels. Producing an image with high dynamic range is one example of an image quality task, while processing a captured image to identify objects of a particular shape in the scene is an example of a very different type of camera or image processing task. Other tasks could include generating a depth map, or producing a semantic understanding of the scene

Returning toFIG.3, the outputs of the two networks are performance values309-1and309-2, respectively, which are then fed into a module (not shown) corresponding to module210of system200, that operates on those values to generate a combination performance value reflective of how well the pair of tasks would be performed for the current version of pixel alignment pattern305for the three pixel types of interest, and for the current internal organization of the neural networks308-1and308-2. The combined performance value is fed back to the neural networks in stack308and to the pixel alignment layer304, generating improved network parameters and pixel alignment patterns by processes well known in the art of deep learning for system parameter optimization.

FIG.4indicates how performance values may be calculated and processed in some embodiments of the present invention where performance values are expressed as loss values which are inversely related to how well the tasks are performed. In the case shown, there are N neural networks in stack408, where the first network outputs a performance value408-1in the form of a task loss L1and so one until the last, Nth network, outputs a performance value408-N in the form of a task loss LN. Then at module410, a combined performance value is calculated as a combination of a function of each performance value and a corresponding weighting parameter α for each task, so that the kth term in the function, where k varies from 1 to N, is represented as f (αk, Lk,). In some embodiments, the function may depend linearly on the individual performance value, so that the kth term would be αk*Lk, for example. In other embodiments, the function may be nonlinear, depending, for example, on (Lk,)2. The weighting parameters, indicating relative priorities of the N tasks, may be predetermined inputs embedded into the system, or as adjustable parameters set by a user training the system according to their preferences.

Other options may readily be envisaged for training systems of the present invention, in which performance values are again be expressed as loss values which are inversely related to how well the tasks are performed, but where the combined performance value is calculated as a combination of a function of each performance value and two or more task-specific weighting parameters. In a 2-parameter case, for example, the kth term in the function may be [(ακ)2*Lk]+[βκ*(Lk)2] for example. Many variations may be envisaged, some of which could involve more than two weighting parameters.

The embodiments described above and shown in the figures all involve the use of a stack of separate neural networks, but the present invention is not necessarily limited to this particular architecture.FIG.5illustrates some other embodiments, in which there is one large ‘super’ neural network508into which all X sensor images are input, and which outputs N values corresponding to each task. Other elements of system500operate just as system200illustrated inFIG.2, as follows.

Pixel response simulator502accepts a training image501from a training image database (not shown) and generates a plurality of X sensor images503, one for each type of pixel whose response is simulated, delivering the images to subsampler506. Trainable pixel alignment layer module504generates a pixel alignment pattern505for the mixture of pixel types, delivering that pattern to subsampler506. Subsampler506operates on each one of the received sensor images503in turn, extracting data for pixels of that corresponding pixel type at each location indicated for that pixel type in pattern505for one block, and then tessellating the resulting fractional part of the simulated sensor image over the sensor array. In this way, subsampler506generates X subsampled images507, one image for each of the X types of pixels.

However, trainable network508receives all the images507, and provides N output values, one for each of the N camera tasks of interest, indicating how well the system currently performs that corresponding task, The individual performance values are input to module510, which calculates a combination performance value512, reflective of how well all N tasks may be performed for the current version of pixel alignment pattern505and the current internal organization of super network508. The calculation involves weighting the individual performance values according to predetermined task priority parameters. In some cases, these parameters may be supplied as another input (not shown) to module510.

Embodiments of the present invention offer a major advantage over prior art in this field, in providing systems and methods for automatic optimization of alignment of pixels of different basic types for a camera intended to perform more than one task, while taking into account a user-determined priority between those tasks. In some embodiments, this is achieved by performing an end-to-end training of a system including a stack of task-specific neural networks and a trainable pixel alignment layer, with the user setting one or more parameters that serve to balance competing aspects of camera performance.

Although the description has been described with respect to particular embodiments thereof, these particular embodiments are merely illustrative, and not restrictive.

Any suitable programming language can be used to implement the routines of particular embodiments including C, C++, Java, assembly language, etc. Different programming techniques can be employed such as procedural or object oriented. The routines can execute on a single processing device or multiple processors. Although the steps, operations, or computations may be presented in a specific order, this order may be changed in different particular embodiments. In some particular embodiments, multiple steps shown as sequential in this specification can be performed at the same time.

Particular embodiments may be implemented in a computer-readable storage medium for use by or in connection with the instruction execution system, apparatus, system, or device. Particular embodiments can be implemented in the form of control logic in software or hardware or a combination of both. The control logic, when executed by one or more processors, may be operable to perform that which is described in particular embodiments.

Particular embodiments may be implemented by using a programmed general purpose digital computer, by using application specific integrated circuits, programmable logic devices, field programmable gate arrays, optical, chemical, biological, quantum or nanoengineered systems, components and mechanisms may be used. In general, the functions of particular embodiments can be achieved by any means as is known in the art. Distributed, networked systems, components, and/or circuits can be used. Communication, or transfer, of data may be wired, wireless, or by any other means.

It will also be appreciated that one or more of the elements depicted in the drawings/figures can also be implemented in a more separated or integrated manner, or even removed or rendered as inoperable in certain cases, as is useful in accordance with a particular application. It is also within the spirit and scope to implement a program or code that can be stored in a machine-readable medium to permit a computer to perform any of the methods described above.

A “processor” includes any suitable hardware and/or software system, mechanism or component that processes data, signals or other information. A processor can include a system with a general-purpose central processing unit, multiple processing units, dedicated circuitry for achieving functionality, or other systems. Processing need not be limited to a geographic location, or have temporal limitations. For example, a processor can perform its functions in “real time,” “offline,” in a “batch mode,” etc. Portions of processing can be performed at different times and at different locations, by different (or the same) processing systems. Examples of processing systems can include servers, clients, end user devices, routers, switches, networked storage, etc. A computer may be any processor in communication with a memory. The memory may be any suitable processor-readable storage medium, such as random-access memory (RAM), read-only memory (ROM), magnetic or optical disk, or other non-transitory media suitable for storing instructions for execution by the processor.

As used in the description herein and throughout the claims that follow, “a”, “an”, and “the” includes plural references unless the context clearly dictates otherwise. Also, as used in the description herein and throughout the claims that follow, the meaning of “in” includes “in” and “on” unless the context clearly dictates otherwise.

Thus, while particular embodiments have been described herein, latitudes of modification, various changes, and substitutions are intended in the foregoing disclosures, and it will be appreciated that in some instances some features of particular embodiments will be employed without a corresponding use of other features without departing from the scope and spirit as set forth. Therefore, many modifications may be made to adapt a particular situation or material to the essential scope and spirit.