Patent ID: 12231767

DESCRIPTION OF EMBODIMENTS OF THE INVENTION

A system1000for tuning a camera ISP of the embodiments of the invention is illustrated inFIG.2, which includes among other modules, an ISP100, a computer vision (CV) detection module200, and an optimizer module300.

The main focus of the optimization process in this patent application is on the parameters of various processing blocks of the ISP100as illustrated inFIG.2. Parameter optimization for manufacturing of optics and sensor hardware is outside of this patent application.

ISP Configuration104. The ISP100may include different components/modules performing different functions. Each component/module is responsible for a specific process to generate a final image. Typical modules of an ISP are a denoiser, demosaicker, white-balancer, color corrector, tone mapper, and JPEG compressor that are functions of some input parameters along with the image. We refer to such set of parameters as ISP configuration104, or ISP configuration set X104. The goal of the proposed optimization framework is to find an ISP configuration104that would result in a maximum performance for one or more target CV application or applications. The ISP configuration set X104is composed of L number of ISP parameters represented by
X=[x1,x2, . . . xL],  (1)
where each ISP parameter xlis bounded within a lower and higher value as z
xl∈[xllow,xlhigh],∀l∈[1,L](2)

ISP parameters are generally discrete or categorical but we map them to the continuous space in this formulation.

Raw Image Stack102. The optimization process of the ISP100requires a set of images captured and processed using the same imaging pipeline. Since the lens system and sensor are not part of the tuning procedure, a set of images102captured with the sensor but not pushed through the ISP are collected and used. This set is in fact a stack of N sensor raw images102, properly tagged/annotated with respect to the target CV task or tasks. The raw images102are processed by the ISP100for all evaluated ISP configurations104during the tuning process. This raw image stack102is denoted by
I(X)=[I1(X),I2(X), . . . ,IN(X)].  (3)

To benefit from the large amount of annotated data available for evaluation of various CV applications, the aforementioned RAW images can be generated via RAW image simulation using existing sensor simulation methods as described in “S. J. Kim, H. T. Lin, Z. Lu, S. Susstrunk, S. Lin, and M. S. Brown. A new in-camera imaging model for color computer vision and its application. IEEE Transaction on Pattern Analysis and Machine Intelligence, 34 (12):2289-2302, 2012.”

Computer Vision Module200. The raw image stack102processed by the ISP100is passed through the CV module200. The CV module200performs the ultimate task or tasks that the entire imaging pipeline and the vision system are designed for. Examples of such tasks can be face, object, text, etc. detection/recognition, human gait analysis, augmented reality, or image retrieval. CV modules are typically pre-trained or derived from some kind of generalized statistics and cannot be easily modified, tuned, and/or re-trained. The CV module200that takes the ISP processed images106as inputs is denoted by CV(I(x)).

Measure of Performance of the CV system. The results of the CV task are passed through evaluation metrics, i.e., key performance indicators (KPIs) mk108are measures of performance of the CV200, and are defined as
mk[x]=KPIk(CV(I(X))).  (4)

Since the CV output can be evaluated using more than one KPI, output of each KPI is indexed as mk(∀k∈[1, K]) in Eq. (4). For instance, in case of object detection, common KPIs include Accuracy, Precision, Recall, Mean-Average-Precision (MAP), Mean-Average-Recall (MAR), Panoptic Quality (PQ) in case of segmentation, or peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM).

The expected and acceptable lower-bound (mklow) and upper-bound (mkhigh) of the range of the KPI target needs to be defined in the tuning process as
mkTarget:mk∈[mklow,mkhigh].  (5)

The choice of this range depends on what we expect from an ISP. For example, for a mid-end ISP, a PSNR range i.e., m1∈[15, 30] would be a reasonable choice.

Metric-Target Loss Function Formulation. The goal of the proposed system1000for tuning the ISP100is to find ISP parameters (and hence a particular configuration or configurations104) that would bring all the CV KPIs mk108to near the KPI target ranges [mklow, mkhigh]. This may be formulated in the form of a loss function as

f⁡(mk;X)={wk⁢❘"\[LeftBracketingBar]"mkhigh-mk[X]❘"\[RightBracketingBar]"nkif⁢mk[X]>mkhighwk⁢❘"\[LeftBracketingBar]"mk[X]-mklow❘"\[RightBracketingBar]"nkif⁢mk[X]<mklow0else,(6)
where wkand nkdenote a scalar weight, and an exponent associated to the metric mk, respectively. With a loss function, a lower value indicates a better result.

We use multiple metrics for the CV task to evaluate it in the ISP tuning.

Hence, the overall goal can be seen as optimizing a multi-objective problem formulated as
minimizeX(ƒ(m1;X), . . . ,ƒ(mK;X)),s·t·xl∈[xllow,xlhigh],∀l∈[1,L].(7)

For example, a camera ISP may have multiple parameters (for example tens of parameters) often not orthogonal with regard to each other. Also, operations performed inside an ISP100are not necessarily linear and may not follow a differentiable functionality. Therefore, the optimization problem (7) is generally non-linear and lacks close-form gradients. Additionally, except for some range of ISP parameters, the ISP operations themselves are not entirely known to a user who tries to solve a problem a black-box optimization. Such specifications of the ISP tuning problem and the foreseeable rugged search landscape (e.g., local optima, outliers, discontinuities, sharp bends, noise) make applications of quasi-Newton or conjugate gradient category of approaches, unusable in solving optimization problem (7).

ISP Optimization Framework for CV (System1000)

FIG.2presents an overview of the ISP optimization framework (system1000) of the embodiments of the invention for tuning the ISP100for an arbitrary CV task. It begins with a set of raw images102fed to the ISP100, and the processed images106after the ISP100are fed to the CV module200.

The performance of the CV task is measured given the ISP processed images106and provided to an optimizer300. The optimizer300iteratively improves the ISP parameters104given the measured CV performance, for example the KPIs mk108.

The Optimizer300is shown in greater detail inFIG.3. The Optimizer300has at least Search Space Modification Module302, including Latin HyperCube Sampling Module (Algorithm2)304and another module306for Search Space Reduction and Selection of ISP Parameters; Re-Mapping Module308; and Global Optimization Module (Algorithm2)310, among other features described in the text below.

The system1000and corresponding modules include a memory device having computer readable instructions stored thereon for execution by a processor312.

We propose an approach to this non-linear non-convex black-box optimization problem based on CMA-ES (Covariance Matrix Adaptation Evolution Strategy), described in “N. Hansen, A. Ostermeier, Completely derandomized self-adaptation in evolution strategies, Evolutionary Computation 9 (2001) 159-195.” The CMA-ES-like optimizer is a suitable tool for the high-dimensional search space that the tuning process has to deal with. Also, it has been shown that it can be adapted to discrete variable space, described in “E. Benhamou, J. Atif, R. Laraki, A discrete version of cma-es, 2018.” One can consider CMA-ES as a second-order iterative approach by estimating a positive covariance matrix on convex-quadratic functions. This matrix is closely related to the inverse Hessian as described in “N. Hansen, A. Ostermeier, Completely derandomized self-adaptation in evolution strategies, Evolutionary Computation 9 (2001) 159-195” and “N. Hansen, Benchmarking a BI-population CMA-ES on the BBOB-2009 function testbed, in: Workshop Proceedings of the GECCO Genetic and Evolutionary Computation Conference, ACM, 2009, pp. 2389-2395.” This makes the method feasible for tuning camera ISP parameters which can be considered as a non-separable and badly conditioned problem.

The method for tuning an ISP occurs inside a memory200with the aid of a processor312. The method steps are shown inFIG.2A, and the combined method and system are shown inFIG.2B. The first step of capturing or simulating a set of raw images901, occurs in the RAW image stack102. The raw images are then processed by the ISP100, in which the step of processing the raw images by the ISP903occurs. The processed images I(X)106are then supplied to the CV200, where the step of supplying the processed images to the CV system905occurs. KPIs m1, m2, . . . , mk108are then used for measuring the performance of the CV system907. Tuning the configuration parameters of the ISP909occurs in the optimizer300and ISP configuration X104.

Algorithm 1 summarizes the steps of the proposed global optimization module310performed by the processor312. This is an evolutionary algorithm whose number of trials (population) and total number of iterations is defined as λ and Niter, respectively. In the algorithm, Xlowand Xhighdenote a set of lower-bounds and higher-bounds of all the ISP parameters as Xlow=[x1low, . . . , xLlow] and Xhigh=[x1high, . . . , xLhigh], respectively. Prior to the main optimization process, i.e., Algorithm 1, these ranges are modified through a search space reduction technique (SSRT). We refer to these modified parameter ranges as current parameter ranges and denote them bylow=[1low, . . . ,Llow],high=[1high, . . . ,Lhigh].

Corresponding search space bounds, normalized in the solver's acceptable range are denoted by Xlowand Xhigh. We also require to keep track of the KPI measurements and the loss function (objective) throughout the iterations. We denote them by M=[m1[X1], . . . , mK[X1]; . . . ; m1[Xλ], . . . , mK[Xλ]] and F=[ƒ(m1; X1), . . . , ƒ(m1; Xλ); . . . ; ƒ(mK; X1), . . . , ƒ(mK; Xλ)], respectively, and use them to calculate fitness Eq. (6), in module318.

Algorithm 1: Proposed ISP parameter optimizer.Require: Niter, λ, Xlow, Xhigh, {tilde over (X)}low, {tilde over (X)}high, P(0), M(0), F(0), Xlow, Xhigh1: X(0)← initialize2: t ← 13: while stop criterion is not satisfied & t ≤ Niterdo4:  for j = 1 to λ do5:   Pj← generate population6:   I(Pj) ← run the ISP for Pj(3)7:   Mj← m1[Pj], ..., mK[Pj](4)8:   Fj← [f(m1; Pj), ..., f(mK; Pj)] (6)9:  end for10:  Update CMA-ES11: t ← t + 112: end while13: X ← X(t)14: return X

Initialization314

Let a trial be a combination of ISP parameters for which the ISP output images and the corresponding CV metrics have been computed, so that its performance with respect to the target CV tasks can be compared to other such ISP parameter combinations. Thus, P(0)is defined as an initial set of trials provided to the optimizer whose objective components are denoted by F(0). We perform a max-rank strategy to initialize an estimate of the optimal solution, for at least some of the trials, or alternatively for all of the trials in the initial set of trials. Let superscript (0) denote the state before iterations begin, we first rank all KPI values per trial and find the maximum among them as Max-Rank(Fj(0)). Then, we assign each element of the initial estimate X(0)as Xl(0)=Pj,l(0)where ĵ is found as
j=arg minj,∀j∈[1,λ][Max-Rank(Fj(0))],s·t·Pj,l(0)∈[llow,lhigh].  (8)

Max-Rank(.) returns max-rank-loss which is the method of computing multi-objective Chebyshev scalarization from the ranks corresponding to KPI values (a lower rank indicating a better KPI value) instead of the KPI or loss values themselves. A description of the Chebyshev scalarization is found in “Michael T. Emmerich and André H. Deutz. A tutorial on multiobjective optimization: Fundamentals and evolutionary methods. 17 (3):585-609, September 2018.” Algorithm 1b illustrates Max-Rank(⋅), the computation of the weighted max-rank loss for a population denoted by P, weights is a vector of weights, one per loss. The losses, K of them per trial, are stored in FP. In lines3to5, one loss at a time, the loss values are ranked across all trials, and the ranks are then associated with the corresponding trial. In line8, for each trial, the ranks are multiplied by the weight corresponding to the loss they were computed from, and the maximum of these weighted ranks for one trial is the max-rank loss for the trial.

Algorithm 1b: Weighted Max-Rank Loss ScalarizationRequire: weights, P, FP1: λ ← total number of trials in P2: for k = 1 to K do3:  for t = 1 to λ do4:   rank[t,k] ← rank of FP[t,k] within { FP[s,k], s=1:λ }5:  end for6: end for7: for t = 1 to λ do8:  max-rank-loss[t] ← max { weights[k] · rank[t,k], k=1:K }9: end for10: return max-rank-loss

FIGS.3,3A,3B,3Cand parts ofFIGS.3G and3Hshow the method and system implementing Algorithm 1, which includes the initialization of an initial set of estimates of the configuration parameters, which occurs inside the initialization module314of the global optimization module310. The step of retrieving an initial set of trials (population)1001aoccurs inside the block for retrieving an initial set of trials (population)1001b, which in turn occurs in the repository module314-1, while the ranking of key performance indicators (KPIs)1003afor indicating the performance of the CV system occurs inside the block for the ranking of key performance indicators (KPIs)1003b, which occurs in the max-ranking module314-2. The method and system implementing Algorithm 1b is shown inFIGS.3C,3Dand parts ofFIGS.3M and3N. Finally, determining those configuration of parameters corresponding to top ranked KPIs1005a, and initializing the X(0) estimates, occurs inside the block for initializing the X(0)1005b, which occurs inside the assignment module314-3.

Generating Population316and Re-Mapping308via Mirroring

A normal distribution with mean equal to the current estimate and a covariance matrix (σ(t))2C(t)is evolved during the iterations of the optimizer. Consequently, at every iteration λ number of parameter sets denoted by P are generated as
Pj=(X(t),(σ(t))2C(t))),∀j∈[1,λ]
in module316where
(·)
is a multivariate normal distribution.

Depending on the value of σ, the population generated by the normal distribution can lie well beyond the bounds defined by the SSRT. The generated population then needs to be remapped, in module308, back to the search bounds Xlow, Xhigh.

FIGS.3,3E,3Fand parts ofFIG.3Gshow the method and system implementing Algorithm 1, which includes generating the population of configuration parameters, which occurs inside the population generation module316of the global optimization module310. The steps of generating the population Pj1101aand running the ISP for Pj=I(Pj)1103aoccur respectively in the blocks for generating the population Pj1101band the block for running the ISP for Pj=I(Pj)1103b, which both are situated in the parameter set generator module316-1, which is shown inFIGS.3E,3Fand part ofFIGS.3G and3H. The step of tracking KPI and loss function Mj1105aoccurs inside the block for tracking KPI and loss function Mj1105b, which in turn occurs inside the tracking module316-2.

Instead of clipping the population at the boundary, the population outside of the search bounds is mirrored back about the bounds. This is done as otherwise for large σ a lot of points would be at the boundary of the search bounds.

In the embodiments of the present invention, re-mapping of the generated population to the reduced search space via mirroring is performed. The mirroring based remapping is shown inFIGS.4A,4B and4Cfor small to large standard-deviation of the generated population. The large value of σ leads to a lot of points away from the mean of the distribution, resulting in a lot of exploration of the search space. This is useful at the start of the optimization process when a large portion of search space is unknown to the optimizer. For small σ, most of the points lie near the mean of the distribution resulting in local exploration of ISP parameter space. This is particularly useful during the final convergence. Finally, the generated population is mapped to the original ISP parameter space i.e.,
Pj,l=llow+Pj,l(lhigh−llow),∀l∈[1,L] and ∀j∈[1,λ].

InFIGS.4A,4B and4C, from left to right, the distribution of the original population is σ=0.05, σ=0.3, and σ=2.0, respectively with the mean equal to 1.6. Red histogram is the remapped distribution of the original population shown in blue.

The operation of the re-mapping module308is illustrated by a flow-chart ofFIG.4D.

Updating CMA-ES (Covariance Matrix Adaptation Evolution Strategy)

The results of building a multi-objective loss function with regard to the measure of performance done by tracking KPIs and loss function Mj1005in the tracking module316-2, are then used for calculating the fitness Fj1200ain the block for calculating fitness Fj1200b, which in turn occurs in the fitness calculation module318, shown inFIG.3Eand parts ofFIGS.3G and3H.

The results of the fitness calculation are then used to apply an evolutionary algorithm (having a number of trials or population) to the multi-objective loss function (6) to determine the configuration parameters of the ISP as they were tuned, which thereby results in a global optimization of at least two or more modules of the ISP simultaneously. A counter3000ensures that λ trials are completed.

This step involves updating mean, covariance and other parameters of CMA-ES based on the fitness values found in the current iteration. This step involves updating the CMAES1250ainFIG.3Gand the block for updating the CMAES1250binFIG.3H. Refer to “N. Hansen, Benchmarking a BI-population CMA-ES on the BBOB-2009 function testbed, in: Workshop Proceedings of the GECCO Genetic and Evolutionary Computation Conference, ACM, 2009, pp. 2389-2395” for more details. The fitness for each trial is first ranked. Then, the maximum rank is used as the final fitness of the trial. Finally, X is returned1107ain the block for returning X1107b(FIGS.3G and3H).

Algorithm 2: Latin Hyper-cube (LH) sampling.Require: Niter, λ, Xlow, Xhigh1: LH ← generate LH2: for j = 1 to λ do3:  I(LHj) ← run the ISP for LHj(3)4:  Mj← m1[LHj],...,mK[LHj] (4)5:  Fj← [f(m1; Pj),...,f(mK; Pj)] (6)6: end for7: FLH← F8: MLH← M9: return LH, FLH, MLH

FIGS.3I,3K and3Lshows the method and system implementing the Algorithm 2, which is performed in the LH generation module320of the Latin Hyper-cube sampling module304, used for reducing a number of combinations of the configuration parameters of the ISP (ISP sets). The steps of generating the population LH1401aand running the ISP for Pj=I(Pj)1403aoccur respectively in the blocks for generating the population Pj1401band the block for running the ISP for Pj=I(Pj)1403b, which both are situated in the LH parameter set generator module320-1. The step of tracking KPI and loss function Mj (LH)1405aoccurs inside the block for tracking KPI and loss function Mj1405b, which in turn occurs inside the LH tracking module320-2.

The results of building a multi-objective loss function with regard to the measure of performance done by tracking KPIs and loss function Mj (LH)1405ain the LH tracking module320-2, are then used for calculating the fitness Fj1500ain the block for calculating fitness Fj1500b, which in turn occurs in the LH fitness calculation module322. A LH counter3001ensures that λ trials are completed, after which LH data is returned1550a(in block for returning LH data1550b). These steps are shown inFIG.3Iand parts ofFIGS.3K and3L.

Reducing Search Space (Module302)

We perform a Latin Hyper-cube (LH) sampling (module304) in order to save some processing time when running the optimizer. The Latin Hyper-cube sampling generates combinations of ISP parameters such that each parameter's range is well sampled and such that sampled pairs of parameters are uncorrelated, as described in “M. D. McKay, R. J. Beckman, W. J. Conover, Comparison of three methods for selecting values of input variables in the analysis of output from a computer code, Technometrics 21 (1979) 239-245” and “T. Torsney-Weir, A. Saad, T. Moller, H.-C. Hege, B. Weber, J.-M. Verbavatz, S. Bergner, Tuner: Principled parameter finding for image segmentation algorithms using visual response surface exploration, IEEE Transactions on Visualization and Computer Graphics 17 (2011) 1892-1901.”

FIG.3Jshow the method of reducing the number of combinations of the configuration parameters of the ISP. The first step is sampling a Latin Hyper-Cube space1301with regard to the configuration parameters of the ISP, then measuring the performance of the CV for sampled sets1303, in other words for the ISP sets sampled in the step1301. The final step is selecting ranges1305for configuration parameters based on those ISP sets which result in the measure of performance of the CV system above a predetermined threshold.

Algorithm 2 presents the procedure to generate LH samples with regard to the ISP parameters. It should be noted that nested hyper cube samples are generated in step 1 in module320. For the first hyper cube, the center of distribution is defined as cl=(xSPl−xllow)/(xlhigh−xllow), ∀l∈[1, L] where xISPldenotes the default parameter value that comes with the ISP. These default values can be the parameters recommended by the ISP manufacturer, or a set of parameters tuned for IQ purposes, or even a set of parameters loosely hand-tuned by a user. For each sample of the LH we need to calculate the fitness in module322which is then consumed by a SSRT.

Algorithm 3: The search space reduction technique (SSRT)Require: pgood, plow, phigh, threshold, nref, K, L, LH, FLH1: λ ← total number of trials in LH2: Ngood← max(2, ceiling(λpgood))3: Nlow← max(2, ceiling(λplow))4: Nhigh← max(2, ceiling(λphigh))5: for l = 1 to L do6:  for k = 1 to K do7:   lossmax← max { FLH[t,k], t=1:λ }8:   if lossmax< threshold9:    weight[k] ← 010:   else11:    for t = 1 to λ do12:     rank[t, k] ← rank of FLH[t,k] within { FLH[s,k], s=1:λ }13:    end for14:    low_rank_vals ← { t | rank[t, k] <= Nlow}15:    high_rank_vals ← { t | rank[t, k] > Nhigh}16:    distance ← supremum( | CDF(low_rank_vals) −CDF(high_rank_vals) | )17:   weight[k] ← KSCDF( distance {square root over (0.5nref)} )18:  end if19: end for20: for t = 1 to λ do21:  max-rank-loss[t] ← max { weight[k] * rank[t,k], k=1:K }22: end for23: S ← argsort{ max-rank-loss[t], t=1:λ }24: B ← S[1:N_good]25: {tilde over (X)}llow← min B26: {tilde over (X)}lhigh← max B27: {tilde over (X)}lbest← S[1]28: end for29: return {tilde over (X)}low, {tilde over (X)}high, {tilde over (X)}best

We then modify the search space for each parameter by applying a SSRT in module306. The steps of SSRT are shown in Algorithm 3.

A key functionality of Algorithm 3 is that, for each parameter and KPI, a non-negative weight that quantifies whether the value of the parameter has an impact on the value of the loss associated with the KPI, is estimated. This weight should be low when the impact of the parameter on the loss is small, and high when the impact is significant. For each parameter-KPI pair, such a weight can be obtained by comparing the distribution of the values of the parameter for the trials that give the best (that is, lowest) loss values loss that corresponds to the KPI, to the distribution of the values for the trials that give the worst (that is, highest) loss values. plowis the proportion of the trials of the population (generated by Latin Hypercube sampling, for example) which is identified, through ranking, as having low (good) loss values for one KPI. Similarly, phighis the proportion of trials which is identified as having high (bad) loss values.

pgoodis an analogous quantity which is, however, used differently: Once the trials have been ranked based on weighted max-rank loss values (this weighted max-rank loss involving all KPIs, with the weights computed for the parameter under consideration), pgoodis the proportion of the trials, specifically those with lowest weighted max-rank, from which we extract parameter values that the reduced search range for the parameter under consideration must contain. (The smallest containing interval is used.) Reducing pgoodleads to more aggressive but less stable search space reduction (stability is restored with additional sampling).

Values of p_good ranging from 2 to 30% have been used successfully. plowand phighcan generally be set to the same value. as pgood. Alternatively, 5%-15% of the best trials may be used, or yet alternatively, 10%-30% of the best trials may be used if required. It is understood that yet another percentage range for selecting best trials may be chosen depending on how much the modified search range needs to be narrowed.

The threshold is a parameter that is used to perform a very simple test of significance. The value of threshold should be set to a loss value which corresponds to KPI values considered satisfactory, in fact, considered equally good. For example, when losses that have been derived from KPIs for which the value 0 is good enough in exact arithmetic, threshold can be set to a small multiple of machine epsilon. Other threshold values, possibly different for each KPI, can be used. Clearly, ranking trials based on a KPI that always “passes requirements” within the sample would be pointless. Consequently, in Algorithm 3, if all the loss values for a KPI are below the threshold, the parameter is deemed to have insignificant impact on loss values, and the corresponding weight is set to 0 in the weighted max-rank loss used to narrow the parameter's search range.

One way to compute a weight quantifying the impact of a parameter on a KPI is to use a p-value. Specifically, 1 minus the p_value (or any non-negative function of the p-value that has a negative slope) can be used to construct a useful weight. For many tests, the p-value is equal to 1 minus the value of a cumulative distribution function (CDF) evaluated over a key statistic. Consequently, any non-negative function of the underlying CDF that has a positive slope can be used as a weight. Algorithm 3 shows a computation based on the Kolgomorov-Smirnov CDF (named KSCDF) of the two sample Kolgomorov-Smirnov distribution comparison test with small number of observations, as described in “James Durbin. Distribution Theory for Tests Based on the Sample Distribution Function. SIAM, 1973.” nrefis a normalization parameter making explicit the number of observations parameterizing the test (nref=10 works well in practice). Because the p-value-based weight is computed directly from the CDF, the p-value itself does not explicitly appear in Algorithm 3.

Other non-parametric two-sample distribution comparison test CDFs than Kolgomorov-Smirnov can be used to compute the weight, for example those of Anderson-Darling, as described in “Scholz, Fritz W., and Michael A. Stephens. K-sample Anderson-Darling tests. Journal of the American Statistical Association 82.399 (1987): 918-924” or in “Kuiper, Nicolaas H. Tests concerning random points on a circle. Nederl. Akad. Wetensch. Proc. Ser. A. Vol. 63. No. 1. 1960.” Instead of using the Kolgomorov-Smirnov CDF one can use the statistic directly (distance in Algorithm 3) to compute weights. Another alternative to using weights derived from p-values is to only keep KPIs for which a two sample comparison test rejects the hypothesis of equality of distribution of parameter values between the set of trials performing well with respect to that KPI and the set of trials performing worse with respect to that KPI. For instance one can use a two-sample Kolgomorov-Smirnov test or a Pearson's chi-squared test, as described in “Plackett, Robin L. Karl Pearson and the chi-squared test. International Statistical Review/Revue Internationale de Statistique (1983): 59-72.” We argue that our approach is better. Because all the corresponding losses are used in the computation of the max-rank loss, albeit with different weights, the computation of these narrowed configuration parameter search ranges is more stable (there is less variation in the results when the population than if KPIs are kept or discarded. The inferiority of all these alternative was verified by comparative testing.

Algorithm 3 shows the computation of the reduced search intervals [llow,lhigh] for all l. A good parameter value,lbest, is also returned for each l. Typically, pgood, plow, and phighare set to the same value, between 0.02 (2%) and 0.1 (10%), and nrefis set to 10. First, one has to determine whether a KPI is significant. In line9, the maximum value of the k-th loss over all the trials is computed. If this maximum loss value is small enough, threshold being a loss value considered acceptable in all cases, the corresponding KPI is ignored in the rest of the process. These steps correspond to the hypothesis-test. In lines11to17, the weight of each loss for the parameter under consideration is computed. First (lines11to13), the rank of each trial with respect to the k-th loss is computed. (This rank is the same for all parameters, and consequently it can be computed exactly once.) In line14, the values of the parameter under consideration for the Nlowbest ranked trials are gathered. In line15, we gather the values of the parameter for the Nhighworst ranked trials. In line16, the L-infinity (max) distance between the CDFs of the two groups of parameter values is computed for each loss. This distance is normalized, and the weight is then set to the corresponding large sample Kolgomorov-Smirnov Cumulative Distribution Function (KSCDF) value (line17). This completes the computation of each loss' weight for the parameter under consideration. The max-rank loss, weighted this time, is then computed for each trial (lines20to22). Lines23and24identify the Ngoodparameter values with best weighted max-rank loss, and the narrowed search interval for the ISP parameter under consideration is set to range from their minimum to their maximum. The parameter value with very best max-rank loss is also returned for each ISP parameter. This will provide a modified range of parameters aslow,highin module326.

An advantage of the above-mentioned steps is that the computations in the global optimization module310converge to an optimal solution quickly and accurately. The global optimization module310works also independently from the SSRT module302and re-mapping module308, especially when the initial configuration is close to the optimal one.

FIGS.3,3C,3D,3M and3Nshow the method and system implementing the Algorithm 3. After calculating Ngood, Nlow, Nhightrials1700ain the block for calculating Ngood, Nlow, Nhightrials1700b, the next step is determining whether a KPI is significant by calculating maximum loss1701ain the block for calculating maximum loss1701b, i.e, if this maximum loss value is small enough, threshold being a loss value considered acceptable in all cases, the corresponding KPI is ignored in the step1703ain the block1703b. The rank of each trial with respect to the k-th loss is computed by calculating rank1705ain the block for calculating rank1705b, and then the weight of each loss for the parameter under consideration is the step for computing weights1707ain the block for computing weights1707b. The computing the (weighted) max-rank loss1709ais done in block for computing (weighted) max-rank loss1709bfor each trial. All of these steps are performed in the loss rank module324, which is situated in the SSRT module306. Identifying the Ngoodparameter values with best weighted max-rank loss, and the narrowed search interval for the ISP parameter under consideration is set to range from their minimum to their maximum in the step for calculating range modification1711ain the block for calculating range modification1711b. Finally determining a modified search space1713acorresponding to the ranges for those configuration parameters is done in the block for determining a modified search space1713b. A new low and high range, and current best values of the configuration parameters are returned in the step for returning a new low and high range, and current best values1715ain the block for returning a new low and high range, and current best values1715b. These steps are performed in the modify range module326, which is situated in the SSRT module306.

Embodiment for ISP Tuning for Object Detection

Consider a traditional ISP that feeds a CV module can consist of a series of signal processing components. These components vary from one imaging system to another. However, a typical set of components common to all ISPs are black level adjustment, demosaicking, denoising, color corrections, tone mapping, etc. as shown inFIG.1. These essential ISP components along with the parameters associated with them represent the common functionality that impacts the performance of the CV.

FIG.5presents the proposed ISP optimization framework set up to tune the ISP100for an arbitrary object detection target task.

We choose the object detection method namely Faster-RCNN (with Resnet101 backend) which is presented in “S. Ren, K. He, R. Girshick, J. Sun, Faster R-CNN: Towards real-time object detection with region proposal networks, in: Advances in neural information processing systems (NeurIPS), pp. 91-99” to detect street objects. However, any other object detection algorithm/method can be used with the proposed ISP optimization framework.

A dataset of raw images annotated with regard to street objects is created. The evaluation of this CV task typically require two KPIs. We use2different objectives, MAP and MAR. Refer to “T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollar, C. L. Zitnick, Microsoft coco: Common objects in context, in: European conference on computer vision (ECCV), Springer, pp. 740-755” for the definitions of MAP (with IoU 0.5) and MAR (given 10 detections per image).

A list of different steps of the tuning process targeting Faster-RCNN object detection application is:1. Collect a dataset of RAW images annotated with regard to street objects.2. From the dataset a subset of images (e.g., N=100) are randomly selected as the tuning set.3. The tuning set was provided to the optimization framework set up for the target CV task, shown inFIG.5.4. The output of Algorithm 1 was used as the CV-tuned parameter set.

The process of ISP tuning is done in two main steps. First, a SSRT method is applied to reduce/determine the optimization search space as shown inFIG.6and explained above. Then, the main optimization process shown inFIG.7is applied given the reduced search space.

Hence, P(0), F(0), and M(0)inFIG.7are assigned with PLH, FLH, and MLHoutputs of the previous step (i.e.,FIG.7), respectively.

FIGS.8A,8B and8Cshow examples of Faster-RCNN object detection for a sample street scene for default ISP parameters, IQ-tuned parameters, and the CV-tuned parameters of the ISP respectively.

FIGS.9A,9B and9Cillustrate results of ISP optimization with real raw data for Faster-RCNN for object detection of another sample street scene, for default prior art ISP, IQ-tuned ISP, and CV-tunes ISP of the embodiment of the invention respectively.

In this exemplary tuning process, we set the weights of KPIs w1and w2, and their corresponding exponents n1, and n2to 2. The KPI range parameters were set as m1low=1, m2low=1, m1high=1 and m2high=1.

This results in fitness Eq. (6) for m1to be ƒ(m1=MAP; X)=(1−MAP(X))2and for m2to be ƒ(m2=MAR; X)=(1−MAR(X))2. This was done to equalize the effect of both MAP and MAR for tuning the ISP parameters. If for some other use-case the MAP score is preferred over the MAR score, the w1can be taken to be higher than w2and vice-versa if MAR is preferred. As both the MAP and MAR have values in similar range, the corresponding exponents are taken to be same. We empirically found λ=4[(4L/3)/4] (i.e., 4L/3 rounded up to the nearest multiple of 4) and λ=about 128L reasonable in Algorithm 1 and Algorithm 2, respectively.

Note that the processing blocks shown inFIG.6andFIG.7are adapted to the targeted object detection task with two KPIs. However, they can be adapted to other CV applications with no further modifications, by replacing the object detection processing block with other desired ones and using the corresponding KPIs.

As a supplementary example,FIGS.10,11and12show results of a face detection CV method applied using the default parameters, IQ-tuned parameters, and the CV-tuned parameters of the ISP where the CV-tuned parameters were obtained using the process shown inFIG.5.

FIGS.10A,10B and10Cillustrate results of ISP optimization with simulated raw data for a first example of face detection, for default ISP, IQ-tuned ISP, and CV-tunes ISP of the embodiment of the invention respectively;

FIGS.11A,11B and11Cillustrate results of ISP optimization with simulated raw data for a second example of face detection, for default ISP, IQ-tuned ISP, and CV-tunes ISP of the embodiment of the invention respectively; and

FIGS.12A,12B and12Cillustrate results of ISP optimization with simulated raw data for a third example of face detection, for default ISP, IQ-tuned ISP, and CV-tunes ISP of the embodiment of the invention respectively.

Advantages, Modifications, Variations

The use of SSRT enables the proposed system of tuning1000to tune all the blocks of the ISP at the same time which was not possible with the previously known ISP tuning methods. This allows for tuning to converge faster within smaller parameter ranges selected by applying SSRT on Latin-HyperCube generated samples. In the previously known automated ISP tuning methods as described in “Lin, Nishimura, Jun, et al. Automatic ISP Image Quality Tuning Using Nonlinear Optimization. IEEE International Conference on Image Processing (ICIP), 2018, pp. 2471-2475,” all the ISP blocks were assumed to be independent of each other enabling them to tune each block separately. This assumption generally led to poor or slower parameter convergence.The embodiments described above were presented targeting a specific object detection application. However, the proposed tuning framework of the present invention can be applied for any CV application including but not limited to segmentation, keypoint detection, image classification etc. by replacing the CV processing block with another suitable block. Depending on the type of CV application, a corresponding evaluation metrics can be used as the KPIs for the tuning. For example, for segmentation, a Panoptic-Quality-Metric can be used as the KPI.The current framework/system1000also allows tuning the ISP100for multiple CV applications at the same time. This is enabled by the use of multiple metrics at the same time that can be normalized in similar range by using equations (5) described above.We presented the embodiment as a series of optimization steps including SSRT as global optimization and CMA-ES as the local optimization. These steps can be replaced with other optimization methods, for example differential equation (DE), particle swarm optimization (PSO) etc. Depending on the complexity of the ISP optimization, steps can also include a fine tuning step like Nelder-Mead method at the end.The presented examples described above were used to tune the ISP component of the camera. It is understood that with proper modeling of the lens and sensor component, a similar CV tuning can be also done for design and operation of the lens and sensor.It should be noted that it is possible to by-pass the CV processing module and perform the tuning process for IQ (image quality) purposes. This can be done by removing object detection process from the processes shown inFIG.6andFIG.7, and applying IQ KPIs directly on the ISP output images. More formally, IQ tuning can be done by modifying equation (4) as
mk[x]=KPIk(I(X)).  (9)where KPIkdenotes an IQ metric such as SNR, SSIM, etc. adapted to a reference image.

Thus, the proposed ISP optimization method leads to a systematic adaptation of an ISP to the desired computer vision application. Such tuned ISP parameters are produced without awareness of the specific implementation of both ISP and the computer vision module in hours (using existing off the shelf computers, for example i7-8700 used in embodiments of the invention) compared to weeks of hand tuning by ISP experts. Our experimental results show that the performance of the computer vision task may be improved by up to a factor of 2 once the ISP is tuned using the proposed optimization framework1000compared with the same ISP is tuned for image quality. This is achieved with only a small amount of tuning data.

According to yet another embodiment of the present invention, there is also provided a system5000having a network5003having one or more nodes, for example node one5001an node two5002, which communicate over the network. Such a network5003is shown inFIG.13, where at least one of the nodes of the network, node one5001, comprises the computer vision system (CVS)1000.

Methods of the embodiment of the invention may be performed using one or more hardware processors, executing processor-executable instructions causing the hardware processors to implement the processes described above. Computer executable instructions may be stored in processor-readable storage media such as floppy disks, hard disks, optical disks, Flash ROMs (read only memories), non-volatile ROM, and RAM (random access memory). A variety of processors, such as microprocessors, digital signal processors, and gate arrays, may be employed.

Systems of the embodiments of the invention may be implemented as any of a variety of suitable circuitry, such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware or any combinations thereof. When modules of the systems of the embodiments of the invention are implemented partially or entirely in software, the modules contain a memory device for storing software instructions in a suitable, non-transitory computer-readable storage medium, and software instructions are executed in hardware using one or more processors to perform the methods of this disclosure.

It should be noted that methods and systems of the embodiments of the invention and data described above are not, in any sense, abstract or intangible. Instead, the data is necessarily presented in a digital form and stored in a physical data-storage computer-readable medium, such as an electronic memory, mass-storage device, or other physical, tangible, data-storage device and medium. It should also be noted that the currently described data-processing and data-storage methods cannot be carried out manually by a human analyst, because of the complexity and vast numbers of intermediate results generated for processing and analysis of even quite modest amounts of data. Instead, the methods described herein are necessarily carried out by electronic computing systems having processors on electronically or magnetically stored data, with the results of the data processing and data analysis digitally stored in one or more tangible, physical, data-storage devices and media.

Although specific embodiments of the invention have been described in detail, it should be understood that the described embodiments are intended to be illustrative and not restrictive. Various changes and modifications of the embodiments shown in the drawings and described in the specification may be made within the scope of the following claims without departing from the scope of the invention in its broader aspect.