Patent ID: 12231772

DETAILED DESCRIPTION

Definitions

Sharpness score: the gradients (dx, dy) of the image are compared (through subtraction) to the gradients of its low pass filtered version. A higher difference indicates a sharper original image. The result of this comparison is normalized with respect to the average variations (for example, sum of absolute gradients) of the original image, to obtain an absolute sharpness score.

Edge score: for each image, the edges are found (for example, using Canny edge detection) and the average intensity of gradients along them is calculated, for example, by calculating the magnitude of gradients (dx, dy) for each edge pixel, summing the results and dividing by the total number of edge pixels. The result is the edge score.

Effective resolution score: this score is calculated only in a region of interest (ROI) and provides a good indication of the effective resolution level in the image. As used herein, “ROI” is a user-defined sub-region of the image that may be exemplarily 4% or less of the image area. The effective resolution score can be derived from a combination of the sharpness scores and edge scores for each image, for example by normalizing both to be between [0, 1] and by taking their average.

FIG.1Ashows schematically a block diagram illustrating an exemplary embodiment of a dual-aperture zoom imaging system (also referred to simply as “dual-camera” or “dual-aperture camera”) disclosed herein and numbered100. Dual-aperture camera100comprises a Wide imaging section (“Wide camera”) that includes a Wide lens block102, a Wide image sensor104and a Wide image processor106. Dual-aperture camera100further comprises a Tele imaging section (“Tele camera”) that includes a Tele lens block108, a Tele image sensor110and a Tele image processor112. The image sensors may be physically separate or may be part of a single larger image sensor. The Wide sensor pixel size can be equal to or different from the Tele sensor pixel size. Dual-aperture camera100further comprises a camera fusion processing core (also referred to as “controller”)114that includes a sensor control module116, a user control module118, a video processing module126and a capture processing module128, all operationally coupled to sensor control block110. User control module118comprises an operational mode function120, a ROI function122and a zoom factor (ZF) function124.

Sensor control module116is connected to the two (Wide and Tele) cameras and to the user control module118and used to choose, according to the zoom factor, which of the sensors is operational and to control the exposure mechanism and the sensor readout. Mode choice function120is used for choosing capture/video modes. ROI function122is used to choose a region of interest. The ROI is the region on which both cameras are focused on. Zoom factor function124is used to choose a zoom factor. Video processing module126is connected to mode choice function120and used for video processing. It is configurable to evaluate a no-switching criterion determined by inputs from both Wide and Tele image data and to make a decision regarding video output. Specifically, upon evaluation of a no-switching criterion, if the no-switching criterion is fulfilled, module126is configurable to output a zoom video output image that includes only Wide image data in a zoom-in operation between a lower zoom factor (ZF) value and a higher ZF value. If the no-switching criterion is not fulfilled, module126is configurable to combine in still mode, at a predefined range of ZF values, at least some of the Wide and Tele image data to provide a fused output image of the object or scene from a particular point of view. Still processing module128is connected to the mode choice function120and used for high image quality still mode images. The video processing module is applied when the user desires to shoot in video mode. The capture processing module is applied when the user wishes to shoot still pictures.

FIG.1Bis a schematic mechanical diagram of the dual-aperture zoom imaging system ofFIG.1A. Exemplary dimensions: Wide lens TTL=4.2 mm and EFL=3.5 mm; Tele lens TTL=6 mm and EFL=7 mm; both Wide and Tele sensors ⅓ inch; external dimensions of Wide and Tele cameras: width (w) and length (l)=8.5 mm and height (h)=6.8 mm; distance “d” between camera centers=10 mm.

Following is a detailed description and examples of different methods of use of dual-aperture camera100.

Still Mode Operation/Function

In still camera mode, the obtained image is fused from information obtained by both cameras at all zoom levels, seeFIG.2, which shows a Wide sensor202and a Tele sensor204and their respective FOVs. Exemplarily, as shown, the Tele sensor FOV is half the Wide sensor FOV. The still camera mode processing includes two stages: the first stage includes setting HW settings and configuration, where a first objective is to control the sensors in such a way that matching FOVs in both images (Tele and Wide) are scanned at the same time, a second objective is to control the relative exposures according to the lens properties, and a third objective is to minimize the required bandwidth from both sensors for the ISPs. The second stage includes image processing that fuses the Wide and the Tele images to achieve optical zoom, improves SNR and provides wide dynamic range.

FIG.3Ashows image line numbers vs. time for an image section captured by CMOS sensors. A fused image is obtained by line (row) scans of each image. To prevent matching FOVs in both sensors to be scanned at different times, a particular configuration is applied by the camera controller on both image sensors while keeping the same frame rate. The difference in FOV between the sensors determines the relationship between the rolling shutter time and the vertical blanking time for each sensor.

Video Mode Operation/Function

Smooth Transition

When a dual-aperture camera switches the camera output between cameras or points of view, a user will normally see a “jump” (discontinuous) image change. However, a change in the zoom factor for the same camera and POV is viewed as a continuous change. A “smooth transition” (ST) is a transition between cameras or POVs that minimizes the jump effect. This may include matching the position, scale, brightness and color of the output image before and after the transition. However, an entire image position matching between the camera outputs is in many cases impossible, because parallax causes the position shift to be dependent on the object distance. Therefore, in a smooth transition as disclosed herein, the position matching is achieved only in the ROI region while scale brightness and color are matched for the entire output image area.

Zoom-In and Zoom-Out in Video Mode

In video mode, sensor oversampling is used to enable continuous and smooth zoom experience. Processing is applied to eliminate the changes in the image during crossover from one camera to the other. Zoom from 1 to Zswitchis performed using the Wide sensor only. From Zswitchand on, it is performed mainly by the Tele sensor. To prevent “jumps” (roughness in the image), switching to the Tele image is done using a zoom factor which is a bit higher (Zswitch+ΔZoom) than Zswitch. ΔZoom is determined according to the system's properties and is different for cases where zoom-in is applied and cases where zoom-out is applied (ΔZoomin≠ΔZoomout). This is done to prevent residual jumps artifacts to be visible at a certain zoom factor. The switching between sensors, for an increasing zoom and for decreasing zoom, is done on a different zoom factor.

The zoom video mode operation includes two stages: (1) sensor control and configuration and (2) image processing. In the range from 1 to Zswitch, only the Wide sensor is operational, hence, power can be supplied only to this sensor. Similar conditions hold for a Wide AF mechanism. From Zswitch+ΔZoom to Zmaxonly the Tele sensor is operational, hence, power is supplied only to this sensor. Similarly, only the Tele sensor is operational and power is supplied only to it for a Tele AF mechanism. Another option is that the Tele sensor is operational and the Wide sensor is working in low frame rate. From Zswitchto Zswitch+ΔZoom, both sensors are operational.

Zoom-in: at low ZF up to slightly above ZFT(the zoom factor that enables switching between Wide and Tele outputs) the output image is the digitally zoomed, unchanged Wide camera output. ZFTis defined as follows:

ZFT=Tan⁡(FOVWide)/Tan⁡(FOVTele)
where Tan refers to “tangent”, while FOVWideand FOVTelerefer respectively to the Wide and Tele lens fields of view (in degrees). As used herein, the FOV is measured from the center axis to the corner of the sensor (i.e. half the angle of the normal definition). Switching cannot take place below ZFTand it can above it.

In some embodiments for the up-transfer ZF, as disclosed in co-invented and co-owned U.S. Pat. No. 9,185,291, the output is a transformed Tele camera output, where the transformation is performed by a global registration (GR) algorithm to achieve smooth transition. As used herein “global registration” refers to an action for which the inputs are the Wide and Tele images. The Wide image is cropped to display the same FOV as the Tele image. The Tele image is passed through a low pass filter (LPF) and resized to make its appearance as close as possible to the Wide image (lower resolution and same pixel count). The outputs of GR are corresponding feature point pairs in the images along with their disparities, and parameters for differences between the images, i.e. shift and scale. As used herein, “feature point” refers to a point such as points10a-dinFIG.3Band refers to a point (pixel) of interest on an object in an image. For purposes set forth in this description, a feature point should be reproducible and invariant to changes in image scale, noise and illumination. Such points usually lie on corners or other high-contrast regions of the object.

Stages of Global Registration

In some exemplary embodiments, global registration may be performed as follows:1. Find interest points (features) in each image separately by filtering it with, exemplarily, a Difference of Gaussians filter, and finding local extrema on the resulting image.2. Find feature correspondences (features in both images that describe the same point in space) in a “matching” process. These are also referred to as “feature pairs”, “correspondence pairs” or “matching pairs”. This is done by comparing each feature point from one (Tele or Wide) image (referred to hereinafter as “image 1”) to all feature points in that region from the other (respectively Wide or Tele) image (referred to hereinafter as “image 2”). The features are compared only within their group of minima/maxima, using patch normalized cross-correlation. As used herein, “patch” refers to a group of neighboring pixels around an origin pixel.3. The normalized cross correlation of two image patches t(x, y) and ƒ(x, y) is

1n⁢∑x,y⁢(f⁡(x,y)-f_)⁢(t⁡(x,y)-t_)σf⁢σt
where n is the number of pixels in both patches,ƒis the average of ƒ and σƒis the standard deviation of ƒ. A match for a feature point from image 1 is only confirmed if its correlation score is much higher (for example, x1.2) than the next-best matching feature from image 2.4. Find the disparity between each pair of corresponding features (also referred to as “matching pair”) by subtracting their x and y coordinate values.5. Filter bad matching points:a. Following the matching process, matches that include feature points from image 2 that were matched to more than one feature from image 1 are discarded.b. Matching pairs whose disparity is inconsistent with the other matching pairs are discarded. For example, if there is one corresponding pair which whose disparity is lower or higher than the others by 20 pixels.6. The localization accuracy for matched points from image 2 is refined by calculating a correlation of neighboring pixel patches from image 2 with the target patch (the patch around the current pixel (of the current matching pair) from image 1, modeling the results as a parabola and finding its maximum.7. Rotation and fine scale differences are calculated between the two images according to the matching points (for example, by subtracting the center of mass from each set of points, i.e. the part of the matching points belonging to either the Wide or the Tele image, and solving a least squares problem).8. After compensating for these differences, since the images were rectified, the disparity in the Y axis should be close to 0. Matching points that do not fit this criterion are discarded. A known rectification process is illustrated inFIG.3C.9. Finally, the remaining matching points are considered true and the disparities for them are calculated. A weighted average of the disparity is taken as the shift between both images. The maximum difference between disparity values is taken as the disparity range.10. At various stages during GR, if there are not enough feature/matching points remaining, the GR is stopped and returns a failure flag.

In addition, it is possible to find range calibration to the rectification process by finding the shiftI=shift for objects at infinity and defining shiftD=shift-shiftI and disparity D=disparity-shiftI. We then calculate

object⁢distance=focalLength·baselinedisparityD·pixelSize,
where “baseline” is the distance between cameras.

Returning now to the Zoom-in process, in some embodiments, for higher ZF than the up-transfer ZF the output is the transformed Tele camera output, digitally zoomed. However, in other embodiments for higher ZF than the up-transfer ZF there will be no switching from the Wide to the Tele camera output, i.e. the output will be from the Wide camera, digitally zoomed. This “no switching” process is described next.

No Switching

Switching from the Wide camera output to the transformed Tele camera output will be performed unless some special condition (criterion), determined based on inputs obtained from the two camera images, occurs. In other words, switching will not be performed only if at least one of the following no-switching criteria is fulfilled:1. if the shift calculated by GR is greater than a first threshold, for example 50 pixels.2. if the disparity range calculated by GR is greater than a second threshold, for example 20 pixels, because in this case there is no global shift correction that will suppress movement/jump for all objects distances (smooth transition is impossible for all objects).3. if the effective resolution score of the Tele image is lower than that of the Wide image. In this case, there is no point in performing the transition because no value (i.e. resolution) is gained. Smooth transition is possible but undesirable.4. if the GR fails, i.e. if the number of matching pairs found is less than a third threshold, for example 20 matching pairs.5. if, for example, that are imaged onto the overlap area are calculated to be closer than a first threshold distance, for example 30 cm, because this can result in a large image shift to obtain ST.6. if some objects (for example two objects) that are imaged in the overlap area are calculated to be closer than a second threshold distance, for example 50 cm, while other objects (for example two objects) are calculated to be farther than a third threshold distance for example 10 m. The reason is that the shift between an object position in the Wide and Tele cameras is object distance dependent, where the closer the objects the larger the shift, so an image containing significantly close and far objects cannot be matched by simple transformation (shift scale) to be similar and thus provide ST between cameras.

Zoom-out: at high ZF down to slightly below ZFT, the output image is the digitally zoomed transformed Tele camera output. For the down-transfer ZF, the output is a shifted Wide camera output, where the Wide shift correction is performed by the GR algorithm to achieve smooth transition, i.e. with no jump in the ROI region. For lower (than the down-transfer) ZF, the output is basically the down-transfer ZF output digitally zoomed but with gradually smaller Wide shift correction, until for ZF=1 the output is the unchanged Wide camera output.

Note that if a no-switching criterion is not fulfilled, then the camera will output without fusion continuous zoom video mode output images of the object or scene, each output image having a respective output resolution, the video output images being provided with a smooth transition when switching between the lower ZF value and the higher ZF value or vice versa, wherein at the lower ZF value the output resolution is determined by the Wide sensor, and wherein at the higher ZF value the output resolution is determined by the Tele sensor.

FIG.3Ashows an embodiment of a method disclosed herein for acquiring a zoom image in video/preview mode for 3 different zoom factor (ZF) ranges: (a) ZF range=1: Zswitch; (b) ZF range=Zswitch: Zswitch+ΔZoomin: and (c) Zoom factor range=Zswitch+ΔZoomin: Zmax. The description is with reference to a graph of effective resolution vs. zoom factor (FIG.4). In step302, sensor control module116chooses (directs) the sensor (Wide, Tele or both) to be operational. Specifically, if the ZF range=1: Zswitch, module116directs the Wide sensor to be operational and the Tele sensor to be non-operational. If the ZF range is Zswitch: Zswitch+ΔZoomin, module116directs both sensors to be operational and the zoom image is generated from the Wide sensor. If the ZF range is Zswitch+ΔZoomin: Zmax, module116directs the Wide sensor to be non-operational and the Tele sensor to be operational. After the sensor choice in step302, all following actions are performed in video processing core126. Optionally, in step304, color balance is calculated if two images are provided by the two sensors. Optionally yet, in step306, the calculated color balance is applied in one of the images (depending on the zoom factor). Further optionally, in step308, registration is performed between the Wide and Tele images to output a transformation coefficient. The transformation coefficient can be used to set an AF position in step310. In step312, an output of any of steps302-308is applied on one of the images (depending on the zoom factor) for image signal processing that may include denoising, demosaicing, sharpening, scaling, etc. In step314, the processed image is resampled according to the transformation coefficient, the requested ZF (obtained from zoom function124) and the output video resolution (for example 1080p). To avoid a transition point to be executed at the same ZF, ΔZoom can change while zooming in and while zooming out. This will result in hysteresis in the sensor switching point.

In more detail, for ZF range 1: Zswitch, for ZF<Zswitch, the Wide image data is transferred to the ISP in step312and resampled in step314. For ZF range=Zswitch: Zswitch+ΔZoomin, both sensors are operational and the zoom image is generated from the Wide sensor. The color balance is calculated for both images according to a given ROI. In addition, for a given ROI, registration is performed between the Wide and Tele images to output a transformation coefficient. The transformation coefficient is used to set an AF position. The transformation coefficient includes the translation between matching points in the two images. This translation can be measured in a number of pixels. Different translations will result in a different number of pixel movements between matching points in the images. This movement can be translated into depth and the depth can be translated into an AF position. This enables to set the AF position by only analyzing two images (Wide and Tele). The result is fast focusing.

Both color balance ratios and transformation coefficient are used in the ISP step. In parallel, the Wide image is processed to provide a processed image, followed by resampling. For ZF range=Zswitch+ΔZoomin: Zmaxand for Zoom factor>Zswitch,+ΔZoomin, the color balance calculated previously is now applied on the Tele image. The Tele image data is transferred to the ISP in step312and resampled in step314. To eliminate crossover artifacts and to enable smooth transition to the Tele image, the processed Tele image is resampled according to the transformation coefficient, the requested ZF (obtained from zoom function124) and the output video resolution (for example 1080p).

FIG.4shows the effective resolution as a function of the zoom factor for a zoom-in case and for a zoom-out case ΔZoomupis set when one zooms in, and ΔZoomdownis set when one zooms out. Setting ΔZoomupto be different from ΔZoomdownwill result in transition between the sensors to be performed at different zoom factor (“hysteresis”) when zoom-in is used and when zoom-out is used. This hysteresis phenomenon in the video mode results in smooth continuous zoom experience.

In conclusion, dual aperture optical zoom digital cameras and associate methods disclosed herein reduce the amount of processing resources, lower frame rate requirements, reduce power consumption, remove parallax artifacts and provide continuous focus (or provide loss of focus) when changing from Wide to Tele in video mode. They provide a dramatic reduction of the disparity range and avoid false registration in capture mode. They reduce image intensity differences and enable work with a single sensor bandwidth instead of two, as in known cameras.

All patent applications mentioned in this specification are herein incorporated in their entirety by reference into the specification, to the same extent as if each individual patent application was specifically and individually indicated to be incorporated herein by reference. In addition, citation or identification of any reference in this application shall not be construed as an admission that such reference is available as prior art to the present disclosure.

While this disclosure has been described in terms of certain embodiments and generally associated methods, alterations and permutations of the embodiments and methods will be apparent to those skilled in the art. The disclosure is to be understood as not limited by the specific embodiments described herein, but only by the scope of the appended claims.