Patent Description:
The subject matter disclosed herein relates in general to Macro images and in particular to methods for obtaining such images with multi-cameras (e.g. dual-cameras).

Multi-aperture digital cameras (or multi-cameras) are standard for mobile electronic devices (e.g. smartphones, tablets, etc.). A multi-camera usually comprises a camera with a wide field-of-view (or "angle") FOVw ("Wide" camera), and at least one additional camera, either with the same FOV (e.g. a depth auxiliary camera), with a narrower (than FOVw) field of view FOVT (Telephoto or "Tele" camera), or with an ultra-wide field of view FOVuw wider than FOVw ("UW camera"). In some embodiments below, a Wide or Ultra-Wide camera may be referred to as a "non-Macro camera".

A "Macro-photography" mode is becoming a popular differentiator for smartphone cameras. "Macro-photography" refers to photographing objects that are very close to the camera, so that the image size of an object recorded on the image sensor is nearly as large as the actual size of the object photographed, i.e. it has a large object : image magnification (M) of e.g. <NUM>: <NUM> to <NUM>:<NUM>. Such an image may be referred to as "Macro image". First smartphones that include a Macro camera with a Macro FOV (FOVM), often based on an UW camera have entered the consumer market. A Macro camera can be realized with a Tele camera. An advantage is the high magnification M of a Macro images captures with a Tele camera, so that one may speak of a "Super Macro Image". In some examples, the Macro camera may be a scanning Tele camera which can scan a scene with its native FOVM, for example as described in co-owned <CIT>.

Macro images are recorded at very small object-camera distances of around <NUM> or less. A UW camera may still be capable to focus to these small distances. Generally, very small objects (e.g., less than <NUM> in size) are targeted as image objects (i.e. are objects of interest or "OOIs"). Such very small OOIs for a Macro image may be occluded by the multi-camera hosting device, making it difficult to point the Macro camera to capture the OOI in a precise manner. This is especially significant when FOVM captures a small area (e.g. a few square millimeters) of an object (e.g., placed <NUM> to <NUM> away from the camera) while being integrated in a large device. It would be beneficial to have a method that supports a quick and user-friendly way to direct a Macro camera FOVM towards very small OOIs.

<CIT> discloses a mobile device with two cameras and a split screen, each half of the screen showing the image of one of the cameras.

In some embodiments there is provided a split screen view on a multi-camera hosting device showing a preview image segment of an ultra-wide camera FOV together with a preview image segment of the FOVM to support the targeting of an OOI with the FOVM.

The image segment of an ultra-wide camera FOV being displayed to the user may contain some differentiating element marking the area of the OOI that is to be captured by the Macro mode. Such differentiating element marking may include a touchable box, for example a rectangular box. The user may get a feedback on the screen in which direction the multi-camera hosting device is to be moved to catch the OOI with the FOVM.

In various embodiments there are provided mobile electronic devices, comprising a first camera with a first field of view FOV<NUM>, a second, Macro camera with a Macro field of view FOVM smaller than FOV<NUM> and a device screen that includes a first screen section configured to display first image data from the first camera and a second screen section configured to display second image data from the Macro camera when both cameras are focused to a distance equal to or smaller than <NUM>.

In some embodiments, the FOVM is shown and marked within the first screen section.

In some embodiments, the first screen section includes a visual indication for guiding a user of the mobile electronic device towards a scene using the FOVM.

In some embodiments, the second screen section displays first image data from the first camera.

In some embodiments, a device further comprises a controller for controlling a change in state of the Macro camera based on the first image data. In some embodiments, the state of the Macro camera is a zoom state. In some embodiments, the state of the Macro camera is a focus state.

In some embodiments, the first camera has a focal length between <NUM> and <NUM>.

In some embodiments, the Macro camera has a focal length between <NUM> and <NUM>.

In some embodiments, both cameras can be focussed to a distance smaller than <NUM>.

In some embodiments, both cameras can be focused to a distance of <NUM> or less.

In some embodiments, at least one camera can be focused to a distance between <NUM> and <NUM>.

In some embodiments, at least one camera can be focused to a distance of <NUM> or less.

In some embodiments, the first camera is an Ultra-Wide camera.

In some embodiments, the first camera is a Wide camera.

In some embodiments, the Macro camera is a scanning Tele camera. In some embodiments, the device further comprises a controller for controlling a change of a scan state of the Macro camera based on the first image data.

In some embodiments, the Macro camera is a Tele camera having different zoom states.

In some embodiments, the first camera is focused to a first distance different from a second distance that the Macro camera is focused to.

In some embodiments, the first screen section and the second screen section are split vertically when the mobile electronic device is held in a landscape orientation, and the first screen section and the second screen section are split horizontally when the mobile electronic device is held in a portrait orientation.

In some embodiments, the device is a smartphone.

In some embodiments, the device further comprises a controller for controlling a change of a scan state of the Macro camera based on the first image data.

In various embodiments there are provided methods, comprising providing a mobile electronic device that includes a first camera with a first field of view FOV<NUM>, a second, Macro camera with a Macro field of view FOVM smaller than FOV<NUM>, and a device screen, focusing the first camera and the Macro camera to a distance ≤ <NUM>, and displaying on a first section of the device screen first image data from the first camera and displaying on a second section of the device screen second image data from the second camera.

In some embodiments, the focusing of the first camera and of the Macro camera to a distance ≤ <NUM> includes focusing the first camera to a first distance ≤ <NUM> and focusing the Macro camera to a distance ≤ <NUM> different from the first distance.

In some embodiments, the focusing of the first camera and of the Macro camera to a distance ≤ <NUM> includes focusing both cameras to a distance of <NUM> or less.

In some embodiments, the focusing of the first camera and of the Macro camera to a distance ≤ <NUM> includes focusing both cameras or one of the cameras to a distance of <NUM> or less.

In some embodiments, a method further comprises showing and marking the FOVM within the first screen section.

In some embodiments, a method further comprises including in the first screen section a visual indication for guiding a user of the mobile electronic device towards a scene using the FOVM.

In some embodiments, a method further comprises displaying the first image data in the second screen section.

In some embodiments, a method further comprises controlling a change in state of the Macro camera based on the first image data.

In some embodiments, the controlling a change in state of the Macro camera includes controlling a change of a state selected from the group consisting of a scan state, a zoom state and a focus state of the Macro camera.

In some embodiments, the Macro camera is a scanning Tele camera and the controlling a change in state of the Macro camera based on the first image data includes automatically controlling a change of a scan state of the scanning Tele camera.

In some embodiments, the Macro camera is a Tele camera having different zoom states and the controlling a change in state of the Macro camera includes automatically controlling a change of a zoom state of the Tele camera.

Non-limiting examples of embodiments disclosed herein are described below with reference to figures attached hereto that are listed following this paragraph. The drawings and descriptions are meant to illuminate and clarify embodiments disclosed herein, and should not be considered limiting in any way. Like elements in different drawings may be indicated by like numerals. Elements in the drawings are not necessarily drawn to scale.

Embodiments disclosed herein solve the problem of occlusion of an object of interest when Macro-photography is performed with multi-cameras included in smartphones and other mobile electronic devices. For simplicity and for example only, the solution is illustrated with a dual-camera, with the understanding that it is also clearly applicable with multi-cameras having three or more cameras.

<FIG> illustrates a typical field of-view (FOV) ratio of smartphone <NUM> including a multi-camera (not shown) that includes a UW camera with a FOVuw <NUM> covering a large segment of a scene and a Macro camera with a FOVM <NUM> covering a small segment of a scene. One can see exemplary sizes and ratios between UW and Macro output images.

In some examples, the Macro camera may be a continuous Tele zoom camera where FOVM changes with changing ZF.

<FIG> illustrates an embodiment numbered <NUM> of a smartphone having a UW camera with a FOVuw <NUM> and a Macro camera with a FOVM <NUM> and a screen (display) <NUM> according to presently disclosed subject matter. A system description of smartphone <NUM> is given in <FIG>. As indicated by four arrows in <FIG>, smartphone <NUM> may be moved by a user in four or more directions for manually moving FOVM towards an OOI/ROI. Screen <NUM> illustrates a first screen example for displaying images of a multi-camera. Macro photography is required to capture small objects from a close range. The Macro camera may include a Tele lens with a focal length much larger (e.g., <NUM> times to <NUM> times) than the focal length of the UW camera. In such a case, the UW camera can easily focus to a close range (e.g. <NUM> to <NUM>), but its spatial resolution is poor since its focal length is small and its FOV is large. For example, consider a UW camera with <NUM> focal length and a Macro camera with <NUM> focal length. The two cameras may include identical or different sensors (e.g., with identical or different pixel count and pixel size). Assume that both cameras include the same sensor, e.g., with <NUM> active image sensor width. When focused to <NUM>, the Macro camera will have a M of <NUM>:<NUM> and will capture an object width of <NUM> (same as the sensor width). The UW camera will have a M of <NUM>:<NUM> and will capture an object width of <NUM>.

In some examples, for performing a method disclosed herein, a smartphone like smartphone <NUM> may comprise, instead of or additionally to the UW camera with FOVuw <NUM>, a W camera with a FOVw (not shown) that is smaller than FOVuw <NUM> but still larger than FOVM. In some examples, the W camera may not be able to focus to an object as close as e.g. <NUM>. In such examples, for performing a method disclosed herein, the W camera may be focused to its minimal focus distance, e.g. to <NUM>.

The UW or W cameras mentioned above have a larger depth of field than a Macro capable Tele camera with FOVM. A ROI is easier to detect in UW or W image data than in Macro image data. Therefore, one may use UW or W or M camera image data for automatic ROI detection and selection.

In an exemplary case, a user wishes to use smartphone <NUM> for capturing an OOI (e.g. a flower <NUM> which forms an image <NUM> in a camera) or a ROI with very high (Macro) resolution. For methods of use as disclosed herein, screen <NUM> is split into two sections, a first section <NUM> (possibly cropped) and a second section <NUM>. The first screen section may display first image data from the first camera with FOV<NUM> and the second screen section may display second image data from the second camera with FOVM. The examples here show a "split screen" view on the screen, i.e. the two screen sections are shown side by side. In other examples (not shown) one may display a "picture-in-picture" view on the screen, i.e. one screen section may be shown as an inlay in the other screen section. In some examples, the second screen section may be shown on the entire screen or on a large segment of the screen except on a segment where the first screen section is shown, and wherein the first screen section covers a smaller area on the screen than the second screen. In the embodiment of <FIG>, the two screen sections have exemplarily a "landscape" orientation, i.e. first screen section and the second screen section are split vertically, beneficial when the device is held in landscape orientation. Screen <NUM> may include additional icons or symbols as known (not shown). First screen section <NUM> displays a cropped FOVuw (and can be therefore called "UW screen section <NUM>") while second screen section <NUM> displays the Macro FOVM (and can be therefore called "Macro screen section <NUM>"). Inside first section <NUM>, FOVM <NUM> can be marked by a physical (i.e. visible on the screen) rectangle <NUM>'. Rectangle <NUM>' indicates the actual position of FOVM <NUM> with respect to FOVuw <NUM>. Optionally, another physical rectangle <NUM>" indicates a UW preview of OOI (flower) <NUM> in order to guide a user towards flower <NUM>. The user can see the position of FOVM <NUM> relative to rectangle <NUM>" at all times in UW screen section <NUM>, while at the same time, Macro screen section <NUM> exhibits the scene within FOVM. In use, the user moves the smartphone with camera towards flower <NUM>. In Macro screen section <NUM>, an arrow <NUM> indicates the direction and distance of movement required to align flower <NUM> with Macro FOVM <NUM>. In some examples, image data from UW FOV <NUM> may be displayed in Macro screen section <NUM>. This is for example beneficial when the Macro camera may not be in focus over entire camera FOVM <NUM> or when there are other optical or image quality issues.

<FIG>, <FIG> show the process of moving the smartphone (and the FOVM) towards flower <NUM> (i.e. toward preview <NUM>). In <FIG>, the UW screen section indicates that the movement of FOVM <NUM> brings it close to rectangle <NUM>" (preview <NUM>). The flower starts to appear in Macro screen section <NUM>. In <FIG>, FOVM <NUM> is seen in the UW screen section as fully overlapping rectangle <NUM>", while flower <NUM> is displayed fully in Macro screen section <NUM>.

<FIG> illustrates an embodiment numbered <NUM> of a smartphone having a UW camera with a FOVuw <NUM> and a Macro camera with a FOVM <NUM> and a screen <NUM> according to presently disclosed subject matter. Screen <NUM> is split into a UW screen section <NUM> and a Macro screen section <NUM>, both in portrait orientation, i.e. first screen section and second screen section are split horizontally, beneficial when device is held in portrait orientation. As in <FIG>, inside first section <NUM>, FOVM <NUM> is marked by a physical rectangle <NUM>', and, optionally, another physical rectangle <NUM>" indicates a UW preview of a flower <NUM> (which forms an image <NUM> in a camera) in order to guide the user towards flower <NUM>.

Such a method or apparatus in which the screen is split and both the ultra-wide FOV and Macro FOV are shown allows the user to find an OOI and capture it with the Macro camera (and possibly also simultaneously with the UW camera) even if the handset (image capture device) occludes the OOI.

<FIG> shows in a flow chart of a method of use of a split screen for Macro-photography in a mobile electronic device, according to embodiments disclosed herein. In step <NUM>, a screen of a mobile electronic device is split to display both a non-Macro (e.g. UW) camera image stream (or cropped version of it) and a Macro camera image stream (or cropped version of it) simultaneously. In step <NUM>, an OOI/ROI as a scene for the Macro image is selected in the FOVuw of the UW camera by a dedicated algorithm running on OOI/ROI selector <NUM> or by a human user. FOVuw and FOVM are calibrated. As known in the art, the position of the OOI/ROI in FOVuw can be translated to a respective OOI/ROI position in FOVM. In step <NUM>, the respective position of the OOI/ROI with respect to FOVM is calculated based on the OOI/ROI location position in FOVuw. In step <NUM>, the user is visually or otherwise guided towards the OOI/ROI's location position with respect to the FOVM. The visual or otherwise indication for the guiding may be visual (e.g. by arrow <NUM> shown in <FIG>) or via a dedicated sound or via some haptic feedback. In some examples, user control unit <NUM> is configured to provide visual or otherwise indication. According to the guiding of step <NUM>, the user moves the camera's hosting device (e.g. smartphone) until the OOI/ROI appears in FOVM. In step <NUM>, the user captures a Macro image (also referred to as "Super Macro image") of the OOI/ROI.

<FIG> shows schematically an embodiment of an electronic device (e.g. a smartphone) numbered <NUM> that includes a multi-camera and is configured to perform methods disclosed herein. Electronic device <NUM> comprises a Macro camera <NUM> with FOVM. Macro camera <NUM> includes a Macro lens module <NUM> with a Macro lens, a Macro image sensor <NUM> and a lens actuator <NUM> for actuating Macro lens module <NUM>. The Macro lens forms a Macro image recorded by Macro image sensor <NUM>.

Optionally, the Macro lens may have a fixed effective focal length (EFL) providing a fixed zoom factor (ZF), or an adaptable (variable) EFL providing an adaptable ZF. The adaption of EFL may be discrete or continuous, i.e. a discrete number of varying EFLs for providing a plurality of discrete or continuous zoom states with respective ZFs. Camera <NUM> may be switched to a beneficial zoom state automatically.

Optionally, Macro camera <NUM> may be a folded camera that includes an OPFE <NUM> and an OPFE actuator <NUM> for actuating OPFE <NUM> for OIS and/or FOV scanning. In some embodiments, the FOV scanning of the Macro camera may be performed by actuating one or more OPFEs. A scanning Macro camera that performs FOV scanning by actuating two OPFEs is described for example in the co-owned <CIT>.

Macro camera module <NUM> further comprises a first memory <NUM>, e.g. in an EEPROM (electrically erasable programmable read only memory). In some embodiments, first calibration data may be stored in memory <NUM>. In other embodiments, the first calibration data may be stored in a third memory <NUM> such as a NVM (non-volatile memory). The first calibration data may comprise calibration data between image sensors <NUM> and <NUM>.

Electronic device <NUM> further comprises a UW camera <NUM> with a FOVuw larger than FOVM of camera <NUM>. UW camera <NUM> includes UW lens module <NUM> with a UW lens and a UW image sensor <NUM>. A lens actuator <NUM> may move lens module <NUM> for focusing and/or OIS. In some embodiments, second calibration data may be stored in a second memory <NUM>. In other embodiments, the second calibration data may be stored in third memory <NUM>. The second calibration data may comprise calibration data between image sensors <NUM> and <NUM>.

The Macro camera may have an effective focal length (EFL) of e.g. <NUM> - <NUM> or more, a diagonal FOV of <NUM> - 40deg and a f number of about f/# = <NUM> - <NUM>. The UW camera may have an EFL of e.g. <NUM> - <NUM>, a diagonal FOV of <NUM> -130deg and a f/# of about <NUM> - <NUM>.

In some embodiments, the Macro camera may cover about <NUM>% of the area of the UW camera's FOV. In some embodiments, the Macro camera may cover about <NUM>% or less of the area of the UW camera's FOV.

Electronic device <NUM> further comprises an application processor (AP) <NUM>. Application processor <NUM> comprises a camera controller <NUM>, a user control unit <NUM>, OOI/ROI selector <NUM> and an image processor <NUM>. Electronic device <NUM> further comprises a screen control <NUM> and a screen <NUM>. Screen <NUM> may display methods as disclosed herein.

Returning now to the method of use as in <FIG>, in some examples, the UW or W camera image data may be used by camera controller <NUM> to automatically change a state of the Macro camera (i.e. control a change of scan state of the Macro camera). The state may be a scan (or steer) state, a zoom state or a focus state. For example, UW or W camera image data may be used to steer (or scan) the FOVM of a scanning Tele camera automatically towards a ROI (change of scan state of the Macro camera). In an example, the Macro camera can steer itself, and step <NUM> of guiding FOVM to the ROI can be done automatically by camera control <NUM>. As indicated above, the selection of an OOI/ROI in the FOVuw as a scene for the Macro image camera may be done by a dedicated algorithm running on OOI/ROI selector <NUM>. In another example, the Macro camera may use UW or W image data to switch between zoom states. Camera controller <NUM> may switch the Macro camera to a beneficial zoom state automatically (i.e. control a change of zoom state of the Macro camera), e.g. while executing steps <NUM> and <NUM>. A beneficial zoom state may be a state where a Macro OOI or ROI fully enters FOVM. In yet another example, the W or UW camera's image data may be used by camera controller <NUM> to focus the Macro camera to a ROI within FOVM automatically (i.e. control a change of focus state of the Macro camera). An OOI or ROI spanning a segment of a scene which is larger than FOVM may be fully captured in two or more sequential frames wherein each frame includes a different segment of the OOI or ROI. The sequential frames together include image data on the OOI or ROI in its entirety and are stitched to a single image by image processor <NUM>.

While this disclosure has been described in terms of certain examples and generally associated methods, alterations and permutations of the examples and methods will be apparent to those skilled in the art. The disclosure is to be understood as not limited by the specific examples described herein, but only by the scope of the appended claims.

It is appreciated that certain features of the presently disclosed subject matter, which are, for clarity, described in the context of separate examples, may also be provided in combination in a single example. Conversely, various features of the presently disclosed subject matter, which are, for brevity, described in the context of a single example, may also be provided separately or in any suitable sub-combination.

Claim 1:
A mobile electronic device (<NUM>, <NUM>), comprising:
a first camera (<NUM>) with a first field of view FOV<NUM> (<NUM>);
a second, Macro camera (<NUM>) with a Macro field of view FOVM (<NUM>) smaller than FOV<NUM>; and
a device screen (<NUM>)
characterized in including
a first screen section (<NUM>) configured to display first image data from the first camera and a second screen section (<NUM>) configured to display second image data from the Macro camera (<NUM>) when both cameras are focused to a distance equal to or smaller than <NUM>,
wherein the first screen section includes a visual indication (<NUM>') for guiding a user of the mobile electronic device towards a scene (<NUM>") using the FOVM.