Device comprising a multi-aperture imaging device for generating a depth map

An inventive device includes a multi-aperture imaging device comprising an image sensor; an array of adjacently arranged optical channels, each optical channel including an optic for projecting at least one partial field of view of a total field of view onto an image sensor area of the image sensor arrangement, a beam deflection means for deflecting an optical path of the optical channels, and a focusing means for setting a focal position of the multi-aperture imaging device. The device further comprises a control means configured to control the focusing means and to receive image information from the image sensor; the control means being configured to control the multi-aperture imaging device into a sequence of focal positions so as to capture a corresponding sequence of image information of the total field of view and to produce, from the sequence of image information, a depth map for the captured total field of view.

BACKGROUND OF THE INVENTION

The present invention relates to devices comprising multi-aperture imaging devices inside a housing, in particular to devices comprising small passage areas which are narrow, in particular, or are arranged in a manner efficient in terms of surface area, for optical paths, so-called viewing windows or exit windows.

Multi-aperture cameras operating on the basis of the principle of channel-wise division of the field of view use mirror facets as beam deflection elements. Having passed the respective areas of the mirror facets, the individual optical paths run through an opening in the housing that may be referred to as an exit window.

What would be desirable are devices which exhibit low consumption of area for such windows and/or which do not utilize the consumption of area in a manner that would be disturbance.

Therefore, it is the object of the present invention to provide a device exhibiting low and/or non-disturbing area consumption of the exit windows used for the multi-aperture imaging device.

SUMMARY

According to an embodiment, a device may have: a housing having two oppositely located main sides which are connected to each other via at least one edge side; a multi-aperture imaging device arranged in an interior of the housing and having: an image sensor arrangement; an array of adjacently arranged optical channels, each optical channel including an optic for projecting at least one partial field of view of a total field of view onto an image sensor area of the image sensor arrangement, and a beam deflection means for deflecting an optical path of the optical channels, the beam deflection means having a plurality of facets, each optical channel having a facet associated with it; wherein one of the main sides has a passage area arrangement with at least one passage area, the passage area arrangement being set up to allow the optical paths to pass; wherein the optical channels have, in relation to the passage area arrangement and along an axial direction, along a course of the optical path of the optical channels between the image sensor arrangement and the beam deflection means, a channel-specific relative position of at least one of: the image sensor area; the optic; a reflecting surface of a facet associated with the optical channel as a distance between the reflecting surface of the facet and the optic of the optical channel, differing from another optical channel.

According to another embodiment, a device may have: a housing having two oppositely located main sides which are connected to each other via at least one edge side; a multi-aperture imaging device arranged in an interior of the housing and having: an image sensor arrangement; an array of adjacently arranged optical channels, each optical channel including an optic for projecting at least one partial field of view of a total field of view onto an image sensor area of the image sensor arrangement, a beam deflection means for deflecting an optical path of the optical channels, the beam deflection means having a plurality of facets, each optical channel having a facet associated with it; wherein one of the main sides has a passage area arrangement with at least one passage area, the passage area arrangement being set up to allow the optical paths to pass; wherein a position of the beam deflection means determines a size of the multi-aperture imaging device along an axial direction between the image sensor arrangement and the beam deflection means, and the beam deflection means is arranged adjacently to the edge side; and wherein the passage area arrangement is arranged within an edge area of the main side adjacently to the edge side.

According to still another embodiment, a method of providing a device may have the steps of: providing a housing having two oppositely located main sides which are connected to each other via at least one edge side, so that one of the main sides has a passage area arrangement having at least one passage area, the passage area arrangement being set up for allowing the optical paths to pass; arranging a multi-aperture imaging device in an interior of the housing, said arranged multi-aperture imaging device having: an image sensor arrangement; an array of adjacently arranged optical channels, each optical channel including an optic for projecting at least one partial field of view of a total field of view onto an image sensor area of the image sensor arrangement, a beam deflection means for deflecting an optical path of the optical channels, the beam deflection means having a plurality of facets, each optical channel having a facet associated with it; so that the optical channels has, in relation to the passage area arrangement and along an axial direction, along a course of the optical path of the optical channels between the image sensor arrangement and the beam deflection means, a channel-specific relative position of at least one of: the image sensor area; the optic; a reflecting surface of a facet associated with the optical channel as a distance between the reflecting surface of the facet and the optic of the optical channel, differing from another optical channel.

One finding of the present invention consists in that the inventive setup of a multi-aperture imaging device may be utilized such that channel-specific local arrangement of individual components of optical channels or of the entirety of individual components along an axial direction may be utilized for obtaining different relative positions so as to keep a required passage opening size for allowing optical paths to pass into a housing (which is equivalent to optical paths exiting the housing) to a minimum, so that small window sizes in housings are possible, which is advantageous, in particular, in mobile devices where a portion of at least one housing side covered by a display (display means) is as large as possible or is to be utilized in as area-efficient a manner as possible.

In accordance with an embodiment, a device includes a housing having two oppositely located main sides which are connected to each other via at least one edge side. The device includes a multi-aperture imaging device which is arranged in an interior of the housing and comprises: an image sensor arrangement; an array of adjacently arranged optical channels, each optical channel including an optic for projecting at least one partial field of view of a total field of view onto an image sensor area of the image sensor arrangement, and a beam deflection means for deflecting an optical path of the optical channels, the beam deflection means comprising a plurality of facets, each optical channel having a facet associated with it. One of the main sides comprises a passage area arrangement with at least one passage area, the passage area arrangement being set up to allow the optical paths to pass through. In relation to the passage area arrangement and along an axial direction along a course of the optical path of the optical channels between the image sensor arrangement and the beam deflection means, the optical channels comprise a channel-specific relative position of at least one of: the image sensor area; the optic; and a reflecting surface of a facet associated with the optical channel.

DETAILED DESCRIPTION OF THE INVENTION

Before embodiments of the present invention will be explained in more detail below with reference to the drawings, it shall be noted that elements, objects and/or structures that are identical, identical in function or in action will be provided with identical reference numerals in the different figures, so that the descriptions of said elements which are provided in different embodiments are interchangeable and/or mutually applicable.

Some of the embodiments described herein relate to a channel-specific relative position of components of optical channels of a multi-aperture imaging device comprising an array of optical channels. What is understood by array is an arrangement, which consists of at least, and preferably, one line, of optical channels next to one another along a line direction. A multi-line array may comprise a number of more than one line, each of which may be arranged along the same line direction, and wherein the individual lines may be mutually offset along a column direction. Even if what can be understood as an optical channel is the course of beams of rays as a whole, said course may be at least partly determined by the arrangement of optical elements, e.g., lenses, associated with the respective channel, so that for the array of optical channels, it may be equivalent to arrange the optical elements next to one another.

Different relative positions therefore relate to different implementations within the optical channels and may disregard a lateral shift in the optical channels. The different relative positions may relate to different distances and/or relative inclinations/orientations of the individual components in relation to one another or within the same optical channel; different inclinations/orientations for facets of a faceted beam deflection means remain left out of consideration since their facets may already be inclined in a channel-specific manner. What is taken into account here, in particular, is that area of the course of the optical paths of the optical channels which is located between an image sensor area associated with the optical channel and a beam deflection which is associated with the optical channel and is caused by reflection or mirroring. The direction between the image sensor area and the beam deflection may be understood, in connection with embodiments described herein, as an axial direction which may be equal in quality notwithstanding possible directional divergences between the optical channels.

The divergences explained may be configured such that the optical channels within a shared plane are not parallel so as to obtain a preliminary divergence which may optionally be amplified (or, alternatively, merely caused) by the beam deflection so as to direct the optical paths towards different fields of view. The shared plane may be spanned open, e.g., by a vector along the axial direction and by a vector along the line extension direction, so that the divergences may occur within the shared plane (in-plane). Alternatively or additionally, provision is made, in inventive embodiments, for divergence to the obtained outside this plane (out-of-plane).

The above-mentioned beam deflection of the optical paths of the optical channels may be caused by a beam deflection means which comprises one or more reflecting areas, e.g., one facet per channel or group of channels, which enables implementing different setting angles between individual optical channels or groups.

The above-mentioned fields of view of an optical channel may be a total field of view or a partial field of view. Partial fields of view may possibly comprise a smaller angle of field or angle of view of the object area, i.e., they may cover a comparatively smaller area of the object area than a total field of view. This may be achieved by joining partial images of partial fields of view to form a total image of the total field of view, e.g. by means of stitching. A field of view of an optical channel may also be described as a total field of view. This total field of view may be captured, for example, by means of an optic that is different than an optic which images partial fields of view, and may enable capturing of the total field of view in addition to imaging of the total field of view that is obtained overall by the partial fields of view, but may also describe a total field of view that is different in terms of size and/or position in relation to the multi-aperture imaging device.

Some embodiments relate to devices comprising at least one multi-aperture imaging device and arranged inside a housing and looking to the outside from the inside through one or more passage areas. This is why potential, but non-limiting implementations of multi-aperture imaging devices will be explained at the beginning before their inventive arrangements within a device will be described.

FIG.1ashows a schematic perspective view of a device101in accordance with an embodiment. The device101includes a multi-aperture imaging device comprising an image sensor12and an array14of adjacently arranged optical channels16a-e. The multi-aperture imaging device further includes a beam deflection means18for deflecting an optical path of the optical channels16a-d. In this manner, the optical paths of the optical channels16a-dmay be deflected, between a lateral course, between the image sensor12by optics22a-dof the array14toward the beam deflection means18toward a non-lateral course. In total, different optical channels16a-dare deflected such that each optical channel16a-dprojects a partial field of view24a-dof a total field of view26onto an image sensor area28a-dof the image sensor12. The partial fields of view24a-dmay be distributed within the space in a one-dimensional or two-dimensional manner; on the basis of different focal widths of the optics22a-d, they may be also distributed in a three-dimensional manner. To further understanding, the total field of view26will be described below such that the partial fields of view24a-dexhibit two-dimensional distribution, it being possible for mutually adjacent partial fields of view24a-dto overlap one another. A total area of the partial fields of view results in the total field of view26.

The multi-aperture imaging device includes optional focusing means32for setting a focal position of the multi-aperture imaging device. This may be effected by changing a relative location or position between the image sensor12and the array14; the focusing means32may be configured to change a position of the image sensor12and/or a position of the array14so as to obtain a variable relative position between the image sensor12and the array14so as to set the focal position of the multi-aperture imaging device.

Setting of the relative position may be effected in a channel-specific manner, for groups of optical channels or in a manner that is global to all the channels. For example, a single optic22a-d, a group of the optics22a-dor all of the optics22a-dmay be moved together. The same applies to the image sensor12.

The device may include optical control means34configured to control the focusing means32. In addition, the control means34is configured to receive image information36from the image sensor12. Said image information36may be, e.g., the partial fields of view24a-dprojected onto the image sensor areas28a-d, and/or information or data which correspond to said images. This does not rule out intermediate processing of the image information36, for example with regard to filtering, smoothing or the like.

The control means34may be configured to control the multi-aperture imaging device to take a sequence of focal positions so as to capture a corresponding sequence of image information of the total field of view26. The control means34is configured to create a depth map38for the total field of view26from the sequence of image information. The depth map38may be provided via a corresponding signal. The control means34may capture, on the basis of the different focal positions obtained due to different relative positions between the image sensor12and the array14, different images of the same field of view26and/or differently focused partial images thereof in correspondence with the segmenting by the partial fields of view24a-d.

Depth maps may be employed for different purposes, for example for image processing, but also for joining (stitching) of images. For example, the control means34may be configured to join individual images (frames), which are obtained of the image sensor areas28ato28d, while using the depth map38so as to obtain image information42which renders the image of the total field of view26, i.e., a total image. For such methods of joining partial images, which is also referred to as stitching, utilization of a depth map is particularly advantageous.

While using the depth map, the control means34may be configured to stitch the partial images of a group of partial images to form a total image. This means that the depth map used for stitching may be generated from the partial images to be stitched themselves. For example, a sequence of total images rendering the total field of view may be generated on the basis of the sequence of image information. Each total image may be based on a combination of partial images of identical focal positions. Alternatively or additionally, at least two, several or all of the total images of the sequence may be combined so as to obtain a total image comprising expanded information, e.g., for creating Bokeh effect. Alternatively or additionally, the image may also be represented such that the entire image is artificially sharp-edged, i.e., a larger number of partial areas is put into focus than is the case in the individual images, e.g., the entire image.

In accordance with an embodiment, the device101is configured to create the image of the total field of view as a mono image and to create the depth map38from the sequence of mono images. Even though multiple scanning of the total field of view26is also possible, the device10may create the depth map from one mono image alone, which may save additional pictures being taken from different viewing directions, e.g. while using multiple capturing with the same device or by means of a redundant arrangement of additional optical channels.

The multi-aperture imaging device101may also be configured without the focusing means32and/or the control means34, as a result of which the multi-aperture imaging device101may be configured, e.g., as the multi-aperture imaging device ofFIG.1. Instead of the control means34, different means may be provided to obtain the image information from the image sensor12, or the control means34may be employed as having said function, but controlling of the focusing means may be dispensed with.

The depicted implementation of the multi-aperture imaging device is to be understood as being exemplary. In embodiments described herein, advantageous configurations of inventive multi-aperture imaging devices are explained. Several drawings, e.g.,FIG.1a, are depicted such that an image sensor having several image sensor areas, an array of optical channels having one shared substrate, and/or a beam deflection means are configured as a shared component, which, however, does not conflict with an inventive channel-specific implementation of the relative positions that will be explained later since the drawings are purely schematic in this respect. All of the embodiments described herein, without any restriction, relate to configuring the image sensor, the array of optical channels and/or the beam deflection means in the form of several individual paths that are positioned differently in relation to one another and/or along an axial direction of the optical paths, and/or to implement, instead of a depicted planar, or flat, implementation, a topography-related configuration of said components that is perpendicular to the axial direction e.g., tilted and/or stepped image sensors having different relative positions of the image sensor areas, tilted and/or stepped arrays comprising different relative positions of the optics, and/or tilted or stepped beam deflection means having different relative positions of the beam deflection areas, or facets.

FIG.1bshows a schematic perspective view of a device102in accordance with an embodiment. As compared to the device101, the device102comprises, instead of the control means34, a control means44configured to control the beam deflection means to take different positons181and182. In the different positions181and182, the beam deflection means18comprises mutually different relative positions, so that in the different positions, imaging information of different total fields of view261and262are obtained since the optical paths of the optical channels16a-dare directed to take different directions which are influenced by the different positions181and182. Alternatively or additionally to the control means34, the device102comprises control means44configured to control the beam deflection means to take the first position181so as to obtain imaging information of the first total field of view261from the image sensor12. Before or after this, the control means44is configured to control the beam deflection means18to take the second position182so as to obtain imaging information of the second total field of view262from the image sensor12. The control means44is configured to insert part of the first imaging information461into the second imaging information462so as to obtain shared or accumulated image information48. The accumulated image information48may in parts render the first total field of view261and, in parts, the second total field of view262, which also involves image-manipulating or image-processing steps. This means that the accumulated image information48is based, in places, on imaging of the total field of view261, and in other places, on the imaging of the total field of view262.

The control means44may be configured to provide a signal52which contains and reproduces the accumulated image information48. Optionally, the image information461and/or462may also be output by the signal52.

FIG.1cshows a schematic view of a device103in accordance with an embodiment, which instead of the control means34ofFIG.1aand instead of the control means44ofFIG.1bcomprises a control means54which combines the functionalities of the control means34and of the control means44and is configured, on the basis of a variable focal position of the device103, to create the depth map38and to provide the accumulated image information48ofFIG.1b.

FIG.2ashows a schematic view of different focal positions561to565, which a device may be controlled to take in accordance with embodiments described herein, e.g., the device101and the device102. The different focal positions561to565may be understood as being positions or distances581to585wherein objects within the captured total field of view are projected onto the image sensor12in a focused manner. In this context, a number of focal positions56may be arbitrary and exhibit a number larger than 1.

Distances621to624between adjacent focal positions may relate to distances within the image space; implementation or transferal of the explanation to distances within the object space are also possible. However, what is advantageous about considering the image space is that the properties of the imaging multi-aperture imaging device are taken into account, in particular with regard to a minimum and/or maximum object distance. The control means34and/or54may be configured to control the multi-aperture imaging device such that same comprises two or a higher number of focal positions561to565. In the respective focal positions, individual images641and642may be captured in accordance with the number of captured partial fields of view24. On the basis of the knowledge of which of the focal positions561to565was set to obtain the respective partial image461and462, the control means may determine, by analyzing the image information in terms of which of the image parts are imaged in focus, the distance in which these objects imaged in a sharp-edged manner are arranged with regard to the device. Said information regarding the distance may be used for the depth map38. This means that the control means may be configured to sense, in the sequence of focal positions561to565, a corresponding number of groups of partial images, each partial image being associated with an imaged partial field of view. The group of partial images may thus correspond to those partial images which image the total field of view in the focal position that has been set.

The control means may be configured to create the depth map from a comparison of local image sharpness information in the partial images. The local sharpness information may designate the areas of the image in which objects are imaged in focus or are imaged in focus within a predefined tolerance range. For example, by determining the edge blurring function and by detecting the distances across which the edges extend one may determine whether a corresponding image area, a corresponding object or a part thereof is imaged in focus or is imaged on the image sensor in a blurred manner. In addition, the point image or line blurring function may be used as a criterion of quality of the sharpness of an image content. Alternatively or additionally, any known optical sharpness metric such as the known modulation transfer function (MTF), for example, may be used. Alternatively or additionally, the sharpness of the same objects may be used in adjacent images of the stack, association of the focus actuator position with the object distance via a calibrated look-up table and/or the direction of the through-focus scan may be used so as to obtain the depth information in a partially recursive manner from adjacent images of the stack and to avoid ambiguities. Thus, when one knows the set focal position, which unambiguously correlates with the sharply imaged object distance, one may therefore deduce, from the knowledge that the object is sharply imaged at least within the previously defined tolerance range, a distance of the area of the image, of the object or of the part thereof, which may be a basis for the depth map38.

While using the depth map, the control means may be configured to stitch the partial images of a group of partial images to form a total image. This means that the depth map used for stitching may be generated from the partial images to be stitched themselves.

The device may be configured to control the focusing means32such that the sequence of focal positions561to565within a tolerance range of ±25%, ±15% or ±5%, preferably as close to 0% as possible, is equidistantly distributed, within the image space, between a minimum focal position and a maximum focal position. In order to save time for setting a focal position it is useful, but not mandatory, to sequentially control the focal positions561to565one after the other, at an increasing or decreasing distance. Rather, an order of the set focal positons561to565is arbitrary.

FIG.2bshows a schematic representation of the utilization of the depth map38and its generation. The partial images641and642may each be used for obtaining partial information381to385of the depth map38from the respective focal position561to565since in each case, the objects which are sharply depicted in the individual images641and642may be accurately determined with regard to their distance. In between the focal positions561and565, interpolation methods may also be used, so that even with slightly blurred objects, one may still obtain sufficiently precise information for the depth map38. The distance information contained in the partial information381to385may be combined by the control means to form the depth map38. The depth map38may be used for combining the frames641and642from the different focal positions561to565to form a corresponding number of total images421to425.

FIG.3ashows a schematic perspective view of a device30in accordance with an embodiment. The image sensor12, the array14and the beam deflection means18may span a cuboid within the space. The cuboid may also be understood to be a virtual cuboid and may comprise, e.g., a minimum volume, and in particular a minimum perpendicular extension along a direction parallel to a thickness direction y, which is parallel to a line extension direction66. The line extension direction66extends along a z direction, for example, and perpendicularly to an x direction which is arranged in parallel with a course of the optical paths between the image sensor12and the array14. The directions x, y and z may span a Cartesian coordinate system. The minimum volume of the virtual cuboid and/or its minimum perpendicular extension may be such that the virtual cuboid nevertheless includes the image sensor12, the array14and the beam deflection means18. The minimum volume may also be understood to mean that it describes a cuboid which is spanned by arranging and/or operatively moving the image sensor12, the array14and/or the beam deflection means18. The line extension direction66may be configured such that along the line extension direction66, the optical channels16aand16bare arranged next to each other, possibly in parallel with each other. The line extension direction66may be arranged to be stationary within the space.

The virtual cuboid may comprise two sides which are mutually opposite and in parallel, in parallel with the line extension direction66of the array14and in parallel with a part of the optical path of the optical channels16aand16bbetween the image sensor12and the beam deflection means18. In simplified terms, however without any limiting effect, said sides may be an upper side and a lower side of the virtual cuboid, for example. The two sides may span a first plane68aand a second plane68b. This means that the two sides of the cuboid may each be part of the plane68aand/or68b. Further components of the multi-aperture imaging device may be arranged to be located completely, but at least partly, within the area located between the planes68aand68b, so that an installation space requirement of the multi-aperture imaging device along the y direction, which is parallel to a surface normal of the planes68aand/or68b, may be small, which is advantageous. A volume of the multi-aperture imaging device may comprise a small or minimum installation space between the planes68aand68b. Along the lateral sides or extension directions of the planes68aand/or68b, an installation space of the multi-aperture imaging device may be large or of any size. The volume of the virtual cuboid is influenced, for example, by an arrangement of the image sensor12, of the array14and of the beam deflection means18; arrangement of said components in accordance with the embodiments described herein may be effected such that the installation space of said components along the direction perpendicular to the planes and, therefore, the mutual distance of the planes68aand68bbecomes small or minimum. As compared to other arrangements of the components, the volume and/or the distance of other sides of the virtual cuboid may be increased.

The device30includes an actuator72for generating a relative movement between the image sensor12, the single-line array14and the beam deflection means18. This may include, e.g., actuation of the beam deflection means18for switching between the positions described in connection withFIG.1b. Alternatively or additionally, the actuator72may be configured to perform the relative movement, explained in connection withFIG.1a, for changing the relative position between the image sensor12and the array14. The actuator72is at least partly arranged between the planes68aand68b. The actuator72may be configured to move at least one of the image sensor12, the single-line array14and the beam deflection means18, which may include rotational and/or translational movements along one or more directions. Examples thereof are a channel-specific change in a relative position between image sensor areas28of a respective optical channel16, of the optic22of the respective optical channel16and of the beam deflection means18and/or of the corresponding segment or facet and/or for a channel-specific change in an optical property of the segment/facet which concerns the deflection of the optical path of the respective optical channel. Alternatively or additionally, the actuator72may at least partly implement auto-focus and/or optical image stabilization.

The actuator72may be part of the focusing means32and may be configured to provide a relative movement between at least one optic of at least one of the optical channels16aand16band the image sensor12. The relative movement between the optic22aand/or22band the image sensor12may be controlled by the focusing means32such that the beam deflection means18performs a simultaneous movement. When a distance between the optic22aand/or22band the image sensor is reduced, the distance between the beam deflection means18and the image sensor12may be reduced accordingly, so that a relative distance between the array14and/or the optic22aand/or22band the beam deflection means18essentially remains the same. This enables the beam deflection means18to be implemented with small beam deflection faces since a cone of rays which grows because of a growing distance between the array14and the beam deflection means18may be compensated for by maintaining the distance from the beam deflection means18.

The focusing means32and/or the actuator72are arranged to project by a maximum of 50% from the area located between the planes68aand68b. The actuator72may comprise a dimension or extension74that is parallel to the thickness direction y. A proportion of a maximum of 50%, a maximum of 30% or a maximum of 10% of the dimension74may project beyond the plane68aand/or68b, starting from an area located between the planes68aand68b, and may thus project out of the virtual cuboid. This means that the projection of the actuator72beyond the plane68aand/or68bis marginal at the most. In accordance with embodiments, the actuator72does not project beyond the planes68aand68b. What is advantageous about this is that an extension of the multi-aperture imaging device along the thickness direction y is not increased by the actuator72.

Even though the beam deflection means18is depicted to be rotationally mounted about an axis of rotation76, the actuator72may alternatively or additionally also generate a translational movement along one or more spatial directions. The actuator72may include one or more single actuators, possibly so as to generate different individual movements in an individually controllable manner. The actuator72or at least a single actuator thereof may be implemented as or may include, e.g., a piezo actuator, in particular a piezoelectric bending actuator, which is described in more detail in connection withFIG.4. A piezo bender enables a fast and reproducible change in position. This property advantageously enables capturing of focus stacks in the sense of several or many images within a short time. Piezo benders as actuators configured to be long along a dimension or direction may be advantageously employed, in particular, in the architecture described since they comprise a form factor that is advantageous for this purpose, i.e., an extension particularly in one direction.

The array14may include a substrate78which has the optics22aand22battached or arranged thereat. The substrate78may be at least partly transparent by means of recesses or by means of a suitable choice of materials for the optical paths of the optical channels16aand16b, which does not rule out that manipulations may be performed in the optical channels, e.g. by arranging filter structures or the like.

Several requirements placed upon the actuator72, including fast adjustability for rapidly setting the different focal positions56, a large force with a small requirement in terms of installation space, and the like may be met by using piezoelectric actuators.

FIG.3bshows a schematic sectional side view of the device30in accordance with an embodiment. The multi-aperture imaging device of the device30may comprise, e.g., a plurality of actuators, e.g., more than one, more than two or a different number >0. For example, actuators721to725may be arranged which may be employed for different purposes, e.g., to adapt the focal position and/or to change the location or position of the beam deflection means18for setting the viewing direction of the multi-aperture imaging device and/or for providing optical image stabilization by means of rotational movement of the beam deflection means18and/or translational movement of the array14.

The actuators721to725may be arranged to be arranged at least partly between the two planes68aand68bwhich are spanned by sides69aand69bof the virtual cuboid69. The sides69aand69bof the cuboid69may be aligned in parallel with each other and in parallel with the line extension direction of the array and of part of the optical path of the optical channels between the image sensor12and the beam deflection means18. The volume of the cuboid69is at a minimum and nevertheless includes the image sensor12, the array14and the beam deflection means18as well as their operational movements. Optical channels of the array14comprise an optic22which may be configured to be identical for each channel or may be different.

A volume of the multi-aperture imaging device may comprise a small or minimal installation space between the planes68aand68b. Along the lateral sides or extension directions of the planes68aand/or68b, an installation space of the multi-aperture imaging device may be large or of any size. The volume of the virtual cuboid is influenced, for example, by an arrangement of the image sensor12, of the single-line array14and of the beam deflection means; arrangement of these components in accordance with the embodiments described herein may be such that the installation space of said components along the direction perpendicular to the planes and, therefore, the distance of the planes68aand68bto each other becomes small or minimal. As compared to other arrangements of the components, the volume and/or the distance of other sides of the virtual cuboid may be increased.

The virtual cuboid69is depicted by dotted lines. The planes68aand68bmay include or be spanned by two sides of the virtual cuboid69. A thickness direction y of the multi-aperture imaging device may be arranged to be normal to the planes68aand/or68band/or to be parallel to the y direction.

The image sensor12, the array14and the beam deflection means18may be arranged such that a perpendicular distance between the planes68aand68balong the thickness direction y, which distance may be referred to, in simplified terms, however without limitation, as the height of the cuboid, is minimal; a minimization of the volume, i.e., of the other dimensions of the cuboid, may be dispensed with. An extension of the cuboid69along the direction y may be minimal and may essentially be determined by the expansion of the optical components of the imaging channels, i.e., of the array14, of the image sensor12and of the beam deflection means18along the direction y.

A volume of the multi-aperture imaging device may comprise a small or minimal installation space between the planes68aand68b. Along the lateral sides or extension directions of the planes68aand/or68b, an installation space of the multi-aperture imaging device may be large or of any size. The volume of the virtual cuboid is influenced, for example, by an arrangement of the image sensor12, of the single-line array14and of the beam deflection means; arrangement of these components in accordance with the embodiments described herein may be such that the installation space of said components along the direction perpendicular to the planes and, therefore, the distance of the planes68aand68bto each other becomes small or minimal. As compared to other arrangements of the components, the volume and/or the distance of other sides of the virtual cuboid may be increased.

The actuators721to725may each comprise a dimension or extension that is parallel to the thickness direction y. A proportion of a maximum of 50%, a maximum of 30% or a maximum of 10% of the dimension of the respective actuator721to725may project beyond the plane68aand/or68b, starting from an area located between the planes68aand68bor may project out of the area. This means that projection of the actuators721to725beyond the plane68aand/or68bis marginal at the most. In accordance with embodiments, the actuators do not project beyond the planes68aand68b. What is advantageous about this is that an extension of the multi-aperture imaging device along the thickness direction, or direction y, is not increased by the actuators.

Even though terms such as top, bottom, left, right, front, or back are used here to improve clarity, these terms are not to have any limiting effect. It shall be understood that said terms are interchangeable on the basis of a rotation or tilting within the space. For example, the x direction from the image sensor12toward the beam deflection means18may be understood to mean at the front or forward. A positive y direction may be understood to be at the top, for example. An area along the positive or negative z direction apart from or adjacent to the image sensor12, the array14and/or the beam deflection means18may be understood as being located next to the respective component. In simplified terms, an image stabilizer may include at least one of the actuators721to725. The at least one actuator may be arranged within a plane71or between the planes68aand68b.

In other words, the actuators721to725may be arranged in front of, behind or next to the image sensor12, the array14and/or the beam deflection means18. In accordance with embodiments, the actuators36and42having a maximum circumference of 50%, 30% or 10% are arranged outside the area located between the planes68aand6b.

FIG.3cshows a schematic sectional side view of the multi-aperture imaging device, wherein different total fields of view261and262may be captured on the basis of different positions of the beam deflection means18since the multi-aperture imaging device then exhibits different viewing directions. The multi-aperture imaging device may be configured to change tilting of the beam deflection means by an angle α, so that alternately, different main sides of the beam deflection means18are arranged to face the array14. The multi-aperture imaging device may include an actuator configured to tilt the beam deflection means18by the axis of rotation76. For example, the actuator may be configured to move the beam deflection means18to a first position in which the beam deflection means18deflects the optical path26of the optical channels of the array14to the positive y direction. To this end, in the first position, the beam deflection means18may exhibit, e.g., an angle α of >0° and <90°, of at least 10° and at the most 80°, or at least 30° and at the most 50°, e.g., 45°. The actuator may be configured to deflect the beam deflection means, in a second position, about the rotational axis76such that the beam deflection means18deflects the optical path of the optical channels of the array14toward the negative y direction, as is depicted by the viewing direction toward the total field of view262and the dashed representation of the beam deflection means18. For example, the beam deflection means18may be configured to be reflecting on both sides, so that in the first position, the viewing direction points toward the total field of view261.

FIG.4ashows a schematic top view of a device40in accordance with an embodiment, wherein the actuator72is configured as a piezoelectric bending actuator. The actuator72is configured to perform bending within the x/z plane, as depicted by the dashed lines. The actuator72is connected to the array14via a mechanical deflection means82, so that upon bending of the actuator72, a lateral shift of the array14along the x direction may occur, so that the focal position may be changed. For example, the actuator72may be connected to the substrate78. Alternatively, the actuator72may also be arranged at a housing which houses at least some of the optics22ato22d, so as to move the housing. Other variants are also possible.

Optionally, the device40may comprise further actuators841and842configured to generate a movement at the array14and/or at the beam deflection means18, for example for placing the beam deflection means18into different positions and/or for the purpose of optical image stabilization by translational shifting of the array14along the z direction and/or by generating a rotational movement of the beam deflection means18about the axis of rotation76.

Unlike the description given in the previous figures, the beam deflection means18may comprise several facets86ato86dwhich are spaced apart from one another but may be moved together, each optical channel being associated with a facet86ato86d. The facets86ato86dmay also be directly adjacent to one another, i.e., may be arranged with little or no distance from one another. Alternatively, a planar mirror may also be arranged.

By actuating the actuator72, a distance881between at least one of the optics22a-dand the image sensor12may be changed, e.g. increased or reduced, from a first value881to a second value882.

FIG.4bshows a schematic sectional side view of the device40for illustrating the arrangement of the actuator72between the planes68aand68b, which are described in connection withFIG.3a. For example, the actuator72is arranged fully between the planes68aand68b, as is the mechanical deflection means82, which may comprise several force-transmitting elements, e.g., connecting bridges, wires, ropes or the like and mechanical bearings or deflection elements.

The mechanical deflection means and/or mechanical means for transmitting the movement to the array14may be arranged on one side of the image sensor12which faces away from the array14, i.e., behind the image sensor12, when starting from the array14. The mechanical means82may be arranged such that a flux of force laterally passes the image sensor12. Alternatively or additionally, the actuator72or a different actuator may be arranged on a side of the beam deflection means18which faces away from the array14, i.e., behind the beam deflection means18, when starting from the array14. The mechanical means82may be arranged such that a flux of force laterally passes the beam deflection means18.

Even though only one actuator72is depicted, it is also possible for a larger number of actuators to be arranged and/or for more than one side of the actuator72to be connected to a mechanical deflection means82. For example, a centrally mounted or supported actuator72may be connected to one mechanical deflection means82on two sides, respectively, and may be effective, e.g., on both sides of the array14so as to enable a homogenous movement.

FIG.5ashows a schematic representation of an arrangement of partial fields of view24aand24bwithin a total field of view26which may be captured, for example, by a multi-aperture imaging device described herein, e.g., the multi-aperture imaging device101,102,103,30and/or40and may correspond, e.g., to the total field of view261and/or262. For example, the total field of view26comprising the optical channel16bmay be projected onto the image sensor area28b. For example, the optical channel16amay be configured to capture the partial field of view24aand to project it onto the image sensor area28a. A different optical channel, e.g., the optical channel16c, may be configured to capture the partial field of view24band to project it onto the image sensor area28c. This means that a group of optical channels may be configured to capture precisely two partial fields of view24aand24b. Thus, simultaneous capturing of the total field of view and of the partial fields of view, which in turn together image the total field of view26, may occur.

Even though the partial fields of view24aand24bare depicted with different extensions to improve distinction, they may have identical or comparable extensions along at least one image direction B1or B2, e.g., along the image direction B2. The extension of the partial fields of view24aand24bmay be identical to the extension of the total field of view26along the image direction B2. This means that the partial fields of view24aand24bmay fully capture the total field of view26along the image direction B2or may capture the total field of view26only partly along a different image direction B1arranged perpendicularly thereto, and may be arranged to be mutually offset, so that in combinatorial terms, complete capturing of the total field of view26results also along the second direction. In this context, the partial fields of view24aand24bmay be mutually disjoint or may mutually overlap in an incomplete manner, at the most, in an overlap area25which possibly fully extends along the image direction B2in the total field of view26. A group of optical channels including the optical channels16aand16cmay be configured to jointly fully image the total field of view26, e.g., by means of complete capturing in combination with partial captures, which jointly image the total field of view. The image direction B1may be a horizontal of an image to be provided, for example. In simplified terms, the image directions B1and B2represent two different image directions that have any arbitrary alignments within the space.

FIG.5bshows a schematic representation of an arrangement of the partial fields of view24aand24b, which are arranged in a mutually offset manner along a different image direction, the image direction B2, and mutually overlap. The partial fields of view24aand24bmay capture the total field of view26in a complete manner along the image direction B1, and in an incomplete manner along the image direction B2. The overlap area25is arranged entirely within the total field of view26along the image direction B1.

FIG.5cshows a schematic representation of four partial fields of view24ato24b, which capture the total field of view26in an incomplete manner in both directions B1and B2in each case. Two adjacent partial fields of view24aand24boverlap within an overlap area25b. Two overlapping partial fields of view24aand24coverlap within an overlap area25c. Similarly, partial fields of view24cand24doverlap within an overlap area25d, and the partial field of view24doverlaps with the partial field of view24awithin an overlap area25a. All four partial fields of view24ato24dmay overlap within an overlap area25eof the total field of view26.

For capturing the total field of view26and the partial fields of view24a-d, a multi-aperture imaging device may be configured in a manner similar as that described in connection withFIGS.1a-c, wherein the array14may comprise five optics, for example—four for capturing partial fields of view24a-dand one optic for capturing the total field of view26. Accordingly, the array may be configured with three optical channels in connection withFIGS.5a-b.

There is a large amount of image information available within the overlap areas25ato25e. For example, the overlap area25bis captured via the total field of view26, the partial field of view24aand the partial field of view24b. An image format of the total field of view may correspond to a redundancy-free combination of the imaged partial fields of view, for example of the partial fields of view24a-dinFIG.5c, the overlap areas25a-ebeing counted once only in each case. In connection withFIGS.5aand5b, this applies to the redundancy-free combination of the partial fields of view24aand24b.

An overlap within the overlap areas25and/or25a-emay include, e.g., a maximum of 50%, a maximum of 35% or a maximum of 20% of the respective partial images.

In other words, in accordance with embodiments described herein, a reduction of the number of optical channels may be obtained, which enables saving costs and reducing the lateral installation-space requirement. In accordance with embodiments described herein, a form of depth information retrieval is enabled which is an alternative to stereoscopic capturing and which makes do without corresponding additional sensors such as Time of Flight, Structured or Coded Light and the like. Time-of-flight sensors enabling low resolution as well as structured-light sensors exhibiting high energy requirements may thus be avoided. Both approaches further exhibit problems with intense ambient lighting, in particular sunlight. Embodiments provide for the corresponding device to be configured without such sensors. In accordance with an embodiment, a piezo bender serves as an extremely fast focus factor with little power consumption. The described architecture of the multi-aperture imaging device enables utilizing such piezo benders since an otherwise cubic form factor of the camera module impedes, or even rules out, utilization of long piezo benders. With short exposure times, this enables capturing of focus stacks, i.e., of numerous images quickly captured one after the other with slightly different focusing of the scene. Embodiments provide for the entire depth of the scene to be sensibly scanned, e.g., from macro, the closest possible way of capturing, to infinitely, which means the furthest possible distance. The distances may be equidistantly arranged within the object space, but preferably within the image space. Alternatively, a different sensible distance may be selected. A number of focal positions is, e.g., at least two, at least three, at least five, at least ten, at least 20 or any other random number.

Other embodiments, however, provide provision of a source of illumination configured to emit an illumination signal within a wavelength range not visible to the human eye, i.e., within wavelengths of less than about 380 nm and/or more than about 700 nm, preferably an infrared range of at least 0.7 μm and at the most 1,000 μm in wavelength, and particularly preferably a near-infrared range of a wavelength of at least 0.75 μm and at the most 3 μm, in the direction of the total field of view to be captured. The multi-aperture imaging device may be configured to image the total field of view within the non-visible wavelength range that is used. To this end, in particular the image sensor may be adapted to the wavelength used by the source of illumination.

Several images42may be presented to the user. Alternatively or additionally, embodiments provide combining the individual image information, so that the user may be presented with an image comprising combined image information. For example, an image comprising depth information, which offers the possibility of digital re-focusing, for example. The image presented may offer a so-called Bokeh effect, a setting to induce a blur. Alternatively, the image may also be presented such that the entire image is artificially in focus, which means that a larger range of distance is put into focus than is the case in the individual images of partial areas, e.g., the entire image. With a small f-number of the used objectives, the object distance of the individual elements of the scene may be reconstructed, and a depth map in image resolution may be created from this, on the basis of the acuity and/or blurriness measured in the individual images and of further information, e.g., the acuity of the same objects in adjacent images of the stack, association of the focus actuator position with an object distance, e.g., while using a calibrated look-up table, a direction of the sequence of focal positions (through-focus scan) in itself but also, in a recursive manner, from other images, so as to avoid any ambiguities.

In accordance with embodiments described herein, one achieves that duplication of the channels for stereo imaging may be dispensed with, while nevertheless a depth map may be created. Said depth map enables image stitching of the different partial images of the multi-aperture imaging device. By, e.g., reducing the number of optical channels by half, one may obtain a clear reduction of the lateral dimensions, e.g., along the line extension direction, and thus, one may achieve a reduction in price as well. By means of other steps, image processing may provide images that are at least as good.

The three optical channels for capturing both partial fields of view24aand24band the total field of view26may be arranged, within the array14, along the line extension direction. The line extension direction may be arranged in parallel with the image direction B1, for example, so that the partial fields of view of the total field of view are arranged in a direction parallel to the line extension direction (FIG.5a) of the array or perpendicular thereto (FIG.5b).

FIG.6shows a schematic perspective view of a device60in accordance with an embodiment. The implementations described readily also apply to the devices101,103,30and/or40. By controlling the beam deflection means to enter into different positions, the device60, or the multi-aperture imaging device of the device60, may capture two mutually spaced-apart total fields of view261and262.

For example, the device60is configured as a portable or mobile device, in particular a tablet computer or a mobile phone, in particular a smartphone.

One of the fields of view261and262may be arranged, e.g., along a user direction of the device60, as is customary, for example, within the context of taking pictures of oneself (selfies) for photos and/or videos.

The other total field of view may be arranged, e.g., along an opposite direction and/or a world direction of the device60and may be arranged, e.g., along that direction along which the user looks when he/she looks at the device60, starting from the total field of view, along the user direction. For example, the beam deflection means18inFIG.1bmay be formed to be reflective on both sides and may deflect the optical path of the optical channels16a-dwith different main sides in different positions, for example, so that starting from the device60, the total fields of view261and262are arranged to be mutually opposite and/or at an angle of 180°.

FIG.7ashows a schematic diagram for illustrating processing of the image information461and462, which may be obtained by imaging the total fields of view261and262. The control means is configured to separate part92of the imaging information461of the field of view261, e.g., to cut it out, to isolate it or to copy exclusively part92. In addition, the control means is configured to combine the separated or segmented part92with the imaging information462, i.e., to insert the part92into the imaging information462so as to obtain the accumulated image information48. Said image information48comprises, in parts, the total field of view262and comprises, in parts, namely where the part92has been inserted, the image information461. It shall be noted that obtaining the accumulated image information48is not limited to inserting a single part92, but that any number of parts92may be segmented from the image information461and that one, several or all of said parts may be inserted into the image information462.

A location or position where the part92is inserted into the second imaging information462may be automatically determined by the control means, e.g., by projecting the part92through the device60into the second field of view262, but may alternatively or additionally also be selected by a user.

In accordance with an embodiment, the control means is configured to identify and segment a person in the first imaging information461, for example via a pattern comparison and/or edge detection, in particular, however, on the basis of the depth map created by the device itself. The control means may be configured to insert the image of the person into the second imaging information462to obtain the accumulated image information48. This means that the part92may a person, e.g., a user of the device60. Embodiments provide for the device to be configured to automatically identify the person and to automatically insert the image of the person, i.e., the part92, into the second imaging information462. This enables automatic creation of a self-portrait or a selfie in front of the or in the second total field of view262without having to go through great effort in terms of positioning the device60and/or the user.

Embodiments provide for the control means to use a depth map, e.g., the depth map38, so as to position the part92in the second imaging information462. The depth map38may comprise a plurality or multitude of depth planes, for example in accordance with the number of focal positons taken into account or with a reduced number obtained therefrom or a larger number interpolated therefrom. The control means may be configured to insert the part92within the predetermined depth plane of the second imaging information462so as to obtain the accumulated image information48. The predetermined depth plane may correspond essentially, i.e., within a tolerance range of ±10%, ±5% or ±2%, to a distance of the first total field of view262from the device60and/or to the distance of the segmented part92from the device60. This may also be referred to as depth-correct insertion of the part92into the second imaging information462.

FIG.7bshows a schematic representation of a scaling of the part92in the accumulated image information48. Alternatively, one may also select a different depth plane, various possibilities of embodiments being provided for this purpose. For example, the predetermined depth plane may be influenced or determined by the placement of the part92in the second imaging information462. Said placement may be automatic or may be effected by a user input. For example, if the user selects a specific place or location within the second imaging information462to insert the part92, the control means may be configured to determine, in the second imaging information462, a distance of that area into which the part92is to be inserted. Having knowledge of the distance of the part92from the device and from the objects in the second imaging information462, for example while using depth maps, a virtual change in distance of the part92, which is caused by the user input, may be compensated for by scaling of the part92.

Thus, a one-dimensional, two-dimensional or three-dimensional size94of the part92may be changed to a size96, e.g., may be reduced, when the distance of the part92from the first imaging information461to the second imaging information462is increased, or may be increased when the distance from the first imaging information461to the second imaging information462is reduced. Irrespective of, but also in combination with, the placement of the part92in the first imaging information461on the basis of an associated user input, the device may be configured to scale the imaging information461so as to obtain scaled imaging information. This scaled imaging information may be inserted into the imaging information462by the control means so as to obtain the accumulated image information48. The device may be configured to determine a distance of an object, which represents the part92and is imaged in the first imaging information461, with regard to the device60. The device may scale the imaging information461and/or the part92thereof on the basis of a comparison of the determined distance with the predetermined depth plane in the second imaging information462. It is advantageous for the two items of imaging information461and462to be captured within short time lags. Advantageously, this time lag within a time interval amounts to 30 ms at the most, 10 ms at the most, 5 ms at the most, or 1 ms at the most, e.g., 0.1 ms. This time may be exploited, for example, for switching or re-positioning of the beam deflection means and may be determined at least partly by a duration of said process.

The accumulated image information48may be obtained as an individual image; however, alternatively or additionally, it may be obtained as a video data stream, e.g. as a multitude of individual images.

In accordance with an embodiment, a device is configured such that the first imaging information461includes an image of a user, and the second imaging information462includes a world view of the device. The control means is configured to segment an image of the user from the first imaging information461and to insert it into the world view. For example, the device may be configured to insert the image of the user into the world view in a manner that is correct in terms of depth.

In other words, in connection with embodiments described herein, taking a selfie picture or making a selfie video may include a depth-based combination of quasi-simultaneous pictures taken with a front-facing camera/view and a rear-side camera/view (main camera/view) of a device, in particular of a mobile phone. In this context, the foreground of the selfie picture, i.e., the self-portrait, may be transferred to the foreground of the picture taken by the main camera. Very fast switching between front-side and main-side picture-taking by changing the position of the beam deflection means enables said quasi-simultaneous capturing of the world-side and the user-side camera image with the same image sensor. Even though in accordance with embodiments described herein, a one-channel imaging device may also be used, embodiments described herein provide advantages in particular with regard to multi-aperture imaging devices since they may already create or use a depth map so as to join (stitch) the individual images. Said depth map may also be used for determining depth information for synthesizing the accumulated imaging information48. A sequence of events is enabled which may be described as follows:1. Use depth map of the selfie picture so as to segment the foreground, i.e., the person(s) taking (a) picture(s) of themselves, from the background;2. Use the depth map of the world-side picture taken so as to ascertain a foreground and a background therefrom, i.e., to separate depth information; and3. Insert the foreground, i.e., the person(s) taking (a) picture(s) of themselves, from the selfie picture into the image of the world-side picture, in particular its foreground.

What is advantageous about this is that the selfie picture may be combined with the world-side picture as the background without having to rotate the phone, as is otherwise necessary by 180° so as to take a picture of oneself in front of said scene. Alternatively or additionally, one avoids picture being taken past oneself in a backward manner, which requires that one always has to think in mirror-inverted terms with regard to the alignment of the phone in relation to the scene. It is possible to create the depth map itself, as is described in connection with embodiments described herein, so that additional arrangement of time-of-flight sensors or structured-light sensors may be dispensed with.

In the following, several advantageous implementations of the multi-aperture imaging device will be addressed in order to illustrate inventive advantages.

FIG.8shows parts of a multi-aperture imaging device80which may be employed in inventive devices; possible focusing means and/or actuating elements for implementing optical image stabilization are not depicted but may be readily implemented.

The multi-aperture imaging device80ofFIG.8includes an array14, which is formed in several lines or preferably is formed in a single line, of adjacently arranged optical channels16a-d. Each optical channel16a-dincludes an optic22a-dfor imaging a respective partial field of view24a-dof a total field of view26, possibly also a total field of view as described in connection withFIG.5. The imaged field of view of the multi-aperture imaging device80is projected onto a respectively associated image sensor area28a-dof an image sensor12.

The image sensor areas28a-dmay be formed of a chip, for example, which includes a corresponding pixel array; as indicated inFIG.8, the chips may be mounted on a shared substrate or circuit board98. Alternatively, it would also be possible, for example, for the image sensor areas28a-dto be formed, respectively, from a part of a shared pixel array which extends across the image sensor areas28a-din a continuous manner or with interruptions, the shared pixel array being formed on a single chip, for example. For example, only the pixel values of the shared pixel array will then be read out in the image sensor areas28a-d. Various combinations of said alternatives are also possible, of course, e.g., the presence of one chip for two or more channels and of a further chip for yet other channels, or the like. In the event of several chips of the image sensor12, said chips may be mounted, e.g., on one or more circuit boards, e.g., all of them together or in groups or the like.

In the embodiment ofFIG.8, four optical channels16a-dare arranged, in a single-line manner, next to one another in the line extension direction of the array14; however, the number “four” is only exemplary and might also be any number larger than one, i.e., N optical channels may be arranged, wherein N>1. In addition, the array14may also comprise further lines which extend along the line extension direction. An array14of optical channels16a-dis understood to mean a combination of the optical channels, or a spatial grouping thereof. The optics22a-dmay each comprise a lens or a group of lenses or a stack of lenses as well as a combination of an imaging optic with further optical elements, including filters, diaphragms, reflective or diffractive elements or the like. The array14may be configured such that the optics22a-dare arranged, fixed or mounted on the substrate78in a channel-specific manner, in groups or in a manner that is global to all the channels, i.e., all channels together. This means that one single substrate78, several parts thereof or even no substrate78may be arranged, e.g. when the optics22a-dare fastened at a different location.

Optical axes and/or the optical paths102a-dof the optical channels16a-dmay extend in parallel with one another in between the image sensor areas28a-dand the optics22a-d, in accordance with an example. To this end, the image sensor areas28a-dare arranged within a shared plane, for example, as are the optical centers of the optics22a-d. Both planes are parallel to each other, i.e., parallel to the shared plane of the image sensor areas28a-d. In addition, with projection perpendicular to the plane of the image sensor areas28a-d, optical centers of the optics22a-dcoincide with centers of the image sensor areas28a-d. In other words, said parallel planes have the optics22a-darranged therein, on the one hand, and have the image sensor areas28a-darranged therein at identical pitches in the line extension direction.

An image-side distance between image sensor areas28a-dand the associated optics22a-dis set such that the projections onto the image sensor areas28a-dare set to a desired object distance. The distance preferably lies within a range equal to or larger than the focal width of the optics22a-dor, for example, within a range between the focal width and double the focal width of the optics22a-d, including both. The image-side distance along the optical axis102a-dbetween the image sensor area28a-dand the optic22a-dmay also be settable, e.g., manually by a user and/or automatically via a focusing means and/or autofocusing control.

Without any additional measures, the partial fields of view24a-dof the optical channels16a-doverlap essentially completely on the grounds of the parallelism of the optical paths and/or the optical axes102a-d. For covering a larger total field of view26, and in order for the partial fields of view24a-dto only partly overlap in terms of space, provision is made of the beam deflection means18. The beam deflection means18deflects the optical paths102a-dand/or optical axes, for example with channel-specific deviation, to a direction of the total field of view104. The direction of the total field of view104extends, for example, in parallel with a plane which is perpendicular to the line extension direction of the array14and is parallel to the course of the optical axes102a-dprior to, or without any, beam deflection. For example, the direction of the total field of view104results from the optical axes102a-dby a rotation about the line extension direction by an angle which amounts to >0° and <180° and preferably ranges between 80 and 100° and may amount to 90°, for example. The total field of view26of the multi-aperture imaging device80, which corresponds to the total coverage of the partial fields of view24a-d, therefore does not lie in the direction of an extension of the series connection of the image sensor12and the array14in the direction of the optical axes102a-d, but due to the beam deflection, the total field of view is located laterally to the image sensor12and the array14in a direction in which the installation height of the multi-aperture imaging device80is measured, i.e., the lateral direction perpendicular to the line extension direction.

In addition, however, the beam deflection means18deflects, e.g., each optical path, or the optical path of each optical channel16a-d, with a channel-specific deviation from the deflection which has just been mentioned and leads to the direction104. To this end, the beam deflection means18includes, for each channel16a-d, e.g., an element set up individually, e.g., a reflecting facet86a-dand/or a reflecting surface. These are mutually slightly tilted. Said mutual tilting of the facets86a-dis selected such that upon beam deflection by the beam deflection means18, the partial fields of view24a-dare provided with a slight divergence such that the partial fields of view24a-dwill only partly overlap. As is indicated by way of example inFIG.8, said individual deflection may also be such that the partial fields of view24a-dcover the total field of view26in a two-dimensional manner, i.e., are arranged within the total field of view26such that they are two-dimensionally distributed.

In accordance with a further embodiment, the optic22a-dof an optical channel may be set up to fully or partly generate the divergence in the optical paths102a-d, which enables fully or partly dispensing with the tilting between individual facets86a-d. If the divergence is provided fully, e.g., by the optics22a-d, the beam deflection means may also be formed as a planar mirror.

It shall be noted that many of the details described so far regarding the multi-aperture imaging device80have been selected to be exemplary only. This refers to, e.g., the above-mentioned number of optical channels. The beam deflection means18may also be formed differently than was described so far. For example, the beam deflection means18does not necessarily act in a reflective manner. It may also be configured differently than being in the form of a facet mirror, such as in the form of transparent prism wedges, for example. In this case, for example, the average beam deflection might amount to V, i.e., the direction104might, e.g., be parallel to the optical paths102a-deven prior to or without any beam deflection, or, in other words, the multi-aperture imaging device18might continue to “look straight ahead” despite the beam deflection means18. Channel-specific deflection by the beam deflection means18in turn would result in that the partial fields of view24a-donly slightly overlap, e.g., in a pairwise manner with an overlap of <10% in relation to the solid-angle ranges of the partial fields view24a-d.

Also, the optical paths102a-d, or the optical axes, might deviate from the parallelism described, and nevertheless, the parallelism of the optical paths of the optical channels might still be sufficiently pronounced so that the partial fields of view, which are covered by the individual channels16a-N and/or are projected onto the respective image sensor areas28a-d, would overlap, for the most part, without any further measures such as, specifically, beam deflection, so that in order to cover a larger total field of view by the multi-aperture imaging device80, the beam deflection means18provides the optical paths with an additional divergence such that the partial fields of view of N optical channels16a-N exhibit less mutual overlap. The beam deflection means18, for example, provides for the total field of view to comprise an aperture angle larger than 1.5 times the aperture angle of the individual partial fields of view of the optical channels16a-N. With some kind of pre-divergence of the optical paths102a-d, it would also be possible that, e.g., not all facet inclinations differ from one another, but that some groups of channels comprise the facets which have identical inclinations, for example. The latter may then be formed in one piece and/or may be formed to continually merge into one another, as a facet, as it were, which is associated with this group of channels which are adjacent in the line extension direction.

The divergence of the optical axes102a-dof said channels16a-dmight then stem from the divergence of these optical axes102a-das is achieved by the lateral offset between optical centers of the optics22a-dand image sensor areas28a-dof the channels16a-dor prism structures or decentered lens sections. The pre-divergence might be restricted to one plane, for example. For example, the optical axes102a-dmight extend, e.g., prior to or without any beam deflection18, within a shared plane, but may extend in a divergent manner within said plane, and the facets86a-donly cause additional divergence within the other transversal plane, i.e., they are all parallel to the line extension direction and are mutually inclined only in a manner that differs from the above-mentioned shared plane of the optical axes102a-d; again, several facets86a-dmay have the same inclination and/or might be jointly associated with one group of channels whose optical axes differ, in a pair-wise manner, for example as early as in the previously mentioned shared plane of the optical axes, prior to or without any beam deflection.

When the beam deflection means18is dispensed with or is configured as a planar mirror or the like, the entire divergence might be accomplished by the lateral offset between optical centers of the optics22a-d, on the one hand, and centers of the image sensor areas28a-d, on the other hand, or by prism structures or by decentered lens sections.

The above-mentioned pre-divergence which may possibly exist may be achieved, e.g., in that the optical centers of the optics22a-dare located on a straight line along the line extension direction, whereas the centers of the image sensor areas28a-dare arranged to deviate from the projection of the optical centers along the normal of the plane of the image sensor areas28a-donto points located on a straight line within the image sensor plane, e.g. at points which deviate from the points located on the above-mentioned straight line within the image sensor plane, in a channel-specific manner, along the line extension direction and/or along the direction perpendicular both to the line extension direction and to the image sensor normal. Alternatively, pre-divergence may be achieved, e.g., in that the centers of the image sensors28a-dare located on a straight line along the line extension direction, whereas the centers of the optics22a-dare arranged to deviate from the projection of the optical centers of the image sensors along the normal of the plane of the optical centers of the optics22a-donto points located on a straight line within the optical center plane, e.g. at points which deviate from the points located on the above-mentioned straight line within the optical center plane, in a channel-specific manner, along the line extension direction and/or along the direction perpendicular both to the line extension direction and to the normal of the optical center plane.

It is preferred for the above-mentioned channel-specific deviation from the respective projection to occur only in the line extension direction, i.e., for the optical axes102a-dwhich are located only within one shared plane to be provided with a pre-divergence. Both optical centers and image sensor area centers will then each be located on a straight line parallel to the line extension direction, but with different intermediate distances. A lateral offset between lenses and image sensors in the direction that is perpendicular and lateral to the line extension direction would result in an increase of the installation height, in contrast. A pure in-plane offset in the line extension direction does not result in a change in the installation height, but will possibly result in fewer facets and/or in that the facets will exhibit a tilt only in one angle orientation, which simplifies the architecture.

Advantageous implementations of the beam deflection means18will be described with reference toFIGS.9a-f. The explanations present a number of advantages which may be effected individually or in any combination, but which are not to have a limiting effect. In particular, it will become clear that the channel-specific relative position may be obtained by means of channel-specific positioning by means of channel-specific axial positioning of the individual facets which may be arranged rigidly, or may be jointly moved via individual axes and/or via a kinked or bent shared axis.

FIG.9ashows a schematic sectional side view of a beam deflection element172as may be employed for a beam deflection means described herein, e.g., the beam deflection means18. For example, the beam deflection element172is a facet86. The beam deflection element172may be operative for one, for a plurality of, or for all of the optical channels16a-dand may comprise a cross-section of the type of a polygonal chain. Even though a triangular cross-section is shown, said cross-section may also be any other polygon.

Alternatively or additionally, the cross-section may also comprise at least one curved surface; in particular with reflecting surfaces, it may be advantageous to have an implementation that is planar at least in portions so as to avoid aberrations.

For example, the beam deflection element172comprises a first side174a, a second side174band a third side174c. At least two sides, e.g., sides174aand174b, are configured to be reflective, so that the beam deflection element172is formed to be reflective on both sides. The sides174aand174bmay be main sides of the beam deflection element172, i.e., sides whose surface area is larger than that of the side174c. For example, the side174cmay be curved, i.e., may be convex starting from the axis176, so that due to the curvature, adaptation of the element to the rotation will occur, which will enable positioning the side174ceven closer to the edge side of the housing, which is advantageous with regard to the position of the passage area arrangement.

In other words, the beam deflection element172may be in the shape of a wedge and may be formed to be reflective on both sides. The face174cmay have a further face arranged opposite it, i.e., between faces174aand174b, which is substantially smaller than the face174c, however. In other words, the wedge formed by the faces174a, bandcdoes not taper in any random manner but is provided with a face at the pointed side and is therefore truncated.

When switching of the viewing directions is dispensed with, it is also possible to arrange a prism or a channel-specific mirror which has only one reflecting side.

FIG.9bshows a schematic sectional side view of the beam deflection element172, wherein a suspension or displacement axis176of the beam deflection element172is described.

The displacement axis176about which the beam deflection element172may be rotationally and/or translationally moveable within the beam deflection means18may be eccentrically shifted with regard to a centroid178of the cross-section. The centroid may alternatively also be a point which describes half of the dimension of the beam deflection element172along a thickness direction182and along a direction184perpendicular thereto.

The displacement axis may be unchanged, e.g., along a thickness direction182and may comprise any offset in a direction perpendicular thereto. Alternatively, an offset along the thickness direction182is also feasible. The displacement may be effected, e.g., such that upon rotation of the beam deflection element172about the displacement axis176, a longer adjustment travel is obtained than with rotation about the centroid178. Thus, the distance by which the edge between the sides174aand174bis moved upon rotation may increase, due to the displacement of the displacement axis176, given the same rotational angle as compared to a rotation about the centroid178. Preferably, the beam deflection element172is arranged such that the edge, i.e., the pointed side of the wedge-shaped cross-section, between the sides174aand174bfaces the image sensor. Thus, by means of small rotational movements, a respectively other side174aor174bmay deflect the optical path of the optical channels. Here it becomes clear that the rotation may be performed such that a spatial requirement of the beam deflection means along the thickness direction182is small since movement of the beam deflection element172in such a manner that a main side be perpendicular to the image sensor is not required.

The side174cmay also be referred to as a secondary side or rear side. Several beam deflection elements may be connected to one another such that a connecting element is arranged on the side174cor extends through the cross-section of the beam deflection elements, i.e., is arranged inside the beam deflection elements, e.g., in the area of the displacement axis176. In particular, the holding element may be arranged so as not to project, or to project to a small extent only, i.e., by a maximum of 50%, a maximum of 30% or a maximum of 10%, beyond the beam deflection element172along the direction182, so that the holding element does not increase or determine the extension of the overall setup along the direction182. Alternatively, the extension in the thickness direction182may be determined by the lenses of the optical channels, i.e., they have the dimension defining the minimum of the thickness.

The beam deflection element172may be formed of glass, ceramic, glass ceramic, plastic, metal or a combination of said materials and/or further materials.

In other words, the mean deflection element172may be arranged such that the tip, i.e., the edge between the main sides174aand174b, points toward the image sensor. A posture of the beam deflection elements may be such that it is effected only on the rear side or inside the beam deflection elements, i.e., the main sides are not covered up. A shared holding or connecting element may extend across the rear side174c. The rotational axis of the beam deflection element172may be arranged to be eccentric.

FIG.9cshows a schematic perspective view of a multi-aperture imaging device190including an image sensor12and a single-line array14of adjacently arranged optical channels16a-d. The beam deflection means18includes a number of beam deflection elements172a-d, which may correspond to the number of optical channels. Alternatively, a smaller number of beam deflection elements may be arranged, for example when at least one beam deflection element is used by two optical channels. Alternatively, it is also possible for a larger number to be arranged, for example when switching of the deflection direction of the beam deflection means18is effected by a translational movement. Each beam deflection element172a-dmay be associated with an optical channel16a-d. The beam deflection elements172a-dmay be imaged as a multitude of elements172in accordance withFIG.11. Alternatively, at least two, several or all of the beam deflection elements172a-dmay be formed to be integral with one another.

FIG.9dshows a schematic sectional side view of the beam deflection element172, the cross-section of which is formed as a freeform surface. Thus, the side174cmay comprise a recess186enabling attachment of a holding element; the recess186may also be formed as a protruding element, e.g., as a key of a slot-and-key system. In addition, the cross-section comprises a forth side174dcomprising a smaller area extension than the main sides174aand174band connecting the latter to each other.

FIG.9eshows a schematic sectional side view of a first beam deflection element172aand of a second beam deflection element172b, which is located behind the former in the representation direction. The recesses186aand186bmay be arranged to be essentially congruent, so that it is possible to arrange a connecting element in the recesses.

FIG.9fshows a schematic perspective view of the beam deflection means18comprising, e.g., four beam deflection elements172a-dwhich are connected to a connecting element188. The connecting element188may be used for being translationally and/or rotationally moveable by an actuator. The connecting element188may be configured to be in one piece and may extend at or within the beam deflection elements172a-dacross an extension direction, e.g., the y direction. Alternatively, the connecting element188may be connected only to at least one side of the beam deflection means18, e.g., when the beam deflection elements172a-dare formed in one piece. Alternatively, connection to an actuator and/or connection of the beam deflection elements172a-dmay also be effected in any other manner, e.g., by means of bonding, wringing or welding. It is also possible for the connecting element188to exhibit kinks or curvatures so as to set different relative positions of the beam deflection elements172a-dwith regard to a reference position (e.g., a distance from the edge side or the image sensor).

Inventive implementations will be explained in more detail with reference to the following embodiments.

FIG.10shows a schematic perspective view of a device100in accordance with an embodiment. The device100includes a housing108which completely or partly encloses or houses an inner volume. The housing108includes two oppositely located main sides108A and108B, which are spaced apart from each other by one or more secondary sides108C. In the case, depicted by way of example, of a cuboidal housing108, e.g., four secondary sides108C to108F may be arranged to connect and to space apart the main sides108A and108B. However, embodiments are not limited to cuboidal housings but may also comprise different shapes, e.g., a type of cylinder, which may result in one individual secondary side, a triangle with three secondary sides or a different polygonal chain having a random number of edge sides.

The device100includes a multi-aperture imaging device112which is arranged inside the housing108and may be configured in line with multi-aperture imaging devices described herein and which includes an image sensor arrangement comprising image sensor areas, the image sensor areas being associated with the respective optical channels. To this end, the multi-aperture imaging device112includes an array of adjacently arranged optical channels, each optical channel including an optic for projecting at least one partial field of view of a total field of view onto an image sensor area of the image sensor arrangement. The multi-aperture imaging device112further includes a beam deflection means for deflecting an optical path102of the optical channels, the beam deflection means comprising a plurality of facets, and each optical channel having a facet associated with it.

At least one of the main sides108A and/or108B comprises a passage area arrangement114. The passage area arrangement114includes one or more passage areas114i, wherein i≥1. What is depicted by way of example is the arrangement of one single passage area1141. The passage area arrangement114is configured to allow the optical paths102to pass. Thus, the passage area114may be at least partly transparent, e.g., for a wavelength relevant to the multi-aperture imaging device112. This means that the multi-aperture imaging device112may look through the housing108, which otherwise might be formed to be opaque, because of the passage area arrangement114. In total, this does not rule out the arrangement of diaphragms, filters or other optically effective elements within the passage area1141or the passage area arrangement114.

What is achieved in accordance with the invention is that the space requirement of the passage area arrangement114in the main side108A is small, preferably within a plane of the main side108A and perpendicular to the line extension direction66, so that a dimension116, e.g., in parallel with the x direction, is small. Alternatively or additionally, a distance118of a proximal edge122and/or a distance124of a distal edge126may be small in relation to the edge side108E so as to provide as large a measure as possible or as large a proportion as possible in terms of surface area of the main side108A for other purposes, e.g., for arranging a display or a display means. The proximal and distal edges122and126in this context relate to edges extending in approximately the same direction as the edge side related thereto, here108E, for example, and which are different from edges extending perpendicular thereto.

To this end, the multi-aperture imaging device112is configured, in accordance with the invention, such that the optical channels, with regard to the passage area arrangement114and along an axial direction, e.g., in parallel with the x direction, of the course of the optical path of the optical channels between the image sensor arrangement and the beam deflection means comprise a channel-specific relative position of the image sensor area of the optical channel, of the optic of the optical channel and/or of a position of the reflecting surface of the facet associated with the optical channel.

Several advantageous embodiments of this invention will be explained below in more detail.

FIG.11ashows a schematic top view of an implementation of the device100, wherein the main side108A of the housing108comprises two passage areas1141and1142, by way of example. Each of the passage areas1141and1142may be effective for at least one, but also for several or all of the optical channels, for example by subdividing the channels, so that one looks through at least two of the passage areas1141and1142(or further possible passage areas) at the same time. Alternatively, it is also possible to associate different passage areas1141and1142with different viewing directions or picture-taking modes, so that several or all of the optical channels at one point in time only look through a subset of all passage areas114i, for example one single passage area.

The passage areas1141and1142may comprise identical dimensions116, but may also be formed to differ from each other, as need may be, for example when a smaller dimension may be implemented for one of the passage areas1141or1142, without limiting the visual range of the multi-aperture imaging device. For example, the dimension124is larger than the dimension116, so that the passage areas1141and1142may exhibit a distance from the edge side108E.

FIG.11bshows a schematic top view of an alternative implementation of the device100, wherein the passage area arrangement comprises only one single passage area1141, which is arranged, e.g., to be centered and adjacent to the edge side108E. Here, too, the distance124between the edge side108E and the distal edge126may be larger than the dimension116of the passage area1141.

It is to be noted that the housing108is depicted as a cuboid, for example, so that the at least one edge side108C to108F may also be implanted to be straight. Alternative implementations comprise curved edge sides108C to108F, e.g., edge sides that are curved in a convex manner, i.e., toward the outside, in relation to the housing108. The distances118and124in this case relate to the edge or to the end of the main side108A, irrespective of how much the top view is increased by convexly curved edge sides.

As compared to the implementation inFIG.11a, the passage area1141may be configured to be larger, along the z direction, i.e., the line extension direction, than the passage areas1141and1142inFIG.11a, e.g., so as to allow a larger number of optical channels to pass at the same time.

FIG.11cshows a schematic top view of a further alternative implementation of the device100, wherein the total passage area1141is not arranged to be centered, unlike inFIG.11b, but to be offset from the edge side108F. In accordance with further alternative implementations, a passage area or several passage areas may be shifted toward any edge side when one uses the configuration ofFIG.11bas the basis. Preferably, the distance124is kept small, and a further additional distance from the edge side108F or from the edge side108D is small or is symmetric in relation to the edge sides108D and108F. This enables a remaining surface area of the main side108A to be either symmetric or largely uninterrupted, which results in a large amount of representable information when said remaining surface area is used as a display.

This means that the effects of the display recess which are referred to as notches may be kept small and may be shifted to a corner of the display or may be at least symmetrically arranged at an edge side.

It shall further be noted that the implementation that the edge sides108C and108E are smaller than the edge sides108D and108F may also be switched or turned around as desired.

FIGS.12ato12cshow implementations of the device100which are equivalent or at least similar to those ofFIGS.11ato11c; the dimension116of the passage areas1141and/or1142may be equal to the distance124, which means that the windows are entirely displaced toward the edge of the main side108A, which may possibly, but not necessarily be implemented such that at least one of the passage areas1141and/or1142extends into the edge side108E.

In other words,FIGS.12ato12cshow a display of a smartphone and the position(s) of the window(s) in a configuration wherein the passage areas are adjacent without an edge.

FIGS.11a,11b,11c,12a,12band12ceach show a view of the main side108A. In accordance with embodiments, the viewing direction of the multi-aperture imaging device is invariable.

In accordance with other embodiments, the viewing direction of the multi-aperture imaging device is variable on the basis of a movement of the beam deflection means, for example toward an opposite direction. Embodiments therefore involve providing the main side, which is not depicted inFIGS.11a,11b,11c,12a,12band12c, with passage areas in an identical or at least comparable manner so as to enable a corresponding viewing direction. A relative location of the beam deflection means may therefore be switchable between a first position and a second position. In the first position, the optical path may be deflected toward a first total field of view, and in the second position, it may be deflected toward a second total field of view; different passage area arrangements, possibly in different main sides of the housing, are passed through in each case. It may be advantageous to provide the passage area arrangement, i.e., the passage windows, with diaphragms. Respectively non-used passage areas, i.e., those passage area arrangements through which the optical paths are not directed in the current picture captured, may be set to be fully or partly non-transparent, i.e., they may be optically closed, by the diaphragm and an associated control means. To this end, mechanical and/or electrochromic diaphragms may be suitable. When the beam deflection means, in a first operating state, deflects the optical paths such that they pass through a first passage area arrangement, the second diaphragm may optically set the second passage area arrangement to be at least partly non-transparent. In the second operating state, the beam deflection means may deflect the optical path of the imaging device such that said optical path extends through the second passage area arrangement. The first diaphragm may then optically set the first passage area arrangement to be at least partly non-transparent, whereas alternately, the second passage area arrangement is set to be transparent.

In the following, reference shall be made to the channel-specific implementation of the multi-aperture imaging device.

To this end,FIG.13ashows an exemplary schematic sectional side view of the device100comprising sections of the main sides108A and1088for depicting the interior127of the housing108.FIG.13ashows, in the depicted sectional plane, a representation of a first optical channel, e.g. of the optical channel16acomprising the optic22a, wherein the distance88is provided between the image sensor12and the optic22a.

FIG.13bshows a schematic sectional side view of the same implementation of the device100as is shown inFIG.13a, but within a different sectional plane, namely within a plane having the value z2instead of z1along the z axis. There the optical channel16bis arranged with the optic22b. The optical channels16aand16bdiffer with regard to a distance128abetween the optic22aand the facet86aas compared to a distance128bof the optical channel16bbetween the optic22band the facet86b. The distance128may refer to the respective distance between the optic22and the reflecting surface of the facet86, while the inclination angle Φ may refer to a plane parallel to the main side108A, e.g., the x/y plane. Channel-specific implementation of the optical channels by means of a channel-specific relative position may include a channel-specific inclination angle of the optical path in the area located between the image sensor area arrangement and the beam deflection means in relation to a plane parallel to the main side, i.e., an out-of-plane course of the optical channels. Channel-specific is understood to mean that at least two channels, several channels or all of the channels differ from at least one other optical channel, several optical channels or all optical channels with regard to the relative position(s).

For example, the distance128bmay be larger, alternatively smaller, than the distance128a. Alternatively or additionally, a setting angle Φ1of the optical channel16amay differ from a setting angle Φ2of the optical channel16b, e.g. it may be larger. As a result, optical paths102aand102bwhich extend in parallel with each other or extend at least within the same x/z plane may be deflected in different directions. A distance132of the image sensor12and/or of the respectively associated image sensor areas from the edge side108E may be identical in the optical channels16aand16b, which is equivalent to a same or identical relative position of the respective image sensor area along the axial direction x. Instead of referring to the edge side108E, said same position may refer to the proximal edge122or the distal edge126.

In other words, embodiments of the present invention relate to the individual designs of the optical channels, so that a minimum window size and/or a minimum distance of the window edge (the distal edge126), which faces the optic, from the edge of the smartphone or its display results. This is achieved, e.g., in that an axial distance128and a setting angle ϕ of the reflecting facets86with regard to the optics22are selected such that a minimum window size (dimension116) and/or a minimum distance124of the window edge126, which faces the optic, from the edge of the smartphone or its display results for a distance, among other things, in accordance with the thickness134of the display of a smartphone. The distances of the optics from the image sensor and the optics may be the same for all of the channels.

FIGS.14ato14cshow schematic top views of different implementations of the device100including the different distances128aand128bbetween the optic22aand the facet86a, on the one hand, and between the optic22band the facet86b, on the other hand, which were mentioned in connection withFIGS.13aand13b. This is obtained in that, for example, the optics22aand22bare arranged at the same axial position, i.e., with the same x coordinate. Alternatively or additionally, the image sensor areas28aand28bmay also be arranged at the same axial position.

While inFIG.14a, the image sensor areas28aand28bare implemented as different chips which are directly adjacent to each other, however, a shared image sensor12may be used in accordance with the configuration shown inFIG.14c, i.e., a constructional gap may or may not be arranged between the image sensor areas28aand28b. In accordance with a distance or a direct neighborhood of image sensor areas, the optics22aand22bmay also be arranged close to each other, as depicted inFIG.14a, or may be arranged at a mutual distance, as explained inFIGS.14band14c, for example. Optionally, the array of optical channels may comprise the shared substrate78which, e.g., is transparent and has the optical paths passing through it.

The facets86aand86bmay also be arranged directly adjacently to each other, as depicted inFIG.14a, or may be spaced apart from each other as depicted inFIGS.14band14c.

The representation of modules consisting of image sensor, optic and facet in accordance withFIG.14a, a distance between the individual channels inFIG.14band a continuous image sensor and a shared array optic inFIG.14cmay be combined in any manner desired.

The different distances128aand128bmay be compensated for, while keeping the distances between the optics and the image sensor areas, by differently formed optics, e.g., with regard to the focal width, so that adaptation of the optics to the different distances, or vice versa, may be implemented.

FIGS.15aand15bshow schematic sectional side views of a configuration of the device100in accordance with an embodiment, wherein a different axial position of the combination, i.e., of the total arrangement, of the image sensor area28, the optic22and the associated facet86of the different optical channels16aand16balong a direction of the optical paths102aand102bin an area located between the image sensor arrangement comprising the image sensor areas28aand28band the beam deflection means provides the channel-specific relative position.

This means that the distance88as well as the distance128may be the same in the optical channels16aand16b, as can also be seen inFIG.15c, which shows a schematic top view of the configuration which is depicted within different planes of the z axis inFIGS.15aand15b. The optical channels16aand16bmay thus be formed to be the same, e.g., with regard to the spacing of the components, but may be mutually offset in total as a group along the x direction, so that different distances136aand136bresult between the facets86aand86band the edge side108E.

Even though the image sensor areas28aand28bare depicted as mutually different chips, this will not prevent the image sensor areas28aand28bfrom being configured within a shared substrate, for example by implementing a step shape or the like. Likewise, the optic22amay be connected to the optic22bwhile using a shared substrate.

This embodiment enables implementing the optics22aand22bin the same manner, for example with the same focal width and/or the same capturing range. However, it is also possible to combine this embodiment with the different distances described in connection withFIGS.13a,13b,14a,14band14c. Alternatively or additionally, for example, only one axial position of the facet along the direction of the optical path102may differ in the area located between the image sensor12and the beam deflection means, and/or a distance between the respective optic22and the image sensor may be adapted.

In other words, alternatively or additionally, the axial position of the entirety consisting of the image sensor, the optic and the deflection facet may be formed differently, i.e., individual channels may comprise the same optics or distances, but may comprise no image sensor within a shared plane, for example no shared continuous image sensor.

FIGS.16aand16bshow schematic sectional side views of a configuration of the device100within different planes z1and z2of an x/y plane, wherein the image sensor areas28aand28bcomprise different distances1321and1322from the edge side108E. In addition, the optics22aand/or22bhave different distances881and882from the respectively associated image sensor areas28aand28b.

Alternatively or additionally, the distances1281and1282between the optic22aand the facet86aand/or between the optic22band the facet86bmay differ from each other so as to implement a corresponding adaptation of the imaging properties of the optical channels16aand16b. Alternatively or additionally, potentially present dissimilar dimensions of the optics22aand22balong the x direction may be compensated for by the different distances, e.g., when using optics22aand22bwhich have different properties.

In other words, optics, axial optic positions and/or axial image sensor positions may be individual for, or specific to, all channels.

Individuality, or specificity, may be exploited in that projections of beams of rays138aand138bof the different optical channels into a shared x/y plane, i.e., with a projection along the line extension direction and/or the z direction, overlap to a large extent, in particular to a larger extent than in an implementation that is global to all channels, i.e., in an implementation wherein the axial positions of the image sensor areas, of the optics and of the facets are the same, even if setting angles of the facets may differ. Due to the large amount of overlap, the window size required, the dimension116, may be small, which is advantageous in particular with large displays, or when a surface is used as a display to a large extent. This means that the passage area arrangement may be adapted to the increased overlap obtained—in particular, it may be reduced in size. The passage area arrangement may be configured such that unimpeded passage of the optical paths is enabled in an unchanged manner.

FIG.17ashows a schematic top view of a configuration of the device100, wherein the optics22aand22bare each formed as a stack of lenses connected to one or more substrates781and/or782within different planes along the x direction. On the basis of the different distances136aand136bof the facets86aand86barranging of the image sensor areas within a shared x plane, possibly as a shared image sensor12, may be enabled; the optics22aand22bmay be individually designed here, e.g., with regard to their focal widths and/or with regard to the obtained distances from the facets and/or from the image sensor. For example, the individual lenses142iof the optics22aand22bmay have different sizes and/or focal widths. As a result, in combination with several individual lenses, any desired optical property of the total optics22aand22bmay be obtained, which are adapted to the mutually different distances136aand136b.

In other words, the optics may comprise shared continuous substrates781and782; arranged on the latter, the lenses142iof the individual channels may be configured in a channel-specific manner and may also comprise different distances from the image sensor12. The substrates781and782may be arranged perpendicularly to the optical axis, which means perpendicularly to the axial direction x.

FIG.17bshows a schematic top view of a configuration wherein the substrate781and/or782is tilted, as compared to the perpendicular arrangement shown inFIG.17a, in a manner that is perpendicular to the axial direction, which also enables adapting the optical paths of the individual optical channels to the mutually different distances of the different channels between the optics22aand22b, respectively, the associated facets86aand86b, respectively, and/or the respectively associated image sensor area, or image sensor12. The optics may comprise rotationally asymmetric bodies while being configured such that the surface profiles of lenses142iof the optics22aand22bmay be described to be rotationally symmetric about a direction parallel to the x direction, i.e., rotationally symmetric to the optical axes of the optical channels. This may be obtained, for example, by compensating for the tilting by means of the substrates781and782by means of wedge shapes144, as a result of which a rotationally asymmetric body is obtained which, however, may result in a rotationally symmetric optic when combined with the tilted substrate781and/or782.

By contrast, adaptation, or channel-specific implementation, in accordance withFIG.17amay be performed such that the optics22aand22bare formed to differ from each other and are arranged along different axial positions along the x direction so as to adapt the optical paths of the individual channels to the mutually differing distances between the optics, the associated facets and/or the respectively associated image sensor area.

In other words, the substrates, as depicted inFIG.17b, may be arranged in a non-perpendicular manner in relation to the optical axes, and the lenses of the optics may be configured to comprise a wedge angle, so that the surfaces of the lenses are formed to be rotationally symmetric to the optical axes of the channels.

WhileFIG.17ashows a continuous image sensor, a shared array optic with continuous substrates,FIG.17bshows a combination of different embodiments.

Embodiments enable a minimum width of the shared exit window (of the passage area arrangement114) of all channels for improved utilization of the surface and maximization of the display area. This may also involve improved aesthetics. Embodiments may be used in multi-aperture imaging systems comprising linear channel encoding and minimal installation size, in particular for cameras for smartphones in the direction facing the user. In accordance with embodiments, a display means, or a display, is arranged on the main side which covers at least 90%, at last 93% or at least 95% of the main side108A and at least partly encloses the passage area arrangement. For example, the passage area arrangement is not enclosed by the display only on that side which faces the edge side108E.

As can be easily seen, e.g., inFIGS.14a,14b,14c,15c,17aand17b, the multi-aperture imaging device may be configured such that an overlap of the facets86, which form the end along the axial direction x, across the passage area arrangement114is small or not required. Therefore, embodiments enable arranging the beam deflection means to be adjacent to the edge side108E and also arranging the passage area arrangement, despite the very small size, to be adjacent to the edge side. The edge126which is distal in relation to the edge side may have a distance from the edge side108E which is influenced by several properties of the device, which include:a: on the side of the optic of the optical channels, a size, e.g., a diameter, of an aperture diaphragm used;b: a thickness of the exit window, see, e.g., the dimension along y inFIGS.13a-b,FIGS.15a-bandFIGS.16a-b;c: a total angle of field of the total field of view along the longest side of the housingd: the number of partial images captured along this longest side of the housing;e: an optional super-resolution factor.

The distance of the distal edge may amount to:
3a+4b*tan(c/2*d*e+(d/e−1)*5°)
or less, which may also be expressed as the sum of three times the diameter of the aperture diaphragm of the optic and the product of four times the window thickness, multiplied by the tangent of the angle, which corresponds to half the visual field angle along the longest side of the housing, divided by the number of partial images in the direction of the longest side of the housing, multiplied by the factor of the super resolution plus the number of images which is divided by the factor of the super resolution, in the direction of the longest side of the housing, minus one, multiplied by 5°.

As is shown inFIGS.12ato12c, the proximal edge122may be made to come close to the edge side108E, so that the distance comprises a value of, at the most, double the diameter of the aperture diaphragm. It may be particularly preferred for the proximal edge to reach or overlap the edge side.

In other preferred embodiments, the proximal edge122has a distance from the edge side108E which amounts to, at the most, double the diameter of the aperture diaphragm.

Even though embodiments described herein may be configured such that the multi-aperture imaging device may be implemented without duplicating the channels for stereo capturing, other embodiments envisage providing at least one second multi-aperture imaging device for capturing the total field of view at least in a stereoscopic manner. Alternatively or additionally, a multi-aperture imaging device or a combination of at least two multi-aperture imaging devices may be configured such that a first group of optical channels is set up for capturing a first partial field of view or the total field of view, whereas a second group of optical channels is set up for capturing a second partial field of view or the total field of view. The device may be configured, while using a corresponding control unit, to join image information of image sensor areas of the first group and/or of the second group for increasing the resolution. Instead of or in addition to generating a depth map, one may therefore obtain an increase in the resolution in the sense of a super resolution.

The position of the beam deflection means may determine a size, at least along one direction, of the multi-aperture imaging device, e.g., along the axial direction x between the image sensor arrangement12and the beam deflection means18. The beam deflection means may be arranged adjacently to the edge side, which enables arranging the passage area arrangement to also be adjacent to the edge side. This may occur alternatively or in addition to channel-specific implementation, so that embodiments refer to a device comprising a housing having two opposite main sides, the main sides being connected to each other via at least one edge side.

The device includes, e.g., a multi-aperture imaging device which is arranged inside the housing and comprises an image sensor arrangement having at least two image sensor areas, an array of adjacently arranged optical channels, each optical channel including an optic for projecting at least one partial field of view of a total field of view onto an image sensor area of the image sensor arrangement. The multi-aperture imaging device includes a beam deflection means for deflecting an optical path of the optical channels, the beam deflection means comprising a plurality of facets, each optical channel having a facet associated with it. The device is configured such that at least one of the main sides comprises a passage area arrangement having at least one passage area, the passage area arrangement being set up for allowing the optical paths to pass through. A position of the beam deflection means determines, at least partly, a size of the multi-aperture imaging device along an axial direction between the image sensor arrangement and the beam deflection means. The beam deflection means is arranged adjacently to the edge side. The passage area arrangement is arranged in an edge area of the main side that is adjacent to the edge side. Said implementation may be readily combined with the size and/or arrangement of the passage areas, of the proximal and distal edges thereof.

Inventive methods of providing a device include providing a housing comprising two oppositely located main sides which are connected to each other via at least one edge side, so that one of the main sides comprises a passage area arrangement having at least one passage area, the passage area arrangement being set up for allowing the optical paths to pass. The method includes arranging a multi-aperture imaging device in an interior of the housing, said arranged multi-aperture imaging device comprising: an image sensor arrangement; an array of adjacently arranged optical channels, each optical channel including an optic for projecting at least one partial field of view of a total field of view onto an image sensor area of the image sensor arrangement, and a beam deflection means for deflecting an optical path of the optical channels, the beam deflection means comprising a plurality of facets, each optical channel having a facet associated with it. The method is performed, in one embodiment, such that the optical channels comprise, in relation to the passage area arrangement and along an axial direction, along a course of the optical path of the optical channels between the image sensor arrangement and the beam deflection means, a channel-specific relative position of at least one of: the image sensor area, the optic, and a reflecting surface of a facet associated with the optical channel. In accordance with a further embodiment, the method may alternatively or additionally be performed such that a position of the beam deflection means determines a size of the multi-aperture imaging device along an axial direction between the image sensor arrangement and the beam deflection means, and the beam deflection means is arranged adjacently to the edge side. The passage area arrangement is arranged within an edge area of the main side adjacently to the edge side.

In accordance with embodiments, the exit windows may be designed to be narrow and/or may be arranged to be close to surface edges of a housing side, so that the display may be arranged to have a large surface area, that interruptions thereof may be dispensed with, and that a large amount of the surface may be used for the display. Embodiments are advantageous, in particular, when the cameras are used in smartphones in the direction facing the user (on the user side, which is located opposite a world side) since the exit window(s) is/are as narrow as possible and/or is/are arranged as close to the edge of the smartphone as possible, so that in total, as large as possible a display area with as little interference, so-called notches, as possible results. The windows may contain one, several or all of the optical paths of the multi-aperture cameras, i.e., there are one or several windows.

The multi-aperture cameras comprising linear channel arrangement comprise several optical channels arranged next to one another and transferring parts of the total field of view or a partial field of view in each case. Advantageously, the imaging lenses may have a mirror mounted in front of them which may be used for beam deflection and contributes to reducing the installation height.

One aspect of the present invention relates to designing the components such that the exit window(s) of the imaging channels is/are as narrow as possible, in particular along the direction of the optical axes in the area located between the imaging optics and the beam deflection means. In addition, the components may also be configured such that the distance, of that edge of the window(s) which is arranged closest to the optics, from the edge of the smartphone becomes as small as possible. The configuration of the windows may be useable for all of the variants of multi-aperture cameras which have been described or derived, i.e., the partial fields of view which are imaged by the individual channels and which partly overlap are possibly, but not necessarily, stitched to form a total field of view. Application as a camera (in the near-infrared field) for face recognition or face identification (face-ID) by generating a depth map/depth information/depth profile of a face (in connection with suitable (structured) illumination). Partial images need not necessarily be stitched (on a depth-based manner) to form a total field of view but may also be individually used for calculating the respective (partial) depth map, the total depth map possibly being stitched at a later point in time. Given a sufficiently specified, limited and known working distance (e.g., distance of the face from the smartphone), parallax-based image stitching is not necessary anyway, or is necessary to a very small extent only, but the partial images may be stitched to form a total image for this corresponding one object distance only.

There may be several channels or modules, so that the total field of view may be scanned several times, so that stereo, triple, quattro and even more often repeated scanned arrangements may result.

The optics of the channels regarding the above-mentioned architectures may achieve a higher spatial resolution than may be resolved by using the pixels of the image sensors used in the channels, so that aliasing results. By scanning the total field of view several times, methods of super resolution may thus be used, and improved spatial resolution may be achieved.

The total fields of view of individual channel groups may differ in size and may fully overlap. Alternatively or additionally, the one-dimensional vectors of the arrangement of the channels and of the partial fields of view may be arranged perpendicularly and/or in parallel.

Even though some aspects were described in connection with a device, it shall be understood that said aspects also represent a description of the corresponding method, so that a block or a component of a device is also to be understood as a corresponding method step or as a feature of a method step. By analogy therewith, aspects that were described in connection with or as a method step also represent a description of a corresponding block or detail or feature of a corresponding device.