Patent Description:
Generally, camera system architectures are known which are used, for example, as wearable cameras. Known cameras typically have a CMOS (Complementary Metal Oxide Semiconductor) image sensor, CCD (Charge Coupled Device) image sensor, or the like.

Moreover, in digital imaging it is known to directly measure, for example, the intensity of a scene with a pixel array sensor which is based on the CMOS, CCD or similar technology mentioned above. In contrast to direct measuring, where every pixel of the sensor is read out, other methods are known, such as compressive sensing.

Compressive sensing (CS) is a signal processing method which typically allows capturing and representing signals with a small number of measurements. CS is useful for image capturing devices (e.g. for lifelog applications) as it allows to reduce battery usage and required storage.

However, in CS, typically, the original image is not available and a computationally intensive reconstruction has to be run in order to retrieve it.

<NPL>] discloses a method for reconstructing a signal that has been measured as a measurement vector in a sparse domain by compressive sensing. The signal is reconstructed by applying a measurement matrix to the measurement vector, wherein the measurement matrix is chosen so that the representation of the reconstructed signal in a sparsity basis is as sparse as possible. The measurement matrix and the sparsity basis are obtained by deep learning.

<NPL> discloses extracting deeply learned features from compressive measurements. Training data is expressed as product of at least one dictionary matrix and a coefficient matrix. The at least one dictionary matrix and the coefficient matrix are learnt by deep matrix factorization from compressed measurements of the training data.

<CIT> discloses an image capture device that incorporates controls for an operation of the image capture device and a learning/predictor element, wherein the learning/predictor element is adapted to analyse use of the controls to learn a user's image capture behaviour and to predict an image capture opportunity of interest to the user.

<NPL> discloses a framework for adaptive and non-adaptive statistical compressive sensing, where a statistical model replaces a standard sparsity model of classical compressive sensing. A two-step adaptive sensing paradigm includes applying online sensing to detect a signal class in a first step, followed by a reconstruction step adapted to the detected class and a observed samples.

<NPL> discloses that convolutional neural networks (CNNs) can be employed to extract discriminative non-linear features directly from Compressive Sensing (CS) measurements. Using these features, effective high-level inference can be performed.

Magalhães, <NPL> discloses tailoring the concept of compressive sensing to assess the intrinsic discriminative capability of this method to distinguish human faces from objects.

<CIT> discloses a face recognition intelligent module based on an embedded type platform. The module comprises an embedded type processor module, as well as a camera interface module and a main controller interface module, which are electrically connected with the embedded type processor module. The camera interface module is connected with an externally-connected camera; and the main controller interface module is connected with an externally-connected controller. The face recognition intelligent module performs face recognition based on visible light images, can utilize a common camera which is easily to be accepted by a tested object, and has higher invisibility than a near-infrared based system, so as to greatly expand the application range.

Although there exist techniques for using compressive sensing in digital imaging, it is generally desirable to improve compressive sensing capturing devices and methods.

According to a first aspect, the disclosure provides a compressive sensing capturing device in accordance with independent claim <NUM>.

According to a second aspect, the disclosure provides a compressive sensing capturing method in accordance with independent claim <NUM>.

Further aspects are set forth in the dependent claims, the drawings and the following description.

Before a detailed description of the embodiments under reference of <FIG>, general explanations are made.

As mentioned in the outset, camera system architectures are known, which are used, for example, as wearable cameras, smartphones, or the like. Known cameras typically have a CMOS (Complementary Metal Oxide Semiconductor) image sensor, CCD (Charge Coupled Device) image sensor, or the like.

In contrast to direct measuring, where every pixel of the sensor is read out, other methods are known, such as compressive sensing. Compressive sensing (CS) is a signal processing method which allows capturing and representing signals with a small number of measurements. CS is especially useful for image capturing devices (e.g. for lifelog applications), as it allows to reduce battery usage and required storage.

CS is based on the principle that a signal that is sparse in a domain (time domain, spatial domain, etc.) can be accurately recovered from much fewer samples as compared to the original number of samples in the sparse domain. A signal is sparse if it contains a small number of nonzero components relative to the total signal length. In other words, the signal is mostly zero. The principles of compressive sensing and of sparsity are well known in the art.

However, after applying CS, the original image is not available and a computationally intensive reconstruction has to be run in order to retrieve it again. The reconstruction may be based on an optimization process, e.g. based on l1-norm- approximation, or by using deep neural networks. An example is the so-called NESTA approach (<NPL>) or any other approach which is known in the art, e.g. matching pursuit algorithms, orthogonal matching pursuit algorithms, etc. Hence, in the prior art, this typically prohibits many algorithms from being directly used on the capturing device (e.g., edge blur measurement) and, therefore, applying them for automatic camera features (e.g., autofocus) may not be easy.

Conventional cameras typically feature automatic functions such as autofocus, but also others, e.g. auto-adjust of luminance, aperture size and the like, etc. However, traditional algorithms are applied on raw images and cannot be used with a CS camera, since, as mentioned, after application of the CS the image is not present anymore, but only a CS representation from which the image has to be reconstructed.

Typically, e.g. for implementing a passive autofocus feature, the blurriness of the original image is to be measured (e.g. by contrast detection). Modern digital cameras also use advanced features to adjust the focus such as the detection and tracking of faces and special objects. In some embodiments, these features are not directly available in the images that are captured by the CS camera. A possible solution would be to reconstruct the original image with CS methods, as mentioned above, and then to use traditional algorithms to perform the automatic functions for the device, but this would be prohibitively slow for functions like autofocus and is also not battery efficient, since the reconstruction of the CS data is rather computational intensive, as mentioned above.

Hence, the invention pertains to the idea of using machine learning, based on an artificial neural network in the compressed domain, which enables automatic features for CS capturing devices, such that, for example, automatic picture camera/video camera features can be implemented. By using machine learning, according to the invention, image features, e.g. the blurriness, are directly measured based on the compressed measurements and therefore a faster automatic adjustment, e.g. of the autofocus, may be achieved. Consequently, the invention pertains to a compressive sensing capturing device, including circuitry configured to obtain compressive sensing image data, and to set a device attribute based on image attribute data, wherein the image attribute data are based on a machine learning algorithm, based on an artificial neural network performed in the compressing domain on the obtained compressive sensing image data.

As mentioned, compressive sensing is generally known and the compressive sensing capturing device or its circuitry is configured to generate compressive sensing image data.

The compressive sensing capturing device may be a digital (video) camera, a smartphone, a laptop computer, a tablet computer, a personal computer, a wearable electronic device, electronic glasses, or the like.

The circuitry includes an image sensor, which may be based on CCD or CMOS technology mentioned above, and may include one or more processors, logical circuits, memory (read only memory, random memory, etc., storage, e.g. hard disk, compact disc, flash drive, etc.), an interface for (wireless, e.g. Bluetooth, infrared) communication via a network (local area network, wireless network, internet), etc..

The compressive sensing image data are generated by the image sensor.

The compressive sensing image data are based on the compressive sensing approach mentioned above. According to the invention, no reconstruction method is performed on the compressive sensing image data before the device attribute is set on the basis of this compressive sensing image data. The compressive sensing image data originate from an image sensor which takes one or more images and generates the compressive sensing image data. Hence, the compressive sensing image data may represent one or more images, while the compressive sensing image data may even represent only parts of an image, e.g. blocks or the like, in some embodiments.

The device attribute includes at least one of: autofocus, luminance, shutter speed, aperture size, shoot mode, flash mode, object detected, etc. According to the invention, the device attribute is used to further control the CS capturing device, e.g. optical elements for autofocus, aperture size for illumination, shutter speed for setting illumination time, tracking an object for autofocus, etc..

The image attribute data, which are used to set the device attribute, are derived with a machine learning algorithm which is performed in the compressing domain on the obtained compressive sensing image data. The machine learning algorithm is based on an artificial neural network, e.g. a convolutional neural network (CNN) or the like, wherein the disclosure herein basically exemplary discusses the convolutional neural network as an example of machine learning algorithm without limiting the disclosure to convolutional neural networks.

The machine learning algorithm may be trained in advance, in some embodiments. For example, default images having defined image attributes, e.g. defined blur, defined objects, defined contrast, defined brightness, etc., may be used for generating the compressive sensing data and for training the machine learning algorithm. In some embodiments, the compressive sensing image data may be reconstructed and based on the reconstructed image a feedback is given to the machine learning algorithm for training of the machine learning algorithm.

The machine learning algorithm may include specific characteristics, classes or the like. In some embodiments, the machine learning algorithm is specialized to identify a specific image attribute, such as identifying predefined objects (people, houses, food, pet, etc.), identifying blur, contrast, brightness, identifying faces, smile in the face, sad face, lucky face, identifying eyes, etc. in the compressive sensing image data. In some embodiments, the machine learning algorithm is able to detect multiple image attributes and the image attribute data is representative of at least one image attribute.

As mentioned, according to the invention, the machine learning algorithm is based on an artificial, e.g. convolutional neural network using compressive sensing image data as input. The compressive sensing image data may be block-based. In some embodiments, the image sensor may provide the compressive sensing image data block-based, e.g. since one image is divided into multiple blocks, which may be an intrinsic feature of a block-based compressive sensing algorithm. In some embodiments, each single block of the multiple blocks included in the compressive sensing image data is used as input for the artificial, e.g. convolutional neural network. For each block of compressive sensing image data at least one measurement is provided to the artificial, e.g. convolutional neural network, wherein in some embodiments at least two different measurements may be provided to the artificial, e.g. convolutional neural network, wherein, for example, the different measurements result from applying respective compressive sensing matrices, as it is known from compressive sensing.

As mentioned, the circuitry further includes a compressive imaging sensor, which provides compressing sensing image data, and the device may be configured as digital camera, which may be configured to capture still images and/or moving (video) images.

As also mentioned above, the circuitry is further configured to control the image sensor (or the device) based on the set device attribute.

The invention pertains to a compressive sensing capturing method, which may be performed by the compressive sensing capturing device described herein, any other electronic device or a processor or other computing means. The method includes obtaining compressive sensing image data; and setting a device attribute based on image attribute data, wherein the image attribute data are based on a machine learning algorithm performed in the compressing domain on the obtained compressive sensing image, as discussed above. As mentioned, the device attribute includes at least one of: autofocus, luminance, shutter speed, aperture size, shoot mode, flash mode, object detected. The machine learning algorithm is based on an artificial, e.g. convolutional neural network using compressive sensing image data as input, as discussed above, without limiting the present disclosure in that regard (see discussion above). The compressive sensing image data may be block-based, wherein one block of compressive sensing image data may be used as input for the artificial, e.g. convolutional neural network, as discussed above. For each block of compressive sensing image data at least one measurement is provided to the artificial, e.g. convolutional neural network, wherein in some embodiments at least two different measurements are provided to the artificial, e.g. convolutional neural network. The method includes generating the image attribute data by performing the machine learning algorithm, which may also be trained in advance, as discussed above. The image attribute data are generated by performing the machine learning algorithm on the obtained compressive sensing image data, which may be obtained from a compressive imaging sensor, as discussed above. The image sensor is controlled based on the set device attribute.

Returning to <FIG>, an exemplary compressive sensing image sensor <NUM> is schematically illustrated, which is based on the CMOS technology. The image sensor <NUM> has a controller <NUM>, and several pixels <NUM>, which are arranged in a pixel array in a sensor plane. Each pixel <NUM> is coupled to the controller <NUM> via a column selection (col. ) and a row selection (row sel. ), via which each pixel is individually selectable by the controller <NUM>. The output from the pixels <NUM> is fed into a read out circuit <NUM> having exemplary a sign inverter <NUM>, an analog adder <NUM> and an ADC <NUM> (Analog to Digital Converter). The output from the pixels <NUM> are fed to the sign inverter, then into the analog adder <NUM>, and from the analog adder <NUM> the image data read out from the pixels <NUM> are fed into the ADC <NUM>, from which image data <NUM> are output by the compressive sensing image sensor <NUM>, which are assumed to be compressive sensing image data in this embodiment.

<FIG> illustrates a compressive sensing capturing device, which is configured as a digital camera <NUM> in this embodiment. The digital camera <NUM> has the image sensor <NUM>, which feeds the compressive sensing image data to a CPU <NUM> (Central Processing Unit). Moreover, a memory <NUM>, an interface <NUM> and a display <NUM> are provided, which are also coupled to the CPU <NUM>.

The memory <NUM> includes a storage medium, e.g. a flash drive, as well as a working memory, which may have a random access memory section and a read-only memory section.

The interface <NUM> is configured to provide an external communication, e.g. via USB (universal serial bus), a wireless network connection, a Bluetooth connection or the like.

The display <NUM> is configured as a touch display for controlling the digital camera <NUM> and displaying images, without limiting the present disclosure in that regard (the display may be based on LCD (liquid crystal display), (O)LED (/organic) light emitting diode), etc. technology).

<FIG> illustrates a processing scheme <NUM> which may be performed by the digital camera <NUM> (image sensor <NUM> and CPU <NUM>, for example) of <FIG>.

A real world scene <NUM> is photographed and the image sensor <NUM> takes an image of the real world scene <NUM> and outputs compressed sensing image data <NUM>. The compressive sensing image data <NUM> includes three image maps <NUM>, <NUM>, <NUM> without limiting the disclosure in that regard. The three image maps <NUM>, <NUM> and <NUM> correspond to three different measurements which are based on a block-based compressive sensing. Each image map <NUM>, <NUM> and <NUM> includes multiple blocks from the block-based compressive sensing method, wherein exemplary three blocks 23a, 23b and 23c are illustrated in <FIG>. Hence, in the present embodiment, the compressive sensing image data <NUM> includes data which may be organized in a tensor-like structure, wherein blocks are vertically and horizontally arranged within one image map (e.g. x- and y- axis), e.g. image map <NUM>, and different measurements, e.g. image maps <NUM> and <NUM>, are arranged in a third dimension (z-axis).

The compressive sensing image data <NUM> are fed to a machine learning algorithm <NUM> which, thus, works in the compressive sensing domain, since the compressive sensing image <NUM> are not reconstructed before they are input into the machine learning algorithm.

As discussed, a possible machine learning method that can be used is a convolutional neural network (CNN) with the compressed measurement, i.e. the compressive sensing image data <NUM>, as input. As mentioned, when measurements, i.e. compressive sensing image data <NUM>, are acquired from block-based compressed sensing, it is possible to arrange the measurements in a three-dimensional tensor where the two first dimensions are the horizontal and vertical blocks and the last dimension (depth dimension) corresponds to each measurement of the corresponding block.

Using the traditional CNN terms, each input map <NUM>, <NUM>, <NUM> corresponds to one measurement for each block. This arrangement allows to preserve the spatial integrity of the original image and is suited for the CNN design.

The principle of the usage of the CNN at <NUM> is also illustrated in <FIG>, which shows generally in the first line the CNN structure, and in the second line the basic principle of building blocks. The principles of a CNN and its application in imaging is generally known and, thus, it is only briefly discussed in the following under reference of <FIG>.

The input image includes for example three maps or layers (exemplary red, green and blue (RGB) color information) and N times N blocks. The CNN has a convolutional layer and a subsequent pooling layer, wherein this structure can be repeated, as also shown in <FIG>. The convolutional layer includes the neurons. By applying a filter kernel (see convolution kernels in the second line) on the input image, a respective feature map can be obtained. The pooling layer, which is based in the present embodiment on the Max-Pooling (see second line, "Max-Pooling"), takes the information of the most active neurons of the convolutional layer and discards the other information. After several repetitions (three in <FIG>), the process ends with a fully-connected layer, which is also referred to as affine layer. The last layer includes typically a number of neurons, which corresponds to the number of object classes (output features) which are to be differentiated by the CNN. The output is illustrated in <FIG>, first line, as an output distribution, wherein the distribution is shown by a row of columns, wherein each column represents a class and the height of the column represents the weight of the object class. The different classes correspond to the output or image attribute features which may be represented by the image attribute data <NUM>, which are output by the CNN <NUM>. The classes are, for example, "people, car, truck, etc." Typically, several hundred or several thousand of classes can be used. In some embodiments, the CNN <NUM> outputs a feature map with estimated classes and/or blur radius values for each image pixel (location), which may be particularly useful in embodiments pertaining to applications such as autofocus.

The output of the CNN <NUM> is used by a device control <NUM> (e.g. formed by the CPU <NUM> in <FIG>) to set one or more device attributes, such that at least one of the following can be realized: autofocus, auto-adjustment of luminance/shutter speed/aperture size, auto-choosing of shoot mode for picture/video mode, auto-choosing of flash mode, capture picture if smile is detected, capture picture if face is detected, capture picture if pet is detected, etc..

For example, the autofocus function can be implemented by detecting an object, such as a face, and detecting the blur of the face with the CNN <NUM> and setting the device attribute such that the detected object (e.g. face) is automatically focused (and thereby an autofocus feature of the digital camera <NUM> is implemented). Hence, for example, the blurriness is directly measured in the compressive sensing domain based on the compressive sensing image data <NUM>, which are directly used as input to the CNN <NUM>. As discussed above, as the blurriness is measured without reconstructing the compressive sensing image, the digital camera <NUM> may provide a faster autofocus.

Similarly, the other camera features or device attributes may be realized. For example, the auto-adjustment of the luminance/shutter speed/aperture size may be based on the measurement of brightness and/or contrast in the compressive sensing image data and based on the CNN <NUM>, etc. The skilled person knows which image features are to be extracted for providing respective device attributes or camera features and, thus, the skilled person is able to provide a respective CNN which is able to extract the needed image attribute data which are needed for setting a respective device attribute (s).

After having set the device attribute accordingly, the device control <NUM> controls the image sensor <NUM> to output compressive sensing image data <NUM>, which are reconstructed at <NUM>, thereby generating a respective output image <NUM>, which may be displayed on display <NUM>, for example.

In the following, a compressive sensing capturing method <NUM> is discussed under reference to <FIG>. At <NUM>, compressive sensing image data are obtained, for example, from the image sensor <NUM>, as discussed above.

On the obtained compressive sensing image data, a machine learning algorithm, such as CNN, is performed at <NUM>, whereby image attribute data are generated at <NUM>. The image attributes included in the image attribute data may be based on the classes including weights as output by the CNN without limiting the present disclosure to this specific example.

At <NUM>, one or more device attributes are set, as mentioned above, such as the autofocus, face-detection, smile-detection, etc..

At <NUM>, the device is controlled based on the device attribute set at <NUM>. For example, the autofocus is controlled for an object, an image is taken if a smile is detected, etc..

At <NUM>, the image, which is taken at <NUM>, is reconstructed based on the compressive sensing image data which are obtained, e.g. from the image sensor <NUM>.

Hence, by using machine learning techniques that directly operate with the compressed measurements, some embodiments avoid to perform a CS reconstruction of the original image. By avoiding such a reconstruction step, which is computationally intensive, e.g. applications where a real-time automatic feature of a device may be improved in some embodiments, as discussed above. The methods as described herein are also implemented in some embodiments as a computer program causing a computer and/or a processor to perform the method, when being carried out on the computer and/or processor. In some embodiments, also a non-transitory computer-readable recording medium is provided that stores therein a computer program product, which, when executed by a processor, such as the processor described above, causes the methods described herein to be performed.

It should be recognized that the embodiments describe methods with an exemplary ordering of method steps. The specific ordering of method steps is however given for illustrative purposes only and should not be construed as binding.

All units and entities described in this specification and claimed in the appended claims can, if not stated otherwise, be implemented as integrated circuit logic, for example on a chip, and functionality provided by such units and entities can, if not stated otherwise, be implemented by software.

Claim 1:
A compressive sensing capturing device (<NUM>), comprising circuitry that includes a compressive imaging sensor (<NUM>), wherein the circuitry is configured to:
obtain compressive sensing image data (<NUM>) from the compressive imaging sensor (<NUM>);
set a device attribute based on image attribute data (<NUM>),
wherein the image attribute data (<NUM>) are based on a machine learning algorithm (<NUM>) performed in the compressing domain on the obtained compressive sensing image data (<NUM>), and
wherein the machine learning algorithm (<NUM>) is based on an artificial neural network configured to use the obtained compressive sensing image data (<NUM>) as input and to output the image attribute data (<NUM>); and
control the compressive sensing capturing device (<NUM>) based on the set device attribute;
wherein the controlling of the compressive sensing capturing device (<NUM>) based on the set device attribute includes at least one of controlling optical elements for autofocus, auto-adjustment of luminance, shutter speed for setting illumination time, aperture size for illumination, auto-choosing of a shoot mode for picture/video mode, auto-choosing of flash mode, and tracking an object for autofocus.