Patent Description:
During operation, a conventional digital camera captures a two-dimensional (2D) image representing a total amount of light that strikes each point on a photo-sensor within the camera. However, this 2D image contains no information about the directional distribution of the light that strikes the photo-sensor.

In contrast, light field cameras sample the four-dimensional (4D) optical phase space or light field and in doing so capture information about the directional distribution of the light rays. Directional information at the pixels corresponds to locational information at the aperture.

This information captured by light field cameras may be referred to as the light field, the plenoptic function, or radiance.

In computational photography, a light field is a 4D record of all light rays in 3D. Radiance describes both spatial and angular information, and is defined as density of energy per unit of area per unit of stereo angle (in radians).

A light field camera captures radiance, therefore enables different post-processing, such as: re-focusing, noise reduction, 3D view construction and modification of depth of field, furthermore has wide applications including 3D TV and medical imaging.

Light fields may be captured with a conventional camera. In one conventional method, M×N images of a scene are captured from different positions with a conventional camera. If, for example, <NUM>×<NUM> images are captured from <NUM> different positions, <NUM> images are produced. The pixel from each position (i, j) in each image are taken and placed into blocks, to generate <NUM> blocks.

<FIG> illustrates an exemplary prior art light field camera <NUM>, or camera array, which employs an array of two or more objective lenses <NUM>. Each objective lens focuses on a particular region of photosensor <NUM>. This light field camera <NUM> may be viewed as a combination of two or more conventional cameras that each simultaneously records an image of a subject on a particular region of photosensor <NUM>. The captured images may then be combined to form one image.

<FIG> illustrates an exemplary prior art plenoptic camera <NUM>, another type of light field camera, that employs a single main objective lens <NUM> and a microlens or lenselet array <NUM> that includes, for example, about <NUM>,<NUM> lenselets.

Lenselet array <NUM> is typically placed at a small distance (~<NUM>) from a photosensor <NUM>, which can be for example a charge-coupled device (CCD). Through the microlens array <NUM>, each point of the 3D scene is projected onto a group of pixels, called macro-pixel, instead of a single pixel as in the traditional 2D images. Each pixel within a macro-pixel corresponds to a specific view angle for the same point of the scene.

<FIG> shows an example of image captured by plenoptic camera <NUM>, also called lenselet image (<FIG>), which is made up of an array of macro-pixels (<FIG>) that are typically hexagonal or circular shaped (<FIG>). The lenselet array <NUM> enables the plenoptic camera <NUM> to capture the light field, i.e. to record not only the image intensity, but also the distribution of intensity in different directions at each point.

Each lenselet splits a beam coming to it from the main lens <NUM> into rays coming from different "pinhole" locations on the aperture of the main objective lens <NUM>.

The plenoptic photograph captured by a camera <NUM> with, for example, <NUM>,<NUM> lenselets, will contain <NUM>,<NUM> macropixels. Captured light fields from light field cameras including plenoptic cameras are commonly saved as a lenselet image (<FIG>).

Compression of light field images is an important problem for computational photography. Due to the 4D nature of light fields, and the fact that 2D slices of light fields are equivalent to conventional pictures, the uncompressed files tend to be big, and may take up to gigabytes of space.

At the same time, there is redundancy in the data: all rays starting from a surface point have approximately the same radiance.

Thus, there is motivation for compression of light field images. Conventionally, light field images have been compressed using existing lossy and lossless image/video compression techniques.

Some conventional image compression approaches treat the 2D slices in a light field image as separate images and compress each separately. In others, the 4D light field image is contained in one 2D image, which is simply compressed by conventional methods as one image.

These approaches do not utilize the information and redundancy specific to light field images, but rather treat them as general images.

JPEG (Joint Photographic Experts Group) is a common conventional image compression standard, which employs block-based compression techniques. JPEG divides images into <NUM>×<NUM> pixel blocks, or more generally block-based compression techniques divide images into m×n pixel blocks, and compresses these blocks using some transform function.

Because of the division of images into blocks, JPEG and other block-based compression techniques are known to have the problem of generating "blocking artifacts", in which the compressed image appears to be composed of blocks or has other introduced vertical/horizontal artifacts (e.g., vertical or horizontal lines, discontinuities, or streaks).

The JPEG standard and other block-based compression techniques may be used to compress light field images directly, without consideration for the specifics of light field data.

However, due to the quasi-periodic nature of light field images, and the blocky nature of the compression, the results tend to be poor, including noticeable blocking artifacts. Such blocking artifacts may severely damage the angular information in the light field image, and therefore may limit the horizontal and vertical parallax that can be achieved using these images.

Several approaches have been proposed to compress specifically the light field images as a frame in a video, by employing video coding standards such as AVC (Advanced Video Codec) or HEVC (High Efficiency Video Coding).

These standards have been developed by the Moving Picture Experts Group (MPEG) and by the Joint Collaborative Team on Video Coding (JCT-VC), and adopt a block based coding approach employing Discrete Cosine Transform (DCT) techniques.

In light field image processing, a lenselet image is usually converted into the so called subaperture images, which is shown in <FIG>.

A subaperture image consists of multiple sub-views, where each of them consists of pixels of the same angular coordinates, extracted from different macro-pixels in the lenselet image.

In <FIG>, the sub-views are arranged according to the relative location within a macro-pixel. In a subaperture image there are in general two types of redundancies that can be exploited for compression, namely the intra-view and inter-view correlation.

The first redundancy type is the spatial correlation within each view, similar to the regular 2D image, where nearby pixels tend to have similar pixel intensities.

The second redundancy type is the inter-view correlation between the neighbouring sub-views. In the literature of light field data compression, these two correlation types have been exploited in the similar way as intra-prediction and inter-prediction in the video coding standard such as AVC and HEVC.

In general, the methods can be classified into two categories.

The first one compresses the subaperture image with modified intra-prediction in the current video codec. <NPL>, propose an extra self-similarity (SS) mode and a SS skip mode, which are included in the current intra-prediction modes to exploit the correlation between neighbouring sub-views in the subaperture image.

In the second approach, sub-views in a subaperture image are rearranged into a pseudo-video sequence, which is then encoded using existing video coding standards like HEVC. Works in which different sub-image re-arrangement schemes are applied. <NPL>, present a light field coding scheme based on a low-complexity pre-processing approach that generates a pseudo-video sequence suitable for standard compression using HEVC.

However, the aforementioned existing works require pre-processing stages which increase data representation redundancy prior to compression.

In order to show the limitation of the current state of art about the compression of light field images as a frame in a video, the architecture of a light field image encoding-decoding system is illustrated in <FIG> in terms of its basic functional units.

The encoder <NUM> includes at least a light field image pre-processing unit <NUM>, a subaperture image processing unit <NUM> and a block based encoder unit <NUM>.

The light field image pre-processing unit <NUM> takes as input the patterned raw lenselet image f, which is generated by a photosensor <NUM> (e.g. CDD sensor).

The patterned raw lenselet image f is a multi-color image, i.e. an image comprising information about different colours (e.g. red, green, blue) which can be generated by employing a color filter array on a square grid of photosensors such as the well-known Bayer filter. <FIG> and <FIG> schematically show an example of said color filter array (element f).

The particular arrangement of color filters is used in most single-chip digital image sensors used in digital cameras, camcorders, and scanners in order to create a color image.

With reference to <FIG>, the pre-processing unit <NUM>, first generates the full-color lenselet image through a RGB color interpolation which is known in the art as demosaicing technique, this operation increases the data volume to three times of the original raw data (<FIG>), because for each color channel R, G, B an image is generated which has the same size of the original raw image.

Successively, the conversion from the full-color lenselet to subaperture image is performed as described by <NPL>). During the conversion, the demosaicked lenselet images are first rotated, translated and scaled so that the estimated locations of the center of the macro-pixel, denoted in dashed line (<FIG>), can fall onto the integer pixel locations. This operations resulting in <NUM>% increase of pixel amount. Finally, the subaperture image f is generated from the converted lenselet image based on the relative position of each pixel to the macro-pixel center.

The subaperture image processing unit <NUM> takes as input the subaperture image f, and arranges the sub-views images, which compose f, into a sequence.

The sequence of the sub-views can obey to various criteria, for example the sequence can be composed in terms of group of pictures, or GOP (Group Of Pictures) structure, specifying the order in which intra-frames and inter-frames are arranged.

The resulting sequence of sub-views f" is such that can be received from a block based encoder.

The block based encoder unit <NUM> takes as input the sub-views sequence f" and encodes it according to a well-known video coding standard such as AVC or HEVC.

Moreover, the methods used by said standards during the compression, require a conversion form <NUM>:<NUM>:<NUM>-RGB format to <NUM>:<NUM>:<NUM>-YUV format.

Although, the down-sampling of U and V components reduces the redundancy introduced by the demosaicing and scaling, however, the rounding effect during the color conversion may introduce other distortion. The output of the encoder unit <NUM> is a bitstream f compliant with said standards.

The encoder <NUM> then transmits the bitstream f to the receiver node over a bandwidth constrained channel or memorizes them on a memory support <NUM> for later use, e.g. for decoding purposes.

The decoder <NUM> includes, at least, a block based decoder unit <NUM> and a post-processing unit <NUM>. For sake of simplicity, we assume that the bitstream f^ available to the decoders <NUM> is identical to that generated by the encoders <NUM>, since in practical applications adequate measures are taken for minimizing read/write or channel errors occurring during information transfer from the encoder to the decoder.

The block based decoder unit <NUM>, takes as input the bitstream f^, and generates the reconstructed sub-views sequence f according to the appropriate video coding standard (AVC, HEVC, etc.).

The post-processing unit <NUM>, takes as input the reconstructed sub-views sequence f and generates a reconstructed light field image f~, using techniques which enables operation such as image re-focusing, noise reduction, 3D view construction and modification of depth of field, as mentioned above.

Finally, the reconstructed light field image f~ is displayed using the display unit <NUM> such as TV-sets, monitors, etc..

In real world applications, the communication takes place over a bandwidth constrained channels, it is hence desirable that light field images can undergo some effective form of compression prior they are put on the channel. The same applies to the memorization of the light field images on a storage unit having limited capacity.

Regarding the problem of compressing light field images, some pre-processing stages increase data representation redundancy prior to compression. In a light field camera, which uses CCD plate at the photo sensor in capturing the color information, each pixel location only contains intensity of single color component (R, G, or B).

However, the existing compression technique all require full colored subaperture images as input.

Therefore, demosaicing is required to produce the full color lenselet images from CCD patterned image, which increases the data volume to three times of the original raw data; another redundancy is introduced during the conversion from lenselet to subaperture image.

During the conversion, the demosaiced lenselet images are rotated, translated and scaled so that the estimated locations of macro-pixels center can fall onto the integer pixel locations, resulting in <NUM>% increase of pixel amount.

Moreover, for methods using compression standard, e.g. AVC and HEVC, the <NUM>:<NUM>:<NUM>-RGB subaperture images need to be converted into <NUM>:<NUM>:<NUM>-YUV images. Although, the downsampling of U and V components reduces the redundancy introduced by the demosaicing and scaling, the rounding effect during the color conversion may introduce other distortion The article proposed by <NPL> describes a process for compressing a lenselet image wherein subaperture images are extracted from said lenselet image. Several approaches are described for ordering the subaperture images, such as the line scan mapping or the rotation scan mapping. The ordered set of subaperture images is encoded by means of H264/AVC encoder obtaining as output a H264/AVC compliant bitstream which represents the compressed lenselet image.

The article proposed by <NPL>, describes a post-processing pipeline to recover accurately the views (light-field) from the raw data of a plenoptic camera such as Lytro and to estimate disparity maps from such light-field. First, the microlens centers are estimated and then the raw image is demultiplexed without demosaicking it beforehand. A new block-matching algorithm to estimate disparities for the mosaicked plenoptic views is presented. The algorithm exploits at best the configuration given by the plenoptic camera: (i) the views are horizontally and vertically rectified and have the same baseline, and therefore (ii) at each point, the vertical and horizontal disparities are the same. The strategy of demultiplexing without demosaicking avoids image artifacts due to view cross-talk and helps estimating more accurate disparity maps.

The article proposed by <NPL>, describes an encoder-driven inpainting strategy to complete disocclusion holes in the depth image-based rendering (DIBR)-synthesized image in a rate distortion (RD) optimal manner. Missing pixel regions that are difficult-to-inpaint are first completed following instructions from the encoder in the form of auxiliary information (AI). The remaining easy-to-fill holes are then completed without encoder's help via nonlocal template matching, which is effective due to the self-similarity characteristics in natural images. Finally, two patch-based transform coding techniques (graph Fourier transform and DCT sparsification) are proposed, so that only missing pixels in a target patch are encoded, avoiding representation redundancy.

The present invention aims to solve these and other problems by providing a method and an apparatus for encoding and/or decoding digital images provided by light field cameras. The basic idea of the present invention is to generate a new compact light field image data representation that avoids redundancy due to demosaicing and scaling; the new representation is efficiently compressed using graph signal processing (GSP) techniques. Conversely, in the decoding stage inverse GSP techniques are performed.

More in detail, at the encoding stage, in order to put the estimated center location of each macro-pixel onto integer pixel locations, the pixels of the raw light field image are spatially displaced in a new, transformed multi-color image, having a larger number of columns and rows with respect to the received raw image. Such displacement introduces dummy pixels, i.e. pixel locations having undefined values. A sequence of sub-views is then obtained, and a bitstream (fd^) is generated by encoding a graph representation of the sub-view images.

At the decoding side, the bitstream (fd^) is graph decoded in a process reversing the GSP technique applied at the encoder side, a reconstructed sub-views sequence (fd") is obtained from the result of the graph decoding. The sub-views of the sequence comprise the dummy pixels introduced at the encoding side for centering the macro-pixels onto integer pixel locations. Then a demosaicing filter is applied to said sub-view sequence, obtaining a demosaiced full-color lenselet image, from which a full-color subaperture image (fd‴) is obtained.

The method disclosed in the present invention can be applied on the original color domain directly, e.g. the RGB color domain, without performing color conversion and rounding during encoding, which typically results in errors.

The characteristics and other advantages of the present invention will become apparent from the description of an embodiment illustrated in the appended drawings, provided purely by way of no limiting example, in which:.

In this description, any reference to "an embodiment" will indicate that a particular configuration, structure or feature described in regard to the implementation of the invention is comprised in at least one embodiment. Therefore, the phrase "in an embodiment" and other similar phrases, which may be present in different parts of this description, will not necessarily be all related to the same embodiment. Furthermore, any particular configuration, structure or feature may be combined in one or more embodiments in any way deemed appropriate.

With reference to <FIG>, an apparatus <NUM> for compressing digital images or video streams (also named encoding apparatus <NUM>) comprises the following parts:.

The video source <NUM> can be either a provider of live images, such as a light field camera, or a provider of stored contents such as a disk or other storage and memorization devices. The Central Processing Unit (CPU) <NUM> takes care of activating the proper sequence of operations performed by the units <NUM>, <NUM>, in the encoding process performed by the apparatus <NUM>.

These units can be implemented by means of dedicated hardware components (e.g. CPLD, FPGA, or the like) or can be implemented through one or more sets of instructions which are executed by the CPU <NUM>; in the latter case, the units <NUM>, <NUM> are just logical (virtual) units.

When the apparatus <NUM> is in an operating condition, the CPU <NUM> first fetches the light field image f from the video source <NUM> and loads it into the memory unit <NUM>.

Next, the CPU <NUM> activates the pre-processing unit <NUM>, which fetches the raw lenselet image f from the memory <NUM>, executes the phases of the method for pre-process the raw lenselet image f according to an embodiment of the invention (see <FIG>), and stores the resulting graph representation of the sequence of sub-views fd' back into the memory unit <NUM>.

Successively, the CPU <NUM> activates the graph coding unit <NUM>, which fetches from the memory <NUM> the graph representation of the sequence of sub-views fd', executes the phases of the method for encode the sequence of sub-views fd' according to a graph signal processing (GSP) techniques such as the Graph Fourier transform (GFT) or the Graph based Lifting Transform (GLT), and stores the resulting bitstream fd^ back into the memory unit <NUM>.

At this point, the CPU <NUM> may dispose of the data from the memory unit <NUM> which are not required anymore at the encoder <NUM>.

Finally, the CPU <NUM> fetches the bitstream fd^ from memory <NUM> and puts it into the channel or saves it into the storage media <NUM>.

With reference also to <FIG>, an apparatus <NUM> for decompressing digital images or video streams (also named decoding apparatus <NUM>) comprises the following parts:.

As for the previously described encoding apparatus <NUM>, also the CPU <NUM> of the decoding apparatus <NUM> takes care of activating the proper sequence of operations performed by the units <NUM>, <NUM> and <NUM>.

These units can be implemented by means of dedicated hardware components (e.g. CPLD, FPGA, or the like) or can be implemented through one or more sets of instructions stored in a memory unit which are executed by the CPU <NUM>; in the latter case, the units <NUM>, <NUM> and <NUM> are just a logical (virtual) units.

When the apparatus <NUM> is in an operating condition, the CPU <NUM> first fetches the bitstream fd^ from the channel or storage media <NUM> via any possible input unit and loads it into the memory unit <NUM>.

Then, the CPU <NUM> activates the graph decoding unit <NUM>, which fetches from the memory <NUM> the bitstream fd^, executes phases of the method for decoding the bitstream fd^ of the sub-views sequence according to a predefined graph signal processing (GSP) technique, such as the Graph Fourier transform (GFT) or the Graph based Lifting Transform (GLT), outputs the reconstructed sub-views sequence fd", and loads it into the memory unit <NUM>.

Any GSP technique can be used according to the invention; important is that the same technique is used in the encoding and decoding apparatus <NUM> for assuring a correct reconstruction of the original light field image.

Successively, the CPU <NUM> activates the demosaicing unit <NUM>, which fetches from the memory <NUM> the reconstructed sub-views sequence fd", and executes phases of the method for generating a full-color subaperture image fd‴ according to the invention, and loads it into the memory unit <NUM>.

Then, the CPU <NUM> activates the post-processing unit <NUM>, which fetches from the memory <NUM> the full-color subaperture image fd‴ and generates a reconstructed light field image fd~, storing it into the memory unit <NUM>.

At this point, the CPU <NUM> may dispose of the data from the memory which are not required anymore at the decoder side.

Finally, the CPU <NUM> fetches from memory <NUM> the recovered light field image fd~ and sends it, by means of the video adapter <NUM>, to the display unit <NUM>.

It should be noted how the encoding and decoding apparatuses described in the figures may be controlled by the CPU <NUM> to internally operate in a pipelined fashion, enabling to reduce the overall time required to process each image, i.e. by performing more instructions at the same time (e.g. using more than one CPU and/or CPU core).

It should also be noted than many other operations may be performed on the output data of the coding device <NUM> before sending them on the channel or memorizing them on a storage unit, like modulation, channel coding (i.e. error protection).

Conversely, the same inverse operations may be performed on the input data of the decoding device <NUM> before effectively process them, e.g. demodulation and error correction. Those operations are irrelevant for embodying the present invention and will be therefore omitted.

Besides, the block diagrams shown in <FIG> are of exemplificative nature only; they allow to understand how the inventions works and how it can be realized by the person skilled in the art.

The skilled person understands that these charts have no limitative meaning in the sense that functions, interrelations and signals shown therein can be arranged in many equivalents ways; for example, operations appearing to be performed by different logical blocks can be performed by any combination of hardware and software resources, being also the same resources for realizing different or all blocks.

The encoding process and the decoding process will now be described in detail.

In order to show how the encoding process occurs, it is assumed that the image f (or a block thereof) to be processed is preferably a color patterned raw lenselet image, where each pixel is encoded over <NUM> bit so that the value of said pixel can be represented by means of an integer value ranging between <NUM> and <NUM>. Of course, this is only an example; images of higher color depth (e.g. <NUM>, <NUM>, <NUM>, <NUM> or <NUM> bit) can be processed by the invention without any loss of generality.

The image f can be obtained applying a color filter array on a square grid of photosensors (e.g. CDD sensors); a well-known color filter array is for example the Bayer filter, which is used in most single-chip digital image sensors.

<FIG> shows some examples of color filter array, where the letters R,G,B indicate respectively the red, green and blue color filters which are applied on a grid of photosensors.

With also reference to <FIG>, it is now described how the different parts of the encoding apparatus <NUM> interact for compressing digital light field images or video streams.

With also reference to <FIG> and <FIG> the pre-processing unit <NUM> preferably comprises the following steps:.

Two distinctive schemes for graph connection can be considered.

The first scheme takes into account only intra-view connections when constructing a graph, where each node is connected to a predefined number K of nearest nodes in terms of Euclidean distance, i.e. the distance between available irregularly spaced pixels (e.g. <NUM>, <NUM>) within the same sub-view of the sequence.

The second scheme takes into account both intra and inter-view correlations among the sub-views of the sequence.

In order to reduce graph complexity, the sub-views sequence is divided into multiple GOPs consists of a predefined number G of sub-views.

<FIG> shows an example of the sub-views sequence fd' subdivided in terms of GOPs structure <NUM>, which is composed by four sub-views comprising a reference sub-view <NUM>.

Successively, a sub-view matching for motion estimation between each sub-view and the previous reference sub-view is performed in the sequence.

The optimal global motion vector can be determined for each sub-view in terms of sum of squared error (SSE), which can be evaluated considering the pixel samples of each sub-view and the previous reference sub-view.

The matching is considered for the whole sub-view, instead of applying the block-based matching employed for example for the motion estimation in HEVC.

Specifically, each m×n sub-view is first extrapolated to the size of (m+2r)×(n+2r) before motion search, where r is the motion search width.

This reduces the overhead in encoding of the motion vectors. The sub-view extrapolation can be performed by employing several techniques, for example by copying the border pixel samples of each sub-view.

After motion estimation, each pixel is connected to a predefined number P of nearest neighbours in terms of Euclidean distance within the same sub-view and the reference view shifted by the optimal motion vector.

With also reference to <FIG> the graph coding unit <NUM> (and the decoding apparatus <NUM> as well, for decoding) encodes the sub-views sequence using graph signal processing (GSP) techniques such as the Graph Fourier transform (GFT) or the Graph based Lifting Transform (GLT) for coding each GOP separately.

A graph G=(E,V) is composed of a set of nodes v∈V, connected with links. For each link ei,j∈E, connecting nodes vi and vj, there is an associated weight of non-negative value wij ∈[<NUM>,<NUM>], which captures the similarity between the connected nodes.

An image f can be represented as a graph where the pixels of the image correspond to the graph nodes, while the weights of the links describe the pixels similarity which can be evaluated using a predetermined non-linear function (e.g. Gaussian or Cauchy function) depending on the grayscale space distance di,j = |fi - fj| between the i-th pixel fi and the j-th pixel fj of the image.

In the Graph Fourier transform (GFT) technique, the graph information can be represented with a weights matrix W which elements are the weights wij of the graph, then the corresponding Laplacian matrix can be obtained as L=D-W where D is a diagonal matrix with elements <MAT>. The GFT is performed by the mathematical expression f̂ = UTf where U is the matrix which columns are the eigenvectors of the matrix L, and f is the raster-scanner vector representation of the image f.

The coefficients f̂ and the weights wij are then quantized and entropy coded. More related work known in the art describe approaches improving the GFT based coding, as shown for example by<NPL>.

The Graph based Lifting Transform (GLT) technique is a multi-level filterbank that guarantees invertibility. At each level m, the graph nodes are first divided into two disjoint sets, a prediction set SPm and an update set SUm.

The values in SUm are used to predict the values in SPm, the resulting prediction errors are stored in SPm, and are then used to update the values in SUm.

The smoothed signal in SUm will serve as the input signal to level m+<NUM>, while the computation for coefficients in SPm uses only the information in SUm, and vice versa.

Carrying on the process iteratively produces a multi-resolution decomposition. For video/image compression applications, the coefficients in the update set SUM of the highest-level M will be quantized and entropy coded. More related work known in the art describe approaches improving the GLT based coding, as shown for example by <NPL>.

Summarizing, with also reference to <FIG> and <FIG>, the method for encoding digital images according to the invention comprises the following phases:.

Finally, the graph-coded bitstream fd^ of the sub-views sequence can be transmitted and/or stored by means of the output unit <NUM>.

With reference to <FIG> and <FIG>, the decoder <NUM> comprises the graph decoding unit <NUM>, the demosaicing unit <NUM> and the post-processing unit <NUM>.

The graph decoding unit <NUM> is configured to receive and decode the bitstream fd^ of the sub-views sequence according to a predefined graph signal processing (GSP) techniques, outputting the reconstructed sub-views sequence fd"(step <NUM>).

The demosaicing unit <NUM> preferably performs the following steps:.

The optional post-processing unit <NUM> is configured to receive the full-color subaperture image fd‴ and to generate a reconstructed light field image fd~, using operations permitted in the light field images such as re-focusing, noise reduction, 3D view construction and modification of depth of field.

Summarizing, with also reference to <FIG> and <FIG>, the method for decoding digital images or video streams according to the invention comprises the following phases:.

Finally, the reconstructed light field image fd~ can be outputted by means of output video unit <NUM> and displayed on the display unit <NUM>.

With reference to <FIG>, <FIG>, the results of performance tests conducted by the inventors are going to be discussed. In this test, an encoder-decoder pair implemented according to an embodiment of the present invention has been evaluated. In order to perform the coding-encoding test, the EPFL database (<NPL>) was used.

The subaperture image consists of <NUM> sub-views of size <NUM>×<NUM>. <FIG> shows the performance of the method described in an embodiment of the present invention, which is compared with the coding approach using HEVC standard.

The ordinate axis denotes the average PSNR for R, G, and B color components. Compared to state-of-the-art schemes, a coding gain is achieved at the high-bitrate region.

For the test, both All-intra and Low delay P configurations were used for the baseline HEVC based scheme.

For Low delay P configuration in HEVC. The sub-views are arranged into pseudo-sequence in the same way as pictured in <FIG>, and divided into multiple size <NUM> GOPs (<FIG>).

The first view in each GOP is compressed as an I-frame, and the remaining frames are coded as P-frames. For the proposed graph based approach, each node is connected to <NUM> nearest neighbours, and the search width r = <NUM> for sub-view matching.

The transformed coefficients are uniformly quantized and entropy coded using the Alphabet and Group Partitioning (AGP) proposed by <NPL>. In order to evaluate the reconstructed lenselet image, using graph based coding, the reconstructed lenselet image is demosaiced and converted to the colored subaperture image in a same way as proposed by <NPL>.

In the baseline method, the reconstructed YUV <NUM>:<NUM>:<NUM> sequences are converted to RGB <NUM>:<NUM>:<NUM>, where the upsampling for U and V components is based on nearest neighbour. Concluding, the obtained results show that the method described in the present invention can outperform the state-of-the-art schemes like a HEVC-based approach.

In an alternative embodiment of the invention, the patterned raw lenselet image f can be generated by employing other color filter arrays placed on a square grid of photosensors, besides the well-known Bayer filter.

In another embodiment of the invention, the patterned raw lenselet image f can be generated by capturing other combinations of color components, for example RGBY (red, green, blue, yellow) instead of RGB.

In other embodiments, the invention is integrated in a video coding technique wherein also the temporal correlation between different light field images is taken into account. To that end, a prediction mechanism similar to those used in the conventional video compression standards can be used in combination with the invention for effectively compressing and decompressing a video signal.

In other embodiments, the encoding and decoding stages described in the present invention can be performed employing other graph signal processing (GSP) techniques instead of the Graph Fourier transform (GFT), or the Graph based Lifting Transform (GLT).

In other embodiments, the graph signal processing (GSP) technique employed at the encoding and decoding stages can be signalled from the encoder apparatus to the decoder apparatus. Alternatively, the GSP technique employed by both the encoder and decoder is defined in a technical standard.

Claim 1:
Method for encoding a raw lenselet image (f), comprising:
- a receiving phase (<NUM>), wherein at least a portion of a raw lenselet image (f) is received, said image (f) comprising a plurality of macro-pixels (<NUM>,<NUM>,<NUM>), each macro-pixel (<NUM>,<NUM>,<NUM>) comprising pixels corresponding to a specific view angle for the same point of a scene;
- an output phase, wherein a bitstream (fd^) comprising at least a portion of an encoded lenselet image (f) is outputted,
wherein said method also comprises:
- an image transform phase (<NUM>), wherein the pixels of said raw lenselet image (f) are spatially displaced in a transformed multi-color image (<NUM>) having a larger number of columns and rows with respect to the received raw lenselet image, wherein said displacement introduces dummy pixels (<NUM>, <NUM>) having undefined color value that are inserted into said raw lenselet image (f) and wherein said displacement is performed so as to put the estimated center location of each macro-pixel (<NUM>,<NUM>,<NUM>) onto integer pixel locations;
- a sub-view generation phase, wherein a sequence of sub-views (fd') is generated, said sub-views (<NUM>,<NUM>,<NUM>) comprising pixels of the same angular coordinates extracted from different macro-pixels (<NUM>,<NUM>,<NUM>) of said transformed raw lenselet image (f);
wherein the sub-views also comprise the dummy pixels introduced in the image transform phase;
- a graph coding phase, wherein a bitstream (fd^) is generated by encoding a graph representation of at least one of the sub-views (<NUM>,<NUM>,<NUM>) of said sequence (fd') according to a predefined graph signal processing (GSP) technique,
wherein said output phase comprises outputting said graph-coded bitstream (fd^) for its transmission and/or storage.