Patent Publication Number: US-2018047185-A1

Title: Light field metadata

Description:
TECHNICAL FIELD 
     The present invention relates to metadata for light field data and processing of light field data. 
     BACKGROUND 
     Conventional cameras capture light from a three-dimensional scene on a two-dimensional sensor device sensitive to visible light. Light sensitive technology used in such imaging devices is often based on semiconductor technology, capable of converting photons into electrons such as, for example, charge coupled devices (CCD) or complementary metal oxide technology (CMOS). A digital image photosensor, for example, typically includes an array of photosensitive cells, each cell being configured to capture incoming light. A 2D image providing spatial information is obtained from a measurement of the total amount of light captured by each photosensitive cell of the image sensor device. While the 2D image can provide information on the intensity of the light and the colour of the light at spatial points of the photosensor(s), no information is provided on the direction of the incoming light. 
     Light field cameras (also known as radiance capturing cameras) can provide directional information on incoming light rays in additional to 2D spatial information. The directional information may be obtained by the use of an array of micro-lenses, often referred to as a microlens array (MLA) associated with an image sensor. A light field array camera, for example, is provided with a microlens array and a photosensor. A plenoptic camera is provided with a main lens focusing light on a MLA, and a photosensor associated with the MLA. In other configuratios of light field cameras a plurality of cameras each provided with its own lens and sensor may be used to obtain light field data. 
     A light field is often defined as a 4D function characterizing the light from different directions at different points in a scene. The information on the directional distribution of the light rays is typically referred to as light field data or 4D data. Since the information provides 4 parameters—two dimensional positional information and two dimensional angular information the directional distribution corresponds to a four-dimensional (4D) function. The light field may be interpreted as a two dimensional collection of 2D images of a scene. 
     The light field data obtained can be processed for many applications, for example, to generate refocused images of the scene, to generate images from different viewpoints, to provide depth information on the captured scene or to generate 3D images. 
     Data obtained from a light field camera should be appropriately managed in order to be properly exploited in the different applications. 
     The present invention has been devised with the foregoing in mind. 
     SUMMARY OF INVENTION 
     According to a first aspect of the invention there is provided a method of providing metadata for captured light field data, the method comprising providing a data format aggregating metadata describing characteristics of at least one of: a light field camera used to capture the light field data, a matrix of views derived from the light field data, and a focal stack derived from the light field data. 
     In an embodiment the method includes obtaining light field data metadata describing characteristics of at least one of: a light field camera used to capture the light field data, a matrix of views derived from the light field data, and a focal stack derived from the light field data; and rendering the light field data based on the information provided by the metadata. 
     In an embodiment, the metadata is aggregated in a light image file. 
     In an embodiment, the metadata is defined in markup language in compliance with XML. 
     In an embodiment, the metadata describing characteristics of the light field camera used to capture the light field data includes data defining one or more of the following technical features of the light field camera:
         a photosensor of the light field camera;   a main lens of the light field camera; and   a micro lens array of the light field camera.       

     In an embodiment, the metadata describing the photosensor of the light field camera includes data representative of one or more of the following parameters:
         spatial resolution of the photosensor;   colour sampling of a colour filter of the photosensor;   transmittance of the colour filter of the photosensor dynamic range of the pixels of the photosensor;   color transform matrix defining the non-linear relationship between the signal output by a pixel and the incoming light;   the pitch between two adjacent pixels of the photosensor; and   scaling factor specifying the shape ratio of non-square pixels.   In an embodiment, the metadata describing the main lens of the light field camera includes data representative of one or more of the following parameters:   focal length of the main lens   the ratio between the pupil and the focal length; and   the vignetting model of the main lens.       

     In an embodiment, the metadata describing the micro lens array of the light field camera includes data representative of one or more of the following parameters:
         pattern of the micro-lens array;   pattern of color filters arranged on the micro-lens array;   the pitch between two adjacent micro-lenses of the micro-lens array;   scaling factor specifying the shape ratio of micro lens; and   the number of calibrated microlens centres   the spatial coordinates of one or more microlens centres       

     In an embodiment, the metadata describing characteristics of a matrix of views derived from the light field data, includes data representative of one or more of the following parameters:
         the number of views in the array;   the spatial resolution of a or each view;   colors available in a view;   the spatial coordinates of a reference point of a view;   the horizontal and vertical focal length of a view       

     In an embodiment, the metadata describing characteristics of a focal stack derived from the light field data, includes data representative of one or more of the following parameters:
         the number of focused pictures in the focal stack;   the spatial resolution of the focused pictures   the distance of focalization of a focused picture       

     A second aspect of the invention provides a device for providing metadata for captured light field data, the device comprising:
         a light field data acquisition module for acquiring light field data captured by a light field camera and   a data formatter for providing aggregating metadata describing characteristics of at least one of:   a light field camera used to capture the light field data,   a matrix of views derived from the light field data, and   a focal stack derived from the light field data.       

     A third aspect of the invention provides a light field imaging device comprising:
         an array of micro lenses arranged in a regular lattice structure;   a photosensor configured to capture light projected on the photosensor from the array of micro lenses, the photosensor comprising sets of pixels, each set of pixels being optically associated with a respective micro lens of the array of micro lenses; and   a device for providing metadata for captured light field data, the device comprising:
           a light field data acquisition module for acquiring light field data captured by a light field camera and   a data formatter for providing aggregating metadata describing characteristics of at least one of:
               a light field camera used to capture the light field data,   a matrix of views derived from the light field data, and   a focal stack derived from the light field data.   
               
               

     A further aspect of the invention provides a device for rendering an image from light field data using metadata describing characteristics of at least one of:
         a light field camera used to capture the light field data,   a matrix of views derived from the light field data, and   a focal stack derived from the light field data, the device comprising   a processor and memory containing a rendering application, the light field data and the metadata wherein the rendering application configures the processor to render the light field data based on information provided in the meta data.       

     Another aspect of the invention provides a data package comprising
         light field data captured from a light field camera;   metadata describing characteristics of at least one of:   a light field camera used to capture the light field data,   a matrix of views derived from the light field data, and   a focal stack derived from the light field data.       

     According to another aspect of the invention there is provided a method of processing light field data, comprising converting light field data from a first light field data format into a second light field data format different to the first light field format, the conversion being performed using parameters obtained from metadata describing the light field data wherein the first light field data format and the second light field data format are selected from the group consisting of: raw light field data captured by a light field camera, a matrix of sub-aperture images, and a focal stack. 
     Yet another aspect of the invention provides a device for processing light field data, comprising a processor and a memory containing a format conversion application, wherein the format conversion application configures the processor to convert light field data from a first light field data format into a second light field data format different to the first light field format, the conversion being performed using parameters obtained from metadata describing the light field data wherein the first light field data format and the second light field data format are selected from the group consisting of: raw light field data captured by a light field camera, a matrix of sub-aperture images, and a focal stack. 
     In an embodiment, the memory further comprises a rendering application, wherein rendering application configures the processor to render the light field data based on information provided in metadata. 
     A further aspect of the invention provides a light field imaging device comprising: an array of micro lenses arranged in a regular lattice structure; a photosensor configured to capture light projected on the photosensor from the array of micro lenses, the photosensor comprising sets of pixels, each set of pixels being optically associated with a respective micro lens of the array of micro lenses; and a device for processing light field data, comprising a processor and a memory containing a format conversion application, wherein the format conversion application configures the processor to convert light field data from a first light field data format into a second light field data format different to the first light field format, the conversion being performed using parameters obtained from metadata describing the light field data wherein the first light field data format and the second light field data format are selected from the group consisting of: raw light field data captured by a light field camera, a matrix of sub-aperture images, and a focal stack. 
     In an embodiment, the light-field metadata describes characteristics of at least one of: a light field camera used to capture the light field data, a matrix of sub-aperture images derived from the light field data, and a focal stack derived from the light field data. 
     In an embodiment, the first light field data format is raw light field data, captured by a light field camera, the raw light field data comprising an array of micro-images, each micro image being projected by a respective micro lens of a micro-lens array and the second light field format is a matrix of sub-aperture images, each sub-aperture image being generated from collocated samples of the micro-images and wherein at least one of the focal lengths and the interaxial baseline distance of the matrix of sub-aperture images is derived using technical parameters of the light field camera included in the metadata. 
     In an embodiment, focal lengths are derived from at least one of the focal length of the main lens of the light field camera, the horizontal and scaling factor of the photosensor of the light field camera, and the pixel pitch of the photosensor. 
     In an embodiment, the interaxial baseline is derived from at least one of the scaling factors of the photosensor of the light field camera, the pixel pitch of the photosensor, the pitch of the microlens array and the scaling factors of the micro lens array. 
     In an embodiment, wherein the extrinsic matrix of a sub-aperture image V k,l  is composed of the following rotation R and translation T matrices: 
     
       
         
           
             R 
             = 
             
               
                 
                   ( 
                   
                     
                       
                         1 
                       
                       
                         0 
                       
                       
                         0 
                       
                     
                     
                       
                         0 
                       
                       
                         1 
                       
                       
                         0 
                       
                     
                     
                       
                         0 
                       
                       
                         0 
                       
                       
                         1 
                       
                     
                   
                   ) 
                 
                  
                 
                     
                 
                  
                 and 
                  
                 
                     
                 
                  
                 T 
               
               = 
               
                 ( 
                 
                   
                     
                       
                         k 
                         · 
                         
                           b 
                           x 
                         
                       
                     
                   
                   
                     
                       
                         l 
                         · 
                         
                           b 
                           y 
                         
                       
                     
                   
                   
                     
                       0 
                     
                   
                 
                 ) 
               
             
           
         
       
     
     where b x  and b y  are the respective horizontal and vertical inter-axial baseline distances, and k and l are integers. 
     In an embodiment, the first light field data format is raw light field data, captured by a light field camera, the raw light field data comprising an array of micro-images, each micro image being projected by a respective micro lens of a micro-lens array and the second light field format is a focal stack, the focal stack being computed by splatting shifted micro images of the raw light field data, wherein the focalisation distance of the focused image is derived from at least one of the focal length of the main lens, the pixel pitch of the photosensor and the lens pitch of the microlens array. 
     In an embodiment, the focalisation distance is determined based on the expression: 
     
       
         
           
             z 
             = 
             
               
                 
                   w 
                   - 
                   p 
                 
                 p 
               
               × 
               F 
             
           
         
       
     
     where w represents the shift defining a focused picture, F is the focal distance of the main lens and 
     
       
         
           
             p 
             = 
             
               
                 r 
                 
                   µ 
                    
                   
                       
                   
                    
                   L 
                 
               
               
                 r 
                 p 
               
             
           
         
       
     
     is the number or pixels or the photosensor corresponding to the diameter of a micro-lens, r μL  denoting the microlens pitch of the light field camera and r p  denoting the photosensor pitch of the light field camera. 
     In an embodiment, the derived focalisation distance is included in the metadata. 
     In an embodiment, the first light field data format is a matrix of sub-aperture images and the second light field data format is a focal stack, the focal stack being computed by splatting shifted sub-aperture images, wherein the focalisation vergence is derived from at least one of the focal lengths and the inter-axial baseline distances of the matrix of sub-aperture images. 
     In an embodiment, the focalisation vergence v is determined by: 
     
       
         
           
             v 
             = 
             
               
                 1 
                 z 
               
               = 
               
                 
                   
                     d 
                     
                       
                         b 
                         x 
                       
                       · 
                       
                         f 
                         x 
                       
                     
                   
                    
                   
                       
                   
                    
                   or 
                    
                   
                       
                   
                    
                   v 
                 
                 = 
                 
                   d 
                   
                     
                       b 
                       y 
                     
                     · 
                     
                       f 
                       y 
                     
                   
                 
               
             
           
         
       
     
     where b x  and b y  are the inter-axial baseline distances, f x  and f y  are the focal lengths of the matrix of sub-aperture images and d is the underlying disparity. 
     A further aspect of the invention provides an electronic device comprising memory, one or more processors; and one or more modules stored in the memory and configured for execution by the one or more processors, the one or more modules including instructions for processing or formatting light field data in accordance with embodiments of the invention. 
     A further aspect of the invention provides a computer readable memory including instructions, that when executed by a processor of a portable device, cause the device to perform operations comprising including instructions for processing or formatting light field data in accordance with embodiments of the invention. 
     Some processes implemented by elements of the invention may be computer implemented. Accordingly, such elements may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a “circuit”, “module” or “system’. Furthermore, such elements may take the form of a computer program product embodied in any tangible medium of expression having computer usable program code embodied in the medium. 
     Since elements of the present invention can be implemented in software, the present invention can be embodied as computer readable code for provision to a programmable apparatus on any suitable carrier medium. A tangible carrier medium may comprise a storage medium such as a floppy disk, a CD-ROM, a hard disk drive, a magnetic tape device or a solid state memory device and the like. A transient carrier medium may include a signal such as an electrical signal, an electronic signal, an optical signal, an acoustic signal, a magnetic signal or an electromagnetic signal, e.g. a microwave or RF signal. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       Embodiments of the invention will now be described, by way of example only, and with reference to the following drawings in which: 
         FIG. 1A  is a schematic diagram of a light field camera according to a first embodiment of the invention; 
         FIG. 1B  is a schematic diagram of a light field camera according to a second embodiment of the invention; 
         FIG. 2A  is a functional diagram of a light field camera according to an embodiment of the invention; 
         FIG. 2B  is a functional diagram of a light field data formator and light field data processor according to an embodiment of the invention; 
         FIG. 3  is an example of a 2D light field image formed on a photosensor array; 
         FIG. 4  is an example of steps of formatting and processing of light field data according to an embodiment of the invention; 
         FIG. 5  and  FIG. 6  are screen shots illustrating examples of data files in accordance with embodiments of the invention. 
     
    
    
     DETAILED DESCRIPTION 
     Light-field cameras are typically used to record a 4D light-field on a sensor composed of an array of pixels. The 4D light-field data provides 2-dimensional spatial information and 2-dimensional angular information on the incoming light. Such light-Field cameras may be for instance: a plenoptic camera  100  comprising a main lens  101 , an array of lenses  110  and a photo sensor  120  as illustrated in  FIG. 1A ; or a multi-camera array comprising an array of lenses  210  and a photosensor  220  without a main lens, as illustrated in  FIG. 1B . A multi-array camera may be considered as a particular case of a plenoptic camera where the mains lens has an infinite focal. 
     Embodiments of the invention provide formatting of light field data for further processing applications such as format conversion, refocusing, viewpoint change and 3D image generation. 
       FIG. 1A  is a schematic diagram of a light field camera for which one or more embodiments of the invention may be applied. 
     The light field camera  100  of  FIG. 1A  is a plenoptic type light field camera comprising a micro lens array  110  and an image sensor  120  associated with the micro lens array  110 . Further optics may be provided such as an aperture  102  for enabling a selectable amount of light to enter into the light field camera. The lens arrangement further includes a main (objective) lens  101  for focusing light towards the micro lens array  110 . 
     The micro lens array  110  is composed of an array of microlenses  111 ,  112 ,  113  . . .  11   n  arranged, in a regular lattice structure. For example, the structure may be a rectangular lattice of rows and columns. The micro lens array may also be referred to as a lenslet array. For purposes of illustratio, the microlens array  110  is shown with a relatively small number of microlenses, but it will be appreciated that the number of microlenses may reach up to several thousand or even several million microlenses. Optionally, spacers might be placed around each micro lens of the micro-lens array  110  between the micro-lens array  110  and the image sensor  120  in order to prevent light from one micro lens from overlapping with light of other microlenses of the microlens array  110  at the image sensor  120 . 
     The image sensor comprises a photosensor array  120  composed of a number m of photo sensors  121 ,  122 ,  123 ,  124  . . .  12   m  arranged in a lattice structure. For example the structure may be a rectangular lattice of rows and columns. Each photosensor  121 ,  122 ,  123 ,  124  . . .  12   m  corresponds to a pixel or a group of pixels of the raw image of the scene captured by the photosensor array  120 , each pixel covering a part (also referred to as a point) of the scene. For purposes of illustratio, the photosensor array  120  is illustrated as having a relatively small number of photosensors  121  to  121   m . It will be appreciated, however, that, the number of photosensors is not limited to that illustrated in  FIG. 1A  but may be extended to any number of photosensors, for example several thousand or several million photosensors. As an illustrative example, an image of 12.4 megapixels may be provided by an array of 4088×3040 pixels/photosensors. The image sensor may be for example a charge coupled device (CCD). 
     The microlenses  111 ,  112 ,  11   n  of the micro lens array  110  are arranged such that each microlens is optically associated with photosensors of the photo sensor array  120 . The photosensor array  120  is generally of finer pitch than the microlens array. Accordingly, each microlens is associated with a plurality of photosensors of the photosensor array  120 . Optical association between a microlens and a group of photosensors signifies that the light rays passing through a given microlens reach at least one of the group of photosensors optically associated with the given microlens. 
     The interposition of the microlens array  110  between the main lens  101  and the photosensor  120  results in multiple images being formed on the photosensor array  120 . Each microlens of the microlens array  110  projects a respective image, onto the associated photosensors of the photosensor array  120 . Accordingly the raw image captured by the image sensor  120  is composed of an array of small images, as for example illustrated in  FIG. 3 , typically referred to as micro-images. Each micro-image corresponds to a partial field of view from a respective different viewpoint. and corresponds to a micro-lens of the microlens array. Each pixel of photosensor  120  may be seen as corresponding to a 4D light field coordinate where two dimensions specify its spatial position on the sensor and two dimensions specify the angular or directional information of light that is incident upon that pixel according to the 2D position of the pixel within the microimage which it belongs to. 
     A color filter array (CFA) may in some cases be arranged on the microlens array  110  or on the photosensor array  120 . The CFA typically arranges RGB (Red, Green and Blue) color filters on the photosensor or microlens array, the RGB arrangement taking, for example, the form of a Bayer filter mosaic. One color filter (red, green or blue filter) may be associated with an MLA according to a predetermined pattern, comprising 50% green, 25% red and 25% blue in the example of a Bayer filter, such a pattern also being referred to as a RGBG, GRGB or RGGB pattern. It will be appreciated that, the arrangement of the color filters on the microlens array  110  or photosensor array  120  is not limited to a RGGB pattern. In other embodiments, the predetermined pattern may be a RGBE pattern with one of the green filters modified to ‘Emerald’ (for a block of four color filters); a CYYM pattern with one ‘Cyan’ filter, two ‘Yellow’ filters and one ‘Magenta’ filter (for a block of four color filters); a CYGM pattern with one ‘Cyan’ filter, one ‘Yellow’ filter, one ‘Green’ filter and one ‘Magenta’ filter; a RGBW pattern with one ‘Red’ filter, one ‘Green’ filter, one ‘Blue’ filter and one ‘White’ filter, several arrangement being possible (for example arranged on a block of four color filters with ‘White’ for the upper left filter, ‘Red’ for the upper right filter, ‘Blue’ for the lower left filter and ‘Green’ for the lower right filter; or arranged on a block of 4×4 color filters with ‘White’, ‘Blue’, ‘White’, ‘Green’ for the first line, ‘Blue’, ‘White’, ‘Green’, ‘White’ for the second line below the first line, ‘White’, ‘Green’, ‘White’, ‘Red’ for the third line below the second line and ‘Green’, ‘White’, ‘Red’, ‘White’ for the fourth line below the third line). 
     The gap between the microlens array and the photosensor array may be composed of air, of an optical material having an index n (for example a glass layer) or of multiple layers comprising at least one layer air layer and at least one optical material layer. Using a glass layer to form the gap has the advantages of keeping the microlens array  110  at a constant distance from the photosensor array  120  uniformly across the photosensor array  120  and of reducing this distance when needed. If d is the distance between the output of the microlens array  110  and the photosensor array  120  along a longitudinal axis, having a layer composed of an optical material with an index n (n&gt;1, for example n=1.5) between the micro lens array  110  and the photosensor array  120  enables the distance to be set to d/n without modifying the distance d. By adapting/modifying the index of the optical material of the layer forming the gap, it is possible to adapt/modify a parameter representative of the distance between the microlens array  110  and the photosensor array  120  without modifying the distance d. 
       FIG. 1B  is a schematic diagram of a light field camera according to a second embodiment of the invention. 
     The light field camera  200  according to the second embodiment of the invention is a multi camera array type light field camera comprising a micro lens array  210  and an image sensor  220  associated with the micro lens array  210 . In this embodiment a main lens  201  for focusing light towards the micro lens array  210  is not present. Elements such as the photosensor array  220  and micro lens array  210  operate in a similar manner to the corresponding elements of the plenoptic type camera of  FIG. 1A . The main difference is that the main lens is not present in the embodiment of  FIG. 1B . Each micro-image corresponds to a full field of view from a respective different viewpoint. 
       FIG. 2A  is a block diagram of a light field camera device in accordance with an embodiment of the invention. The light field camera comprises an aperture/shutter  102 , a main (objective) lens  101 , a micro lens array  110  and a photosensor array  120  in accordance with the light field camera of  FIG. 1A . In some embodiments the light field camera includes a shutter release that is activated to capture a light-field image of a subject or scene. It will be appreciated that the functional features may also be applied to the light field camera of  FIG. 1B . 
     The photosensor array  120  provides light field image data which is acquired by LF Data acquisition module  140  for formatting by a light field data formatting module  150  and/or processing by light field data processor  155 . Light field data may be stored, after acquisition and after processing, in memory  190  in a raw data format, or as sub aperture images or focal stacks. 
     In the illustrated example, the light field data formatting module  150  and the light field data processor  155  are disposed in or integrated into the light field camera  100 . In other embodiments of the invention the light field data formatting module  150  and/or the light field data processor  155  may be provided in a separate component external to the light field capture camera. The separate component may be local or remote with respect to the light field image capture device. It will be appreciated that any suitable wired or wireless protocol may be used for transmitting light field image data to the formatting module  150  or light field data processor  155 ; for example the light field data processor may transfer captured light field image data and/or other data via the Internet, a cellular data network, a WiFi network, a BlueTooth communication protocol, and/or any other suitable means. 
     The light field data formatting module  150  is configured to provide a data format, in accordance with embodiments of the invention, aggregating metadata describing characteristics of at least one of a light field camera used to capture the light field data, a matrix of views derived from the light field data, and a focal stack derived from the light field data. The light field data formatting module  150  may be implemented in software, hardware or a combination thereof. 
     The light field data processor  155  configured to operate on raw light field image data received directly from the LF data acquisition module  140  for example to generate focal stacks or a matrix of views in accordance with embodiments of the invention. Output data, such as, for example, still images, 2D video streams, and the like of the captured scene may be generated. The light field data processor may be implemented in software, hardware or a combination thereof. 
     In at least one embodiment, the light field camera  100  may also include a user interface  160  for enabling a user to provide user input to control operatio of camera  100  by controller  170 . Control of the camera may include one or more of control of optical parameters of the camera such as shutter speed, or in the case of an adjustable light field camera, control of the relative distance between the microlens array and the photosensor, or the relative distance between the objective lens and the microlens array. In some embodiments the relative distances between optical elements of the light field camera may be manually adjusted. Control of the camera may also include control of other light field data acquisition parameters, light field data formatting parameters or light field processing parameters of the camera. The user interface  160  may comprise any suitable user input device(s) such as a touchscreen, buttons, keyboard, pointing device, and/or the like. In this way, input received by the user interface can be used to control and/or configure the LF data formatting module  150  for controlling the data formatting, the LF data processor  155  for controlling the processing of the acquired light field data and controller  170  for controlling the light field camera  100 . 
     The light field camera includes a power source  180 , such as one or more replaceable or rechargeable batteries. The light field camera comprises memory  190  for storing captured light-field data and/or rendered final images or other data such as software for implementing methods of embodiments of the invention. The memory can include external and/or internal memory. In at least one embodiment, the memory can be provided at a separate device and/or location from camera  100 . In one embodiment, the memory includes a removable/swappable storage device such as a memory stick. 
     The light field camera may also include a display unit  165  (e.g., an LCD screen) for viewing scenes in front of the camera prior to capture and/or for viewing previously captured and/or rendered images. The screen  165  may also be used to display one or more menus or other information to the user. The light field camera may further include one or more I/O interfaces  195 , such as FireWire or Universal Serial Bus (USB) interfaces, or wired or wireless communication interfaces for data communication via the Internet, a cellular data network, a WiFi network, a BlueTooth communication protocol, and/or any other suitable means. The I/O interface  195  may be used for transferring data, such as meta data formatted by LF data formatting module in accordance with embodiments of the invention and light field data such as raw light field data or data processed by LF data processor  155 , to and from external devices such as computer systems or display units, for rendering applications. 
       FIG. 2B  is a block diagram illustrating an embodiment of a potential implementation for of light field data formatting module and the light field data processor. 
     The circuit  300  includes memory  390 , a memory controller  345  and processing circuitry  340  comprising one or more processing units (CPU(s)). The one or more processing units  340  are configured to run various software programs and/or sets of instructions stored in the memory  390  to perform various functions including light field data formatting and light field data processing. Software components stored in the memory include a data formatting module (or set of instructions)  350  for providing a data format aggregating light field metadata in accordance with embodiments of the invention and a light field data processing module (or set of instructions)  355  for processing light field data in accordance with embodiments of the invention, including a format conversion module  3555  (or set of instructions) for converting one format of light field data to another different format of light field data in accordance with embodiments of the invention. Other modules may be included in the memory for applications of the light field camera device such as an operating system module  351  for controlling general system tasks (e.g. power management, memory management) and for facilitating communication between the various hardware and software components of the device  300 , and an interface module  352  for controlling and managing communication with other devices via I/O interface ports. 
       FIG. 3  illustrates an example of a 2D image formed on the photosensor array  120  of  FIG. 1A  or the photosensor array  220  of  FIG. 1B . The image is composed of an array of micro images MI, each micro image being produced by the respective micro lens (i,j) of the microlens array  110 , 210 . The micro images are arranged in the array in a rectangular lattice structure defined by axes i and j. A micro lens image is referenced by the respective micro lens coordinates (i,j). A pixel PI of the photosensor  120 ,  220  is referenced by its spatial coordinates (x,y). 4D data associated with a given pixel may be referenced as (x,y,i,j). 
     Embodiments of the invention provide a data format that aggregates metadata describing for raw light field data content, for matrices of views, and for focal stacks. In some embodiments the metadata may be used for processing light field data, for example for converting the light field data content from one format to another, e.g. from raw light field data into matrix of views or focal stacks; or from matrix of views into focal stacks; or for common light field applications such as viewpoint change and refocusing. 
       FIG. 4  outlines steps of a method of processing and formatting light field data in accordance with one or more embodiments of the invention. In step S 401  light field data is acquired by data acquisition module  140  from a plenoptic camera such as the camera shown in  FIG. 1A  or a multi camera array such as the camera shown in  FIG. 1B . Lightfield data processor  155  receives the light field data in raw format. In step S 402  the raw light field data is converted into a matrix of views (sub-aperture images) by a process of de-mosaicking and re-gridding. In step S 403  the raw light field data is converted into a focal stack (set of images at different focal planes) by a process of splatting (accumulating shifted microimages). In step S 404  the matrix of views is converted into a focal stack by a process of splatting (accumulating shifted sub-aperture images). 
     In step S 410  metadata for the acquired light field data is provided and formatted. The metadata includes camera metadata defining technical characteristics of the camera, view matrix metadata defining the matrices of views and focal stack metadata defining the focal stacks; View point changes may be performed in step S 420  using the matrix of views and light field meta data defining the matrix of views. Refocusing may be performed in step S 430  using the focal stack and metadata defining the focal stack. A 2D image is generated from the micro-images by shifting the micro images by a given disparity and adding the images to form a single 2D image. In order to refocus the image at a given distance z, the micro-images are shifted by a disparity associated with the distance z. In order to perform a change of viewpoint only a part of the micro-images are added (e.g. left/right portions for left/right viewpoints) 
     In embodiments of the invention formatting syntax for metadata characterising the light field data content is provided. The metadata characterising the light field data content includes: 
     Camera characteristic data: characterising the sensor, the main lens and the p-lens array of the light field camera capturing the light field data and in particular the positions of the centres of the p-lens; The metadata may include data representative of sensor resolution, color sampling, color filter transmittance, dynamic range, color transform matrix, pixel pitch, lens pitch, focal length, vignetting, for example 
     View characteristic data—data characterising matrices of views such as metadata that characterizes the respective intrinsic &amp; extrinsic parameters of each view/sub-aperture image; and color information of each view/sub-aperture image; 
     Focal stack data Data characterising focal stacks such as metadata that characterizes the focalization distance of each focused picture; for example an index map. 
     The formatting syntax may also include basic information needed to recover file location and file names for the management of video and sequence of frames. 
     The metadata may be provided as digital data stored in a file or a set of files. In one embodiment of the invention the formatting syntax is presented as a markup language complying for example with XML rules. In what follows the formatting syntax is referred to as light field markup language (abbreviated to lfml). In the example that follows the formatting syntax for the metadata includes the definition of sections (enclosed with tags) and of the corresponding attributes (the metadata itself). 
     Root Section and Type Attribute 
     Table 1 illustrates an example of a root section and attribute of a light field metadata file. The body of an *.lfml file is enclosed in a &lt;LightFieldContent&gt; section. It contains an attribute specifying the type of content, denoted in the example as LFtype=“RAW” indicates raw plenoptic picture, indicates “MATRIX_OF_VIEWS” for arrays of views, and “FOCAL_STACK” for focal stacks. 
     
       
         
           
               
             
               
                 TABLE 1 
               
               
                   
               
               
                 Root section and attribute of an exemplary *.lfml file 
               
               
                   
               
             
            
               
                   
               
            
           
           
               
               
            
               
                   
                 &lt;LightFieldCcntent 
               
               
                   
                 lfml:LFtype=“RAW” 
               
               
                   
                 &gt; 
               
               
                   
                      [...] 
               
               
                   
                 &lt;/LightFieldContent&gt; 
               
               
                   
                   
               
            
           
         
       
     
     Folder and File Attributes 
     File naming attributes can be found at the root level if every file is stored and named consistently. File naming attributes can also be found in lower-level sections if some files do not share the same path or prefix or numbering. 
     The path attribute is a string specifying the folder path.
 
The filePrefix attribute is a string specifying the prefix of filenames.
 
The storageMode attribute is a two-part string separated by a colon that describes the file storage mode, and the file format used. It may either of the following: “FILE_SEQUENCE:PNG”; “FILE_SEQUENCE:PGM”; “FILE_SEQUENCE:RAW”; “VIDEO FILE:YUV”; “VIDEO FILE:AVI”.
 
     The firstFrameIdx and lastFrameIdx attributes are indices of the first and the last frame of the light field sequence respectively. The digitSuffix attribute is a string, for example of hash symbols, specifying the number of digits dedicated to frame numbering in the filename suffix. In the case of still-imaging, these three fields may be missing or empty (see Table 2 and  FIG. 5 ). In the case of sequence of images, filenames are suffixed with the index of the frames (see  FIG. 3  and  FIG. 6 ). 
     
       
         
           
               
             
               
                 TABLE 2 
               
               
                   
               
               
                 File naming attributes for a raw light field still image 
               
               
                   
               
             
            
               
                   
               
            
           
           
               
               
            
               
                   
                 &lt;LightFieldCcntent 
               
               
                   
                 lfml:LFtype=“RAW” 
               
               
                   
                 lfml:path=“.\Lytro\Gallery” 
               
               
                   
                 lfml:filePrefix=“Snake” 
               
               
                   
                 lfml:storageMode=“FILE_SEQUENCE:RAW” 
               
               
                   
                 lfml:firstFrameIdx=“” 
               
               
                   
                 lfml:lastFrameIdx=“” 
               
               
                   
                 lfml:digitSuffix=“” 
               
               
                   
                 &gt; 
               
               
                   
                 [...] 
               
               
                   
                 &lt;/LightFieldContent&gt; 
               
               
                   
                   
               
            
           
         
       
     
     
       
         
           
               
             
               
                 TABLE 3 
               
               
                   
               
               
                 File naming attributes for light field video 
               
               
                 (here a CGI multi-view video sequence) 
               
               
                   
               
             
            
               
                   
               
            
           
           
               
               
            
               
                   
                 &lt;LightFieldCcntent 
               
               
                   
                   lfml:LFtype=“MATRIX_OF_VIEWS” 
               
               
                   
                   lfml:path=“.\CGI\Blender” 
               
               
                   
                   lfml:filePrefix=“outdoor” 
               
               
                   
                   lfml:storageMode=“FILE_SEQUENCE:PNG” 
               
               
                   
                   lfml:firstFrameIdx=“0” 
               
               
                   
                   lfml:lastFrameIdx=“249” 
               
               
                   
                   lfml:digitSuffix=“####” 
               
               
                   
                 &gt; 
               
               
                   
                 [...] 
               
               
                   
                 &lt;/LightFieldContent&gt; 
               
               
                   
                   
               
            
           
         
       
     
     It may be noted that in the case of matrices of views and focal stacks, files may be fetched by default in folders named for instance respectively \view# and \focus# (see for example  FIG. 6 ). Filenames may also be built by default following the prefix and suffix rules predefined. However the attributes path, folder, filename and suffix can be used in subsections in order to specify alternative locations and names. 
     Coordinate System Attributes 
     Dealing with light field imaging requires coordinate system conventions. The coordinate system may be implicit with default values being pre-determined, or explicitly stated in the description file with the attributes CSx and CSy, which can respectively have the following values: “leftward”/“rightward” and “upward”/“downward”. 
     Coordinate systems attributes can be located at any level of the *.lfml file, up to the root level provided that all the coordinate systems used are consistent with each other. 
     Raw Plenoptic Pictures 
     In the case of raw plenoptic pictures (LFtype=“RAW”), a lightfield metadata file(*.lfml file) present a section &lt;RawPlenopticPictures&gt;, that may include one or more of the following sections, depending on the completeness of the calibratio: &lt;Sensor&gt;, &lt;MainLens&gt; and &lt;μLensArray&gt; (see Table 4). 
     The &lt;Sensor&gt; section may contain one or more of the following attributes describing technical features of the light field camera. Such attributes may be provided by the manufacturer: 
     The raysResolution attribute specifies the resolution of the sensor (“Width×Height”, e.g. “4008×2672” for the Raytrix R11 camera, “3280×3280” for the first Lytro ‘LightField’ camera, etc.). 
     The colorFilter attribute specifies the color sampling on the sensor (e.g. “BayerGRBG” or “Foveonx3RGB” or “Monochrome”). 
     The colorFilterTransmittance attribute specifies the transmittance of the color filters of the sensor. It has one or three or four values, depending on the color filter. This attribute may be missing or empty. 
     The dynamicRange attribute specifies the range of the pixels. It has two values, the lower and upper bounds beyond which the signal can be considered as noise. 
     The colorTransformMatrix is a 3×3 matrix that specifies the non-linear relationship between the signal output by a pixel and the incoming light. 
     The pixelPitch attribute specifies the distance between two adjacent pixels, in meters. 
     The scaleFactorX and scaleFactorY attributes can be used to specify the shape ratio of rectangular (not square) pixels. 
     The &lt;MainLens&gt; section contains one or more of the following attributes, that may be provided by the manufacturer: 
     The focalLength attribute specifies the focal length of the main lens, in meters. 
     The numericalAperture attribute specifies the ratio between the pupil and the focal length. That attribute may be missing or empty. 
     The whiteImage attribute specifies a folder location from where to retrieve white images that correspond to the actual focal length. White images are used to remove the effect of vignetting. That attribute may be missing or empty. 
     The vignetting attribute specifies a model for main lens vignetting. That attribute may be missing or empty. 
     The &lt;μLensArray&gt; section contains the following attributes. Some may be provided by the manufacturer and some may be determined by further calibratio. 
     The pattern attribute specifies the pattern of the μdens array (“Square”, or “HexagonalRowMajor”, or “HexagonalColumnMajor”). 
     If applicable, a colorArrangement attribute specifies the pattern of color filters arranged on the μ-lens array. That attribute may be missing or empty. 
     The μLensPitch attribute specifies the distance between two adjacent μdens, in meters. 
     The μLensArrayRotation attribute specifies the rotation angle (in degrees) of the μ-lens array with respect to the sensor grid, in the coordinate system defined. 
     The scaleFactorX and scaleFactorY attributes can be used to specify the shape ratio of μ-lenses. 
     The nbμCentersCalibrated attribute specifies the numbers of centers of micro-images (μ-centers) (i.e. the number of centers of μ-image) calibrated. It can be equal to zero. 
     The &lt;μLensArray&gt; section contains the corresponding number of μCenter&gt; subsections (see Table 5), that contain the following attributes: 
     The μCenterId attribute is the index of the μ-center. 
     The cx and cy attributes respectively specify the x- and y-coordinates (in pixels) of the μ-center. 
     
       
         
           
               
             
               
                 TABLE 4 
               
               
                   
               
               
                 An example of *.lfml file for a raw light field picture 
               
               
                   
               
             
            
               
                   
               
            
           
           
               
            
               
                 &lt;LightFieldCcntent 
               
               
                   lfml:LFtype=″RAW″ 
               
               
                   lfml:path=″.\Lytro\Gallery″ 
               
               
                   lfml:filePrefix=″Snake″ 
               
               
                   lfml:storageMode=″FILE_SEQUENCE:RAW″ 
               
               
                   lfml:firstFrameIdx=″″ 
               
               
                   lfml:lastFrameIdx=″″ 
               
               
                   lfml:digitSuffix=″″ 
               
               
                   lfml:CSx=″rightward″ 
               
               
                   lfml:CSy=″downward″ 
               
               
                 &gt; 
               
               
                   &lt;RawPlencpticPictures&gt; 
               
               
                     &lt;Sensor 
               
               
                       lfml:raysResolution=″3280×3280″ 
               
               
                       lfml:colorFilter=″BayerGRBG″ 
               
               
                       lfml:colorFilterTransmittance=″″ 
               
               
                       lfml:dynamicRange=″″ 
               
               
                       lfml:colorTransformMatrix=″″ 
               
               
                       lfml:pixelPitch=″0.000001399999976158141876680929″ 
               
               
                       lfml:scaleFactorX=″1.0″ 
               
               
                       lfml:scaleFactorY=″1.0″ 
               
               
                     &gt;&lt;/Sensor&gt; 
               
               
                     &lt;MainLens 
               
               
                       lfml:focalLength=″0.05131999969482421875″ 
               
               
                       lfml:numericalAperture=″1.9″ 
               
               
                       lfml:whiteImage=”” 
               
               
                       lfml:vignetting=”” 
               
               
                     &gt;&lt;/MainLens&gt; 
               
               
                     &lt;μLensArray 
               
               
                       lfml:pattern=″HexagonalRowMajor″ 
               
               
                       lfml:μLensPitch=″0.00001389861488342285067432158″ 
               
               
                       lfml:μLensArrayRotation= 
               
               
                       ″−0.002579216146841645240783691406″ 
               
               
                       lfml:scaleFactorX=″1.0″ 
               
               
                       lfml:scaleFactorY=″1.00024712085723876953125″ 
               
               
                       lfml:nbμCentersCalibrated=″0″ 
               
               
                     &gt; 
               
               
                     &lt;/μLensArray&gt; 
               
               
                   &lt;/RawPlenopticPictures&gt; 
               
               
                 &lt;/LightFieldContent&gt; 
               
               
                   
               
            
           
         
       
     
     
       
         
           
               
             
               
                 TABLE 5 
               
               
                   
               
               
                 An example of μ-center subsection 
               
               
                   
               
             
            
               
                   
               
            
           
           
               
               
            
               
                   
                 LightFieldContent 
               
               
                   
                   lfml:LFtype=“RAW” 
               
               
                   
                   [...] 
               
               
                   
                 &gt; 
               
               
                   
                   &lt;RawPlencpticPictures&gt; 
               
               
                   
                     [...] 
               
               
                   
                     &lt;μLensArray 
               
               
                   
                       [...] 
               
               
                   
                       lfml:nbμCentersCalibrated=“108900” 
               
               
                   
                     &gt; 
               
               
                   
                       [...] 
               
               
                   
                       &lt;μCenter 
               
               
                   
                         lfml:μCenterId=“27224” 
               
               
                   
                         lfml:cx=“1633.087277” 
               
               
                   
                         lfml:cy=“1633.490847” 
               
               
                   
                       &gt;&lt;/μCenter&gt; 
               
               
                   
                       [...] 
               
               
                   
                     &lt;/μLensArray&gt; 
               
               
                   
                   &lt;/RawPlenopticPictures&gt; 
               
               
                   
                 &lt;/LightFieldContent&gt; 
               
               
                   
                   
               
            
           
         
       
     
     Matrices of Views 
     In the case of matrices (or arrays) of views (LFtype=“MATRIX_OF_VIEWS”), *.lfml files present a section &lt;MatrixOfViews&gt;, that encloses several &lt;SubApertureImage&gt; subsections. The syntax suits both a set of sub-aperture images reconstructed from raw plenoptic material (see Table 6) and the content acquired from a camera array (see Table 7). 
     The &lt;MatrixOfViews&gt; section contains the following attributes: 
     The nbSubApertureImages specifies the number of views/sub-aperture images. The &lt;MatrixOfViews&gt; section contains the corresponding number of &lt;SubApertureImage&gt; subsections. 
     The pixel Resolution attribute specifies the resolution of views/sub-aperture images in pixels (“Width×Height”), assuming that it is the same for all views. If not, this attribute is is filled in every subsequent &lt;SubApertureImage&gt; subsection. That attribute may be missing or empty if available in the headers of corresponding picture files. 
     The &lt;SubApertureImage&gt; subsections contain the following attributes: 
     The subApImId attribute is the index of the view/sub-aperture image. 
     The extrinsics attribute specifies the extrinsic matrix of the view/sub-aperture image in the coordinate system defined. 
     The color attribute specifies the color(s) available in the view/sub-aperture image (“R”, “G”, “B”, “RGB”, “L”). This attribute can be filled in the &lt;MatrixOfViews&gt; section if shared by every view/sub-aperture image. 
     The cx and cy attributes respectively specify the x- and y-coordinates (in pixels) of the principal point. These attributes may be missing or empty. They can also be filled in the &lt;MatrixOfViews&gt; section if shared by every view/sub-aperture image. 
     The fx and fy attributes respectively specify the horizontal and vertical focal length (in pixels) of the view/sub-aperture view. These attributes may be missing or empty. They can also be filled in the &lt;MatrixOfViews&gt; section if shared by every view/sub-aperture image. 
     The numericalAperture attribute specifies the ratio between the pupil and the focal length, like in the &lt;MainLens&gt; section. That attribute may be missing or empty. It can also be filled in the &lt;MatrixOfViews&gt; section if shared by every view/sub-aperture image. 
     The distortionCoefficients attribute specifies a distortion model (for example, Bouguet&#39;s 5-parameter model of radial and tangential distortions). That attribute may be missing or empty. It can also be filled in the &lt;MatrixOfViews&gt; section if shared by every view/sub-aperture image. 
     
       
         
           
               
             
               
                 TABLE 6 
               
               
                   
               
               
                 An example of an excerpt of *.lfml file describing the matrix of views 
               
               
                 reconstructed from a Pelican Imaging camera device 
               
               
                   
               
             
            
               
                   
               
            
           
           
               
            
               
                 &lt;LightFieldCcntent 
               
               
                   lfml:LFtype=“MATRIX_OF_VIEWS” 
               
               
                   lfml:path=“.\PelicanImaging\Gallery” 
               
               
                   lfml:filePrefix=“GoldenGate” 
               
               
                   lfml:storageMode=“FILE_SEQUENCE:PGM” 
               
               
                   lfml:firstFrameIdx=“” 
               
               
                   lfml:lastFrameIdx=“” 
               
               
                   lfml:digitSuffix=“” 
               
               
                   lfml:CSx=“rightward” 
               
               
                   lfml:CSy=“downward” 
               
               
                 &gt; 
               
               
                   &lt;MatrixOfViews 
               
               
                     lfml:nbSubApertureImages=“16” 
               
               
                     lfml:pixelResolution=“1000×750” 
               
               
                   &gt; 
               
               
                     &lt;SubApertureImage 
               
               
                       lfml:subApImId=“0” 
               
               
                       lfml:extrinsics=“[[1, 0, 0, 0], [0, 1, 0, 0], [0, 0, 1, 0], 
               
               
                       [0, 0, 0, 1]]” 
               
               
                       lfml:color=“G” 
               
               
                       lfml:cx=“499” 
               
               
                       lfml:cy=“374” 
               
               
                       lfml:fx=“ 1175.0” 
               
               
                       lfml:fy=“1175.0” 
               
               
                       lfml:numericalAperture=“” 
               
               
                       lfml:distortionCoefficients=“” 
               
               
                     &gt;&lt;/SubApertureImage&gt; 
               
               
                     &lt;SubApertureImage 
               
               
                       lfml:subApImId=“1” 
               
               
                       lfml:extrinsics=“[[1, 0, 0, 0.005], [0, 1, 0, 0], [0, 0, 1, 0], 
               
               
                 [0, 0, 0, 1]]” 
               
               
                       lfml:color=“R” 
               
               
                       lfml:cx=“500” 
               
               
                       lfml:cy=“374” 
               
               
                       lfml:fx=“1175.4” 
               
               
                       lfml:fy=“1175.4” 
               
               
                       lfml:numericalAperture=“” 
               
               
                       lfml:distortionCoefficients=“” 
               
               
                     &gt;&lt;/SubApertureImage&gt; 
               
               
                     &lt;SubApertureImage 
               
               
                       lfml:subApImId=“2” 
               
               
                       lfml:extrinsics=“[[1, 0, 0, 0.01], [0, 1, 0, 0], [0, 0, 1, 0], 
               
               
                 [0, 0, 0, 1]]” 
               
               
                       lfml:color=“G” 
               
               
                       lfml:cx=“499” 
               
               
                       lfml:cy=“375” 
               
               
                       lfml:fx=“1175.9” 
               
               
                       lfml:fy=“1175.9” 
               
               
                       lfml:numericalAperture=“” 
               
               
                       lfml:distortionCoefficients=“” 
               
               
                     &gt;&lt;/SubApertureImage&gt; 
               
               
                     &lt;SubApertureImage 
               
               
                       lfml:subApImId=“3” 
               
               
                       lfml:extrinsics=“[[1, 0, 0, 0.015], [0, 1, 0, 0], [0, 0, 1, 0], 
               
               
                 [0, 0, 0, 1]]” 
               
               
                       lfml:color=“B” 
               
               
                       lfml:cx=“500” 
               
               
                       lfml:cy=“375” 
               
               
                       lfml:fx=“1175.0” 
               
               
                       lfml:fy=“1175.0” 
               
               
                       lfml:numericalAperture=“” 
               
               
                       lfml:distortionCoefficients=“” 
               
               
                     &gt;&lt;/SubApertureImage&gt; 
               
               
                      [...] 
               
               
                   &lt;/MatrixCfViews&gt; 
               
               
                 &lt;/LightFieldContent&gt; 
               
               
                   
               
            
           
         
       
     
     
       
         
           
               
             
               
                 TABLE 7 
               
               
                   
               
               
                 An excerpt of */lfml file describing a computer-generated matrix of views. 
               
               
                   
               
             
            
               
                   
               
            
           
           
               
            
               
                 &lt;LightFieldCcntent 
               
               
                   lfml:LFtype=“MATRIX_OF_VIEWS” 
               
               
                   lfml:path=“.\CGI\Blender” 
               
               
                   lfml:filePrefix=“outdoor” 
               
               
                   lfml:storageMode=“FILE_SEQUENCE:PNG” 
               
               
                   lfml:firstFrameIdx=“0” 
               
               
                   lfml:lastFrameIdx=“249” 
               
               
                   lfml:digitSuffix=“####” 
               
               
                   lfml:CSx=“rightward” 
               
               
                   lfml:CSy=“downward” 
               
               
                 &gt; 
               
               
                   &lt;MatrixOfViews 
               
               
                     lfml:nbSubApertureImages=“121” 
               
               
                     lfml:pixelResolution=“1920×1080” 
               
               
                     lfml:color=“RGB” 
               
               
                     lfml:cx=“960” 
               
               
                     lfml:cy=“540” 
               
               
                     lfml:fx=“3026.218853” 
               
               
                     lfml:fy=“3026.218853” 
               
               
                     lfml:numericalAperture=”0.” 
               
               
                 &gt; 
               
               
                     &lt;SubApertureImage 
               
               
                       lfml:subApImId=“0” 
               
               
                       lfml:extrinsics=“[[1, 0, 0, 0], [0, 1, 0, 0], [0, 0, 1, 0], 
               
               
                       [0, 0, 0, 1]]” 
               
               
                       lfml:distortionCoefficients=“” 
               
               
                     &gt;&lt;/SubApertureImage&gt; 
               
               
                     &lt;SubApertureImage 
               
               
                       lfml:subApImId=“1” 
               
               
                       lfml:extrinsics=“[[1, 0, 0, 0.1], [0, 1, 0, 0], [0, 0, 1, 0], 
               
               
                       [0, 0, 0, 1]]” 
               
               
                       lfml:distortionCoefficients=“” 
               
               
                     &gt;&lt;/SubApertureImage&gt; 
               
               
                     [...] 
               
               
                     &lt;SubApertureImage 
               
               
                       lfml:subApImId=“249” 
               
               
                       lfml:extrinsics=“[[1, 0, 0, 1.0], [0, 1, 0, 1.0], [0, 0, 1, 0], 
               
               
                 [0, 0, 0, 1]]” 
               
               
                       lfml:distortionCoefficients=“” 
               
               
                     &gt;&lt;/SubApertureImage&gt; 
               
               
                   &lt;/MatrixCfViews&gt; 
               
               
                 &lt;/LightFieldContent&gt; 
               
               
                   
               
            
           
         
       
     
     Focal Stacks 
     In the case of focal stacks (LFtype=“FOCAL_STACK”), *.lfml files present a section &lt;FocalStack&gt;, that encloses several &lt;FocalStack&gt; subsections and optionally a subsection &lt;IndexMap&gt; (see Erreur ! Source du renvoi introuvable.  8 ). 
     The &lt;FocalStack&gt; section contains the following attributes: 
     The nbFocusedPictures specifies the number of focused pictures in the stack. The &lt;FocalStack&gt; section contains the corresponding number of &lt;FocusedPicture&gt; subsections. 
     The focusedPicldxMap attribute specifies whether an index map is available or not (“AVAILABLE” or “UNAVAILABLE”). 
     As in the case of matrices of views, the pixel Resolution attribute specifies the resolution of the focused pictures in pixels (“Width×Height”), assuming that it is the same for everyone. If not, this attribute is filled in every subsequent &lt;FocusedPicture&gt; subsection. That attribute may be missing or empty if available in the headers of corresponding picture files. 
     The &lt;FocusedPicture&gt; subsections contain the following attributes: 
     The focusPicld attribute is the index of the focus picture. 
     The distance attribute specifies the distance of focalization of the focused picture, in meters. That attribute may be missing or empty. 
     The vergence attribute specifies the inverse of the distance of focalization of the focused picture (in m −1 ). That attribute may be missing or empty. 
     The &lt;IndexMap&gt; subsection contain the attributes needed to retrieve the corresponding file (filename or suffix, optionally path or folder). 
     
       
         
           
               
             
               
                 TABLE 8 
               
               
                   
               
               
                 An example of *lfml file for a focal stack 
               
               
                   
               
             
            
               
                   
               
            
           
           
               
               
            
               
                   
                 &lt;LightFieldCcntent 
               
               
                   
                   lfml:LFtype=“FOCAL_STACK” 
               
               
                   
                   lfml:path=“.\Lytro\Gallery” 
               
               
                   
                   lfml:filePrefix=“Snake” 
               
               
                   
                   lfml:storageMode=“FILE_SEQUENCE:PNG” 
               
               
                   
                   lfml:firstFrameIdx=“” 
               
               
                   
                   lfml:lastFrameIdx=“” 
               
               
                   
                   lfml:digitSuffix=“” 
               
               
                   
                   lfml:CSx=“rightward” 
               
               
                   
                   lfml:CSy=“downward” 
               
               
                   
                 &gt; 
               
               
                   
                   &lt;FocalStack 
               
               
                   
                     lfml:nbFocusedPictures=“8” 
               
               
                   
                     lfml:focusedPicIdxMap=“AVAILABLE” 
               
               
                   
                     lfml:pixelResolution=“1000×1000” 
               
               
                   
                   &gt; 
               
               
                   
                     &lt;FocusedPicture 
               
               
                   
                       lfml:focusPicId=“0” 
               
               
                   
                       lfml:distance=“” 
               
               
                   
                       lfml:vergence=“” 
               
               
                   
                     &gt;&lt;/FccusedPicture&gt; 
               
               
                   
                     &lt;FocusedPicture 
               
               
                   
                       lfml:focusPicId=“1” 
               
               
                   
                       lfml:distance =“” 
               
               
                   
                       lfml:vergence=“” 
               
               
                   
                     &gt;&lt;/FccusedPicture&gt; 
               
               
                   
                     &lt;FocusedPicture 
               
               
                   
                       lfml:focusPicId=“2” 
               
               
                   
                       lfml:distance =“” 
               
               
                   
                       lfml:vergence=“” 
               
               
                   
                     &gt;&lt;/FccusedPicture&gt; 
               
               
                   
                     &lt;FocusedPicture 
               
               
                   
                       lfml:focusPicId=“3” 
               
               
                   
                       lfml:distance =“” 
               
               
                   
                       lfml:vergence=“” 
               
               
                   
                     &gt;&lt;/FccusedPicture&gt; 
               
               
                   
                     &lt;FocusedPicture 
               
               
                   
                       lfml:focusPicId=“4” 
               
               
                   
                       lfml:distance =“” 
               
               
                   
                       lfml:vergence=“” 
               
               
                   
                     &gt;&lt;/FccusedPicture&gt; 
               
               
                   
                     &lt;FocusedPicture 
               
               
                   
                       lfml:focusPicId=“5” 
               
               
                   
                       lfml:distance =“” 
               
               
                   
                       lfml:vergence=“” 
               
               
                   
                     &gt;&lt;/FccusedPicture&gt; 
               
               
                   
                     &lt;FocusedPicture 
               
               
                   
                       lfml:focusPicId=“6” 
               
               
                   
                       lfml:distance =“” 
               
               
                   
                       lfml:vergence=“” 
               
               
                   
                     &gt;&lt;/FccusedPicture&gt; 
               
               
                   
                     &lt;FocusedPicture 
               
               
                   
                       lfml:focusPicId=“7” 
               
               
                   
                       lfml:distance =“” 
               
               
                   
                       lfml:vergence=“” 
               
               
                   
                     &gt;&lt;/FccusedPicture&gt; 
               
               
                   
                     &lt;IndexMap 
               
               
                   
                       lfml:suffix=“indexMap” 
               
               
                   
                     &gt;&lt;/IndexMap&gt; 
               
               
                   
                   &lt;/FocalStack&gt; 
               
               
                   
                 &lt;/LightFieldContent&gt; 
               
               
                   
                   
               
            
           
         
       
     
     The metadata may be used for processing light field data, for example converting the light field data content from one format to another, e.g. from raw light field data into matrix of views or focal stacks; or from matrix of views into focal stacks; and for common light field applications such as viewpoint change and refocusing. Examples of processing performed, in accordance with embodiments of the invention, on the LF field data by the light field data processor will now be described. 
     Format Conversion 
     From Raw Plenoptic Content to Matrices of Views 
     Arrays of views (micro images) can be reconstructed by de-mosaicking and re-gridding raw light field data acquired from the light field camera. 
     (R i,j ) denotes a set of micro-images in the raw light field picture. A sub-aperture image V k,l  is composed of collocated samples from each p-image: V k,l [i,j]=R i,j [k,l] 
     The focal lengths (in pixels) of the corresponding array of views, i.e. the horizontal focal length fx and the vertical focal fy attributes of the metadata describing the matrix of views, are given by: 
     
       
         
           
             
               
                 
                   { 
                   
                     
                       
                         
                           
                             f 
                             x 
                           
                           = 
                           
                             F 
                             
                               
                                 s 
                                 x 
                               
                               · 
                               
                                 r 
                                 p 
                               
                             
                           
                         
                       
                     
                     
                       
                         
                           
                             f 
                             y 
                           
                           = 
                           
                             F 
                             
                               
                                 s 
                                 y 
                               
                               · 
                               
                                 r 
                                 p 
                               
                             
                           
                         
                       
                     
                   
                 
               
             
           
         
       
     
     where
         F denotes the focal length of the main lens, in meters (i.e. the value of the attribute focalLength of the section &lt;MainLens&gt;);   s x  and s y  respectively denote horizontal and vertical scaling factors of the sensor (i.e. the values of the attributes scaleFactorX and scaleFactorY of the section &lt;Sensor&gt;);   and r p  denotes the pixel pitch, in meters (i.e. the value of the attribute pixelPitch of the section &lt;Sensor&gt;).       

     And the underlying inter-axial baseline distances are given by: 
     
       
         
           
             
               
                 
                   { 
                   
                     
                       
                         
                           
                             b 
                             x 
                           
                           = 
                           
                             
                               
                                 s 
                                 x 
                               
                               · 
                               
                                 s 
                                 y 
                               
                               · 
                               
                                 r 
                                 p 
                                 2 
                               
                             
                             
                               
                                 σ 
                                 x 
                               
                               · 
                               
                                 r 
                                 
                                   µ 
                                    
                                   
                                       
                                   
                                    
                                   L 
                                 
                               
                             
                           
                         
                       
                     
                     
                       
                         
                           
                             b 
                             y 
                           
                           = 
                           
                             
                               
                                 s 
                                 x 
                               
                               · 
                               
                                 s 
                                 y 
                               
                               · 
                               
                                 r 
                                 p 
                                 2 
                               
                             
                             
                               
                                 σ 
                                 y 
                               
                               · 
                               
                                 r 
                                 
                                   µ 
                                    
                                   
                                       
                                   
                                    
                                   L 
                                 
                               
                             
                           
                         
                       
                     
                   
                 
               
             
           
         
       
     
     where
         r μL  denotes the p-lens pitch, in meters (i.e. the value of the attribute μLensPitch);   σ x  and σ y  respectively denote horizontal and vertical scaling factors of the μ-lens array (i.e. the values of the attributes scaleFactorX and scaleFactorY of the section &lt;μLensArray&gt;).       

     The extrinsic matrix of the view V k,l  (that corresponds to the extrinsics attribute of the section &lt;SubApertureImage&gt;) is therefore made of the following rotation and translation matrices: 
     
       
         
           
             
               
                 
                   R 
                   = 
                   
                     
                       
                         ( 
                         
                           
                             
                               1 
                             
                             
                               0 
                             
                             
                               0 
                             
                           
                           
                             
                               0 
                             
                             
                               1 
                             
                             
                               0 
                             
                           
                           
                             
                               0 
                             
                             
                               0 
                             
                             
                               1 
                             
                           
                         
                         ) 
                       
                        
                       
                           
                       
                        
                       and 
                        
                       
                           
                       
                        
                       T 
                     
                     = 
                     
                       ( 
                       
                         
                           
                             
                               k 
                               · 
                               
                                 b 
                                 x 
                               
                             
                           
                         
                         
                           
                             
                               l 
                               · 
                               
                                 b 
                                 y 
                               
                             
                           
                         
                         
                           
                             0 
                           
                         
                       
                       ) 
                     
                   
                 
               
             
           
         
       
     
     assuming that the optical center of view V 00  is chosen as the origin of the coordinate system. 
     From Raw Plenoptic Content to Focal Stacks 
     Focal stacks can be computed by accumulating (“splatting”) shifted μ-images. 
     w denotes the shift defining a focused picture. Considering the contribution of a μ-image R i,j , for each of its pixel 
     
       
         
           
             
               ( 
               
                 
                   
                     x 
                   
                 
                 
                   
                     y 
                   
                 
               
               ) 
             
             , 
           
         
       
     
     new coordinates in the focal stack are given by: 
     
       
         
           
               
             
               { 
               
                 
                   
                     
                       X 
                       = 
                       
                         
                           u 
                           · 
                           x 
                         
                         - 
                         
                           u 
                           · 
                           w 
                           · 
                           i 
                         
                       
                     
                   
                 
                 
                   
                     
                       Y 
                       = 
                       
                         
                           u 
                           · 
                           y 
                         
                         - 
                         
                           u 
                           · 
                           w 
                           · 
                           j 
                         
                       
                     
                   
                 
               
             
           
         
       
     
     where 
     
       
         
           
               
             
               ( 
               
                 
                   
                     x 
                   
                 
                 
                   
                     y 
                   
                 
               
               ) 
             
           
         
       
         
         
           
             is expressed in the whole raw image coordinate system; 
             and u is a zooming factor. 
           
         
       
    
     The underlying focalization distance (in meters), i.e. the distance attribute of the section &lt;FocusedPicture&gt;, is given by: 
     
       
         
           
             
               
                 
                   z 
                   = 
                   
                     
                       
                         w 
                         - 
                         p 
                       
                       p 
                     
                     × 
                     F 
                   
                 
               
             
           
         
       
     
     where 
     
       
         
           
             p 
             = 
             
               
                 r 
                 
                   µ 
                    
                   
                       
                   
                    
                   L 
                 
               
               
                 r 
                 p 
               
             
           
         
       
     
     is the number of pixels corresponding to the diameter of a p-lens. 
     The vergence attribute is then its inverse. 
     From Matrices of Views to Focal Stacks 
     Focal stacks can also be computed by accumulating (“splatting”) shifted views, provided that they are registered. 
     The same formula can be used, considering this time the collection of sub-aperture images (V k,l ): 
     
       
         
           
               
             
               { 
               
                 
                   
                     
                       X 
                       = 
                       
                         
                           u 
                           · 
                           i 
                         
                         - 
                         
                           u 
                           · 
                           d 
                           · 
                           k 
                         
                       
                     
                   
                 
                 
                   
                     
                       Y 
                       = 
                       
                         
                           u 
                           · 
                           j 
                         
                         - 
                         
                           u 
                           · 
                           d 
                           · 
                           l 
                         
                       
                     
                   
                 
               
             
           
         
       
     
     where 
     
       
         
           
               
             
               ( 
               
                 
                   
                     i 
                   
                 
                 
                   
                     j 
                   
                 
               
               ) 
             
           
         
       
     
     is expressed in the coordinate system of each view. 
     The underlying focalization vergence (in m −1 ), i.e. the vergence attribute of the section &lt;FocusedPicture&gt;, is given indifferently by: 
     
       
         
           
             v 
             = 
             
               
                 1 
                 z 
               
               = 
               
                 d 
                 
                   
                     b 
                     x 
                   
                   · 
                   
                     f 
                     x 
                   
                 
               
             
           
         
       
     
     or 
     The corresponding distance attribute is the inverse of vergence v. 
     The proposed format of metadata describing light field data also suits common applications of light field data, such as viewpoint change and refocusing. 
     3D effects can be achieved by changing the viewpoint. This is obtained by view interpolation within the matrix of views. 
     Refocusing is a common application for focal stacks. The purpose involves stressing the depth of field, in which case the user picks one or several areas of the picture that he wants to be in focus (the remaining objects of the scene being as blurred as they are far from the focalized depth plane(s)). Another purpose consists in delivering all-in-focus pictures. In both ways, index maps are locally retrieved in order to fetch samples in one focalized picture of the focal stack. 
     Although the present invention has been described hereinabove with reference to specific embodiments, the present invention is not limited to the specific embodiments, and modifications will be apparent to a skilled person in the art which lie within the scope of the present invention. 
     Many further modifications and variations will suggest themselves to those versed in the art upon making reference to the foregoing illustrative embodiments, which are given by way of example only and which are not intended to limit the scope of the invention, that being determined solely by the appended claims. In particular the different features from different embodiments may be interchanged, where appropriate.