[107] G. Lippmann, « La photographie integrale », *Comptes-Rendus*, vol. 146, pp. 446-451, 1908.

[108] J. C. Yang, M. Everett, C. Buehler, and L. McMillan, « A real-time distributed light field camera. », *Rendering Techniques*, vol. 2002, 77-86, p. 2, 2002.

[109] B. Wilburn, N. Joshi, V. Vaish, M. Levoy, and M. Horowitz, « High-speed videography using a dense camera array », in *Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004.*, IEEE, vol. 2, 2004, pp. II-II.

[110] K. Venkataraman, D. Lelescu, J. Duparré, et al., « Picam: an ultra-thin high performance monolithic camera array », *ACM Transactions on Graphics (TOG)*, vol. 32, 6, pp. 1–13, 2013.

[111] C.-T. Huang, J. Chin, H.-H. Chen, Y.-W. Wang, and L.-G. Chen, « Fast realistic refocusing for sparse light fields », in *2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*, IEEE, 2015, pp. 1176–1180.

[112] E. H. Adelson and J. Y. Wang, « Single lens stereo with a plenoptic camera », *IEEE transactions on pattern analysis and machine intelligence*, vol. 14, 2, pp. 99–106, 1992.

[113] R. Ng, M. Levoy, M. Brédif, G. Duval, M. Horowitz, and P. Hanrahan, Ph.D. dissertation, Stanford university, 2005.

[114] R. A. Farrugia, C. Galea, and C. Guillemot, « Super resolution of light field images using linear subspace projection of patch-volumes », *IEEE Journal of Selected Topics in Signal Processing*, vol. 11, 7, pp. 1058–1071, 2017.

[115] M. Rossi and P. Frossard, « Geometry-consistent light field super-resolution via graph-based regularization », *IEEE Transactions on Image Processing*, vol. 27, 9, pp. 4207–4218, 2018.

[116] Y. Wang, F. Liu, K. Zhang, G. Hou, Z. Sun, and T. Tan, « Lfnet: a novel bidirectional recurrent convolutional neural network for light-field image super-resolution », *IEEE Transactions on Image Processing*, vol. 27, 9, pp. 4274–4286, 2018.

[117] R. A. Farrugia and C. Guillemot, « Light field super-resolution using a low-rank prior and deep convolutional neural networks », *IEEE transactions on pattern analysis and machine intelligence*, vol. 42, 5, pp. 1162–1175, 2019.
[118] T. E. Bishop, S. Zanetti, and P. Favaro, « Light field superresolution », in *2009 IEEE International Conference on Computational Photography (ICCP)*, IEEE, 2009, pp. 1–9.

[119] N. K. Kalantari, T.-C. Wang, and R. Ramamoorthi, « Learning-based view synthesis for light field cameras », *ACM Transactions on Graphics (TOG)*, vol. 35, 6, pp. 1–10, 2016.

[120] E. Penner and L. Zhang, « Soft 3d reconstruction for view synthesis », *ACM Transactions on Graphics (TOG)*, vol. 36, 6, pp. 1–11, 2017.

[121] J. Flynn, M. Broxton, P. Debevec, et al., « Deepview: view synthesis with learned gradient descent », in *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, 2019, pp. 2367–2376.

[122] S. Wanner and B. Goldluecke, « Variational light field analysis for disparity estimation and super-resolution », *IEEE transactions on pattern analysis and machine intelligence*, vol. 36, 3, pp. 606–619, 2013.

[123] Y. Yoon, H.-G. Jeon, D. Yoo, J.-Y. Lee, and I. S. Kweon, « Light-field image super-resolution using convolutional neural network », *IEEE Signal Processing Letters*, vol. 24, 6, pp. 848–852, 2017.

[124] N. Meng, X. Wu, J. Liu, and E. Lam, « High-order residual network for light field super-resolution », in *Proceedings of the AAAI Conference on Artificial Intelligence*, vol. 34, 2020, pp. 11757–11764.

[125] S. D. Babacan, R. Ansorge, M. Luessi, R. Molina, and A. K. Katsaggelos, « Compressive sensing of light fields », in *2009 16th IEEE International Conference on Image Processing (ICIP)*, IEEE, 2009, pp. 2337–2340.

[126] E. J. Candès, J. Romberg, and T. Tao, « Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information », *IEEE Transactions on information theory*, vol. 52, 2, pp. 489–509, 2006.

[127] D. L. Donoho, « Compressed sensing », *IEEE Transactions on information theory*, vol. 52, 4, pp. 1289–1306, 2006.

[128] R. G. Baraniuk, « Compressive sensing [lecture notes] », *IEEE signal processing magazine*, vol. 24, 4, pp. 118–121, 2007.
[129] K. Marwah, G. Wetzstein, Y. Bando, and R. Raskar, « Compressive light field photography using overcomplete dictionaries and optimized projections », *ACM Transactions on Graphics* (TOG), vol. 32, 4, pp. 1–12, 2013.

[130] E. Miandji, J. Unger, and C. Guillemot, « Multi-shot single sensor light field camera using a color coded mask », in *2018 26th European Signal Processing Conference (EUSIPCO)*, IEEE, 2018, pp. 226–230.

[131] H.-N. Nguyen, E. Miandji, and C. Guillemot, « Multi-mask camera model for compressed acquisition of light fields », *IEEE Transactions on Computational Imaging*, vol. 7, pp. 191–208, 2021.

[132] M. Gupta, A. Jauhari, K. Kulkarni, S. Jayasuriya, A. Molnar, and P. Turaga, « Compressive light field reconstructions using deep learning », in *Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops*, 2017, pp. 11–20.

[133] A. K. Vadathya, S. Cholleti, G. Ramajayam, V. Kanchana, and K. Mitra, « Learning light field reconstruction from a single coded image », in *2017 4th IAPR Asian Conference on Pattern Recognition (ACPR)*, IEEE, 2017, pp. 328–333.

[134] O. Nabati, D. Mendlovic, and R. Giryes, « Fast and accurate reconstruction of compressed color light field », in *2018 IEEE International Conference on Computational Photography (ICCP)*, IEEE, 2018, pp. 1–11.

[135] M. Guo, J. Hou, J. Jin, J. Chen, and L.-P. Chau, « Deep spatial-angular regularization for compressive light field reconstruction over coded apertures », in *Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16*, Springer, 2020, pp. 278–294.

[136] R. Ng, « Fourier slice photography », in *ACM Siggraph 2005 Papers*, 2005, pp. 735–744.

[137] J. R. Alonso, A. Fernández, and J. A. Ferrari, « Reconstruction of perspective shifts and refocusing of a three-dimensional scene from a multi-focus image stack », *Applied optics*, vol. 55, 9, pp. 2380–2386, 2016.

[138] S. K. Nayar and Y. Nakagawa, « Shape from focus », *IEEE Transactions on Pattern analysis and machine intelligence*, vol. 16, 8, pp. 824–831, 1994.
[139] L. McMillan and G. Bishop, « Plenoptic modeling: an image-based rendering system », in *Proceedings of the 22nd annual conference on Computer graphics and interactive techniques*, 1995, pp. 39–46.

[140] N. Asada, H. Fujiwara, and T. Matsuyama, « Edge and depth from focus », *International Journal of Computer Vision*, vol. 26, pp. 153–163, 1998.

[141] A. S. Malik, T. L. Song, and T.-S. Choi, « Depth map estimation based on linear regression using image focus », *International Journal of Imaging Systems and Technology*, vol. 21, 3, pp. 241–246, 2011.

[142] A. Mousnier, E. Vural, and C. Guillemot, « Partial light field tomographic reconstruction from a fixed-camera focal stack », *arXiv preprint arXiv:1503.01903*, 2015.

[143] A. Levin and F. Durand, « Linear view synthesis using a dimensionality gap light field prior », in *2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition*, IEEE, 2010, pp. 1831–1838.

[144] Z. Huang, J. A. Fessler, T. B. Norris, and I. Y. Chun, « Light-field reconstruction and depth estimation from focal stack images using convolutional neural networks », in *ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*, IEEE, 2020, pp. 8648–8652.

[145] X. Yin, G. Wang, W. Li, and Q. Liao, « Iteratively reconstructing 4d light fields from focal stacks », *Applied optics*, vol. 55, 30, pp. 8457–8463, 2016.

[146] C. Liu, J. Qiu, and M. Jiang, « Light field reconstruction from projection modeling of focal stack », *Optics express*, vol. 25, 10, pp. 11377–11388, 2017.

[147] K. Takahashi, Y. Kobayashi, and T. Fujii, « From focal stack to tensor light-field display », *IEEE Trans. on Image Processing*, vol. 27, 9, pp. 4571–4584, 2018.

[148] S. Gao and G. Qu, « Filter-based landweber iterative method for reconstructing the light field », *IEEE Access*, vol. 8, pp. 138340–138349, 2020.

[149] M.-B. Lien, C.-H. Liu, I. Y. Chun, et al., « Ranging and light field imaging with transparent photodetectors », *Nature Photonics*, vol. 14, 3, pp. 143–148, 2020.

[150] M. H. Kamal, B. Heshmat, R. Raskar, P. Vandergheynst, and G. Wetzstein, « Tensor low-rank and sparse light field photography », *Computer Vision and Image Understanding*, vol. 145, pp. 172–181, 2016.
[151] C. J. Blocker, Y. Chun, and J. A. Fessler, « Low-rank plus sparse tensor models for light-field reconstruction from focal stack data », in *2018 IEEE 13th Image, Video, and Multidimensional Signal Processing Workshop (IVMSP)*, IEEE, 2018, pp. 1–5.

[152] M. Le Pendu, C. Guillemot, and A. Smolic, « A fourier disparity layer representation for light fields », *IEEE Trans. on Image Processing*, vol. 28, 11, pp. 5740–5753, 2019.

[153] S. Gao, G. Qu, M. Sjöström, and Y. Liu, « A tv regularisation sparse light field reconstruction model based on guided-filtering », *Signal Processing: Image Communication*, p. 116 852, 2022.

[154] X. Jiang, M. Le Pendu, R. A. Farrugia, and C. Guillemot, « Light field compression with homography-based low-rank approximation », *IEEE Journal of Selected Topics in Signal Processing*, vol. 11, 7, pp. 1132–1145, 2017.

[155] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng, « Nerf: representing scenes as neural radiance fields for view synthesis », *Communications of the ACM*, vol. 65, 1, pp. 99–106, 2021.

[156] D. Scharstein and R. Szeliski, « A taxonomy and evaluation of dense two-frame stereo correspondence algorithms », *International journal of computer vision*, vol. 47, pp. 7–42, 2002.

[157] A. Saxena, M. Sun, and A. Y. Ng, « Make3d: learning 3d scene structure from a single still image », *IEEE transactions on pattern analysis and machine intelligence*, vol. 31, 5, pp. 824–840, 2008.

[158] D. Eigen, C. Puhrsch, and R. Fergus, « Depth map prediction from a single image using a multi-scale deep network », *Advances in neural information processing systems*, vol. 27, 2014.

[159] Y. Cao, Z. Wu, and C. Shen, « Estimating depth from monocular images as classification using deep fully convolutional residual networks », *IEEE Transactions on Circuits and Systems for Video Technology*, vol. 28, 11, pp. 3174–3182, 2017.

[160] Y. Kuznietsov, J. Stuckler, and B. Leibe, « Semi-supervised deep learning for monocular depth map prediction », in *Proceedings of the IEEE conference on computer vision and pattern recognition*, 2017, pp. 6647–6655.
[161] S. Evain and C. Guillemot, « A lightweight neural network for monocular view generation with occlusion handling », *IEEE Transactions on Pattern Analysis and Machine Intelligence*, vol. 43, 6, pp. 1832-1844, 2019.

[162] O. Veksler, « Stereo matching by compact windows via minimum ratio cycle », in *Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001*, IEEE, vol. 1, 2001, pp. 540–547.

[163] C. L. Zitnick and T. Kanade, « A cooperative algorithm for stereo matching and occlusion detection », *IEEE Transactions on pattern analysis and machine intelligence*, vol. 22, 7, pp. 675–684, 2000.

[164] T. Taniai, Y. Matsushita, and T. Naemura, « Graph cut based continuous stereo matching using locally shared labels », in *Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition*, 2014, pp. 1613–1620.

[165] J. Zbontar and Y. LeCun, « Computing the stereo matching cost with a convolutional neural network », in *Proceedings of the IEEE conference on computer vision and pattern recognition*, 2015, pp. 1592–1599.

[166] Z. Chen, X. Sun, L. Wang, Y. Yu, and C. Huang, « A deep visual correspondence embedding model for stereo matching costs », in *Proceedings of the IEEE International Conference on Computer Vision*, 2015, pp. 972–980.

[167] A. Kendall, H. Martirosyan, S. Dasgupta, et al., « End-to-end learning of geometry and context for deep stereo regression », in *Proceedings of the IEEE international conference on computer vision*, 2017, pp. 66–75.

[168] R. C. Bolles, H. H. Baker, and D. H. Marimont, « Epipolar-plane image analysis: an approach to determining structure from motion », *International journal of computer vision*, vol. 1, 1, pp. 7–55, 1987.

[169] S. Zhang, H. Sheng, C. Li, J. Zhang, and Z. Xiong, « Robust depth estimation for light field via spinning parallelogram operator », *Computer Vision and Image Understanding*, vol. 145, pp. 148–159, 2016.

[170] S. Heber, W. Yu, and T. Pock, « Neural epi-volume networks for shape from light field », in *Proceedings of the IEEE international conference on computer vision*, 2017, pp. 2252–2260.
[171] C. Shin, H.-G. Jeon, Y. Yoon, I. S. Kweon, and S. J. Kim, « Epinet: a fully-convolutional neural network using epipolar geometry for depth from light field images », in *Proceedings of the IEEE conference on computer vision and pattern recognition*, 2018, pp. 4748–4757.

[172] H.-G. Jeon, J. Park, G. Choe, et al., « Accurate depth map estimation from a lenslet light field camera », in *Proceedings of the IEEE conference on computer vision and pattern recognition*, 2015, pp. 1547–1555.

[173] J. Navarro and A. Buades, « Robust and dense depth estimation for light field images », *IEEE Transactions on Image Processing*, vol. 26, 4, pp. 1873–1886, 2017.

[174] X. Jiang, M. Le Pendu, and C. Guillemot, « Depth estimation with occlusion handling from a sparse set of light field views », in *2018 25th IEEE international conference on image processing (ICIP)*, IEEE, 2018, pp. 634–638.

[175] M. W. Tao, S. Hadap, J. Malik, and R. Ramamoorthi, « Depth from combining defocus and correspondence using light-field cameras », in *Proceedings of the IEEE International Conference on Computer Vision*, 2013, pp. 673–680.

[176] H. Lin, C. Chen, S. B. Kang, and J. Yu, « Depth recovery from light field using focal stack symmetry », in *Proceedings of the IEEE International Conference on Computer Vision*, 2015, pp. 3451–3459.

[177] M. W. Tao, P. P. Srinivasan, J. Malik, S. Rusinkiewicz, and R. Ramamoorthi, « Depth from shading, defocus, and correspondence using light-field angular coherence », in *Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition*, 2015, pp. 1940–1948.

[178] T. Herfet, K. Chelli, and M. Le Pendu, « Light field representation: the dimensions in light fields », in *Immersive Video Technologies*, Elsevier, 2023, pp. 173–199.

[179] S. Zheng, Y. Liu, Z. Meng, et al., « Deep plug-and-play priors for spectral snapshot compressive imaging », *Photonics Research*, vol. 9, 2, B18–B29, 2021.

[180] R. Shah, G. Wetzstein, A. S. Raj, and M. Lowney, *Stanford lytro light field archive*, 2018.

[181] K. Honauer, O. Johannsen, D. Kondermann, and B. Goldluecke, « A dataset and evaluation methodology for depth estimation on 4d light fields », in *Asian conf. on computer vision*, Springer, 2016, pp. 19–34.
[182] E. Miandji, H.-N. Nguyen, S. Hajisharif, J. Unger, and C. Guillemot, « Compressive
hdr light field imaging using a single multi-iso sensor », *IEEE Transactions on
Computational Imaging*, vol. 7, pp. 1369-1384, 2021. DOI: 10.1109/TCI.2021.
3132191.
<figure><img src="image_41.png" /></figure>
<figure><img src="image_42.png" /></figure>
**Titre :** Apprentissage profond pour l'acquisition et la restauration de champs de lumière

**Mot clés :** Problème inverse, champs de lumière, apprentissage profond, algorithmes déroulés, restauration d'images

**Résumé :** L'acquisition d'une image est restreinte par les limitations du matériel d'acquisition et est soumise à des perturbations. La reconstruction d'images à partir de mesures dégradées est un problème inverse, souvent mal conditionné et demandant donc la présence d'une connaissance à priori sur les images à reconstruire. Les algorithmes déroulés ont prouvé leur efficacité en matière de résolution de problèmes inverses, mais leur coût en mémoire et en temps de calcul est très élevé.

Notre première contribution est une méthode d'entraînement pour les algorithmes déroulés, permettant de considérablement réduire les coûts et les contraintes liées à l'entraînement de ces méthodes.

Nous nous intéressons ensuite plus particulièrement aux problèmes inverses liés à l'acquisition et à la reconstruction de champs de lumière. Ceux-ci permettent d'obtenir l'information 3D cruciale pour une variété de tâches en imagerie, qui est perdue lors de l'acquisition d'une image avec une caméra traditionnelle. Un champ de lumière est généralement capturé via des appareils coûteux et non accessibles au grand public.

Notre deuxième contribution est une méthode basée sur les algorithmes d'optimisation déroulés, permettant de reconstruire un champ de lumière à partir d'un empilement de mises au point, contenant peu d'images capturées avec une caméra traditionnelle.

**Title:** Deep learning for light field acquisition and restoration

**Keywords:** Inverse problem, light fields, deep learning, unrolled algorithms, image reconstruction

**Abstract:** The acquisition of an image is restricted by the limitations of the acquisition device and subject to perturbations. Image reconstruction from degraded measurements is an inverse problem, usually ill-conditioned, hence requiring the presence of an image prior. Unrolled optimization algorithms have achieved state-of-the-art results for a variety of image reconstruction tasks, but suffer from a high computational burden.

Our first contribution is a training method for unrolled optimization algorithms which considerably reduces the computational burden and constraints of the training.

We then focus on inverse problems related to the acquisition and restoration of light fields. This representation of scenes contains the 3D information that is crucial for a variety of image processing tasks and that is generally lost in the acquisition of an image with a traditional camera. Light fields are generally captured with complex devices that are not accessible to the general public.

Our second contribution is a method, based on the unrolled algorithms, which reconstructs light fields from a focal stack containing very few images captured with a traditional camera.