Title: From Pixels to Views: Learning Angular-Aware and Physics-Consistent Representations for Light Field Microscopy
Venue: NeurIPS 2025
Accepted: yes
Keywords: Physics-Guided Learning; Microscopy; Biomedical Imaging; Neural Imaging
Abstract: Light field microscopy (LFM) has become an emerging tool in neuroscience for large-scale neural imaging in vivo, with XLFM (eXtended Light Field Microscopy) notable for its single-exposure volumetric imaging, broad field of view, and high temporal resolution.  However, learning-based 3D reconstruction in XLFM remains underdeveloped due to two core challenges: the absence of standardized datasets and the lack of methods that can efficiently model its angular–spatial structure while remaining physically grounded. We address these challenges by introducing three key contributions. First, we construct the XLFM-Zebrafish benchmark, a large-scale dataset and evaluation suite for XLFM reconstruction. Second, we propose Masked View Modeling for Light Fields (MVM-LF), a self-supervised task that learns angular priors by predicting occluded views, improving data efficiency. Third, we formulate the Optical Rendering Consistency Loss (ORC Loss), a differentiable rendering constraint that enforces alignment between predicted volumes and their PSF-based forward projections. On the XLFM-Zebrafish benchmark, our method improves PSNR by 7.7\% over state-of-the-art baselines. Code and datasets are publicly available at: https://github.com/hefengcs/XLFM-Former.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work deals with the 3D reconstruction of eXtended Light field microscopy(XLFM) data. The authors argue that the traditional way of volumetric annotations is computationally intensive (Richardson–Lucy deconvolution) and so they bring in a self-supervised component to their work (MVM-LF). To enforce predictions to adhere to the forward model of the image formation process, ORC loss is introduced. This work introduces a dataset, which will be made public, on which all experiments are conducted.  A sufficient number of ablation studies have been conducted (for ORC loss, MVM-LF, masking ratio, etc), which justifies the presence of individual components of the training setup.\n\nSTRENGTHS AND WEAKNESSES:\n## Stengths:\n\n1. Ablations have been done in a comprehensive manner. \n2. The manuscript reads well. \n3. XLFM is a relatively new field, and so, contributions of this work will have more impact to this field, and in this sense, this work is valuable. \n4. A new dataset will be made publicly available. \n\n## Weakness.\n\n1. Relevant baselines are missing. At line 117, paper refers to 3 works [25,30,9] which work in this field. However, none of them are present as a baseline in Table 1. The work does mention that those methods reconstruct only neural activity, but this work also reconstructs entire volumetric structures. However, the baselines mentioned in Table 1 were also not developed for XLFM 3D reconstruction task, but the authors did the work to adapt them to this task. So, I donot see why such an effort was not made for any of these three works, given the fact that they are much closer to the domain than the used baselines. Also, since richardson-lucy is used for 3D reconstruction in light field microscopy, it is expected that that should be present as a baseline, arguably along with a quantitative estimate of its compute requirements (the relevant ones from time, gpu, cpu or RAM).  \n2. The authors themselves mention that only one dataset, that has been generated by them, was used for the experiments. The authors should explain the reason: Are there no other datasets which could be adapted to this task? In such a case, even simulated data will hold value, in my opinion. \n3. the loss function for XLFM reconstruction tasks (equation 7) arguably has redundancies. It has a loss component using PSNR as well as using MSE. Since PSNR internally uses MSE, it is not clear why both are needed. Indeed the analysis done in the supplement shows that the PSNR loss component provides limited benefit. The other issue with having five loss components is that one has 5 hyper-parameters, and then it is not clear whether the same hyperparameter set can be used for other datasets. Since there are no other datasets on which the same hyperparameter set has been tested, this becomes a bit more problematic.\n\nQUESTIONS:\n1. At Line 230, the authors mention that the batch size is set to 1 and mention that it aligns with the requirements of the volumetric task. How does batch size of 1 align with the requirements of the volumetric task? \n2. In eq 5, it has not been explained how the network f() is able to take in a dynamic number of views as input. Also, what is the shape of a view tensor Ui? \n3. At line 66, the authors say that Richardson-Lucy deconvolution should not be used because of being computationally intensive. It will be helpful to know quantitatively how much the computational requirements are. For example, a statement like ‘using it on a XxYxZ sized data takes A amount of time with B GPU/CPU specs’ can be helpful.  \n4. There are a few typos in Lines 254-261 (5070%, 2030%). \n5. Not a question, but rather a minor suggestion for Fig. 6b: I think it would make this work stronger if you had another entry for 1% labelled data. In that case, one expects to see a relatively increased gap between the two plots.\n6. In Line 287, the phrase 'confirming strong generalization' was used. Since 'strong' cannot be justified quantifiably, it might make sense not to use such additional adjectives.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper tackles 3-D reconstruction for eXtended Light-Field Microscopy (XLFM) and introduces a complete learning framework that is both data-driven and physics-aware.\n- New benchmark: The authors curate XLFM-Zebrafish, a public dataset of 22,581 light-field frames with an accompanying evaluation protocol, filling the long-standing gap of standardized data for this modality.  \n- Self-supervised pre-training (MVM-LF): They propose Masked View Modeling for Light Fields, in which 70 % of the sub-aperture views are masked and the network learns to predict them from the remaining views, capturing angular correlations without labels.  \n- Physics-consistent fine-tuning: A Swin-Transformer encoder plus lightweight CNN decoder, dubbed XLFM-Former, is fine-tuned with an Optical Rendering Consistency (ORC) loss that renders the predicted volume through the system PSF and enforces agreement with the measured views, ensuring physical plausibility.  \n- Results: On the new benchmark the method attains 54.0 dB PSNR and 0.994 SSIM, improving average PSNR by 7.7 % over seven recent learning-based baselines.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- The pipeline is carefully decomposed into (i) self-supervised pre-training (MVM-LF) and (ii) physics-consistent fine-tuning (ORC loss). Extensive ablations isolate the gain from each component and show a clear ∼35 % MSE reduction over a strong scratch baseline. \n- The paper is well-structured; figures 1 – 3 cleanly illustrate the view-masking idea and the physics loop, and the benchmark release is clearly documented.\n- Provides the first large-scale public XLFM dataset with a community evaluation protocol—likely to standardise future work in this niche. The physics-guided loss adds a reproducible recipe for combining differentiable optics with transformers.\n\nWeaknesses\n- Scope of evaluation is narrow: all quantitative tests are on zebrafish larvae; no cross-species or out-of-distribution studies. Runtime/throughput and GPU memory footprints—critical for high-speed microscopy—are omitted, so practical efficiency is unclear.\n- Several implementation details are under-specified (decoder up-sampling stages, PSF calibration procedure, learning-rate schedule). Readers cannot reproduce the exact 54 dB result without digging into supplementary code.\n- The two core ideas (masked prediction, differentiable rendering) are themselves well-known in vision/graphics. Their combination is novel for XLFM, but may be judged more as thoughtful integration than fundamental algorithmic innovation.\n\nQUESTIONS:\n1.\tGeneralisability beyond zebrafish\nCould you add—or at least simulate—experiments on a different specimen (e.g., mouse brain slice or a thicker adult zebrafish) to show that the gains hold outside the current benchmark?\nScore up if cross-domain PSNR/SSIM or qualitative slices confirm similar improvements; score down if performance drops sharply without a clear explanation.\n2.\tRuntime, throughput, and memory footprint\nPlease report FPS and peak GPU memory for one full-resolution volume on a single A100 and compare with the strongest baseline.\nShowing that XLFM-Former meets real-time (>30 fps) or near-real-time speeds with ≤80 GB will raise confidence; opaque or unfavourable numbers will lower it.\n3.\tMasked-view vs. standard MAE baseline\nInclude a control experiment where an off-the-shelf pixel-patch MAE (ViT-MAE) is pre-trained on the same data and fine-tuned identically.\nIf MVM-LF still wins by a clear margin, originality concerns are alleviated; if the gap is marginal, the novelty claim weakens.\n4.\tRobustness to PSF mis-calibration\nDemonstrate reconstruction quality when the forward PSF used in ORC is intentionally perturbed (e.g., ±10 % axial FWHM, mild aberration).\nStable performance would validate the physics-loss design; sensitivity would suggest the need for adaptive PSF estimation.\n5.\tReproducibility details\nProvide (in supplementary material or repo) the exact decoder architecture diagram, LR schedule, and data-normalisation script that yield the 54 dB result.\n\nLIMITATIONS:\nMostly yes: the manuscript candidly notes (i) restriction to larval zebrafish and (ii) reliance on a calibrated PSF.\nTwo points that could still be added:\n1.\tComputational cost / carbon footprint of the 1 k-GPU-hour pre-training stage.\n2.\tData-privacy implications if the approach is transferred to vertebrate animal or clinical specimens (e.g., brain-activity maps).", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a new dataset and a Swin Transformer based 3D reconstruction method for eXtended Light Field Microscopy (XLFM). The collected dataset include 13 fixed zebrafish for reconstruction task, and 3 free moving zerbrafish for pre-training the transformer model, by recording videos, a total of 22,581 images are captured. As the XLFM image can be divided into sub-aperture images specific for different viewing angle, the Swin Transformer is first pre-trained to learn the inter-aperture information by masking-and-completing task, and then the extracted multi-resolution feature is coupled with a convolutional network to reconstruct the 3D volume of the biological sample being imaged.\n\nSTRENGTHS AND WEAKNESSES:\nQuality: The paper is technically reasonable in a general sense; however, the implementation seems suspicious, and the experimental results appear contradictory. The quantitative results in Table 1 are excessively high, showing a PSNR of 45–54 dB for all methods (while the maximum PSNR for 8-bit signals is 48.13), essentially making them “perfect” equally. Such high PSNR values do not match the poor visualization results in Figure 5.\n\nClarity: The paper does not present the technical details clearly. The design of the lightweight decoder is unclear, and the pipeline presented in Appendix A is poorly written. Additionally, the details of the data, such as image resolution, sub-aperture image resolution, and reconstruction volume size, are not specified.\n\nSignificance: The proposed method has the potential to speed up the 3D reconstruction of XLFM through a Swin Transformer-based approach, along with a unique dataset for this specific task. However, the speed of inference is not mentioned or compared, and the significance of reconstructing the full 3D biological signal over simply reconstructing functional (neural) signals is not well established.\n\nOriginality: The work proposes both a unique dataset and a method specifically tailored for the task of XLFM reconstruction. Although it is rooted in masked pretraining and the Swin Transformer architecture, it introduces a fair rendering consistency loss to inject domain-specific prior knowledge.\n\nQUESTIONS:\n* Implementation of XLFM-Transformer: It is unclear what does the “cropping operation” does. According to appendix A, it raises the image to a 3D representation of shape H x W x D x S ( the channel S =1). I wonder in what sequency does the sub-aperture images are stacked in the D dimension, how to ensure the 2D neighboring information of sub-images is reflected in this one dimensional stacking? Moreover, what is changed in the positional embedding for this added dimension? The equation 13 and 14 contradicts each other, and it seems duplicated illustration in the last part in Appendix A. \n* The idea of masked pretraining doesn’t seems to fit quite well with such 3D Swin Transformer architecture. For masked pretraining in [1,2], each sub-image is treated as a token and are assigned a unique positional embedding, which is clearly not the case for this implementation of “3D Swin Transformer”. Does the pretraining implemented in this paper simply set the masked sub-image to zero? A more detailed implementation should be included, and the rationality to do that should be discussed.\n* In the section “efficacy of  pretraining Strategies” ablation study, how does this “3D Swin Transformer” being pre-trained in ImageNet-1k/22k? \n\n[1] Masked Autoencoders Are Scalable Vision Learners\n\n[2] Boosting light ﬁeld spatial super-resolution via masked light ﬁeld modeling\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nFourier light field microscopy is a kind of 3D snapshot microscopy that allows reconstruction of a 3D sample from a single 2D image. The authors present a new method to reconstruct volumes from images captured by an XLFM (a kind of Fourier light field microscope). Reconstruction of Fourier light field images is a significant and useful problem to solve using deep learning, because volume reconstruction is typically an expensive and slow process that requires an iterative deconvolution process to solve. Deep learning-based reconstructions allow reconstructions to be obtained significantly faster than iterative optimization algorithms, and can also improve reconstruction quality by accounting for sample priors based on the training data. However, it is more expensive to collect the training data of zebrafish volumes than it is to collect the 2D images. The authors' new method presents a promising technique to scale up the training of reconstruction neural networks by pretraining a feature extractor in a self-supervised manner using just the 2D image data, while also providing a valuable training dataset of zebrafish volumes imaged through a Fourier light field microscope to the community. The pretrained feature extractor serves to significantly improve the reconstruction quality of a trained reconstruction neural network, providing higher reconstruction quality with just 20% of the volumetric ground truth training data (52.67 dB PSNR) than a neural network trained without the pretrained features using 60% of the data (52.66 dB PSNR).\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\nThe authors appear to have an original and useful technique for reconstruction of Fourier light field/XLFM images with a pretraining step that significantly improves the training efficiency of volumetric ground truth (which is experimentally expensive to collect and computationally expensive to train with). This method, along with the accompanying standard dataset of volumes, could be significant for the field.\n\n# Weaknesses\n\n## Major Issues\n\n1. *Explanation of ORC Loss and presentation issues*: While the authors have a promising method, the paper suffers from widespread issues in its presentation quality that limit the clarity of the paper. The most major issue with presentation is that one of the major contributions of the paper as claimed by the authors, the \"ORC Loss,\" does not obviously appear in any explanation of the losses used for the reconstruction phase of the training process. The authors later introduce a \"PSF Loss\" which is not explained, which is likely the \"ORC Loss,\" but this is not clear. Other issues with the presentation are listed in the minor issues section below.\n\n2. *Analysis of ORC Loss and hallucinations*: Further, to substantiate the claim that the ORC/PSF Loss can reduce artifacts in the reconstructions and improve cross-sample generalization, the authors should include some examples of reconstructions from networks trained with and without the ORC Loss. While it is apparent that penalizing the forward projections of the reconstruction and the ground truth can enforce some consistency between the reconstruction and the measured snapshot image, there can still be hallucinated cells/artifacts in the reconstruction. Indeed, some cells/background appear in the XLFM-Former in Figure 5 (hindbrain) which are not present in the ground truth. Also, some explanation of why this ORC Loss does not use the measured image instead of a forward projection would be useful (e.g. whether this helps avoid reconstruction artifacts due to some background/dark frame in the measurement).\n\n3. *Appropriate baselines*: The authors compare their reconstructions against reconstructions from a range of other neural network architectures. However, they do not compare their reconstructions using the methods described by prior work in  snapshot imaging (Fourier light field or mask-based ) with deep learning, for example: Yanny et al.: \"Deep learning for fast spatially varying deconvolution\", Optica, 2022, https://doi.org/10.1364/OPTICA.442438 and references [25, 30, 9] from their paper. If memory is a concern, an explanation of why these other methods are not appropriate for these data would be useful. It would also be useful to compare against a reconstruction using an iterative deconvolution algorithm such as Richardson-Lucy.\n\n4. *Ground truth in datasets*: Another major issue is that the collected dataset as described, though it is valuable for the computational optics community, does not describe the ground truth data collection process or resolution, though some kind of volumetric ground truth must be present and is shown in Figure 5. The authors should clarify the process of collecting the ground truth for the dataset.\n\n## Minor Issues\n\n* The second paragraph of the introduction does not motivate the need for learning-based reconstruction. While this may be obvious to experts of both machine learning and microscopy, it would be helpful to briefly explain why learning-based reconstructions for LFM are desirable at the beginning of the paper.\n* Figure 1, the caption describes splitting the raw image into 27 viewpoints \"based on physical coordinates.\" This language is unclear, but based on the figure it appears that the 27 projections of the fish from each beam of the XLFM PSF have been cut out, presumably centered on the x-y pixel coordinate on the camera of the focal point from each microlens. The caption explaining this process could be made more clear (or point the reader to a more detailed explanation), especially because this is a core contribution of the proposed method.\n* Line 38 - 39, the sentence seems to be missing a \"while\" or \"but\" to complete the explanation for the lack of volumetric training data.\n* Line 42-43, the authors state that \"most existing approaches overlook the wave-optical nature of the XLFM forward model.\" However, other deep learning 3D snapshot reconstruction methods do explicitly take into account the wave-optics nature of the forward model: reference [9] in the paper and also the work of Yanny et al.: \"Deep learning for fast spatially varying deconvolution\n\", Optica, 2022, https://doi.org/10.1364/OPTICA.442438. The authors' intent here may be to claim that other methods typically do not include the forward projection in the loss function, but this is not the only way to take into account the forward model in the formulation of the reconstruction method.\n* Line 48, the term \"XLFM-Former\" is introduced with no prior explanation.\n* Line 53, \"Fourier neural operators\" refers to learning mappings between function spaces, typically to solve PDEs. The authors likely meant global convolutions implemented as Fourier convolutions, which do require large amounts of memory.\n* Points 1 and 2 of the key insights motivating the XLFM-Former in the introduction seem to be making the same point.\n* Figure 2 would benefit from some text labels on the image triplets.\n* Line 75, there seems to be a missing comma or an extra space.\n* Line 62, \"MVM-LF\" is introduced for the first time in the main text without a definition of the acronym aside from the abstract.\n* Line 84, the authors construct a benchmark using rather low FPS acquisitions (1 or 10 fps), which seems to imply high SNR. Could the authors comment on the effect of low SNR on the reconstruction quality using XLFM-Former?\n* Figure 3, this figure would also benefit from more clear text labels or titles across the images. This figure is also missing an explanation of the layout in the caption.\n* Figure 4, many of the layers have abbreviated or short form names which are not clear without additional labels or a caption. The paper would greatly benefit from bringing some summary of the detailed explanation of the architecture in the supplement into the main text and the caption of Figure 4.\n* Section 3.1, the paper would benefit from a pointer to a detailed explanation of the real-time tracking and treatment of motion artifacts (either in the supplementary material or through a citation).\n* Line 172, what is meant by \"high-precision light field reconstruction?\"\n* Supplement Table 5, Is Gaussian noise an appropriate augmentation? Would including Poisson noise as an augmentation make more sense due to the fact that this paper is working with fluorescence? Similarly, some discussion on the choice of other augmentations (especially rotation, since rotating the system would presumably change the subimages on the camera) would be useful.\n\nQUESTIONS:\nMy questions/suggestions are all related to the \"Major Issues\" listed above. With added evaluations/explanations that address the numbered points raised in the \"Major Issues\" section, I would definitely increase my score. The significance and originality of this work seem apparent, but it is currently being limited by the clarity of the presentation/evaluation.\n\nFor convenience, I am pasting those Major Issues again:\n\n1. *Explanation of ORC Loss and presentation issues*: While the authors have a promising method, the paper suffers from widespread issues in its presentation quality that limit the clarity of the paper. The most major issue with presentation is that one of the major contributions of the paper as claimed by the authors, the \"ORC Loss,\" does not obviously appear in any explanation of the losses used for the reconstruction phase of the training process. The authors later introduce a \"PSF Loss\" which is not explained, which is likely the \"ORC Loss,\" but this is not clear. Other issues with the presentation are listed in the minor issues section below.\n\n2. *Analysis of ORC Loss and hallucinations*: Further, to substantiate the claim that the ORC/PSF Loss can reduce artifacts in the reconstructions and improve cross-sample generalization, the authors should include some examples of reconstructions from networks trained with and without the ORC Loss. While it is apparent that penalizing the forward projections of the reconstruction and the ground truth can enforce some consistency between the reconstruction and the measured snapshot image, there can still be hallucinated cells/artifacts in the reconstruction. Indeed, some cells/background appear in the XLFM-Former in Figure 5 (hindbrain) which are not present in the ground truth. Also, some explanation of why this ORC Loss does not use the measured image instead of a forward projection would be useful (e.g. whether this helps avoid reconstruction artifacts due to some background/dark frame in the measurement).\n\n3. *Appropriate baselines*: The authors compare their reconstructions against reconstructions from a range of other neural network architectures. However, they do not compare their reconstructions using the methods described by prior work in  snapshot imaging (Fourier light field or mask-based ) with deep learning, for example: Yanny et al.: \"Deep learning for fast spatially varying deconvolution\", Optica, 2022, https://doi.org/10.1364/OPTICA.442438 and references [25, 30, 9] from their paper. If memory is a concern, an explanation of why these other methods are not appropriate for these data would be useful. It would also be useful to compare against a reconstruction using an iterative deconvolution algorithm such as Richardson-Lucy.\n\n4. *Ground truth in datasets*: Another major issue is that the collected dataset as described, though it is valuable for the computational optics community, does not describe the ground truth data collection process or resolution, though some kind of volumetric ground truth must be present and is shown in Figure 5. The authors should clarify the process of collecting the ground truth for the dataset.\n\nLIMITATIONS:\nYes (aside from the critiques raised in this review).", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 1, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}