From Pixels to Views: Learning Angular-Aware and Physics-Consistent Representations for Light Field Microscopy
Authors: Feng He
Guodong Tan
Qiankun Li
Jun Yu
Quan Wen
Venue: NeurIPS 2025 poster ()
Keywords: Physics-Guided Learning; Microscopy; Biomedical Imaging; Neural Imaging
Primary area: machine_learning_for_sciences
TL;DR: We propose XLFM-Former, a physics-guided and self-supervised Transformer framework for efficient volumetric reconstruction in light field microscopy, achieving state-of-the-art performance with minimal supervision.

Abstract: Light field microscopy (LFM) has become an emerging tool in neuroscience for large-scale neural imaging in vivo, with XLFM (eXtended Light Field Microscopy) notable for its single-exposure volumetric imaging, broad field of view, and high temporal resolution.  However, learning-based 3D reconstruction in XLFM remains underdeveloped due to two core challenges: the absence of standardized datasets and the lack of methods that can efficiently model its angular–spatial structure while remaining physically grounded. We address these challenges by introducing three key contributions. First, we construct the XLFM-Zebrafish benchmark, a large-scale dataset and evaluation suite for XLFM reconstruction. Second, we propose Masked View Modeling for Light Fields (MVM-LF), a self-supervised task that learns angular priors by predicting occluded views, improving data efficiency. Third, we formulate the Optical Rendering Consistency Loss (ORC Loss), a differentiable rendering constraint that enforces alignment between predicted volumes and their PSF-based forward projections. On the XLFM-Zebrafish benchmark, our method improves PSNR by 7.7\% over state-of-the-art baselines. Code and datasets are publicly available at: https://github.com/hefengcs/XLFM-Former.

From Pixels to Views: Learning Angular-Aware and Physics-Consisten Representations for Light Field Microscopy
Feng He, Guodong Tan, Qiankun Li", Jun Yu', Quan Wen" University of Science and Technology of China
hefenges @ gmail.com, tagodong @ mail.utcc edu.en, qklee mail mail.ustc.edu.co harryjun ustc. enu qwen ustc.edu.cn *Corresponding Author
Abstract
Light field microscopy (L.FM) has become an emerging tool in neuroscience for large-scale neural imaging in vivo, with XLFM (eXtended Light Field Microscopy) notable for single-exposure volumetric imaging, broad field of view, and high temporal resolution. However, learning-based 3D reconstruction in XLFM remains underdeveloped due to two core challenges: the absence of standardized datasets and the lack of methods that can efficiently model its angular-spatial structure while remaining physically grounded. We address these challenges by introducing three key contributions. First, construct the XLFM-Zebrafish benchmark, large-scale dataset and evaluation suite for XLFM reconstruction. Second, we pro- pose Masked View Modeling for Light Fields (MVM-LF), self-supervised task that learns angular priors by predicting occluded views, improving data efficiency. Third, formulate the Optical Rendering Consistency Loss (ORC Loss), differ- entiable rendering constraint that enforces alignment between predicted volumes and their PSF-based forward projections. On the XLFM-Zebratish benchmark, our method improves PSNR by 7.7% over state-of-the-art baselines. Code and datasets are publicly available at: http::/giihbbboom///fee////////
Introduction
Light Field Microscopy (LFM) has emerged as crucial technique for rapid volumetric imaging of nervous systems [18, 36, 19]. Notably, eXtended Light Field Microscopy (XLFM) [7], due to its graceful balance between speed, scale and resolution, is considered one of the most suitable LFM techniques for large-scale neural activity recording in several model organisms, including fish and mouse [1]. XLFM offers several advantages: 1)XLFM enables single-exposure acquisition of complete light field information at 100 Hz, whereas conventional microscopy techniques (e.g., two-photon microscopy [10], light-sheet microscopy [16]) require sequential layer-by-layer scanning, making it challenging to capture sub-second large-scale neural dynamics simultaneously. 2) The XLFM system incorporates point spread function (PSF) that is approximately spatially invariant. Consequently, the reconstruction of volumes through 3D deconvolution free from artifacts. 3) The rapid speed of XLFM allows real-time observation of large-scale population neural activity in vivo. Integrating volumetric imaging with optogenetic manipulation [3, 35] will enable optical brain-machine interface, namely closed-loop optical interrogation of brain-widd activity in both immobilized [28] and freely behaving animals [7, 4].
Despite the promise of eXtended Light Field Microscopy (XLFM) for rapid volumetric neural imaging, progress in learning-based 3D reconstruction remains limited not only due the unique physics of XLFM, but also because of lack of standardized datasets and evaluation protocols. First,
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

XLFM data differs fundamentally from conventional image data: each frame encodes dense angular sampling of the 3D scene microlens array, creating highly entangled multi-view observations. Traditional convolutional models struggle to model these angular correlations and view-dependent cues effectively. In addition, raw XIFFM acquisitions are abundant, producing high-quality volumetric ground truth (e.g., via Richardson-LLucy deconvolution [24]) is computationally expensive. This makes supervised learning pipelines costly at scale. Moreover, there is currently no public benchmark dataset or reproducible evaluation protocol for XLFM reconstruction. As result, comparisons across methods are often anecdotal, and progress in the field fragmented. Finally, most existing approaches overlook the wave-optical nature of the XLFM forward model. Without incorporating physics-guine guided constraints such as point spread function (PSF) priors, reconstructions may be visually plausible but physically inconsistent.
ez loss
Lightweight Decoder
Swin Transformer
Figure 1: Our pretraining pipeline for XLFM. The raw light field acquired from the microscope is separated into 27 distinct viewpoints based on physical coordinates. With a 70% probability, randomly mask a subset of these viewpoints and task the model with reconstructing them. The training is supervised by fz loss comparing the predicted and ground-truth views.
To address the unique challenges of XLFM reconstruction, revisit the problem not merely as a supervised regression task, but as a structured prediction problem grounded in physical optics, spatial geometry, and data asymmetry. Our design of XLFM-Former is driven by four key insights: 1) XLFM reconstruction inherently requires long-range dependency modeling across a large volumetric field-of-view with densely entangled angular observations. While increasing the receptive field in convolutional models like U-Net can improve performance, also incurs steep memory overhead, scaling linearly with spatial resolution and depth. Alternative global modeling strategies such as Fourier neural operators introduce even higher complexity, often transforming real-valued tensors into complex-valued representations that exceed the memory capacity single 80GB GPU. In contrast, we adopt Swin Transformer encoder with hierarchical windowed attention, which efficiently captures both local and global dependencies with significantly reduced memory costs that making it a natural fit for large-scale 3D light field modeling. 2) Unlike conventional image data, XLFM views are not independent. They exhibit occlusion patterns, spatial redundancy, and angular continuity, much like dependencies observed in natural language or multiview stereo systems. Modeling these view-wise interactions is essential for resolving fine 3D structures from ambiguous projections. We argue that the view, not the pixel, should be treated as the atomic modeling unit. Our MVM-LF pretraining task (Figure 1) reconstructs masked views from their angular neighbors, allowing the model to internalize structural priors specific to the XLFM sampling pattern.

---

Figure 2: The multi-view images used for pretraining encoder model. For each triplet, show the masked image (left), our MVM-LF regenerated image (middle), and the ground-truth (right). The masked regions are generated by applying the binary mask complement the original image.
original
mask 75%
mask 80%
mask 90%
Figure 3: Regeneration of XLFM light field images via MVM-LE. The model still accurately predict the view under appropriate occlusion, indicating that it has learned the global view relationship. Excessive occlusion (90%) causes prediction crash, indicating that MVM-LF requires reasonable occlusion ratio to balance information loss and network learning ability.
3) The data economics of XLFM present a natural motivation for self-supervised learning. Capturing light field images fast, inexpensive, and non-destructive, but generating high-quality volumetric annotations (e.g., via Richardson-Lucy deconvolution) is computationally intensive. Our pretraining strategy allows the model to scale with data availability while reducing reliance on expensive labels. improving both generalization and transferability to new imaging settings. Our pretraining strategy masks random subset angular views and tasks the model with reconstructing them. This view-level self-supervision enables the model to scale with unlabeled XLFM data while improving generalization and transferability to unseen imaging conditions. Figure illustrates the model's is ability to reconstruct masked views across diverse zebrafish samples, while Figure shows its robustness and limitations under varying occlusion levels. 4) Even visually accurate reconstructions may diverge from the underlying physics of light propagation. Models trained purely pixel- level losses often hallucinate plausible but optically inconsistent structures undermining the scientific validity of the output. To enforce physical plausibility, we introduce the Physically Optical Rendering Consistency Loss (ORC Loss), which ensures that reconstructed volumes, when passed through the known point spread function (PSF) of the imaging system, yield synthetic light field that matches the observed measurements. This alignment with the optical forward model constrains the network to produce predictions that are both data-aligned and physics-consistntt.
Our main contributions are summarized as follows:
I A standardized benchmark for XLFM reconstruction. We construct the first large-scale and standardized XLFM dataset, comprising 22,581 light field images captured under varying acquisition rates (10 fps fps) across three free-swimming zebrafish, seven immobilized zebrafish, and six unseen test samples. This benchmark enables reproducible evaluation and systematic advancement of XLFM reconstruction methods.
transformer-based framework tailored for light field microscopy. We develop XLFMM Former, hierarchical Swin Transformer backbone adapted to the structural characteristics of XLFM, capable of modeling both spatial and angular dependencies efficiently across large volumetric fields.

---

A view-masked pretraining strategy aligned with angular geometry. We introduce Masked View Modeling for Light Fields (MVM-LF) that a sulf-supe task that masks and reconstructs angular viewpoints rather than pixels. To our knowledge, this is the first pretraining strategy explicitly designed to capture inter-view structure XI.FM data, reducing dependence on costly volumetric labels.
A physically grounded loss via differentiable rendering. We formulate the Optical Rendering Consistency Loss (ORC Loss), which enforces that reconstructed 3D volumes, when forward- projected through the microscope's 's point spread function (PSF), match the observed light field. This loss imposes wave-optical constraints on learning, improving physical plausibility and caoss- generalization.
2 Related Work
2.1 Unsupervised Pretraining Methods for Light Field Microscopy
In the computer vision community, unsupervised pretraining methods have gained widespread attention [12, 13, 21, 25, 2]. For example, Masked Autoencoders [14] learn by randomly masking parts of the input help the model understand spatial relationships and global context. Contrastive learning [5, 6, 30, 33, 8] creates positive and negative pairs of samples to bring similar samples closer and push dissimilar ones apart. However, in the context of LFM, unsupervised methods are still underexplored. A recent approach [34], Masked LF Modeling (MLFM), introduces self- supervised pre-training scheme to enhance Light Field Super-Resolution (LFSSR). This method uses a transformer- based structure, KLFM-Former to learn inter-view correlations. While this approach significantly improves performance, its reliance on random masking does not fully capture the interdependencies between views, which are critical for high-quality super-resolution. Although both approaches are applied to light fields, our proposed method fundamentally different. First, we focus on the specific task of XLFM reconstruction. Second, our approach is based on view reconstruction, whereas theirs relies on random pixel masking.
2.2 3D Reconstruction in XLFM
Recent work [26, 31, 9] has demonstrated the potential of deep learning in addressing computational bottlenecks in XLFM. A recent approach [31] combines two neural networks, SLNet and XLFMNet, for real-time sparse 3D volumetric reconstruction in light field microscopy. SLNet extracts the spatio-temporally sparse components from image sequences, while XLFMNet performs -ideliiylitty 3D reconstruction. Another recent approach [26] proposes using a conditional normalizing flow architecture for fast 3D reconstruction of neural activity in immobilized zebrafish. However, like XLF,NNet, this method remains constrained by its sparsity- approach, reconstructing only neural signals while disregarding complete biological morphology. Unlike these prior meth- ods, XLFM-Former is designed for full-volume imaging, reconstructing not only neural activity but also entire volumetric structures. By leveraging Swin Transformer backbone for hierarchical feature extraction and MVM-LF pretraining to learn global context dependencies. XLFM-Forree M-FFormer provides a comprehensive reconstruction of biological samples. This distinction is critical in applications where both functional (neural signals) and anatomical (morphological structures) information are necessary for deeper biological insights.
recent end-to-end approach [9] combines differentiable simulations of optical systems with deep learning-based reconstruction networks for nigh-performante computational imaging. The key insight is that global information crucial for such problems, which is achieved by using Fouricr-Nets, a shallow neural network architecture based on global kernel Fourier convolution. However, mapping to the Fourier domain results in a substantial increase in memory usage, requiring multiple GPUs for lorge-volume reconstruction. This limits the method's 's scalability and applicability to more general imaging tasks. This method is particularly expensive in terms of video memory and not suitable for XLFM reconstruction because the final output of XLFM exceeds 100 million pixels and cannot be made into patches due to system design issues.
To achieve such global information extraction: 1) we use the Swin Transformer [22] as feature extraction module and then apply a CNN-based decoder for feature fusion. Using self-attention more efficient than convolution mapped to the Fourier domain. 2)To force the network to understand

---

the dependencies between different views, we propose proxy task. By masking 70% of the views, we force the network reconstruct the masked views, enabling unsupervised pretraining.
3
XLFM-Zebrafish Dataset
3.1 Data Collection
To construct the XLFM- Dataset, we utilized an advanced XLFM system designed to capture nigh-resolution volumetric neural activity zebrafish. The data collection process was carefully structured to ensure diversity in motion states, imaging conditions, and biological variability. For free-swimming zebrafish, we recorded neural activity an unconstrained environment, allowing for the study of brain-wide dynamics during naturalistic behaviors. A real-time tracking system was employed to continuously adjust the imaging field of view, ensuring that the zebrafish remained within the microscope's s focal range. Additionally, motion artifacts caused by rapid movement were mitigated through dual-color fluorescence imaging and adaptive filtering techniques. In contrast, fixed zebrafish were embedded in a stabilizing medium to facilitate high-precision 3D structural reconstruction. This setting eliminated motion-induced distortions, enabling the extraction detailed neural architecture. The immobilized specimens were further divided into different groups for training, validation, and testing purposes, ensuring a structured dataset for benchmarking reconstruction algorithms. To capture both rapid neural dynamics and long-term activity trends, employed multiple imaging conditions that varied in temporal resolution and sampling strategies. This approach allowed us to balance high-fideltyy reconstruction with the need for extended observation periods.
3.2 Dataset Statistics
To construct high-quality XLFM-ZZerrafih Dataset, we collected zebrafish in different motion states and set various sampling conditions to ensure diversity and applicability. This dataset is the first standardized XLFM zebrafish 3D reconstruction dataset, designed to evaluate the performance of deep learning models XLFM-3D reconstruction tasks. The SLFM-Zebrafish Dataset consists of two categories of zebrafish data: Free-swimming Zebrafish, includes 3 individual zebrafish, used for studying dynamic neural activity analyzing the impact motion blur on 3D reconstruction. Fixed Zebrafish, includes 13 individual zebrafish, suitable for high-precision 3D structural reconstruction, with 7 individuals for training and validation, and 6 for testing. Additionally, we introduced two different sampling rates; 10fps (High sampling rate): Suitable for temporal neural activity modeling and high-precision light field reconstruction. 1fps (Low sampling rate): Used for long-term dynamic tracking and reconstruction stability analysis under low frame rates. The dataset comprises 22,581 light field images. The detailed statistics are presented the Supplementary Section
Stage
Stage
Stage
Linear Eibbodding
Patcc Pourrnn comooo
Paich Same
Paud Fach Lenging
Pal Faucl Massiny Tne
Swin
Swin Black
Swin Block
Swin
Transfomer Block
Trans formerrrer Block
Convolution
Desomvolutinn
Up
Decone woloinnn
S
Figure Overview of the Swin-XLFM architecture.
5

---

4 Methodology
The overall architecture of XLFM-Former is illustrated in Figure 4. It consists of Swin Transformer- based encoder and CNN decoder for progressive 3D volume reconstruction. While the encoder and decoder structures follow standard hierarchical modeling and upsampling designs, we refer the reader to Appendix A for implementation and architectural details. Here, we focus on the three core innovations: view-masked pretraining (MVM-LF), optical rendering consistency loss (ORC Loss), and the design rationale tailored to XLFM geometry.
4.1 Optical Rendering Consistency Loss (ORC Loss)
To ensure that reconstructed volumes not only match the ground truth structurally but also remain consistent with the underlying physical imaging process, we introduce a differentiable rendering- based supervision term: the Optical Rendering Consistency Loss (ORC Loss). This loss enforces that the predicted 3D volume, when passed through the XLFM system's 's forward model characterized by its Point Spread Function (PSF) produces a synthetic light field image that aligns with the observation derived from ground truth. Although one might consider comparing the reconstructed volume directly to the measured light-field image, we empirically found this approach to be highly unstable. The raw measurement contains substantial sensor noise, dark current, and scattering artifacts, which introduce non-phy non-physical gradients during training. In contrast, using the PSF-based forward projection of the ground-truth volume provides a clean and structured supervision signal, ensuring that the network learns the optical consistency without being distracted by measurement noise.
Let Vreed and VGT denote the predicted and ground truth 3D volumes, respectively. Let h be the known system-specife PSF, modeled 3D convolution kernel. The forward-rendered images are obtained by convolving each volume with h:
pred h Vpred:
LTT = h Var-
The ORC Loss defined as the mean squared error between these two forward projections:
Corc
Ired
Iorll 1122 = ||h Vpred - h Varll
By minimizing LoRC, the model is regularized to produce volumetric outputs that not only reconstruct anatomical structure but also render physically plausible observations under the XLFM imaging model effectively bridging data-driven learning with wave-optical consistency.
4.2 Masked View Modeling for Light Fields (MVM-LF)
To enhance superviper learning and inter-view modeling in XLFM, we propose Masked View Modeling for Light Fields (MVM-LFF) as a pretraining strategy, enabling the model to reconstruct missing views and capture global scene structures.
The encoder architecture and input representation in MVM-LF are identical to those used in XLFM reconstruction. This consistency ensures that the learned features during pretraining are directly transferable to the supervised reconstruction task. The pretrained encoder, denoted as fo. serves the initialization for the XLFM reconstruction model.
Pretrained Encoder:
Inspired by self-supervised masked reconstruction frameworks, we adopt lightweight decoder consisting of series of convolutional layers. The decoder is responsible for predicting the missing views during pretraining. Once training is completed, the decoder is discarded, and only the pretrained encoder is retained for fine-tuning in the XLFM reconstruction task.
Lightweight Decoder:
Masking Strategy: The core principle of MVM-LF is to randomly mask proportion rm of the input views and force the network to reconstruct them based solely on the unmasked views. This proxy task compels the model to learn a joint representation of the global structure and inter-view dependencies. In practice, the masked sub-aperture views are zero-filled while their positions are preserved in the input tensor. Formally, given a set of sub-aperture views:
U UU, U2.... UN...
we define the masked subset as:
Umask -U: iie M).
(4)

---

where M is randomly sampled index set satisfying | = Nu with I' 0.7 (i.e., 70% of the views are masked). The network is trained to reconstruct the missing views as:
Ummss
fo Umack)
(5)
The loss function for MVM-LF defined as the mean squared error (MSE) between the predicted and ground truth masked views:
MMMLLF =
Σ -U.13.
No architecture-specifi changes were made during pretraining except adapting the output head, making the MVM-LF strategy applicable to any backbone capable of handling multi-view inputs.
4.3 Loss Function
For the pre-training task, we only use & loss. For the XLFM reconstruction task, we complete the reconstruction by minimizing the following loss combination.
1 LMS. LMM_SSSM
1
Guotl
SSIM + - LEdge + PPNNR
A1
(7)
1
1
LMSE + LoRC.
X4
As
The detailed loss function presented in the Supplementary Section C.
Experiments
5.1 Implementation Details
For the MVM-LF task, we employ a batch size of 8 to facilitate stable training dynamics. To enhance convergence and mitigate the risk of the model becoming trapped in local optima, utilize the learning rate scheduler, with an initial learning rate set to le-4. The training process is conducted for 250 epochs ensure robust feature learning. All experiments on distributed computing setup with four NVIDIA A A100-80GB SMX4 GPUs. For the XLFM reconstruction task, the training configuration remains largely consistent, with the primary exception that the batch size is set to 1, aligning with the requirements of volumetric reconstruction. The detailed experimental setup presented in the Supplementary Section D.
performed
49.48-0.9851
45.54
46.46%0 9648
47.20
48.81:0.9807
03.97.09930
PSNRSSIM
ConvNeXt
EffrientNet
PVT
Remnst50
Remnet 01)
U-Net
Ours
Lubel
Figure 5: Comparison with state-of-the--rt architectures the XLFM-Zebrafish Dataset. For visualization of Zebrafish sample #1, the PSNR/SSIM values are shown the top-left corner each image. Additional examples samples #2 #6 provided in the supplementary (Figure 7).
5.2 Main Results
We selected the state-of-the-art architectures for comparative experiments, including ConvNeXt [23], ViT [11], PVT [32], EfficientNet [29], ResNet-50 [15], ResNet-101 [15], and U-Net [27].
As shown in Table 1, our method significantly outperforms existing state-of-the-ar architectures

---

across all evaluation metrics. Specifically, our approach achieves the highest Peak Signal-to-Noiss Ratio (PSNR) and Structural Similarity Index Measure (SSIM) on all test samples, demonstrating superior reconstruction fidelity and perceptual quality. Compared to ConvNeXt, which achieves an average PSNR of 50.16 dB and SSIM of 0.9876, our model achieves 54.04 dB and 0.9944, respectively, highlighting substantial improvement. Similarly, our approach surpasses transformer- based models such as ViT and PVT, as well as widely used CNN architectures including EfficientNet and ResNet. Notably, U-Net, which performs competitively on some samples, is still outperformed by our model in all cases, demonstrating the effectiveness of our proposed framework. To further validate the qualitative performance of our model, Figure 5 presents visual comparisons between different methods. It is evident that our method produces reconstructions that are sharper and better aligned with the ground truth, particularly in fine structural details. In contrast, competing methods suffer from various artifacts, such as excessive blurring, structural distortions, and loss of fine details. These results indicate that our proposed approach not only achieves the best numerical performance but also generates reconstructions that are more perceptually faithful to the original structures. The integration of physics-gu guideed constraints and transformer-basee hierarchical feature modeling plays crucial role in achieving these improvements.
Table Comparison Methods on XLFM-Zebrafish Dataset. The best results are highlighted in bold, while the second-best are underlined.
#2
3
2-4
E.5
66
Avg.
Mothod
SSIM
PSNK*
SSIM
PSNK
SSIM
PSNKT
SSIM
PSNKT
SSIM
PSNK
SSIM1
PSNK
SSIM
Com-NeXi [23]
49.48
0.9851
53.88
0.9867
44.87
0.9633
$1.38
0.9882
51.52
0.9892
49.79
0.9935
50.16
0.9676
[i1]
55.29
0.9531
0.9888
51.35
0.9905
0.9893
0.9676
[32
47.21
0.9804
47.93
0,9760
44.50
0.9607
49.46
0.9851
48.32
0.984]
46.60
0.9910
47.34
0.9629
EdficienNNt (29)
54.65
0,9851
42.13
0,9541
49.56
0.9801
27.16
0.7261
44.53
0.926
RosNer-50 [15]
46.46
0.9688
54,89
[19851
41.46
0,3388
49.47
0.9790
48.82
0.9786
39.98
0,3304
46.85
0:9634
ResNer-101 [15]
0,9728
$4.90
0,9851
41.33
0.9366
40:47
0.9787
49.09
0.980)
39.50
0,8893
46.91
0.9554
U-Net (27)
48.81
0.9807
57.23
0.9928
44.41
0:9608
52.61
0.9908
52,06
0.9904
41,47
0,9725
49.43
0.9647
Ours
53,97
0.9930
59.83
0.9963
49.31
0.9910
54.55
0.9951
54.65
0.9955
$1.95
0.9956
54,04
0,9944
toning
53.5
$3.00
530
53.0
11.
113
52.0
110
Screes
500
Masking ratio
Parcertage
(a) Effect of masking ratio during MVM-LF pretrain- ing.
(b) Label-effcienccy (w/ w/o pretraining).
Figure 6: Effectiveness of MVM-LF pretraining, (Left) PSNR under different masking ratios.
(Right) PSNR under varying percentages of labeled data, comparing pretraining vs. scratch.
5.3 Ablation Study
Masking ratio: As shown in Figure 6a, we analyze the impact of different masking ratios in MVM- LF pretraining on XLFM reconstruction. The results indicate that moderate masking (50-70%) achieves optimal performance, with PSNR peaking at 53.38 dB at 70% masking. Lower masking ratios (20-30%) provide insufficient representation learning, leading to suboptimal fine-tuning results, while excessive masking (90%) reduces PSNR to 52.30 dB, indicating difficulty in reconstructing missing views with limited context. These findings highlight the importance of balancing information removal and reconstruction difficulty, and adopt 70% masking as the default setting for maximal pretraining efficiency.
Efficacy of Pre-training: We assess data efficiency by comparing models trained from scratch and those with pretraining under varying labeled data proportions (Figure 6b). Pretraining provides a significant boost, especially in low-data regimes, with a PSNR of 51.92 dB at 10% labeled data, surpassing the 50.73 dB of training from scratch. While the performance gap narrows as more labeled data is available, the pretrained model consistently outperforms the ccratch-trained counterpart, even
8

---

at 80% labeled data. These results confirm that MVM-LF pretraining enhances feature generalization, making the model more data fata-efficient and robust across different data availability scenarios.
Efficacy of different components: We conduct an ablation study to assess the impact of ORC loss and MVM-LF pretraining (Table 2). 'The baseline model achieves 52. dB PSNR, while adding ORC loss improves reconstruction fidelity to 52.96 dB. MVM-LF pretraining further enhances global view dependency learning, reaching 53,38 dB. Integrating both components into the full model yields the highest performance (54.04 dB PSNR, 0.9944 SSIM), confirming that combining physics-based constraints with superviee pretraining results the most effective XLFM reconstruction.
Efficacy of Pretraining Strategies: We compare MVM-LF pretraining with alternative methods, including ImageNet-based initialization and pixel-level masked pretraining (Table 2). Training from scratch (Baseline) achieves 52. 14 dB PSNR, while ImageNet- k/22k pretraining provides only marginal improvements (52.70 dB / 52.38 dB), indicating that conventional pretraining suboptimal for XLFM data. Random-masked pretraining performs slightly better (52.97 dB PSNR), but MVM- LF pretraining achieves the best results (54.04 dB PSNR, 0.9944 SSIM), demonstrating its superior ability to model multi-view dependencies. These findings highlight the importance of task-specific pretraining in optimizing XLFM reconstruction quality.
We evaluate the pretrained model's ability to infer scratch-trained trained model achieves 52.14 dB PSNR with full views, whereas the pretrained model surpasses it even with only 60% of views, reaching 52.54 dB PSNR. The best performance (53.26 dB PSNR) occurs 80% input views, confirming strong generalization. These results demonstrate that MVM-LF pretraining enables robust multi-view reconstruction, allowing high-fidelity reconstruction even with incomplete input, making it highly adaptable to real-world imaging constraints.
Generalization Under Reduced Input Views:
missing views by progressively reducing available inputs (Table 2). The scratch-
Table 2: Unified evaluation of model components, pretraining, and view-missing robustness. PSNR/SSIM results across all configurations. Our method outperforms across all settings. The best results are highlighted in bold.
Group
Setting
Notes
PSNR1
SSIM
baseline
no PSF, no MVM
52.14
0.9924
ORC loss only
w/ physics loss w/ view pretraining
52.96
0.9931
Ablation
MVM-LF only
53.38
0.9938
Full (Ours)
PSF + MVM-LF
54.04
0.9944
ImageNet Ik
vision-domain weights
52.70
0.9931
ImageNet 22k
large-scale weights
52.38
0,9923
Pretraining
Random mask
pixel-masked MAE
52.97
0.9934
MAE MVM-LF (Ours)
VIT backbone
46.55
0,9752
view-aware masking
54.04
0.9944
100% (scratch)
full input, pretrain
52.14
0.9924
90%
w/ MVM pretrain
52.97
0,9933
Missing Views
80%
53.26
0.9936
70%
52.67
0.9928
60%
52.54
0.9928
5.4 Cross-Domain Evaluation on the H2B-Nemos Dataset
We further evaluate our model on a newly collected zebrafish dataset, H2B-NeMOs, which utilizes NeMOs [20], new genetically encoded calcium indicator, to assess cross-domain generalization across different biological conditions. This dataset involves a distinct zebrafish line and optical setup, allowing us to directly test reconstruction robustness beyond training domain.
As shown in Table 3, our XLFM-Former consistently outperforms a range of representative architec- tures under identical training settings with ResNet-101 as the common baseline. Notably, our model achieves a +0.92 dB PSNR gain in the supervised setting and a +2.29 dB gain the seot setting, confirming its ability generalize across imaging domains.

---

Table Comparison on the H2B-Nemos dataset (baseline: ResNet-101).
Method
PSNR f
SSIM
APSNR R101 f
ASSIM R101 1
EfficientNet
49.01
0,9826
-2.42
-0.0118
ViT-tiny
50.87
0.9904
-0.55
-0,0041
PVT-tiny
50.89
0.9903
-0.53
-0.0042
ConyNeXt-tiny
50.88
0.9903
-0.54
-0.0042
U-Net
51.00
0.9910
-0.42
-0.0035
ResNet-50
51.34
0.9931
-0.09
-0.0014
ResNet- 101 (baseline)
51.42
0.9945
XLFM-Former (Ours, Full)
52.34
0.9955
+0.92
+0.0010
XLFM-Former (Zero-Shot)
53.72
0.9930
+2.29
-0.0015
5.5 Robustness to PSF Mis-Calibration
We further assess the robustness our Optical Reconstruction Consistency (ORC) loss to inaccuracies in the forward model by intentionally perturbing the point spread function (PSF) used in the forward projection. Specifically, we vary the axial full-width at half-maximum (FWHM) of the PSF by 100% to simulate mild miscalibration or optical aberrations. As shown in Table 4, the reconstruction performance remains highly stable under these perturbations, with PSNR fluctuations within =0.12 12.1226 dB and SSIM deviations within =0.0002.
This stability arises because the ORC loss enforces consistency between the reconstructed and measured light-field projections a global level, making it less sensitive to small inaccuracies in PSF calibration. Such robustness particularly desirable for real-world microscopy setups, where slight deviations in PSF shape system alignment are inevitable.
Table 4: Robustness of ORC loss to PSF perturbations the H2B-Nemos dataset.
PSF Setting
PSNR
APSNR
SSIM 1
ASSIM 1
Baseline PSF +10% FWHM
52.3440
0.9955
52.3369
-0.0071
0.9953
-0.0002
PSF 10%% FWHM
52.4647
+0.1207
0.9957
+0.0002
These results demonstrate that XLFM-Former does not rely on perfect PSF calibration, and can tolerate mild optical misalignments, reducing the need for exact calibration in practical deployments.
Conclusion
We introduce XLFM-Former, unified learning framework that combines physics-based constraints and self-supervised angular modeling to enable high-speed, high-fidelity volumetric imaging in eXtended Light Field Microscopy (XLFM). By integrating a hierarchical transformer backbone, differentiable rendering loss via PSF, and view-masked pretraining strategy (MVM-LF), our method captures the geometric and optical structures inherent in light field data with minimal supervision. Empirical results demonstrate that XLFM-Former significantly improves over existing approaches in PSNR and SSIM, especially under limited labels or incomplete input views. Our ablations further reveal the complementary roles of physical priors and angular-aware self-supervision in robust 3D reconstruction. Beyond the scope of XL.FM, this work highlights scalable path toward physics-aligned, data-efficient learning for scientific imaging. We believe our findings bridge neural imaging and modern vision learning, offering a transferable foundation for future exploration of self-supervised, physically informed learning paradigms in neuroscience, biomedicine, and beyond.
Linitations: Despite promising results, our approach is evaluated only zebrafish datasets, limiting generalization to other organisms such as mouse or drosophila. We focus on demonstrating the physical and computational efficiency of XLFM- Mor-Forerr rather than directly assessing functional trace extraction. As neural signal extraction involves complex pipelines with registration, segmentation, and clustering, future work will explore direct trace extraction enhance biological analysis.
10

---

Acknowledgments
We would like thank Professor Pengcheng Zhou for his helpful discussions during the development of the PSF-guided loss function. This research was supported by the ST12030-Major Projects (Grant No. 2022ZD021 1900), "Brain Science and Brain-Inspired Intelligence", under the subproject "New Technologies for Whole-Brain-Scale Neuronal Mesoscopic Atlas" (Grant No. 2022ZD021 1994). The numerical calculations in this paper were performed on the supercomputing system at the Supercomputing Center of the University of Science and Technology of China.
References
[1] Lu Bai, Lin Cong. Ziqi Shi, Yuchen Zhao, Yujie Zhang, Bin Lu, Jing Zhang, Zhi-Qi Xiong, Ninglong Xu, Yu Mu, et al. Volumetric voltage imaging of neuronal populations in the mouse brain by confocal light-field microscopy. Nature Methods, 21(11):2160-2170 2024.
[2] Amir Bar, Xin Wang, Vadim Kantorov, Colorado Reed, Roei Herzig, Gal Chechik, Anna Rohrbach, Trevor Darrell, and Amir Globerson. Detreg: Unsupervised pretraining with region priors for object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14605-14615, 2022.
[3] Edward S Boyden, Feng Zhang, Ernst Bamberg, Georg Nagel, and Karl Deisseroth. Millisecond- timescale, genetically targeted optical control of neural activity. Nature neuroscience, 809):1263- 1268, 2005.
[4] Yuming Chai, Kexin Qi, Yubin Wu, Daguang Li, Guodong Tan, Yuqi Guo, Jun Chu, Yu Mu, Chen Shen, and Quan Wen. All-optical interrogation of brain-wide activity in freely swimming larval zebrafish. iScience, 27(1):108385, 2024.
[5] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pages 1597-1607. PMLR, 2020.
[6] Ching- Cao Chuang. Joshua Robinson, Yen-Chen Lin, Antonio Torralba, and Stefanic Jegelka. Debiased contrastive learning. Advances in neural information processing systems, 33:8765- 8775, 2020.
[7] Lin Cong, Zeguan Wang, Yuming Chai, Wei Hang, Chunfeng Shang, Wenbin Yang. Lu Bai, Jiulin Du, Kai Wang, and Quan Wen. Rapid whole brain imaging of neural activity in freely behaving larval zebrafish (danio rerio). elife, 6:e28158, 2017.
[8] Jiequan Cui, Zhisheng Zhong, Shu Liu, Bei Yu, and Jiaya Jia. Parametric contrastive learning. Proceedings of the IEEE/CVF international conference on computer vision, pages 715-724, 2021.
[9] Diptodip Deb, Zhenfei Jiao, Ruth Sims, Alex Chen, Michael Broxton, Misha B Ahrens, Kaspar Podgorski, and Srinivas C Turaga. Fouriernets enable the design of highly non-local optical encoders for computational imaging. Advances Neural Information Processing Systems, 15:25224-55236, 2022.
[10] Winfried Denk, James H Strickler, and Watt W Webb. Two-photon laser scanning fluorescence microscopy. Science, 248(4951):73-76, 1990.
[11] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x words: Transformers for image recognition at scale. In International Conference on Learning Representations.
[12] Dumitru Erhan, Aaron Courville, Yoshua Bengio, and Pascal Vincent. Why does unsupervised pre-training help deep learning? In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 201-208 JMLR Workshop and Conference Proceed- ings, 2010.

---

[13] Dengpan Fu, Dongdong Chen, Jianmin Bao, Hao Yang. Lu Yuan, Lei Zhang, Houqiang Li, and Dong Chen. Unsupervised pre- tre-trainin for person re-identification. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 14750-14759, 2021.
[14] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000-16009, 2022.
[15] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[16] Jan Huisken, Jim Swoger, Filippo Del Bene, Joachim Wittbrodt, and Emst HK Stelzer. Optical sectioning deep inside live embryos by selective plane illumination microscopy. Science, 305(5686):1007-1009, 2004.
[17] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in neural information processing systems, 33:18661-18677 2020.
[18] Marc Levoy, Ren Ng, Andrew Adams, Matthew Footer, and Mark Horowitz. Light field microscopy. In Acm siggraph 2006 papers, pages 924-934. 2006.
[19] Haoyu Li, Changliang Guo. Deborah Kim-Holzapfel. Weiyi Li, Yelena Altshuller, Bryce Schroeder, Wenhao Liu, Yizhi Meng, Jarrod B French, Ken-Ichi Takamaru, et al. Fast, volumet- ric live-cell imaging using high-resolution light-field microscopy. Biomedical optics express, 10(1):29-49) 2018.
[20] Jia Li, Ziwei Shang, Jia-Hui Chen, Wenjia Gu, Li Yao, Xin Yang, Xiaowen Sun, Liuqing Wang, Tianlu Wang, Siyao Liu, al. Engineering of nemo as calcium indicators with large dynamics and high sensitivity. Nature methods, 20(6):918 82118-9222 924, 2023.
[21] Hao Liu and Pieter Abbeel. Behavior from the void: Unsupervised active pre-training. Advances in Neural Information Processing Systems, 18444499-,, 2021.
[22] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012-10022, 2021.
[23] Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xic. A convnet for the 2020s. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1 1976-119866 2022.
[24] Leon B Lucy. An iterative technique for the rectification of observed distributions. Astronomical Journal, Vol. 79, 745 (1974), 79:745, 1974.
[25] Oscar Manas, Alexandre Lacoste, Xavier Giró-i Nieto, David Vazquez, and Pau Rodriguez. Sea- sonal contrast: Unsupervised pre-training from uncurated remote sensing data. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9414-9422 2021.
[26] Josué Page Vizcaíno. Panagiotis Symvoulidis, Zeguan Wang, Jonas Jelten, Paolo Favaro, Edward S Boyden, and Tobias Lasser. Fast light-field 3d microscopy with out-of -fitt--disttibution detection and adaptation through conditional normalizing flows. Biomedical optics express, 15(2):1219-2232, 2024.
[27] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted intervention-MICCA. 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedinss, part III 18, pages 234 241. Springer, 2015.
[28] C. F. Shang. Y. F. Wang, M. T. Zhao, Q X. Fan, S. Zhao, Y. Qian, S. J. Xu, Y. Mu, J. Hao, and J. L. Du. Real-time analysis of large-scale neuronal imaging enables closed-loop investigation of neural dynamics. Nat Neurosci, 1014)1001-1 2024.
12

---

[29] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning, pages 6105-6114, PMLR, 2019.
[30] Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? Advances in neural information processing systems, 33:6827-6839, 2020.
[31] Josue Page Vizcaino, Zeguan Wang, Panagiotis Symvoulidis, Paolo Favaro, Burcu Guner- Ataman, Edward S Boyden, and Tobias Lasser. Real-time light field 3d microscopy via sparsity- driven learned deconvolution. In 2021 IEEE International Conference Computational Photography (ICCP), pages 1-11. IEEE, 2021.
[32] Wenhai Wang, Enze Xie, Xiang Li, Beng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In Proceedings of the IEEE/CVF international conference computer vision, pages 568-578, 2021.
[33] Xiao Wang and Guo-Jun Qi. Contrastive learning with stronger augmentations. IEEE transactions on pattern analysis and machine intelligence, 45(5):5549-5555 5560, 2022.
[34] Da Yang, Hao Sheng. Sizhe Wang, Shuai Wang, Zhang Xiong, and Wei Ke. Boosting light field spatial super-resolution via masked light field modeling. IEEE Transactions on Computational Imaging, 2024.
[35] Feng Zhang, Li-Ping Wang, Edward S Boyden, and Karl Deisseroth. Channelrhodopsin-2 and optical control of excitable cells. Nature methods, 3(10):785-72, 2006.
[36] Zhenkun Zhang, Lu Bai, Lin Cong, Peng Yu. Tianlei Zhang, Wanzhuo Shi, Funing Li, Jiulin Du, and Kai Wang. Imaging volumetric dynamics at high speed in mouse and zebrafish brain with confocal light field microscopy. Nature biotechnology, 39(1):74- 2021.
A XLFM-Former
Figure 4 illustrates the overall architecture of XIFM--Forme.. We describe the details of encoder and decoder in this subsection.
The input to XLFM-Former is a raw light field captured by the XLFM system. The original light field data is represented as a collection of sub-aperture channels, each encoding spatial information the observed scene and collectively forming a multi-view representation. These sub-aperture channels correspond to distinct angular responses induced by the XLFM optical system, without assuming a regular grid of physical viewpoints as in standard MLA-based light field imaging. To avoid ambiguity, we clarify that all sub- sub-aperture channels concatenated along the channel axis, forming a tensor of size x W x C. cropping operation is applied to align the data with the optical coordinate system, yielding X' H xDx where H and W denote spatial dimensions, D corresponds to the axial depth resolution from cropping, and S = is the channel number in XLFM imaging. Importantly, the sub-aperture index is treated as channel information rather than spatial dimension.
Encoder:
To facilitate self-attention computation in the transformer- based encoder, the patch partitioning layer divides the input into a sequence of non-overlapping 2D patches of size (H', W') for each depth H slice D. The resulting tokenized feature map has dimensions H' x W x D, where each patch is projected into C-dimensional embedding space using a lincar embedding layer. We explicitly note that the encoder adopts 2D positional embedding along the (H, W' axes only; no positional encoding is assigned to the sub-aperture dimension since it is folded into channels. This patch- wise tokenization enables efficient modeling of spatial and depth information without ambiguity about angular indexing.
W
To effectively capture multi-scale features, XLFM-Former employs a hierarchical encoding strategy based on the Swin Transformer. The encoder consists of four stages, each with two consecutive Swin Transformer blocks. At the end of each stage, a patch merging layer downsamples the resolution while increasing the feature dimension. Given input feature map S xxxexxxe at stage T, the
13

---

patch merging operation groups adjacent 2 x 2 patches in the (h, w) plane (applied per depth slice) and concatenates their features, yielding S' x A linear projection then reduces the feature dimension to 2c:
ST+1 WX'+ b,
where W and b are learnable parameters.
The four hierarchical feature maps {S1, Ss. S are passed to the decoder via skip connections. Each decoder stage upsamples the features and fuses them with the corresponding encoder outputs, ensuring that both high-level semantics and fine-grained spatial details are preserved.
Decoder: The decoder reconstructs the final high-resolution 3D structure by progressively integrating multi-scale features extracted from the encoder. This process consists of two key components: progressive upsampling and cross-level feature fusion. The decoder takes the hicrarchical feature maps from the encoder, denoted as S,, i 2, 3). and reconstructs the high-resolution output through a sequence of deconvolutional operations. The highest-level feature map S1 serves as the starting point, and at each stage, upsampling operation is applied gradually recover spatial details. Fused feature maps Fj,
i E (1,2,3} are obtained by adding features of different scales:
Fo = S1, A = Up((i))) + S₁, i (1,
(8)
The upsampling is performed using deconvolution layers:
S, = Deconv (1,2,3, 84 S4-
(9)
S1 St, S, = Deconv i (1,2,3)
(10)
To enhance reconstruction quality, cross-level feature fusion is applied each stage, allowing fine-grained spatial information from lower-level features to be combined with high-level semantic information. The fusion is performed by summing the upsampled feature maps with the corresponding encoder feature maps, followed by convolutional refinement:
F; Conv(S; + St), E (1,
(11)
The final reconstructed 3D structure is obtained by applying reconstruction head, which maps the fused feature F3 to the target 3D volume: First, convolution is performed to match the number of channels:
F
ConV 1x (Fiusion),
(12)
Then two deconvolutions are performed to match the target resolution:
Prred = Convix1 (Deconv2 (Deconv (F))
(13)
Vpred = Deconv
Deconv
(Conv1x1
(14)
This approach ensures that the final output preserves both deep semantic features and high-frequency spatial details, achieving high-quality 3D reconstruction.
XLFM-Former leverages Swin Transformer to extract multi-scale features and semantic information. The extracted multi-scale features undergo global and local feature fusion within the encoder. Figure
4 illustrates the overall framework of XI.FM-Former.
Feature Extractor: We adopt Swin Transformer [22] as the backbone network to extract hierarchical features Fo. Fi. F2, F3, which represent multi-scale features from shallow to deep. Among them, Fo has higher spatial resolution but weaker semantic information, while F3 contains strong semantic information but fewer spatial details.
Fencoder = Swinnrranffomer((]
where I is the input image, and Fencoder (Fo, F1, F2, F3) represents the feature pyramid extracted by the Swin Transformer.
Pyramid Fusion Module: Pyramid Fusion uses Progressive Upsampling and Cross-Level Feature Fusion to achieve the interaction of multi-scale information.
The highest-level feature Fy is upsampled through Deconvolution, then fused with Fy:
Fg Conv(F2 + (econv3-2(FF))
14

---

The fusion continues layer by layer until the highest resolution is restored:
f = Conv(Fi + Deconv2-=1(F.))
Fiusion Conv(P' + Deconvi-o(Fii)
The final fused feature map Fiusion simultaneously contains deep semantic information and fine- grained details.
B Dataset Statistics Details
Table 5 summarizes the statistics of the XLFM-Zebrafish dataset, which consists of image data from zebrafish in both free-swimming and fixed conditions. The datasct categorized into Pre-training Set, Training/Valiin "raining/Valiaation Set, and Test Set to facilitate different stages of model development. The dataset carefully designed to ensure diversity motion complexity, viewpoint variations, and temporal resolutions. By pretraining on large-scale, complex free-swimming zebrafish data, the model gains stronger ability to generalize and better reconstruct simpler fixed zebrafish data, leading to improved performance. This structured dataset and training methodology provide a standardized benchmark for evaluating XLFM-Former and contribute to the advancement of XLFM-based 3D reconstruction techniques.
Pre-training Set (Free-swimming Zebrafish): The Pre-training Set comprises three free-swimming zebrafish (ml, m2, m3), totaling 20,123 images, all captured at a 10 fps sampling rate. Free- swimming zebrafish exhibit highly dynamic and complex motion, leading to significant variations in viewpoint and pose. This complexity presents challenge for 3D reconstruction but also offers an opportunity for the model to learn richer geometric structures. To leverage this complexity, we employ unsupervised pretraining this large-scale free-swimming dataset before training the fixed zebrafish dataset. By learning from diverse, naturally occurring light field transformations, the model develops robust understanding of light field geometry and depth relationships. This approach significantly improves performance when fine-tuned on simpler fixed zebrafish data, demonstrating the benefits of pretraining large, diverse datasets before supervised learning smaller, more controlled datasets.
Training/Validatioon Validation Set (Fixed Zebrafish): The Training/Validation Set contains seven fixed zebrafish (fl-f7) with a total of 1,761 images. Most samples were collected at 10 fps, while some (e.g., f3) were acquired swimming zebrafish, the fixed zebrafish dataset presents a more structured and constrained setting, making it ideal target for supervised training once the model has been pretrained on more complex free-swimming data.
fps to introduce variations in temporal resolution. Compared
free-
Test Set (Fixed Zebrafish): The Test Set consists of six fixed zebrafish (t1-16) with total of 1,011 images. Some samples (e.g., 12, 14, 15) were acquired at fps, allowing comprehensive evaluation of XLFFM-Frrer's reconstruction performance under different sampling conditions.
C Loss Function Details
In our proposed XLFM-Former framework, we employ different loss functions tailored for pretrain- ing and reconstruction tasks to ensure robust and high-quality 3D volume generation.
Pretraining Loss: For the self-supervised pretraining task, where Masked View Modeling- Light Field (MVM-L.F), we adopt simple & loss to enforce the consistency between the predicted and ground truth light field views:
Cpeetrain
|| 7
(15)
where i represents the predicted light field views, and I denotes the original (ground truth) views before masking. The choice of &, loss is motivated by its stability in regression tasks and its ability to ensure smooth reconstructions during pretraining.
XL.FM Reconstruction Loss: For the final 3D reconstruction task, we minimize the following composite loss function:
15

---

Table 5: The XLFM-Zebrafish dataset statistics.
Dataset Name
Number Images Sampling Rate (fps)
Free-swimming Zebrafish (Pre-training Set)
moving fish (ml)
4000
10
moving fish2 (m2)
7332
10
moving Fixed Zebrafish ( TTatning/ Valinationiinninnnnnnnn Set)
hxed. hxed_hshh hihl (f1)
240
10
fish2
117
Exed lish3 (13)
318
1
lied fish4
10
fixed. fish5 (f5)
374
10
fixed. fishó
214
10
fixed_fisha fish7 (17)
184
10
Fixed Zebrafish (Test
hixed_sshh
300
10
test, fixed. fish2 (12)
1
lest [ixed_fish3
334
10
test. fixed fish4 (14)
lest. lixed___eeee fish5 (15)
61
1
test. fishe__ddh6h (16)
214
10
H2B Nemos
lest_ fixed_fsst (11)
73
10
lest lixed fish2 (12)
73
10
test fixed_fish3 (13)
73
10
test. fixed. (14)
1
1
1
Lital
LMs_SSIM
+
Crage + CPSNR
A1
A2
A3
(16)
1
1
LMSE
LoRC.
A4
A5
Each term in the loss function contributes to a different aspect of the 3D reconstruction quality, ensuring sharpness, accuracy, and optical consistency. Below, we provide detailed breakdown of each component:
Multi-Scale Structural Similarity (MS-SSIM) Loss: Structural Similarity Index (SSIM) is widely used to measure the perceptual similarity between images. We employ multi-scale SSIM (MS-SSIM) loss capture both local and global structural fidelity:
LMS_SSIM = MS-SSIM(V, V),
(17)
where V and V represent the predicted and ground truth 3D volumes. MS-SSIM helps preserve structural details and enhances the perceptual quality of the reconstruction.
Edge-Aware Loss: To enhance edge sharpness and suppress blurring, we define the edge-preserving loss weighted combination of edge loss and multi-scale MSE loss:
Lidge Aware
1
(IV=
V.Vil + I VVVil
Audgo
+ V.Vlll)
5
v(+) 12-
(18)
Ame
Here, Ledges ensures sharpness by penalizing gradient differences along spatial dimensions, while Lmulti scale enforces consistency across multiple resolutions, improving global structure and fine details.
Peak Signal-to-Noise Ratio (PSNR) Loss: PSNR is a widely used metric for measuring signal fidelity. We define loss function that penalizes low PSNR values:
16

---

Table 6: Loss function. 'Full' denotes the complete model with all loss terms. 'w.o. ms ssim' removes the multi-scale SSIM loss. "w.o. Edge Aware' omits the edge-awar aware loss. "w.o. PSNR' excludes the PSNR loss. w.o. MSE Loss' removes the MSE loss. The best results are highlighted in bold.
Method
PSNRT
SSIM
w.o. LMS_SSI
53.2787
0.9937
w.o. LEdge_Aware
52.1870
0.9922
W.O. LISNR
53.0741
0.9935
W.O. LMSE
53.2521
0.9937
Full
54.0435
0.9944
CPSNR -PSNR V).
(19)
where a higher PSNR corresponds to higher-quality reconstructions. By minimizing this loss, we encourage the model to reduce noise and artifacts. The PSNR term included as an auxiliary compo- nent that provides consistent yet minor gain in reconstruction stability, rather than representing core novelty our approach.
Mean Squared Error (MSE) Loss: The MSE loss ensures pixel-wise intensity similarity between the predicted and ground truth volumes:
MASE 1|V V112.
(20)
While MSE is commonly used in image restoration, it is prone to blurring. Thus, we use it in combination with perceptual losses (e.g., MS-SSIM and edge loss) to balance fine details and overall similarity.
C.1 Ablation Study on Loss Functions
To evaluate the impact of different loss components on the overall performance of our model, we conduct an ablation study by removing individual loss terms and measuring the performance in terms of PSNR and SSIM. The results are summarized in Table 6.
From the table, we observe that the complete model (Full) achieves the highest PSNR of 54.0435 and SSIM of 0.9944, demonstrating the effectiveness integrating all loss terms.
Effect of Multi-Scale SSIM Loss: Removing the multi-scale SSIM loss (w.o. Cas_ssia results in performance drop, reducing PSNR to 53.2787 and SSIM to 0.9937. This highlights the importance of SSIM-based perceptual loss in preserving structural information.
Effect of Edge-Aware Loss: When the edge-aware loss is removed (w:o. L Aware). the PSNR decreases significantly to 52. 1870, while SSIM drops 0.9922. This indicates that the edge-aware term plays crucial role in maintaining sharp details and edge consistency.
Effect of PSNR Loss: Excluding the PSNR- based loss (w.o. LPSNR) results in minor degradation in performance, with PSNR reducing to 53.0741 and SSIM to 0.9935. This suggests that optimizing directly for PSNR provides some benefits but not the dominant factor.
Effect of MSE Loss: When the MSE loss is removed (w.o. LMSE), the PSNR slightly drops to 53.2521, while SSIM remains 0.9937. This indicates that MSE contributes to pixel-wise accuracy but is less critical than the other loss terms.
D Training Configuration
For the MVM-LF task, we adopt batch size of 8 to ensure stable training dynamics while maintaining an efficient balance between computational cost and convergence stability. To further enhance the training process and prevent the model from becoming trapped local optima, we employ the ReduceeLOnnlltteer learning rate scheduler. The initial learning rate is set to Ie-4, and the scheduler
17

---

Table 7: Data augmentation techniques applied during training. Each transformation applied with probability of 0.5.
Augmentation Type
Probability
Random Flip
0.5
Random Rotation
0.5
Random Gaussian Noise&Blur
0.5
Random Brightness Conntrast
0.5
dynamically adjusts the learning rate based on validation loss fluctuations, reducing it when no improvement is observed for a predefined number of epochs. This adaptive learning rate strategy helps in maintaining steady convergence while preventing premature stagnation in suboptimal solutions.
To achieve robust feature learning and ensure generalization across diverse data distributions, we train the model for 250 epochs. Given the complexity of the task and the high-dimensional nature of the input data, prolonged training allows the model to capture intricate spatial and structural information effectively. All experiments conducted in distributed computing environment equipped with four NVIDIA A100-80GB SMX4 GPUs, leveraging multi-GPU parallelism to accelerate training and optimize resource utilization.
For the XLFM reconstruction task, the training setup remains largely consistent with the MVM-LF configuration. However, a notable difference is the use of batch size of 1, which aligns with the requirements of volumetric reconstruction. Given that volumetric data often involves higher memory footprints due to its ditee-dimensional representation, smaller batch size ensures that computations remain feasible within available GPU memory constraints. All experiments are implemented using PyTorch Lightning, high-level deep learning framework that simplifies training and enhances reproducibility.
D.1 Data Augmentation Strategy
To improve the model's robustness and generalization capability, we apply series of data augmen- tation techniques during training. These augmentations are applied with a probability of 0.5, as summarized in Table 7.
These augmentation techniques enhance the model's s ability to handle variations in real-world data, reducing overfitting and improving generalization performance.
D.2 Model Architecture
For all experiments, we adopt the Swin Transformer framework in its tiny configuration. The Swin Transformer is hierarchical vision transformer that efficiently models long-range dependencies while maintaining computational efficiency. The tiny variant provides a lightweight architecture suitable for our training setup, balancing performance and efficiency.
D.3 Extended Training on Additional Datasets
To further explore the scalability and generalization capability of XIFMM-Former, we conducted large-scale training on an extended dataset. This additional training aimed improve the model's ability to reconstruct fine-graines details while enhancing robustness across diverse volumetric data.
The large-scale training process required substantial computational resources, consuming approxi- mately 1344 A A100-80GB GPU hours. This extensive training allowed the model to refine its feature representation and leverage broader data distribution, leading improved reconstruction quality.
To demonstrate the effectiveness of this large-scale training, we provide a visualized demo showcasing the enhanced view synthesis capability. The qualitative results highlight the model's ability to generate high-fidelity reconstructions with improved structural consistency and perceptual quality, further validating the benefits of extended training.
18

---

D.4 Evaluation Details
All reconstructed volumes are normalized within the physical intensity range of [0, 2000], correspond- ing to the effective dynamic range of the neural fluorescence signals captured by our XLFM system. This differs from conventional RGB-based normalization (0,, 255]), and may lead slightly higher absolute PSNR/SSIM values compared with typical computer vision benchmarks. To ensure faimesss, all compared methods are evaluated under the same normalization setring.
E
Qualitative Comparison on XLFM-Zebrafish Dataset
To better understand the reconstruction quality beyond numerical metrics, we present qualitative comparisons in Figure 7. For each zebrafish sample, we visualize the XY projection and zoomed-in region-of-interest (ROI) across baseline methods, our proposed model, and the ground truth.
We observe that convolution- based methods (e.g.. ConvNeXt, EfficientNet) tend to oversmooth fine neural structures or introduce artifacts. Transformers such as PVT improve spatial continuity but still suffer from blurry ROIs. In contrast, XLFM-Former accurately recovers elongated dendritic shapes and figuenyyyyency details, closely matching the ground truth.
F
H2B-Nemos datasets
F1
Cross-Dataset Ablation Study on the H2B-Nemos Dataset
To further assess the robustness and generalizability of model design, we conduct ablation study on the newly collected H2B-Nemos dataset, which differs from the original G8S dataset in imaging conditions and noise statistics. All architectural and loss configurations remain unchanged, except for a slight adjustment to the ORC loss weight (from x 10 to 3 x 10-5) to account for minor differences in noise level.
As shown in Table 8, our approach maintains consistent improvement trends across modules and loss components. Each component contributes positively to reconstruction quality, demonstrating that the design principles established on the G8S dataset generalize well to new imaging domains.
Table Cross-dataset ablation study on the H2B-Nemos dataset.
Method
PSNR
APSNR
SSIM
ASSIM
Bascline
50.87
0.9903
+ MVM-LF only
51.66
+0.79
0.9940
+0.0037
+ ORC Loss only
51.77
+0.90
0.9935
+0.0032
Full (Ours)
52.34
+1.47
0.9955
+0.0052
These results confirm that the proposed components both architectural and loss-based remain effective across different zebrafish datasets, highlighting the robustness of XLFM-Former.
F.2 Robustness to ORC Loss Weight Variation
We further examine the sensitivity of our model to the weighting coefficient of the Optical Recon- struction Consistency (ORC) loss. As shown in Table the performance remains highly stable across a wide range of ORC weights, demonstrating that the method is not fragile or over-tuned to a specific configuration.
Specifically, sweeping the ORC weight from 1 to 10-2 results in only minor fluctuations in PSNR and SSIM. Weights that too small (e.g., 10-4) may lead to unstable convergence, while larger values (e.g.. 5 x 10-3 to 10 10-2) yield consistently strong performance and stable optimization behavior. These results confirm that the ORC formulation provides reliable supervision across diverse imaging conditions without requiring fine-graine ime-grained tuning.
Slight tuning is expected when transferring to new datasets due to differences in fluorescence properties or noise characteristies. Importantly, no changes to the architecture or training strategy are required, underscoring the generality and robustness of the proposed ORC formulation.
19

---

Zebrafish Sample #2
53.88-119900 98667
54.6k-89651
47.53
54.89809851
54,06009551
57.23009928
SU.3319993
PSNRASIM
ROI
ROI
ROI
ROI
ROI
ROI
ROI
R001
CravNNee
FificherNeer
PVT
Besnaróo
Renet10r
U-rtt
Our
Label
Zebrarich Sample 43
PSNRASIM
ROI
ROI
ROI
ROI
ROI
ROI
Cany
U-Net
Zebraiishh Sample 44
$1,300 9WR3
49.5669 9801
99.56119951
49.4770.9790
49.47.1.9787
51.1
54,44199911
PENRASIM
Conn NoM
KifissorNNer
PVT
Resnet50
Renncto1
L-Not
Qurs
Label
Zebra Tish Semple #5
11.52/119912
48.639899772
48.32N1-984
45.82207766
47.09-11 5M00
5L.1699090
54,51199955
PSNRANSIM
ROI
ROI
ROI
ROI
ROI
ROI
ROI
ROI
Com
PVT
Resns150
Resnil10
C-Nel
Qui
Label
49,7N11
41,47/00715
51.95119956
PSNRSSIM
ROI
ROI
ROI
ROI
BOI
ROI
BOI ROI
CORUNeXI
Figure 7: Qualitative comparison on XLFM-Zebrafish samples #2-#6. Each row shows a different zebrafish sample. For each method, the XY projection and ROI zoom-in are shown. PSNR/SSIM values are displayed at the left corner of each full-frame image. The visualizations confirm that our method consistently preserves neural structures across diverse samples and acquisition conditions.
20

---

Table 9: ORC loss weight sweep showing the stability of XLFM-Former under different loss configurations.
ORC Loss Weight
PSNR 1
SSIM t
Observation
1 x 10-1
50.98
0,991
Unstable convergence
1 x 10-3
51.70
0.994
Lower performance
x 10
52.34
0,995
Best, stable convergence
5 x 10-2
52.23
0,995
Slightly lower than optimal
1 x 10-2
52.35
0.995
Similar but less stable
Comparison with Classical Iterative Deconvolution
We further compare method with the classical iterative Richardson-Lucy Deconvolution (RLD) algorithm under different iteration counts (20, 30, and 40) on the challenging 212B-Nemss dataset. While RLD achieve very high PSNR and SSIM when run for many iterations, it incurs orders of magnitude higher memory usage and significantly lower speed, making impractical for real-time or high-throughput applications.
Table 10: Comparison with the classical Richardson-Lucy Deconvolution (RLD) on the H2B-Nemos dataset.
Method
PSNR t
SSIM
FPS 1
Peak Memory J. (MiB)
RLD-20
67.36
0.9987
0.068
20899
RLD-30
69.85
0.9994
0.046
20899
RLD-40
72.31
0.9998
0.035
20899
XLFM-Former (Ours)
52.34
0.9955
48.44
2631
Despite the numerical advantage of RLD at high iteration counts, our method achieves more than 700x faster inference and 8x lower memory consumption, while maintaining high structural fidelity. This highlights XLFM-Former's practicality for large-scale or real-time microscopy reconstruction.
F4 Inference Efficiency and Practical Deployment
To evaluate practical efficiency of our method, we benchmarked XLFM-Former against four widely used backbone models on single NVIDIA A100 GPU (80 GB), reporting inference speed (FPS), peak memory usage, and reconstruction accuracy (PSNR). As shown in Table 11, XLFM- Former achieves 48 48.44 FPS, well above the 30 FPS real-time threshold, while maintaining competitive memory consumption (2631 MiB) and the highest reconstruction fidelity (PSNR 52.34 dB) among all compared methods.
These results demonstrate that XLFM-Former is not only accurate but also efficient enough for real-time and high-throughput 3D microscopy applications.
Table 11: Inference efficiency and reconstruction accuracy on the H2B-Nemos dataset (single A.00 GPU).
Method
FPS 1
Peak Memory (MiB)
PSNR
ConvNeXt
98.56
2565
50.88
ViT
94.79
2875
50.87
U-Net
74.34
2423
51.00
ResNet-50
41.34
6467
51.34
XLFM-Former (Ours)
48.44
2631
52.34
XLFFM-Former provides an effective balance between reconstruction fidelity, runtime speed, and memory efficiency, making suitable for real-time 3D reconstruction large-scale biological imaging systems. Qualitative comparisons in Figure further demonstrate that applying the ORC
21

---

Loss suppresses background hallucinations and yields clearer neuronal boundaries across multiple orthogonal planes (XY, XZ. YZ).
F.5 Extended analysis of ORC Loss
To further validate the contribution of the Optical Reconstruction Consistency (ORC) Loss, we provide detailed quantitative comparison across multiple held-out zebrafish samples from the H2B-G8S and H2B- B2B-Nemos datasets. The ORC Loss encourages consistency between the forward projections reconstructed and ground-truth volumes through the PSF. effectively constraining hallucinated structures. As shown in Table 12, incorporating the ORC (PSF) Loss consistently improves both PSNR and SSIM across all samples, confirming that it reduces reconstruction artifacts and enhances cross- generalization, particularly in challenging anatomical regions such as the hindbrain.
Table 12: Quantitative evaluation of ORC (PSF) Loss independent datasets. Incorporating the loss improves both fidelity (PSNR) and perceptual quality (SSIM).
Sample PSNR w/ PSF
PSNR wio PSF
SSIM PSF
SSIM w/o PSF
APSNR
ASSIM
G8S-#1
52.97
51.83
0.9913
0,9899
+1.14
+0.0014
G8S-#2
59.34
59.13
0.9957
0.9955
+0.21
+0.0002
G8S-#3
48.27
47.00
0.9894
0,9879
+1.27
+0.0015
G8S-#4
54.12
53.90
0.9944
0.9940
+0.21
+0.0005
G8S-#5
54.17
53.73
0.9949
0,9942
+0.43
+0.0008
G8S-#6
48.88
47.26
0.9931
0.9930
+1.62
+0,0002
Nemos-#1
52.88
52.17
0.9942
0.9914
+0.71
+0.0028
Nemos-#2
50.62
49.55
0.9928
0.9892
+1.07
+0.0030
YZ View Gide View
VZ. View (iid View)
(Top
(Tep
(4) w/ ORC Loss
(b) w/o ORC Loss
Figure 8: Qualitative comparison of reconstructions with and without the ORC Loss. Each column shows orthogonal views (XY, XZ, YZ) of the same zebrafish sample. The ORC Loss visibly suppresses background hallucinations and enhances structural clarity, especially in the hindbrain.
G
Future Work
In addition, several preliminary experiments indicate that satisfactory reconstruction quality can still be achieved even when only subset of microlenses (i.e., partial angular views) is used. This observation opens up another promising research direction: (1) how to adaptively determine the optimal number of lenses or sub-aperture views to balance throughput and reconstruction fidelity; and (2) whether a subset of the remaining views could be reserved as an evaluation signal to serve as an absolutely accurate ground truth for self supervied or cross- cross-view validation.
Beyond these specific directions, our findings suggest that future XLFM reconstruction research may benefit from dual emphasis physical interpretability and functional usability bridging optical modeling with neuroscience analysis in a unified framework.
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The abstract and introduction clearly state the core contributions: (1) self- supervised pretraining strategy (MVM-LF), (2) a physics-guided rendering loss (ORC Loss), and (3) benchmark dataset for XLFM reconstruction.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The current study is focused on XLFM imaging, where zebrafish is the most established and widely used biological model due to its optical transparency and large- scale neural activity observability. While our method is general in formulation, the dataset is constrained by available imaging setups. Extension to other modalities (e.g., light-sheet, two-photon) and species is left for future work.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model vell-specifiiattin, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
23

---

3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: The paper does not contain any theoretical results formal proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
The paper provides all necessary details for reproduction. The dataset, code, and pretrained checkpoints will be released upon paper acceptance upon request during review.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility.

---

In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [No]
Justification: We plan release the full dataset, source code, and pretrained checkpoints upon paper acceptance. All implementation and training details are described in the main paper and will be supported by a reproducibility-readd codebase. We can also provide anonymized access to reviewers upon request during the review period.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
//ipps.c.. cttpssccccpccccccccccccccccccccc gww.d/s///////eeg//ee CCodeeubeeeeeeeeeeeee issiiissii asssonnolll onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper provides all necessary details for reproduction.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Additional training details and ablation protocols are provided in the appendix. Guidelines:
The answer NA means that the paper does not include experiments.
25

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errоrs is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: All experiments were conducted on a machine with 4xNVIDIA A100 80GB GPUs. Training single SLFM-Former model takes approximately hours. Pretraining using MVM-LF takes around 20 hours.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported didn't 'ttnt make it into the paper).
paper (e.g., preliminary or failed experiments that NeurlPS Code of Ethics https http///eurrr..ree ct/pppbliccttiiccctiedeeeeeeleee
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the
Answer: [Yes]
Justification: This research complies with the NeurIPS Code Ethics. All datasets were collected in accordance with institutional and ethical standards for animal imaging.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
26

---

Justification: This work contributes to foundational research in high-speed volumetric neural imaging, with potential positive impact in neuroscience and biomedical research. The improved efficiency and data efficiency of XI.FM reconstruction may facilitate large-scale brain activity studies in model organisms, aiding basic scientific discovery. We are not aware of direct negative societal impacts at this stage, as the method is domain-specifie, non-pe non-persone personal, and not intended for deployment in high-stakes decision-making.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymeent of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
The released models and datasets are domain-specific scientific tools for zebrafish light field microscopy and do not pose risks of malicious misuse. No internet- scraped or human-related data are involved.
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
27

---

Justification: We use publicly available Swin Transformer implementation under the MIT license and cite the original paper accordingly. All other data and code used this paper were developed by the authors and will be released under a permissive license (e.g., CC-BY 4.0 or MIT) upon publication.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the assct's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We introduce the XLFM-Zebrafish dataset and the XLFM-Formed model. Documentation for dataset structure, data acquisition protocol, and code usage will be provided alongside the release.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve human subjects or crowdsourcing. All experiments are performed on microscopy data of zebrafish.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
28

---

15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This work does not involve human subjects and does not require IRB or ethics board approval. All data is obtained from in vivo zebrafish imaging under standard ethical protocols for animal research.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: No large language models (LLMs) were used as part of the core methodology.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: //neuriiss.../eeeee 220 /LLM) for what should or should be described.
29


Reviews:
[{"review_id": "ptL0BvaAid", "reviewer": "Reviewer_1jCx", "summary": "Fourier light field microscopy is a kind of 3D snapshot microscopy that allows reconstruction of a 3D sample from a single 2D image. The authors present a new method to reconstruct volumes from images captured by an XLFM (a kind of Fourier light field microscope). Reconstruction of Fourier light field images is a significant and useful problem to solve using deep learning, because volume reconstruction is typically an expensive and slow process that requires an iterative deconvolution process to solve. Deep learning-based reconstructions allow reconstructions to be obtained significantly faster than iterative optimization algorithms, and can also improve reconstruction quality by accounting for sample priors based on the training data. However, it is more expensive to collect the training data of zebrafish volumes than it is to collect the 2D images. The authors' new method presents a promising technique to scale up the training of reconstruction neural networks by pretraining a feature extractor in a self-supervised manner using just the 2D image data, while also providing a valuable training dataset of zebrafish volumes imaged through a Fourier light field microscope to the community. The pretrained feature extractor serves to significantly improve the reconstruction quality of a trained reconstruction neural network, providing higher reconstruction quality with just 20% of the volumetric ground truth training data (52.67 dB PSNR) than a neural network trained without the pretrained features using 60% of the data (52.66 dB PSNR).", "questions": "My questions/suggestions are all related to the \"Major Issues\" listed above. With added evaluations/explanations that address the numbered points raised in the \"Major Issues\" section, I would definitely increase my score. The significance and originality of this work seem apparent, but it is currently being limited by the clarity of the presentation/evaluation.\n\nFor convenience, I am pasting those Major Issues again:\n\n1. *Explanation of ORC Loss and presentation issues*: While the authors have a promising method, the paper suffers from widespread issues in its presentation quality that limit the clarity of the paper. The most major issue with presentation is that one of the major contributions of the paper as claimed by the authors, the \"ORC Loss,\" does not obviously appear in any explanation of the losses used for the reconstruction phase of the training process. The authors later introduce a \"PSF Loss\" which is not explained, which is likely the \"ORC Loss,\" but this is not clear. Other issues with the presentation are listed in the minor issues section below.\n\n2. *Analysis of ORC Loss and hallucinations*: Further, to substantiate the claim that the ORC/PSF Loss can reduce artifacts in the reconstructions and improve cross-sample generalization, the authors should include some examples of reconstructions from networks trained with and without the ORC Loss. While it is apparent that penalizing the forward projections of the reconstruction and the ground truth can enforce some consistency between the reconstruction and the measured snapshot image, there can still be hallucinated cells/artifacts in the reconstruction. Indeed, some cells/background appear in the XLFM-Former in Figure 5 (hindbrain) which are not present in the ground truth. Also, some explanation of why this ORC Loss does not use the measured image instead of a forward projection would be useful (e.g. whether this helps avoid reconstruction artifacts due to some background/dark frame in the measurement).\n\n3. *Appropriate baselines*: The authors compare their reconstructions against reconstructions from a range of other neural network architectures. However, they do not compare their reconstructions using the methods described by prior work in  snapshot imaging (Fourier light field or mask-based ) with deep learning, for example: Yanny et al.: \"Deep learning for fast spatially varying deconvolution\", Optica, 2022, https://doi.org/10.1364/OPTICA.442438 and references [25, 30, 9] from their paper. If memory is a concern, an explanation of why these other methods are not appropriate for these data would be useful. It would also be useful to compare against a reconstruction using an iterative deconvolution algorithm such as Richardson-Lucy.\n\n4. *Ground truth in datasets*: Another major issue is that the collected dataset as described, though it is valuable for the computational optics community, does not describe the ground truth data collection process or resolution, though some kind of volumetric ground truth must be present and is shown in Figure 5. The authors should clarify the process of collecting the ground truth for the dataset.", "limitations": "Yes (aside from the critiques raised in this review).", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strengths\nThe authors appear to have an original and useful technique for reconstruction of Fourier light field/XLFM images with a pretraining step that significantly improves the training efficiency of volumetric ground truth (which is experimentally expensive to collect and computationally expensive to train with). This method, along with the accompanying standard dataset of volumes, could be significant for the field.\n\n# Weaknesses\n\n## Major Issues\n\n1. *Explanation of ORC Loss and presentation issues*: While the authors have a promising method, the paper suffers from widespread issues in its presentation quality that limit the clarity of the paper. The most major issue with presentation is that one of the major contributions of the paper as claimed by the authors, the \"ORC Loss,\" does not obviously appear in any explanation of the losses used for the reconstruction phase of the training process. The authors later introduce a \"PSF Loss\" which is not explained, which is likely the \"ORC Loss,\" but this is not clear. Other issues with the presentation are listed in the minor issues section below.\n\n2. *Analysis of ORC Loss and hallucinations*: Further, to substantiate the claim that the ORC/PSF Loss can reduce artifacts in the reconstructions and improve cross-sample generalization, the authors should include some examples of reconstructions from networks trained with and without the ORC Loss. While it is apparent that penalizing the forward projections of the reconstruction and the ground truth can enforce some consistency between the reconstruction and the measured snapshot image, there can still be hallucinated cells/artifacts in the reconstruction. Indeed, some cells/background appear in the XLFM-Former in Figure 5 (hindbrain) which are not present in the ground truth. Also, some explanation of why this ORC Loss does not use the measured image instead of a forward projection would be useful (e.g. whether this helps avoid reconstruction artifacts due to some background/dark frame in the measurement).\n\n3. *Appropriate baselines*: The authors compare their reconstructions against reconstructions from a range of other neural network architectures. However, they do not compare their reconstructions using the methods described by prior work in  snapshot imaging (Fourier light field or mask-based ) with deep learning, for example: Yanny et al.: \"Deep learning for fast spatially varying deconvolution\", Optica, 2022, https://doi.org/10.1364/OPTICA.442438 and references [25, 30, 9] from their paper. If memory is a concern, an explanation of why these other methods are not appropriate for these data would be useful. It would also be useful to compare against a reconstruction using an iterative deconvolution algorithm such as Richardson-Lucy.\n\n4. *Ground truth in datasets*: Another major issue is that the collected dataset as described, though it is valuable for the computational optics community, does not describe the ground truth data collection process or resolution, though some kind of volumetric ground truth must be present and is shown in Figure 5. The authors should clarify the process of collecting the ground truth for the dataset.\n\n## Minor Issues\n\n* The second paragraph of the introduction does not motivate the need for learning-based reconstruction. While this may be obvious to experts of both machine learning and microscopy, it would be helpful to briefly explain why learning-based reconstructions for LFM are desirable at the beginning of the paper.\n* Figure 1, the caption describes splitting the raw image into 27 viewpoints \"based on physical coordinates.\" This language is unclear, but based on the figure it appears that the 27 projections of the fish from each beam of the XLFM PSF have been cut out, presumably centered on the x-y pixel coordinate on the camera of the focal point from each microlens. The caption explaining this process could be made more clear (or point the reader to a more detailed explanation), especially because this is a core contribution of the proposed method.\n* Line 38 - 39, the sentence seems to be missing a \"while\" or \"but\" to complete the explanation for the lack of volumetric training data.\n* Line 42-43, the authors state that \"most existing approaches overlook the wave-optical nature of the XLFM forward model.\" However, other deep learning 3D snapshot reconstruction methods do explicitly take into account the wave-optics nature of the forward model: reference [9] in the paper and also the work of Yanny et al.: \"Deep learning for fast spatially varying deconvolution\n\", Optica, 2022, https://doi.org/10.1364/OPTICA.442438. The authors' intent here may be to claim that other methods typically do not include the forward projection in the loss function, but this is not the only way to take into account the forward model in the formulation of the reconstruction method.\n* Line 48, the term \"XLFM-Former\" is introduced with no prior explanation.\n* Line 53, \"Fourier neural operators\" refers to learning mappings between function spaces, typically to solve PDEs. The authors likely meant global convolutions implemented as Fourier convolutions, which do require large amounts of memory.\n* Points 1 and 2 of the key insights motivating the XLFM-Former in the introduction seem to be making the same point.\n* Figure 2 would benefit from some text labels on the image triplets.\n* Line 75, there seems to be a missing comma or an extra space.\n* Line 62, \"MVM-LF\" is introduced for the first time in the main text without a definition of the acronym aside from the abstract.\n* Line 84, the authors construct a benchmark using rather low FPS acquisitions (1 or 10 fps), which seems to imply high SNR. Could the authors comment on the effect of low SNR on the reconstruction quality using XLFM-Former?\n* Figure 3, this figure would also benefit from more clear text labels or titles across the images. This figure is also missing an explanation of the layout in the caption.\n* Figure 4, many of the layers have abbreviated or short form names which are not clear without additional labels or a caption. The paper would greatly benefit from bringing some summary of the detailed explanation of the architecture in the supplement into the main text and the caption of Figure 4.\n* Section 3.1, the paper would benefit from a pointer to a detailed explanation of the real-time tracking and treatment of motion artifacts (either in the supplementary material or through a citation).\n* Line 172, what is meant by \"high-precision light field reconstruction?\"\n* Supplement Table 5, Is Gaussian noise an appropriate augmentation? Would including Poisson noise as an augmentation make more sense due to the fact that this paper is working with fluorescence? Similarly, some discussion on the choice of other augmentations (especially rotation, since rotating the system would presumably change the subimages on the camera) would be useful.", "was_revised": true, "final_justification": "I believe the method is sound and interesting for the field of light field volume reconstructions. Using deep learning to accelerate these reconstructions is an important task, and the presented method addresses a few challenges with using deep learning here: (1) fast, real-time reconstructions (typically a slow iterative process without deep learning), (2) a memory efficient architecture, (3) a pre-training strategy to make more effective use of the fact that lots of light field snapshot image data can be collected, but not nearly as much volumetric ground truth data.\n\nThe authors also seem to be addressing some of the issues with clarity and presentation during the rebuttal process. However, their evaluations against prior work in this field are limited and their ground truth is not actually the ground truth structure, but the result of a standard iterative deconvolution algorithm (Richardson-Lucy). While these prior deep learning methods may require lots of memory/multiple GPUs for their largest volumes, it would still have been possible to demonstrate performance comparisons on smaller simulated volumes on a single GPU.\n\nDue to these limitations in their evaluation, I am only increasing my score to a 4.", "rebuttal": "**We sincerely thank the reviewer for their detailed and thoughtful feedback.** \n\n### Comment 1:\n\n> *Explanation of ORC Loss and presentation issues*: While the authors have a promising method, the paper suffers from widespread issues in its presentation quality that limit the clarity of the paper. The most major issue with presentation is that one of the major contributions of the paper as claimed by the authors, the \"ORC Loss,\" does not obviously appear in any explanation of the losses used for the reconstruction phase of the training process. The authors later introduce a \"PSF Loss\" which is not explained, which is likely the \"ORC Loss,\" but this is not clear. Other issues with the presentation are listed in the minor issues section below.\n\n### Authors’ response:\n\n\"ORC\" stands for Optical Reconstruction Consistency, which encourages consistency between the forward projection of the reconstructed volume and the measured snapshot image. Specifically, this is implemented by comparing the forward projections (using the PSF) of both the reconstructed volume and the ground-truth label, thereby enforcing alignment between the predicted and real-world optical measurements. Thus, we introduced it as \"PSF Loss\" in the implementation section, but this is indeed the same as the ORC Loss.\n\n\n\n### Comment 2.1:\n\n> *Analysis of ORC Loss and hallucinations*: Further, to substantiate the claim that the ORC/PSF Loss can reduce artifacts in the reconstructions and improve cross-sample generalization, the authors should include some examples of reconstructions from networks trained with and without the ORC Loss. \n\n### Authors’ response:\n\nWe appreciate the reviewer’s suggestion to provide visual examples comparing models trained with and without the ORC Loss. We have prepared visualization results similar to Figure 5 in the main paper. However, due to the NeurIPS 2025 policy which prohibits images and figures in the rebuttal, we are unable to include them here.\n\nTo address this concern within the policy constraints, we instead provide quantitative comparisons on multiple held-out zebrafish samples from the H2B-G8S and H2B-Nemos datasets. As shown in the table below, incorporating the ORC (PSF) Loss consistently improves both PSNR and SSIM across all test samples. These improvements demonstrate that the ORC Loss helps reduce artifacts and enhances cross-sample generalization, particularly in challenging anatomical regions such as the hindbrain.\n\n\n\n| Sample   | PSNR w/ PSF ↑ | PSNR w/o PSF ↑ | SSIM w/ PSF ↑ | SSIM w/o PSF ↑ | ΔPSNR ↑ | ΔSSIM ↑ |\n| -------- | ------------- | -------------- | ------------- | -------------- | ------- | ------- |\n| G8S-#1   | 52.9684       | 51.8263        | 0.9913        | 0.9899         | +1.1421 | +0.0014 |\n| G8S-#2   | 59.3393       | 59.1274        | 0.9957        | 0.9955         | +0.2119 | +0.0002 |\n| G8S-#3   | 48.2672       | 46.9963        | 0.9894        | 0.9879         | +1.2709 | +0.0015 |\n| G8S-#4   | 54.1151       | 53.9047        | 0.9944        | 0.9940         | +0.2104 | +0.0005 |\n| G8S-#5   | 54.1677       | 53.7337        | 0.9949        | 0.9942         | +0.4340 | +0.0008 |\n| G8S-#6   | 48.8784       | 47.2617        | 0.9931        | 0.9930         | +1.6167 | +0.0002 |\n| Nemos-#1 | 52.8825       | 52.1697        | 0.9942        | 0.9914         | +0.7128 | +0.0028 |\n| Nemos-#2 | 50.6205       | 49.5483        | 0.9928        | 0.9892         | +1.0723 | +0.0030 |\n\n\n\n### Comment 2.2:\n\n> While it is apparent that penalizing the forward projections of the reconstruction and the ground truth can enforce some consistency between the reconstruction and the measured snapshot image, there can still be hallucinated cells/artifacts in the reconstruction. Indeed, some cells/background appear in the XLFM-Former in Figure 5 (hindbrain) which are not present in the ground truth. \n\n### Authors’ response:\n\nWe appreciate the reviewer’s observation. While some background or cellular structures appear in the XLFM-Former output but not in the ground truth, we believe these are not necessarily artifacts. Instead, they may reflect limitations of the imaging and labeling process.\n\nSpecifically, the ground truth may miss low-SNR or ambiguously localized structures—especially in dense regions like the hindbrain—where XLFM captures signals beyond what is confidently annotated. Since downstream analysis focuses on neuronal clusters rather than individual cells, such discrepancies are unlikely to affect practical utility.\n\nUltimately, these differences likely arise from imperfect ground-truth alignment, not reconstruction failure. Resolving this would require single-cell-resolution annotations, which XLFM currently lacks.\n\n\n\n\n\n### Comment 2.3:\n\n> Also, some explanation of why this ORC Loss does not use the measured image instead of a forward projection would be useful (e.g. whether this helps avoid reconstruction artifacts due to some background/dark frame in the measurement).\n\n### Authors’ response:\n\nWe appreciate the reviewer’s question. In the early stages of developing XLFM-Former, we did explore using the raw measured light field image directly in the ORC Loss. However, we found this approach to be significantly less stable during training due to **high levels of noise** and **background artifacts** inherent in the measured data (e.g., dark current, scattering, and sensor noise).\n\nBy contrast, using the forward projection of the ground-truth volume via the PSF offers a cleaner and more structured supervision signal, ensuring that the network focuses on learning the physical consistency between volume and measurement, without being distracted by low-level measurement noise. \n\n\n\n### Comment 3:\n\n> *Appropriate baselines*: The authors compare their reconstructions against reconstructions from a range of other neural network architectures. However, they do not compare their reconstructions using the methods described by prior work in snapshot imaging (Fourier light field or mask-based ) with deep learning, for example: Yanny et al.: \"Deep learning for fast spatially varying deconvolution\", Optica, 2022, https://doi.org/10.1364/OPTICA.442438 and references [25, 30, 9] from their paper. If memory is a concern, an explanation of why these other methods are not appropriate for these data would be useful. \n\n### Authors’ response:\n\n- **(1) Incompatible forward model (Yanny et al. [Optica 2022])**: Their method assumes a spatially variant convolutional model, which does not hold for XLFM. XLFM relies on complex wavefront propagation and angular multiplexing, violating the convolution assumption. Even recent XLFM-specific works (e.g., Ref. 25) are not compatible with Yanny et al.'s formulation.\n- **(2) Different reconstruction goals (Refs. 25, 30)**: These works focus only on sparse neural signals, ignoring structural context. In contrast, our method reconstructs full tissue volumes, critical for tasks like optogenetic targeting. Moreover, their approaches require immobilized samples (e.g., Ref. 25 fuses three frames), making them unsuitable for dynamic settings.\n- **(3) FourierNet (Ref. 9) is infeasible at our scale**: Operating fully in the Fourier domain, it demands >80GB memory even for smaller outputs. Our volumes (up to 300×600×600) render this approach impractical. Instead, we use Swin Transformer for efficient global modeling with far lower memory cost.\n\n\n\n### Comment 3.1:\n\n> It would also be useful to compare against a reconstruction using an iterative deconvolution algorithm such as Richardson-Lucy.\n\n### Authors’ response:\n\nWe appreciate the reviewer’s suggestion to compare against classical iterative deconvolution algorithms. As shown below, we benchmarked RLD at different iteration counts (20/30/40) on the challenging H2B-Nemos dataset. While (Richardson-Lucy Deconvolution)RLD can achieve high PSNR/SSIM given sufficient iterations, it incurs orders of magnitude higher memory usage and lower speed, making it impractical for real-time or high-throughput applications.\n\n| Method                 | PSNR ↑ | SSIM ↑ | FPS ↑     | Peak Memory ↓ (MiB) |\n| ---------------------- | ------ | ------ | --------- | ------------------- |\n| RLD-20                 | 67.36  | 0.9987 | 0.068     | 20899               |\n| RLD-30                 | 69.85  | 0.9994 | 0.046     | 20899               |\n| RLD-40                 | 72.31  | 0.9998 | 0.035     | 20899               |\n| **XLFM-Former (Ours)** | 52.34  | 0.9955 | **48.44** | **2631**            |\n\nWe appreciate the reviewer’s suggestion to compare against classical iterative deconvolution algorithms. As shown below, we benchmarked RLD at different iteration counts (20/30/40) on the challenging H2B-Nemos dataset. While (Richardson-Lucy Deconvolution)RLD can achieve high PSNR/SSIM given sufficient iterations, it incurs orders of magnitude higher memory usage and lower speed, making it impractical for real-time or high-throughput applications.\n\n\n\n### Comment 4:\n\n> *Ground truth in datasets*: Another major issue is that the collected dataset as described, though it is valuable for the computational optics community, does not describe the ground truth data collection process or resolution, though some kind of volumetric ground truth must be present and is shown in Figure 5. The authors should clarify the process of collecting the ground truth for the dataset.\n\n### Authors’ response:\n\nWe appreciate the reviewer’s attention to dataset clarity. We follow the established practice in XLFM literature and generate ground truth volumetric labels using 60 iterations of Richardson-Lucy Deconvolution (RLD-60), a widely adopted baseline in previous works. This iterative deconvolution algorithm is considered the gold-standard for recovering high-resolution volumes from XLFM snapshot measurements, and serves as our supervision target."}, {"review_id": "LNVhZnUYWt", "reviewer": "Reviewer_ioxC", "summary": "This paper presents a new dataset and a Swin Transformer based 3D reconstruction method for eXtended Light Field Microscopy (XLFM). The collected dataset include 13 fixed zebrafish for reconstruction task, and 3 free moving zerbrafish for pre-training the transformer model, by recording videos, a total of 22,581 images are captured. As the XLFM image can be divided into sub-aperture images specific for different viewing angle, the Swin Transformer is first pre-trained to learn the inter-aperture information by masking-and-completing task, and then the extracted multi-resolution feature is coupled with a convolutional network to reconstruct the 3D volume of the biological sample being imaged.", "questions": "* Implementation of XLFM-Transformer: It is unclear what does the “cropping operation” does. According to appendix A, it raises the image to a 3D representation of shape H x W x D x S ( the channel S =1). I wonder in what sequency does the sub-aperture images are stacked in the D dimension, how to ensure the 2D neighboring information of sub-images is reflected in this one dimensional stacking? Moreover, what is changed in the positional embedding for this added dimension? The equation 13 and 14 contradicts each other, and it seems duplicated illustration in the last part in Appendix A. \n* The idea of masked pretraining doesn’t seems to fit quite well with such 3D Swin Transformer architecture. For masked pretraining in [1,2], each sub-image is treated as a token and are assigned a unique positional embedding, which is clearly not the case for this implementation of “3D Swin Transformer”. Does the pretraining implemented in this paper simply set the masked sub-image to zero? A more detailed implementation should be included, and the rationality to do that should be discussed.\n* In the section “efficacy of  pretraining Strategies” ablation study, how does this “3D Swin Transformer” being pre-trained in ImageNet-1k/22k? \n\n[1] Masked Autoencoders Are Scalable Vision Learners\n\n[2] Boosting light ﬁeld spatial super-resolution via masked light ﬁeld modeling", "limitations": "Yes", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 1, "clarity": 1, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Quality: The paper is technically reasonable in a general sense; however, the implementation seems suspicious, and the experimental results appear contradictory. The quantitative results in Table 1 are excessively high, showing a PSNR of 45–54 dB for all methods (while the maximum PSNR for 8-bit signals is 48.13), essentially making them “perfect” equally. Such high PSNR values do not match the poor visualization results in Figure 5.\n\nClarity: The paper does not present the technical details clearly. The design of the lightweight decoder is unclear, and the pipeline presented in Appendix A is poorly written. Additionally, the details of the data, such as image resolution, sub-aperture image resolution, and reconstruction volume size, are not specified.\n\nSignificance: The proposed method has the potential to speed up the 3D reconstruction of XLFM through a Swin Transformer-based approach, along with a unique dataset for this specific task. However, the speed of inference is not mentioned or compared, and the significance of reconstructing the full 3D biological signal over simply reconstructing functional (neural) signals is not well established.\n\nOriginality: The work proposes both a unique dataset and a method specifically tailored for the task of XLFM reconstruction. Although it is rooted in masked pretraining and the Swin Transformer architecture, it introduces a fair rendering consistency loss to inject domain-specific prior knowledge.", "was_revised": true, "final_justification": "The authors’ response clarifies technical details that were previously unclear or misleading. I have raised my score to borderline reject, but I still have concerns:\n\n* The proposed “view masking” differs from standard approaches used in masked autoencoders and similar transformer architectures. Overlapping regions allow information leakage between views. This seems like an engineering choice tied to the Swin Transformer backbone; additional ablation studies are needed to justify the design. (The pixel-masked MAE ablation appears not to follow the standard protocol.)\n\n* PSNR may be an inappropriate metric given the data’s original sparsity. Evaluating neurological signal recovery would better demonstrate the pipeline’s practical significance.\n\nOverall, I acknowledge the dataset contribution, the PSF constraint for 3D recovery, and the effort toward faster reconstruction for near real-time whole-brain imaging. However, the technical contributions and evaluation are not sufficiently justified and fall slightly below the NeurIPS bar.", "rebuttal": "### Comment 1:\n\n> The paper is technically reasonable in a general sense; however, the implementation seems suspicious, and the experimental results appear contradictory. The quantitative results in Table 1 are excessively high, showing a PSNR of 45–54 dB for all methods (while the maximum PSNR for 8-bit signals is 48.13), essentially making them “perfect” equally. Such high PSNR values do not match the poor visualization results in Figure 5.\n\n### Authors’ response:\n\n**Summary.** \n\n- Our PSNR values are computed on 32-bit floating-point data normalized to [0, 2000], not 8-bit integers, making values >48.13 dB valid and consistent with prior XLFM work.\n\n\n\nWe would like to respectfully but firmly correct a misunderstanding in this comment.\n\nThe claim that PSNR values “cannot exceed 48.13 dB” is based on an incorrect assumption that the evaluation is performed on 8-bit integer images in the [0, 255] range. This is not the case in our setting. In line with standard practice in XLFM and calcium imaging, we work with 32-bit floating-point volumetric data, and normalize signals to a biologically meaningful range of [0, 2000]. Under this dynamic range, PSNR values significantly above 48 dB are not only mathematically valid, but widely reported in prior XLFM literature. For example, recent works such as *Fast light-field 3D microscopy with out-of-distribution detection and adaptation through Conditional Normalizing Flows* (Fast et al. [1]) clearly report PSNR values exceeding 50 dB under the same normalization convention, as shown in their Figure 3.\n\nWe acknowledge the reviewer’s concern regarding the visual appearance of Figure 5, but would like to clarify that Figure 5 highlights a challenging anatomical region (the hindbrain) with sparse neural signals. In such cases, PSNR—which evaluates fidelity across the entire 3D volume—may remain high even when small, localized discrepancies are visible. This is a known property of PSNR and does not indicate implementation issues.\n\nDue to this year’s NeurIPS policy prohibiting the release of anonymized links during the rebuttal phase, we are unable to share our implementation at this stage. However, we will additionally release our code upon acceptance, under the supervision of the Area Chair, to ensure full transparency and reproducibility.\n\n\n\n### Comment 2:\n\n> The paper does not present the technical details clearly. The design of the lightweight decoder is unclear, and the pipeline presented in Appendix A is poorly written. Additionally, the details of the data, such as image resolution, sub-aperture image resolution, and reconstruction volume size, are not specified.\n\n### Authors’ response:\n\nWe thank the reviewer for highlighting these points. Below, we provide the requested clarifications.\n\n**(1) Design of the Lightweight Decoder**\n\nOur decoder follows a simple yet effective architecture to upsample the Swin Transformer output to a full-resolution 3D volume. It consists of three key stages:\n\n- **Projection to depth**: A 1×1 convolution maps the Swin features (channel dimension 96) to the target depth size (D=300).\n- **Spatial upsampling**: Two successive transposed convolutions upscale the spatial resolution from 150×150 → 300×300 → 600×600.\n- **Final refinement**: A 1×1 convolution is applied to preserve local semantics and reduce boundary artifacts.\n\nThe corresponding PyTorch implementation is provided below for clarity:\n\n```python\nclass Lightweight3DDecoder(nn.Module):\n    def __init__(self):\n        super().__init__()\n        self.project_to_depth = nn.Conv2d(96, 300, kernel_size=1)\n        self.upsample_1 = nn.ConvTranspose2d(300, 300, kernel_size=4, stride=2, padding=1)\n        self.upsample_2 = nn.ConvTranspose2d(300, 300, kernel_size=4, stride=2, padding=1)\n        self.refine_output = nn.Conv2d(300, 300, kernel_size=1)\n\n    def forward(self, x):\n        x = self.project_to_depth(x)       # [B, 96, 150, 150] → [B, 300, 150, 150]\n        x = F.relu(self.upsample_1(x))     # → [B, 300, 300, 300]\n        x = F.relu(self.upsample_2(x))     # → [B, 300, 600, 600]\n        x = self.refine_output(x)          # → [B, 300, 600, 600]\n        return x\n```\n\n**(2) Clarification of Dataset Specifications**\n\n- **Raw image resolution**: 2048 × 2048, captured by an XLFM system with a microlens array.\n- **Sub-aperture view resolution**: Each of the 27 angular views is aligned and cropped to 600 × 600 pixels.\n- **Reconstructed 3D volume size**: 600 × 600 × 300 voxels.\n\n\n\n\n\n### Comment 3:\n\n>  The work proposes both a unique dataset and a method specifically tailored for the task of XLFM reconstruction. Although it is rooted in masked pretraining and the Swin Transformer architecture, it introduces a fair rendering consistency loss to inject domain-specific prior knowledge.\n\n### Authors’ response:\n\nWe thank the reviewer for recognizing the novelty of our dataset and the design of our domain-specific loss function, as well as its conceptual connection to masked pretraining.\n\nWhile our pretraining strategy is indeed inspired by masked autoencoders (e.g., MAE), our core contribution lies in a fundamentally different view-level masking mechanism tailored to light field microscopy. Instead of masking random image patches, we mask out entire angular views from the 27-view XLFM input and train the model to reconstruct the missing views from the remaining ones.\n\nThis design forces the network to learn explicit spatial-angular dependencies across views—an essential property for XLFM systems where angular consistency and optical geometry play a critical role. Unlike standard MAE, which operates on spatial patches, our approach requires the model to reason about structured angular correlations inherent in the light field representation.\n\nFurthermore, as shown in Table 1 of our manuscript, our view-masked pretraining strategy significantly outperforms random-patch masking (MAE-style) approaches, especially in low-supervision settings and when generalizing to new zebrafish strains. These results validate the advantages of our domain-aware masking paradigm and its alignment with the underlying physics of the imaging system.\n\n\n\n### Comment 4:\n\n> Implementation of XLFM-Transformer: It is unclear what does the “cropping operation” does. According to appendix A, it raises the image to a 3D representation of shape H x W x D x S ( the channel S =1). I wonder in what sequency does the sub-aperture images are stacked in the D dimension, how to ensure the 2D neighboring information of sub-images is reflected in this one dimensional stacking? Moreover, what is changed in the positional embedding for this added dimension? The equation 13 and 14 contradicts each other, and it seems duplicated illustration in the last part in Appendix A.\n\n### Authors’ response:\n\nWe appreciate the reviewer’s close reading and raise the following clarifications:\n\n- (1) “Cropping operation” explanation\n  We extract 27 sub-aperture views of size 600×600 from the 2048×2048 input image based on known microlens coordinates. This operation corresponds to spatially aligning and cropping the raw XLFM measurement into angular sub-images.\n\n- (2) Stacking order and spatial adjacency\n  The 27 views are stacked into a tensor following their spatial layout on the microlens array (i.e., consistent with optical geometry). While we maintain this order for reproducibility, we note that our 3D Swin Transformer possesses a large receptive field, making the model robust to minor changes in view order.\n\n- (3) Positional embedding for the D dimension\n  We do not alter the positional embedding mechanism of the Swin Transformer. During pretraining, we randomly zero out a subset of views (angular-wise masking), which does not require architectural modification. The model learns to complete missing views from visible ones, guided by inherent spatial-angular dependencies.\n\n- (4) Appendix A redundancy and equations\n  We acknowledge that Equation (13) in Appendix A is redundant and may have caused confusion. The decoder’s structure is accurately described by Equation (14)。\n\n\n\n### Comment 5:\n\n> The idea of masked pretraining doesn’t seems to fit quite well with such 3D Swin Transformer architecture. For masked pretraining in [1,2], each sub-image is treated as a token and are assigned a unique positional embedding, which is clearly not the case for this implementation of “3D Swin Transformer”. Does the pretraining implemented in this paper simply set the masked sub-image to zero? A more detailed implementation should be included, and the rationality to do that should be discussed.\n\n### Authors’ response:\n\nWe respectfully disagree with the reviewer’s concern, and appreciate the opportunity to clarify our design.\n\nWhile our MVM-LF framework may superficially resemble pixel-level masked autoencoders such as MAE, its **inductive bias, masking granularity, and objective** are fundamentally different and tailored to XLFM.\n\n- **Masking strategy:** Instead of masking image patches, we mask entire sub-aperture views (i.e., angular slices from XLFM’s 27-view input). This aligns with the physical imaging model of light field microscopy, where each view encodes angular information.\n- **Network compatibility:**  During pretraining, we randomly zero out a subset of views (i.e., set entire channels to zero) and train the model to reconstruct the missing views using the remaining ones.\n- **No change to positional embedding:** Since Swin Transformer already uses relative positional encoding within each view, and our masking is done at the view level, we **do not modify the embedding scheme**. The model naturally learns to associate angular correlations across channels.\n- **Empirical evidence:** Our ablation study (Table 2) shows that MVM-LF pretraining achieves **54.04 dB PSNR**, outperforming random pixel masking (**52.97 dB**) by a clear margin (+1.07 dB). Given the high baseline, this is a **significant improvement** in 3D volume reconstruction tasks."}, {"review_id": "lgYBWK3xq4", "reviewer": "Reviewer_c2EJ", "summary": "The paper tackles 3-D reconstruction for eXtended Light-Field Microscopy (XLFM) and introduces a complete learning framework that is both data-driven and physics-aware.\n- New benchmark: The authors curate XLFM-Zebrafish, a public dataset of 22,581 light-field frames with an accompanying evaluation protocol, filling the long-standing gap of standardized data for this modality.  \n- Self-supervised pre-training (MVM-LF): They propose Masked View Modeling for Light Fields, in which 70 % of the sub-aperture views are masked and the network learns to predict them from the remaining views, capturing angular correlations without labels.  \n- Physics-consistent fine-tuning: A Swin-Transformer encoder plus lightweight CNN decoder, dubbed XLFM-Former, is fine-tuned with an Optical Rendering Consistency (ORC) loss that renders the predicted volume through the system PSF and enforces agreement with the measured views, ensuring physical plausibility.  \n- Results: On the new benchmark the method attains 54.0 dB PSNR and 0.994 SSIM, improving average PSNR by 7.7 % over seven recent learning-based baselines.", "questions": "1.\tGeneralisability beyond zebrafish\nCould you add—or at least simulate—experiments on a different specimen (e.g., mouse brain slice or a thicker adult zebrafish) to show that the gains hold outside the current benchmark?\nScore up if cross-domain PSNR/SSIM or qualitative slices confirm similar improvements; score down if performance drops sharply without a clear explanation.\n2.\tRuntime, throughput, and memory footprint\nPlease report FPS and peak GPU memory for one full-resolution volume on a single A100 and compare with the strongest baseline.\nShowing that XLFM-Former meets real-time (>30 fps) or near-real-time speeds with ≤80 GB will raise confidence; opaque or unfavourable numbers will lower it.\n3.\tMasked-view vs. standard MAE baseline\nInclude a control experiment where an off-the-shelf pixel-patch MAE (ViT-MAE) is pre-trained on the same data and fine-tuned identically.\nIf MVM-LF still wins by a clear margin, originality concerns are alleviated; if the gap is marginal, the novelty claim weakens.\n4.\tRobustness to PSF mis-calibration\nDemonstrate reconstruction quality when the forward PSF used in ORC is intentionally perturbed (e.g., ±10 % axial FWHM, mild aberration).\nStable performance would validate the physics-loss design; sensitivity would suggest the need for adaptive PSF estimation.\n5.\tReproducibility details\nProvide (in supplementary material or repo) the exact decoder architecture diagram, LR schedule, and data-normalisation script that yield the 54 dB result.", "limitations": "Mostly yes: the manuscript candidly notes (i) restriction to larval zebrafish and (ii) reliance on a calibrated PSF.\nTwo points that could still be added:\n1.\tComputational cost / carbon footprint of the 1 k-GPU-hour pre-training stage.\n2.\tData-privacy implications if the approach is transferred to vertebrate animal or clinical specimens (e.g., brain-activity maps).", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n- The pipeline is carefully decomposed into (i) self-supervised pre-training (MVM-LF) and (ii) physics-consistent fine-tuning (ORC loss). Extensive ablations isolate the gain from each component and show a clear ∼35 % MSE reduction over a strong scratch baseline. \n- The paper is well-structured; figures 1 – 3 cleanly illustrate the view-masking idea and the physics loop, and the benchmark release is clearly documented.\n- Provides the first large-scale public XLFM dataset with a community evaluation protocol—likely to standardise future work in this niche. The physics-guided loss adds a reproducible recipe for combining differentiable optics with transformers.\n\nWeaknesses\n- Scope of evaluation is narrow: all quantitative tests are on zebrafish larvae; no cross-species or out-of-distribution studies. Runtime/throughput and GPU memory footprints—critical for high-speed microscopy—are omitted, so practical efficiency is unclear.\n- Several implementation details are under-specified (decoder up-sampling stages, PSF calibration procedure, learning-rate schedule). Readers cannot reproduce the exact 54 dB result without digging into supplementary code.\n- The two core ideas (masked prediction, differentiable rendering) are themselves well-known in vision/graphics. Their combination is novel for XLFM, but may be judged more as thoughtful integration than fundamental algorithmic innovation.", "was_revised": true, "final_justification": "Thank you for the substantial rebuttal and the careful new experiments. I appreciate the clear effort the authors put into addressing prior concerns—these additions materially strengthen the paper.\n\nWhat improved\n•\tAdded cross-strain (zebrafish) evaluation, a ViT-MAE control, and runtime benchmarks; clarity and positioning are better.\n•\tReported >30 FPS on 600×600×300 volumes on a single GPU and a comparison indicating much slower classical deconvolution, supporting real-time use.\n\nWhat still limits a higher score\n•\tPlease provide matched, hardware-controlled speed/memory comparisons (same GPU, volume size, stopping criteria) against strong classical baselines, plus peak GPU memory and a transparent benchmarking protocol.\n•\tBiological scope remains zebrafish-only; PSF/aberration mis-calibration robustness is not directly tested.\n\nDecision. Merits now outweigh remaining gaps; I keep Borderline Accept (4).\nIf accepted (camera-ready): include the full runtime/memory table with matched configs, explicit PSF/aberration stress tests, and documentation of the protocol supporting the real-time claim.", "rebuttal": "### Comment 1:\n\n> Generalisability beyond zebrafish Could you add—or at least simulate—experiments on a different specimen (e.g., mouse brain slice or a thicker adult zebrafish) to show that the gains hold outside the current benchmark? Score up if cross-domain PSNR/SSIM or qualitative slices confirm similar improvements; score down if performance drops sharply without a clear explanation.\n\n### Authors’ response:\n\n**Summary.** We validated generalization using a new dataset from a distinct zebrafish strain (H2B-NemoS); our method showed strong cross-strain and OOD performance, confirming robustness to physiological and imaging variations.\n\n\n\nWe appreciate the reviewer’s suggestion to evaluate the generalizability of our method. While publicly available XLFM datasets beyond zebrafish are currently lacking, we took an important step by introducing a new dataset—XLFM-H2B-NemoS—captured using a genetically distinct zebrafish strain and under varied stimulation protocols.\n\nSpecifically, our original dataset is based on the H2B-G8S strain [1], which expresses a fast-responding GCaMP calcium indicator optimized for high signal-to-noise imaging in freely behaving animals. In contrast, the new H2B-NemoS dataset is based on an indicator engineered for larger dynamic range and higher sensitivity to weak signals [2], introducing more sparse and heterogeneous neural activity along with higher background noise. This substantial difference in imaging properties and biological characteristics makes the two datasets effectively distinct in terms of signal distribution, sample noise, and dynamic response profiles.\n\nTo simulate domain shift, we collected 4 zebrafish samples from H2B-NemoS under four levels of electrical stimulation (5V, 10V, 15V, 20V), each producing ~70 volumetric frames at 10 fps. The 10V and 15V data were used for training and in-distribution validation, while 5V and 20V data were held out for out-of-distribution (OOD) testing. Apart from a minor increase in PSF regularization weight (from 1e-4 to 5e-3 to adapt to the brighter dynamics), all network and training settings were kept unchanged.\n\nOur method demonstrated strong generalization across this challenging domain shift. As shown in Tables 1 and 2, it outperformed all baselines with clear margins on PSNR and SSIM. Moreover, the zero-shot version of XLFM-Former (trained only on the G8S and tested directly on H2B-NemoS) still delivered 53.72 dB PSNR, indicating impressive robustness to cross-strain variation.\n\nThese results validate that our method generalizes well to specimens with distinct physiological properties and imaging conditions, and support its broader applicability to future XLFM data beyond the current benchmark.\n\n\n\n**Table 1: Ablation Study on the H2B-Nemos Dataset**\n\n| Method          | PSNR ↑    | ΔPSNR ↑   | SSIM ↑     | ΔSSIM ↑     |\n| --------------- | --------- | --------- | ---------- | ----------- |\n| Baseline        | 50.87     | –         | 0.9903     | –           |\n| + MVM-LF only   | 51.66     | +0.79     | 0.9940     | +0.0037     |\n| + ORC Loss only | 51.77     | +0.90     | 0.9935     | +0.0032     |\n| **Full (Ours)** | **52.34** | **+1.47** | **0.9955** | **+0.0052** |\n\n\n\n**Table 2: Comparison with Existing Methods on the H2B-Nemos Dataset (baseline: ResNet-101)**\n\n| Method                        | PSNR ↑ | SSIM ↑ | Δ PSNR vs. ResNet-101 ↑ | Δ SSIM vs. ResNet-101 ↑ |\n| ----------------------------- | ------ | ------ | ----------------------- | ----------------------- |\n| EfficientNet                  | 49.01  | 0.9826 | –2.42                   | –0.0118                 |\n| ViT-tiny                      | 50.87  | 0.9904 | –0.55                   | –0.0041                 |\n| PVT-tiny                      | 50.89  | 0.9903 | –0.53                   | –0.0042                 |\n| ConvNeXt-tiny                 | 50.88  | 0.9903 | –0.54                   | –0.0042                 |\n| U-Net                         | 51.00  | 0.9910 | –0.42                   | –0.0035                 |\n| ResNet-50                     | 51.34  | 0.9931 | –0.09                   | –0.0014                 |\n| ResNet-101 (baseline)         | 51.42  | 0.9945 | –                       | –                       |\n| **XLFM-Former (Ours — Full)** | 52.34  | 0.9955 | +0.92                   | +0.0010                 |\n| **XLFM-Former (Zero-Shot)**   | 53.72  | 0.9930 | +2.29                   | –0.0015                 |\n\n\n\nReferences:\n[1]Zhang, Y., Rózsa, M., Liang, Y., Bushey, D., Wei, Z., Zheng, J., ... & Looger, L. L. (2023). Fast and sensitive GCaMP calcium indicators for imaging neural populations. *Nature*, *615*(7954), 884-891.\n[2]Li, J., Shang, Z., Chen, J. H., Gu, W., Yao, L., Yang, X., ... & Wang, Y. (2023). Engineering of NEMO as calcium indicators with large dynamics and high sensitivity. *Nature methods*, *20*(6), 918-924.\n\n\n\n\n\n### Comment 2:\n\n> Runtime, throughput, and memory footprint Please report FPS and peak GPU memory for one full-resolution volume on a single A100 and compare with the strongest baseline. Showing that XLFM-Former meets real-time (>30 fps) or near-real-time speeds with ≤80 GB will raise confidence; opaque or unfavourable numbers will lower it.\n\n### Authors’ response:\n\nWe appreciate the reviewer’s concern regarding the practical efficiency of our method. To address this, we benchmarked XLFM-Former against four widely used baselines on a single A100 GPU (80 GB), reporting inference speed (FPS), peak memory usage, and reconstruction accuracy (PSNR).\n\nAs shown below, XLFM-Former achieves 48.44 FPS, well above the 30 FPS real-time threshold, while maintaining competitive memory usage (2631 MiB) and the highest reconstruction fidelity (PSNR 52.34 dB) among all compared methods. These results demonstrate that our approach is not only accurate but also efficient enough for real-time 3D microscopy applications.\n\n### **Inference Efficiency and Reconstruction Accuracy on H2B-Nemos Dataset (Single A100 GPU)**\n\n| Method             | FPS ↑ | Peak Memory ↓ (MiB) | PSNR ↑ |\n| ------------------ | ----- | ------------------- | ------ |\n| ConvNeXt           | 98.56 | 2565                | 50.88  |\n| ViT                | 94.79 | 2875                | 50.87  |\n| U-Net              | 74.34 | 2423                | 51.00  |\n| ResNet-50          | 41.34 | 6467                | 51.34  |\n| XLFM-Former (Ours) | 48.44 | 2631                | 52.34  |\n\n\n\n### Comment 3:\n\n> Masked-view vs. standard MAE baseline Include a control experiment where an off-the-shelf pixel-patch MAE (ViT-MAE) is pre-trained on the same data and fine-tuned identically. If MVM-LF still wins by a clear margin, originality concerns are alleviated; if the gap is marginal, the novelty claim weakens.\n\n### Authors’ response:\n\nWe thank the reviewer for the valuable suggestion. To assess the effectiveness and originality of our proposed Masked View Modeling for Light Fields (MVM-LF), we conducted a control experiment using a standard ViT-based pixel-patch MAE [He et al., 2022] pre-trained on the same dataset (∼20k unlabelled frames) and fine-tuned with the same downstream setup.\n\nThe results are as follows:\n\nViT-MAE: PSNR = 46.55, SSIM = 0.9752\n\nMVM-LF (Ours): PSNR = 54.04, SSIM = 0.9944\n\nThis substantial performance gap highlights that our MVM-LF design, which incorporates prior knowledge of the multi-view structure of XLFM (e.g., angular correlations), is significantly more effective than standard MAE when operating on limited-scale microscopy data. We believe this validates both the originality and practical value of MVM-LF in the context of structured light field inputs.\n\n\n\n### \n\n\n\n### Comment 4:\n\n> Robustness to PSF mis-calibration Demonstrate reconstruction quality when the forward PSF used in ORC is intentionally perturbed (e.g., ±10 % axial FWHM, mild aberration). Stable performance would validate the physics-loss design; sensitivity would suggest the need for adaptive PSF estimation.\n\n### Authors’ response:\n\nThese results suggest that the reconstruction performance of our ORC loss is largely stable under ±10% perturbations in the axial full-width at half-maximum (FWHM) of the PSF. Specifically, the PSNR fluctuates within ±0.12 dB and SSIM remains within ±0.0002, indicating that the ORC loss is not overly sensitive to small mismatches in PSF calibration.\n\nThis robustness arises from the fact that the forward projection consistency enforced by the ORC loss penalizes global mismatches between reconstructed and measured views, while allowing for small inaccuracies in the PSF shape. We believe this property is crucial for real-world applicability, where slight PSF variations are inevitable due to experimental imperfections.\n\nMoreover, these findings suggest that our method does not require exact PSF estimation and can tolerate mild miscalibration, reducing the burden on the calibration process in practical deployments.\n\n**Robustness of ORC Loss to PSF Perturbation on H2B-Nemos Dataset**.\n\n| PSF Setting   | PSNR ↑  | ΔPSNR ↑ | SSIM ↑ | ΔSSIM ↑ |\n| ------------- | ------- | ------- | ------ | ------- |\n| Baseline      | 52.3440 | –       | 0.9955 | –       |\n| PSF +10% FWHM | 52.3369 | –0.0071 | 0.9953 | –0.0002 |\n| PSF –10% FWHM | 52.4647 | +0.1207 | 0.9957 | +0.0002 |\n\n\n\n\n\n### Comment 5:\n\n> Reproducibility details Provide (in supplementary material or repo) the exact decoder architecture diagram, LR schedule, and data-normalisation script that yield the 54 dB result.\n\n### Authors’ response:\n\nThank you for raising this important point. Due to the NeurIPS 2025 rebuttal policy, we are unable to include external links in our response. However, we will submit an anonymized link containing the full training code (including the decoder architecture diagram, learning rate schedule, and data normalization pipeline) to the Area Chair. If you are interested, please feel free to request it via the AC. We apologize for this inconvenience and appreciate your understanding."}, {"review_id": "ZuKfKgK0fU", "reviewer": "Reviewer_2sPe", "summary": "This work deals with the 3D reconstruction of eXtended Light field microscopy(XLFM) data. The authors argue that the traditional way of volumetric annotations is computationally intensive (Richardson–Lucy deconvolution) and so they bring in a self-supervised component to their work (MVM-LF). To enforce predictions to adhere to the forward model of the image formation process, ORC loss is introduced. This work introduces a dataset, which will be made public, on which all experiments are conducted.  A sufficient number of ablation studies have been conducted (for ORC loss, MVM-LF, masking ratio, etc), which justifies the presence of individual components of the training setup.", "questions": "1. At Line 230, the authors mention that the batch size is set to 1 and mention that it aligns with the requirements of the volumetric task. How does batch size of 1 align with the requirements of the volumetric task? \n2. In eq 5, it has not been explained how the network f() is able to take in a dynamic number of views as input. Also, what is the shape of a view tensor Ui? \n3. At line 66, the authors say that Richardson-Lucy deconvolution should not be used because of being computationally intensive. It will be helpful to know quantitatively how much the computational requirements are. For example, a statement like ‘using it on a XxYxZ sized data takes A amount of time with B GPU/CPU specs’ can be helpful.  \n4. There are a few typos in Lines 254-261 (5070%, 2030%). \n5. Not a question, but rather a minor suggestion for Fig. 6b: I think it would make this work stronger if you had another entry for 1% labelled data. In that case, one expects to see a relatively increased gap between the two plots.\n6. In Line 287, the phrase 'confirming strong generalization' was used. Since 'strong' cannot be justified quantifiably, it might make sense not to use such additional adjectives.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "## Stengths:\n\n1. Ablations have been done in a comprehensive manner. \n2. The manuscript reads well. \n3. XLFM is a relatively new field, and so, contributions of this work will have more impact to this field, and in this sense, this work is valuable. \n4. A new dataset will be made publicly available. \n\n## Weakness.\n\n1. Relevant baselines are missing. At line 117, paper refers to 3 works [25,30,9] which work in this field. However, none of them are present as a baseline in Table 1. The work does mention that those methods reconstruct only neural activity, but this work also reconstructs entire volumetric structures. However, the baselines mentioned in Table 1 were also not developed for XLFM 3D reconstruction task, but the authors did the work to adapt them to this task. So, I donot see why such an effort was not made for any of these three works, given the fact that they are much closer to the domain than the used baselines. Also, since richardson-lucy is used for 3D reconstruction in light field microscopy, it is expected that that should be present as a baseline, arguably along with a quantitative estimate of its compute requirements (the relevant ones from time, gpu, cpu or RAM).  \n2. The authors themselves mention that only one dataset, that has been generated by them, was used for the experiments. The authors should explain the reason: Are there no other datasets which could be adapted to this task? In such a case, even simulated data will hold value, in my opinion. \n3. the loss function for XLFM reconstruction tasks (equation 7) arguably has redundancies. It has a loss component using PSNR as well as using MSE. Since PSNR internally uses MSE, it is not clear why both are needed. Indeed the analysis done in the supplement shows that the PSNR loss component provides limited benefit. The other issue with having five loss components is that one has 5 hyper-parameters, and then it is not clear whether the same hyperparameter set can be used for other datasets. Since there are no other datasets on which the same hyperparameter set has been tested, this becomes a bit more problematic.", "was_revised": true, "final_justification": "I am satisfied with the utility of this method even when classical approaches like RL outperforms their method. Also, the hyper-parameter ORC weight seems rather stable.\n\nI noticed that other reviewers have pointed out that the PSNR values are high, and yet the performance is poor. However, since the comparisons are being made between different baseline methods, this should not be a concern. In fact, a better prediction would naturally result in a higher PSNR.\n\nRegarding PSNR in the loss, PSNR and MSE have been around since ages. PSNR internally computes MSE. However, this is the first work I have seen which uses both, and that too without any convincing intuitive reason. I would prefer if only one of the two is used in the loss.", "rebuttal": "### Comment 1.1:\n\n> Relevant baselines are missing. At line 117, paper refers to 3 works [25,30,9] which work in this field. However, none of them are present as a baseline in Table 1. The work does mention that those methods reconstruct only neural activity, but this work also reconstructs entire volumetric structures. However, the baselines mentioned in Table 1 were also not developed for XLFM 3D reconstruction task, but the authors did the work to adapt them to this task. So, I donot see why such an effort was not made for any of these three works, given the fact that they are much closer to the domain than the used baselines.\n\n**Summary.** While Refs. [9, 25, 30] are closer to our imaging domain, they fundamentally assume sparse, static settings or require impractical memory, making them inapplicable to our dense, dynamic full-volume reconstruction task. In contrast, our selected model-agnostic baselines enable fair, scalable, and reproducible comparisons.\n\nWe appreciate the reviewer’s thoughtful suggestion to include prior domain-specific methods as baselines. After thorough analysis, we found that these methods—despite their relevance—cannot be meaningfully adapted to our setting due to the following limitations:\n\n- **Refs. 25 & 30** focus exclusively on sparse neural signal extraction under the assumption of fully immobilized specimens. Both adopt the SLNet architecture to extract sparse signals prior to reconstruction and are fundamentally incompatible with our goal of reconstructing dense volumetric tissue. Moreover, they process temporally fused multi-frame inputs, which cannot be applied to our single-frame, dynamic zebrafish recordings.\n- **Ref. 9 (FourierNet)** performs all computation in the Fourier domain and demands excessive GPU memory (>80GB) even for modest output sizes. In our case (300×600×600 volumes), FourierNet is not practically reproducible or scalable, even with aggressive approximations.\n\n\n\n### Comment 1.2:\n\n> Also, since richardson-lucy is used for 3D reconstruction in light field microscopy, it is expected that that should be present as a baseline, arguably along with a quantitative estimate of its compute requirements (the relevant ones from time, gpu, cpu or RAM).\n\n### Authors’ response:\n\n**Summary.** RLD achieves higher PSNR but is prohibitively slow and memory-intensive; our method offers a practical tradeoff, enabling real-time, scalable 3D reconstruction essential for biological applications.\n\n\n\nWe appreciate the reviewer’s suggestion to compare against classical iterative deconvolution algorithms. As shown below, we benchmarked RLD at different iteration counts (20/30/40) on the challenging H2B-Nemos dataset. While (Richardson-Lucy Deconvolution)RLD can achieve high PSNR/SSIM given sufficient iterations, it incurs orders of magnitude higher memory usage and lower speed, making it impractical for real-time or high-throughput applications.\n\n| Method                 | PSNR ↑ | SSIM ↑ | FPS ↑     | Peak Memory ↓ (MiB) |\n| ---------------------- | ------ | ------ | --------- | ------------------- |\n| RLD-20                 | 67.36  | 0.9987 | 0.068     | 20899               |\n| RLD-30                 | 69.85  | 0.9994 | 0.046     | 20899               |\n| RLD-40                 | 72.31  | 0.9998 | 0.035     | 20899               |\n| **XLFM-Former (Ours)** | 52.34  | 0.9955 | **48.44** | **2631**            |\n\n\n\n\n\n### Comment 2:\n\n> The authors themselves mention that only one dataset, that has been generated by them, was used for the experiments. The authors should explain the reason: Are there no other datasets which could be adapted to this task? In such a case, even simulated data will hold value, in my opinion.\n\n### Authors’ response:\n\nWe appreciate the reviewer’s suggestion about including additional datasets, including simulated ones, to strengthen the evaluation. Since the submission, we have extended our experiments to a newly collected dataset—H2B-Nemos, which involves a different zebrafish line under distinct experimental conditions. This allows us to directly test the generalizability and robustness of our method.\n\nAs shown below, we re-run all methods under identical settings using ResNet-101 as a common baseline, and observe consistent performance advantages of our XLFM-Former. Notably, our method achieves +0.92 dB PSNR gain over ResNet-101 in the supervised setting, and a +2.29 dB gain in the zero-shot setting, further demonstrating its cross-domain generalization.\n\n\n\n**Comparison on the H2B-Nemos Dataset (baseline: ResNet-101) **\n\n| Method                        | PSNR ↑ | SSIM ↑ | Δ PSNR vs. ResNet-101 ↑ | Δ SSIM vs. ResNet-101 ↑ |\n| ----------------------------- | ------ | ------ | ----------------------- | ----------------------- |\n| EfficientNet                  | 49.01  | 0.9826 | –2.42                   | –0.0118                 |\n| ViT-tiny                      | 50.87  | 0.9904 | –0.55                   | –0.0041                 |\n| PVT-tiny                      | 50.89  | 0.9903 | –0.53                   | –0.0042                 |\n| ConvNeXt-tiny                 | 50.88  | 0.9903 | –0.54                   | –0.0042                 |\n| U-Net                         | 51.00  | 0.9910 | –0.42                   | –0.0035                 |\n| ResNet-50                     | 51.34  | 0.9931 | –0.09                   | –0.0014                 |\n| **ResNet-101 (baseline)**     | 51.42  | 0.9945 | –                       | –                       |\n| **XLFM-Former (Ours – Full)** | 52.34  | 0.9955 | +0.92                   | +0.0010                 |\n| **XLFM-Former (Zero-Shot)**   | 53.72  | 0.9930 | +2.29                   | –0.0015                 |\n\nWe will include this dataset and evaluation in the final version of the paper and commit to releasing both datasets (G8S and Nemos) publicly under the supervision of the AC.\n\n\n\n### Comment 3.1:\n\n> The loss function for XLFM reconstruction tasks (equation 7) arguably has redundancies. It has a loss component using PSNR as well as using MSE. Since PSNR internally uses MSE, it is not clear why both are needed. Indeed the analysis done in the supplement shows that the PSNR loss component provides limited benefit. \n\n### Authors’ response:\n\n\n\n**Summary.** Despite being related, PSNR and MSE offer complementary benefits—our ablation shows that including both improves fidelity and contrast, supporting their joint use for high-quality reconstruction.\n\nWe thank the reviewer for the thoughtful comment. While it is true that PSNR is derived from MSE, we argue that including both explicit PSNR and MSE losses encourages the model to balance overall numerical accuracy (via MSE) with perceptual contrast and dynamic range (via PSNR).\n\nImportantly, our ablation in Table 4 shows that removing the PSNR loss results in a noticeable drop in PSNR (from 54.04 to 53.07 dB) and SSIM (from 0.9944 to 0.9935), which demonstrates non-negligible improvement.\n\n\n\n### Comment 3.2:\n\n> The other issue with having five loss components is that one has 5 hyper-parameters, and then it is not clear whether the same hyperparameter set can be used for other datasets. Since there are no other datasets on which the same hyperparameter set has been tested, this becomes a bit more problematic.\n\nAuthors’ response:\n\n**Summary.**  Our loss configuration generalizes well across datasets with only minimal tuning, as confirmed by strong performance on H2B-Nemos in new ablation results.\n\n\n\nWe thank the reviewer for raising this important concern regarding the generalizability of our loss configuration across datasets.\n\nTo verify this, we conducted additional experiments on the H2B-Nemos dataset, using the same loss configuration as in our main experiments on the original dataset. We only slightly adjusted the ORC loss weight (from 1e-4 to 3e-5) to account for minor noise differences; all other hyperparameters were kept unchanged.\n\nAs shown in the table, the results remain consistently strong:\n\n**Ablation Study on the H2B-Nemos Dataset. **\n\n| Method          | PSNR ↑    | ΔPSNR ↑   | SSIM ↑     | ΔSSIM ↑     |\n| --------------- | --------- | --------- | ---------- | ----------- |\n| Baseline        | 50.87     | –         | 0.9903     | –           |\n| + MVM-LF only   | 51.66     | +0.79     | 0.9940     | +0.0037     |\n| + ORC Loss only | 51.77     | +0.90     | 0.9935     | +0.0032     |\n| **Full (Ours)** | **52.34** | **+1.47** | **0.9955** | **+0.0052** |\n\n\n\n\n\n### Comment 4:\n\n> At Line 230, the authors mention that the batch size is set to 1 and mention that it aligns with the requirements of the volumetric task. How does batch size of 1 align with the requirements of the volumetric task?\n\n### Authors’ response:\n\nThank you for pointing this out. Due to the large size of each reconstructed 3D volume (up to 300×600×600) and the complexity of our Swin-Transformer-based architecture, the peak memory usage approaches 79GB on an A100 GPU, leaving little room for increasing the batch size. Therefore, we set the batch size to 1 to ensure stable training and inference under these memory constraints.\n\n\n\n\n\n### Comment 5:\n\n> At line 66, the authors say that Richardson-Lucy deconvolution should not be used because of being computationally intensive. It will be helpful to know quantitatively how much the computational requirements are. For example, a statement like ‘using it on a XxYxZ sized data takes A amount of time with B GPU/CPU specs’ can be helpful.\n\n### Authors’ response:\n\nWe appreciate the reviewer’s suggestion and now provide quantitative justification. Applying Richardson-Lucy deconvolution (RLD) on a single 600×600×300 volume takes over 42 seconds for 60 iterations and consumes more than 20 GiB of GPU memory, even on an A100 GPU (80GB)."}]

Author rebuttal:

Meta-review: (a) This paper introduces a method for volumetric reconstruction from 2D light field data (with the XLFM architecture).
A general benefit of such LF methods is the data capture efficiency, which, combined with the use of deep learning models,
allows an even faster 2D to 3D data processing. The authors employ a different model architecture for this task (the Swin Transformer) 
and introduce a new training procedure (masking and view completing for pre-training + an accurate and differentiable image formation model) that yields sota performance.

(b) The paper introduces a curated dataset, which can be beneficial to research in this domain (although the ground truth is obtained
through another algorithm rather than through a direct measurement), and a novel training method (although it uses ideas from related 
domains -- but has never been applied to LF data). Finally, its performance is sota, although there were issues with the evaluation procedure (an unusually high PSNR).

(c) The reviewers raised several concerns: 1) The presentation of the paper is lacking in several aspects; 2) There are missing comparisons with competing methods; 3) The experimental results are unusually high and not well-justified; 4) The proposed training has several components borrowed from vision and computer graphics.

(d) Despite all the concerns, the authors managed to clarify the majority of the concerns during the discussion with the reviewers; also, in general, the overall contribution is found to be sufficiently significant in terms of solution design (the combination of architecture, loss and pre-training procedure) and is experimentally validated by a significant performance. In fact, most reviewers were convinced by the replies and raised their scores.

(e) The discussion between the reviewers and authors revolved around the concerns already mentioned above. The authors provided explanations for the missing technical details and better clarified the experiments; they also showed new results under the ablations and settings that the reviewers mentioned as missing or unclear. The reviewers were mostly satisfied with the replies and raised their scores.

In conclusion, this paper is technically solid, with meaningful dataset and methodological contributions, but is held back by incomplete evaluation and presentation clarity.
Decision: Accept (poster)