Information-Driven Design of Imaging Systems
Authors: Henry Pinkard
Leyla A Kabuli
Eric Markley
Tiffany Chien
Jiantao Jiao
Laura Waller
Venue: NeurIPS 2025 poster ()
Keywords: Mutual information estimation
Computational imaging
Information theory
entropy estimation
Encoder and sensor design
Imaging system evaluation
computational photography
Microscopy
Astronomy
Primary area: machine_learning_for_sciences
TL;DR: A universal method to evaluate and optimize any imaging system using only noisy measurements of unknown objects, enabling efficient design optimization and evaluation of real systems where traditional approaches cannot be applied.

Abstract: Imaging systems have traditionally been designed to mimic the human eye and produce visually interpretable measurements. Modern imaging systems, however, process raw measurements computationally before or instead of human viewing. As a result, the information content of raw measurements matters more than their visual interpretability. Despite the importance of measurement information content, current approaches for evaluating imaging system performance do not quantify it: they instead either use alternative metrics that assess specific aspects of measurement quality or assess measurements indirectly with performance on secondary tasks. 

We developed the theoretical foundations and a practical method to directly quantify mutual information between noisy measurements and unknown objects. By fitting probabilistic models to measurements and their noise characteristics, our method estimates information by upper bounding its true value. By applying gradient-based optimization to these estimates, we also developed a technique for designing imaging systems called Information-Driven Encoder Analysis Learning (IDEAL). 

Our information estimates accurately captured system performance differences across four imaging domains (color photography, radio astronomy, lensless imaging, and microscopy). Systems designed with IDEAL matched the performance of those designed with end-to-end optimization, the prevailing approach that jointly optimizes hardware and image processing algorithms. These results establish mutual information as a universal performance metric for imaging systems that enables both computationally efficient design optimization and evaluation in real-world conditions.

A video summary of this work can be found at: https://waller-lab.github.io/EncodingInformationWebsite/

Information-Driven Design of Imaging Systems
Henry Pinkard
Leyla Kabuli
Eric Markley
Tiffany Chien
Jiantao Jiao
Laura Waller
Department of Electrical Engineering and Computer Sciences, University of California, Berkeley
Abstract
Imaging systems have traditionally been designed mimic the human eye and produce visually interpretable measurements, Modern imaging systems, however, process raw measurements computationally before instead of human viewing. As a result, the information content of raw measurements matters more than their visual interpretability. Despite the importance of measurement information content, current approaches for evaluating imaging system performance do not quantify it: they instead either use alternative metrics that assess specific aspects measure- ment quality or assess measurements indirectly with performance on secondary tasks. We developed the theoretical foundations and a practical method to directly quantify mutual information between noisy measurements and unknown objects. By fitting probabilistic models to mcasurements and their noise characteristics, our method estimates information by upper bounding its true value. By applying gradient-bar based optimization to these estimates, we also developed a technique for designing imaging systems called Information-Drivee Encoder Analysis Learn- ing (IDEAL). Our information estimates accurately captured system performance differences across four imaging domains (color photography, radio astronomy, lensless imaging, and microscopy). Systems designed with IDEAL matched the performance of those designed with end-to-end optimization, the prevailing ap- proach that jointly optimizes hardware and image processing algorithms. These results establish mutual information universal performance metric for imag- ing systems that enables both computationally efficient design optimization and evaluation in real-world conditions.
A video summary of this work can be found at:
https://wallee lab. lib.tuhhuu....iiuub...........
Introduction
The increasing utilization computational processing in imaging systems removes the constraint that they must produce human-interpretable measurements. Traditional imaging system designs produce measurements that contain the specific visual patterns recognizable to human observers [1]. However, these patterns unnecessary for modern algorithms like deep neural networks, which can utilize information in measurements regardless of how appears: they can either transform measurements into forms that make sense humans, computational imaging systems do or they analyze measurements directly and bypass humans altogether.
This shift fundamentally changes what limits performance: not how well systems encode information into human- perceptible patterns, but how much information they encode, regardless of its visual appearance.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Existing metrics to quantify hardware performance fail to capture information content comprehen- sively. Traditional measurement metrics like resolution, signal-to-noise ratio, sampling rate, and field-of-view characterize individual system aspects separately, making it difficult to assess their combined effect on overall performance or compare systems that make different trade-offs among these factors.
As result, performance is often assessed by computationally decoding measurements to perform secondary tasks. This decoder-basee approach uses tasks like image reconstruction or object clas- sification. It quantifies success by comparing algorithm outputs against ground truth using metrics like mean squared error or classification accuracy 4 This approach furthermore enables system design in simulation through "end-to-end" methods that jointly optimize both encoder parameters (like lens shapes or sensor configurations) and decoder algorithms (like image reconstruction). [M
However, decoder-based evaluation and end-to-end optimization have few limitations. First, decoder- based evaluation metrics require full ground truth knowledge of the object or some of its properties, which limits their application simulations and laboratory settings that may not generalize to real- world conditions. Second, this joint evaluation conflates encoder and decoder performance, since low performance could result from either insufficient encoded information or the decoder's 's failure to utilize it. Finally, neural network- decoders require substantial compute and may create backpropagation challenges in end-to-ent optimization
12
Directly quantifying information in mcasurements provides an alternative approach that avoids the limitations of decoders. Mutual information quantifies how much object information survives the encoding and measurement processes [33 37 3. Equivalently, it quantifies how much a measurement
reduces observer's uncertainty about the object.
Mutual information also unifies traditionally separate aspects of measurement quality, enabling mean- ingful comparison between systems that make different trade-off's among these factors (Sec. (S1.2). For example, it captures the combined effect of resolution, signal-to-noise ratio, sampling, and quantization information preservation [17 25 38)
Despite these benefits, practical estimation of mutual information has been challenging. Previous approaches fall into two categories, each with significant limitations.
The first approach overestimates information because it models imaging system as unconstrained communication system 13. 15 18
23 28 37). While communication systems assume encoders can transform any input into any output without physical constraints, imaging systems cannot achieve this because lenses, sensors, and other hardware impose fundamental limitations. These theoretical calculations can therefore produce highly inaccurate estimates (Fig. (S6).
The second approach explicitly models the objects being imaged, relying on assumptions about object properties that are difficult to verify in practice [5). These object models must also be developed case-by-case for each object type, limiting their generality.
To address these limitations, we developed framework for estimating mutual information directly from measurement of unknown objects. Our approach requires only two inputs: a dataset of measurements and a model of the system's noise. By fitting models to actual measurements rather than assuming object properties or ignoring encoders' physical constraints, the method readily adapts to diverse imaging modalities and performance trade-offs. Our framework enables computationally efficient optimization when designing systems through simulation, as well as quantitative evaluation of real imaging systems operating in the field.
This paper is organized follows: Section presents the theoretical framework for estimating mutual information from measurements. Section describes implementation through three probabilistic models and validates their accuracy. Section 4 demonstrates that information estimates predict system performance across four imaging applications: color photography, radio astronomy, lensless imaging, and microscopy. Section 55) introduces our method for automated imaging system design through gradient-based optimization, Information-Driver Encoder Analysis Learning (IDEAL). Section 6 considers implications and future research directions.
2

---

2 Mathematical framework
This section shows how to overcome the inherent statistical and computational challenges of mutual information estimation by progressively decomposing the estimation problem using key properties of imaging systems. First, deterministic encoding makes it possible to focus the relationship between clean images and noisy measurements rather than the more complex abject-measuremeet relationship. Second, the known statistical structure of noise in imaging systems enables decomposing the estima- tion problem into two simpler subproblems. Finally, the constraint of only having measurement data (on real imaging systems) informs the selection of estimation approaches for each subproblem.
To formalize this approach, we model imaging systems as a chain probability distributions (Fig, []. Consider photographing trees: probability distribution shape, and species. An encoder (a camera with particular lens and position) maps each possible tree noiseless image the sensor. Detection noise corrupts these ideal images, creating the measurements that sensors actually record 13
objects captures how trees vary in size,
(Sec. ST.
Mutual information provides a metric for comparing encoder designs by quantifying how well measurements can distinguish different objects. Information losses occur at two stages: during encoding (e.g. from finite resolution sampling) and during measurement (from noise).
Unknown objects
Images
Measurements
Noise
Y
Encoder
Mutual information estimate
Information estimator
(x; Y)
Noise model
Evaluation + design
Figure 1: Information estimation for imaging systems. encoder (e.g., an optical system) maps unknown objects to noiseless images X. Noise corrupts these images to produce measurements Y. The information estimator uses these measurements and a noise model to quantify how well measurements distinguish noiseless images (and thus objects under deterministic encoding). The estimate f(;; Y) enables both encoder evaluation and design optimization.
Mathematically, model images and measurements probability distributions vectors. An image with D pixels is vector x (æ1, ø2, ID), where Ią represents the energy at the kth pixel. The probability distribution p(x) p(#1, F2. ........777 Tp)) describes the likelihood of each possible image. Similarly, a noisy measurement is vector = (y1, 32...., .D) with distribution p(y).
Quantifying object information in measurements becomes tractable through the widely-used sumption of deterministic imaging systems. The object- measuremet mutual information I(O; Y) requires working with the joint distribution of objects and measurements, which is unknown in real systems. However, deterministic encoders, which always produce the same noiseless image for each object, enable estimation of 7(X; Y) instead. This deterministic encoder assumption applies broadly in practice and holds by construction in simulated design.
Intuitively, this equality holds because all variations in noiscless images come from variations in objects when encoders are deterministic. Thus, any image information that survives the measurement process must have also come from the object. Mathematically;
Deterministie Information Transfer Theorem. For a Markov chain O - Y, for some deterministic function f, then I(O; Y) = I(X;
(See supplement for proof (Sec. SI.1
Even the more accessible 7(X; Y): estimation problem requires handling the high-dimensional joint distribution p(x, y). Estimating mutual information between high-dimensional variables from

---

samples is notoriously challenging [41:50]. Without strong assumptions, estimation requires sample sizes that grow exponentially with dimensionality, suffers from high bias and variance, and lacks formal guarantees 15 46
Furthermore, the majority of existing estimation methods assume access dataset of paired samples (x(*), y()) While these available in simulated systems, real systems only provide noisy measurements { without corresponding noiseless images.
Fortunately, imaging systems have well-characterized noise processes p(y x) from physical models or calibration 40) This knowledge, combined with the deterministic encoder property, enables decomposing T(X; Y) into entropy terms that can be estimated from noisy measurements alone:
- - H
(1)
HY) quantifies total variation in measurements from both objects and noise. XX quantifies variation from noise alone. Their difference 7(X; Y) isolates noiseless image-induced variation in measurements, which equals the object information preserved when encoders are deterministic.
These entropy terms can be expressed as expectations:
H(Y) = E |- lo (Y))
(2)
H(Y log p( x)]
The H({ term further simplifies under the assumption of independent noise across pixels, reducing the conditional entropy H(Y (X) from high-dimensional integral sum single-pixel
entropies: H XX = ΣP H(Y:(X,) (Sec. S2.3.1 Imaging systems commonly satisfy this
assumption 40
The HY) term requires learning from data. The true distribution p(y) is unknown but its entropy (Y) can nonethcless be upper bounded by fitting a parametric model pa (y) to training set of N measurements {y J and computing the cross-entropy on held-out test set of M additional measurements:
M
H(Y) S pog
1
Σ
log Pa ()())
M
This upper-hounding approach reliably estimates entropy in complex real-world distributions without knowing the true probability [22]
Furthermore, cross-entropy provides a principled way to compare models. Since cross-entropy between true and estimated distributions always exceeds true entropy, models achieving lower cross-entropy values produce more accurate estimates.
Subtracting our estimate of HY( from this upper bound on H(Y) yields our estimate of the mutual information [(x; Y).
Probabilistic models for information estimation
Practical implementation of our framework requires choosing probabilistic models fit both mea- surements and noise processes. For measurements, users face fundamental trade-off: expressive models capture complex distributions accurately but demand substantial data and compute, while simpler models train faster using fewer resources but may overestimate information. To address this trade-off, we developed and tested three models spanning the expressivity-effcieney spectrum: stationary Gaussian process, full Gaussian process, and autoregressive PixelCNN model 55 For noise modeling, users must select an appropriate model and (on real systems) fit its parameters using only noisy measurements.

---

This section details the three measurement models and validates their performance through experi- ments that establish practical guidance for model selection in different scenarios.
Measurement model assumptions. Our models differ in two key assumptions: whether measure- ments follow multivariate Gaussian distribution and whether pixel statisties translation-invariint (stationary). Violated assumptions yield worse fits that overestimate information. For example, Gaus- sian models cannot capture many types of statistical structure that are present in real measurement distributions, such as the bimodal statistics of measurements of MNIST handwritten digits (Fig. (S1)) Meanwhile, stationary models fail with gradients.
artifacts like sensor defects or illumination
We processed images as patches rather than full images for computational efficiency. Larger patches capture more spatial relationships (Fig. (812) but require more computation; smaller patches provide more training samples and permit smaller models.
Model types. We tested three models with different expressivity-effcieney trade-offs.
Table 1: Comparison of probabilistic models for information estimation. Training times measured on 20 x 20 patches using an NVIDIA RTX A6000 GPU. Stationary Gaussian training time range reflects optional optimization for numerical stability.
Model
Data Efficiency
Training Time
Expressivity
Gaussian Stationary
Stationary Gaussian
Highest
0011110s
Lowest
Gaussian (full cov.)
Medium
0.1s
Medium
PixeICNN
Lowest
100ss
Highest
Estimator consistency. We validated convergence behavior on both simulated data with known ground truth and real imaging data. On simulated data from a stationary Gaussian process with additive noise, all three models converged to the analytically-calulaate stationary Gaussian converging fastest since the test data matched its assumptions (Fig. (ig.). On real imaging data, all models converged to stable, though different, values. PixcICNN consistently achieved the lowest entropy bounds, indicating the most accurate estimates, while Gaussian models plateaued at higher values due to their inability to capture non non-Gaussian statistics (Fig. S16p,c). Based on these convergence results, used ~110000-1.000 10.000 patches of ~ 20 x 20 pixels for subsequent experiments, balancing computational efficiency with estimation accuracy.
mutual information, with the
Model selection. Based on our validation results, we recommend selecting models according to available resources and data characteristies. The stationary Gaussian provides reliable estimates with limited data, PixeICNN maximizes accuracy when computational resources and data abundant, and the standard Gaussian balances ease of computation with reliable performance. While PixcICNN generally provides the most accurate estimates, the performance difference may be smaller for data with approximately Gaussian statistics, making simpler Gaussian models reasonable when computational resources are limited. Fitting multiple models and selecting the lowest entropy estimate yields the most reliable results when feasible.
Noise model validation. Noise processes in imaging systems proved far simpler to model than measurement distributions, making H H(Y estimation straightforward. While this required iden- tifying the correct noise model and fitting its parameters, the structure of imaging noise allows the framework to remain robust to somewhat inaccurate noise modeling. Object-independen noise (thermal, dark current) affects all measurements equally, so any model error simply shifts all estimates by the same constant, preserving encoder rankings. Object-dependent noise (shot noise) arises from well-understood quantum processes with established mathematical models, facilitating accurate characterization.
We tested one key challenge specific to shot noise and confirmed our framework handles it success- fully: shot noise parameters depend on the noiseless image that real systems cannot observe. We validated that accurate entropy estimation remained possible using only noisy measurements. Our estimators succeeded across wide range of photon counts, deviating from the true value only below 20 photons per pixel (Fig. S18.
5

---

4 Validation across imaging applications
Theoretical results establish that mutual information limits achievable decoder performance [56-59], but estimates of information inevitably contain error. If information estimates accurately capture what limits real decoder performance, they should correlate with decoder-based evaluation across diverse imaging systems. We tested this relationship by comparing information estimates against decoder performance across four imaging systems spanning different modalities, noise regimes, and tasks, with decoders ranging from classical signal processing to modern neural networks.
Across all four systems, information estimates predicted decoder performance, validating their use for additional settings where decoders face computational challenges or cannot be used because ground truth data is unavailable.
Color photography. Digital cameras encode color on monochrome sensors using color filter arrays in front of their pixels. These arrays filter incoming light so that each pixel (traditionally) detects only red, green, or blue light, and demosaicing algorithms decode the raw measurements to reconstruct full-color images the full sensor resolution. Beyond the traditional Bayer pattern (a 2x2 grid with two green, one red, and one blue pixel) (Fig, Lh, bottom), recent end-to-end learned designs incorporate white (i.e. color filter) pixels together with neural-network--hase decoders 60 61 to improve reconstruction quality.
Our information estimates predicted reconstruction quality across three color filter designs, enabling the evaluation of designs without reconstruction algorithms or ground truth data (Fig. (2)). We tested the traditional Bayer pattern, random arrangement of red, green, blue, and white filters, and a learned arrangement of these filters [61]. Using natural images [62] 663 with simulated photon shot noise, estimated information content for each design and reconstructed full-color images using neural network demosaicing [61]. Higher mutual information consistently predicted better reconstruction across all quality metrics-ment squared error (MSE), peak signal-to-noise ratio (PSNR), and structural similarity index measure (SSIM) (Fig. S20).
Black hole imaging. Radio telescope arrays achieve the angular resolution of an Earth-sized telescope by combining data from global sites, enabling images like the Event Horizon Telescope's s M87 black hole image (64). Selecting optimal telescope locations for next next-generation arrays remains computationally challenging: each site's value depends on all others, and reconstructing images for every possible configuration is compurationally intractable 65) Thus, current approaches use simplified metrics that can miss important imaging features 65
Our information estimates predicted reconstruction quality across telescope array configurations, enabling telescope site selection without computationally expensive image reconstruction (Fig. 2b). Using four-telescope subsets from the original Event Horizon array, we simulated measurements with additive Gaussian noise and compared information estimates to the error in black hole images generated by iterative inverse problem optimization. Higher information estimates consistently produced more accurate reconstructions (Fig. S20).
Lensless imaging. Lensless imagers replace traditional lenses with gigttmooulating masks, offering simple hardware, wide field-of-view, and the ability to encode depth and time information single measurements [66]. Their encoded measurements bear visual resemblance to scenes and vary dramatically between designs, requiring computational reconstruction for both image recovery and performance evaluation.
Our information estimates predicted reconstruction quality across three optical designs, enabling evaluation of unconventional optics without image reconstruction (Fig. (b). We tested traditional lens, random microlens array [67]. and Gaussian diffuser (88 using natural images with simulated photon shot noise at various light levels. Higher information estimates consistently correlated with better reconstruction accuracy across all designs and noise conditions (Fig. S20).
Coded illumination microscopy. LED array microscopy replaces standard illumination with programmable light sources to flexibly generate different contrast modes at low cost by varying angle, intensity, and coherence [69 70 Optimal illumination patterns depend heavily on imaging tasks, making theoretical evaluation difficult and necessitating empirical comparison through mas--specific
decoders 72).

---

Objects
Encoder
Measurements
Decoder
a)
Neural network
Filter
Reconstruction
Scenes
Learned (end-to-ond)
Sensor
Color photography
Random
+
11.5-
Mean squanne eenot
Bayer
Bayer
Rendom
Mutuel information
per pizel)
b)
Black holes
Radio telescopes
Inversion
Reconstruction
oepmii Cip. f(x))
(R
Radio astronomy
Jansky per pical 2x10
Munusl Information
200
(bils)
C)
Phase mask
Point spread function
Deconvolution Reconstruction
lens
Lensiess imaging
Natural images
Arg min AAx -E
Four lens
Posk torallto ross ralo (dB)
Diffuser
per pixel
40
160 320
information (bits poxel)
0.6
Photons 250
Label-free microscopy g
Illumination pattern
Neural
Protein expression prediction
network
Leukocytes
Single LED
Brightfield Differetiall
0.075
contrast
Phase Contrast
Singla
Predidion log
LED
Brightfield
pixol
-0.075
00
100
460
Mutuel information
0.4
Photons 250
(Dits per pixel)
Figure 2: Information estimates predict decoder performance across four imaging applica- tions. Each row shows representative objects, three encoder designs, example measurements. and the relationship between information estimates and a representative decoder performance metric (see Fig, /20 for all metries). a) Color photography: Bayer, learned, and random filter arrays with neural network demosaicing [61] b) Radio astronomy: Three telescope array configurations with inverse problem reconstruction. e) Lensless imaging: Lens, microlens array, and diffuser with Wiener deconvolution. d) LED array microscopy: Brightfield, differential phase contrast, and single-LED illumination with neural network protein expression prediction.

---

Our information estimates correlated with protein prediction accuracy across three illumination patterns, enabling evaluation of microscopy designs without time-consuming and expensive protein labeling experiments (Fig. D We tested brightfield, differential phase contrast 73 74], and single-LED illumination measurements of white blood cells [75) with added simulated photon shot noise to equalize photon counts. Higher information estimates generally predicted better neural network classification performance, though single-LED showed inflated estimates due to optical imperfections (Sec. (..1).
4.1 Task-specific information
While information estimates strongly predict reconstruction performance, they may correlate less strongly with specialized tasks that rely only on specific features of measurements. Systems that preserve specific lask-relevan information can outperform those capturing more total information indiscriminately 76].
Our experiments confirmed that specialized tasks show weaker correlations between information estimates and decoder performance. We tested this by switching from reconstruction classifi- cation task for the lensless imaging system (Fig. S22 This classification task uses only ~ % of the information in measurements (Sec. S4.2 We found that designs with similar total information achieved different classification accuracies, likely reflecting different amounts of task-specific infor- mation captured (Fig. (S0). similar pattern may explain the mieroscopy results (Fig. 2 where single-LED illumination yielded less accurate protein predictions than differential phase contrast despite higher information estimates.
These findings motivate extensions that optimize task-specific rather than total information. By decomposing information into task-specific and task-irrelevant components (T(Y; 0) I(;TT) O|T)) future work could target specialized tasks while maintaining decodeer-nndppendnnt evaluation (Sec. $2.8
Encoder design via information maximization with IDEAL
Having validated that information estimates can assess encoder quality directly from measurements, we tested their ability to guide computationally efficient design optimization. To realize this capability, we developed Information-Driver Encoder Analysis Learning (IDEAL), which iteratively improves encoder designs through gradient ascent information estimates (Fig. 3). IDEAL requires only differentiable encoder model and a dataset or model representing the objects to be imaged, avoiding the memory and compute requirements and backpropagation challenges of complex neural network decoders.
IDEAL successfully optimized color filter arrays, creating encoders that capture more information and enable better reconstruction. Testing on photography filters with red, green, blue and white pixels, IDEAL progressively improved filter designs compared to random initial filter (Fig. (b). To validate the optimization, we trained decoders on measurements from designs at different optimization stages, and found that reconstruction accuracy increased from beginning to end (Fig. 3r). Furthermore, our PixeICNN estimator independently confirmed that information content increased throughout optimization, validating that the faster Gaussian model used as objective successfully guided the design process. IDEAL achieved comparable information content and decoder error to end-to-end optimization (6Л. which jointly trains both encoders and decoders. This encoder-only approach offers advantages in simplicity, memory, and runtime, which are studied in greater depth subsequent work (77).
Discussion
In this work, we developed broadly applicable and computationally efficient method that comprehen- sively captures imaging system performance by quantifying the information content of measurements. The method applies broadly because it uses only measurement data and noise characterization, with- out requiring any knowledge or assumptions about objects or system-specific mathematical models. captures comprehensive performance because mutual information naturally synthesizes resolution, noise, sampling, and all other factors that affect the ability to distinguish different objects into single
8

---

a) Information-Driven Encoder Analysis Learning (IDEAL) for color filter design
Objects
Differentiable encoder model
Measurements
fe
Information estimator
IX; Y)
Gradients
c) IDEAL design matches end-to-end design performance
b) Optimization process
Measurements Encoder
x10*
95% confidence
-2.00
11.5
Random
-Mutual information
Mean squared error
Initialization
End-to-end
IDEAL
-2.08
Optimization step
3714
1.15
1.20
1.25
Mutual Information (bits per pixel)
Figure 3: Information-Driin Driven Encoder Analysis Learning (IDEAL) designs encoders through gradient ascent on information estimates. a) IDEAL framework applied to color filter design: gradient feedback from information estimates updates parameters of differentiable filter model to maximize information capture. b) Estimated information increases monotonically during optimization. c) The final IDEAL-optimized encoder matches the performance of jointly optimizing encoders and decoders end-to-end [61] in terms of both downstream reconstruction error and measurement mutual information, while avoiding decoder complexity during training. We further study IDEAL in comparison end-to-end subsequent work [77].
metric. It achieves computational efficiency by evaluating encoders directly, bypassing the traditional approach of using decoder algorithms to perform secondary tasks and then measuring success those tasks.
We validated that our approach realizes these advantages while matching traditional decoder-based evaluation and design strategies across color photography, radio astronomy, lensless imaging, and microscopy. Information estimates consistently predicted which designs would enable decoders to reconstruct unknown objects most accurately. Additionally, we demonstrated that information estimates can guide automated design of imaging systems using IDEAL (Information-Drivc Encoder Analysis Learning), achieving comparable performance to the prevailing approach of end-to-end optimization through encoder and decoder simultancously.
Information-bass evaluation offers computational advantages that suggest promising capabilities for designing previously intractable imaging systems in simulation. By avoiding decoders, the approach eliminates the memory requirements of large neural networks and the training complexity of joint optimization, such vanishing gradients that arise during backpropagation through complex archi- tectures [ [2]. This reduced computational burden enables more effective exploration of design spaces and may allow for tackling new problems where end-to-end design becomes computationally intractable. We explore IDEAL's capabilities and limitations more extensively across diverse imaging systems in subsequent work 77

---

Beyond simulated design, our method creates entirely new possibilities for rigorously evaluating imaging systems operating in real-world conditions. Current evaluation approaches each have significant limitations: subjective visual assessment of raw measurements or algorithm outputs suffers from observer bias and inconsistency; heuristic quality metries applied to algorithm outputs are unreliable for powerful neural network decoders that can hallucinate convincing but incorrect details [78); assessments like signal-to-noise ratio capture only isolated system aspects rather than overall capability; and standardized test objects like resolution targets may not represent the diversity approach addresses these limitations by providing
of real-world objects. Our information-bast an objective, unified metric that quantifies measurement quality without requiring knowledge or assumptions about the objects being imaged. Our approach thus enables rigorous and comprehensive evaluation of imaging systems capturing unknown objects in their intended environments.
More broadly, our method may aid in discovering signal in measurements that appear unintelligible to human observers. Information theory suggests that optimal encoders produce measurements that appear random or noisy to maximize entropy and ensure small object changes remain distinguish- able [79 80] Such measurements would naturally appear unintuitive to human observers who are selectively sensitive to specific visual features [Л. Laser speckle patterns hint at this possibility: once routinely suppressed as unwanted noise 44), they can now be effectively interpreted by neural networks 83
Extensions and future directions
Beyond our framework's s primary applications the evaluation and design of imaging systems. several promising directions could improve its capabilities, extend its theoretical foundations, and enable its application to new classes of design problems.
Performance improvements. Better probabilistic models could improve the estimator's accuracy. Transformer architectures (4 exhibit predictable scaling laws [52 85], enabling either deployment of larger models for more accurate estimates or extrapolation of true information content using scaling relationships on smaller models. Alternatively, specialized architectures might achieve equivalent accuracy with fewer computational resources.
Stochastic encoders. Extending the framework handle con-deterministic imaging systems would broaden its applicability. Two scenarios require this extension: systems with unpredictable varia- tions like illumination fluctuations or mechanical drift, and quantum imaging regimes where image formation is inherently random [86-88). We outline key challenges for this extension in the supple- ment (Sec. (2.7). though practical validation remains future work.
Task-specific imaging. Extending the framework to specialized tasks would address applications where maximizing total information may be suboptimal. While our approach captures performance well on reconstruction tasks that require recovering objects complete detail, information estimates less accurately characterize performance when decoders perform tasks such as classification that prioritize specific object features. Modifying the objective function our estimator (Sec. (2.88 might address this limitation. Alternatively, optimization for general performance with IDEAL could provide a starting point for task-specific refinement using end-to-end design methods 12 that reduces labeled data requirements.
Design in non-differemtiable systems. Information estimation could enable optimization of non- differentiable imaging systems where neither end-to-end design nor IDEAL would apply. This includes combinatorial design problems in simulation and real-time parameter adjustments de- ployed imaging systems. Both applications require efficient objective functions that can be evaluated repeatedly during gradient- free optimization. Subsequent work has applied information estimation to non-differentiable imaging system design [89].
Other sensing modalities. Information-theoreti design principles could extend beyond imaging to optimize performance in electronic, biological, geological, and chemical sensors. Our method could potentially apply to any sensing system that can be modeled as deterministic encoding with a known noise model, suggesting possibilities for systematic sensor design across diverse domains.
10

---

Project website
Code
Acknowledgements
For helpful feedback and discussions about this work we thank T. Courtade, E. Aras, K. Lee, K. Bouman, A. Gao, M. Foxxe, C. Degher, S. Degher, D. Degher, S. Baker, J. Goodman, A. Ashok and the UC Berkeley Computational Imaging Lab.
Author Contributions
Conceptualization: H.P. Methodology: H.P., L.K., and J.J.
Data, Experiments and Software: L.K., H.P., and E.M.
Funding: L.W.
Supervision: H.P. and L.W.
Visualization and figures: H.P., T.C., L.K., E.M., and L.W.
Writing: The original draft was written by H.P. with assistance from T.C. and L.K.;; all authors contributed to review and editing.
Funding
L.W. is a Chan Zuckerberg Biohub investigator. L.K. was supported by the National Science Foundation Graduate Research Fellowship Program under Grant DGE 2146752.
This work was supported by STROBE: National Science Foundation Science and Technology Center under Grant No. DMR 1548924, ONR Grant N00014-17-1-2400 and the U.S. Air Force Office Multidisciplinary University Research Initiative (MURI) program under award no. FA9550- 23-1-0281.

---

References
[1] David Moshe Fass. Human sensitivity to mutual information. Ph.D., Rutgers The State University of New Jersey, School of Graduate Studies, United States New Jersey, 2006. URL http:s//www.ppppsss...cooooooooa
(r551777f.4aap///eeeeeeeeesessss 9780542513411.
ISBN:
[2] Ayush Bhandari, Achuta Kadambi, and Ramesh Raskar. Computational imaging. The MIT Press, Cambridge, Massachusetts, 2022. ISBN 978-0-262-04647-3,
[3] Shaofan Yuan, Chao Ma, Ethan Fetaya, Thomas Mueller, Doron Naveh, Fan Zhang, and Fengnian Xia. Geometric deep optical sensing. Science, 379 1220, March 2023. doi: 10.1 222///ienecee.eeeeeeoo 1220. htttth:tt///rtt paatts://oor.ssceencc www.scciiccc...ccc.cco wwto//////ooo. /oi/10.iii./ 1wt.0.111/sscnnccccco Publisher: American Association for the Advancement of Science.
ade1220
[4] Gordon Wetzstein, Aydogan Ozcan, Sylvain Gigan, Shanhui Fan, Dirk Englund, Marin Soljačić, Cornelia Denz, David A.B. Miller, and Demetri Psaltis. Inference in artificial intelligence with deep optics and photonics. Nature, 588(7836):39 47, 2020. ISSN 14764687, doi:
10. 1011000/1116-----1
[5] Vincent Sitzmann, Steven Diamond, Yifan Peng, Xiong Dun, Stephen Boyd, Wolfgang Hei- drich, Felix Heide, and Gordon Wetzstein. End-to-end optimization of optics and image processing for achromatic extended depth of field and super-resolution imaging. ACM Trans- actions on Graphics, 37 August 2018. ISSN 0730-0301, 1557-7368. doi: 10.1 145/ 3197517.320133.. URL https: acm. www./oo//o//o/o.////oo 11t.///90.111///7/11171 32013333
[6] Qilin Sun, Ethan Tseng, Qiang Fu, Wolfgang Heidrich, and Felix Heide. Learning Rank- Diffractive Optics for Single-Shot High Dynamic Range Imaging- In 2020 IEEF/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 1383-1393, Seattle, WA. USA, June 2020, JEEE. ISBN 978- 1-1-1--22117--666 ieee org/ http///ttmmmmttmmeet//..///
doi: N0.1109/CVPP442000022222
00146. htRp:t/t::///eee
[7] Christopher A. Metzler, Hayato Ikoma, Yifan Peng. and Gordon Wetzstein. Deep Optics for Single-Shot High-Dynamic-Rane Imaging. In 2020 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1372-1382, Seattle. WA, USA, June 2020. URL. https
IEEE. ISBN 1-881-7228171666 //ieeexplore 1eee. org/ /ttp//mmmmmmtt1566777//////
doi: 10. 111.///V2242002222.000000
[8] Qilin Sun, Congli Wang.
Qiang Fu, Xiong Dun, and Wolfgang Heidrich. End-to-end complex lens design with differentiate ray tracing. ACM Transactions on Graphics, 40(4): August 2021. ISSN 0730-0301, 1557-7368. doi: 10. 00.1145/3450026635556744 URL https///: acm
org/doi/100
[9] Ethan Tseng, Shane Colburn, James Whitehead, Luocheng Huang. Seung-Hwan Back, Arka Majumdar, and Felix Heide. Neural nano-optics for high-quality thin lens imaging. Nature Communications, 12(1):6493, November 2021. ISSN 2041- 1723. doi: 10. (1038//41447-0222264444 URL https:://ssett nature. www.eoooaicceeeeee article//
[10] Ethan Tseng, Ali Moslch, Fahim Mannan, Karl St-Arnaud, Avinash Sharma, Yifan Peng, Alexander Braun, Derek Nowrouzezahrai, Jean-François Lalonde, and Felix Heide. Differen- tiable Compound Optics and Processing Pipeline Optimization for End-to-end Camera Design. ACM Transactions on Graphics, 40(2):1-19 April 2021. ISSN 0730-0301, 1557-7368. doi: 10. 1 10.1344/344.91 BRLphttps:/// //d1 acm. org /tw//0.o11111///// 111.1..114444777777
[11] Diptodip Deb, Zhenfei Jiao, Ruth Sims, Alex B. Chen, Michael Broxton, Misha B. Ahrens, Kaspar Podgorski, and Srinivas C. Turaga. FourierNets enable the design of highly non-local optical encoders for computational imaging, November 2022. URL http: /rxxvv .org/i.rg///.//....
aba/2104. 10611 arXiv:2104. 10611 [cs, eess].
[12] Xinge Yang, Qiang Fu, Yunfeng Nie, and Wolfgang Heidrich. Image Quality Is Not All You Want: TrikeDriven Lens Design for Image Classification, May 2023. URL http:
//arxiv. org/ abs abs/ 220// 17185 arXiv:2305.17185 [physics].
12

---

[13] P.B. Fellgett and E.H. Linfoot. On the assessment of optical images. Philosophical Transactions of the Royal Society of London. Series A, Mathematical and Physical Sciences, 247(931); 369-407, February 1955. ISSN 0080-4614. doi: 10. 10.t/aa/ 1assaa.1555.0000 URL https //ww.jjstor org/ sta.le/lbll...... 1tt
Publisher: Yale University Press.
[14] G. Toraldo di Francia. Resolving Power and Information. Journal of the Optical Society of America, 45(7):497, July 1955. ISSN 0030-3941. doi: 255.:////S.A...0000000.. URL
https: /tops://www..r www. www.:hoopppubsiisii.e. org/ abstract. cfm?URI=josa -0- 46-1-72
[15] G. Toraldo di Francia. Degrees of Freedom of an Image. Optical Soc America-J, 59(7):
799-804, 1969.
[16] Neil Bershad. Resolution, Optical-Channel Capacity and Information Theory. Journal of the Optical Society of America, 59(2):157, February 1969. ISSN 0030-3941. doi: 10. 1364/J0SA.55.00000
URL htllhttps://wwt wwt.:/wwwwww..hn........o.....oa org/ www.srracttccc?uu
59-2-99-2-155
[17] Carl L. Fales, Friedrich O. Huck, and Richard W. Samms. Imaging system design for improved information capacity. Applied Optics, 236))872-888,, March 1984. ISSN 2155-3165. doi: 10. 10.1336-66 1364/A0.233 23.0000722. URL/ttpps: htttps://opp opti org/ao Publisher: Optica Publishing Group.
cfm/uri abstract. org/ao/abstract.ccii
a0-23-6- 2---23--880 872
[18] I. Cox and C. J. R. Sheppard. Information capacity and resolution in an optical system. Journal of the Optical Society of America 3()):112, August 1986. ISSN 1084-7529. doi: 10. 1364/JOSAA.3.00 A.0,001 152. URL hwtp://wwwww.eom http://wwww.aabb...oooogggrgaraa josaa-3-8-115;
[19] R. Linsker. Self-organization in perceptual network. Computer, 21(3):05-1177, March 1988. ISSN 0018-9162. doi: 10.1 109/2.36. URL http://ieeeeplorrs leee org /org///cumett//// 36/
[20] Ralph Linsker. An application of the principle of maximum information preservation to linear systems. In D. Touretzky, editor, Advances in neural information processing systems, volume 1.
1988. URL htt.::::/poooceeeaaiiii hewpe/ddiiinnnnnuuispeee.e httpps.ccpppppppppe cc/paper. .pdf
[21] D. Van Dyck and A.F. de Jong. Ultimate resolution and information in electron microscopy: general principles. Ultramicroscopy, (7(1-)):266-281 November 1992. ISSN 03043991. doi: 10. p11/030439919290202U
elsevier wtt.svieer..cm//erreeeeeeee URL http:://www.inggggaaaas........
[22] Friedrich O. Huck, Carl L. Fales, and Zia-ur Rahman. An information theory of visual communication. Philosophical Transactions of the Royal Society of London. Series A: Mathematical, Physical and Engineering Sciences, 354(1716)2193--223 org abs/ doi: 10.1098/rsta. 1996.0098.
January 1997.
10. 10910/9rttt... 1996. 0098
Publisher: Royal Society.
[23] Mark A. Neifeld. Information, resolution, and bance-bandwidth product. Optics Letters. 23 (18):1477, 1998. ISSN 0146-9592. doi: 10. 10.1364/01.23.001477,
[24] J.A. O'Sulliaan, R.E. Blahut, and D.L. Snyder. Information-thhoretie image formation. IEEE Transactions on Information (4teory,,4(6):2000 21233 October 1998. ISSN 001 00189448. doi: 10. 109/18.720533. URL htt.::tttte/eepppt/tt
ieee. www.ee..rr/.oomeeee///00000
[25] Rachel Alter-Gartenberg Information metric as a design tool for optoelectronic imaging systems. Applied Optics, 39(11):143311760, April 2000. ISSN 2155-3165. doi: 10.136//AO. 39.001743. URL httpps://ope potpriia..rg////aassrrraa abstractt............ cff?uri 39- 39-1 Publisher: Optica Publishing Group.
[26] Sudhakar Prasad. Information capacity a seeing-limited imaging system. Op- tics Communications, 177(1):119-134, April 2000. ISSN 0030-4018. doi: 10.1016/
$00300-4118000000555-8 http://onnnoo..........ree//eeea http:///www..cccoecceeeeeee/o//s
pii/ pi1//00330401800055533
13

---

[27] Luyin Cao, Stephen E. Reichenbach, and Ram M. Narayanan. Information theoretic assessment and design of hyperspectral imaging systems with nonuniform bandwidths. Stephen K. Park, Zia-ur Rahman, and Robert A. Schowengerdt, editors, Aerospuce/Defense Sensing, Simulation, and Controls, pages 78,78, Orlando, FI., August 2001. doi: 10. 10.1117/12.43844.3 URIpttpp/ V/proceedings.. spiediggeee giggllllllii talribrary .rg/ wwtggp/oceeding.oom asp..aatciiicciii.1111111055.
[28] Christoph Wagner and Gerd Häusler. Information theoretical optimization for optical range sensors. Applied Optics, 42(27):5418, September 2003. ISSN 0003-6935, 1539- 4522. doi: 10. 00.1364/A0.442.05541 URI=ao- 42- 42-27-5418
URL htttps://og..ooon optica. artaaa/arrpppooooaarra.aaaa..a.. cfm?
[29] Mark A. Neifeld. Information-optimmize imaging system design. In Stephen K. Park, Zia-ur Rahman, and Robert A. Schowengerdt, editors, Aerospace/Defense Sensing, Simulation, and Controls. pages 113-116, Orlando. FL. August 2001. doi: 10.1117/12.438248 / //proceedings.
[30] Amit Ashok and Mark Neifeld. Information-based analysis of simple incoherent imaging systems. Optics Express, 1115333, September 2003. ISSN 1094-4087, doi: 10.1364/OE. 11.002153. https:s:///oog optica org/oe /e/ awstaaatt (atprracttccmmrrrr-------111 11--1-------11------ 18-2153
[31] Mark A. Neifeld, Amit Ashok, and Pawan K. Baheti. Task-specific information for imaging system analysis. Journal of the Optical Society of America A, 24( 1): December 2007. ISSN 1084-7529, 1520-8532, doi: 113364/0AAA..4.0000555. URL https://opg...... optica
org/abstract cff.ccmmu...mmm wwwpuml=mmiieeoeoee. -saa----josaa--- 24-a----4--1 12-
[32] Yihong Wu and Sergio Verdú. Optimal Phase Transitions in Compressed Sensing, July 2012.
URR.ptttp//arxxiv.org http/bbs/1111 6822 arXiv:1111.688 [cs, math, stat].
[33] Amit Ashok, Liang-Chil Huang, and Mark A. Neifeld. Information optimal compressive
sensing: static measurement design. Journal of the Optical Sociery of America A, 30(5): 831. May 2013. ISSN 1084-7529, 1520-8532. doi: 10. 355.4//0SAA.33.0000000000
URL
https://opg.. optica. org abstract. SEnTPURI- 30-
[34] James Huang and Amit Ashok. Information optimal Compressive X-ray Threat Detection. Imaging and Applied Optics, 2015.
[35] Timur E. Gureyev, Alexander Kozlov, Yakov Nesterets, David M. Paganin, Andrew Martin, and Harry M. Quineya. Signal-to-noise, spatial resolution and information capacity of coherent diffraction imaging. IUCrJ, 5:716-726, 2018. ISSN 20522525. doi: 10.1 10.1 2205225251801094 Publisher: International Union of Crystallography.
[36] Evgenii Narimanov. Resolution limit of label-free far-field microscopy. Advanced Photonics, 1055):1, 2019. ISSN 2577-5421. doi: 10. 1t1.11/........00556 1.5
arXiv: 1903.05254.
[37] Florian Willomitzer. Single-Shot 3D Sensing Close to Physical Limits and Information Limits. Springer Theses. Springer International Publishing, Cham, 2019. ISBN 978-3-030- 10903-5. doi: 10.1007/778-3-00 10904-2. URL httt:://link wwr.ik...ringrrr.....er. com/ 978-3-030-10904-22
cwm/10. 10. 1007/
[38] Amit Ashok, James Huang, Yuzhang Lin, and Ronan Kerviche. Information optimal compressive imaging: design and implementation. In Harrison H. Barrett, John Greivenkamp. and Eustace L. Dereniak, editors, SPIE Optical Engineering + Applica- tions, page 91860K, San Diego, California, United States, October 2014. doi: 10.1117/12. 2063947. URL http::://ooceeepnggggee/// www.sspiiiaellbbraeee... wwwpprocceeeingg.g.oooooooogpoep aspx? doi=10. 1117/11111111 116312 2063947
[39] Hyun Sung Chang, Yair Weiss, and William T. Freeman. Informative Sensing, January 2009.
URL http:://rxiii org/ abs/ab///// 00001 4275 arXiv:0901.4275 [cs, math).
[40] Joseph W. Goodman. Statistical optics. Wiley series in pure and applied optics. John Wiley Sons Inc, Hoboken, New Jersey, second edition edition, 2015. ISBN
14

---

[41] Alexander Kraskov, Harald Stögbauer, and Peter Grassberger. Estimating mutual information. Physical Review 6666)66611,,,, June 2004, doi: 10. www.1o0oyyeeee..9..ooo URLhttps Publisher: American Physical
//link. aps www.oorg////o/ooo 1wt.:/1110//////./ 69. 066138
Society.
[42] Georges A. Darbellay and Igor Vajda. Estimation of the information by an adaptive partitioning of the observation space. IEEE Transactions Information Theory, 45(4): 1999.
ISSN 00189448. doi: 10.1109/18.761999
[43] Brian C. Ross. Mutual information between discrete and continuous data sets. PLoS ONE, 9 0087357..
(2), 2014. ISSN 19326203. doi: 10. 137
[44] Ben Poole, Sherjil Ozair, Aaron van den Oord, Alexander A. Alemi, and George Tucker. On Variational Bounds of Mutual Information, May 2019, URL http:://arxiv.org/aaaea
06922
arXiv:1905.0692 [cs, stat].
[45] Ziv Goldfeld and Kristjan Greenewald. Sliced mutual information: scalable mea- sure of statistical dependence. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wort- man Vaughan, editors. Advances in neural information processing systems, 2021. URL attps://openrevieet net/ forum?id-SvrYl FDq2
[46] Jiaming Song and Stefano Ermon. Understanding the limitations of variational mutual in- formation estimators. In International conference on learning representations, 2020. URL
https://openreviert http::///ppeerrrrieeeeeeeerrrrrt
[47] Pawel Czyz, Frederic Grabowski, Julia E Vogt, Niko Beerenwinkel, and Alexander Marx. Beyond normal: On the evaluation of mutual information estimators. In Thirly-seventh conference on neural information processing systems, 2023. hattps://oppnreete net/forumt iisrummidd2555555177
[48] Gokul Gowri, Xiao-Kang Lun, Allon M. Klein, and Peng Yin. Approximating mutual in- formation high-dimensional variables using learned representations. A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang, editors, Advances in neural information processing systems, volume 37, pages 132843-132875. Curran As- sociates, Inc., 2024. URL https: o00//f/n10ee00033ccc41a1aa10e113
neurips cc/ httplppaprraapppppplpppppee Confereceeeeeeeee.pp.c.
[49] Nunzio A. Letizia, Nicola Novello, and Andrea M. Tonello. Mutual information esti- mation via f-Divergence and data derangements. In A. Globerson, Mackey, D. Bel- grave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang, editors, Advances in neural infor- mation processing systems, volume 37, pages 1051 / 2024. URL https::/pproceddiiggg neurips. cc/ wwtp:ccccpppppp.eeeeeeee paper_ffllee files/lll///eeeo pappr//202//tilll bccr850aaaa4111101111888882222/2 oore pdf
Curran Associates, Inc..
[50] Butakov I. D., Tolmachev A. D., Malanchuk S. V., Neopryatnaya A. M., and Frolov A. A. Mutual information estimation via normalizing flows. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in neural information processing systems, volume 37, pages 3027-3057. Curran Associates, Inc., 2024. URL httpp:://proceedd/ 0fa2d9ef [522000f0aef//9977777/CC7/77-220 Papere Conf crence. pdf
wttppppr.cppp...pcceepp neur ips. cc/ papp/aaperrr___ snmerrffilss fel.ffillsppaaaeee httles/papprr222//fieeee
[51] David McAllester and Karl Stratos. Formal limitations on the measurement of mutual in- formation. In Silvia Chiappa and Roberto Calandra, editors, Proceedings of the twenty third international conference on artificial intelligence and statistics, volume 108 of Pro- ceedings of machine learning research, pages 875-884. PMLR, August 2020. URL https://prooeedingss mlr wwwpppeessvviii pttpsee//vv0/mmmllleeeee////
[52] Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B. Brown, Prafulla Dhariwal, Scott Gray, Chris Hallacy, Benjamin Mann, Alec Radford, Aditya Ramesh, Nick Ryder, Daniel M. Ziegler, John Schulman, Dario Amodei, and Sam McCandlish. Scaling Laws for Autoregressive Generative Modeling, October 2020. http://///arrr org////aaaa hngg/bs//an 2tw/aab///0...//// 2010 14701. arXiv: 2010.14701.
15

---

[53] Aaron van den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel Recurrent Neural Networks, 2016. URL htttttttp/a/x// wttp:////aaa...0000007111 9781510882008..
http:/161..0000 06759 arXiv: 1601.06759 ISBN:
[54] Aäron van den Oord, Nal Kalchbrenner, Oriol Vinyals, Lasse Espeholt, Alex Graves, and Koray Kavukcuoglu. Conditional Image Generation with PixeICNN Decoders. arXiv, pages 4790-4798, 2016. ISSN 10495258. URL https:// wips. cc/ 6aper/6627-conditt -ew.raaa-aa-aaeeeeeeeriiiooooo-e with wwtp:piiiicccndddciieie 1606.05328 ISBN: 9781510829008.
arXiv:
[55] Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik Kingma. Pixel CNN++: Im- proving the Pixel CNN With Discretized Logistic Mixture Likelihood. Iclr, pages 1-10, 2017. URL httpps://///.mmm /ttp/ppeeeeveieeeee net abs/ 1701. 05517
ottpar/eeeeeeeeeoooooooonoooofrr http:///wnn/derrrrooo/oooog///rs org/
Xiv: 1701.05517.
[56] Eric Clarkson and Johnathan B. Cushing. Shannon information and receiver operating charac- teristic analysis for multiclass classification in imaging. Journal of the Optical Society of Amer- ica A, 33(5);930, May 2016. ISSN 1084-7529, 1520-8532. doi: 10.
URL Utt.htts:: http:/:://oog.pppcc..roo/r/////s wtt.aa/ggg/bbbbbaaaeeeeeeee wttp:/atttaacccc.I---aaaaa..- wwtp:c-=mmmsmaaaaaa (asiiiojssaa----33 5-30
[57] Eric Clarkson and Johnathan B. Cushing. Shannon information and ROC analysis in imaging. Journal of the Optical Society of America A, 32(7):1 July 2015. ISSN 1084-7529, 1520- 8532. doi: 10. 10.1364/JOSSAA.32.01222
Uttphtss::/ http::/::///p.pppii.....aaap optica org wttg/abstract..com cfm?
URI= jil=josaa-32--1 1-saa-32-7-111 1288
[58] D. Guo, S. Shamai, and S. Verdu. Mutual Information and Minimum Mean-Square Error in Gaussian Channels. IEEE Transactions on Information Theory, 5164)122111212222, April 2005.
ISSN 0018-9448. doi: 10. 1LL10//TIT.20055844400 document/1412024/
URL http::/ ieee. org/
[59] Dongning Guo, Shlomo Shamai, and Sergio Verdu. Mutual Information and Conditional Mean Estimation in Poisson Channels. IEEE Transactions on Information Theory, 54(5): 1837-1849, May 2008. ISSN 0018-9448, 1557-9654. doi: 10.109//TT.2008.922220 URL
http://ieeexplore. leee. org...eeeee....eooooee http///ocmmtt/////eeee 4494688/
[60] Bernardo Henz, Eduardo L. Gastal, and Manuel M. Oliveira. Deep Joint Design of Color
Filter Arrays and Demosaicing. Computer Graphics Forum, 37(2):389-399) May 2018. ISSN 0167-7055, 1467-8659. doi: 10. 10.111 10.111 111l//ff..333/0 URL hoLlhttts///onnllebrrry wiley
com/doi/10 1111/cgf..... 13370
[61] Ayan Chakrabarti. Learning Sensor Multiplexing Design through Back- pock-propagation. In Advances in Neural Information Processing Systems 29 (NeurIPS 2016), 2016.
[62] Peter Vincent Gehler, Carsten Rother, Andrew Blake, Tom Minka, and Toby Sharp. Bayesian color constancy revisited. In 2008 IEEE Conference on Computer Vision and Pattern Recogni- tion, pages 1-8, Anchorage, AK, USA, June 2008. IEEE. ISBN 978- 978-1-4244-224--5 doi: 10. wWWpCCVVR.220.84587776 URL http:://ieeeepppoo//////// ieee
[63] Lilong Shi and Brian Funt. Re-processsed Version of the Gehler Color Constancy Dataset of 568 Images, 2012. URL http:/ttt///w...cc.aa http://w........ccccooooo//aaaaa
colour
[64] The Event Horizon Telescope Collaboration, Kazunori Akiyama, Antxon Alberdi, Walter Alef, Keiichi Asada, Rebecca Azulay, Anne-Kathrin Baczko, David Ball, Mislav Baloković, John Barrett, Dan Bintley, Lindy Blackburn, Wilfred Boland, Katherine L. Bouman, Geoffrey C. Bower, Michael Bremer, Christiaan D. Brinkerink, Roger Brissenden, Silke Britzen, Avery Broderick, Dominique Broguiere, Thomas Bronzwaer, Do- YouYo Byun, John E. Carlstrom, Andrew Chael, Chi-kwan Chan, Shami Chatterjee, Koushik Chatterjee, Ming-Tang Chen, Yongjun Chen, Ilje Cho, Pierre Christian, John E. Conway, James M. Cordes, Geoffrey B. Crew, Yuzhu Cui, Jordy Davelaar, Mariafelicia De Laurentis, Roger Deane, Jessica Dempsey, Gregory Desvignes, Jason Dexter, Sheperd S. Doeleman, Ralph P. Eatough, Heino Falcke, Vincent I.. Fish, Ed Fomalont, Raquel Fraga-Encinas, William Freeman, Per Friberg, Christian M. Fromm, José L. Gómez, Peter Galison, Charles F. Gammie, Roberto García,
16

---

Olivier Gentaz, Boris Georgiev Ciriaco Goddi, Roman Gold, Minfeng Gu, Mark Gurwell, Kazuhiro Hada, Michael H. Hecht, Ronald Hesper, Luis C. Ho, Paul Ho. Mareki Honma, Chih-Wei I.. Huang, Lei Huang, David H. Hughes, Shiro Ikeda, Makoto Inoue, Sara Issaoun, David J. James, Buell T. Jannuzi, Michael Janssen, Britton Jeter, Wu Jiang, Michael D. Johnson, Svetlana Jorstad, Taehyun Jung, Mansour Karami, Ramesh Karuppusamy, Tomohisa Kawashima, Garrett K. Keating, Mark Kettenis, Jac- Young Kim, Junhan Kim, Jongsoo Kim, Motoki Kino, Jun Yi Koay, Patrick M. Koch, Shoko Koyama, Michael Kramer, Carsten Kramer, Thomas Krichbaum, Cheng-Yu Yu Kuo, Tod R. Lauer, Sang-Sung Lee, Yan-Rong Li, Zhiyuan Li, Michael Lindqvist, Kuo Liu, Elisabetta Liuzzo. Wen-Ping Lo, Andrei P. Lobanov, Laurent Loinard. Colin Lonsdale, Ru-Sen Lu, Nicholas R. MacDonald, Jirong Mao, Sera Markoff, Daniel P. Marrone, Alan P. Marscher, Iván Martí- Vidal, Satoki Matsushita, Lynn D. Matthews, Lia Medeiros, Karl M. Menten, Yosuke Mizuno, Izumi Mizuno, James M. Moran, Kotaro Moriyama, Monika Moscibrodzka, Cornelia Müller, Hiroshi Nagai, Neil M. Nagar, Masanori Nakamura, Ramesh Narayan, Gopal Narayanan, Iniyan Natarajan, Roberto Neri, Chunchong Ni, Aristeidis Noutsos, Hiroki Okino, Héctor Olivares, Tomoaki Oyama, Feryal Özel, Daniel C. M. Palumbo, Nimesh Patel, Ue-Li Pen, Dominic W. Pesce, Vincent Piétu, Richard Plambeck, Aleksandar PopStefanija. Oliver Porth, Ben Prather, Jorge A. Preciado- López, Dimitrios Psaltis, Hung-Yi Yi Pu, Venkatessh Ramakrishnan, Ramprasad Rao, Mark G. Rawlings, Alexander W. Raymond, Luciano Rezzolla, Bart Ripperda, Freek Roelofs, Alan Rogers, Eduardo Ros. Mel Rose, Arash Roshanineshat, Helge Rottmann, Alan L. Roy, Chet Ruszczyk, Benjamin R. Ryan, Kazi L. Rygl, Salvador Sánchez, David Sänchez-Arguelles Mahito Sasada, Tuomas Savolainen, F. Peter Schloerb, Karl-Friedrich Schuster, Lijing Shao, Zhiqiang Shen, Des Small, Bong Won Sohn, Jason SooHoo, Fumie Tazaki, Paul Tiede, Remo P. J. Tilanus, Michael Titus, Kenji Toma, Pablo Torne, Tyler Trent, Sascha Trippe, Shuichiro Tsuda, llse Van Bemmel, Huib Jan Van Langevelde, Daniel R. Van Rossum, Jan Wagner, John Wardle, Jonathan Weintroub. Norbert Wex, Robert Wharton, Maciek Wielgus, George N. Wong, Qingwen Wu, André Young, Ken Young, Ziri Younsi, Feng Yuan, Ye-Fei Yuan, Anton Zensus, Guangyao Zhao, Shan-Shan Zhao, Ziyan Zhu, Joseph R. Farah, Zheng Meyer-Zhao, Daniel Michalik, Andrew Nadolski, Hiroaki Nishioka, Nicolas Pradel, Rurik A. Primiani, Kamal Souccar, Laura Vertatschitsch. and Paul Yamaguchi. First M87 Event Horizon Telescope Results. IV. Imaging the Central Supermassive Black Hole. The Astrophysical Journal Letters, 875(1):L4, April 2019. ISSN 2041-8205, 2041-8213, doi: 10.384772041-8213331 3/abee88. URL http:/ddeeee...oooooorrrrcco//oo 1tt.///////1..0022221-1-11111110
[65] Sheperd S. Doeleman, John Barrett, Lindy Blackburn, Katherine L. Bouman, Avery E. Brod- erick, Ryan Chaves, Vincent L. Fish, Garret Fitzpatrick, Mark Freeman, Antonio Fuentes, José I.. Gómez, Kari Haworth, Janice Houston, Sara Issaoun, Michael D. Johnson, Mark Kettenis, Laurent Loinard, Neil Nagar, Gopal Narayanan, Aaron Oppenheimer, Daniel C. M. Palumbo, Nimesh Patel, Dominic W. Pesce, Alexander W. Raymond, Freek Roelofs, Ranjani Srinivasan, Paul Tiede, Jonathan Weintroub, and Maciek Wielgus. Reference Ar- ray and Design Consideration for the Next-Generation Event Horizon Telescope. Galax- ies, 11(5):107, October 2023. ISSN 2075-4434. doi: 10. n..3390galaxiies URL https://wwi mdpi www./0000-m0 21112004754442111 44144/4//4111107777
[66] Vivek Boominathan, Jacob T. Robinson, Laura Waller, and Ashok Veeraraghavan. Recent advances in lensless imaging. Optica, 91):1-16, January 2022. ISSN 2334-2536. doi: 10. 434-0PPTICA.413661 URL hattps://opt optica. org http///ticcaaaaaaa Publisher: Optica Publishing Group.
ottp/ptiic/abssracttt.cctoo abstractt..tt.......... cfm?uri=
optica-9-1-1
[67] Leyla Kabuli, Gina Wu, and Laura Waller. Bigh-Qualing Lensless Imaging with Random Multi-Focal Lenslet Phase Mask. In Optica Imaging Congress (3D, COSI, DH, FLarOptics, IS. PcAOP), page CW3B.2 2, Boston, Massachusetts, 2023. Optica Publishing Group. ISBN 978-1-957171-28-9 doi: 10. 1344//3002222222
20..........02...333.2..222 URL httth:://p//pp optica...ptic.rr.g..
abstract www::RI-COSI---2 - -0 CW3B.
[68] Nick Antipa, Grace Kuo, Reinhard Heckel, Ben Mildenhall, Emrah Bostan, Ren Ng, and Laura Waller. DiffuserCam: lensless single-exposure 3D imaging. Optica, 5(1):1, January 2018. ISSN 2334-2536. doi: 10. 10.1364/OPTICA.5..OM 5.0000.. URL. https: /ttts:://op..ptiiee org/ abstract. cfn
17

---

[69] Guoan Zheng, Christopher Kolner, and Changhuei Yang. Microscopy refocusing and dark- field imaging by using simple LED array. Optics Letters, 36(20):3987, 2011. ISSN 0146-9592.
doi: 10.1 10.1364/01... 00..9..7.6.007877. URL https: www osapublishing org/abstract. cfm?
URI=ol-36- 20--66-2--9---8
ISBN: 0090-8258 (Print)r00090-8255) (I.innin (Linking).
[70] Zachary Phillips, Regina Eckert, and Laura Waller. Quasi-Dome: A Self-Calibrated High-
NA LED Illuminator for Fourier Ptychography. In Imaging and Applied Optics 2017 (3D, AIO. COSI, IS. MATH. pcAOP), volume IW4E.5, 2017. doi: 10.1364/isa.22 2tliiia..201...vi iww...
[71] Michael R. Kellman, Emrah Bostan, Nicole A. Repina, and Laura Waller. Dussccs-Based Learned Design: Optimized snded-Illumination for Quantitative Phase Imaging. IEEE Transactions Computational Imaging, 5(3):344 353, 2019. ISSN 2573-0436. doi: 10.1109/
tci. Ici.2019.2904444.
arXiv: 1808.03571.
[72] Roarke Horstmeyer, Richard Y. Chen, Barbara Kappes, and Benjamin Judkewitz. Convolu- tional neural networks that teach microscopes how image. arXiv, September 2017. URL http:// arwiv. wwg/ ab////a///// 1109 .70010722713
arXiv: 1709.07223.
[73] Shalin B Mehta and Colin R Sheppard. Quantitative phase-ggaadien imaging at high resolution with asymmetric illumination-based differential phase contrast. Optics Letters, 34(13):1924, 2009. ISSN 01-9995. doi: 10.1 (/644/o..34.01999 ISBN: 0146-9592.
[74] Lei Tian and Laura Waller. Quantitative differential phase contrast imaging in LED array microscope. Optics Express, 23(9):11394, 2015. ISSN 1094-4087. doi: 10. 00.13//0E..333111339 URL https://////ww htttps//www..oaaubbiishii.. www.:oosppbblsissis...
org/ oe/ abstract. cfm/uri 00- 23- 9-1199
[75] Henry Pinkard, Cherry Liu, Fanice Nyatigo, Daniel A. Fletcher, and Laura Waller. The Berkeley Single Cell Computational Microscopy (BSCCM) Dataset, February 2024. URL
http://arxiv. org/ abs 2202 06191 arXiv:2402.06191 [cs, q-bio].
[76] Amit Ashok, Pawan K. Baheti, and Mark A. Neifeld. Compressive imaging system design using task-specific information. Applied Optics, 47(25):4477, September 2008. ISSN 0003-6935, 1539-4522. doi: 11./////0.4..004440 -7
URL https://opg httpp:////ppppppc...ooo.o org/ wttpica.....bbsaaaaaat.t......t. absttacct
[77] Eric Markley, Henry Pinkard, Leyla Kabuli, Nalini Singh, and Laura Waller. Computationally Efficient Information-Drivee Optical Design with Interchanging Optimization, 2025. URL
http://arxivv htwpaaawwii.o..aaaaaaaaaaaeooeee 2507 07789
[78] Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Yejin Bang, Delong Chen, Wenliang Dai, Ho Shu Chan, Andrea Madotto, and Pascale Fung. Survey of Hallucination in Natural Language Generation. ACM Computing Surveys, 55(1)):1-38 December 2023. ISSN 0360-0300, 1557-7341. doi: 10.1145/3571730. URL http:://arax org/abs/ 03629 arXiv:2202.0366 [cs].
[79] Claude Shannon. A mathematical theory of communication. The Bell System Tech- nical Journal, 5(1):3, January 1948. ISSN 15591662,
doi: (0.11455540991.584400
URL hittp://portat acm. atg/citaaion..oom wttp:/ooddddoooo.....1 584093 dyn/9411012 ISBN: 0252725484.
arXiv: chao-
[80] Thomas M Cover and Joy A. Thomas. Elements of Information Theory. Wiley, New York, USA, September 2005. ISBN 978-0-471-24195.99 doi: 10. 10.1002/047174882X URL /7 Series Title: Wiley Series Telecommunications
doi Roi.wileys www.w.ccoowwlll cow.commmoo/aa 1022004712000111
[81] Hengameh Mirzaalian, Mohamed Hussein, and Wael Abd-Almageed. On the Effectiveness of Laser Speckle Contrast Imaging and Deep Neural Networks for Detecting Known and Unknown Fingerprint Presentation Attacks. In 2019 International Conference on Biometrics (ICB), pages 1-8, Crete, Greece, June 2019. IEEE. ISBN 978- 1--2811--14444 URL
doi: 10.1109) leee. org/ htt///dcccmntttt/eee
[82] Yunzhe Li, Yujia Xue, and Lei Tian. Deep speckle correlation: a deep learning approach toward scalable imaging through scattering media. Opticu, 1(10) October 2018. ISSN 2334-2536. doi: 10. 1364/OPTICA...000
http::tts:://ppt http:////p.pp.p.......po wwtp:epopcccccaaapeeeeeeeeeeeeee www./bstraatt......oooeeoeee....
cfm?URI=optica 10- 1181
18

---

[83] Ulas Kürüm, Peter R. Wiecha, Rebecca French, and Otto L. Muskens, Deep learning enabled real time speckle recogmition and hyperspectral imaging using multimode fiber array. Optics Express, 27(15))2096--29999 July 2019. ISSN 1094-4087. doi: 000.66//......00996666
URI, /tttttttt/:://ppoppttt optica. cacllcc.rrrg//oo org/ 00/ hwrgg/e/abbstren abstract........... 2 20965 Optica Publishing Group.
Publisher:
[84] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in neural information processing systems, volume 30. Curran Asso- ciates, Inc., 2017. URL https:://proceeediiii neurips...........p... pdf
filess/aper
[85]
OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Iige Akkaya, Flo- rencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji. Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett- Shaapiirrt- Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen. Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung. Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Lontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Hcewoo Jun. Tomer Kaftan, Lukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Lukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis. Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkinn Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reichhro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyconwoo Noh, Long Ouyang. Cullen O' Oeeeeee Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri. Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shich, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky. Yang Song. Natalic Staudacher, Felipe Petroski Such, Natalie Summers, llya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang. Ben Wang. Jonathan Ward, Jason Wei, C. J. Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojeiech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao
19

---

Zheng. Juntang Zhuang, William Zhuk, and Barret Zoph. GPT-4 Technical Report, March 2024. URL htlLhhtp::/eee /ttp/vaaxvvvv.. wwg.i.oo.......oaooooooo.oooooot www.//gg/////////ee/...//...eee/ aa/aa/b//////// 08774 arXiv:2303.08774 [cs].
[86] B. L. Higgins, D. W. Berry, S. D. Bartlett, H. M. Wiseman, and J. Pryde. Entanglement- free Heisenberg-limited phase estimation. Nature, 450(7168))333-3993 November 2007. ISSN 1476-4687. doi: 10. 1001038/natrr00655555 URL htthhtt:::. pinttp://viie http:///wwwaa......eem/.mam nature. www. com/articles/ nature06257 Number: 7168 Publisher: Nature Publishing Group.
[87] Vittorio Giovannetti, Seth Lloyd, and Lorenzo Maccone. Quantum-Enhancee Mcasure- ments: Beating the Standard Quantum Limit. Science, 30665700) November 2004. doi: 10.1 10.1126/scieeee.. 1 1001199. URL https:///ww/ science. wraeccerrrg/oooooo htg//oii/1.... 1126/ science. 1104149)
Publisher: American Association for the Advancement of Science.
[88] Yonatan Israel, Shamir Rosen, and Yaron Silberberg. Supersensitive Polarization Microscopy Using NOON States of Light. Physical Review Letters, 11((10)103604, March 2014. doi: 10. 10.1100/hyyseeeeet.... 11..Lettt.11111366.. 12.103604. URL sttps://link.. aps org//oi/10..o.. 1103/ Publisher: American Physical Society.
PhysRevLett. 112. 103604
[89] Leyla A. Kabuli, Nalini M. Singh, Henry Pinkard, and Laura Waller. Information - bayesian optimization of imaging systems. Optica Imaging Congress (3D, COSI, DH, FLatOptics, IS, pcAOP), page Paper CTulB.4. Optica Publishing Group, 2025.
[90] Henry Pinkard and Laura Waller. A visual introduction information theory, June 2022. URL http://arxia httpg//aabb//..///ee/22 arXiv:2206.07800 [cs, math].
[91] David C MacKay. Information Theory, Inference, and Learning Algorithms David J.C. MacKay. Cambridge University Press, 2003. ISBN 978-0-521-64298-9) doi: 10.5555/971145. ISSN: 01621459.
[92] Vasco Ronchi. Resolving Power of Calculated and Detected Images. Journal of the Optical So- ciety of America, 51(4):588 460, April 1961. ISSN 0030-3941. doi: 10.1364/JOSAA.51.04555 URL hwwp:htttp//ww..apppppteee sempoublissinng ishing.hnng.. wwwp/bsttaa.......ooooooo/ -ttp://aacttccurP111-oooo-------
[93] C.E. Shannon. Communication in the Presence of Noise. Proceedings of the IRE, 37(1):10-21 21, January 1949. ISSN 2162-6634. doi: 10.1009/IPPRCC.1449222222 Conference Name: Proceedings of the IRE.
[94] Massimo Franceschetti. Wave Theory of Information. Cambridge University Press, edition, November 2017. ISBN 978 978-1-139-13633-- doi: 10.1017/97811131116334 URL https
/wwww.cambrr ottp:///ororreeoooo//o//e//eeeee
[95] J. L. Harris. Resolving Power and Decision Theory. Journal of the Optical Sociery of America, 54(5):606, May 1964. ISSN 0030-3941. doi: 10.1364/JOSA.54.000060 URL
attps://opa optica. org/ awstractt.t..m. http://atttttttcaattttoooo 50--:os--555------ 5--6-
[96]
Heintzmann and Sarafis. Two point resolution in incoherent imaging. Optik, 112(3):114 1 118, 2001. ISSN 00304026. doi: 10. 10.10778003004402-00022 URL http::://iikkinnaeenee elsevier http////ttrreee////////000000000
[97] Joseph W. Goodman. Introduction to Fourier optics. Roberts & Co, Englewood, Colo, 3rd ed edition, 2005. ISBN 978-0097447077-2-. OCLC: 0cm56632414.
[98] Benigno Uria, Marc Alexandre Cote, Karol Gregor, lain Murray, and Hugo Larochelle. Neural autoregressive distribution estimation. Journal of Machine Learning Research, 17:1-37, 2016. ISSN 15337928. arXiv: 1605.02226.
[99] Mathieu Germain, Karol Gregor, Iain Murray, and Hugo Larochelle, MADE: Masked Au- toencoder for Distribution Estimation. In Francis Bach and David Blei, editors, Proceedings of the 32nd International Conference of Machine Learning Research, pages 881-889, Lille, France, July 2015. PMLR. URL sttps://prooceedings mlr, www.://ww......mmmm http///33//eemaainn5mmmmm html
Machine Learning, volume 37 of Proceedings
20

---

[100] Jongchan Park, David J. Brady, Guoan Zheng, Lei Tian, and Liang Gao. Review of bio- optical imaging systems with a high space-bandwide product. Advanced Photonics, 3(0)): 369-407, June 2021. ISSN 2577-5421. doi: 10.1117/1.AP344 1...7//11......444400.0 URL https:/7
1955.0001
[101] H. Nyquist. Certain Topics in Telegraph Transmission Theory. Transactions of the American Institute of Electrical Engineers, 47(2):117-644, April 1928. ISSN 2330-9431, doi: 10.
URL ptcttpp:/iieexpl.oooo ieee. Conference Name: Transactions of the American Institute of Electrical Engineers.
wwweee.or///cuueeeeeeeeeeeeeeeee org/document/5055022
[102] J. Beirlant, E. Dudewicz, L. Györfi, and Denes. Nonparametric entropy esti- mation. An overview. International Journal of Mathematical and Statistical Sci- ences, 6(1):17-39, 1997. URL https: https://www. semanticscholast oww.aaprrrgpppppprrr/e
[103] Lyudmyla F. Kozachenko and Nikolai N. Leonenko. Sample estimate of the entropy of a random vector. Problemy Peredachi Informatsii, 23(2):9-16, 1987. Publisher: Russian Academy of Sciences, Branch of Informatics, Computer Equipment and
[104] Damiano Lombardi and Sanjay Pant. A non parametric k-nearest neighbour entropy estimator. Physical Review E, 93(1):013310, January 2016. ISSN 2470-0045, 2470-0053. doi: 10.1 10.1/ PhysRewE.93.0133100 URL httt::/tpp://t// org/abs/ httg/abb/15000 1506.065011 arXiv:1506.0650) [cs, math].
[105] Jiantao Jiao, Weihao Gao, and Yanjun Han. The Nearest Neighbor Information Estimator is Adaptively Near Minimax Rate-Optimal, September 2018. URL URL htttp://axxi..org////////
1711. 08824 arXiv:1711.0883 [cs, math, stat].
[106] Kristina Monakhova, Stephan R. Richter, Laura Waller, and Vladlen Koltun. Dancing under the stars: video denoising in starlight. In 2022 IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 16220-16230, New Orleans, LA, USA, June 2022. URL https
IEEE. ISBN 978-1-665 378-1-66546-94633
doi: (tw.///CVV2228822022222202 //ieeexplore ieee wwtpeee.rr///oooooriieeee argg/doccmentt/87799777
[107] Alexander Krull, Tomáš Vičar, Mangal Prakash, Manan Lalit, and Florian Jug. Probabilistic
Nois:2Void: Unsupervised Content-Aware Denoising. Frontiers in Computer Science, 2:5, February 2020. ISSN 2624-9898.
doi: 10.3389/fcomp..200 wwwp::mmmmp.0.000000 20....00.0000.... URL https://www
frontiersin oaaeloorgggrrer. org/article/10 http:///...///ffommpppo/ 2020. 0000000000/fu11
[108] Ivan Kobyzev, Simon J. Prince, and Marcus A. Brubaker. Normalizing Flows: An In- troduction and Review of Current Methods. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(11):39644397,, November 2021. ISSN 0162-8828, 2160-9292, 1939- 3539. doi: 10. 109/TPAMM..00022299092 arXiv:1908.0925: [cs, stat].
URL http:t//t///ttttie org abs/ 1908 09257
[109] Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep Unsupervised Learning using Nonequilibrium Thermodynamics, November 2015. URL 03585 arXiv:1503.03523 [cond-mat, q-bio, stat].
[110] Yang Song and Diederik P. Kingma. How to Train Your Energy-Bast Based Models, 2021. URL arXiv:
http://arxiv.org/Baaa abs/a/as///00
[111] Lucas Theis, Aäron van den Oord, and Matthias Bethge. A note on the evaluation of generative models, April 2016. URL /RLp/ttp///arii http:///axxxvv.og/bbbbas/ htggabs//1111 01844 arXiv:1511.018.. [cs. stat].
[112] Diederik P Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. density estimation with diffusion models. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in neural information processing systems, 2021. URL. httphhttp:://tttt htttps://openrevier forum?ide iemm?id==2Lao 2dd=LLdBBxclW
21

---

[113] Christopher M. Bishop. Mixture Density Networks. Technical Report NCRG/94/004, Aston University, 1994.
[114] Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are universal approximators. Neural Networks, 2(5):399-666, January 1989. ISSN 08936080. doi: 10. 10.101600892--000089999000000000 URL /ttp:////// http://aiiiaaaaaaeeeeeeeeerrrrea p11/08836080899002088
elsevier cesevvier.com/retrieeve.
[115] Matthew D. Hoffman, David M. Blei, Chong Wang, and John Paisley. Stochastic variational inference. Journal of Machine Learning Research, 14(2):1303-132) 2013. ISSN 15324435. arXiv: 1206.7051.
[116] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied document recognition. Proceedings of the IEEE, (6(11)) 1998. ISSN 00189219. doi: 10.1109/5.72679
[117] Dongning Guo. The Interplay Between Information and Estimation Measures. Foun- dations and Trends@ in Signal Processing, 6(4):243-429, 2013. ISSN 1932-8346, 1932-8354. doi: 10. 10.1561/20000000018, URL http:://://o foundations- aatenns--nddtrdndd--aee -iipaddd-i--iinn--p--aaa--- ptw.ggaall--porssiinnn/S--------
com/ anticlss
[118] E.J. Candes and M.B. Wakin. An Introduction To Compressive Sampling. IEEE Signal Processing Magazine, 25(2):11-30, 2008. ISSN 1053-5888. doi: 10.1 1.100///SP22000.11773... arXiv: 1360v1.
[119] Sri Rama Prasanna Pavani and Rafael Piestun. High-efficiency rotating point spread functions. Optics Express, 16(5):3484, March 2008. ISSN 1094-4087, doi: 10. 1.64/0E.16.003444.. URL https://opol
awwt//atstttt.ccco wfw?URI=oe 1-----4-6---44
[120] Jerry Chao, E. Sally Ward, and Raimund J. Ober. Fisher information theory for parameter estimation in single molccule microscopy: tutorial. JOSA A, 33(7):B3-BB77, July 2016. ISSN 1520-8532. doi: 10. (0.133//0SAAA.33.00 33.0003.... URL httph::t///// //og. opti ca. jrg/josaa/ abstract. dffmuui=jossan
7-3-7-336
Publisher: Optica Publishing Group.
[121] Sri Rama Prasanna Pavani and Rafael Piestun. Three dimensional tracking of fluorescent microparticles using a limten-limited double-helix response system. Optics Express, 16 (26):22048-22557, December 2008. ISSN 1094-4087. doi:
URL Publisher: Op-
https://opg optica. org/ hcc/ www.aaatt.tfftruaaaaaaa.aa.. 16- 16-26---222 26-26-22048 lica Publishing Group.
[122] Yoav Shechtman, Steffen J. Sahl, Adam S. Backer, and W. E. Moerner. Optimal point spread function design for 3D imaging. Physical Review Letters, 13(3), September 2014. ISSN 107971 14. doi: 10. 10. 1..110//hyssell..c
113. 133902. Publisher: American Physical Society.
[123] Ginni Grover, Sri Rama Prasanna Pavani, and Rafael Piestun. Performance limits on three- dimensional particle localization in photon-limited microscopy. Optics Letters, 3301)) October 2010. ISSN 0146-9592, doi: 10. 1...33//OL.35.0033000 httpss//://// o/tp::////pppppcccaaaa.oo optica
org/ absttactt........ cfm?URI= 35- 19-119-3300 -306
[124] Sean Quirin, Sri Rama Prasanna Pavani, and Rafael Piestun. Optimal 3D single-molecule localization for superresolution microscopy with aberrations and engineered point spread functions. Proceedings of the National Academy of Sciences, 109(3):675-679, January 2012. ISSN 0027-8424. doi:
11133/paa.110901111
BRRphttps://pnaa www.ooggg//ooooooooooooo frg/diifuul111
1073/ 1073/pnas. wwtpppa......... 1109011109
[125] Elias Nehme, Daniel Freedman, Racheli Gordon, Boris Ferdman, Lucien E. Weiss, Onit Alalouf, Tal Naor, Reut Orange, Tomer Michaeli, and Yoav Shechtman. DeepSTORM3D: dense 3D localization microscopy and PSF design by deep learning. Nature Methods, 17(7): 734-740, July 2020. ISSN 1548-7105. doi: 10. (028/441592-000-00555- URL //www nature. hww.uree.ooo/aticle///////eeeeee
0853-5
Number: 7 Publisher: Nature Publishing
Group.
22

---

[126] Dorian Bouchet, Jonathan Dong, Dante Maestre, and Thomas Juffmann. Fundamental Bounds on the Precision of Classical Phase Microscopes. Physical Review Applied, 15(2):024047, February 2021. ISSN 2331-7019. doi: 10. Rew//hyyseeeApppleeeee 15.024........4......... 15.024047. URL https
//link. www.nkk.ps.orggg....oooo wwt.oo/g//ooo/oooooooooo wtw.:///./...haaeapppeee........
024047
[127] David Lee, Michael C. Roggemann, and Byron M. Welsh. Cramér-Rae Rao analysis of phase- diverse wave-front sensing. Journal of the Optical Society of America A. 16(5):1005, May 1999. ISSN 1084-7529, 1520-8532. doi: N0.334//OSAA.16.0000
URL https://opg
optica. org abstract cfm/URI=josaa-
[128] Jonathan Dong, Dante Maestre, Clara Conrad-Billrothh and Thomas Juffmann. Fundamental bounds on the precision of iSCAT, COBRI and dark-field microscopy for 3D localization and photometry. Journal of Physics D: Applied Physics, 54(39):394002, September 2021. ISSN 0022-3727, 1361-6463. doi: 10. 1088/ 000008811360 111///31116-1111 -64433/ac0122.
URL https://
10000.108811111 000000081111100 6463/ ac0f22
[129] Zhou Wang and A.C. Bovik. Mean squared enror: Love it leave it? A new look at Signal Fidelity Measures. IEEE Signal Processing Magazine, 26(1):98-117, January 2009. ISSN 1053-5888. doi: 10. 109/MSP2208.. 930649.
URL URI hattpp:/ieeexploos weee..eeerr..
document/4775863
[130] Richard D. Gill and Boris Y. Levit. Applications of the van Trees Inequality: A Bayesian Cramér Rao Bound. Bernoulli, 1(1/2))59, March 1995. ISSN 13507265, doi: 10.2307/3318881.
URL https::/:://ww: jttp:////wwww..orrrooooo jstor org. wttgsttalee//e/eooeegggogge htt.a//////11ee11oggii-------e-
[131] Efe Aras, Kuan- Yun Lee, Ashwin Pananjady, and Thomas A. Courtade. A Family of Bayesian Cramer-Rao Bounds, and Consequences for Log-Concave Priors, February 2019. URL
http://arxiv. org/ abaaabas/112 08582 arXiv:1902.08533 [cs, math].
[132] Trevor Hastie, Robert Tibshirani, and Jerome Friedman. The Elements of Statistical Learning. Springer Series in Statistics. Springer New York, New York, NY, 2009. ISBN 978-0-387- 84858-7. doi: 001107/978-0--87-848888 7. URL htRp:ttt://1ee springer. com/ 10. 1007/ 778-0-387-8433
[133] W. James and Charles Stein. Estimation with Quadratic Loss. In Samuel Kotz and Norman L. Johnson, editors, Breakthroughs in Statistics, pages 443-460, Springer New York, New York, NY, 1992. ISBN 978 177-0-387-9403777 doi: 10. 1007/9777 14411-199999555
URL Series Title: Springer
http://link: http:///ink.sprinneer com/ 10. 1007/ 1- 401-- 0919-8 5_30
Series in Statistics.
[134] Yann Lecun, Yoshua Bengio, and Geoffrey Hinton.
Deep learning.
Nature, URL
521(7533)436-444, 2015. ISSN 14764687.
doi:
10. 1tt38//nature44333
http: http://wwwr
http///wwwwcp..... scopus. http://wwwadrrrerrr.....eeeeeee
84930630277&
arXiv: 1312.6184v5
ISBN: 3135786504.
[135] Harry L. Van Trees. Detection, estimation, and modulation theory. Wiley, New York, 2001. ISBN 978-0-471-099117--0
[136] Z. Wang, A.C. Bovik, H.R. Sheikh, and E.P. Simoncelli. Image Quality Assessment: From Error Visibility to Structural Similarity. IEEE Transactions on Image Processing, 13(4): doi: 10.1109/TIP22003311666. URL http://
600-612, April 2004. ISSN 1057-7149.
ieeexplore ieee. .ww...rgeeeeeeeee.e.o........ htte//doummnnt1122 1284395/
[137] Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The Unreasonable Effectiveness of Deep Features a Perceptual Metric, April 2018. URL
http://arxiv.org abaaaaaaa//00 wttpa//www....gaaa////....1 03924 1rXiv:1801.03922 [cs].
[138] S Yu Efroimovich. Information contained in a sequence of observations. Problems in Information Transmission, 15:24 39, 1980.
[139] Kuan-Yur Lee. New information inequalities with applications to statistics. phd, University of California, Berkeley, 2022.
23

---

[140] D. Guo, S. Shamai, and S. Verdu. Mutual Information and Minimum Mean-Square Error in Gaussian Channels. IEEE Transactions on Information Theory, 1164)12211-122 April 2005. ISSN 0018-9448. doi: 10.1 0119/TTT.2205884400777 URL http:::///eeeeellrrrtttt ieee org/
document/1412024/
[141] Emmanuel J. Candès and Mark A. Davenport. How well can we estimate sparse vector?, March 2013. URL http:///axxii org/ aa//abbb//111 5246 arXiv:1104.5246 [es, math, stat].
[142] Ashish Bora, Ajil Jalal, Eric Price, and Alexandros G. Dimakis. Compressed Sens- ing using Generative Models, March 2017. http:/tp://are wwwgorgggaa...o.oo..oo.oooooee 1tte/1703.0322 MrXiv:1703.023 (es, math, stat].
[143] Xiaohan Wei, Zhuoran Yang, and Zhaoran Wang. On the statistical rate of nonlinear recovery in generative models with heavy-tailed data. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th international conference on machine learning, volume 97 Proceedings of machine learning research, pages 6697-6706. PMLR, June 2019. URL sttps://proceed/:
mlr. wwwppprssssse.pp wtt/wwi19b...... html
[144] Dustin G. Mixon and Soledad Villar. SUNLayer: Stable denoising with generative networks, March 2018. URL http::/arxivi org/ abw/aaab....././//// 1tt.s/9900000999999 arXiv:1803.09311 [cs, stat].
[145] Richard G. Baraniuk, Volkan Cevher, Marco Duarte, and Chinmay Hegde. Model- Based Compressive Sensing, December 2009. URL http://///aaxii www.orgg//bbbbeeeeeeeeee abaaabas////// 3572 arXiv:0808.3572 [cs, math].
[146] Martin J. Wainwright. Information-theerrtiicc limits on sparsity recovery in the high- dimensional and noisy setting, February 2007. URL 0702301 arXiv:math/0702301.
[147] Yihong Wu and Sergio Verdu. Rényi Information Dimension: Fundamental Limits of Almost Lossless Analog Compression. IEEE Transactions on Information Theory, 173:1322-3748 August 2010. ISSN 0018 9448, 1557-9654 doi: 10.1 00.11099/12200000000000. URL, http
//ieeexplore leee. org/ http///cctttttteeeeee
[148] D. Donoho, Adel Javanmard, and Andrea Montanari. Compressed Sensing via Spatial Coupling and Approximate Message Passing, January 2013,
Optimal
URL URL http:///aaaa org/ 1112/ 0708 arXiv:1112.074 [cond-mat, stat].
[149] Galen Reeves and Michael Gastpar. The Sampling Rate-Distortion Tradeoff for Sparsity Pattern Recovery in Compressed Sensing, June 2012. URL arXiv:1006.3128 [cs, mathj.
3128
[150] Aviad Levis, Daeyoung Lee, Joel A. Tropp, Charles F. Gammie, and Katherine L. Bouman. Inference of Black Hole Fluid-Dynamics from Sparse Interferometric Measurements. In 202/ IEEE/CVF International Conference Computer Vision (ICCV), pages 2320-2329, Montreal, QC, Canada, October 2021. IEEE. ISBN 978-1-66542-812-1 doi: 10. 109/ICCV48922.20011 00234. http::s////e/eeeeelllooo ieee orwpeee.rg/oooooeuee 99711339/
[151] Andrew A. Chael, Michael D. Johnson, Katherine L. Bouman, Lindy L. Blackburn, Kazunori Akiyama, and Ramesh Narayan. Interferometric Imaging Directly with Closure Phases and Closure Amplitudes. The Astrophysical Journal, 857(1):23, April 2018. ISSN 0004- 637X, 1538-4357. doi: 10.3847/1538. 4357/aab6a8. URL htttts:tt/tppsseee atticle/10.3847/1538-8 111111...4411550 01111113884477778 aab6a8
iop org/
[152] Yvette Y. Lin, Angela F. Gao, and Katherine L. Bouman. Imaging An Evolving Black Hole By Leveraging Shared Structure. In ICASSP 2024 2024 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages Republic of, April 2024. IEEE, doi: 10.11090CASSSP4445..220000
Scoul, Korea, 2044444482224..445866 URL https
//ieeexplore iee.eerrrreeeeee....... ieee. org/ dtcument//////o//o//om/c hete//114459//
[153] Alex Krizhevsky. Learning Multiple Layers of Features from Tiny Images. PhD thesis, University of Toronto, 2009.
24

---

[154] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q. Weinberger. Densely connected convolutional networks. Proceedings 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, 2017-lanua:2611-2266 2017. ISSN 0022-4790. doi: ISBN: 9781 9781538604571.
10.1 2099/CV22201..244. arXiv: 1608.0699 06993
25

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction accurately present the paper's contributions: developing a mutual information estimation framework for imaging systems that works without ground truth data, validating it across diverse applications, and demonstrating it can both evaluate existing systems and optimize new ones. Each claim substantiated with experimental evidence in the corresponding sections of the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The paper discusses key limitations including: 1) our approach helps optimize data capture but doesn't solve the decoding challenge ((Sec. (6)) 2) not all information contributes equally to specific tasks, shown in Section S2.8 future work could improve the accuracy-comput tradeoff's of our estimators ((Sec. 6). and 4) our supplementary mate- rials (particularly Figure S15) experimentally demonstrate specific failure cases of different models, showing when simpler Gaussian models are inadequate and more sophisticated models like PixelCNN become necessary.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they seale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
26

---

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
We did not prove any theoretical results this work
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We' ve provided extensive supplementary materials and documentation of methods, and all data and code is open source. The code is anonymized and included along with the submission.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
27

---

(c) If the contribution is a new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We include with our submission a zip file of the full (anonymized) codebase, including a README that links every experiment/fiure to the particular code and data needed to reproduce it.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://niis.cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips. wttpccccpppppup....eeeeee..o.cpt weeiii/gguiese//ee /ooeeSubmm ssiinni asionPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: In methods section, as well as included supplementary code that produces all results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
28

---

Answer: [Yes]
Justification: We provide error bars on all experiments where applicable, or we note that error bars are smaller than the size of markers shown.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi-
dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Figure S17 shows the compute needed. This is representative of all of our information estimation experiments. We also provide details about computational resources needed for other experiments in the methods section.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: This research complies with the NeurlPS Code of Ethics by ensuring proper data attribution and licensing, addressing limitations transparently, providing reproducible results with open-source code, and considering potential applications across domains without introducing harmful capabilities. The paper focuses fundamental research in informa- tion theory for imaging systems with applications in beneficial areas like microscopy and astronomy.
Guidelines:
29

---

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there is a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Since this work is foundational research that is not tied to any particular application, is unclear to us what societal impacts might be. However, we would gladly discuss any concerns the reviewers might point out.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out, For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
It's not clear to us what would be needed for this research.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
12. Licenses for existing assets
30

---

Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Licenses are explicitly mentioned the supplement/methodd where each dataset is first described.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's ss creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Yes, include all code and documentation of how to use it.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
31

---

15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
32

---

Supplementary materials
33

---

Contents
S1
Information encoding formalism
36
SI.1
Minimal probabilistic model of imaging system
36
ST..11 Separation of encoding and noise
37
S1.1.2
Visualizing high- nigh-dimensional distributions in energy coordinates
38
S1.1.3
Mathematical formalism for encoders
38
ST.2
Probabilistic two-point resolution
40
S1.3
Encoder inefficiency in 1D)
$1.4
Effects of signal-to-noise ratio, resolution, and sampling information
45
$1.4.1 Full mathematical treatment two-point resolution
47
S1.5
Expanded model
49
S2
Estimating information: Theory
50
$2.1
Background
50
$2.2
2
Mutual information estimator
................................
50
S2.3
Estimating conditional entropy measurement noise
................................
$2.3.1
Conditionally independent noise at each pixel
52
$2.3.2
Conditional entropy with additive Gaussian noise
52
$2.3.3
Conditional entropy with shot noise
53
$2.4
Estimating entropy of noisy images
54
$2.5
Probabilistic models
55
$2.5.1
Full Gaussian process
55
$2.5.2
Stationary Gaussian process
55
S2.5.3
PixcICNN
................................
56
$2.6
Sampling and likelihood evaluation
57
S2.7
Stochastic encoders
58
1.1
Encoder uncertainty
S2.7.2 Estimating information with stochastic encoders
S2.8
Task-specific information
60
83
Estimating information: Experiments
60
$3.1
Fitting stationary Gaussian processes
60
S3.2
Comparing stationary Gaussian process and Pixe CNN estimates
S3.3
Failures of stationary Gaussian estimates on highly non non-Gaussian data
63
S3.4
Model training times
67
$3.5
Conditional entropy estimates on noisy data
67
S3.6
Comparing analytic and upper bound entropy estimates
68
$4
Additional decoder experiments
68
S4.1
Experimental evidence of stochastic encoder effects
68
34

---

S4.2
spek-specific information in classification
68
S5
Comparison other frameworks
69
$5.1
Estimation theory and the Cramér-Rao lower bound
72
S5.1.
Background: The Cramér-Rao lower bound in imaging
72
S5.1.2
Cramér-Rao lower bound-based design and its limitations
73
S5.1.3
The challenges of generalizing estimation- theorettcc design
S5.1.4
Connections between estimation and information theory
75
S5.2
Compressed Sensing: Assumptions and Limitations
77
S6
Methods
77
$6.1
Model training
77
S6.2
Confidence intervals
78
S6.3
Sample generation
78
S6.4
Generation of synthetic experimental noise
79
$6.5
Applications
79
$6.5.1
Color Filter Array
79
S6.5.2
Black hole imaging
79
S6.5.3
Lensless imaging
80
$6.5.4
LED array microscopy
81
S6.6
Information Driven Encoder Analysis Learning (IDEAL)
S6.7
ID dimulations
82
35

---

S1 Information encoding formalism
Information theory enables quantification of uncertainty and randomness. A formal mathematical model of uncertainty and randomness requires probability. Thus in order to use information theory to analyze imaging systems, we must model them probabilisticlll.
This section assumes familiarity with probability and information theory fundamentals. A tutorial introduction these concepts can be found in previous work [90] as well as the textbooks 91 80 and Shannon's 's original paper [79].
S1.1 Minimal probabilistic model of an imaging system
The minimal probabilistic model present has been widely used in information -eemaatioo-thorettie analyses of imaging systems [13 22 30 39] and builds on established principles in statistical optics [40].
This minimal imaging system model involves three random variables (Fig. S1
The object has physical properties we want to measure
The noiseless image is created when the imaging system encodes object properties through physical processes (e.g., electromagnetic fields in optical systems, pressure waves in ultra- sound)
The noisy measurement results from detection noise corrupting the noiseless image
Each variable is modeled as random to capture our uncertainty about its true value.
Information can be lost at two stages: During encoding, information may be irretrievably lost when creating the noiseless image (e.g. spatial frequencies beyond the system's passband). During detection, measurement noise further corrupts information present in the noiseless image. The goal of imaging system is preserve much object information as possible through both encoding and detection.
Mathematically, these variables form Markov chain with joint distribution:
plo x,) = p())p(x o)p(y x)
where:
p(o) is the distribution possible objects
p(x o) a mapping from objects to noiseless images
p(y x) represents the detection noise process
The relationship between object, noiseless image, and noisy measurement can be understood in two directions:
Forward: Object Noiseless Image Noisy Measurement
p(o, y) = p())p(x)
Reverse: Noisy Measurement + Noiseless Image + Object
p(o, y. = p(y)) p(x))x(x())p(ox)
The Data Processing Inequality
80 reveals key limitations in each direction:
Forward: I(O:X) 2 I(O: Y) noise can only reduce information about the object.
Reverse: X(Y;X) > I(Y;)) the measurement's s ability to carry information about the object is limited by the information it carries about the noiseless image.
our model, encoding deterministic- each object maps to exactly one noiseless image (p(x () is a Dirac delta function). This means the noiseless image's 's randomness comes entirely from object uncertainty. Consequently, any information the measurement contains about the object must equal the information preserved through noise: I(Y: I(Y; O).
36

---

Deterministic Information Transfer Theorem. For a Markov chain O - X Y, if f(0) for some deterministic function f. then I(0; Y) [[: Y).
Proof. Since -) H(Y H ( X).
H and T(X = H(Y) H(Y))), we need to show
From the deterministic condition X s(O): knowing O completely determines X. Therefore, conditioning O is equivalent conditioning on both O and X:
H(Y X
From the Markov property O + X +Y
H(YOO, X
(6)
Therefore, H(YOO) = H(Y)X), which implies 7(O; Y) = (X;Y).
This equality is powerful means we can evaluate imaging system performance by measuring infor- mation between noiseless images and noisy measurements (X( Y)), without having to characterize more challenging distribution over possible objects (p(o)). While simplified object models can provide valuable insights as in the two-point resolution case (Sec. (S..4.) real objects are rarely known well enough to create accurate models. By focusing on / 7(X; Y), we can develop practical methods for estimating information in imaging systems.
Noiseless image
Noisy measurement
Object
O
X
Y
Encoder
Detector
O
Figure S1: A minimal probabilistic graphical model an imaging system. Three key variables describe the imaging process: (I) the object, which has unknown physical properties we want to measure, (2) the noiseless image, created when the imaging system encodes object properties through deterministic process, and (3) the noisy measurement, produced when detection adds random noise to the noiseless image. The arrows show how information flows: the measurement can only reveal information about the object if that information was preserved in the noiseless image.
S1.1.1 Separation of encoding and noise
This minimal model, which separates image encoding from detection noise, has been widely used in analyses of imaging systems [13] 22 30 39] and general imaging models [92] For optical systems, this separation justified by the classicaa approach common statistical optics [40] This method treats light classically until it reaches the detector, where quantum effects introduce noise. Optical systems can experience two types of noise:
Classical noise from fluctuations in light intensity (e.g., thermal light from bulb intensity variations in laser light)
Quantum "shot" noise from the random arrival times of individual photons the detector
In the visible spectrum, quantum noise typically dominates classical noise. This justifies treating light propagation as deterministic until reaches the detector, where quantum effects become significant. This physical reality supports separating encoding and detection in optical systems.
37

---

S1.1.2 Visualizing high-dimensional distributions in energy coordinates
The probability distributions over images (p(y)) and measurements (p(x)) are high- nigh-dimensiona,, and thus not possible to visualize directly. In microscopy, for example (Fig. (22a), microscope transforms a distribution of possible cells into noiseless images, which detection then converts into noisy measurements. To provide insight into these complex distributions, adapt representation from [93] (Fig. (S2)). This representation, which we call energy coordinates, offers a complementary perspective to the traditional spatial coordinate representation, enabling a more comprehensive understanding of image distributions and information flow in imaging systems.
Traditionally, images are represented in spatial coordinates the normal way to show images, where each pixel's 's intensity is plotted at its corresponding position in 2D space. In this representation, each displayed image is single sample drawn from the underlying probability distribution of all possible images. While intuitive and familiar, this representation doesn't 't directly convey the statistical properties of image distributions, it only shows or few instances a time rather than the full range of possibilities and their relative probabilities.
Alternatively, image distributions can be visualized in the energy coordinate representation, which shows the probability mass over pixel values. In this representation, each image of D pixels is a D-dimensional vector, and the D-dimensional probability density function describes how likely each image is. low-dimensional projections of the full distribution, such as the marginal distribution p(zx+) or the joint distribution p(zR. xy) of two pixels can be plotted and provide insight into the full D-dimensional distribution's 's behavior.
Visualizing the distribution of noisy measurements for a single noiseless image (i.e., ply x)) in energy coordinates demonstrates the effect of measurement noise (Fig. (S2c). Noise spreads the probability mass further from the point corresponding to the noiseless image, which increases the overlap between noisy distributions of different images. This overlap makes it harder to determine the true object from noisy measurement, with the extent of overlap determining how much object information is lost due to noise corruption.
The energy coordinate representation is particularly useful for comparing different imaging modali- ties (Fig. (S.)). For example, when comparing two microscope illumination patterss with different spatial coherence, energy coordinates reveal how the encoders affect image distinguishability. An encoder using spatially incoherent illumination maps different objects to more similar images, re- sulting in overlapping measurement distributions that are difficult to distinguish. In contrast, an encoder using spatially coherent illumination creates more distinct images, leading to measurement distributions that remain separable even in the presence of noise. This visualization demonstrates how different illumination patterns affect the information content of the resulting measurements, with more distinct encoded images being more robust to noise corruption. This concept of distinguishability in the presence of noise identical to the two-point resolution problem (See Probabilistic two-point] resolution). only with spatial coherence the variable parameter instead of signal-to to-niss ratio and resolution. This demonstrates how the ceermatiionthhoretee approach comprchensively captures imaging system performance while abstracting away the specific physical details of different imaging modalities.
$1.1.3 Mathematical formalism for encoders
There is a family of encoders & consisting of functions ég O - X, where O is the domain the space of possible objects, X is the codomain the space of possible noiseless images, and 0 is the parameter(s) that define a particular encoder. For example, in the case of a linear, shift-invariant encoder, E would be the set of all linear, shift-invariant functions and 0 would define a specific point spread function.
The action of an encoder is to take an object E O and form noiseless image E X of it. This noiseless image will then undergo a measurement process, resulting in a noisy measurement y E Y. The measurement process is modeled as conditional probability distribution Pvix, which describes the probability of observing a particular noisy measurement given a particular noiseless image.
The information carried by noisy measurements is determined by the distributions PYzx and Px. quantified by the mutual information between the noiseless image and the noisy measurement, I (X; Y). which can be expressed as:
38

---

a) Probabilistic model of imaging system
c) Visualizing measurement noise
Measurement
Noise level
Object
Image (noiseless)
Probabillty
(noisy)
20 Energy Coordinates Coordinates
Spatial
Probability
Encoder
Detector
x (photons))
Probability
X (photons)
X, (photons)
(photons)
d) Dissimilar images are more robust to noise
b) Visualizing distributions
information
More information
Spatial Coordinates
Energy Coordinates
Image
Image
Measurements
Few samples
All samples
All pixels
Few pixels
Probabllity
pixel
Photons
2 pixelss
(photons)
Probaslity
x, (photons)
pixets
100
Probabiity
(photons)
100 (photons) x, (photons)
100 0
x, (photons)
100 0
100 K, (photons)
Figure S2: Probabilistic modeling and visualization of imaging systems. a) Probabilistic model of microscopy, which a distribution of objects (cells) are encoded to distribution of images by microscope and detected as noisy measurements. b) Two complementary ways to visualize measure- ment distributions: spatial coordinates show all pixels for few samples, while energy coordinates show the full probability distribution for selected pixels. Visualization of how measurement noise affects distributions: as noise increases, measurement probability spreads further from the noiseless value. d) Demonstration of how encoder design affects information preservation: more distinct noiseless images (right) create measurements that remain separable even with noise, while similar images (left) produce overlapping distributions that make object discrimination harder.
P(X, Y) p(X)p(Y))
Y(; Ex,y log
where y) is the joint probability of observing particular noiseless image and particular noisy measurement, and p(x) and p(y) are the marginal probabilities of observing a particular noiseless image and particular noisy measurement, respectively.
Better encoders will produce noisy measurements that contain higher mutual information. it thus of interest to investigate what limits the mutual information. One way of doing this is based on decomposition of the mutual information into two terms:
Y) = H(X Y)
(X) is the entropy of the noiseless images, and H(X Y) is the conditional entropy of the noiseless images given the noisy measurements, which quantifies the uncertainty about the noiseless images that remains after observing the noisy measurements.
39

---

There are multiple ways of mathematically modeling the space of noiseless images X, which depends on whether the images are continuous or discrete over space and in energy. For example, model of noiseless images over continuous space and energy would be space of continuous functions over space which output real numbers corresponding to an amount of energy, whereas a discrete model would be space of finte-dimensional vectors that take discrete values (the number of photons) at discrete set of locations (the pixels). Combinations of these are also possible, such as a continuous model over space and discrete model over energy, or discrete model over space and continuous model over energy.
With any model, the space of noiseless images is ultimately finite in some sense. Energy cannot be infinitely concentrated in single point" and the physics of pave-propagation effectively constrain electromagnetic waves to finite number of degrees of freedom [94]. This means that even in the continuous/contiiuuus case, any noiseless image can be represented to an arbitrary level of precision by finite number of samples.
It thus of interest to understand the limits of the entropy of the noiseless images, because this will determine the limits of the mutual information. The space X will have either finite volume or finite cardinality, as dictated by physical constraints. A uniform distribution over this space, in which all noiseless images are equally likely. will have the maximum possible entropy. However, due to their physical constraints, encoders will not in general be able to produce noiseless images that are uniformly distributed over X, leading to an inefficiency in the amount of information that can be carried by the noisy measurements.
S1.2 Probabilistic two-point resolution
To illustrate information- thhoreeic analysis of imaging systems, we first provide simplified example: the classic two-point resolution problem. Typically, resolution determined by the Abbe diffraction limit, which says that two inffraction-limited point sources will be "resolved" if they are sufficiently far apart (at least 2/(2N A), where NA the numerical aperture). The Abbe criteria, however, does not take into account the effects of noise, aberrations, coherence, pixel sampling, among other factors. Our mutual information analysis can naturally incorporate all these effects into a single comprchensive metric.
We demonstrate with a simplified scenario, where the task is to distinguish between one point source or two closely spaced dimmer point sources, in the presence of measurement noise. We can derive the information content analytically for this idealized case (see (S14.1)(S1.4.1) in which we model the object as either one point or two half-energy points, with equal probability [95] (66.. We assume that the imaging system has a diffraction-limite by additive Gaussian noise (Fig, S3). This simplified setup allow us to examine how information content accounts for both resolution and signal-to -o-ine ratio.
point spread function, with measurements corrupted
We seek to find the minimum numerical aperture needed to resolve two points that are spaced öd apart. The Abbe diffraction limit says the numerical aperture should be at least \/(2)d). However, without noise we can theoretically resolve any arbitrarily spaced points. Conversely, if there is lot of noise, we need larger numerical aperture than the Abbe limit predicts to resolve these same two points. Below, we show how mutual information can quantify this interplay between resolution and noise (Fig.
predicting the achievable classification accuracy (Fig. S3D).
With two equally probable object states, the maximum possible information is bit. Since the optical system is deterministic, it introduces randomness and /(O; Y) I(X; Y) exactly, because the randomness in the object distribution is the only source of randomness in the noiseless image. This allows us to directly calculate the information preserved through the imaging process. Unlike the general case where we must estimate the entropy of measurements H (Y) through a cross-entropy upper bound, in this scenario we can derive the probability density of measurements analytically:
1 ply) -
N(yxx,, 021) +
+ Ny: 021)
1
w..thohuut collapsing space into black hole 94
40

---

Here, X1 is a vector of pixels for noiseless image of single point, X₂ is vector corresponding to two points, and (y: p. Σ) is the multivariate normal probability density function. o² represents diagonal covariance matrix with additive Gaussian noise with variance o² at each pixel.
Using this probability density, we can generate N samples from the distribution of measurements and use them to compute the entropy of measurements H (Y) using a Monte Carlo approximation:
HY)) = E |- logp Y)
N
log logp(y:)
N
In this case of additive Gaussian noise, H( also has a closed form analytical expres- sion (Sec. (S2.32).. Subtracting these two quantities enables computation of mutual information with error converging to zero as the number of samples increases.
In Figure (S3,, we plot the mutual information for a range of diffraction-limited resolutions and signal-to-noise ratios. As expected, there a trend towards more information for higher resolution (higher numerical aperture) systems and higher signal-to-noise ratio measurements. However, a high- resolution measurement can contain less information than low-resolution one, the low-resolution one has a sufficiently better signal-to-noist ratio. When mutual information is equal to bit, the measurements can be classified as coming from one or two point sources with perfect accuracy. When mutual information is zero, the measurement contains no information about which of the two scenarios it is, so the best classifier will only achieve 50 percent accuracy through random guessing. Figure S3b shows this relation between mutual information and classification accuracy, after deriving an optimal classifier analytically. As expected, the more information preserved in the measurement, the better an optimal classifier can perform.
This two-point resolution example demonstrates how our unified treatment with information theory can evaluate resolution limits with noise effects taken into consideration. Such analysis can be extended more complex imaging scenarios, where many different factors influence the encoder (e.g. aberrations, filters) and the measurement quality (e.g. spectral sensitivity. coherence). While our analysis here can provide valuable intuition about when two points will be "distinguishable" " real imaging scenarios involve high- high-dimensional measurements of complex objects. The key insight car- ries over: better encoders create measurements that are more distinguishable the presence of noise. The geometric interpretation of these high- -ishddimensoonll probability distributions provides additional insight into why information theory is well-suited for analyzing imaging systems (Sec. (Sec.(S1.1.2) (1.112)..
S1.3 Encoder inefficiency in 1D
To understand how physical constraints limit information encoding, we expand on the ID two-point resolution example to analyze a more general ID model system of an imaging system. This example reveals a fundamental concept we call "encoder inefficiency" the gap between theoretically optimal encoding and what physical constraints allow.
The family of encoders E studied were 1D bandlimited, nonnegative, linear-shift invariant, infinitely periodic point spread functions. Figure S4aa shows the outputs of representative encoder in this family (i.e. a specific point spread function) acting on a distribution of delta function objects. This system be thought of simplified version of an imaging system that uses spatially incoherent illumination, such as in photography or microscopy (9].
The set of possible output signals X for this family of encoders is identical to the set of possible point spread functions: bandlimited, nonnegative, and infinitely periodic signals (Fig, (Fig.(S6b). These output signals (which are analogous to the noiscless images in discussed in Mathematicall Tramework) can be viewed either in spatial coordinates, in which their energy density is plotted as a function of space, or in energy coordinates (Fig. (2). with values found by integrating areas of the signal corresponding to "pixels." "
Signals cannot have both finite bandwidth and finite extent in space, and they thus require infinite samples to represent with arbitrary accuracy in the absence of further constraints. This is typically handled in one of two ways: treating signals as both band-limited and spatially finite and sampling
41

---

Probabilistic model of 2-point resolution
Information limits
optimal classifier
Probability
Object
Point spread function
Image Measurement
Optimal Classifier
Binary
One
classifier
point
Accuracy
0.5
Ad
Energy
Two point
Position
Position
Position
Mutual Information
(bits)
c) Information captures effects of resolution and noise
Measurement
Image
Sample
Distribution
10
Signal-to-Nooss Ratio
Low resolution, low noise
x, Ohhtoons ss
Photons
Probability
10
I Ad (true
Low resolution, high noise
separation)
X, (photons)
Photons
600
400
200
Diffraction-lmiited
resolution (nm)
High resolution, high noise
Mutual Information (bits)
x, (photons)
Photons
X,X
Position
x, (photons)
Position
Figure S3: Mutual information captures imaging trade-ofl's between resolution and noise for simplified case of two-point resolution. a) Probabilistic model of two-point resolution: an object (one point two half-energy points) is blurred by a point spread function and corrupted by noise. binary classifier attempts to determine the object type from the measurement. b) The accuracy of an optimal classifier is fundamentally limited by the information content of the measurements. e) Information analysis reveals how resolution and noise interact: (Left) information in measurement as a function of signal-to-noise ratio and diffraction-limtted resolution. (Right) Example images, measurements, and measurement distributions (at pixel locations # and 22). The top row and bottom row achieve equivalent information with low-resolution, low-noise measurements and highuressouutin, high noise measurements.
42

---

a) Model system
b) Output space: positive, bandlimited, periodic signals
Object
Encoded signals
(delta function)
Energy density
Energy density
Point spread
o
function
Space
Space
Space
Energy densiyy
Energy in x,
Space
Energy density
Valid signals
Energy dersity
Space
Energy in X
Space
Space
Figure $4: 1-dimensional model system. a) An object distribution (e.g. randomly-located delta function) and a linear, shift-invariant point spread function encoder. b) M-dimensional energy coordinate representation of the set of all encodable signals in which each signal corresponds to single point.
finite density over finite extent; or considering band-limited signals with infinite spatial extent, which asymptotically have finite degrees of freedom and can thus be sampled at finite rates with arbitrary accuracy 194]
Here, we avoid these complications by considering band-limited signals that are infinitely periodic in space. This means the signal can be represented exactly by sampling at the Nyquist rate over a single period (because sampling beyond this period would yield the same values).
As a result of this simplification, there is a bijective mapping between the set of possible signals and non-negative vectors in D-dimensional space-thal that is, each signal in the set corresponds to point in D-dimensional space (similar to the argument made in [33). This allows us to computationally analyze this mime--dimeniioall space with insights that can be applied to the more complicated space of continuous signals.
As discussed in Visualizing high- nigh-dimensional distributions in energy coordinates the effect of measurement noise in the energy coordinate representation is turn point (i.e. a signal/image) into cloud of probability mass representing the possible noisy realizations of that signal/image. Here we assume, without loss of generality, that measurement noise additive Gaussian (See Conditional entropy with additive Gaussian noisep. The amount of information that can be encoded is determined by how well dispersed the distribution of encoded signals can be in this space such that the noisy versions of different signals minimally overlap. Thus, the volume of the space of possible signals is critical: it determines how much room there is to map different objects to non-overlapping signals. Though there distinguished with given level of certainty in the presence of noise.
infinite number of signals in the set, only finite number can be
What is the volume of the space of possible signals? Given that all signals have energy S 1, the vector that defines their representation energy coordinates must have In norm also S 1. Thus, all signals must correspond to points that lie inside the positive orthant of the In unit ball. However, not every point in this space will correspond to a valid signal: for example, a vector with a single 1 and the rest Os will not be possible, because this would entail concentrating all of the signal's energy within a single pixel.
It is unclear to if there is an analytical expression that defines the set of possible signals, but we can investigate the size of this set empirically. To do so, we set up an optimization problem in which we pick a fixed object and target energy coordinate representation of signal (e.g. a vector with a single and the rest Os). We then find the optimal point spread function that brings the object closest to the target signal by optimizing an encoder to minimize this distance using gradient descent. Repeating this experiment over grid of target signals shows which signals can be reached, and which cannot, thereby revealing the limits of the space of possible signals.
43

---

Repeating this experiment with different fixed objects illustrates an important insight about encoders: Their range is object dependent (Fig. (S5). This is direct result of their physical constraints. The ID encoder in this simulation is representative of imaging systems governed by intensity point spread functions [97]. Such encoders have at least two important physical constraints: 1) they can only reduce energy (if they attenuate light) or preserve it. They cannot, for example, encode dim object to signal with greater energy. 2) They can only disperse, and not re-concentrate, energy. Every point spread function (under the constraints of non-negativity and linear shift invariance) can only map objects to signals that are blurrier versions themselves.
Figure [S5] shows the consequence of the second constraint for three different objects with equal energy. The single delta function in the top row can be encoded to the broadest range possible signals, since it is the most concentrated to begin with. More dispersed objects, like the 8 delta functions each with g energy the bottom row can only be encoded to smaller volume of possible signals.
Object
Possible Signals
Target signal Closest encodable
Energy density
Energy at x
Energy density
Energy x, X,
Any point spread function
Energy density
Energy a X,
Space
Energy at
Figure S5: Object-dependence of encoder range. The set of signals encodable from an object depends on that object's 's properties. More dispersed objects like the 8 delta functions (bottom) can only reach subset of the signals reachable from concentrated delta function (top), despite equal energies. This is due to physical constraints preventing reconcentrating energy. The closest encodable signals to the target signals were found by solving an optimization problem to find the optimal encoder that encodes the object to the target signal.
The range of a family of encoders for fixed object of particular type determines the volume of the set of possible encoded signals, and thus places upper limit on the amount of encoded information. Physical constraints limit the functions an encoder family can implement. As result, even optimal encoders generally cannot achieve the theoretically ideal signal distribution (which would be uniform over all possible signals when dealing with additive Gaussian noise). An encoder must handle not just one object, but an entire distribution of objects. While we might find different encoders that each map a specific object to a desired signal, no single encoder can optimally map all objects simultaneously. This fundamental limitation prevents achieving the thcoretically ideal signal distribution.
44

---

Given three constraints an object distribution, an energy limit on signals, and noise model the optimal signal distribution maximizes mutual information between objects and encoded signals. However, physical constraints and object-dependence often prevent encoder families from achieving this optimum. We define "encoder inefficiency" as the gap between this theoretical optimum and the best distribution achievable by given encoder family.
We can measure encoder inefficiency experimentally using simplified test case with delta functions of unit energy at random positions as objects, and additive Gaussian measurement noise. Under these conditions, the distribution of signals carrying maximum information has uniform probability over the set of possible signals.
To find the best achievable distribution, first use Information Driveen Encoder Analysis Learning (IDEAL) (See Encoder design via information maximization with IDEAL) to learn the optimal encoder. We then generate random objects and encode them with this optimal encoder to sample from the distribution of encoded signals, add noise, and estimate the mutual information. We use the FixlCNN-basee estimato treating the signals as 2D images match the estimator's 's design.
Signal distribution
Example signals
Optimal signal distribution
100
Energy a
Probability
Energy Meesann
Mutual informmaion (bits)
80-
Optimal signal distribution
Encoder inefficiency:
Gap between optimal and physically achievable
60
40
Best encoded distribution
Best encoded distribution
Probability
20
Energy
Energy
10
20
Signal bandwidth (Number of Nyquist samples)
Energy
Space
Figure S6: Physical constraints limit the amount of information that can be encoded. (Top left) The optimal distribution of energy limited signals for additive, signal-independent Gaussian noise is uniform (it appears non-uniform in this 2D projection of a 4D space). (Bottom left) The best encodable distribution using optimal point spread function is far from uniform. (Right) The gap between the information in the optimal signal distribution and the best encodable distribution quantifies the inefficiency of the encoder family.
The results of this experiment are shown in Figure S6 The optimal point spread function encodes objects to only a small fraction of the total volume of possible signals, resulting in encoded signals that are significantly less random in appearance the the optimal uniform distribution of signals. (Note: the energy coordinate representation of these signals shown on the left side a 2-dimensional projection of D--dimensional space, with D 4 for the picture shown, which is why the distribution appears non-uniform.) The mutual information estimates shown on the right side demonstrate the gap between the optimal distribution of signals and the best encodable distribution given the family of linear, shift-invariant encoders.
S1.4 Effects of signal-to-noise ratio, resolution, and sampling on information
This same experimental setup can used to investigate how different properties of the en- coder/detection process, such bandwidth, signal-to-noiss ratio, and sampling density, affect the amount of information that can be encoded. These experiments provide insight into the tradeoffs between different system parameters well connections the existing literature analyzing information in imaging systems.
We used PixeICNN for convenience, though estimators designed for ID signals 98 better accuracy.
99
might provide
45

---

hignal-to-noise ratio Signal-to Signal-to-noise noise ratio is key parameter in imaging systems, and it is widely- appreciated that higher hignal-to-noise ratio is a desirable characteristic. For the additive Gaussian noise measurement model, noise is fixed, and the signal-to-noise ratio is determined by the amount of energy in the signal. simplest to consider the average signal-to-noiss ratio over each signal, which can be found by dividing the energy of the signal by the standard deviation of the noise. Choosing an maximum average signal-to-noise ratio defines a set of signals that can be encoded, which consists of all possible signals with this average signal-to-noise ratio less. This set has finite volume, and thus a finite maximum amount of information that can be encoded. Within this set, there are subsets of signals that each have the same signal-to-an -o-iee ratio.
a) Experimental framework
Noisy measurements
Object distributions
Clean signals
0.04
Dela Energy den
0.01
IDEAL-deseign ed point spread function
0.01
While nolse
Energy density Energi denslty
0.04
8 Dellas
Space
A00
Space
Space
Space
Space
Space
b) Effects of measurement properties on information
3004
10"
500
Deltas
600
Nyquist sampled
Signal-to-noise ratio
Deltas
Mutual Information
Mutual Information
Mutual information
Delta
(bits)
White noise
(bits)
(bits)
Delta
White
10"
10P
10a
10+
10
20
30
Signal-to-noise ratio
Bandwidth
Number of samples
(Number Nyquist samples)
Figure S7: Effects of signal-to-noise space-bandwidtt product, and sampling density on en- coded information. a) Samples from three different object distributions and the signals and noisy measurements which they encoded with optimal point spread function. b) The amount of encoded information increases (Left) logarithmically with the average signal-tonnoise ratio with object-dependent rates, (Middle) linearly with the space- macce-badwwith product of the signal with object- dependent rates. (Right) Sampling signals of fixed bandwidth at increasing densities increases the amount of encoded information, but with diminishing returns across different signal-to-noise ratios (for the delta object distribution).
Sets of signals with higher average signal-to agnise ratios have increasing large volumes, and can thus carry larger amounts of information. described in the previous section, physical constraints of encoders impose object-dependent limits on the the maximum amount of information that can be en- coded. To test how the maximum average signal-to-noise ratio affects the amount of information that can be encoded, we repeated the procedure described on three different distributions of objects: single, citoomly-locctedd unit energy delta functions, randomly-located, delta functions each with & energy, and unit energy white noise patterns (note, these objects, not measurement noise) (Fig, S7a). The
46

---

results show that for all three object distributions, the best encodable distribution of signals grows logarithmically with the average signal-to-non to-nosse ratio, with different objects having different absolute amounts of information for a given signal-to-nois ratio (Fig. (S7p, left). This is consistent with the intuition that higher signal-to-noise ratio allows for more information to be encoded, and that the amount of information that can be encoded is object-dependent.
produet Next, we tested the effect of signal bandwidth on information capacity, Optical imaging systems are often characterized in terms of their space-bandwidth product [TOO], and the mpace-bandwidth product is often used synonymously with the word "information." A more accurate term for the bandwdban product "degrees of freedom," since it quantifies the potential complexity of an electromagnetic field wave propagating through the system [94]. Information (in Shamnon's 's entropy/mutual information sense) also depends on the signal-to-noise ratio and the object- dependent ability of encoders to map to distinct signals. In our numerical simulation, the spatial extent of signals is fixed, so increasing the bandwidth of the signal increases the space- bandwidth product. We found that captured information increases linearly with the space-bane bandwbh product, with rate of increase depending on the object distribution (Fig. S7p, center).
Sampling density Finally, we examined the effect of sampling density on the amount of information that can be encoded. For fixed bandwidth, the sampling density determines the number of pixels in the signal. The Nyquist sampling theorem [101] [93] states that a bandlimited signal can be perfectly reconstructed from its samples if the sampling density is at least twice the bandwidth. However, in the presence of noise, even when sampling at the Nyquist rate, there remains residual uncertainty about the signal, and additional samples can reduce this uncertainty. Experimentally, we found that, as expected, increasing the sampling density increases the amount of information, even beyond the Nyquist rate (Fig. S7 p, right). However, the additional increases in information were progressively smaller. These results were consistent across both different object distributions and different average signal-to-nen ratios (for delta function object distribution).
S1.4.1 Full mathematical treatment of two-point resolution
In this section we describe the example of I-dimensional two-point resolution (See (Peobbbbbaii two- point resolution) in full mathematical detail. By making assumptions about the object distribution, the encoder, and the noise model, we can write down the exact probability density functions of the object, the noiseless image, and the noisy measurement. This enables writing exact expression for the mutual information between object and noisy measurement, as well as the classification accuracy of the optimal binary classifier decoder that uses the noisy measurement classify the object as being a single point source or two point sources.
Object The object is a mixture of two possibilities that each occur with probability }: Either a single point source with energy 1 or two point sources with energy 1 and separation distance A with their midpoint the same location as the single point source. We represent these objects mathematically as and og respectively. Using to denote spatial position, with the single-point source object located 0:
(r))) = 8(r)
1
+ 0(r + )
o2 (r) d(r
++2200
2
2
where 5 is the Dirac delta function.
The random object O has probability 1 of being 0₁ and | of being og. Thus, its probability density function can be written as:
1 5(0 01) 5(0
p(o)
+,10000000 o2)
2
8(0 ((o)))) is 1 when the object is 0r and 0 otherwise, and similarly for 5(o -02).
47

---

Encoder The encoder is a 1-dimensional linear shift-invariant imaging system with diffraction- limited intensity point spread function h(r):
sin( 2
h(r)
2aNA
where NA is the numerical aperture of the system, À is the wavelength of light, and is the spatial coordinate. The noiseless image is the convolution of the object with the point spread function:
x = O h
This gives rise to two possible noiseless images, X and X2, corresponding to the two possible objects 0r and O2 respectively. The probability density function of the random noiseless image X is thus:
1 2 20(x x1) ++700x x2)
1
pxx)
Detector The noisy measurement is formed by adding independent Gaussian noise with variance o² to each pixel of the noiseless image. We assume pixel size much smaller than the minimum pixel size dictated by the Nyquist sampling theorem, so sampling effects minimally influence the results.
The random noisy measurement Y is thus a length D vector of pixels. Its probability distribution is found by taking the mixture of two deltas distribution of the noiseless images and adding Gaussian noise to each pixel. This gives a mixture of two multivariate Gaussian distributions with mean vectors given by the noiseless images and a diagonal covariance matrix with variances equal the noise variance. The probability density function of the noisy measurement is thus:
1
1
p(y) = - + NN; o21)
where M(y:x, is the multivariate Gaussian distribution with mean vector = X and covariance matrix E = I.
Mutual information The mutual information between the object and the noisy measurement T(O; Y) is equal to the mutual information between the noiseless image and the noisy measurement T(X; Y), since the object is fully determined by the noiseless image. We focus on the mutual information between the noiseless and noisy images, which can be calculated by decomposing it into a difference of entropies:
H(Y) H(Y()) where H(Y) is the entropy the noisy measurement and (Y (X) is the conditional entropy of the noisy mcasurement given the noiscless image.
is constant that a function of the the variance o² and the number of dimensions (pixels) D. It can be analytically simplified as shown in Equation 24
Under the additive Gaussian noise model, H H(Y X)
The entropy of the noisy measurement H (Y) can be expanded as:
H(Y) = Ellog p(y)]
Since casily generate samples from the distribution of the noisy measurements and we know the true probability density p(y), we can estimate this entropy with a Monte Carlo approximation of N samples:
N
H(Y)
log p(y:)
48

---

Decoder Since the goal of the imaging system in this simple example is to classify the object as being single point source or two point sources [95], the decoder is binary classifier that takes in the noisy measurement and outputs a decision as to whether the object was single point or two points.
The optimal decoder is the Bayes classifier, which chooses the object class that maximizes the posterior probability given the noisy measurement. In this case, since the prior probabilities of the two object classes are equal, the Bayes classifier is equivalent to the maximum likelihood classifier, which chooses the object class that maximizes the likelihood of the noisy measurement.
Given the probability density of the noisy measurement in Equation 9 the Bayes/maximum likelihood classifier decides the object is two points if:
X2 d²1) > N(y;
Plugging in the expressions for the multivariate Gaussian distributions, this simplifies to:
ly X2 ly ||
In other words, the noisy measurement y is classified two points if its Euclidean distance the noiseless image X₂ is less than its distance to X1, and classified as one point otherwise.
Using the analytic expressions for mutual information and classification accuracy, they can be shown to have a monotonic relationship with each other (Fig. (88). This demonstrates the fundamental link between the information content of the measurements and the achievable performance of downstream decoding in this minimal example.
Optimal Classifier Accuracy
0.5
Mutual Information (bits)
Figure S8: Classification accuracy vs information in two-point resolution. Performance of the optimal binary classifier decoder that uses the noisy measurement to classify the object being single point source or two point sources. The classification accuracy this simple example has monotic relationship with the mutual information between the object/noiseless image and the noisy measurement.
$1.5 Expanded model
Our basic framework works for many imaging systems, including many optical systems whose purpose is to capture the object in much detail as possible. Here we present expanded model that accounts for specialized cases involving stochastic encoders and task-specific information, with detailed treatments provided in subsequent sections.
The minimal model assumes deterministic encoders where the same object always produces the same noiseless image. In specialized cases, this assumption may not hold due quantum effects in image formation classical system variations like mechanical drift and illumination fluctuations. We provide theoretical framework for handling such stochastic encoders in Section S2.7
49

---

Noiseless image
Noisy measurement
Task estimate
Object
X
Y
T
Encoder
Detector
Decoder
T
E
Task
Figure S9: An expanded probabilistic graphical model of an imaging system. A generalization of the minimal probabilistic model Figure(ST which in addition to modeling the object, noiseless image, and noisy measurement as random variables, also models randomness of the encoder, and the true and estimated values of the decoder task.
Additionally, the minimal model treats all object information equally valuable, but many imaging applications focus on specific tasks that may require only a subset of the total object information. We develop extensions for task- information estimation in Section (22.8
S2
Estimating information: Theory
In this section, we develop the mathematical framework for estimating information in imaging systems.
S2.1 Background
As described in Mathematical framework] our probabilistic model represents noiseless images as a random 2D array X with distribution p(x). where each element quantifies the energy single pixel. The corresponding noisy measurements form array Y with distribution p(y). These distributions determine how much object-relevant information survives the measurement process.
We estimate (X; Y). the mutual information between noiseless images and noisy measurements. The units are bits, which have concrete operational meaning: in ideal conditions, each bit enables perfect discrimination between 2x(:Y) different objects. In practice, noise prevents such perfect dis- crimination-inn instead, each bit quantifies how well multiple object states can be partially distinguished from each other.
While imaging system performance typically varies across the field of view, information content using a single value: the mutual information rate (information per pixel). This par- allels how optical resolution is often characterized by a single number despite spatial variations 97
quantify overall
S2.2 Mutual information estimator
Mutual information estimation a well-studied problem in many fields, including machine lcarning, neuroscience, and computational biology, and number of different approaches have been proposed, some of which attempt to estimate mutual information directly 4145) and others which try infer its value by computing upper lower bounds [51 44 It is challenging statistical problem to solve in general, and many estimators and bounds are known to suffer from high bias and/or variance, particularly digh-dimensional problems [51
Many mutual information estimators rely on first estimating entropy, which is a measure of uncertainty of random variable. The outcome of random variables with higher entropy are more uncertain and harder to predict than the outcome of those with lower entropy, because they are more random. Equivalently, the higher distribution's 's entropy is, the more spread out it is, and the harder
50

---

it would be to digitally compress samples from it. Mathematically. entropy is defined as the expected value of the negative log of the probability density/mass function:
H (Y) = -E [oog p Y)]
(10)
Like mutual information estimation,, there is a large body of literature on entropy estimation and many different approaches have been proposed [I02], including those that form estimate of the probability density function p(y) and plug it into the definition of entropy and those that estimate entropy based on the similarity of samples from p(y) [103/105]. Both approaches, however, face difficulties in high dimensions.
There are multiple ways of decomposing mutual information into difference of entropies. Our approach is based upon the decomposition:
- HY x)
(11)
Here, H (Y) is the entropy of the noisy image distribution p(y). Both variations in the object and measurement noise contribute to the randomness of measurements. We are interested in the information about the object, thus to quantify how much of these variations the measurements contain, we must subtract the entropy contributed by the noise, H (Y X).
This decomposition is especially valuable for optical imaging because measurement noise typically acts independently at each pixel, greatly simplifying the calculation of the conditional entropy H (YY) X).
(A technical note: in optical systems, the outcomes of the random variables X and Y in our model are discrete, because they are 2D arrays of pixels, where each pixel takes on an intensity value that is an integer number of photons. However, for computational simplicity, we will approximate them as continuous random variables and use differential entropy instead discrete entropy [80, 9I] These approximations break down very low photon counts, less than 20 photons, so in this paper, use only data with photon counts greater than this. A possible direction of future work to extend approach to work with discrete random variables, which would allow it to be applied to data with lower photon counts.)
We begin by describing the simpler of the two terms to estimate, the conditional entropy H (Y X).
S2.3 Estimating conditional entropy of measurement noise
The conditional entropy of noisy measurements given noiseless images can be written out in terms of expectations over the logarithm conditional probability of noisy measurements given a noiseless image p(y x):
H (Y
X = Ex, [logp(Y) X)]
(12)
By the Law of Total Expectation, this can be written as:
Ex, [log P Y X)] = Ex Ex Pog p( (Y
X) X)]]
X]]
(13)
Ex EY [logp(Y
(14)
p(y x) embodies the various sources of noise in the detection process, including photon shot noise, detector read noise, etc. Here, we utilize established analytic models of detection noise in optical imaging [40]. Empirical results suggest that the true noise in systems deviates from these models in low-light conditions [106), but since our experiments are conducted in the high-light regime, we will assume that these models are accurate for the purposes of this paper. A possible direction for future work is to learn more accurate noise models from the data, as was done in [107].
Assuming have access to dataset of N samples from the distribution of noiseless images (x(1) x ..... x (N)) the outer expectation can be estimated through Monte Carlo approximation:
51

---

Ex By llog(( Xi]] ~
1
By
logplY x(1)]
(15)
N
i=1
N
(16)
Here, H (Y x(0)
is the conditional entropy of the distribution of noisy images given the ith noiseless
image.
S2.3.1
Conditionally independent noise each pixel
H (Y (x()) is a function of the noise introduced in the detection process, which is modeled by the probability distribution p(y x x) In optical imaging, common analytic noise models like additive Gaussian and Poisson shot noise typically assume that the noise each pixel is conditionally independent of the noise at other pixels, given the true (noiseless) pixel value. When this is true, p(y x²)., which a joint distribution over all D pixels in the noisy measurement, can simplified by factoring it into product of scalar distributions for each pixel, where ys and w/6 are the intensity values at the ktth pixel in the noisy measurement and the ith noiseless image, respectively:
(1) ply x) ply1, 12. VD ar) 1
(6)
T (8)
p(us x
This factorization simplifies the calculation of conditional entropy, because it is much easier to compute D scalar conditional entropies than single joint conditional entropy over D variables. Mathematically, this simplification can be seen by plugging the factorized distribution into the definition of conditional entropy:
H Y x(1) = --
log II
plux )
(17)
k=1
=
Σ log p(yk 1xb
(18)
D
Σ
Ey [oo p(yx (x(2))
(19)
k=1
Σ H R=1
YN xx(2))
(20)
Here, H(Y | xx is the conditional entropy of the ktth pixel the noisy image given the intensity of the kth pixel in the noiseless image.
(0))
This is scalar quantity and can calculated analytically for many common noise models. We will discuss two such models here: additive Gaussian noise and Poisson noise.
$2.3.2 Conditional entropy with additive Gaussian noise
Additive Gaussian noise is simple noise model often used in optical imaging, especially in low-light conditions where the read noise of the detector is the dominant source of noise. In this model, the noise at each pixel is drawn from a Gaussian distribution with mean zero and variance o². Mathematically:
52

---

YA Xx N5
N₁ ~ (N(,0000))
The entropy of a (scalar) Gaussian distribution N(H, o) is 80]
H (N+) 2 logg loggg22ee (2Tre²²)
Since the noise is independent of the noiseless image, the conditional entropy the noise each pixel is the same, and the full conditional entropy (Equation 20) simplifies to:
D
H (
Σ H
YE
(21)
k=1
D
H (Nx)
(22)
k=1
D
loga (2Tee²)
(23)
2
Plugging this result into Equations 16 and 12 yields:
D
H H X) loga (2mea²)
(24)
To summarize, the conditional entropy of the noisy image given the noiseless image is a constant, independent of the intensity values of the noiseless images, and is equal to the number of pixels in the image times the entropy of the noise distribution at each pixel.
S2.3.3 Conditional entropy with shot noise
Images with high photon counts are fundamentally limited by shot noise- randomness in photon arrival times due to the quantum nature of light. This shot noise follows a Poisson distribution with rate parameter equal to the expected number of photons each pixel. When shot noise the dominant source of noise, it can be accurately approximated by Gaussian distribution with equal mean and variance 40
Thus, the conditional entropy of the noise at pixel for the ith noiseless image can be approximated with the entropy of gaussian distribution:
H
YY
1 (1)
= 1ogg(2mexf)) (0)
(25)
Once again making use of the fact that the measurement noise at each pixel is independent of the noise at other pixels conditional on the intensity of the noiseless image at that pixel (Section S2.3. we can write the conditional entropy for noiseless image xi) as:
Y (x())
= Σ
Yv (0(0
(26)
1=1
= Σ
log eaT (2)
(27)
2
km 1
53

---

The full conditional entropy (Equation 20 simplifies to:
N
H(Y X)
loga (2meak) (0))
(28)
To summarize, the conditional entropy under Poisson noise model can be approximated the log of the intensity values of the noiseless image each pixel, averaged over N noiseless images. This approximation is accurate when the photon counts high, and breaks down when the photon counts are low, and is discussed further in Section S3.5
of
S2.4 Estimating entropy of noisy images
The second term in the mutual information decomposition, H Y presents a greater estimation challenge. Unlike the conditional entropy calculated in Section S2.3 the joint probability distribution p(y) cannot be factored into independent distributions for each pixel, pixels throughout both noisy and noiseless images exhibit complex dependencies.
We estimate this entropy by computing an upper bound, an approach that has proven more accurate than alternative bounds in nigh-dimension
settings [51]. Our method fits parametric model po(y) to the empirical distribution of noisy images using maximum likelihood estimation. The optimal parameters MLL are found by minimizing the negative log likelihood of the observed data:
0 MMLE arg min - llog pe (Y)
(29)
0
This loss function, -E [log ps Y)], p)(y) and the empirical distribution p(y).
[]. is also known as the cross-entropy between the model distribution
In practice, is fit using dataset of samples from the empirical distribution p(y):
i = arg min
log (y
(30)
N
When the model distribution ) identical to the empirical distribution p(y), the data has been fit perfectly, and the value of the cross-entropy loss function is equal to the entropy of the noisy images, H (Y).
EE 1o (Y)) = -E [logp( (Y)
(31)
(Y)
(32)
In practice, the model will not be able to fit the true distribution exactly, and the average value the loss function will be greater than the entropy of the noisy images. The gap between the entropy that we are interested estimating and the cross-entropy loss function is given by the Kullback-Leibler divergence between the empirical distribution and the model distribution:
E log YY) = H (Y) + DKL Pe)
(33)
The Kullback-Leible divergence is measure of the difference between two probability distributions. It is always neg-negative and is zero only when the two distributions are identical. Thus, the cross- entropy loss function is an upper bound on the entropy of the noisy images. The better our model fits the data, the tighter this bound will be. Finding the right model that balances the accuracy of this bound with the computational cost of fitting the data is an important choice that discussed further in Section S3.2
54

---

In practice, the cross-entropy loss function is evaluated on separate test set of samples from the empirical distribution, avoid overfitting to a subset of the data and generating a model that is overly optimistic about its performance.
This process is mathematically equivalent to data compression in information theory In data compression, the goal is to map each outcome to bit string while minimizing the average string length. Optimal compression assigns shorter bit strings more probable outcomes and longer ones to less probable outcomes. The achievable compression measured by the average bit string length- has a fundamental lower bound equal the entropy of the data distribution. Just as no compression scheme can achieve shorter average bit length than that dictated by the true distribution's 's entropy. no model we can achieve lower entropy than the true distribution any mismatch between our model and reality can only increase our entropy estimate.
S2.5 Probabilistic models
In this section, we describe three probabilistic models po (y) used to estimate the entropy of noisy measurements via upper bounds, as described in Section S2.4 The models offer different tradeoff's between accuracy and computational efficiency.
The simplest model assumes stationary Gaussian statistics, providing fast estimation with minimal data requirements. A more complex full-covariance Gaussian model removes the stationarity assump- tion, offering improved accuracy while maintaining computational efficiency. The most sophisticated model, based on the PixeICNN neural network architecture, provides the tightest bounds but requires substantially more computation time and training data.
Other models could be used within this framework, provided they allow direct evaluation of likelihood functions. Recent neural network architectures such as transformers [84), normalizing flows [08]. and diffusion models [199 [10
offer promising directions for future work.
While generative models are often evaluated by the visual quality of their samples, this metric does necessarily correlate with cross-entropy performance [II though some tfate-oftthe-a art models can achieve both [[12]. We selected PixeICNN for its demonstrated effectiveness in minimizing test set tross-entropy [55], combined with its relative simplicity and computational efficiency. However, the rapid advancement of generative models suggests that more efficient architectures providing tighter entropy bounds for given computational efficiency likely emerge.
$2.5.1
Full Gaussian process
The full Gaussian process model approximates the distribution of noisy images with multivariate Gaussian distribution, which is specified by a mean vector μ and covariance matrix Σ. The mean vector contains D parameters describing the average value at each pixel, while the covariance matrix requires D(D+1) parameters to capture all possible correlations between pixels.
We fit this model by directly estimating the mean and covariance from the data.
$2.5.2 Stationary Gaussian process
To reduce model complexity and quantify performance over the full field of view with single scalar quantity, we can make the simplifying assumption that images are stationary stochastic processes. This is the probabilistic analog of assuming constant optical resolution across the field of view (97).
stationary stochastic process with distribution p(y) is one in which the joint distribution of any set of pixels is invariant to translations across the field of view. For ID process (i.c., single row of pixels), this means that the joint distribution of pixels depends only on their relative locations, not absolute positions.
Mathematically, for any vector of pixels (y1, .2..... ... .pb),. the joint distribution remains unchanged when all pixels are offset by a constant amount k:
'Technically, this only completely true when the model distribution is discrete, because continuous data cannot be losslessly compressed. For example, it would take an infinite number of bits on computer to represent arbitrary real number exactly. But the intuition remains the same.
55

---

p(y V2..... .D) = B2-k WD+k)
(34)
This stationarity assumption constrains our Gaussian model in ways. First, the mean vector must be constant across all pixels, reducing it from D parameters to just one. Second, the covariance between any two pixels must depend only their relative positions. For ID process, this makes the covariance matrix Toeplitz (constant along diagonals). For 2D processes, it becomes a doubly Toeplitz. matrix: blocks of Toeplitz matrices arranged in Toeplitz pattern (Fig. (SO). This structure reduces the number of covariance parameters from D(++1) to just
D
Covariance (photons?)
Figure S10: The doubly Toeplitz D D covariance matrix of stationary 2D Gaussian process.
Stationary processes have another useful property: a constant entropy per pixel, called the entropy rate 80) (explained in Figure 8 of (90)). This rate can be defined in two equivalent ways:
lim H(11, V2... V2...
......,, .D)
lim H (yD y1, V2. ... .DD11)
(35)
D-s00
The second formulation reveals why the entropy rate decreases with D: as we observe more pixels, it is caiser to predict the value at the next pixel, because more context about neighboring pixels is available.
$2.5.3 PixeICNN
Our most flexible model is the PixeICNN 55, which uses neural networks to capture complex pixel dependencies through an autoregressive approach.
Autoregressive models factorize a joint distribution product of conditional distributions:
ply) = II p(u W1. WasBe:.... (.-11) R-1
(36)
p(m )p(12
yı lp(ss (1. pluM Vi, "................... .2.1...9D-1))
(37)
This factorization does not require specific assumptions about the joint distribution, so in theory can be applied to any distribution. Creating the full model requires D conditional distribution models,
56

---

each taking between and D previous pixel values imput and outputting ID probability distribution for the next pixel. While this could be implemented with D separate models, PixeICNN uses masked convolutions in a single neural network to model all conditionals simultaneously, dramatically reducing computational complexity.
Following previous work [53 54], our architecture uses a series of masked convolutions to maintain the autoregressive ordering, producing conditional probability distribution at each pixel.
However, we make one crucial modification: instead the standard softmax output layer, we use mixture of Gaussians parameterized by the network (a mixture density network (113).. This strategy capable in theory of approximating any conditional probability distribution, in the same way that a neural network is capable of approximating any function [114]. This modification is essential for estimating entropy in our framework, because it means the output distributions will be continuous probability densities, instead of discrete probability mass functions. Since noise models p(y x) are continuous (Gaussian), our entropy estimates for H(Y) must also use continuous distributions to allow proper subtraction of the conditional entropy I/ (Y] X).
S2.6 Sampling and likelihood evaluation
While our models' primary purpose is likelihood evaluation for entropy estimation (Section (2..4) generating samples provides valuable insight into model behavior. For simple models like the stationary Gaussian process, sample quality correlates well with likelihood accuracy, making visual inspection useful diagnostie tool. However, this correlation breaks down for more complex models like PixelCNN [[T]] though samples remain helpful for understanding model behavior.
Both PixeICNN and stationary Gaussian models are trained on fixed-size patches (VD VD pixels), limiting their ability to capture longer-range dependencies. While increasing patch size allows modeling of longer-range correlations, both models can handle arbitrary-sized images despite their fixed patch size. This is possible because their stationary nature allows them to be factored into conditional distributions, enabling evaluation over larger images by sliding the fixed-extent window (though this iterative process is substantially slower than processing native-sized patches). For I-dimensional model with 1) pixels, this factorization means the joint distribution can be written as:
p = II ply V2....,......................... 11- k=1
(38)
= p(y) (p(2 y1)p(ys y1. v2) P(yD 91. V2. 1D)11)
(39)
Taking the log likelihood turns this product into sum:
logp(y)
Σ
log p(uk y1, U2. Vk-1)
(40)
k=1
log log vn) + logp(ys 11, 82) + .... log popp y1, V2. ....... VD-1)
(41)
Computing the likelihood over an image larger than the length D model the pixel was trained on can be accomplished by adding additional terms to this sum. For example, computing the log likelihood of an D + 1 length image would require adding an additional term to the sum of the log likelihood of the the + pixel conditioned on the previous that preceded it:
pixels (the maximum extent of the model)
wK-1) log Aogp(ynt) y2, V3......... yD)
loy)(y)= Σ log plyx I yn, U2, v2.......eii)
(42)
Details of the likelihood computations for PixeICNNs are described in 53 processes, there is closed form solution for finding the mean uD and variance of of a l-dimensional
For stationary Gaussian
57

---

Gaussian distribution conditioned on D-1 previous values. This involves decomposing the covariance matrix into top left (D- 1) x (D 1) block 11,1. top right (D 1) column vector 1
and bottom left 1 x (D 1) row vector E²,
Σ 1,1
Σ
112
Σ
(43)
2,1
M
Given previous values yn-1, the mean and variance of the Dth value can computed as:
PD - 22 o = 0² Σ,, -
(44)
(45)
The likelihood the the Dth pixel can then be evaluated using the probability density function of a 1D Gaussian distribution with mean μap and variance r.
Similarly, sampling images larger than the patch size on which the models were trained can be accomplished by iteratively sampling each pixel conditioned on the previous D pixels.
S2.7 Stochastic encoders
The deterministic encoder assumption underlying our framework breaks down in two scenarios:
First, in quantum imaging regimes, image formation becomes inherently random and cannot be predicted from Maxwell's equations. These are beyond the scope of our present work.
Second, systems with encoder tion fluctuations or mechanical drift that cause the same object produce different noiseless images across measurements.
but unpredictable variations like illumina-
The most effective approach to reduce encoder uncertainty is minimize system variations through careful engineering: building mechanically stable optical systems, using temperature control to reduce thermal drift, and implementing precise electronic control of components like illumination sources. Nonetheless, some level of system variation often remains unavoidable in practice.
Here we describe how our framework can be expanded to stochastic encoders to account for encoder uncertainty.
S2.7.1 Encoder uncertainty
Stochastic encoders affect information estimates two ways. First, different encoder states preserve different amounts of object information. For example, defocus destroys sharp features images. Average object information thus depends on the focus distribution.
Second, encoder variations create measurement diversity beyond object variations. With focus drift, measurements vary in both content and sharpness. Without modification, our method would interpret both object information, thereby upwardly biasing its estimates.
One potential way to adapt our method is using calibration data to identify encoder-inducee variations. This would likely require either additional compute simplifying assumptions to handle the fact the the measurement diversity induced by encoder variations may be object dependent. Mathematical formulation
We model encoder variations with state variable
(0, (O,E) +X
Now X = f(O,E) for some deterministic function f. For example, for focus drift, would describe how defocus E blurs object O produce image X.
The quantity of interest the original deterministic setting now written with the encoder explicit I(O; Yle).
58

---

In the stochastic case, averaging over all encoder states gives:
I(0,Y)= =( [lle) Ye)
Where T(O; Y) denotes the average object information across all encoder states. Extending the deterministic information transfer theorem to X = f(O,E),
1 = I((,EEY))
Rewriting using the chain rule of mutual information:
((; I(0; (E; Y|O)
The first term captures effect (average object information across encoder states); the second captures effect two (diversity from encoder variations).
Example: random focal drift Consider a microscope with mechanical drift causing focus to vary between perfect focus and various degrees of defocus. Compared stable, in-focus system, we have T(O: Y) < I(O; Y (/emm.ozuu) because defocus destroys high-frequency information. Meanwhile, [EE; Y O) > O because focus variations add measurement diversity.
As shown in the Deterministic Information Transfer Theorem, a deterministic in-focus system has [(;; Y) IO; Yoein-focus) With stochastic focus variations, [(X; Y) can increase decrease relative to this baseline depending on which of the two effects dominates.
S2.7.2 Estimating information with stochastic encoders
When encoders vary stochastically, our method estimates 1(X; Y) I(O; Y) + (E; Y|O). To isolate object information I(O; Y), we must estimate and subtract the encoder contribution.
The central challenge is that (E; YJO) depends on the object-differen different objects may reveal encoder variations differently. We propose two approaches:
Approach Fix object, vary encoder states, estimate [E; Y = o) for multiple objects. This would be feasible in IDEAL using increased compute, but in experimental systems only possible if we can fix objects while varying encoder states (e.g., imaging a static sample while inducing focus drift).
Section S4.1 shows an example non-object information the measurement.
Approach 2: Assume EEE Y|O) ~ [E; Y). Computationally and experimentally simpler, but may give incorrect results if encoder effects are highly object-dependen
Taking Approach 2 as an example:
(E: - - H(Y
Our method already estimates HYY). To estimate HYY fit conditional models (y(y))) instead of (y),, where e represents encoder state information. For focus drift, conditioning on timestamps would help predict measurements since focus varies smoothly with time.
For Approach 1, we would additionally need to take an outer expectation over objects. In IDEAL, since we know the true object, the models could be modified to condition on it directly. In experimental conditions, we would likely need static objects to take multiple measurements of.
By estimating both terms we can recover the quantity of interest, object information in the measure- ment.
I(0: Y) 1(X; Y) I(E, Y
59

---

S2.8 Tpsk-specific information
While information estimates strongly predict reconstruction performance, they may correlate less strongly with specialized tasks that rely only on specific features of measurements. Systems that preserve specific task-relevant information can outperform those capturing more total information
indiscriminately [31 76
Total information naturally decomposes into task-specific and task-irrelevant components: I(Y
TY:T) + T(Y; OJT), where T represents task variables like classification labels. This enables weighted objectives TY; + ( for specialized applications:
Balanced systems (a, 3 > 0): Control relative importance of task-specific versus general information
Compressed sensing (a 0, 0): Maximize task-relevant while suppressing task- irrelevant information for bandwidth-limited applications
(a < 0, 3 > 0): Suppress task-specific information while preserving
general scene information
Estimation requires computing TY = HYY) (T),)) Our existing approach estimates H(Y), while H (Y))) can be estimated by conditioning models on task information. Section (S.22 provides experimental evidence of when information estimates not fully predict task performance.
S3 Estimating information: Experiments
Having developed a framework for estimating information in imaging systems, we now validate its performance through experiments on both simulated and real-world data.
S3.1 Fitting stationary Gaussian processes
Fitting a stationary Gaussian process to N image patches (vD x vD pixels) requires estimating both the mean vector and covariance matrix E. While the mean is easily computed by averaging all pixels across patches, estimating the covariance matrix presents a greater challenge.
As described in Section S2.5.2, the covariance matrix must be both symmetric positive definite and doubly Toeplitz, with repeated patterns along diagonal blocks. Our initial approach was to compute the sample covariance matrix from vectorized patches and enforce the Toeplitz structure by averaging along diagonals and blocks. However, this averaging operation sometimes produced matrices with negative eigenvalues, violating the positive definite requirement and creating invalid covariance matrices.
To address this issue, we initially tried enforcing a minimum positive value (floor) for the eigenvalues. However, this approach proved problematic for two reasons. First, modifying eigenvalues disrupted the doubly Toeplitz structure, violating the stationarity requirement. More critically, since the likelihood evaluation used for entropy estimation highly sensitive to small eigenvalues in the covariance matrix, the arbitrary choice of eigenvalue floor significantly impacted our entropy bounds. This sensitivity varied across datasets, making the approach unreliable for comparing different imaging systems.
To overcome these limitations, we developed an iterative optimization procedure that improves the covariance matrix fit while maintaining required properties. Given N image patches (each
vD x vD pixels) vectorized to length- vectors y (0) 1=1, we minimize the negative log likelihood of multivariate Gaussian distribution ([: (t E):
N Σ
arg min Σ
ly
(46)
Where a is the fixed mean vector estimated from the data described above.
60

---

We solved this optimization problem using proximal gradient descent with momentum, initializing from the direct sample estimate. Each iteration consisted of three steps:
Computing the eigendecomposition of the covariance matrix and applying gradients to cigenvalues while keeping cigenvectors fixed
Applying proximal operator that enforced the doubly Toeplitz structure by averaging along diagonals and blocks
Enforcing the minimum eigenvalue constraint
The procedure was regularized through early stopping: optimization terminated when the loss hadn't 't decreased for a fixed number of iterations, with parameters from the lowest-loss iteration selected as the final estimate.
Empirically, we found that the optimization procedure was unstable and prone to divergence. This seemed to be because the likelihood was very sensitive to small eigenvalues of the covariance, which tended to produce extremely large gradients for these eigenvalues. To account for this, we implemented gradient clipping to limit the magnitude of the gradients for the eigenvalues.
The likelihood of probability distribution can be very sensitive small changes in parameters because the Euclidean distance between parameter vectors does not always accurately reflect the dissimilarity of the resulting distributions (as described section 2.3 of (15)). In theory this can be corrected by multiplying the gradient by the inverse of the Fisher information matrix to reorient it from the steepest direction in Euclidean space to the steepest direction in Riemannian space, which accounts for the natural geometry of the parameters. However, in practice we found this to be expensive to compute without obvious performance benefits over gradient clipping.
Figure ST1 shows the results when fitting stationary Gaussian processes to images from the BSCCM dataset 775] The direct estimate contained several negative eigenvalues, which were then an arbitrary eigenvalue floor. The optimization procedure was able to correct these negative eigenvalues, and produced robust results for a variety of settings of the eigenvalue floor parameter (provided it was smaller than the true minimum eigenvalue) (Fig, SS1pp). As a consequence of the small, incorrect eigenvalues in the direct estimate, samples produced from the model that were larger than the patch size on which they were trained (Sec. (22.6) exhibited numerical instability that created large oscillation in the samples. In contrast, samples from the optimized model were stable and did not exhibit this oscillation (Fig, b). The optimization quickly converged in a few iterations (Fig. (STTe), taking ~ 3 seconds to complete on an NVIDIA GeForee RTX 3090 GPU. Comparing estimates of a non-stationary Gaussian process, the direct estimate stationary Gaussian process, and the optimized stationary Gaussian process showed that the optimized stationary Gaussian process produced more robust results in terms of its eigenvalue distribution using smaller datasets (Fig, SIId).
S3.2 Comparing stationary Gaussian process and PixeICNN estimates
We evaluated model performance through two approaches: negative log likelihood on held-out test data and qualitative assessment of model-generated samples.
An important consideration for both models is the choice of training patch size. Larger patches capture longer-range dependencies and improve per-pixel likelihood, but require more training data and computational resources. We therefore sought to determine the minimum patch size that would provide reliable entropy estimates while maintaining computational efficiency.
Figure S12 shows samples produced when fitting models to patches of different sizes. As ex- pected, larger patch sizes were able to capture more complex statistical relationships between pixels (Fig. S12a), decrease the per-pixel loss (Fig. S12b), and more tightly upper bound the entropy of the noisy images (Fig, (S12)). The gains in performance were minimal beyond a patch size of 35 x 35 pixels for the Gaussian model, while the PixelCNN was able to continue to produce more accurate estimate as patch size increased further. This is presumably due to its much greater flexibility in modeling complex dependencies between pixels. However, the magnitude of the gains was small relative to the differences between the estimates mutual information for different contrast modalities (Fig, (13) for these experimental images.
When comparing the samples produced by the two models when fit to images with different illumi- nation patterns on the LED array, both the stationary Gaussian process and PixeICNN models were
61

---

Optimization improves covariance matrix conditioning
b) Improved conditioning prevents numerical instability when sampling
10%
Initial
Elgenvalue (photons")
10
Optimized
10
Initial fit
Optimized
10"4
10%
Eigenvalue floor hyperparameter
25
50
75
100
500
1000
1500
2000
Eigenvalue index
Photons
c) Optimization progression
d) Optimizing gives better fit with less data
Training set size (number of images)
5.442
10
10a
10a
10
Non-stationary
10
Validation negative
Eigenvalus
(phatons")
10
log likelihood
10a
10
10"
5.432
10
Stationary
Eigervallee
(phatons))
10a
10
10
10
0
25
50
75
100
lteration
Stationary
(optimized)
Elgenvalue
(photors))
100
100
100
100
Eigenvalue index
Eigenwalue index
Eigenvalue index
Eigenvalue index
Figure S11: Fitting Gaussian Process models. a) Eigenvalues of the covariance matrix before and after optimization, indicating improved conditioning post-optimization. The conditioning of the covariance matrix of non-optimized fits highly dependent the choice of the eigenvalue floor hyperparameter. b) Comparison of numerical stability during sampling pre- and post-optimization, demonstrating that optimization prevents the instability that manifests in the initial fit. e) The opti- mization process takes only few iterations to converge and shows an extremely large improvement in the negative log likelihood of the data for poor initial fits. d) Efficacy of the optimization across varying training set sizes, illustrating that optimized fitting requires fewer images to achieve a compa- rable or better fit than the non-optimized approach and outperforms the full Gaussian model when training data is limited.
62

---

a) Samples from trained models
b) Test set loss
Stationary Gaussian Process
likellihood per pixel
1750
Negative log
True distribution
6
1500
1250
Photons
c) Estimated mutual information
1000
PixelCNN
750
Estimated information (bits per pixel)
3
500
2-
80x80
1-
Gaussian process
250
PixalCNN
10x10 20x20 30x30 40x40 50x50
2x2
5x5
20x20
50x50
1x1
3x3
10x10
35x35
Patch size
Training data patch size
Figure $12: The effects of patch size model fit. Samples generated from Stationary Gaussian Process and PixeICNN, alongside samples from the the true distribution. As the patch size increases, the models are able capture more long range dependencies between pixels, and b) achieve lower negative log likelihood per pixel and thus c) tighter upper bound on mutual information.
able to capture statistical patterns to each type of contrast modality (Fig. S14a). Samples from the trained models appear to show that the stationary Gaussian model was able to learn the statistical patterns of the texture of the images of objects under different illumination, while the PixeICNN model additionally learned higher order structures like the edges of cells.
S3.3 Failures of stationary Gaussian estimates on highly non-Gaussian data
The comparable performance of stationary Gaussian and PixeICNN models the BSCCM dataset (single cells under coded-angle illumination) likely reflects the near-Gaussian statistics of these images. However, this similarity cannot be assumed for all datasets. This becomes evident when testing on the MNIST handwritten digits dataset [16] (GNU General Public License), where image statistics deviate significantly from Gaussian distributions due the nearly bimodal nature of pixel values (predominantly 0 or I).
To demonstrate this, take the MNIST dataset and simulate minimal optical encoder (i.c. single lens) by convolving the images with a Gaussian point spread function. We then add Poisson noise to simulate noisy measurements, and fit both the stationary Gaussian process and PixeICNN models to the data. Sampling from the fitted models shows that the PixelCNN model can produce images that appear similar to measurements, unlike the Gaussian process model (Fig, S155))
This occurs because the best approximating stationary Gaussian fit very dissimilar to the true distribution of the measurements. This can be seen by looking at the histograms of image pixels of
63

---

Photons
10
30
100
150
200
200
400
600
800
Single LED
Differential Phase Contrast Brightfield
0.7
95%
confidence
Single LED
Estimated information
0.6
0.5
(bits/pixel)
tixel)
0.4
Differential
Phase Contrast
0.3
0.2
0.1
Brightfield
0.0
100
200
300
400
500
Photons per pixel
Figure S13: Mutual information and photon count. As the average number of photons per pixel increases, the signal-to-non toiise ratio and mutual information both increase. (Top) Example images of three different contrast modalities at varying photon counts. (Bottom) The mutual information per pixel for each contrast modality function of photon count.
the true data and samples from the models, pooled across all many images (Fig. S15p). The true data and PixelCNN samples have similar histograms, while the histogram from the stationary Gaussian process samples has very different shape. As result of this poor fit, the upper bound given by the mutual information estimator quite loose for the stationary Gaussian model compared to the PixelCNN model (Fig. S1)c).
These findings demonstrate that stationary Gaussian process model is inadequate to capture certain image distributions. Since the tightness of the upper bound in our information estimation procedure depends on accurately fitting the distribution noisy measurements, this suggests that achieving a tight upper bound/accurate estimate in many cases may require more flexible probabilistic models, like the PixeICNN.
An important and desirable property of an estimator is its consistency: whether converges to the true value of the parameter being estimated given enough data. This has practical implications for determining how much data is needed to achieve desired accuracy level.
To evaluate our estimators high-dimensional data, first tested them on samples from known distribution: a stationary multivariate Gaussian with independent additive Gaussian noise at each pixel (Equation (4).. In this case, (YY X) constant, and (Y) can be computed analytically by adding the noise variance to the diagonal of the data covariance matrix and analytically calculating the entropy of the resultant multivariate Gaussian distribution.
Figure S16 shows that all three estimators converge to the true value given sufficient samples, with the stationary Gaussian estimator being most accurate for given sample size. This is expected since the test data perfectly matches the model's assumptions.
To evaluate performance real microscopy data, we tested two different patch sampling strategies. When patches were taken from fixed locations across different images (S66)), the full Gaussian model outperformed the stationary model. This suggests the presence of location-specific image
64

---

True distribution
450
Single LED
Gaussian
400
PixeICNN
350
True
distribution
Brightfield
300
Photons
Gaussian
PixelCNN
250
True distribution
Differential phase contrast
200
Gaussian
150
PixelCNN
Figure S14: Model samples for different contrast modalities. Samples from the stationary Gaussian process and PixeICNN models for Brightfield, Differential phase contrast, and Single LED illumination.
a) Images
b) Pixel distributions
c) Mutual information estimates
True distribution
Count
286 a
3.5
Bits per pixe!
Gaussian
2.5
Count
1o to. & a
Model samples
PixeeCNN Gaussian
2.0
1.5
Count
2 अ क a.
0.5
PixelCNN
50
100
150
200
250
300
500 1000
Mean photons per pixel
400
700 800 900 Photons
Pixel Intensity
(Photons)
Figure S15: PixelCNN can fit non-Gaussian data better than stationary Gaussian processes. a) Samples from the stationary Gaussian process and PixeICNN models for MNIST digits. The Gaussian process samples do not resemble the true distribution, indicating poor fit. b) Histograms of pixel values for the true distributions and samples from the two models indicate that the marginal distribution of pixels is non-Gaussian, which the PixeICNN, unlike the Gaussian model, is able fit. с) As a result of the poor fit, the upper bound on mutual information given by the Gaussian process estimator is much looser than the PixeICNN estimator.
65

---

statistics that violate the stationarity assumption. However, when patches were sampled from random locations (Fgg))(ik)) the stationary Gaussian model performed better than the full Gaussian model, as the random sampling effectively enforces stationarity in the data distribution.
The PixelCNN model demonstrated good performance across all scenarios due its flexibility, though the stationary Gaussian model was able to outperform it for small training dataset sizes.
a) Stationary Gaussian data
Full Gaussian
10*
PixelCNN
Mutual information (bits per pixel)
process
10*
True value
10a
Stationary Gaussian process
0.5
95% confidence
10
b) Real data (fixed location patches)
10*
Mutual information (bits per pixel)
10*
10a
10
10
0.3
c) Real data (random location patches)
10
Mutual information (bits per pixel)
10*
10
10
101
0.4
109
10a
10a
10*
10a
10
10*
Number of samples
Number of samples
Figure S16: Estimator consistency. a) On simulated data from stationary Gaussian process, all estimators converge to the true mutual information value, with the stationary Gaussian being most accurate per sample. b) With fixed-location patches from real data, the full Gaussian model outperforms the stationary model due to location-specific image statistics. e) With random-location patches, the stationary model performs better sampling enforces effective stationarity in the data distribution.
66

---

$3.4 Model training times
The three models offer different tradeoffs between accuracy and computational efficiency. As shown
in Figure S17 training times typical 20 x 20 pixel patch dataset vary significantly: the full Gaussian process is fastest (~ 0.1s), while the stationary Gaussian process requires longer (~ 10s) due to its iterative optimization procedure. The PixelCNN is most computationalyy intensive (~ 100s), reflecting the cost its greater modeling flexibility.
I
Standard deviation
10
Training time (s)
10'
10"
10
Stationary Gaussian process
Full
PixelCNN
Gaussian
process
Figure S17: Model training times. Comparison of training times for each model type on a typical dataset of 20x20 pixel patches on an NVIDIA RTX A6000 GPU. Error bars show standard deviation across multiple training runs.
S3.5 Conditional entropy estimates on noisy data
The accuracy of mutual information estimates depends on both HY) and H(Y X). For additive Gaussian noise, the conditional entropy is independent of the noiseless images and can be computed analytically (Sec. (S2.3.2). However, for signal- Poisson noise (Sec. (2.3.33, estimation becomes more challenging.
Using simulated data derived from the BSCCM 75 dataset, we first examined the consistency of our conditional entropy estimator. We created ground truth data by applying a 3 x 3 median filter to produce noiseless images, then added simulated shot noise. Our estimator rapidly converges to the true value (Fig. S18a), with variations on the order of 10 differential entropy per pixel- significantly smaller than the variations in H (Y) estimation.
In experimental settings, noiseless images are typically unavailable, Instead, we must estimate conditional entropy directly from noisy measurements, replacing noiseless pixel values in Equation 28 with noisy ones:
H(Y X
loga (2rea (0)
(47)
N
log (2mey (1)
(48)
N
To evaluate this approximation, we compared mutual information estimates using either noiseless or noisy images for conditional entropy estimation across different imaging modalities (Fig. SI8p). The estimates show good agreement except at very low photon counts (below ~40 photons per pixel). At these low counts, using noisy measurements leads to slight overestimation of mutual information due to underestimation of conditional entropy. While this bias could potentially be corrected, other approximations in our framework (such as the Gaussian approximation Poisson noise (Sec. S2.3.3 also break down in this regime.
67

---

a) Consistency of shot noise conditional entropy estimate
b) Estimating from noisy samples
Noisy samples only
Differential entropy per pixel
4.28
Estimated Mutual Information (bits per pixel)
0.5
95% confidence
Clean and noisy samples
95% confidence
Single LED
4.26
0.4
4.24
True value
0.3
Differential Phase Contrast
0.2
4.22
0.1
Brightfield
4.20
0.0
10
10a
10a
0
100
200
300
Number of samples
Photons per pixel
Figure S18: Conditional entropy estimation. a) The estimator converges rapidly the true conditional entropy for shot noise as sample size increases, with variations much smaller than those seen in entropy estimation (Figure S16). b) Comparison of mutual information estimates using noisy versus noiseless measurements for conditional entropy estimation across different imaging modalities, showing good agreement except very low photon counts.
S3.6 Comparing analytic and upper bound entropy estimates
For Gaussian models, we can estimate entropy in two ways: by computing an upper bound using test negative log likelihood, by analytically calculating the entropy the fitted model. While the upper bound approach provides theoretical guarantees, the relationship between analytic estimates and true entropy less clear.
We compared both approaches empirically across three imaging modalities in the BSCCM dataset using three estimators: analytic stationary Gaussian entropy, stationary Gaussian upper bound, and PixelCNN upper bound (Fig. (Fig.SS9)), While we lack theoretical guarantees for the analytic stationary Gaussian estimates, they consistently track close to the PixelCNN bounds, which represent best estimate of the true entropy. The differences between estimation methods remain small compared the variations across imaging modalities and photon counts. This close agreement particularly relevant for our IDEAL framework (See Encoder design via information maximization with IDEL where we use the analytic form our optimization loss function.
S4
Additional decoder experiments
S4.1 Experimental evidence of stochastic encoder effects
We examined LED array microscopy with different illumination patterns to demonstrate how sensitive systems capture both object and system information. By estimating mutual information empty image patches, we measured information arising purely from system variations.
As shown in Figure S21 differential phase contrast or brightfield illumination. This demonstrates that more sensitive encoders capture more system information alongside object information. When substantial encoder uncertainty is present, the theoretical framework from Section S2.7 may be needed to separate these contributions.
single-LED illumination exhibited higher background information than
S4.2 Task-specific information in classification
For tasks like deconvolution that aim to recover full image structure, any additional information helps reduce reconstruction error. However, simpler tasks like classification may only need specific features. We tested this using a 10-class object classification task on our lensless imaging measure- ments (Fig. S22 Unlike deconvolution, where higher information consistently predicted better
68

---

Gaussian (analytic)
Gaussian (upper bound)
x PixelCNN (upper bound)
95% confidence
Single
0.6
LED
0.5
Estimated information (bits per pixel)
0.4
Differential
Phase
0.3
Contrast
0.2
0.1
Brightfield
0.0
100
200
300
400
500
Photons per pixel
Figure S19: Comparison of entropy estimation methods. Three approaches entropy estimation analytic stationary Gaussian, stationary Gaussian upper bound, and PixeICNN upper bound - across different illumination patterns and photon counts. The error bars for the upper bound methods (Gaussian and PixeICNN) are computed by bootstrapping both the marginal and conditional entropy estimates from the test set. For the analytic estimates, multiple Gaussian models were fit to boot- strapped versions of the full dataset to estimate H (Y), while H X) was bootstrapped as before. While the relationship between analytic estimates and true entropy lacks theoretical guarantees, they remain close to our best estimates across conditions.
performance across all encoders, classification accuracy varied significantly between encoders with similar total information content.
This disparity arises because classification requires only the information that distinguishes between classes. Most of the measured information (~307 bits for our 32x32 pixel images at 0.3 bits/pixel) cap- tures within-class variations irrelevant to the classification task, which requires most log, 10 3.32 bits. Different encoders may preserve this class flas-dissriminative information more or less effectively, even while capturing similar total information.
S5 Comparison to other frameworks
Mathematical assumptions about object structure underlie several established frameworks for imaging system design. Like our approach, estimation theory [117] and compressed sensing [118) offer ways to design imaging systems without requiring ground truth data numan-interprette
measurements. Estimation theory assumes objects can be described by few key parameters and uses the Cramér- Rao lower bound [199 120] to optimize system performance. Compressed sensing assumes objects sparse some basis, enabling efficient measurement with theoretical guarantees.
These frameworks take deductive approach: they start with specific assumptions about objects, encoders, and decoders to prove performance guarantees. While powerful, this approach has inherent limitations. The theoretical guarantees only apply when all assumptions can be difficult or impossible to verify in practice. Moreover, these frameworks often focus on best or worst-case scenarios for individual objects, which may not reflect the average performance over range of objects.
met-a condition that
Our information based framework can complement these approaches. Like estimation theory and compressed sensing, it can used deductively to derive theoretical limits when specific assumptions hold, as demonstrated in our analysis of two-point resolution (See Probabilistic two-point resolu fion). However, also offers a unique advantage: the ability to work inductively from experimental data. By estimating information content directly from measurements, our approach automatically
69

---

Mean squared error (MSE)
Structural similarity index (SSIM)
Peak signal-to-noise ratio (PSNR)
x10
0.971
95% confidence
47.4
11.5
Color filters
0.970
47.2-
MSE
SSIM
0.969
PSNR
10.5
47.0-
0.968
10
46.8-
9.5
46.6
1.0
1.2
0.9
1.1
0.9
1.1
Mutual information (bits per pixel)
Mutual information (bits per pixel)
Mutual information (bits per pixel)
Telescope configurations
x10*
0.80
0.75
MSE
6-
SSIM
0.70
PSNR
0.65
0.60
72-
2-
0.55
704"
100
200
300
200
300
100
200
300
Mutual information (bits)
Mutual information (bits)
Mutual information (bits)
x10
8-
Photons per pixel
25.0-
0.8
Phase masks
20
40
80
160
320
22.5-
SSIM
0.6
PSNR
20.0
MSE
4
17.5
0.4
15.0
2-
12.5-
0.2
0.2
0.4
0.6
0
0.2
0.4
0.6
0.2
0.4
0.6
Mutual information (bits per pixel)
Mutual information (bits per pixel)
Mutual information (bits per pixel)
Figure S20: Performance across multiple metries correlates with information estimates. Compar- ison of mean squared error (MSE), structural similarity index (SSIM), and peak signal-to-noise ratio (PSNR) for the color photography, radio telescope, and lensless imaging systems shown in Fig, 2 For the radio telescope configurations (middle row), colored dots correspond to the three example configurations shown in the main text, while black dots show additional tested configurations. Each metric shows consistent correlation with information estimates across different imaging modalities.
70

---

Example images
Mutual information estimates
Bits per pixel
320
0.0
0.1
0.2
0.3
0.4
Background Object
Brightfield
Object
Background
95% confidence
500
Back ground Oboject
Differential Phase Contrast
200
800
Background Object
Single LED
Figure S21: The effect of stochastic encoders on information estimates. By computing mutual information estimates on empty image patches with no objects them, the information about the system present in the measurements Differential Phase Contrast, and Brightfield illumination patterns on an LED array microscope, and estimating the mutual information between noiseless image and noisy measurement with and without an object. Encoders that capture more information about the object also capture more information about the system.
be estimated. This can be seen by comparing Single LED,
0.5
Mean photons per pixel
Classificttion accuracy
One Lens
150
0.4
80
Four Lens
0.3
Diffuser
0.2
20
0.1
0.2
Mutual information (bits per pixel)
Figure S22: Classification accuracy vs. information content in lensless imaging. Unlike de- convolution D)) classification performance does not show consistent relationship with total information content across different optical designs, suggesting the importance of preserving task- specific information. Points represent median over five classification networks with different random weight initializations.
71

---

adapts to the actual properties of objects being imaged, without requiring explicit mathematical models of the imaging process. The only assumption we rely on is a noise model, which itself could be learned from data rather than specified analytically.
This flexibility enables novel capabilities: comparing different imaging modalities, identifying subtle performance tradeoff's and understanding complex relationships between encoding physics and measurement quality. By learning from data rather than relying on predetermined assumptions, our framework can evaluate and optimize a broader range of imaging systems.
Detailed comparisons between our approach and these established frameworks are provided in Sections S5.1 and S5.2
S5.1 Estimation theory and the Cramér-Rao lower bound
Estimation theory provides tools for optimizing imaging systems by minimizing measurement uncertainty. Its central tool, the Cramér-Rao lower bound [117), defines the minimum achievable error when estimating object parameters from noisy measurements. This bound enables systematic comparison of different optical designs by identifying those with the lowest theoretical error.
While both estimation theory and our information-based framework aim to reduce measurement uncertainty (Sec. (S...44. our approach offers several key advantages:
Data-driven adaptation: Rather than requiring explicit mathematical models, our frame- work learns system behavior directly from measurements, automatically capturing complex optical effects.
Task flexibility: Estimation theory requires expressing imaging goals as parameter estima- tion problems (e.g. modeling point source localization in terms of position coordinates). Many practical imaging tasks lack such tractable parameterizations.
Minimal assumptions: We require only a noise model, which can be measured empirically, reducing the risk of model misspecification.
Decoder generality: The Cramér-Ree Raa bound traditionally applies only to unbiased estima- tors, which are rarely optimal in practice. While extensions to biased estimators exist, they introduce additional complexity without clear benefits (Sec. S5.1.3). Our framework places restrictions decoder type.
Our framework's flexibility allows it to operate in both deductive and inductive modes. When analytical object models are available, it can provide theoretical guarantees similar estimation theory. However, it can also learn directly from experimental data when such models are impractical. This dual capability enables our framework to handle wider range of imaging scenarios, from well-modeled systems to those with complex, difficult-to-modee behavior.
S5.1.1 Background: The Cramér-Rao lower bound in imaging
The Cramér-Rao lower bound has guided the design of numerous imaging systems. Its most successful application has been in fluorescence microscopy, where it has helped optimize point spread functions
for localizing single molecules [12]] 119 [24 120
[25]. Similar approaches have proven
valuable in phase microscopy [126] and wavefront sensing 127
At its core, the Cramér-Rao bound provides fundamental limit measurement precision. Consider measuring single parameter 0 (such as particle's 's position) from noisy data X. The measurement process is described by probability distribution p(x; 9), which captures both the imaging physics and measurement noise. For instance, in fluorescence microscopy, this distribution might describe how likely different intensity patterns are when imaging a point source position 0.
The bound states that any unbiased estimator 0(X) (one that gets the right answer average) must have a mean squared error greater than or equal to the inverse of the Fisher information:
E (0(x) (0(x)-0)} 0)² 2 (0)
(49)
where (0) is the Fisher information of 0, defined as:
72

---

Tr(0) = E (Vo log log
For many common distributions, the Fisher information behaves like the inverse of the distribution's ' variance. Intuitively, this means that noisier measurements (higher variance) make it harder to accurately estimate the true value of 6.
This mathematical framework has direct practical applications. In single- molecule microscopy, for example, the Cramér-Ral bound quantifies how precisely point source's position can be determined from an image. By calculating this bound for different optical designs (such various point spread functions), researchers can systematically identify and optimize systems that enable more precise localization.
It's important to note that Fisher information, despite sharing the word "information," is distinct from Shannon's mutual information and entropy used elsewhere in this work. While both concepts quantify aspects of measurement uncertainty, they arose from different and information theory- and were developed independently. Though theoretical connections exist between them in certain cases [17], they serve different purposes in analysis and design.
Bilds-estimation theory
S5.1.2 Cramér-Rao lower bound-based design and its limitations
The approach of designing physical systems based on the Cramér-Rao lower bound has inherent limitations that prevent application outside narrow set applications in which the decoding problem is limited to estimation of one few parameters. Furthermore, even when it can be applied, requires developing a complete mathematical model of the physics of image formation with respect to the parameter of interest. These limitations have thus far prevented its application outside a narrow range of problems, which include estimation of the position or mass of a single particle [12 119 124 120 [28), or estimating single optical aberration such as defocus [127] Below, we discuss these limitations in more detail.
The difficulties of creating parametric models Creating parametric models is essential for calculating the Cramér-Rao lower bound, but it becomes challenging when dealing with high- dimensional objects due to the difficulty of parameterizing objects in a way that captures important aspects of decoders, while also retaining computational tractability.
The computation of the Cramér-Ran lower bound requires calculation of Fisher Information, which in turn requires parametric model of the data distribution, denoted p(x; This parametric model stipulates the probability of a specific outcome given a certain parameter 0.
In certain scenarios, p(x; 0) can be analytically defined. For example, in the point localization problem. 0 denotes a 3-dimensional vector that marks the XYZ location of a point emitter (the bold type indicates it is vector of parameters), and denotes the intensity of specific pixel. Given known equations for the system's 's noise characteristics and image formation mechanism, p(r; 0) can be precisely defined. This can be used to compute Fisher Information and the Cramér-Rar Rao lower bound for single pixel, and the process can be repeated over each pixel x to get an averaged bound.
However, creating straightforward equation-based based model for imaging most types of objects is usually very challenging due to the difficulty and increasing complexity of adding additional parameters. Most objects being imaged lack the simplicity of single point emitter, which can be fully described by its XYZ location. One potential way to circumvent this issue is a more general purpose parameterization of the object in which it is represented by discrete array of pixel values [126). However, unbiased estimators are rarely used on high-dimend -igh-dimnssionl estimation problems like this. Furthermore, the Cramér Rao lower bound is in terms of mean squared error, which, when applied to arry of pixels, generally does not effectively capture semantically meaningful information about objects [29].
Generalizing from the simple case In light of the the difficulties of extending this approach to objects with sigh-dimensional parameterizations, another possibility is to design imaging systems on simple classes of objects with the hope that they will generalize to other classes of objects. Taking such an approach requires making additional assumptions with unknown effects on results. Our experiments in minimal I-Dimensional simulations show that the deeenende of imaging systems can be readily demonstrated (Sec. (Sec.SS1.3 which may in part explain why empirical solutions to particle localization problems deviate from theoretical predictions [[5]]
73

---

Unbiased estimators are usually suboptimal In addition to the practical difficulties of formulating complex parametric models and computing corresponding Cramér- lower bounds, there remains the limitation that the (standard) Cramér-Rao lower bound quantifies the minimum mean squared error only of unbiased estimators. While constraining an estimator to be unbiased arguably a reasonable choice for simple, low-dimensional parameters like the location of single point in 3D space, most estimators used in practice are in fact biased [130 [31 and state-of-the-at methods for solving image processing tasks almost exclusively use biased estimators achieve high performance.
The bias-variance tradeoff 132 provides a useful perspective: biased estimators, while possessing higher bias, can have lower variance, thereby potentially reducing overall error. Thus, with bias, estimators with better error than the standard Cramér-Rao lower bound can be achieved. A classic illustration of this principle is the James- Stein estimator [133], which improves the estimation of multiple parameters simultaneously by shrinking individual estimates towards a common mean. is based on the counterintuitive principle that, under certain conditions, an estimator that partially pools the data towards central value can produce overall estimates that are closer to the true values than those obtained by estimating each parameter independently, especially when dealing with small sample sizes or high-dimensional data. It results lower average error than the unbiased sample mean approach for Gaussian random variables when the number of dimensions is 2 3.
Furthermore, the advantages of biased estimators are clear on empirical problems, particularly high- dimensional ones. State of the art methods on amage-to-imase estimation problems like denoising and deconvolution are usually achieved using deep neural networks [34 or with iterative optimization procedures that use regularization to bias the estimates towards certain classes of solutions.
Since computational imaging relies heavily on biased estimators for most image processing tasks, designing systems focused solely on minimizing the error of unbiased estimators, or assessing empirical performance based on this criterion, can provide only narrow range of guarantees, and is unclear if the conclusions reached by these guarantees can be generalized to a broader range of applications. This raises of the question of what additional theoretic tools can be used address this more general case.
S5.1.3 The challenges of generalizing estimation-theoretie design
This section explores alternative approaches to estimation theory. focusing on the use of biased estimators and Bayesian Cramér-Rao lower bounds to address the limitations of the Cramér-Rao lower bound.
The biased Cramér-Rao lower bound Though the (standard) Cramér- lower bound only pertains to unbiased estimators, there are variants and related inequalities that can be applied more broadly.
The first is the biased version of the Cramér- Rao lower bound:
(1 Vab(o)) I+(0)
F [((()) - 0)2]
+ b(0)
Where b(0) is the bias of the estimator as a function of 6:
b(0) E 0
While this form may appear promising. it is in practice challenging for similar reasons to those described in section S5.1..2 Computing the gradient of the expectation of the estimator in high dimensions is a challenging statistical problem in its right, and this process would need to be repeated for each value of 0, necessitating another high-dimensional integration. Furthermore, this bound is not universal-ir it changes depending the bias of the estimator in question. This makes it more difficult to determine the best performance of an ideal theoretical estimator, and thus more difficult determine how close real estimator come to achieving that performance.
The Bayesian Cramér-Rao lower bound (van Trees inequality) One way of addressing the limitations of the (standard) Cramér-Rao lower bound for unbiased estimators and its biased estimator
74

---

variant can be found by generalizing the estimation problem to consider not just single value of the parameter but instead consider to also be random (the upper case è used denote the corresponding random variable).
Like the standard Cramér-Rao lower bound, the van Trees inequality (also known the Bayesian Cramér-Ra Rao lower bound) [35 130 provides a lower bound on the squared error that can be achieved in parameter estimation problem. Mathematically:
E (0(x) 2 [T(r J(0)
(50)
Compared to the standard Cramér-Rao lower bound (Equation 49 this inequality makes two important changes. First, the Fisher information of a particular parameter value @ has been replaced with probability-weighted average over all possible values of f. Second, there is an additional term ((), defined as:
J(O) = E (VV(()))
(51)
where p(.) is the probability of a particular value of O. This can be approximately understood as quantifying how concentrated the distribution of the random variable θ is. The more concentrated the distribution the parameter θ is, the more precisely it can be estimated from noisy measurements. Biasing estimates towards more probable values of @ enables estimation error to be lowered on average.
This inequality formalizes important intuition: The theoretical limit of the average error with which a parameter of interest can be estimated (such as some property of an object being imaged) is dependent upon the distribution of that parameter. Changing the distribution of the parameter can change the theoretical limits of performance, as well as the the form of optimal estimators.
S5.1.4 Connections between estimation and information theory
Originally, information-theorete quantities like entropy and mutual information were developed for noise-affected message transmission. However, these tools have significant theoretical links to estimations of the precision of noise-corrupted random variables. Though these connections are mostly known only for simpler analytical cases such as Gaussian random variables, they nonetheless provide insights into the relationships between these fields [117. Here we highlight some of these connections.
An important insight is that where these connections are recognized, designing imaging systems using either estimation information measures tends have similar objectives. However, estima- tion measures have inherent limitations in their applicability, as previously discussed. In contrast, information theory tools don't share these restrictions, positioning them potentially universal tools for designing physical imaging systems across a variety of applications. Several known inequalities capture the known relationships between these findings and goals of information and estimation theory.
Much of estimation theory centers limits defined in terms of mean squared error of signals. While mean squared error has many appealing properties, its shortcomings, particularly in the context of quantifying the perceptual and semantic quality of images readily apparent [12 This has, for example, motivated work on alternative ways of quantifying error [136] 137].
Efroimovich inequality contrast, rate distortion theory, a branch of information theory, can be used to understand the fundamental limits and behavior of wide variety of loss functions. The connection from the lower bounds used in estimation theory and information theory can be readily
seen in the Efroimovich inequality [138), which generalizes the van Trees inequality (Equation 50 from providing bound on only mean squared error to providing a bound on more general way of quantifying uncertainty, the entropy of parameter given noisy measurement X:
2M(e)))
2Te
E (1)
75

---

Here, /(() X) is the differential entropy of a parameter given data X. This inequality can be used derive bounds on loss functions other than mean squared error in terms of cooommatinnthhoreee
quantities 131 139
I-MMSE formula Another known relationship with particular relevance this work is known as the I-MMSE formula (short for Information minimum mean squared error), which states that [140] 117:
1 2 mmse (X yX + N)dy
(Xx N)
In this equation, X signal of interest, and Y V N is a noisy measurement of that signal, created by adding independent Gaussian noise N to the original signal such that the resultant signal-to-noise ratio vs. mmse(-) is the minimum mean squared error of estimating (i.e. the Bayesian Cramér-Rao lower bound shown in Equation 50). From this formula, the mutual information / (X; Y) is equal to the minimum mean squared error the optimal estimator, averaged over all achievable signal-to-noise ratios.
given Y
This formula shows that mutual information quantifies the same operational idea the (Bayesian) Cramér-Rao lower bound in the case of additive Gaussian noise, which strongly suggests that the quamtities may serve similar purposes under more general noise models. This relationship has been leveraged to develop task-specific information estimators [3..
This relationship intuition as to why these quantities are closely related. Figure S23 shows the distributions of noisy measurements for 6 different signals with measurement system that imparts additive Gaussian noise. Since mutual information is operationally defined as the number of signals that can be reliably distinguished in the presence of noise, decreasing the maximum signal-to-noise ratio lowers the mutual information. Simultaneously from the estimation theory perspective, it impedes the ability to estimate the original signal from noisy measurement of because increases the ambiguity as to which input signal gave rise to the measurement.
be visualized in the signal coordinate representation. which provides further
High signal-to-noise ratio
Low signal-to-noise ratio
Maximum energy constraint
Photons at X
Photons at x,
Photons at X,
Figure $23: Visualizing the connection between information and estimation. Shown noisy measurements of distinct signals with high and low signal-to-noise rations. With less noise, the signals remain more distinguishable, increasing mutual information. Simultancously. lower noise reduces uncertainty in inferring the true signal, improving estimability.
This theoretical connection, combined with the developments in the present work that enable estima- tion of mutual information across many types of imaging systems provides means of generalizing
76

---

the successes of estimation thhooretcc design criteria to a wide variety of imaging systems, without the requirements for detailed, gystem-specifee mathematical modeling.
S5.2 Compressed Sensing: Assumptions and Limitations
Compressed sensing theory showed that certain signals can be accurately reconstructed from fewer measurements than traditionally thought possible. While conventional sampling theory (Nyquist) requires sampling at twice the highest signal frequency, compressed sensing demonstrates that sparse signals- those with few nonzero components in some basis can be recovered from fewer measurements.
In its original formulation, the framework relies specific assumptions:
Signal sparsity: Most signal components must be zero in basis
The measurements of the signal should not be sparse. The degree of incoher- ence between the mcasurement matrix and the sparsity basis gauges the level to which this requirement is met.
Incoherence:
Gaussian noise: Measurement noise must be normally distributed and signal Aggal-independent
Specific reconstruction algorithms: Recovery requires optimization methods with provable guarantees
When these assumptions hold, compressed sensing approaches theoretical performance limits [41 However, real- world imaging often violates these assumptions. Natural objects rarely exhibit perfect sparsity, and measurement noise may not be Gaussian. Studies have shown that compressed sensing strategies become suboptimal when objects don't 't match the assumed structure (39 38
Recent work has shown that incorporating additional object knowledge improve upon basic compressed sensing. When signals can be generated by passing low-dimensional vectors through nonlinear models, reconstruction accuracy improve by an order of magnitude compared to traditional compressed sensing approaches [12
More broadly, probabilistic models offer framework for describing signal structure beyond sparsity, These models enable analysis of average-cas case performance and can incorporate various forms of prior knowledge about signals [14 [46]. Theoretical work has shown how measurement limits
depend fundamentally on the complexity of the objects being measured [147. and how probabilistic
approaches can achieve optimal performance for specific signal classes 148 149
S6
Methods
S6.1
Model training
Memory, computational, and data constraints often made it impractical to estimate information from full images. We therefore randomly sampled fixed-size patches from across the images for training.
Full Gaussian model The full (non-stationary) Gaussian model estimates complete covariance matrix directly from vectorized image data without assuming stationarity. The mean vector is computed the average across all training samples. The covariance matrix is estimated using the empirical covariance of the centered data. To ensure numerical stability and prevent degenerate solutions, we enforce positive definiteness by applying an eigenvalue floor - any eigenvalues below this threshold (typically 10 10-3) are set to the floor value. If numerical issues persist, the floor is automatically increased until the covariance matrix becomes positive definite. This model offers greater ficxibility than the stationary version but requires more training data and computation due to the larger number of parameters. For discrete-val valued data, small uniform noise is added during training prevent infinite likelihoods.
Stationary Gaussian process model The stationary Gaussian process model enforces translation invariance through a doubly Toeplitz structure in the covariance matrix. The model is initialized by computing the empirical covariance matrix from vectorized image patches and averaging along diagonals and block diagonals to impose the Toeplitz constraint. The mean is estimated as the
77

---

average pixel value across all patches. We then refine these estimates through an optimization procedure that maximizes the data likelihood. The optimization uses parameterization based on eigendecompasition, where only the eigenvalues are updated while keeping the eigenvectors fixed. We employ stochastic gradient descent with momentum and gradient clipping to prevent numerical instability. After each gradient step, a proximal operator ensures the eigenvalues remain above minimum floor value (typically 10-3) to maintain positive definiteness. The optimization typically runs for 60 epochs with early stopping based on validation likelihood, using batch sizes of 12 images and a learning rate 100. During both training and evaluation, small uniform noise is added to discrete-valle valued data to enable proper likelihood computation with the continuous model. The stationary assumption significantly reduces the number of parameters compared to the full Gaussian model, enabling more efficient training and better generalization with limited data.
PixeICNN model The images were modeled using a PixeICNN architecture adapted from prior work [53] [4.] The model consists of vertical and horizontal masked convolution layers arranged in stack, enabling modeling of the conditional distribution of each pixel given the previous pixels in raster scan order. The network includes seven gated masked convolution layers with alternating dilation to increase the receptive field while maintaining computational efficiency. Each gated layer combines information from vertical and horizontal stacks through masked convolutions followed by a gating mechanism using tanh and sigmoid activations.
rates (1,2,1,4,1,22,
We modified the original categorical output distribution to instead use Gaussian mixture den- sity [113] with 40 components at each pixel. The mixture parameters (means, standard deviations, and mixing weights) were computed through separate dense layers. The Gaussian means were initialized randomly uniformly between the training image minimum and maximum values and clipped to this range during inference. The standard deviations were computed the softplus output of a dense layer, clipped to be between 1 and the training set standard deviation to avoid degenerate solutions.
During training, small uniform noise was added to the discrete-valued images to account for modeling them with continuous distribution. The noise prevents the likelihoods from going to infinity and overfitting to the exact training values. Input images were normalized by subtracting the mean and dividing by the standard deviation of the training set. The model was trained using the Adam optimizer with learning rate of 0.01 on image patches, with early stopping based on validation likelihood to prevent overfitting. All models were implemented in JAX/Flax for efficient training GPUs.
S6.2 Confidence intervals
Our framework supports estimation of confidence intervals through bootstrapping quantify uncer- tainty from finite test set size. For a specified confidence level (e.g.. 90%), we repeatedly resample the test with replacement and recompute both the marginal entropy (using the trained model) and conditional entropy. The mutual information is computed for all combinations of these resampled estimates. The confidence interval is then determined by taking appropriate percentiles of these bootstrap estimates for example, the 5th and 95th percentiles for a 90% confidence interval. This approach captures uncertainty in both the marginal and conditional entropy estimates.
S6.3 Sample generation
Images are generated using ancestral sampling, where pixels are drawn sequentially in raster scan order (left to right, top bottom). For each new pixel, its value is sampled from a probability distribution conditioned on all previously generated pixels within its local neighborhood. For the Gaussian models, this is a Gaussian distribution whose parameters are computed using the Schur complement of the covariance matrix. For PixeICNN, this is mixture of Gaussians whose param- eters are computed by the neural network. To ensure physical plausibility in imaging applications, generated values can optionally be constrained to be non-negative by clipping negative values to zero. While slower than parallel generation methods, ancestral sampling maintains the local statistical dependencies captured by the models while enabling generation of arbitrarily large images.
78

---

S6.4 Generation of synthetic experimental noise
Experimental data was simulated to have been collected with fewer photons by adding simulated photon shot noise. However, since the experimental data already contains some shot noise, it is necessary to determine how noisy the images already are, and then only add additional noise as needed.
To simulate lower photon count, each pixel with photon count p was multiplied by a fraction f to reduce the photon count to fp. Assuming the dominant source of noise in the original image is photon shot noise, the variance of the noise in the original image is approximately p. and the variance in the reduced photon count image is approximately fºp. The desired variance of the reduced photon count image is equal to its mean, [p. Since the sum of two independent Gaussian random variables with variances a and b is a Gaussian random variable with variance a - b, we can add noise to the reduced photon count image to achieve the desired variance. Additional zero-mean Gaussian noise was added with variance fp f2p. The standard deviation of the added noise was then v ffp(1 - f).
S6.5 Applications
S6.5.1 Color Filter Array
Color imaging dataset We conducted experiments using the Gehler-Shi dataset [62] 63 (implied license via explicit permission use). which comprises 568 high-quality natural images. The dataset was partitioned into 461 training. 51 validation, and 56 test images, with each image subdivided into 24 24 pixel patches. To establish a consistent baseline for illumination, we computed a white channel by summing the red, green, and blue channels, then normalized the dataset to achieve a mean of 1000 photons per pixel in the white channel, based on the training split mean. Signal-dependent Poisson noise was subsequently applied to all patches.
Information content analysis For mutual information estimation, we sampled 100, 000 patches from the test set for fitting PixeICNN, reserving 10, 000 for evaluation. Conditional entropy calculations were performed on clean images prior to synthetic Poisson noise application. We evaluated three distinct color filter array (CFA) designs: a Bayer pattern, a random configuration, and a learned pattern using the architecture in [61] For each CFA design, we selected the minimum information estimate across the 20 replicates (since our estimator provides an upper bound) and the best prediction performance (lowest negative log likelihood on test data) to characterize system performance. Each replicate took ~3. of memory. All replicates were trained 5 hours on an Nvidia RTX A6000 GPU.
Neural network architecture and training Color image reconstruction was implemented using bifurcated network architecture as proposed by Chakrabarti (6]). This approach processes 24 x 24 pixel measurement patches through two parallel paths. The first path performs multiplicative operations in log-space with linear mixing, while the second employs convolutional layers with 128 filters. Each path generates 24 estimates per color value, which are then combined through learned weights for final reconstruction.
The network implementation features an x 8 repeated CFA pattern incorporating four channels (RGB and panchromatic). Training used the Adam optimizer (3) 0.9, B2 0.999) with learning rate of 1 x 10-5 and a batch size of 128 patches, running for a maximum of 100, 000 training steps with checkpoints every 5, 000 steps. Training progress was monitored using validation set comprising 10% of the training patches (100, 000 patches total). The checkpoint achieving the lowest validation loss was selected for final reconstruction performance evaluation.
$6.5.2 Black hole imaging
Black hole dataset For our site selection experiments, we generated a dataset of 100,000 synthetic black holes based on the physical models for black hole dynamics from [150]. Each synthetic black hole had a static envelope, right ascension, and source declination matching that of M87. The dynamic portion forming the accretion disk surrounding each black hole was modeled as a Gaussian random field generated by different solution to an underlying anisotropic spatio-temporal diffusion partial differential equation with fixed diffusion and advection fields. Each black hole was formed
79

---

by applying a Gaussian random field as multiplicative perturbation to the static envelope, and normalized to have total flux of Jansky. To simulate black hole imaging we selected combinations of four telescopes from the eight original telescope locations used in the Event Horizon Telescope 2017 array. We also included comparisons between reconstruction quality and information estimates for all 70 possible combinations of four telescopes (Fig. (520). For each combination we simulated radio telescope measurements including thermal noise using the methods described in [[5T]
Information content analysis For mutual information estimation from complex-va valuud radio telescope measurements we used the full Gaussian process model. Each complex-valued measurement with thermal noise was separated into real and imaginary components and concatenated into a single vector before being input to the model. The full Gaussian process was fit with 50,000 measurements and evaluated on a test set of 10,000 measurements with 100 bootstrapped estimates to form a 95% confidence interval. The conditional entropy was calculated based on the thermal noise statistics for each telescope array, modeled as Gaussian noise with independent and varying standard deviations at each measurement sample.
Imaging inverse problem Black hole images were reconstructed from simulated radio telescope measurements using a standard regularized maximum likelihood method with total variation regular- ization [3I], with hyperparameters matching the implementation in [152). Images were reconstructed for a 2,000 measurement subset of the test set used for information estimation. Each reconstruction was performed on the CPU and took 10s.
S6.5.3 Lensless imaging
The CIFARIO dataset [153] (implied license via permission to use) was used for the lensless imaging experiments. This dataset consists of 60,000 total images across 10 classes. To convert CIFARIO image pixel values synthetic photon counts, the dataset was scaled to have mean value equivalent to the desired photons per pixel. Encoded measurements for each lensless imaging system were generated by convolving (valid region, no zero padding) with the corresponding point spread function. The point spread functions for the one lens and four lens encoders were modeled as aouusiaan-blurte points. For the diffuser encoder, the point spread function is a downsampled version of an experimentally captured point spread function from DiffuserCam [67] After convolution, synthetic Poisson noise was added to form the noisy encoded measurements.
Natural image dataset
Information content analysis For mutual information estimation, images from the CIFARIO dataset were randomly selected in sets of 9 and tiled into 3x3 grids before convolution with the encoding point spread functions. This tiling was implemented to prevent intensity falloff at the edges of a measurement, which occurs when convolving a non-tiled image due to the zero padding necessary to maintain measurement size. With tiling, image content was brought in uniformly at every point in the field of view, forming a spatially consistent texture. Information content was estimated using 32 x 32 image patches, the same size as the original non-tiled CIFARIO 10 images. Patches were randomly sampled over the 65 x 65 region corresponding to the valid convolution output and 10,000 patches were used for information estimation. Conditional entropy was calculated using clean images without synthetic Poisson noise. For each encoder and photon count combination, trained four separate PixelCNN models to estimate information content, each including 100 bootstrap estimates with a test set of 1,500 images to form 95% confidence intervals. We selected the minimum information estimate across the replicated models to characterize system performance. Estimates consistent across replicates, with less than 0.01 bits per pixel difference across replicates and confidence intervals smaller than the marker size in graphs.
Image deconvolution For the deconvolution task, images from the CIFARIO dataset were con- volved with encoder. Then, Wiener deconvolution with automatically tuned regularization parameter was used to reconstruct the original image from each convolved measurement. This was performed on the CPU, taking 10 100 ms per reconstructuion.
Deconvolution is an ill-posed process and successful reconstructions with Wiener deconvolution require finite image extent. Therefore, instead of random tiling, we use non-tiled images with zero padding for this task. For each encoder and photon count combination, we evaluated image reconstruction metries test set of 1,500 images. The images in this test set correspond to the
80

---

center image of each of the 3 x 3 tiled measurements used in the information estimation test set. The mean value for each metric was reported and 95% confidence intervals generated from 100 bootstrap estimates with the test set were smaller than the marker size.
Image classification In addition to the deconvolution task in the main paper, we study object classification using the same encoders. A simple CNN architecture, sufficiently powerful for regular CIFARIO image classification is used. This consists of two convolutional layers (64 and 128 filters respectively with kemne size 5), each followed by a MaxPool, and two densely-connecte first with 128 nodes and a ReLU activation, and the second with 10 layers and a Softmax activation for classification into the 10 classes. Classifiers were trained <5GB memory, 10 minutes training time per model.
layers, the
NVIDIA TITAN XP GPU using
The random tiling process used in mutual information estimation is used for the images in this task as well. The label for classification based on the center image in the x 3 grid, for which maximum image content is present in the measurement. Classification is repeated 10 times for each photon count and encoder combination, from which the 90% confidence interval is generated.
S6.5.4 LED array microscopy
Cell imaging dataset We analyzed single leukocyte images and corresponding protein expression measurements from the Berkeley Single Cell Computational Microscopy (BSCCM) dataset [75) (CCO 1.0 Universal license). The dataset includes images acquired under multiple illumination conditions using an LED array microscope along with measurements of eight protein markers (CD3, CD 19, CD56, CD123, HLA-DR, CD14, CD16, and CD45) obtained through antibody staining. To study performance at different signal levels, we simulated lower photon counts by adding synthetic Poisson noise to the original images.
Neural network architecture and training We developed a neural network to predict protein expression levels from single-cell images. The network uses a DenseNet121 [I54] backbone pre- trained on ImageNet, modified to accept single-channel input by averaging the first convolutional layer's RGB weights. The backbone feeds into a global average pooling layer followed by eight parallel fully-connected networks, each with two hidden layers. These networks output parameters for Gaussian mixture models corresponding to each protein marker, enabling direct evaluation of the negative log likelihood for target protein levels. Training used the Adam optimizer with a learning rate of 5 x 10-5 and batches of 16 images. A composite loss function summed the negative log likelihood across protein markers, with missing values masked out. Training proceeded for 4,000 steps per epoch and employed early stopping when validation loss failed to improve for 20 epochs. Typical trainings took 1-2 days on NVIDIA TITAN Xp GPUs with 12GB memory. 2-3 networks were trained in parallel sharing a single GPU.
Information content analysis To evaluate system performance, we performed 15 independent train-test splits of the data. For each split, we trained both a PixeICNN model to estimate information content and protein prediction network. Information content was estimated using 40 x 40 pixel patches uniformly sampled from the normalized images. For each illumination condition and photon count, we selected the minimum information estimate across the 15 replicates (since our estimator provides an upper bound) and the best prediction performance (lowest negative log likelihood on test data) to characterize system performance.
S6.6 Information-Driver Encoder Analysis Learning (IDEAL)
partitioned into training (461 images), validation (51 images), and test (56 images) sets, we extracted 100,000 patches of size 24 x 24 pixels for training, along with 10,000 patches from the validation set. To establish consistent illumination conditions, the dataset was normalized to achieve a mean of 1000 photons per pixel in the white channel (sum of RGB channels) based on the training set statistics.
Using the Gehler-Shi dataset 162 63
The mutual information between noiseless and noisy measurements was calculated using an analytic Gaussian approximation to ensure differentiability of the loss function with respect to the learnable parameters (1). At each optimization step, we process a batch of 2304 measurement patches (24 x 24 pixels each), selected randomly from the Gehler-Shi dataset. These measurements were vectorized to
81

---

our loss compute their covariance matrix. whose log cigenvalues are used to estimate H().. For numerical stability, any non-po non-positive cigenvalues were set 10 S The batch size was chosen be 2304, times the number of pixels in a patch (576 pixels), as we found a batch of at least this size produces the most stable estimation. H(Y X) is calculated using the analytic formula detailed earlier (Sec. (4).
For the IDEAL optimization, we designed an 8 x 8 repeating color filter array (CFA) pattern with four possible filters each pixel: red, green, blue, and white. To learn which filter place each location, we used temperature-based softmax approach [[]]
m m(n) = Softmax(a
(52)
Here, m(n) is a 4-element vector that represents the probability of selecting each color filter at pixel location n. The learnable weights w(n) determine these probabilities through a softmax function whose sharpness is controlled by the temperature parameter Oa iteration and 0.001. Early in training when a₂ is small, the softmax produces soft probabilities that allow gradient-based learning. As training progresses and at increases, these probabílities become increasingly binary, eventually forcing the network to select a single filter at each location.
(74)², where is the training
The training procedure begins by passing clean image patches through the CFA pattern generate noiseless measurements. Signal-dependent Poisson noise is then approximated using Gaussian noise with variance matching the signal intensity. We the negative mutual information function, optimized using the AdamW optimizer (learning rate 10-4 31 = 0.9, .2 = 0.999) until convergence. Training took ~3 hours and ~ GB Nvidia RTX A6000 GPU.
To evaluate IDEAL's ability to optimize a CFA, we computed the mutual information on the test set for the optimized pattern, as well as initial and intermediate patterns, using the estimation procedure described previously (See (Color Filter Array). Reconstruction networks were subsequently trained for these three filters and evaluated following the protocol detailed above SCeoColor Filter Array).
S6.7 1D simulations
We studied a simplified 1D imaging system with three key components: objects, encoders (point spread functions), and measurements. Objects were represented as periodic signals over a fixed spatial domain of 512 samples. Encoders were constrained to be bandlimited, non-negative, and infinitely periodic point spread functions that acted on objects through convolution. To ensure bandlimiting and enable stable optimization, encoders were parameterized the Fourier domain using amplitude and phase components up to specified bandwidth. The encoding process generated signals by convolving objects with these point spread functions. These signals were then integrated over fixed intervals (" ("pixels") to simulate detector sampling, with independent additive Gaussian noise applied to each pixel to model measurement noise.
Information content was estimated using our PixelCNN-based estimator, requiring 1D signals to be reshaped into 2D arrays. Training datasets were generated by encoding random object instances drawn from specified distributions. To characterize encoder constraints, we mapped the space of achievable signals by optimizing encoders to match target signals, defined as points in the space of possible pixel measurements. This optimization minimized the L2 distance between encoded and target signals. We analyzed how information capacity varied with key system parameters. Signal- to-noise ratio effects were studied by varying noise standard deviation while keeping signal energy fixed. Bandwidth impact was assessed by adjusting the number of nonzero Fourier components the encoder. Sampling density analysis involved changing the number of pixels while maintaining fixed bandwidth. These relationships were examined across different object distributions including single delta functions, multiple delta functions, and bandlimited noise patterns.
Encoder optimization employed stochastic gradient ascent on a Gaussian -aprriianappoooimatee mutual information objective, with a proximal operator enforcing non-negativity and energy constraints on the point spread functions. All simulations were implemented in JAX for automatic differentiation and GPU acceleration.
82


Reviews:
[{"review_id": "IILvmGQeQz", "reviewer": "Reviewer_HkSv", "summary": "This paper introduces a data-driven framework for estimating mutual information in imaging systems, enabling information-theoretic design without ground truth data. The approach fits probabilistic models to noisy measurements and quantifies information content by subtracting noise entropy. It validates across diverse applications—color photography, radio astronomy, lensless imaging, and microscopy—showing that mutual information reliably predicts system performance. The Information-Driven Encoder Analysis Learning (IDEAL) framework optimizes imaging systems via gradient-based methods to maximize information capture, achieving up to 3.8× faster training with 97% performance retention compared to full-data training.", "questions": "1. The framework focuses on static images, but real-world imaging (e.g., video) involves temporal correlations unaddressed in the current model. How does the method handle dynamic scenes with time-varying noise?\n\n2. The study lacks validation on multi-spectral data, where cross-channel dependencies may alter mutual information.\nCan IDEAL optimize systems for real-time processing constraints? \n\n3. The framework prioritizes information maximization but doesn’t address latency-accuracy trade-offs in real-time applications.\nWhat is the impact of multi-modal imaging (e.g., RGB-NIR) on information estimation?\n\n4. Most experiments use supervised decoders (e.g., neural networks), but unsupervised scenarios (e.g., self-denoising) remain untested. How does the method perform with unsupervised or self-supervised decoding?", "limitations": "see weakness.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths：\n1. No ground truth required: Estimates information directly from measurements, avoiding reliance on labeled data .\n2. Broad applicability: Works across imaging modalities (photography, astronomy, microscopy) without task-specific assumptions .\n3. Quantitative design metric: Unifies resolution, SNR, and sampling trade-offs into a single information-theoretic metric .\n4. Gradient-based optimization: IDEAL enables automated system design (e.g., color filter arrays) with up to 3.8× training speedup .\n\nWeaknesses：\n1. Computational complexity: PixelCNN models require more training data and time compared to Gaussian models .\n2. Noise model dependency: Relies on accurate noise characterization (e.g., Gaussian/Poisson), which may fail in complex noise scenarios .\n3. Task generality trade-off: Total information may not align with task-specific needs (e.g., classification vs. reconstruction) .", "was_revised": false, "final_justification": "", "rebuttal": "**Notice: This review appears to be AI-generated and has been reported to the Area Chair.**\n\nThe review contains fabricated statistics not present in our paper (e.g. \"3.8× faster training with 97% performance retention\"), which don't make sense in the context of our work. Furthermore, the review demonstrates no substantive critical analysis, instead merely paraphrasing content from our manuscript without evaluation."}, {"review_id": "jWnGfIIPQQ", "reviewer": "Reviewer_Sgt1", "summary": "This paper introduces an information-driven design framework for imaging systems, focusing on mutual information estimation between unknown objects and noisy measurements. It proposes a data-driven method to quantify the information content of measurements, and combines this information estimation approach with gradient-based optimization to design imaging systems that maximize information capture.", "questions": "Questions for the authors to address in the rebuttal: \n1.\tAs mentioned in weakness, how does the proposed method address the challenge of continuously evolving measurements during imaging system optimization, especially without real-time adjustable hardware for data collection, which may still require ground truth data for training?\n2.\tDoes the generalization of maximizing information content sacrifice task-specific accuracy, and can the method be experimentally validated against previous end-to-end optimization approaches involving decoders?\n3.\tHow does the proposed method handle cases where the noise model may not be fully characterized or differs from the assumed model , such as those complex noise models in thermal infrared imaging.\nAddressing these questions in the rebuttal would greatly enhance the clarity and applicability of the proposed method, thereby improving the paper's evaluation.", "limitations": "The method assumes static measurements, but in real-world imaging system optimization, measurements continuously evolve with changing imaging parameters. Without real-time adjustable hardware to collect new data, the training process may still rely on ground truth data to simulate new measurements. The authors should clarify how this challenge is addressed.\nWhile the approach avoids post-processing decoders, it remains unclear whether this generalization sacrifices task-specific accuracy. The authors should provide experimental validation comparing this method with previous end-to-end optimization approaches involving decoders.\nThe noise model assumed in the paper may not fully account for complex noise scenarios, such as those encountered in thermal infrared imaging. The authors should explain how their method handles such cases, where noise characteristics may differ significantly from the assumed models.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 2, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths.\n+ The interesting method effectively bridges a critical gap in imaging system design, by quantifying information content directly from measurements without needing post-processing decoders. \n+ The paper demonstrates the applicability of the approach in a wide range of imaging systems, including real-world use cases like black hole imaging and biological microscopy. \n+ The theoretical foundation of the paper is robust, offering a clear mathematical formulation for mutual information and its estimation. \n+ The IDEAL method for optimizing encoder designs based on information content shows great promise, especially in terms of reducing computational overhead by focusing on the encoder alone before considering the decoder.\n\nWeakness. \nNevertheless, I believe the authors still need to address the following critical concerns. \n- The authors claim that the information content can be quantified directly from a dataset of measurements without requiring ground truth data. However, in the process of optimizing imaging systems, the imaging parameters are constantly changing, which implies that the measurements are also continuously evolving. Without adjustable imaging hardware to collect new training data in real-time (which is relatively difficult to achieve), this inevitably requires the training process to rely on ground truth data to simulate new measurements. \n- Furthermore, although the method does not require post-processing decoders, it raises the question of whether this generalization compromises the accuracy for specific tasks. It needs to be experimentally demonstrated whether maximizing information content ultimately leads to the optimal task performance. For instance, in an imaging modality experiment, a comparison between this method and previous end-to-end optimization approaches involving decoders should be provided to show the effectiveness.", "was_revised": true, "final_justification": "I have always recognized the originality of this article, and the response satisfactorily addressed my doubts; therefore, I raised my vote.", "rebuttal": "Thank you for your thorough review and thoughtful questions. We appreciate the time you invested in engaging with our work. Your insights have helped us clarify both our presentation and our thinking about future directions for information-driven design.\n\n\n\n# W1/Q1: changing measurements and ground truth data\n\n\n> The method assumes static measurements, but in real-world imaging system optimization, measurements continuously evolve with changing imaging parameters. Without real-time adjustable hardware to collect new data, the training process may still rely on ground truth data to simulate new measurements. The authors should clarify how this challenge is addressed.\n\n> The authors claim that the information content can be quantified directly from a dataset of measurements without requiring ground truth data. However, in the process of optimizing imaging systems, the imaging parameters are constantly changing, which implies that the measurements are also continuously evolving. Without adjustable imaging hardware to collect new training data in real-time (which is relatively difficult to achieve), this inevitably requires the training process to rely on ground truth data to simulate new measurements.\n\n> How does the proposed method address the challenge of continuously evolving measurements during imaging system optimization, especially without real-time adjustable hardware for data collection, which may still require ground truth data for training? \n\n\nThis question highlights an important distinction about deployment scenarios for information-driven design. Consider three key scenarios with different data availability and optimization constraints:\n\n**1) Ground truth-free system comparison (field measurements):** Our method can estimate information content directly from real-world measurements without ground truth data. This enables quantitative performance evaluation in real deployment conditions that were previously inaccessible to systematic analysis. Figure 2d exemplifies this: we can quantitatively rank microscopy illumination patterns without knowing the molecular structure of the cells being imaged. Similarly, a self-driving car company could compare three prototype cameras by collecting field images and comparing encoded information from each camera. \n\n**2) Pre-fabrication design (simulation):** This scenario requires ground truth data and system simulation to optimize designs before fabrication using IDEAL. While end-to-end optimization methods can achieve similar goals, IDEAL offers advantages in this simulated design scenario: it optimizes only the encoder rather than the full pipeline, significantly reducing computational and memory requirements during the design process.\n\n**3) Adaptive optimization (real-time):** Using information estimates to optimize system parameters in real-time represents largely unexplored territory with both significant challenges and significant potential. This scenario faces fundamental challenges the reviewer correctly identifies: datasets may contain heterogeneous combinations of objects and parameter settings, and during real-time operation, distinguishing whether measurement changes arise from object variation or parameter adjustments requires additional constraints such as assuming objects remain stationary. Despite these challenges, information offers potential advantages as a universal optimization signal that doesn't require algorithm-specific objective functions. This could enable simultaneous optimization of multiple interacting parameters—capabilities difficult to achieve with conventional approaches. We plan to explore this direction in future work.\n\n# W2/Q2: Specialized tasks and end-to-end\n\n> Does the generalization of maximizing information content sacrifice task-specific accuracy, and can the method be experimentally validated against previous end-to-end optimization approaches involving decoders? \n\nYes, we have validated IDEAL against end-to-end methods—it achieves comparable performance while using less memory and compute. Our general information-maximization approach achieves optimal performance for image reconstruction tasks, which are ubiquitous across imaging applications. Specialized tasks would require a natural extension of our framework that we outline below.\n\n**IDEAL vs end-to-end:** IDEAL matches end-to-end optimization performance as shown in Figures 2a and 3c (compare blue points). We acknowledge this comparison is presented confusingly across multiple figures and will clarify it in our revision. Follow-on work has compared IDEAL and end-to-end across additional systems, providing additional evidence of IDEAL's competitive or superior performance (see anonymized PDF in supplementary materials).\n\n\n**General task performance:** For reconstruction tasks, maximizing information achieves optimal performance because reconstruction requires preserving all object details. Figures 2a-c and S20 demonstrate tight correlations between information estimates and reconstruction quality across all tested systems, aligning with theoretical results [`1`] that establish this fundamental relationship (further discussed in Section S5.1.4).\n\nHowever, information maximization offers unique advantages over direct reconstruction metrics like mean squared error: computational efficiency in simulated design and ground-truth-free evaluation for field measurements.\n\nReconstruction is ubiquitous across imaging applications—from astronomy to consumer photography to medical imaging—because\ncapturing maximum detail is often the primary goal. Accordingly, most existing end-to-end approaches also optimize for reconstruction performance. Thus, our general information maximization approach is already broadly applicable without modification.\n\n\n**Specialized task performance:** For specialized tasks, maximizing total information may be suboptimal. Section 4.1 and Figure S22 demonstrate this: classification tasks using only ~1\\% of total information show weaker correlation between information and performance compared to reconstruction tasks.\n\nOur framework naturally extends to these cases by decomposing information into task-specific and task-irrelevant components:\n$I(\\mathbf{Y};\\mathbf{O}) = I(\\mathbf{Y};\\mathbf{T}) + I(\\mathbf{Y};\\mathbf{O}|\\mathbf{T})$. This enables weighted objectives for\nvarious applications:\n\n- Balancing task-specific and general information for flexible systems\n- Maximizing task-relevant while suppressing irrelevant information for bandwidth-limited scenarios\n- Preserving general information while suppressing task-specific information for privacy\n\nWe provide additional mathematical and implementation detail in Section S1.5.2 and in our response to Reviewer NSW5 below.\n\n`1` *The Interplay Between Information and Estimation Measures* (2013) \n\n# Q3: Noise model mismatch\n\n> The noise model assumed in the paper may not fully account for complex noise scenarios, such as those encountered in thermal infrared imaging. The authors should explain how their method handles such cases, where noise characteristics may differ significantly from the assumed models.\n\n> How does the proposed method handle cases where the noise model may not be fully characterized or differs from the assumed model, such as those complex noise models in thermal infrared imaging?\n\nYou've identified an important consideration that needs clearer explanation in our revision. \n\nOur method remains robust to noise model mismatch in most practical scenarios. When accurate noise modeling does become essential, this requirement is not unique to our approach.\n\nImaging noise typically exhibits one of two properties that provide robustness to noise model mismatch: (1) statistical independence from the object, or (2) conditional independence given the noiseless image. The additive Gaussian and Poisson models we tested exemplify these cases and provide a blueprint for other noise types, including those in thermal infrared imaging.\n\nWhen noise is object-independent (property 1), an incorrect noise model affects mutual information estimates by an additive constant, which means comparisons between encoders and IDEAL optimization gradients remain unchanged. The complex noise encountered in thermal imaging is primarily object-independent, for example thermal (Johnson) noise, dark current, fixed pattern noise, and 1/f noise [`1`, `2`, `3`].\n\nWhen noise is object-dependent (property 2), an incorrect noise model can affect relative encoder evaluations. However, object-dependent noise typically arises from well-understood\nquantum processes with simple mathematical structure, making estimation of its entropy reliable and tractable. Poisson shot noise is the most common example. In the supplement (S2.3.3), we demonstrate how its conditional independence given the noiseless signal allows its estimation to be reduced to a tractable 1D calculation.\n\nAccurate noise modeling becomes essential only when comparing across encoder designs that directly affect noise patterns, such as comparing thermal sensors with different fixed pattern noise. Even then, property 1 enables straightforward empirical characterization of the noise without having to consider the complexities of the objects being imaged.\n\nIn this setting, the need for accurate noise models is not unique to our approach—it is commonplace for methods applied to applications sensitive to complex noise patterns. For example, low-light denoising methods require careful physics-based noise modeling to handle extreme conditions [4].\n\n\n`1` *Modeling Noise in Thermal Imaging Systems* (1993)\n\n`2` *Exploring Video Denoising in Thermal Infrared Imaging: Physics-Inspired Noise Generator, Dataset, and Model* (2024)\n\n`3` *Thermal Image Processing via Physics-Inspired Deep Networks* (2021)\n\n`4` *Dancing under the stars: video denoising in starlight* (2022)"}, {"review_id": "VXAGaGjB8y", "reviewer": "Reviewer_NSW5", "summary": "This paper introduces a data-driven framework for estimating mutual information in imaging systems using only measurement data and a noise model, without requiring ground truth or object assumptions. The method fits probabilistic models to estimate entropy and information content, enabling objective evaluation across different imaging modalities. It is validated on applications like color photography, black hole imaging, and microscopy, showing strong correlation with task performance. The authors also propose IDEAL, an optimization pipeline that designs imaging systems by maximizing information capture through differentiable models. This work provides a general tool for evaluating and optimizing imaging systems based on information theory.", "questions": "1. How sensitive is the proposed estimator to model mismatch or data scarcity? The paper uses parametric models (e.g., Gaussian, PixelCNN) to estimate entropy, but does not analyze the sensitivity of the mutual information estimates under model mismatch (e.g., non-Gaussian data fit with Gaussian models) or limited data regimes. Could the authors provide quantitative or empirical analysis (e.g., bias/variance, convergence with sample size) to assess estimator robustness? Demonstrating estimator reliability under varying conditions would strengthen confidence in practical applicability and improve the overall evaluation.\n\n2. Can the authors report the runtime and memory cost of different estimators, especially within IDEAL? PixelCNN is noted as the most accurate but computationally intensive option. However, the paper lacks detailed comparisons of training time, memory usage, and scalability across models. This is especially important for real-world deployment and adoption of IDEAL. A clear runtime analysis would validate the claim of practicality and support the scalability of the method in larger design spaces.\n\n\n3. How does the method perform when mutual information is not aligned with task relevance? The authors acknowledge that total mutual information may not fully predict performance in tasks like classification (e.g., microscopy application). Could the authors expand on potential extensions to task-aware information metrics or suggest how IDEAL could incorporate task-specific weighting? Outlining a concrete path toward task-adaptive extensions would improve the method’s flexibility and broaden its relevance.\n\n4. Can IDEAL be compared with task-specific or end-to-end design pipelines? While IDEAL is promising, its comparison is limited to internal metrics (e.g., mutual information or MSE). Can the authors compare IDEAL-based encoder designs with those from task-specific or end-to-end learned systems in terms of downstream task accuracy and efficiency? Empirical evidence of IDEAL's competitive or superior performance versus baselines would significantly enhance the strength of the design contribution.\n\n5. Could you clarify assumptions behind the equivalence I(O;Y) = I(X;Y)? The estimator substitutes I(O;Y) with I(X;Y) under the assumption that X deterministically depends on O. Could the authors provide practical examples or guidance on when this assumption fails and how the method might adapt (e.g., when the encoder is stochastic)? Better explanation of this core assumption would improve theoretical clarity and help readers understand the estimator’s limits and adaptation potential.", "limitations": "The authors address key limitations, including the gap between total and task-specific information, and the computational cost of expressive models. To strengthen this section, it would help to include runtime comparisons, clarify estimator reliability under limited data, and discuss cases where the I(O;Y) = I(X;Y) assumption may fail.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 4, "originality": 3, "strengths_and_weaknesses": "### **Strengths**\n\nThis paper introduces a practical, well-founded method to estimate mutual information in imaging systems without ground truth, and validates it across diverse applications. The proposed IDEAL framework adds a novel, differentiable way to optimize system design. Overall, the work is clear, technically sound, and broadly impactful for imaging system evaluation and optimization.\n\n\n### **Weaknesses**\n\n1. While the method relies on parametric entropy estimation, the paper does not quantify the estimator’s bias or variance across models, particularly in high-dimensional or low-sample regimes. This leaves some uncertainty about reliability in practice.\n\n2. The PixelCNN-based estimator is significantly more accurate but also more computationally expensive (\\~100s training time). However, detailed runtime or memory comparisons between models (especially within IDEAL) are not provided, making it hard to assess practical scalability.\n\n3. Sections on model implementation (e.g., entropy estimation using parametric fitting) could benefit from more concrete examples or pseudocode. Additionally, the notation between I(O;Y) and I(X;Y) may be confusing to readers unfamiliar with the assumptions.", "was_revised": true, "final_justification": "Although the authors’ response has addressed some of my concerns, I feel that the rebuttal lacks sufficient depth, particularly in the answer to Question 5, which is not convincing to me. That said, I do appreciate this work, despite some flaws.", "rebuttal": "Thank you for your thorough review and thoughtful questions. It's clear you've deeply understood our work, and your insights have helped us refine both our thinking and presentation.\n\n# W1/Q1: estimator bias/variance\n\n> While the method relies on parametric entropy estimation, the paper does not quantify the estimator’s bias or variance across models, particularly in high-dimensional or low-sample regimes. This leaves some uncertainty about reliability in practice.\n\n> How sensitive is the proposed estimator to model mismatch or data scarcity? The paper uses parametric models (e.g., Gaussian, PixelCNN) to estimate entropy, but does not analyze the sensitivity of the mutual information estimates under model mismatch (e.g., non-Gaussian data fit with Gaussian models) or limited data regimes. Could the authors provide quantitative or empirical analysis (e.g., bias/variance, convergence with sample size) to assess estimator robustness? Demonstrating estimator reliability under varying conditions would strengthen confidence in practical applicability and improve the overall evaluation.\n\nFigure S16 in the supplement analyzes estimator consistency and the estimator performance with dataset size.\n\n**Convergence:**\n All three estimators (stationary Gaussian, full Gaussian, PixelCNN) converge to true values when tested on known distributions, with performance depending on how well data matches model assumptions.\n\n**Model mismatch:** On real data, Gaussian models converge to higher estimates than PixelCNN because they cannot accurately fit complex data distributions, demonstrating predictable bias patterns under model mismatch.\n\n**Data scarcity:** PixelCNN performs well across scenarios but is outperformed by simpler models on small datasets.\n\nWe welcome suggestions on expanding this analysis to better serve practitioners. \n\n\n# W2/Q2: runtime/memory comparisons\n\n> The PixelCNN-based estimator is significantly more accurate but also more computationally expensive (~100s training time). However, detailed runtime or memory comparisons between models (especially within IDEAL) are not provided, making it hard to assess practical scalability.\n\n> Can the authors report the runtime and memory cost of differbent estimators, especially within IDEAL? PixelCNN is noted as the most accurate but computationally intensive option. However, the paper lacks detailed comparisons of training time, memory usage, and scalability across models. This is especially important for real-world deployment and adoption of IDEAL. A clear runtime analysis would validate the claim of practicality and support the scalability of the method in larger design spaces.\n\n\nWe agree that these runtime and memory comparisons are important. The supplement contains some of this information, and we will expand it to provide the comprehensive analysis suggested.\n\n\n# W3: model implementation examples/pseudocode\n\n> Sections on model implementation (e.g., entropy estimation using parametric fitting) could benefit from more concrete examples or pseudocode. Additionally, the notation between I(O;Y) and I(X;Y) may be confusing to readers unfamiliar with the assumptions.\n\n\nWe will link to our Python implementation to provide working code examples. We welcome \nspecific suggestions about which aspects would benefit most from additional clarification.\n\n\n# Q3: Extensions to task-specific information\n\n> How does the method perform when mutual information is not aligned with task relevance? The authors acknowledge that total mutual information may not fully predict performance in tasks like classification (e.g., microscopy application). Could the authors expand on potential extensions to task-aware information metrics or suggest how IDEAL could incorporate task-specific weighting? Outlining a concrete path toward task-adaptive extensions would improve the method's flexibility and broaden its relevance.\n\n\nIn its current form, our method's performance with specialized tasks depends on the overlap between task-specific and total information, which likely varies across applications. However, we envision several promising pathways to extend our framework for task-specific optimization.\n\nThe difference between Figure S22 and Figure 2c demonstrates how maximizing total information may be suboptimal for specialized tasks: for a reconstruction task (Fig 2c) where all information aids performance, mutual information correlates strongly with error. For a classification task where only ~1\\% of the total information is task-relevant, mutual information correlates less strongly with accuracy.\n\nThere are multiple promising avenues to extend information estimation to task specificity in future work:\n\nIn its current form, IDEAL-optimized systems could serve as pre-trained starting points for task-specific refinement with end-to-end optimization, analogous to how LLMs are pre-trained as a first step. This approach could reduce labeled data requirements for task-specific imagers.\n\n\nWe can also directly extend our mathematical framework to estimate task-specific information. This requires estimating $I(\\mathbf{Y};\\mathbf{T})$, where $\\mathbf{T}$ represents task variables like classification labels or segmentation masks. The quantity decomposes as $I(\\mathbf{Y};\\mathbf{T}) = H(\\mathbf{Y}) - H(\\mathbf{Y}|\\mathbf{T})$. We could estimate $H(\\mathbf{Y})$ with our existing approach and $H(\\mathbf{Y}|\\mathbf{T})$ by conditioning our models on task information. This requires only modifying our models to take both measurements and task labels as input.\n\nSince total information naturally decomposes into task-specific and task-irrelevant components: $I(\\mathbf{Y};\\mathbf{O}) = I(\\mathbf{Y};\\mathbf{T}) + I(\\mathbf{Y};\\mathbf{O}|\\mathbf{T})$, we can design systems with weighted objectives $\\alpha I(\\mathbf{Y};\\mathbf{T}) + \\beta I(\\mathbf{Y};\\mathbf{O}|\\mathbf{T})$. This could be used for applications such as:\n\n**Balanced systems** ($\\alpha,\\beta > 0$):} Control the relative importance of task-specific versus general information, potentially enabling systems that balance current task performance with future flexibility.\n\n**Compressed sensing** ($\\alpha > 0, \\beta < 0$):} Maximize task-relevant information while suppressing task-irrelevant information, which could be valuable for bandwidth-limited applications.\n\n**Privacy-preserving** ($\\alpha < 0, \\beta > 0$):} Suppress task-specific information while preserving general scene information for other analyses.\n\n# Q4: IDEAL vs End-to-End\n\n> Can IDEAL be compared with task-specific or end-to-end design pipelines? While IDEAL is promising, its comparison is limited to internal metrics (e.g., mutual information or MSE). Can the authors compare IDEAL-based encoder designs with those from task-specific or end-to-end learned systems in terms of downstream task accuracy and efficiency? Empirical evidence of IDEAL's competitive or superior performance versus baselines would significantly enhance the strength of the design contribution.\n\nThis comparison already exists in our current results: IDEAL achieves comparable performance to end-to-end optimization methods (compare blue points in Figures 2a and 3c) while using less memory and compute. We acknowledge that this comparison is presented confusingly across multiple figures and will clarify the presentation in our revision.\n\nFollow-on work has systematically investigated this across a wider variety of systems, providing empirical evidence of IDEAL's competitive or superior performance versus end-to-end methods, with results included as an anonymous PDF in the supplementary materials.\n\n\n\n# Q5: $I(\\mathbf{O};\\mathbf{Y}) = I(\\mathbf{X};\\mathbf{Y})$ assumptions\n\n> Could you clarify assumptions behind the equivalence I(O;Y) = I(X;Y)? The estimator substitutes I(O;Y) with I(X;Y) under the assumption that X deterministically depends on O. Could the authors provide practical examples or guidance on when this assumption fails and how the method might adapt (e.g., when the encoder is stochastic)? Better explanation of this core assumption would improve theoretical clarity and help readers understand the estimator’s limits and adaptation potential.\n\n\nThis assumption is key to our framework. We want to make it as clear as possible and welcome suggestions on whether more of this discussion should be moved from the supplement to main text.\n\nAs described in Section S1.5.1, the equivalence $I(\\mathbf{O};\\mathbf{Y}) = I(\\mathbf{X};\\mathbf{Y})$ holds when the encoder function is deterministic—noiseless images $\\mathbf{X}$ depend deterministically on objects $\\mathbf{O}$. This enables practical information estimation without access to the objects themselves.\n\nThe assumption fails when encoders vary between different objects. Examples include changing dust patterns across measurements, illumination fluctuations that differ for each object, or mechanical drift in focus across objects. Fixed system imperfections (e.g. constant dust patterns or defocus) do not violate this assumption because the encoder remains deterministic.\n\nWhen encoder variations occur, measurements contain information about both objects and the system itself, leading to overestimated object information. Section S4.1 demonstrates this effect experimentally by estimating information of measurements without objects.\n\nWe describe how this could theoretically be addressed by estimating $H(\\mathbf{Y}|\\mathbf{E})$ instead of $H(\\mathbf{Y})$, where $\\mathbf{E}$ represents encoder state in Section S2.7."}]

Author rebuttal:

Meta-review: The paper presents a framework for the data-driven design of imaging systems, with experiments on designing color sensors, optimizing the packing of radio telescopes for black hole imaging, and related applications. The framework is based on approximating the mutual information between the underlying signal and the captured data—a task that is particularly challenging in high-dimensional settings such as images. To address this, the authors develop Gaussian Process and neural network–based approximations, and demonstrate that the framework leads to improved design choices.

The paper was well received by the reviewers. While there are aspects that could be further refined, the AC concurs with the reviewers that the work makes solid contributions and recommends acceptance. The authors are encouraged to improve the presentation of the results and address the specific issues raised during the review process. The AC also notes the broader context of concerns around LLM-generated reviews, but emphasizes that this did not factor into the decision-making for this submission.
Decision: Accept (poster)