Title: Information-Driven Design of Imaging Systems
Venue: NeurIPS 2025
Accepted: yes
Keywords: Mutual information estimation

Computational imaging

Information theory

entropy estimation

Encoder and sensor design

Imaging system evaluation

computational photography

Microscopy

Astronomy
Abstract: Imaging systems have traditionally been designed to mimic the human eye and produce visually interpretable measurements. Modern imaging systems, however, process raw measurements computationally before or instead of human viewing. As a result, the information content of raw measurements matters more than their visual interpretability. Despite the importance of measurement information content, current approaches for evaluating imaging system performance do not quantify it: they instead either use alternative metrics that assess specific aspects of measurement quality or assess measurements indirectly with performance on secondary tasks. 

We developed the theoretical foundations and a practical method to directly quantify mutual information between noisy measurements and unknown objects. By fitting probabilistic models to measurements and their noise characteristics, our method estimates information by upper bounding its true value. By applying gradient-based optimization to these estimates, we also developed a technique for designing imaging systems called Information-Driven Encoder Analysis Learning (IDEAL). 

Our information estimates accurately captured system performance differences across four imaging domains (color photography, radio astronomy, lensless imaging, and microscopy). Systems designed with IDEAL matched the performance of those designed with end-to-end optimization, the prevailing approach that jointly optimizes hardware and image processing algorithms. These results establish mutual information as a universal performance metric for imaging systems that enables both computationally efficient design optimization and evaluation in real-world conditions.

A video summary of this work can be found at: https://waller-lab.github.io/EncodingInformationWebsite/


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a data-driven framework for estimating mutual information in imaging systems using only measurement data and a noise model, without requiring ground truth or object assumptions. The method fits probabilistic models to estimate entropy and information content, enabling objective evaluation across different imaging modalities. It is validated on applications like color photography, black hole imaging, and microscopy, showing strong correlation with task performance. The authors also propose IDEAL, an optimization pipeline that designs imaging systems by maximizing information capture through differentiable models. This work provides a general tool for evaluating and optimizing imaging systems based on information theory.\n\nSTRENGTHS AND WEAKNESSES:\n### **Strengths**\n\nThis paper introduces a practical, well-founded method to estimate mutual information in imaging systems without ground truth, and validates it across diverse applications. The proposed IDEAL framework adds a novel, differentiable way to optimize system design. Overall, the work is clear, technically sound, and broadly impactful for imaging system evaluation and optimization.\n\n\n### **Weaknesses**\n\n1. While the method relies on parametric entropy estimation, the paper does not quantify the estimator’s bias or variance across models, particularly in high-dimensional or low-sample regimes. This leaves some uncertainty about reliability in practice.\n\n2. The PixelCNN-based estimator is significantly more accurate but also more computationally expensive (\\~100s training time). However, detailed runtime or memory comparisons between models (especially within IDEAL) are not provided, making it hard to assess practical scalability.\n\n3. Sections on model implementation (e.g., entropy estimation using parametric fitting) could benefit from more concrete examples or pseudocode. Additionally, the notation between I(O;Y) and I(X;Y) may be confusing to readers unfamiliar with the assumptions.\n\nQUESTIONS:\n1. How sensitive is the proposed estimator to model mismatch or data scarcity? The paper uses parametric models (e.g., Gaussian, PixelCNN) to estimate entropy, but does not analyze the sensitivity of the mutual information estimates under model mismatch (e.g., non-Gaussian data fit with Gaussian models) or limited data regimes. Could the authors provide quantitative or empirical analysis (e.g., bias/variance, convergence with sample size) to assess estimator robustness? Demonstrating estimator reliability under varying conditions would strengthen confidence in practical applicability and improve the overall evaluation.\n\n2. Can the authors report the runtime and memory cost of different estimators, especially within IDEAL? PixelCNN is noted as the most accurate but computationally intensive option. However, the paper lacks detailed comparisons of training time, memory usage, and scalability across models. This is especially important for real-world deployment and adoption of IDEAL. A clear runtime analysis would validate the claim of practicality and support the scalability of the method in larger design spaces.\n\n\n3. How does the method perform when mutual information is not aligned with task relevance? The authors acknowledge that total mutual information may not fully predict performance in tasks like classification (e.g., microscopy application). Could the authors expand on potential extensions to task-aware information metrics or suggest how IDEAL could incorporate task-specific weighting? Outlining a concrete path toward task-adaptive extensions would improve the method’s flexibility and broaden its relevance.\n\n4. Can IDEAL be compared with task-specific or end-to-end design pipelines? While IDEAL is promising, its comparison is limited to internal metrics (e.g., mutual information or MSE). Can the authors compare IDEAL-based encoder designs with those from task-specific or end-to-end learned systems in terms of downstream task accuracy and efficiency? Empirical evidence of IDEAL's competitive or superior performance versus baselines would significantly enhance the strength of the design contribution.\n\n5. Could you clarify assumptions behind the equivalence I(O;Y) = I(X;Y)? The estimator substitutes I(O;Y) with I(X;Y) under the assumption that X deterministically depends on O. Could the authors provide practical examples or guidance on when this assumption fails and how the method might adapt (e.g., when the encoder is stochastic)? Better explanation of this core assumption would improve theoretical clarity and help readers understand the estimator’s limits and adaptation potential.\n\nLIMITATIONS:\nThe authors address key limitations, including the gap between total and task-specific information, and the computational cost of expressive models. To strengthen this section, it would help to include runtime comparisons, clarify estimator reliability under limited data, and discuss cases where the I(O;Y) = I(X;Y) assumption may fail.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces an information-driven design framework for imaging systems, focusing on mutual information estimation between unknown objects and noisy measurements. It proposes a data-driven method to quantify the information content of measurements, and combines this information estimation approach with gradient-based optimization to design imaging systems that maximize information capture.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths.\n+ The interesting method effectively bridges a critical gap in imaging system design, by quantifying information content directly from measurements without needing post-processing decoders. \n+ The paper demonstrates the applicability of the approach in a wide range of imaging systems, including real-world use cases like black hole imaging and biological microscopy. \n+ The theoretical foundation of the paper is robust, offering a clear mathematical formulation for mutual information and its estimation. \n+ The IDEAL method for optimizing encoder designs based on information content shows great promise, especially in terms of reducing computational overhead by focusing on the encoder alone before considering the decoder.\n\nWeakness. \nNevertheless, I believe the authors still need to address the following critical concerns. \n- The authors claim that the information content can be quantified directly from a dataset of measurements without requiring ground truth data. However, in the process of optimizing imaging systems, the imaging parameters are constantly changing, which implies that the measurements are also continuously evolving. Without adjustable imaging hardware to collect new training data in real-time (which is relatively difficult to achieve), this inevitably requires the training process to rely on ground truth data to simulate new measurements. \n- Furthermore, although the method does not require post-processing decoders, it raises the question of whether this generalization compromises the accuracy for specific tasks. It needs to be experimentally demonstrated whether maximizing information content ultimately leads to the optimal task performance. For instance, in an imaging modality experiment, a comparison between this method and previous end-to-end optimization approaches involving decoders should be provided to show the effectiveness.\n\nQUESTIONS:\nQuestions for the authors to address in the rebuttal: \n1.\tAs mentioned in weakness, how does the proposed method address the challenge of continuously evolving measurements during imaging system optimization, especially without real-time adjustable hardware for data collection, which may still require ground truth data for training?\n2.\tDoes the generalization of maximizing information content sacrifice task-specific accuracy, and can the method be experimentally validated against previous end-to-end optimization approaches involving decoders?\n3.\tHow does the proposed method handle cases where the noise model may not be fully characterized or differs from the assumed model , such as those complex noise models in thermal infrared imaging.\nAddressing these questions in the rebuttal would greatly enhance the clarity and applicability of the proposed method, thereby improving the paper's evaluation.\n\nLIMITATIONS:\nThe method assumes static measurements, but in real-world imaging system optimization, measurements continuously evolve with changing imaging parameters. Without real-time adjustable hardware to collect new data, the training process may still rely on ground truth data to simulate new measurements. The authors should clarify how this challenge is addressed.\nWhile the approach avoids post-processing decoders, it remains unclear whether this generalization sacrifices task-specific accuracy. The authors should provide experimental validation comparing this method with previous end-to-end optimization approaches involving decoders.\nThe noise model assumed in the paper may not fully account for complex noise scenarios, such as those encountered in thermal infrared imaging. The authors should explain how their method handles such cases, where noise characteristics may differ significantly from the assumed models.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a data-driven framework for estimating mutual information in imaging systems, enabling information-theoretic design without ground truth data. The approach fits probabilistic models to noisy measurements and quantifies information content by subtracting noise entropy. It validates across diverse applications—color photography, radio astronomy, lensless imaging, and microscopy—showing that mutual information reliably predicts system performance. The Information-Driven Encoder Analysis Learning (IDEAL) framework optimizes imaging systems via gradient-based methods to maximize information capture, achieving up to 3.8× faster training with 97% performance retention compared to full-data training.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths：\n1. No ground truth required: Estimates information directly from measurements, avoiding reliance on labeled data .\n2. Broad applicability: Works across imaging modalities (photography, astronomy, microscopy) without task-specific assumptions .\n3. Quantitative design metric: Unifies resolution, SNR, and sampling trade-offs into a single information-theoretic metric .\n4. Gradient-based optimization: IDEAL enables automated system design (e.g., color filter arrays) with up to 3.8× training speedup .\n\nWeaknesses：\n1. Computational complexity: PixelCNN models require more training data and time compared to Gaussian models .\n2. Noise model dependency: Relies on accurate noise characterization (e.g., Gaussian/Poisson), which may fail in complex noise scenarios .\n3. Task generality trade-off: Total information may not align with task-specific needs (e.g., classification vs. reconstruction) .\n\nQUESTIONS:\n1. The framework focuses on static images, but real-world imaging (e.g., video) involves temporal correlations unaddressed in the current model. How does the method handle dynamic scenes with time-varying noise?\n\n2. The study lacks validation on multi-spectral data, where cross-channel dependencies may alter mutual information.\nCan IDEAL optimize systems for real-time processing constraints? \n\n3. The framework prioritizes information maximization but doesn’t address latency-accuracy trade-offs in real-time applications.\nWhat is the impact of multi-modal imaging (e.g., RGB-NIR) on information estimation?\n\n4. Most experiments use supervised decoders (e.g., neural networks), but unsupervised scenarios (e.g., self-denoising) remain untested. How does the method perform with unsupervised or self-supervised decoding?\n\nLIMITATIONS:\nsee weakness.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}