Experimental Design for Multi-Channel Imaging via Task-Driven Feature Selection
Authors: Stefano B. Blumberg
Paddy J. Slator
Daniel C. Alexander
Venue: ICLR 2024 poster (2024)
Keywords: Experimental Design
Supervised Feature Selection
Multi-Channel Imaging
Hyperspectral Imaging
Magnetic Resonance Imaging (MRI)
Task-based Image Channel Selection
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
TL;DR: This paper presents a data-driven, task-specific paradigm for experimental design, to shorten acquisition time, reduce costs, and accelerate the deployment of imaging devices.

Abstract: This paper presents a data-driven, task-specific paradigm for experimental design, to shorten acquisition time, reduce costs, and accelerate the deployment of imaging devices.  Current approaches in experimental design focus on model-parameter estimation and require specification of a particular model, whereas in imaging, other tasks may drive the design.  Furthermore, such approaches often lead to intractable optimization problems in real-world imaging applications. Here we present a new paradigm for experimental design that simultaneously optimizes the design (set of image channels) and trains a machine-learning model to execute a user-specified image-analysis task. The approach obtains data densely-sampled over the measurement space (many image channels) for a small number of acquisitions, then identifies a subset of channels of prespecified size that best supports the task.  We propose a method: TADRED for TAsk-DRiven Experimental Design in imaging, to identify the most informative channel-subset  whilst simultaneously training a network to execute the task given the subset. Experiments demonstrate the potential of TADRED in diverse imaging applications: several clinically-relevant tasks in magnetic resonance imaging; and remote sensing and physiological applications of hyperspectral imaging. Results show substantial improvement over classical experimental design, two recent application-specific methods within the new paradigm, and state-of-the-art approaches in supervised feature selection.  We anticipate further applications of our approach.  Code is available: https://github.com/sbb-gh/experimental-design-multichannel

Published as a conference paper at ICLR 2024
EXPERIMENTAL DESIGN FOR MULTI-CHANNEL IMAGING VIA TASK-DRIVEN FEATURE SELECTION
Stefano B. Blumberg' Paddy J. Slator Daniel C. Alexander?
Cennre for Artificial Intelligence, Department of Computer Science, University College London "Centre for Medical Image Computing, Department of Computer Science, University College London
Cardiiff University Brain Research Imaging Centre and School of Computer Science, Cardiff University stefand wlw.berg.g...................... 17uucl...... ac. uk
ABSTRACT
This paper presents a data-driven, task-specifi, paradigm for experimental design, to shorten acquisition time, reduce costs, and accelerate the deployment of imaging devices. Current approaches in experimental design focus on model-paramees estimation and require specification of particular model, whereas in imaging, other tasks may drive the design. Furthermore, such approaches often lead to intractable optimization problems in real-world imaging applications. Here we present a new paradigm for experimental design that simultaneously optimizes the design (set of image channels) and trains a machine-learning model to execute a user- specified image-analysia task. The approach obtains data densely- the measurement space (many image channels) for a small number of acquisitions, then identifies a subset of channels of prespecified size that best supports the task. We propose a method: TADRED for TAsk-DRiver Experimental Design imaging, to identify the most informative channel-subsel whilst simultaneously training a network to execute the task given the subset. Experiments demonstrate the potential of TADRED in diverse imaging applications: several clinically- relevant tasks in magnetic resonance imaging; and remote sensing and physiological applications of hyperspectral imaging. Results show substantial improvement over classical experimental design, two recent application-specifee methods within the new paradigm, and state-of-the-art approaches in supervised feature selection. We anticipate further applications of our approach. Code is available: Code Link
over
INTRODUCTION
Experimental design seeks sampling scheme or design D {d' ,..., (d.....dd). where each d', i 1,... C, is a combination of experimental variables that are under the control of the experimenter, that provides data optimally informative for some criteria task Antony 2003); Pukelsheim (2006). The experimental outcome (measured data) of design is matrix X D E R with C' corresponding measurements from each of samples. The optimal choice of design depends on the experimental task, which we express as a function T that maps Xp to a corresponding matrix Y of labels. Experimental design optimization seeks the design that maximizes the ability to perform the task. subject to constraints of time or cost, i.c.
D arg min L(T(XD), subject |D] C
D
where L is loss function. Here we limit cost simply to the size of D; can be any task, but often in imaging involves aetimating/mappiinnn model parameters e.g. via gradient-descent model-fitiin in every pixel/voxel. as in inlexexnde (2008); Cerciggaai & Alexander (2006), or machine learning as in Gyori et al. (2022): Waterhouse & Stoyanov (2022).
In imaging, as illustrated in figure b. Xp is typically collection of n pixels or voxels with C channels (e.g. RGB images have C = 3). The choice of d' E D controls the contrast in channel i and is global to the whole channel. Compact (small C') but informative designs often critical in reducing acquisition or development costs in real-world applications. Examples include acquiring magnetic resonance imaging (MRT) contrasts, e.g. to estimate and map microstructural tissue

---

Published as a conference paper at ICLR 2024
Legod
a) Expirinental Design for Clali-Cl Iunging Oijertion Chiose d' mesiaare trugg Xo's Interraiion
bj Paradiguu Exprrimntaal Dasiga Aeynis Xy stth 2 chuzaaa too shown Indedroona feature alotion C'dauzdds
Reaugia Tooks
Typiol Pspermeets Drdd
4%) Design
d
d's Sempey-Scuplod Design
Xn R
2 Combinetion Independent Variables
Xa g
Xx Poots of 8.0 Chanmile Misoonmmenn.
Targe of Sumpos (Poook)
Spatee Wics
Tragg
Xp Detu Roouseenciio
8
7
Model wark
Selcction
Clennel
T
Modd
Pit
D D
Figure 1: a) An example of experimental design for imaging, In remote sensing hyperspectral imaging
(see table 3), each observed wavelength d' is chosen by the experimenter. The outcome of each d' is a grayscale image- a channel of the resultant data X (RGB has channels). b) The new paradigm for experimental design illustrated for qMRI. First, obtain image data X D with a large number of C channels. Next, train a user-chosen task network, which drives design optimization to select C' < Ò channels we propose TADRED for this. We consider three distinct example tasks in experiments.
properties within the time a patient can stay still in scanner scanner/Alexane (Alexander (2008), or manufacturing affordable hyperspectral imaging devices including few well-chosen spectral filters, e.g. for estimating tissue oxygenation Waterhouse & Stoyanov (2022).
Standard approaches for experimental design typically optimize D over a continuous space, for the task of model parameter estimation. For example, a classical approach still widely deployed uses the Fisher matrix Montgomery (2001), whilst more recent approaches use the paradigm of sequential Bayesian experimental design Blau et al. (2022) : Foster et al. (202T): Ivanova et al. (202T). Both require priori model choice, limiting consideration to model-based tasks, and choices or assumptions on their prior distribution. Moreover, such approaches rapidly become computationally intractable the dimension of the optimization increases.
specific
Here we suggest a new task-driven paradigm for experimental design for real-world imaging ap- plications, illustrated in figure [b, that does require priori model specification and replaces high-dimensional continuous search with a subsampling problem. First, the paradigm requires train- ing data Xn with C' channels/measuremmntt acquired using design D that densely samples the measurement space. Secondly the paradigm selects subset of size C' < C image channels from XD (optimizing the design and choosing XD C p)), coupled with the training of a high- gegh-perfommng neural network that executes the task T driving the experimental design. Thus, the new paradigm replaces the optimization in equation with:
D", = arg min L(T(Xp), Y) subject to D D.
D,T
In this paradigm, the task must be specified a priori, but may go beyond standard model-based tasks that drive classicalBayyssian experimental design, to include 'model free" tasks such as missing data reconstruction. The training data requires only a small number of subjects/samples, so may use specialized hardware, lengthy acquisitions, or even simulations. In practice, such acquisitions are often made during early development phases of imaging technologies to explore the range of sensitivity, which informs the choice of, and often provides, D. The paradigm we propose formalizes the exploitation of such data in experimental design for downstream systems designed for wide deployment and directly supports the use of deep learning for T.
In the new paradigm, the experimental design problem becomes similar to supervised feature selection, where the C' image channels of X x are considered features. In supervised feature selection, state- of-the-art approaches Wojtas & Chen (2000); Lee et al. (2022) couple feature selection with task optimization, however the structure of the data in typical supervised feature selection problems differs from those in experimental design for imaging. Feature selection algorithms typically assume most features are uninformative and the task is to 'identify a small, highly discriminative subset' Kuncheva al. (2020) c.g. genes associated with drug response from the entire genome. In experimental design for imaging, however, most channels individually offer similar amounts of information to support task performance, since they view the same scene/sample but with often-subtle differences in contrast (see e.g. figure (). Design optimization seeks a compact combination that covers all important aspects.
Therefore propose TADRED, a novel method for TAsk-DRiven Experimental Design in imaging. TADRED couples feature scoring and task execution in consecutive networks. The scoring and subsampling procedure enables efficient identification of subsets of complementarily informative
2

---

Published as a conference paper at ICLR 2024
channels jointly with training high-pprforming network for the task. TADRED also gradually reduces the full set of samples stepwise to obtain the subsamples, which improves optimization.
Key contributions are:
1. A new coupled subsampling- tlsk paradigm (feature selection) for experimental design in imaging.
2. TADRED: a novel approach for supervised feature selection tuned specifically for experimental
design in imaging. TADRED performs task-based image channel selection.
3. A demonstration of our approach on six datasets/tasks in both clinically-relevant MRI and remote sensing and physiological applications in hyperspectral imaging. TADRED outperforms (i) Classical experimental design, (ii) Recent application-specife published results, (iii) State-of-the- art approaches in supervised feature selection.
2 RELATED WORK
Approaches in Experimental Design A typical task in experimental design is optimize the design D for estimating model parameters. The most widely used classical approach in imaging uses the Fisher information matrix PPukelsheim (2006). However, for non-linear models, the optimization requires pre-specification of parameter values of interest, leading to circularity, e.g. the standard design for VERDICT model with primary application prostate cancer detection and classification Panagiotaki et al. (2015a) (used as baseline in table is computed by optimizing the Fisher-matrix for one specific combination of parameter values, despite aiming to highlight contrast in those parameters throughout the entire prostate. Approaches in the sequential Bayesian experimental design paradigm Blau et al. (2022); Foster et (202T): [vanova et al. (202T) reduce this circularity by optimizing over combinations ranges of parameter values. Recently Blau et al. (2022) also implemented an experimental design optimization in a discrete space and obtained tfate-of-thh-ar performance and deployment time, by using reinforcement learning to map history of designs and outcomes to the next design. However, the tasks driving experimental design in imaging are often 'model free' supervised tasks such as missing data reconstruction (tables 2 3 to recover missing image channels. Classical Fisher-matrix experimental design or sequential Bayesian techniques do not apply in such problems. Furthermore, the sequential Bayesian techniques have been deployed on only small-scale experiments with simulated data e.g. simple localization problem for two sources. For example, experiments in Blau al. (2022) have C S and D E Rdii dim 6. In contrast, e.g. the real-world experiment in table has C' {110, 55, 28, 14) and D E R Preliminary experiments suggest the application of these approaches to the high dimensional problems is not computationally tractable with the published code/methods. These issues motivate the reformulation of the experimental design paradigm and the introduction of TADRED. Appendix EE] is a broader review of experimental design for quantitative MRI (qMRI) and hyperspectral imaging.
Supervised Feature Selection operates either at the instance level e.g. identifying different salient parts of different images; or at the population level by selecting across all the instances. In imaging, each combination of acquisition parameters d' E D is global across all image pixels/voxels, so channel-selection for experimental design must be population wide. Recursive feature climination (RFF) / backward selection Guyon et al. 2002); /SiikitLLear (2023; ; Kohavi & John (1977)
frameworks that seek the most informative set of features among a superset to inform a model or task. They work by eliminating the least informative features stepwise to reach a prespecified
feature-set size. 'Feature Importance Ranking for Deep Learning' (FIRDL) Wojtas & Chen (2020). "Self-Supervision Enhanced Feature Selection with Correlated Gates' (SSEFS) are considered state-of- tftte--f-th-att the-art art in feature selection, outperforming both classical (e.g. RFE) and recent approaches outlined in appendix E| Both techniques are specifically designed to 'identify small, highly discriminative' subset Kuncheva et al. (2020) of features from larger group of mostly uninformative features. SSEFS, a first step, uses a probabilistic approach to search for this subset, whilst also exploiting the presence of correlated subsets for enhanced performance. A second step then trains a network on the chosen subset to exccute the task. FIRDL instead has a complex optimization procedure involving are detailed in appendix A and are baselines in later experiments.
al. (2022)
stochastic local search. SSEFS and FIRDL
In contrast to typical feature selection problems, most candidate choices in experimental design set exists. SSEFS's first step seeks groups of correlated features, which is less useful in experiment design, as
are informative: few, if any, features are uninformative so no single small discriminative
3

---

Published as a conference paper at ICLR 2024
most image channels correlate strongly (examples in figure 5). FIRDL incorporates global information by performing multiple evaluations on different feature combinations. However, FIRDL's search for a discriminative subset is inappropriate in the experimental design application: its multiple evaluations of the task-execution network are redundant and result in covariate shift and overfitring.
Nevertheless, TADRED builds upon the basic principles of task-driven feature selection, which is the foundation of FIRDL and SSEFS's success. TADRED adopts the same dual-network architecture, but with different optimization procedure tailored the experimental design problem. Specifically, TADRED's implements a novel combination of the dual selection/task network optimization within the paradigms of RFE/backwards selection. As such, it adopts a comparatively simple scoring procedure, which avoids the complicated and suboptimal joint optimization FIRDL/SSEFS require to search for distinctively discriminative subset. TADRED's end-to-end dual networks avoids FIRDL's multiple evaluations on different feature combinations, and TADRED's passing of information through the optimization procedure improves on both SSEFS and FIRDL.
Finally, PROSUB Blumberg et al. (2022) (baseline in table 2 is previous attempt to equate experimental design with feature selection and also uses RFE. It uses a customized neural architecture search at every step and was designed specifically to address measurement-selection problem in qMRI (data in table (2 where it achieves state-of-the-
performance. However, the technique does not naturally generalize to other tasks, which is a key motivation for TADRED. TADRED avoids PROSUB's cumbersome neural architecture search and implements instead a novel four-phase procedure in each step, which keeps the gradient updates smooth and allows feature selection each step. Also, beyond standard RFE, TADRED efficiently passes information from the optimization on larger feature sets to smaller sets by passing information on the network weights across the steps, unlike PROSUB. These advances combine to enhance substantially the performance, portability and generalizability of the algorithm across diverse experimental design problems.
3 TADRED:
EXPERIMENTAL DESIGN FOR IMAGING
TADRED presents a novel approach to supervised feature selection, tailored to the particularities of the experimental design problem in imaging, and aims to solve equation 2 Section 3.1 describes an outer loop of the procedure, which is inspired by classical paradigms Kohavi & John 1997 Guyon et al. 2002). that gradually eliminates elements from the densely-sampler design 1) in
steps to obtain designs D D1 D ... Dr. This corresponds performing supervised feature selection for decreasing sizes C = >> Cr, where {Co)²= are chosen by the user a priori. Then section 3.2 outlines an inner loop for training with fixed 1 S t S T. Inspired by recent supervised feature selection advances Imrie et al. (2022); Wojtas & Chen (202)). TADRED trains two coupled networks at each step: a scoring network S., which scores individual elements of X for importance to inform the subsampling, and task network Tr, which performs the task driving the design, i.c. estimates Y from chosen feature subset XD, C Xp- The training procedure is split into four phases that allows feature selection at each step and is inspired by Karras et al. (2018); Blumberg ctt al. (002)) which produced enhanced optimization. The full procedure is outlined in algorithm
3.1 OUTER LOOP
Cr Across steps 1. 1,.... T consider decreasing feature set sizes C = Ci C2 [.2)p. Reducing and perform supervised feature selection at each step in an inner loop (see section feature set sizes stepwise aids the optimization procedure compared to c.g. training on all features then subsampling all at once (see table 5). The procedure passes information from the optimization on larger feature sets to smaller sets. Finally, the stepwise procedure efficiently produces set of optimized designs (as is typical in supervised feature selection see e.g. Wojtas & Chen (2020) and (2022)) which can be useful for post-hoe selection of design size to balance economy (small C) with task performance. Whilst iterative subsampling also increases computational time, this is comparable to other supervised feature selection approaches (appendix D).
also in Waterhouse Stoyanov]
3.2 INNER LOOP: FOUR-PIAASE DEEP LEARNING TRAINING
At step 1 < t < T of the outer loop the inner loop constructs (i) binary mask m; (0. 1)0, ||m C. to subsample the features; (ii) a weight vector for the features 5, RQ;
4

---

Published as a conference paper at ICLR 2024
(iii) trained network Te to perform the task, which corresponds to solving the optimization problem:
minimize
(XT, 3 St), subject to ||ma|/e = Ce. may Te: where XD. = m. @ XD + (Le - my) @ b
the @ operation element-wise dot product which follows broadcasting rules when inputs have mismatched dimensions, I1-1|o is the L norm, 16 a vector with C ones, and 'feature fill' X R is a hyperparameter that fills the removed features (we take the data median, see appendix C.2) The weight vector S, contains feature scores, which the training procedure uses to remove low-scoring features by setting corresponding values of the mask m, to 0.
Scoring, Subsampling, and Task Execution The core of the training procedure uses the for- ward/backward pass in algorithm T The full procedure in algorithm 2 uses the forward/backward pass to update feature scoring gradually in tandem with improving label prediction.
The procedure aims to learn a meaningful sample-independent feature score to rank the features. In practice, deep-learning training is performed in batches and not across the whole data. Therefore we first learn a sample-dependent feature score (S, (Xp)) D)) = 3 E R where St is a neural network and o + [0, oo) is an activation function to ensure positive scores (we take initialization (0) 1). We then compute a sample-ing cenple-indeeendent score S, E R as an average of across the n samples in Xp. We also compute combined score that aids task execution
sigmoid and
s = a=00 + (1 aa) 2 St, a E [0,
(4)
which balances the current learned sample-depentee independent score and allows smooth integration between the two. The mix parameter a is set in the optimization procedure shift the balance from sample-dependent to sample-independent scores.
score with fixed global estimate of the sample-
We use a mask m, E [0, 1l to subsample the features Xp. m, 3 Xp +1c m.) C X D
and replace the removed features with default values X to retain the shape of the data structures throughout training. Rather than learning the mask m, end-to-end e.g. using sparsity term/prior as in Lee et al. (2022), we modify elements of m, during our training procedure. This is important to enable the outer loop of the procedure output candidate designs at each step.
D
We now estimate the target Y with P = TT((s() XD.) from the subsampled data weighted feature-wise by the score, then calculate the loss LY,Y). This weighting allows gradients to flow end-to-end.
Training Procedure The key challenges in the design of the training procedure in the inner loop are how to (i) obtain meaningful global sample-independent scores s, from learnt sample-dependent scores St. (ii) differentiate through masking operation to compute m, TADRED's four-phase procedure, inspired by Karras et al. (2018); Blumberg et al. (2022) gradually modifies the neural network structure during deep learning training, moving from learning simpler task (learning sample-dependent scores and retaining most features) to a more complex task (learning sample- independent scores and removing more features) by linear interpolation of network components. This improves optimization over directly learning the more difficult task. Thus we address (i) by first learning and then progressively reducing the final score to the average of 8 its average i.e. s = 5, (in algorithm 2 phase 2), and (ii) by progressively setting elements of m, to zero i.e., during training, mask elements are real valued but gradually reduce binary values (in phase 3).
The training procedure is different for the first outer loop step t = 1 compared to steps 2 2. This is because for step & we train all Ć features and do not have information from previous steps and for steps t 2 2 we perform supervised feature selection for user-chosen C. (solve equation 3) and training is initialized from step t 1. We describe each step with reference algorithm 2
Training for Step t t= 1 In the first step (lines D we simply train S1, Ti on full information i.e. on all features for total (chosen) F. epochs. At completion, we set the first sample-independent score 8) (line[4) to be the mean of the sample-dependent scores 8 across samples/batches. We found training solely on a sanple-dependent score results in faster optimization.
Training for Steps t = 2,...T The four phases require choosing the number of epochs for each phase:
1 <= E < E2 < Eg < for total number epochs E, training proceeds as follows:
5

---

Published as a conference paper at ICLR 2024
Phase 1) Initialize St and Ti from St-1 and Te-1,
Algorithm TADRED Forward & Back- ward Pass (FBP) in Step t
to St-1, my to m,-1.e and a = 2 to balance learning a new score for this step and using information from the learnt score from step - 1. Run En epochs to refine scores and task execution with and m, fixed.
Requires:
Input and Target Data Xp. Y. Mask m, Scoring and Task Networks St, Te, Loss L Feature Score 8, Mix Parameter a E [0, 1]. Feature Fill X
Update the sample-independen score s, with the learnt score from phase (line [T].. Run Eg E, epochs progressively linearly modifying a (line 13 so training moves gradually from using sample-dependent scores to
Phase 2)
I: (X)(
2: s = a a (1 a) O S; # Equation
3: XD. - m 3 XD + (1c m.) о X m
Phase 3) Choose the C5-1 C. lowest-ccored features
4: y = -T((s o XD.)
to remove (lines 16 17 p. Run Es E2 epochs lin-
5: Compute L(Ý, Y) and backpropagate
early modifying the mask for subsampling (line [9). This alters the C--1 - -C elements of m, correspond- ing to the lowest-scored features gradually to O. Thus ||m C--1 goes to ||mello = C.. Separating this phase from phase 2 increases the stability of the optimization, as modifying the mask and score simul- taneously results in large gradients.
Algorithm 2 TTADRED Optimization
Requires:
Input and Target Data XD E RnxC, ,Y Loss L. Feature Fill X E RC
Feature Set Sizes C = C1 ... CT Training Steps S E1 < E2 < Eg < E Initial Scoring and Task Networks S1, Ti
Phase 4) Train Te for final refinement for E - Es epochs with the score weights fixed and features chosen. At completion return Tr, m., Sz,
1: t + 1; m 1o: a + 1 Step t=1
2: for e + 1,.... E' do
Implementation Details and Hyperparameters TADRED's 's hyperparameters fixed across experi-
3: FBPO
Algorithm
4: 81 - mean of across data
ments and different application areas. They are detailed in appendix A
5: for / + 2,... do # Steps / 2 2
6:
# Phase
7:
S; + 5 a a+
m.
m
8:
for E + I,...
.... Ej do
4 EXPERIMENTS AND RESULTS
9:
FBPQ
10:
5 + mean of 8 on data
# Phase
11:
5, + } (5: + 5)
This section demonstrates the benefits of TADRED in multiple scenarios, with example applications in qMRI and hyperspectral imaging. First. in table I we con- sider the standard experimental design task of model
12:
for E + E1 + +1,
Eg do
13:
t- max
2 B.) 0}
14:
FBPO
parameter estimation and outperform classical Fisher-
15:
# Indices that sort an array; Phase
matrix approaches. Within the new paradigm, we also
16:
R = argsort (5. mm [i]
1)
show improvements over recent supervised feature se-
17:
D. < {R(0)
R(C,
C.I.
lection approaches. We then show TADRED's efficacy
18:
for E t E3 +1 Eg do
in 'model-free' experimental design scenario: recon- 19:
m: <- max (m. Es E2 0z)
struction of a densely sampled data set from a sparse 20:
FBPO
subset, where Fisher-matrix recent Bayesian exper-
21:
for e 4 Eg + 1,... do # Phase
imental design cannot operate and TADRED outper- 22:
FBPO
forms best published results in an MRI challenge in table In figure 2 we consider reconstruction task
23:
Cache Te, m, St for equation
to then estimate multiple clinically-relevant downstream metrics from model fitting extending the model-parameter
traditional model- parameter estimation task to estimate multiple quantities. TADRED outperforms recent supervised feature selection techniques in this task that has immediate deployment potential. We then show the generalizability of TADRED by performing similar sets of experiments on hyper- spectral images, outperforming both supervised feature selection baselines for earth remote sensing in table 3 and recent work in tissue oxygenation estimation in table 4 Tables 5 show ablation study and that TADRED is mostly robust to randomness in deep learning training.
Appendix provides additional analysis. Appendix D provides details on experimental design in qMII and hyperspectral imaging and how to implement our paradigm in real-world scenarios. Frrrndixx(summmizzze summarizes and visualizes the resultant densely-sampled data X,. Following standard practice in MR parameter estimation Alexander et al. (2019); |eeccignan et al. (218)) and hyperspec- tral image filter design Waterhouse & Stoyanov (2022). data samples are individual pixels/voxels.
Appendix [

---

Published as a conference paper at ICLR 2024
Table 2: Performance comparison on MUDI:
Table 1: Performance comparison of feature se- lection approaches for VERDICT-MRI designs: MSE x 10ah between estimated model parameters and ground truth for various C' and = 220.
MSE between C 1344 reconstructed MRI channels/measuremmntss and C ground-truth mea- surements for various C. PROSUB results from Blumberg et al. (2022) table
C= 110
55
28
14
C = 500
250
100
50
Random
1.54
2.24
3.25
6.10
PROSUB
0.49
0.61
0.89
1.35
SSEFS
1.06
1.28
1.89
4.58
TADRED
0.22
0.43
0.88
1.34
FIRDI.
2.22
2.14
3.09
4.05
C = 40
30
20
10
TADRED
1.03
1.18
1.80
2.64
PROSUB
1.53
1.87
2.50
3,48
TADRED
1.52
1.76
2.12
2.88
Figure 2: Downstream MRI metrics (see appendix F3 estimated from the full set of chan- nels/measurements on HCP data C = 288, and C from C 18 reconstructed measurements. Left: MSE for various metrics; Right: Qualitative comparison where arrows highlight closer agreement from TADRED's design with the gold standard than those from the best performing baseline.
DTI
Baseline
Gold Standard
Proporrd
FA
MD
AD
RD
Random
2.22
6.09
22.7
6.97
SSEFS
2,86
12.9
31.2
14,9
FA
FIRDL
9.83
23.2
77.7
26.8
TADRED
1.29
2.55
13.4
2.60
DKI
MSDKI
MK
AK
RK
MSI)
MSK
Random
9.03
7.83
13.3
6,82
7.59
SSEFS
12.0
9.26
20.3
8.96
8.17
RK
FIRDL
11.9
10.9
21.3
10.8
6.03
TADRED
7.67
6.73
13.9
6.37
4.94
Baselines and Comparisons We compare TADRED with standard model-based approaches such as the classical Fisher-matrix. Within the subsampling- -task paradigm we use i) recent application- specific published results optimized by the respective authors; ii) ofate-of-the-art supervised feature selection approaches FIRDL, SSEFS (see section 2 and random selection then deep learning training (denoted by 'random') mimic random baselines used in experimental design papers. Each feature selection approach conducts an extensive hyperparameter search, for faimess, the same number of evaluations are used for each feature subset C. All details are in appendix A As this requires multiple training run (SSEFS in table [
requires >400 runs), we examine the effect of the random seed on performance in table We compare the computational costs of different approaches in appendix D
TADRED Outperforms Classical Experimental Design and Baselines in Model Parameter Estimation A standard task experimental design is selecting the design D to maximize the precision of model parameters. We evaluate strategies for this using the VERDICT-ME model which aids early detection and classification of prostate cancer Panagiotaki et al. 2015a). We sample parameters 0; for voxel i 1..., n. from biologically plausible range, add synthetic noise representative of clinical gMRI, and the task is to estimate Y (01. ..., 0n) with performance metric classical Fisher-matrix experimental design (see section 2) to compute the design 1) with C 20. The design produces a root-mean square error of 15.0 10 2 in this experiment. The TADRED design with C 20 has corresponding error of 2.04 x 10 2 The supervised feature selection approaches in the new paradigm use a densely-sampled design D, where C 220 from Panagiotaki et al. (2115a) and use deep learning to estimate 0;- Appendix F.I documents all designs, models, and data. Table shows TADRED outperforms the feature selection basclines where C C C C C Thus it can better estimate parameters shown to 7, T reduce unnecessary biopsies Singh et al. (2022) in shorter scan times, spurring wider deployment in clinical settings. Similar results the well-known NODDI model are in appendix B
MSE. The first baseline Panagiotaki et al. (2015b)
Best Performance on qMRI Challenge Data The aimed to identify an informative subset of data, from which reconstruct the original full dataset X (i.e. Y Xf)) which had C = 1344 measurements. This task provides a generic challenge that tests the ability of an experimental design supervised feature selection algorithm to identify
Challenge Pizzolato et al. 2020)
7

---

Published as a conference paper at ICLR 2024
Table Performance comparison RMSE x10²,
Table 3: Performance comparison of feature se- lection approaches for remote sensing AVIRIS hyperspectral data, MSE between C 220 re- constructed and C' ground-truth measurements.
estimating abundance of HbO2, Hb (top), SO₂ (bottom). Experimental settings and baseline from Waterhouse & Stoyanov 2022 figure 5.
C = 110
55
28
14
C=
Baseline
4.54
4.91
5.33
6.17
Random
1.81
2.60
3.99
8.27
SSEFS
2.03
4.49
5.77
10.8
TADRED
2,80
2.89
3.23
4.36
C=
5
4
3
FIRDL.
8.10
9.87
10.6
10.3
Baseline
4.45
4.43
5.10
6.36
TADRED
0.87
1.82
2.84
5,80
TADRED
2.76
2.94
3.46
5.64
subset with maximal information content. As discussed in section 2 neither classical Fisher matrix nor Bayesian experimental design approaches can perform this task. Data are brain scans of five human subjects, which were acquired from a state-of- state-of-the-ar technique that acquires multiple MRI modalities simultancously in high- digeenimeniional space where d' E R.. Thus experimental design is important, as sampling in a time budget realistic in climical settings is difficult Slator et al. (2021). The first experiment follows Blumberg et al. (022)) which has the best performance the data and table shows TADRED outperforms this approach. We also show TADRED outperforms the supervised feature selection baselines in appendix B All details are in appendix F.2
Surpassing the Baselines in Estimation of Multiple Downstream Metries DTI, DKI, and MSDKI
Basser et al. (1994); JJensen & Helpern (2TT0); [enrrques] (2018) are widely-used qMRI methods. They quantify tissue microstructure and show promise for extracting imaging biomarkers for many medical applications, such as mild brain trauma, epilepsy, stroke, and Alzheimer's disease Jensen & Helpern| (2010): Ranzenberger & Snyder (2022); Tae et al. (2018). Reducing acquisition requirements (picking small C') whilst obtaining more accurate quantification will enable their usage in wider range of clinical application areas. We use publicly available, rich, high-resolution HCP data with C' = 288 measurements from six human subjects, corresponding to = 30 minute scan times in the
clinic - too long for general deployment. The task is to subsample sizes C = C then reconstruct the data, where the models are then fitted using standard techniques. Further details the models, data, and model fitting techniques are in appendix F3 Quantitative and qualitative results are in figure and appendix B and show TADRED outperforms the baselines on 1 17/18 comparisons on clinically useful downstream metrics. Furthermore, the downstream metrics produced by TADRED are visually closer to the gold standard than those from the best baseline, potentially enhancing the diagnosis of aberrations in tissue microstructure.
8: T6
Outperforming Baselines in Reconstructing Remote Sensing Ground Images The JPL's Airborne Visible/ Infrared Imaging Spectrometer (AVIRIS) Thompson et al. (2017) remotely senses elements of the Earth's atmosphere and surface from aeroplanes, and has been used to examine the effect and rehabilitation of forests affected by large wildfires. Purdue University Agronomy Department obtained AVIRIS data to support soils research and we use this publicly available 'Indian Pine' data Baumgardner et al. (2015). obtained from two flights which acquired ground images from C' 220 different wavelengths. Details are in appendix [:4 This experiment follows experiment table 2 and examines sumpling-reconstrun reconstruction task where we investigate if we can obtain the same quality data with fewer wavelengths - which would in practice require fewer sensors. Table 3 shows TADRED
C C C outperforms the supervised feature selection baselines with subsample sizes C = 2 g To- These improvements demonstrate the potential for using fewer filters in AVIRIS. In the development of next- eext-generation airborne hyperspectral devices, TADRED may be used to choose the filters. Further results are in appendix B promising additional applications are outlined in appendix
Improving the Estimation of Oxygen Saturation This experiment follows Waterhouse & Stoyanov (002)) Tissue oxygen saturation levels provide information regarding chemical and heat burns, along with the likelihood of healing. However, techniques such as spectrophotometees and pulse oximetry do not provide the spatial resolution to observe differences in blood saturation in neighboring tissue. Hyperspectral imaging is non-invasive and real-time alternative to improve oxygenation estimation, yet application-speciffi spectral band selection is required to reduce the high cost of imaging sensors, allowing widespread clinical adoption. To address this, Waterhouse & Stoyanov (2022) adapted the model in Can & Ülgen (2019) for simulations and the objective is to estimate the pixel-wise abundance of oxyhemoglobin HbO, and deoxyhemoglobin Hb; and oxygen saturation SO2. Design
8

---

Published as a conference paper at ICLR 2024
Table 6: Standard deviation of performance x10²
Table 5: Ablation study TADRED's ' compo- nents.
across 10 random seeds settings.
C = 110
55
28
14
C
110
55
28
14
Random
0.11
0.23
0.37
0.76
w/o Scoring Network S iterative suhsampling
7.23
10.7
11.5
11.
SSEFS
0.01
0.02
0.05
0.18
1,03
1.83
2.80
TADRED
1,03
1.18
1,80
2.64
FIRDL
0.44
0.34
0.37
0.44
TADRED
0.01
0.02
0.01
0.12
elements d' E D are chosen from 4 filters of different widths applied to 87 wavelengths (center), producing C = 348 measurements. Table 4 shows that TADRED produces directly outperforms all approaches and results published and optimized in Waterhouse & Stoyanov (2022) for estimating the abundance of 2 Hb, SO2; for feature sizes C' = 6, 5, 4, This suggests that using TADRED during the development of clinically-viable hyperspectral devices may be beneficial to reduce costs.
Component Analysis and the Effect of Randomness We use the experimental settings in table [ Table 5 examines the impact of removing TADRED's components on performance. First it considers TADRED without iteratively removing features in the optimization procedure, fixing = 2 and C2 = Č, C, showing that iterative subsampling has better performance than subsampling all features one iteration. As the feature scoring is key element of TAARED, we also show that removing the scoring network S, whilst still learning score, results in extremely poor performance, as training is destabilized when progressively setting the score from sample-dependen to sample- independent (recall equation 4). Table 6 shows how changing the random seed affects network initialization and data shuffling impacts performance; TADRED performs favorably compared to alternative approaches, and TADRED is mostly robust to the randomness inherent in deep learning.
5 DISCUSSION
This paper proposes TADRED, a feature selection algorithm that enables a new subsampling paradigm for experimental design particularly in multi-channel imaging applications. We demonstrate sub- stantial performance benefits over standard Fisher-matrix approaches at the heart of widely used quantitative MRI techniques, as well strong potential in multiple hyperspectral-imagge cations. "Standard" data sets for testing TADRED do not exist, as its new paradigm is largely unexplored. but in the few available examples (dataset used in table and hyperspectral datasets in tables 3 4
appli-
TADRED strongly outperforms existing algorithms, even on datasets for which those baselines were specifically designed, and without -eeeiff hyperparameter tuning.
TADRED combines the dual selection/task network training strategy in state-of mate-of-theeart the-art feature selection algorithms (SSEFS and FIRDL) with an RFE framework better suited identifying complementary subsets among many informative candidate features. Thus, TADRED outperforms SSEFS and FIRDL. on the imaging experimental design problems we consider. In fact, random supervised feature selection often outperforms SSEFS when there are informative/correlan feature subsets to identify and FIRDL's complex optimization procedure is often not beneficial when there is no such subset to identify and it underperforms simpler approaches. On the other hand, TADRED is likely to underperform SSEFS and FIRDL on typical applications in supervised feature selection where small sets of discriminative features reside among many uninformative features. One possible limitation is that TADRED's iterative subsampling in the paradigm of RFE and backward selection decreases the upper bound on performance the optimal feature sets for sizes Ct C.-1 may not be nested. Future work will consider alternative strategies. This iterative subsampling also increases computational time compared to random supervised feature selection. However, appendix shows TADRED's ' computational time is comparable to SSEFS and FIRDL. Here, we consider all image channels to have equal cost, but in practice some measuremeens//hhen measurementt/channeh may be more expensive than others; TADRED's formulation adapts naturally to more complex cost functions on the experimental design. Also here we consider only tasks that treat each image pixel/voxel independently which is typical in quantitative imaging Alexander et al. (2019): Cercignani al. (2018), so use only fully-connected networks (as the baselines), again TADRED'S s formulatin adapts naturally to e.g. CNN for T. TADRED has further applications to other imaging problems e.g autofocus for specialized equipment Lightley et al. (2022) and potentially beyond imaging to e.g. studies of cell populations Sinkoe & Hahn (2017).

---

Published as a conference paper at ICLR 2024
REPRODUCIBILITY STATEMENT
We provide the code: Code Link, which contains the entire source code for our algorithm TADRED. The code also contains the script to create simulations used in tables 7 and downloading and preprocessing the data in for results presented in tables 2 and figure Further details on all data and preprocessing are in appendix We also provide detailed information on the implementation of TADRED and the baselines in appendix A
ACKNOWLEDGEMENTS
Tristan Clark, James O'Connor, Edward Martin; Ahmed Abdelkarim, Daniel Beechey, George Blumberg, Räzvan Cäramaluău, Amy Chapman, Alice Cheng, Luca Franceschi, G-Research (for previous grant), Fredrik Helltröm, Jessica Hoang, Chen Jin, Jean Kaddour, Marcus Keil, Johannes Kirschner, Marcela Konanova, Eve Levy and Michael Salvato, Hongxiang Lin, Nina Montaña-Brown, Luca Morreale, MUDI Organizers, Raymond Ojinnaka, Gabriel Oon, Brooks Paige, David Pérez- Suárez, Stefan Piatek, Reviewers, Oliver Slumbers, Dennis Soemers, Danail Stoyanov, Shinichi Tamura, Dale Waterhouse, Tom Young, An Zhao, Yukun Zhou. Funding: EPSRC grants M020533 R006032 R014019, Microsoft scholarship, NIHR UCLH Biomedical Research Centre, Research Initiation Project of Zhejiang Lab (No.2021 N0OPID)),, Data were provided [in part] by the Human Connectome Project, MGH-USC Consortium (Principal Investigators: Bruce R. Rosen, Arthur W. Toga and Van Wedeen; (H1M9933765) funded by the NIH Blueprint Initiative for Neuroscience Research grant; the National Institutes of Health grant HIIEB15586 and the Instrumentation Grants S10RR023043, 1S10RR023401, 1S10RR019307.
HPC:
REFERENCES
Abubakar Abid, Muhammed Fatih Balin, and James Zou. Concrete autoencoders: Differentiable feature selection and reconstruction. In: International Conference on Machine Learning (ICML), 2019.
Daniel C. Alexander. A general framework for experiment design in diffusion MRI and its application in measuring direct fissue-microstructure features. Magnetic resonance medicine, 60(2):439-448
2008.
2 19 22
Daniel C. Alexander, Tim B. Dyrby, Markus Nilsson, and Hui Zhang. Imaging brain microstructure with diffusion MRI: practicality and applications. NMR in Biomedicine, 32(4))e841, 2019. 6
Fidel Alfaro-Almagro, Mark Jenkinson, Neal K. Bangerter, Jesper L. R. Andersson, Ludovica Grif- fanti, Gwenaëlle Douaud, Stamatios N. Sotiropoulos, Saad Jbabdi, Moises Bermandez-Fernandee Emmanuel Vallee, Diego Vidaurre, Matthew Webster, Paul McCarthy, Christopher Rorden, Alessan- dro Daducci, Daniel C. Alexander, Hui Zhang, Iulius Dragonu, Paul M. Matthews, Karla L. Miller, and Stephen Smith. Image processing and quality control for the first 10,000 brain imaging datasets from UK biobank. Neurolmage, 166:400-424, 2018. 24
Yashas Annadani, Panagiotis Tigas, Desi R. Ivanova, Andrew Jesson, Yarin Gal, Adam Foster, and Stefan Bauer. Differentiable multi-target causal bayesian experimental design. In: International Conference on Machine Learning (ICML), 2023. 22
Jiju Antony. Design of Experiments for Engineers and Scientists. Oxford: Butterword--Heinemmnn. 2003. 0
Boaz Arad and Ohad Ben-Shahar. Filter selection for hyperspectral estimation. In: International
Conference on Computer Vision (ICCV), 2017. 23
David Arbour, Drew Dimmery, Tung Mai, and Anup Rao. Online balanced experimental design. In:
International Conference of Machine Learning (ICML), 2022. 22
Peter J. Basser and Carlo Pierpaoli. Microstructural and physiological features of tissues elucidated by quantiaaive-diffuuiin--- MRI. Journal of Magnetic Resonance, 11 111:209-219 1996. 25
10

---

Published as a conference paper at ICLR 2024
Peter J. Basser, James Mattiello, and Denis LeBihan. MR diffusion tensor spectroscopy and imaging. Biophysical journal, 66( 1):259 267 1994. 25
Marion F. Baumgardner, Larry L. Bichl, and David A. Landgrebe. 220 band AVIRIS hyperspectral
image data set: June 12, 1992 indian pine test site 3. Purdue University Research Repository doi: 10.4233/R7RR2099
2015.
23
Marion F. Baumgardner, Larry L. Biehl, and David A. Landgrebe. Aviris hyperspectral image data
set. https: /tops://purr.. purdue wdw/ cat patiions/1i 947/1 2022. 26
Tom Blau, Edwin V. Bonilla, ladine Chades, and Amir Dezfouli. Optimizing sequential experimental design with deep reinforcement learning. In: International Conference Machine Learning (ICML), 2022.
Stefano Blumberg. Hongxiang Lin, Francesco Grussu, Yukun Zhou, Matteo Figini, and Daniel C. Alexander. Progressive subsampling for oversampled data application to quantitative MRI. 2022.
18
25
I.eo Breiman. Random forests. Machine Learning, 45:5-32, 2001 22
Romain Camilleri, Kevin Jamieson, and Julian Katz-Samuels. High-dimensional experimental design and kernel bandit. 2021. 22
Osman Melih Can and Yekta Ülgen. Modeling diffuse reflectance spectra donated blood with their hematological parameters. Clinical and Preclinical Optical Diagnostics 11, 2019.
Timothy Castiglia, Yi Zhou, Shiqiang Wang, Swanand Kadhe, Nathalie Baracaldo, and Stacy Patter- LESS-VFL VFL: Communication--fficee feature selection for vertical federated learning. In:
International Conference on Machine Learning (ICML), 2023. 22
Mara Cercignani and Daniel C. Alexander. Optimal acquisition schemes for in vivo quantitative magnetization transfer MRL. Magnetic Resonance in Medicine, 56(4))803-810, 2006. [
Mara Cercignani, Nicholas G. Dowell, and Paul S. Tofts. Quantitative MRI of the Brain: Principles of Physical Measurement. CRC Press, second edition, 2018.
Jianbo Chen, Mitchell Stern, Martin J. Wainwright, and Michael I. Jordan. Kernel feature selection via conditional covariance minimization. In: Neural Information Processing Systems (NIPS), 2017.
22
Code Link, Code for this Paper by Stefano B. experimental -esign multichannel
sbboom//bbbb/.. gh/
10
24
25
26
David Cohen, Tal Shnitzer, Yuval Kluger, and Ronen Talmon. Few-sample feature selection via
feature manifold learning. In: International Conference on Machine Learning (ICML), 2023. 22
Bethany Connolly, Kim Moore, Tobias Schwedes, Alexander Adam, Gary Willis, Ilya Feige, and Christopher Frye. Task-specific experimental design for treatment effect estimation. In: Interna- tional Conference on Machine Learning (ICML), 2023. 22
Ian Covert, Wei Qiu, Mingyu Lu, Nayoon Kim, Nathan White, and Su-In Lee. Learning to maximize mutual information for dynamic feature selection. In: International Conference on Machine
Learning (ICML), 2023. 22
Nick Doudchenko, Khashayar Khosravi, Jean Pouget-Abadie, Sebastien Lahaie, Miles Lubin, Vahab Mirrokni, Jann Spiess, and Guido Imbens. Synthetic design: An optimization approach to experimental design with synthetic controls. In: Neural Information Processing Systems (NIPS), 2021. 22
David C. Van Essen, Stephen M. Smith, Deanna M. Barch, Timothy E. J. Behrens, Essa Yacoub, Kamil Ugurbil, and WU-Minn HCP Consortium. The WU-Minn Human Connectome Project: an overview. Neuroimage, 80:62-79, 2013. 25

---

Published as a conference paper at ICLR 2024
Zalan Fabian, Berk Tinaz, and Mahdi Soltanolkotabi. HUMUS-Ne Hybrid unrolled multi-scale network architecture for accelerated mri reconstruction. In: Neural Information Processing Systems (NIPS), 2022. 23
Uran Ferizi, Benoit Scherrer, Torben Schneider, Mohammad Alipoor, Odin Eufracio, Rutger J. Fick, Rachid Deriche. Markus Nilsson, Ana K. Loya-Olivas, Mariano Rivera, Dirk H. Poot, Alonso Hamirez-Manzanaree Jose L. Marroquin, Ariel Rokem, Christian Pötter, Robert F. Dougherty, Ken Sakaie, Claudia Kingsht- Simon K. Warfield Thomas Witzel, Lawrence L. Wald, José G Raya, and Daniel C. Alexander. Diffusion MRI microstructure models with in vivo human brain connectome data: results from a multi-group comparison. NMR in biomedicine, 30(9), 2017.
19
22
24
Rutger Fick, Demian Wassermann, and Rachid Deriche. The Dmipy toolbox:
Diffusion MRI multi- compartment modeling and microstructure recovery made easy. Frontiers in Neuroinformatics, 13
(64), 2019.
24
Xavier Fontaine, Pierre Perrault, Michal Valko, and Vianney Perchet. Online A-Optimal design and active linear regression. In: International Conference on Machine Learning (ICML), 2021. 22
Adam Foster, Desi R. Ivanova, Iyys Malik, and Tom Rainforth. Deep adaptive design: Amortizing sequential bayesian experimental design. In: Neural Information Processing Systems (NIPS), 2021.
18
Eleftherios Garyfallidis, Matthew Brett, Bagrat Amirbekian, Ariel Rokem, Stefan van der Walt, Maxime Descoteaux, Ian Nimmo-Smith, and Dipy Contributors. DIPY, a library for the analysis of diffusion MRI data. Frontiers Neuroinformatics, 8(8), 2014, 25
Peter W. Glynn, Ramesh Johari, and Mohammad Rasouli. Adaptive experimental design with temporal interference: A maximum likelihood approach. In: Neural Information Processing
Systems (NIPS), 2020.
2
Francesco Grussu, Torben Schneider, Carmen Tur, Richard L. Yates, Mohamed Tachrount, Andrada lanuş, Marios C. Yiannakas, Jia Newcombe, Hui Zhang, Daniel C. Alexander, Gabriele C. DeLuca, and Claudia A. M. Gandini Cheeler-Kingsheen Neurite dispersion: new marker of multiple sclerosis spinal cord pathology? Annals of Clinical and Translational Neurology, 4(9), 2017. 24
Francesco Grussu, Stefano B. Blumberg, Marco Battiston, Lebina S. Kakkar, Hongxiang Lin, Andrada Januş, Torben Schneider, Saurabh Singh, Roger Bourne, Shonit Punwani, David Atkinson, Claudia A. M. Gandini Wheeler- Kingssstt,, Eleftheria Panagiotaki, Thomy Mertzanidou, and Daniel C. Alexander. Feasibility of data-driven, model-free quantitative MRI protocol design: Application to brain and prostate iiffusion-relaxation imaging. Frontiers in Physics, 9:615, 2021.
Hákon Gudbjartsson and Samuel Patz. The Rician distribution of noisy MRI data. Magnetic Resonance Medicine, (4(6))910-911 1995. 24
Isabelle Guyon, Jason Weston, Stephen Barnhill, and Vladimir Vapnik. Gene selection for cancer classification using support vector machines. Journal of Machine Learning Research, 46(1): 389-422, 2002.
3 4 22
Noemi G. Gyori, Marco Palombo, Christopher A. Clark, Hui Zhang, and Daniel C. Alexander. Training data distribution significantly impacts the estimation of tissue microstructure with machine learning. Magnetic Resonance in Medicine, 87(2):932-947, 2022.
Derek Hansen, Brian Manzo, and Jeffrey Regier. Normalizing flows for knockoff-fre free controlled feature selection. In: Neural Information Processing Systems (NIPS), 2022. 22
Xiaofei He, Deng Cai, and Partha Niyogi. Laplacian score for feature selection. In: Neural
Information Processing Systems (NIPS), 2005.
22
Rafael Neto Henriques. Advanced methods for diffusion MRI data analysis and their application to the healthy ageing brain. Ph. Thesis, 2018.
8 25
12

---

Published as a conference paper at ICLR 2024
Jana Hutter, Paddy J. Slator, Daan Christiaens, Rui Pedro Teixeira, Thomas Roberts, Laurence Jackson, Anthony N. Price, Shaihan Malik, and Joseph V. Hajnal. Integrated and efficient diffusion- relaxometry using ZEBRA. Scientifie reports, 8(1):1-13, 2018.
24 25
Fergus Imrie, Alexander Norcliffe, Pietro Liò, and Mihaela der Schaar. Composite feature selection using deep ensembles. In: Neural Information Processing Systems (NeurIPS), 2022. 4
22
Desi R. Ivanova, Adam Foster, Steven Kleinegesse, Michael U. Gutmann, and Tom Rainforth. Implicit deep adaptive design: Policy-based experimental design without likelihoods. In: Neural Information Processing Systems (NeurIPS), 2021.
18 22
Desi R. Ivanova, Joel Jennings, Tom Rainforth, Cheng Zhang. and Adam Foster. Co-bed: Information- theoretic contextual optimization via bayesian experimental design. In: International Conference
on Machine Learning (ICML), 2023. [2
Jens H. Jensen and Joseph Helpern. MRI quantification of non-Gaussian water diffusion by
kurtosis analysis. NMR in Biomedicine, 23(7):698-710, 2010. 25
Jet Propulsion Laboratory (JPL). AVIRIS web page. https: Savviiii ris. 3pl www..aaa....mmm gov/
2023. 26
Shali Jiang, Henry Chai, Javier Gonzalez, and Roman Garnett. BINOCULARS for efficient, nonmy-
opic sequential experimental design. In: International Conference Machine Learning (ICML),
2020. 22
Edward W. Johnston, Elisenda Bonet-Carne, Uran Ferizi, Ben Yvernault, Hayley Pye, Dominic Patel, Joey Clemente, Wivijin Piga, Susan Heavey, Harbir S. Sidhu, Francesco Giganti, James O'Callaghan, Mrishta Brizmohun Appayya, Alistair Grey, Alexandra Saborowska, Sebastien Ourselin, David Hawkes, Caroline M. Moore, Mark Emberton, Hashim U. Ahmed, Hayley Whitaker, Manuel Rodriguez-Juste Alexander Freeman. David Atkinson, Daniel Alexander, Fleftheria Panagiotaki, and Shonit Punwani. VERDICT MRI for prostate cancer: Intracellular volume fraction versus apparent diffusion coefficient. Radiology, 291(2):391-397 2019. 24
Jean Kaddour, Steindór Sæmundsson, and Mare Peter Deisenroth. Probabilistic active meta-learning. In: Neural Information Processing Systems (NeurIPS), 2020.
Kouhei Kamiya, Masaaki Hori, and Shigeki Aoki. NODDI in clinical research. Journal of Neuro- science Methods, 346: 108908, 2020. ISSN 0165-0270.
22
Shahid Karim, Akeel Qadir, Umar Farooq, Muhammad Shakir, and Asif Laghari. Hyperspectral imaging: A review and trends towards medical imaging. Current Medical Imaging, 10, 2022. 23
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of GANs for im- proved quality, stability, and variation. In: International Conference on Learning Representations (ICLR), 2018.
Muhammad Khan, Hamid Khan, Adeel Yousaf, Khurram Khurshid, and Asad Abbas. Modern trends in hyperspectral image analysis: A review. IEEE Access, 6:14118-14122 14129, 2018. 23
Steven Kleinegesse and Michael U. Gutmann. Bayesian experimental design for implicit models by mutual information neural estimation. In: International Conference on Machine Learning (ICML),
2020. 2
Florian Knoll, Tullie Murrell, Anuroop Sriram, Nañssa Yakubova, Jure Zbontar, Michael Rabbat, et al. Advancing machine learning for mr image reconstruction with an open competition Overview of the 2019 fastMRI challenge. Magnetic resonance medicine, 84(6):355430700 2020. 23
Ron Kohavi and George H. John. Wrappers for feature subset selection. Artificial Intelligence, 97(1): 273-324, 1997.
Atsutoshi Kumagai, Tomoharu Iwata, and Yasutoshi Ida and. Few-shot learning for feature selec- tion with Hilbert-Schmidt independence criterion. In: Neural Information Processing Systems (NeurIPS), 2022. 22
13

---

Published as a conference paper at ICLR 2024
Ludmila I. Kuncheva, Clare E. Matthews, Álvar Arnaiz-Gonzllez, and Juan José Rodríguez Diez. Feature selection from nigh-dimensional data with very low sample size: cautionary tale. arXiv preprint arXiv:2008.12000 2020. 2
Changhee Lee.
sulf-super enhanced feature Code for
selection gil
with commit
correlated gates.
https://gi thub. com/ch////oo//oo 111188855555
2022. [8]
Changhee Lee, Fergus Imrie, and Mihaela van der Schaar. Self-supervision enhanced feature selection with correlated gates. International Conference on Learning Representations (ICLR), 2022. 2
19 22
Yifeng Li, Chih- Yu Chen, and Wyeth W. Wasserman. Deep feature selection: Theory and application to identify enhancers and promoters. Journal of Computational Biology, 23(5):322 -336,, 2016. 22
Jonathan Lightley, Frederik Görlitz, Sunil Kumar, Ranjan Kalita, Arinbjorn Kolbeinsson, Edwin Gar- cia, Yuriy Alexandrov, Vicky Bousgouni, Riccardo Wysoczanski, Peter Barnes, Louise Donnelly, Chris Bakal, Christopher Dunsby, Mark A. A. Neil, Seth Flaxman, and Paul M. W. French. Robust deep learning optical autofocus system applied to automated multiwell plate single molecule localization microscopy. Journal of Microscopy, 288(2): 2022.
Ofir Lindenbaum, Uri Shaham, Erez Peterfreund, Jonathan Svirsky, Nicolas Casey, and Yuval Kluger. Differentiable unsupervised feature selection based on gated laplacian. In: Neural Information Processing Systems (NeurIPS), 2021. 22
Guolan Lu and Baowei Fei. Medical hyperspectral imaging: a review. Journal of Biomedical Optics, 19(1), 2014. 23
Clare Lyle, Arash Mehrjou, Pascal Notin, Andrew Jesson, Stefan Bauer, Yarin Gal, and Patrick Schwab. DiscoBAX: Discovery of optimal intervention sets in genomic experiment design. In:
International Conference on Machine Learning (ICML), 2023. 22
Gustavo Malkomes, Bolong Cheng, Eric Lee, and Mike Mccourt. Beyond the pareto efficient
frontier: Constraint active search for multiobjective experimental design. In: International Conference on Machine Learning (ICML), 2021. 22
Dimitris G. Manolakis, Ronald B. Lockwood, and Thomas W. Cooley. Hyperspectral Imaging Remote Sensing: Physics, Sensors, and Algorithms. Cambridge University Press, 2016. 23
Arash Mehrjou, Ashkan Soleymani, Andrew Jesson, Pascal Notin, Yarin Gal, Stefan Bauer, and Patrick Schwab. Genedisco: A benchmark for experimental design in drug discovery. In: Interna- tional Conference on Learning Representations (ICLR), 2022. 22
Viraj Mchta, Biswajit Paria, Jeff Schneider, Stefano Ermon, and Willie Neiswanger. An experimental design perspective on model-basee reinforcement learning. In: International Conference on Learning Representations (ICLR), 2022. 22
Douglas C. Montgomery. Design and analysis of experiments. John Wiley & Sons, fifth edition,
2001. 2
Matthew J. Muckley, Benedikt Riemenschneider, Alaleh Radmanesh, Sooyoung Kim, Gukyeong Jeong, Jacho Ko, et al. Results of the 2020 fastMRI challenge for machine learning MR image reconstruction. IEEE transactions on medical imaging, 40(9):2306-25) 2021. 23
MUDI Organizers. MUlti-dimensional DIffussin (MUDI) MRI challenge 2019 data.
https
7/ 7/www. www..ddeeeeoiinggriiinn..c co. uk/datal 2022. 24
Mojmir Mutny and Andreas Krause. Experimental design for linear functionals reproducing kernel Hilbert spaces. In: Neural Information Processing Systems (NeurIPS), 2022. 22
Preetam Nandy, Divya Venugopalan, Chun Lo, and Shaunak Chatterjee. A/B testing for recommender systems in a two-sided marketplace. In: Neural Information Processing Systems (NeurIPS), 2021.
22
14

---

Published as a conference paper at ICLR 2024
Eleftheria Panagiotaki, Rachel W. Chan, Nikolaos Dikaios, Hashim U. Ahmed, James O D'Callagaan, Alex Freeman, David Atkinson, Shonit Punwani, David J. Hawkes, and Daniel C. Alexander. Microstructural characterization of normal and malignant human prostate tissue with vascular, extracellular, and restricted diffusion for eytometry in tumours magnetic resonance imaging. Investigative Radiology, (0(4):21-222) 2015a.
24
Eleftheria Panagiotaki, Andrada lanuş, Edward Johnston, Rachel W. Chan, Nicola Stevens, David Atkinson, Shonit Punwani, David J. Hawkes, and Daniel C. Alexander. Optimised VERDICT MRI protocol for prostate cancer characterisation. In: International Society for Magnetic Resonance in Medicine (ISMRM), 2015b.
21 24
Fletheria Panagiotaki, Simon Walker-Samuel, Bernard Siow, Peter Johnson, Vineeth Rajkumar, Barbara R. Pedley, Mark F. Lythgoe, and Daniel C. Alexander. Noninvasive quantification of solid tumor microstructure using VERDICT MRL. Cancer research, 74(7):1902-1912, 2014. 24
Hanchuan Peng, Fuhui Long, and Chris Ding. Feature selection based on mutual information: criteria of dependden,,,, max-relevance, and min-redundancy. IEEE Transactions on Puttern Analysis and Machine Intelligence, 27(8): 1226-1238, 2005,
22
Marco Pizzolato, Marco Palombo, Elisenda Bonet- Bonet-Carne, Francesco Grussu, Andrada Lanuş, Fabian Bogusz, Tomasz Pieciak, Lipeng Ning. Stefano B. Blumberg, Thomy Mertzanidou, Daniel C. Alexander, Maryam Afzali, Santiago Aja-Fernández, Derek K. Jones, Carl-Fredrik Westin, Yogesh Rathi, Steven H. Baete, Lucilio Cordero-Grande, Thilo Ladner, Paddy J. Slator, Daan Christi- aens, Jean-Philippe Thiran, Anthony N. Price, Farshid Sepehrband, Fan Zhang, and Jana Hutter. Acquiring and predicting Multiidimennionnll DIffusion (MUDI) data: open challenge. In:
International Society for Magnetic Resonance in Medicine (ISMRM). 2020. 7 25
Friedrich Pukelsheim. Optimal Design of Experiments. Society for Industrial and Applied Mathemat- ics, 2006.
Francesco Quinzan, Ashkan Soleymani, Patrick Jaillet, Cristian R. Rojas, and Stefan Bauer. DRCFS: Doubly robust causal feature selection. In: International Conference on Machine Learning (ICML), 2023. 22
Logan R. Ranzenberger and Travis Snyder. Diffision Tensor Imaging. StatPearls Publishing, fifth edition, 2022.
Scikit-Lear.. Scikit-leam recursive feature elimination http:t/ttts/ssctttteeeeeee org otbp::/ccotteeeeeeeereeeaaaaaaee RFt. 2023.
David Simchi-Len Levi and Chonghuan Wang. Pricing experimental design: Causal effect, expected revenue and tail risk. In: International Conference Machine Learning (ICML), 2023. 22
John J. Simmonds and Robert O. Green. Current status, preformance and plans for the NASA airbome visible and infrared imaging spectrometer (AVIRIS). 1996. URL httph::///// // htwp:://wwww.. www.wwovt.gooo
biblio/379497 26
Saurabh Singh, Harriet Rogers, Baris Kanber, Joey Clemente, Hayley Pye, Edward W. Johnston, Tom Parry, Alistair Grey, Eoin Dinneen, Greg Shaw, Susan Heavey, Urszula Stopka-Farooqui, Aiman Haider, Alex Freeman, Francesco Giganti, David Atkinson, Caroline M. Moore, Hayley C. Whitaker, Daniel C. Alexander, Eleftheria Panagiotaki, and Shonit Punwani. Avoiding unnecessary biopsy after multiparametric prostate MRI with VERDICT analysis: The INNOVATE study. Radiology, pp- 212536, 2022.
24
Andrew Sinkoe and Juergen Hahn. Optimal experimental design for parameter estimation of an IL-6 signaling model. Processes, 5(3), 2017.
Paddy J. Slator, Marco Palombo, Karla L. Miller, Carl-Fredrik Westin, Frederik Laun, Daeun Kim, Justin P. Haldar, Dan Benjamini, Gregory Lemberskiy, Joao P. de Almeida Martins, and Jana Hutter. Combined iiffusion-relaxomeet microstructure imaging: Current status and future prospects. Magnetic Resonance in Medicine, 86(6))29873011, 2021.
15

---

Published as a conference paper at ICLR 2024
Ghada Sokar, Zahra Atashgahi, Mykola Pechenizkiy, and Decebal Constantin Mocanu. Where to pay attention in sparse training for feature selection? In: Neural Information Processing Systems (NeurIPS), 2022. 22
Le Song, Alexander J. Smola, Arthur Gretton, Justin Bedo, and Karsten M. Borgwardt. Supervised feature selection via dependence estimation. In: International Conference on Machine learning
(ICML), 2007. 22
Le Song, Alexander J. Smola, Arthur Gretton, Justin Bedo, and Karsten M. Borgwardt, Feature selection via dependence maximization. Journal of Machine Learning Research, 13(5)113331444,
2012. 22
Mary B. Stuart, Andrew J. S. McGonigle, and Jon R. Willmott. Hyperspectral imaging in environ- mental monitoring: A review of recent developments and technological advances in compact field deployable systems. Sensors, 19(14):3071, 2019. 23
Mary B. Stuart, Leigh Stanger, Matthew Hobbs, Tom D. Pering, Daniel Thio, Andrew J.S. McGonigle, and Jon R. Willmott. Low-cost hyperspectral imaging system: Design and testing for laboratory-basee environmental applications. Sensors, 20( 1):3239, 2020. 23
Woo Suk Tae, Byung Joo Ham, Sung Bom Pyun, Shin Hyuk Kang, and Byung Jo Kim. Current clinical applications of diffusion-tensor imaging in neurological disorders. Journal of Clinical Neurology, 14(2):129-140, 2018.] 25
Ali Ahmadi Teshnizi, Saber Salehkaleybar, and Negar Kiyavash. Lazyiter: A fast algorithm for counting markov equivalent dags and designing experiments. In: International Conference on Machine Learning (ICML), 2020. 22
David R. Thompson, Joseph W. Boardman, Michael L. Eastwood, and Robert O. Green. A large airborne survey of earth's 9195, 2017.
spectral dimensionality. Optics Express, 25(8);9188
g 26
Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistics Society, Series B (Methodological), pp. 267-288, 1996. 22
Panagiotis Tigas, Yashas Annadani, Andrew Jesson, Bernhard Schölkopf, Yarin Gal, and Stefan Bauer. Interventions, where and how? Experimental design for causal models at scale. In: Neural Information Processing Systems (NeurIPS), 2022.
Panagiotis Tigas, Yashas Annadani, Desi R. Ivanova, Andrew Jesson, Yarin Gal, Adam Foster, and Stefan Bauer. Differentiable multi-target causal bayesian experimental design. In: International Conference on Machine Learning (ICML), 2023.
22
Dale J. Waterhouse and Danail Stoyanov. Optimized spectral filter design enables more accurate estimation of oxygen saturation in spectral imaging. Biomedical Optics Express, 13(4):1156-2173, 2022.
18 23 26
Maksymilian A. Wojtas. Code for feature importance ranking for deep learning, git com-
mit 3260066ubbBs222100aeeeeeeeccrera
bttts::/gitta wwom/maaksae
Feature 1.2 19
Maksymilian A. Wojtas and Ke Chen. Feature importance ranking for deep learning. In: Neural
Information Processing System (NeurIPS), 2020.
19 22
Renjie Wu, Yuqi Li, Xijiong Xie, and Zhijie Lin. Optimized multi-spectral filter arrays for spectral reconstruction. Sensors, 19(13), 2019.
23
Yutaro Yamada, Ofir Lindenbaum, Sahand Negahban, and Yuval Kluger. Feature selection using stochastic gates. In: International Conference on Machine Learning (ICML), 2020. 22
Burhaneddin Yaman, Seyed Amir Hossein Hosseini, and Mehmet Akçakaya. Zero-shot self- supervised learning for MRI reconstruction. In: International Conference on Learning Rep- resentations (ICLR), 2022. 23
16

---

Published as a conference paper at ICLR 2024
Vincent D. Zaballa and Elliot E. Hui. Stochastic gradient bayesian optimal experimental designs for simulation- sraslattoo--beedd inference. In: Differentiable Almost Everything Workshop of the International Conference Machine Learning (ICML), 2023. 22
Jure Zbontar, Florian Knoll, Anuroop Sriram, Tullie Murrell, Zhengnan Huang, Matthew J Muck- ley, et al. fastMRI: An open dataset and benchmarks for accelerated mri. arXiv preprint arXiv: 1.08839, 2018.
X
Hui Zhang, Torben Schneider, Claudia A. Wheeler-Kingshott, and Daniel Alexander. NODDI: practical in vivo neurite orientation dispersion and density imaging of the human brain. Neurolmage, 61(4):1000-1016 2012.
19 20 24
Junzhe Zhang, Jin Tian, and Elias Bareinboim. Partial counterfactual identification from observational and experimental data. In: International Conference Machine Learning (ICML), 2022. 22
Sue Zheng, David Hayden, Jason Pacheco, and John W Fisher III. Sequential bayesian experimental design with variable cost structure. In: Neural Information Processing Systems (NeurIPS), 2020.
22
APPENDIX STRUCTURE
The appendices are structured as follows:
1. Appendix provides comprehensive details on all approaches utilized in this paper, including the specific hyperparameters employed.
2. Appendix B are supplementary experimental results.
3. Appendix offers further experimental analysis of our method, TADRED.
4. Appendix compares the computational cost of all the approaches used in this paper and outlines the computational resources employed.
5. Appendix is comprehensive description of prior work related to our problem.
6. Appendix details all data for each experiment, along with specifics of each task.
A KEY APPROACHES, HYPERPARAMETERS, AND SETTINGS
This section describes the different supervised feature selection approaches used in this paper and details the choice of parameter settings within each.
GENERAL EXPERIMENTAL SETTINGS
For every experiment comparing TADRED with baselines, split the data into training, valida- tion/development, and test sets. This is described in detail section Following Lee et al. (2022) (paper of SSEFS), we conducted extensive hyperparameter search for each approach (using the validation set), for different experimental settings and subsample values C. This described in detail below for every approach. For fairness, we have the same number of evaluations on the validation set for each different feature set size C', i.c. number of trials for model selection. The best model was then applied the test set and we reported the performance.
Other general hyperparameters are: batch size 1500, learning rate 10-4 (10-5 for the experiment in figure 2}. ADAM optimizer, and default network weight initialization. The default option for early stopping used 20 epochs for patience (i.e training stops if validation performance does not improve in 20 epochs).
TADRED TASK-DRIVEN EXPERIMENTAL DESIGN FOR MULTI-CHANNEL IMAGING
This subsection details the hyperparameters for the method we present in this paper: TADRED for TAsk-DRiven Experimental Design in imaging, as outlined in section 3 Figure provides a graphical representation of TADRED's structure and computational graph.
17

---

Published as a conference paper at ICLR 2024
Gamplekcspandar
S₁
Meon
Vourg PurencAr
X D
Task Execution
Scoring
Loop
No Backpropagation
Doss
Vesk
T Iask Natwork
Backpropagation
x
Tepat
X D
x
X Succampled
Dama
Feature Subsampling
Remroved
60
Fauaue
Figure 3: TADRED's structure. During training TADRED concurrently performs feature scoring, feature subsampling, and task execution. During training we progressively set the score to be sample- independent by setting a to 1. We score features with s, E IRC and remove features with low score by setting corresponding values of the mask to O, in this example removed feature 2.
We conducted brief search for TADRED hyperparameters and fixed these hyperparameters across all experiments. We set the numbers of epochs in the four-phase inner loop training procedure Ei 25, Es E1 + 10, E3- Eg + 10. Following other baselines. do not fix the total number of training epochs E, but keep training beyond e E' in algorithm phase 4 until early stopping criteria (on the validation set) are met.
For fairness, when comparing TADRED with other supervised feature selection baselines, we chose simple set of hyperparameters for the feature set sizes ) and number of outer loop steps T.
Here we fixed 7 = 5 and C1. C2, Cg, C1, = C 2: 4 8 16" two recent application-spedtee
C When comparing TADRED against
published results optimized by the authors of Blumberg al. (2022); Waterhouse & Stoyanov (2022). we followed standard practice and performed brief hyperparameter search the validation set. We used Ci Cg = { (1344, 500, 250, 100, 50, 40, 30, 20, 10), T
in table and C1, C19 = [348] + [250:45::-500 + [45::8:-5] [8,6.5.4.3,21 T (start:stop::step] in table
19 (notation is
We perform a grid search to find the optimal network architecture hyperparameters for each task. The Scoring Network S and Task Network T have the same number hidden layers 21, 3J. number of units E {30, 100, 300, 3000), and for each combination we obtain task performance on the feature set sizes C1 > C2 > > Cr. The best performing network on the validation is deployed on the test data.
RANDOM SUPERVISED FEATURE SELECTION
This baseline is inspired by the random design baselines used in experimental design papers e.g. Foster et al. (2021) Ivanova al. (02II). For particular design size, C. repeat the following process: i) randomly select C' features/channels: ii) perform grid search on the task network (mapping subsampled data Xp target Y), with number of hidden layers E {1, 2, 3), number of units {30, 100, 300, 1000, 3000}: iii) train until early stopping criteria specified on the validation set are met; iv) evaluate the best trained model on the test set.
SELF-SUPPERVSSON ENHANCED FEATURE SELECTION WITH CORRELATED GATES (SSEFS)
LEE AL. (2022)
This approach has lengthy hyperparameter search detailed in Appendix of et al. (2022). which consists of three-phase procedure and four neural networks. Note that this required multiple training steps e.g. obtaining results for table I requires >400 runs. SSEFS exploits task mask-pefformance self-supervision, additional unlabeled data, and correlated feature subsets. It scores the features then subsequently trains a task-based network (analogous to T in TADRED) on subsampled data. We the official repository Lee (2022) and verified our implementation by replicating results in the
18

---

Published as a conference paper at ICLR 2024
paper Lee al. (0022). The full optimization procedure follows Lee et al. (2022) and is split into i) self suffrvuperviioon phase, ii) supervision phase, iii) training on selected features only.
The self-supervision phase finds the optimal encoder network hyperparameters. We follow Appendix of Lee et al.] (2022) and perform grid search. Similar to other approaches in this paper. we consider the encoder network, feature vector estimator network, gate vector estimator network, all have same number of hidden layers E {1. 2..3.3 3) number of units, including hidden dimension {30, 100, 300, 1000, 3000). Directly following Lee al. (2022) table S.I, other hyperparameters 10, 100 E {0.2, 0.4, 0.6, 0.83. The self-supervisory dataset is input data X
E (0.01, 1.0, 1.0, On the best validation performance (with early stopping). this returns trained encoder network, cached for the supervision phase.
The supervision phase scores the features. The pretrained encoder is loaded from the previous phase. We then perform grid search, where the predictor network has number of hidden layers 3), number of units {30, 100, 300, 1000, 3000), following Lee al. (2022) table S. 3 E {0.01, 1.0, 10, 1003. On the best validation performance with early stopping, the process returns a score for all features.
The final phase is repeated for different number of subset sizes C. We extract the C' highest scored features from the previous phase and perform grid search on the task network (mapping subsampled data Xp to target Y). with number of hidden layers E (, 3). number of units {30, 100, 300, 1000, 3000]. Training is until early stopping on the validation set. The best trained model is evaluated on the test set.
FEATURE IMPORTANCE RANKING FOR DEEP LEARNING (FIRDL) WOJTAS & CHEN (2020)
This approach has three-stage procedure detailed in Appendix D of Wojtas & Chen (2200) and uses two neural networks. One of the networks scores the masks (analogous to mask m in TADRED). and the other trains task network to perform task on the subsampled data (analogous We use the official repository Wojtas (202T) and verified our implementation by replicating results in the paper Wojtas & Chen 2020). The following process is repeated for different feature subset sizes C.
in TADRED),
We perform a grid search to find the optimal hyperparameters. The operator network (analogous to task network 'T in this paper) and the selector network have the same number of hidden layers (1,2,3), number of units E (30, 100, 300, 1000, 3000), Ap = 5, F 15000. The joint training uses early stopping the validation set, and returns optimal feature set size, of size C' and trained operator network. The best performing operator network on the validation set is deployed on the test data.
B ADDITIONAL RESULTS
This section provides additional results supporting the experiments presented in the main paper.
Table 7 contains results that repeat the experiment in table using the NODDI model Zhang et al. (2012) instead of VERDICT. The first bascline uses the classical Fisher-matris experimental design Alexander (2088) compute the design 1) from Zhang al. (2012) where C 99. For the supervised feature selection approaches we use densely-sampled designs 5 where Ö = 3612 Ferizi al. (2017). Similar to results in table [ table 7 TADRED outperforms classical experimental design with C' set to 99 following the classical approach used in current practice. In addition,
TADRED outperforms the supervised feature selection baselines where C = 2. 4 8 16:16 optimized designs enable us to estimate the widely used NODDI parameters shorter sean times opening the potential for wider range of clinical applications. All information designs, models and data is in section ET
C C C C The
Table 8 shows extra results within the experiment documented in figure We consider additional feature set subsample size C = 36 which extends the results in figure 2 and show TADRED outperforms the baselines 1 17/18 comparisons
clinically useful downstream metries. This is beneficial as pressure for time in clinical MRI protocols is intense as many different MR contrasts are informative, but patient-time in the scanner is limited. Therefore shorter acquisition protocols for
19

---

Published as a conference paper at ICLR 2024
Table 8: MSE for downstream MRI metrics (see appendix F.3 estimated from the full set of mea-
Table 7: MSE x x0 between estimated NODDI model parameters and ground truth parameters used to simulate the data. Classical experimental design is from Zhang et al. (2012) and uses a Fisher-matrix approach. Additional results for table
surements on HCP data C 288, and C from
C 36 reconstructed measurements. Addi- tional results to figure
DTI
FA
MD
AD
RD
Random
1.17
3.13
9.3
3.75
Classic Experimental Design
99
8.00
SSEFS
5.96
10.4
43.7
TADRED
99,
3612
4.51
HIRDL.
10.4
68.8
106
81.4
TADRED
0.94
1.73
8.15
1.50
C 3612
C
1806
452
226
DKI
MSDKI
Random
2.99
3.34
3.88
4.31
MK
AK
RK
MSD
MSK
SSEFS
2.95
3.39
3.73
4.39
Random
6.37
6.28
13.3
2.50
4.15
FIRDL
4.21
4.61
4.96
5.14
SSEFS
8.74
7.27
16.9
3.87
10.8
TADRED
2.59
2.92
3.33
3.85
FIRDL
9.39
10.3
18.6
11.1
8.82
TADRED
6.15
5.92
12.6
2.75
3.78
Table 10: Performance comparison of feature selection approaches for remote sensing AVIRIS hyperspectral data (east-to-west flight of Indian Pine), MSE between C 220 reconstructed and C ground-trute measurements. Additional results table B
Table 9: MSE between C
1344 recon-
structed measurements and C ground-trutt mea- surements Multi-Diffusion challenge subjects. Additional results to table 2
C' 500
250
50
Random
0.93
1.41
2.12
5.23
C 110
55
28
14
SSEFS
0.63
0.86
1.24
1.61
Random
1.22
1.76
2.91
5.61
FIRDL
1.67
1.72
2.17
2.34
SSEFS
1.36
3.11
3.77
7.61
TADRED
0.21
0.44
0.94
1.34
FIRDL
6.34
7.16
7.78
TADRED
0.60
1.42
2.33
4.49
these widely informative downstream metrics (parametric maps) enables their exploitation in wider range of clinical studies and applications.
Table Jssows additional results on the MUDI data in table This experiment compares TADRED with the supervised feature selection baselines following settings in the original MUDI challenge. Evaluation uses MSE metric as original challenge. Further details are appendix F.2
Table TO shows additional results on the AVIRIS data presented in table Here, only use data from the north-to-south flight. Improvements of TADRED over the supervised feature selection baselines are similar to that in table
C FURTHER ANALYSIS
C.I ANALYZING THE EFFECT ON RANDOMNESS ON FEATURE SET CHOSENN
Table II examines how the changing the random seed that affects network initialization and data shuffling, impacts the feature set chosen. Results show TADRED performs favorably compared to alternative approaches and mostly chooses the same features
C.2 EVALUATION OF THE CHOICE OF FEATURE FILL XFIII.
Table 12 examines the effect of varying the values D that fill the unsubsampled features. FIRDL used zeros for its equivalent of X-ill and SSEFS used the data mean (per channel/feature). Results show even if we set the values of X to that of the baselines, TADRED has large improvements the baselines.
C.3 HOW DOES THE SIZE OF THE DENSELY-SAMPLED DESIGN AFFECT PERFORMANCE?
20

---

Published as a conference paper at ICLR 2024
Table 11: Mean Jaccard Index between chosen measurements, across 10 random seeds, experi- mental settings table VERDICT simulations.
Table 12: Comparison of the choice of X Ex-
perimental settings follow table
Xp
C 110
55
28
14
C = 110
55
28
14
SSEFS
data mean
1.06
1.28
1.89
4.38
Random
32.8
15.2
6,82
2.87
FIRDL
zeros
2.22
2.14
3.09
4.05
SSEFS
81.2
71.4
62.2
75,9
TADRED
data median
1.03
1.19
1.80
2.55
FIRDL
34.3
18.1
48.8
41.0
TADRED
data mean
1.03
1.20
1.79
2.80
TADRED
74.3
82.1
84.6
59.1
TADRED
zeros
1.03
1.19
1.79
2.51
We examine how varying the size of densely- sampled design D (used to create Xp) affects performance. Across 10 random seeds, ran- domly sample the design from Panagiotaki et al.) (2015b) to create custom D with C' elements. Training on fixed network sizes for a single
masuuremeent Over hursion Seeads
Valldttion Mean Squares pmora2102
3.75
subsampling rate C2 C 14. We use 10% of the training data within the experimental settings
3.50
of table Results in figure and exemplify typical behavior that while performance is rea- sonably stable for large C phase change occurs as C' nears C' and performance decreases rapidly, as the set of samples to choose from becomes too sparse.
3.25
3.00
2.75
Size Donacly Sampled Design CBar
Figure 4: Analyzing the performance on different densely- designs D where [D] = C and
C.4 TADRED VARIANT WITH RANDOM SELECTION
= 14. Settings follow table
We tested modified training procedure that works in the 'same manner' as the original im- plementation of TADRED whilst the scores chosen are random. Across different modifications, in settings of table 5) results (MSE) are more than 10% worse, even more than 'w/o iterative subsam- pling'. Thus, although the "gradual aspect' of TADRED's training procedure improves performance (in fact line in the ablation study table 5 already demonstrates this with 'less gradual' scenario without iterative subsampling decreases performance), the learning of the scoring network is working intended and further improves results.
D COMPUTATIONAL COST OF DIFFERENT APPROACHES AND INFRASTRUCTURE
It is difficult to compare the computational cost of TADRED against SSEFS, FIRDL. The official implementations, described in appendix use different machine learning frameworks and all use customized early stopping. In particular, SSEFS has three stage procedure (first two large hyperparameter searches) which are completed consecutively; TADRED does not require this. distinct networks, whilst SSEFS uses four, such, training costs
TADRED and FIRDL train are somewhat comparable if network sizes are taken to be the same. Practical requirements in all cases were reasonable and training for all methods for each experiment were performed within 24 hours. As an example, we compare the time to run the various methods for the results in table per C' with the network sizes fixed and no hyperparameter search over different network sizes. Training speeds are: Random supervised feature selection (baseline) 505s; SSEFS (baseline) 1934s (using only run single run per seed for the first two stages; as previously noted, for other results in the paper, this is much slower the method proposes computationally expensive sequential hyperparameter search); FIRDL (baseline) 1756s; TADRED (the new approach) 1988s. The random supervised feature selection baseline is by far the most computationally economical, no iterative search. TADRED's ' computational cost similar to the two state-of-the- supervised feature selection baselines, SSEFS and FIRDI..
expect, because uses
21

---

Published as a conference paper at ICLR 2024
Exploratory analysis and development was conducted on mid-range (as of 2023) machine with AMD Ryzen Threadripper 2950X CPU and single Titan GPU. All experimental results reported in this paper were computed on low-to-mid range (as of 2023) graphical processing units (GPU); GTX 1080 Ti, Titan Xp. Titan X, Titan V. RTX 2080 Ti. We ran jobs on high-performance computing cluster shared with other users, allowing multiple jobs to run in parallel.
E EXTENDED RELATED WORK
This section provides further information to section 2 detailing previous work related to our problem.
Classical and Other Recent Supervised Feature Selection Approaches Supervised feature selection approaches are either i) 'filter methods', which select features using some proxy metric independent of the final task, ii) "wrapper methods', which use the task to evaluate feature set performance, or iii) 'embedded methods', which couple the feature selection with the task training. Embedded methods FIRDL Wojtas & Chen (2020), SSEFS [ee et al. (2022) are state-of-the-art outperforming classical approaches e-g recursive feature elimination (orgnnal Guyon et al. (2002). BAHSIC Song et al.
2007 2012), mRMR Peng et al. (2005). CCM Chen et al. (2017). )) RF Breiman (200T). DFS
al. (2016). LASSO Tibshirani (1996), Scoree He al. (2005) and recent deep learning-bass based CE Abid et al. (2019) STG Yamada et al. (2020). DUFS Lindenbaum et al.] (202T). More recent
approaches extend the supervised feature selection paradigm to limit false discovery rate Hansen et al.
(2022), , few-shot learning Kumagai et al. (2022), discovering groups of predictive features Imrie et al.
(2022), the unsupervised setting Sokar et al. (2022), few-sample classification problems| Cohen et al.
(2023), dynamic feature selection Covert et al. (2023), for federated learning Castiglia et al. (2023). for identifying high-dime high-dimensional causal features Quinzan et al. (2023). They are not designed for the standard regression- based supervised feature selection problem considered in this paper.
Other Recent Experimental Design Approaches Techniques for experimental design have been
developed for causal modeling Tigas et al. (2022); Zhang et al. (2022) Teshnizi et al. 2020) linear
models Fontaine et al. (202T): Mutny & Krause (2022), online learning Arbour et al. (2022) active
learning Kaddour et al. (2020), drug discovery Mehrjou et al. (2022), reinforcement learning Mehta
iet al. (2022), A/B testing Nandy et al. (2021), panel-data settings Doudchenko et al.) (2021), bandit
problems Camilleri al. (2021), balancing competing objectives with uncertainty Malkomes et al.
(2021), temporal treatment and control Glynn et al. (2020). causal discovery when interventions can be costly or risky Tigas et al. (2023), designing pricing experiments Simchi-Levi & Wang (2023).
contextual optimization for Bayesian experimental design [vanova et al. (2023), genomies Lyle et al. (2023), treatment effects in large randomized trials CiinollCy et al. (002)), learning causal models with Bayesian approaches Annadani et al. (2023). These are not applicable to the problem setting we consider. Approaches Zheng et al. (2020 : Kleinegesse & Gutmann (2020); Jiang et al. (2020) are older sequential experimental design approaches, whilst Zaballa & Hui (2023) is contemporary to this work - they face the same issues as Blau et al. 2202),, Foster et al. (2021) Ivanova et al. (2021) (discussed in section [ which focus on estimating model parameters and are mostly demonstrated in small-scale problems which do not scale up to the high dimensional problems we face in experimental design for image-channel selection.
Experimental Design in qRRI In qRI the design D is known as an 'acquisition scheme". One standard task is 'parameter mapping'. first estimating dollogcally-informaa. voxel-wise model fitting, to then obtain downstream metrics Alexander et al. (2019). This provides information that is not visible directly from the images, such as microstructural properties of tissue. However, acquisition time (corresponding to C' Di) is limited by factors of cost and the ability of (often sick) subjects to remain motionless in the noisy and claustrophobic environment of the scanner. Thus experimental design can be crucial to support the most accurate image-driven diagnosis, prognosis, or treatment choices. Many clinical scenarios use based on intuition loosely guided by understanding of the physical systems under examination, but this can lead to highly suboptimal designs particularly for complex models. However, some studies optimize the design using the Fisher information matrix, e.g. Alexander (2008); Cercignani & Alexander (0066.
model parameters by
Lengthy MRI acquisitions corresponding to D to enable our new experimental design paradigm made easily few subjects, but not feasible in routine patient imaging. However, such lengthy acquisitions are often made in the design phase of quantitative imaging techniques, e.g. as in Ferizi al. (017). We note also that several distinct experiment design problems arise in MRI.
22

---

Published as a conference paper at ICLR 2024
VERDICT Simulations
MUDI
Indian Pine
0.5
1948
219
1343
219
0.0
NODDI Simulations
HCP Retest
Oxygen Saturation
-0.5
-1.0
3611
Figure 5: Correlation coefficient between the messuremennt//eaaurscccaamel of the data.
Here we focus on estimating per-voxel parameter values, but others e.g. Zbontar et al. (2018); Knoil et al. (2020); Muckley et al. (2021); Fabian et al. (2022); Yaman et al. (2022), focus on how best to subsample the k-space. Our approach is complementary to and may combined with those; they expedite the acquisition of each individual channel, we identify compact/cconomical set of channels.
Experimental Design in Hyperspectral Imaging Hyperspectral imaging (a.k.a. imaging spec- troscopy) obtains pixel-wise information of an object-of across multiple wavelengths the electromagnetic spectrum from specialized hardware Manolakis et al. (2016). This produces 'image cube' 2D image with C channels, as in qMRI, image channels correspond to the measure- ments. Experimental design involves choosing a design consisting of wavelengths and/or filters Arad & Ben-Shhhar 2017); Waterhouse & Stoyanov 2202);; ) Wu et al. (2019) which controls the image channels, where most current practice uses uniform spacing for the wavelengths Thompson et al.) (0117). For our paradigm, expensive devices can acquire large numbers of images with different spectral sensitivity simultaneously to provide training data for the design of much cheaper deployable devices Stuart et al. (2020). Recovering high-quality information from the few wavelengths chosen for particular applications by experimental design, reduces acquisition cost, increases acquisition speed, avoids misalignment, reduces storage requirements, and speeds up clinical adoption. Hyperspectral imaging has many applications Khan et al. (2018) from multiple modalities in medical imaging z Lu & Feil (2014) Karim al. (022). remote sensing (Baumaadnner al. (2015). and environmental monitoring Stuart et al. (2019)
F
DATA AND TASK EVALUATION
Table 13: Summary data used in this paper.
Channels C
Target Regressoon
Pixel/Vaxel Size
No. Independent Viriables
No. pixelolvoxels x x10"
Name
Results
Data Type
Train
Val
Test
VERDICT
Table
Simulated gMRI
220
d' C
1000
100
100
NODDI
Table
Simulated gMRI
3612
d' eR"
100
MUDI
Table
4MRI Scan
1344
1344
2.5mm²
CRR
321
132
105
HCP
Scan
288
288
1.25mm²
2182
774
674
Indian Pine
Table
Remote Sensing Hyperspectal
220
220
20m"
d' C.
1480
164
1135
Osygen Saturation
Simulated Hyperspectral
This section includes additional details about the experimental data. Table 13 provides summary, fig- ure 6 visualizes various examples,figee 5 a correlation plot of the messurementt/chaamellffelnee
We follow Grussu et al. 2021) and normalize each chamee/measurement//cuuuu by dividing by its 99th percentile value calculated from the training set. This is performed in both the input and output of the neural network.
23

---

Published as a conference paper at ICLR 2024
Table 14: Parameter ranges for simulating synthetic VERDICT and NODDI model data.
VERDICT Model
NODDI Model
Parameter
Minimum
Maximum
Parameter
Minimum
Maximum
fr
0.01
0.99
fie
0.01
0.99
fv
0.01
0.99
Do (umss 3
3.05
10
Jiso
0.01
0.99
ODI
0.01
0.99
R (Jm)
0.01
20
n
1-1-1-11
1 1]
[-1-1-1]
11]]]
F.1 SIMULATIONS WITH THE VERDICT AND NODDI BIOPHYSICAL MODELS.
This section describes the the VERDICT and NODDI models and the experimental settings used in tables Exact code to perform the simulations is in Code Link
The VERDICT (Vascular, Extracellular and Restricted Diffusion for Cytometry in Tumors) model Panagiotaki et al. (2014), maps histological features of solid-cance tumors particularly for early
detection and classification of prostate cancer Panagiotaki et al.) (2015a); Joonnsoonn al.. (2019): (Singh et al. (2022). The VERDICT model includes parameters: Jr the intra-cellular volume fraction, Jv the vascular volume fraction, D, the vascular perpendicular diffusivity, the mean cell radius, and 3D vector defining mean local vascular orientation.
The NODDI (Neurite Orientation Dispersion and Density Imaging) model Zhang et al. (0012) maps parameters of the cellular composition of brain tissue and is widely used in neuroimaging studies in neuroscience such as the UK Biobank study Alfaro-Almagro et al. (2018), and neurology e.g. in Alzheimer's disease Kamiya et al. (2020) and multiple sclerosis Grussu al. (2017). The NODDI model includes five tissue parameters: Jic the intra-cellular volume fraction, Jiso the isotropic volume fraction, the orientation dispersion index (ODI) that reflects the level of variation in neurite orientation, and n a 3D vector defining the mean local fiber orientation.
To conduct the simulations the VERDICT and NODDI models, we employ the widely- used, open-source dmipy toolbox Fick et al. (201)). The code is available: [ooee Link In each case, data simulation a known, fixed, acquisition scheme, i.e. experimental design, combination with set of ground truth model parameters. We chose the ground truth model parameters {0),... ...., On for voxel/sample i 1,.... n by uniformly sampling parameter combinations from the bounds given in We choose these bounds as they approximate the physically feasible limits of the parameters.
table 14
The VERDICT data has number of samples n 1000K, 100K, 100K the train, validation, test split, with target data Y E R 0 E R = 1.... n. The classical experimental design approach yields an acquisition scheme derived from the Fisher information matrix Panagiotaki et al. (2015b) and here X, E R C = 20. The approaches in supervised feature selection (including TADRED) also use densely-sampled empirical acquisition scheme, designed specifically for the VERDICT protocol from Panagiotaki et al. (2015a) and here XD x with C 220 measurements.
The NODDI data has number of samples n = 100K, 10K, 10K in the train, www.vaaiiiiintt.. split, with target data Y E Rax7 02 E RT, 1, ..., The classical experimental design approach yields an acquisition scheme derived from the Fisher information matrix Zhang et al. (2012) and so XG, * C 99. The approaches in supervised feature selection use densely censely-samples empirical acquisition scheme from an extremely rich acquisition from Ferizi et al. (2017). This was designed for the ISBI 2015 White Matter Challenge, which aimed collect the richest possible data to rank biophysical models, and required single subject to remain motionless for two uncomfortable back-to-back 4 hour scans. Here XD E xax3612 with C' 3612 measurements.
We added Rician noise to all simulated signals, which is standard for MRI data Gudbjartsson & Patz (1955.. The signal to noise ratio of the unweighted signal 50, which is representative of clinical MMR.
F.2 MULIIDDIFFUSON (MUDI) CHIALLNGE DATA
Data used in tables 2 9 are images from five in-vivo human subjects, and are publicly available MUDI Organizers (2022) ) and was acquired with the state-of-the-ar ZEBRA sequence Hutter et al.)
2018). This difuusion-relaxation MRI dataset has 6D acquisition parameter space d' R
24

---

Published as a conference paper at ICLR 2024
echo time (TE), inversion time (TD), b-value, and b-vector directions in 3 dimensions: bey by. bz. Data has 2.5mm isotropic resolution and field-of-view 220 x 230 x 140mm and resulted in 3D brain volumes (i.e. images) with C 1344 measurements//hameee diffusion- 72" and 71- 71- weighting contrasts. More information is in Hutter et al. (2018); Pizzolato al... (2020). Each subject has associated brain mask, after removing outlier voxels resulted in 104520, 110420, 105743, 132470. 105045 voxels for respective subjects 11, 12, 13, 14, 15. For the experiment in table 2 we follow Blumberg et al. (2022) and perform 5-fold cross validation on the subjects. For the experiment in [ we followed the original challenge Pizzolato et al. (2020) and took subjects 11, 12, 13 as the training and validation set, and subjects 14, 15 as the unseen test set, where 90% 10% of the training/validation set voxels are respectively, for training and validation.
which here unique
F.3 HUMAN CONNECTOME PROJECT (HCP) TEST-RETEST DATA
This section describes the data and model fitting procedure used in figure and table g
This section utilizes WU-Minn Human Connectome Project (HCP) diffusion data, which is publicly available at www. www.nconnectoomm org (Test Retest Data Release, release date: Mar 01, 2017) Essen et al. (2013). The data comprises C' = 288 volumes (i.e. mcasureeeentt/chameeeB)
with 18 b=0 s mm 2 (i.e. non-diffusion weighted) volumes, 90 gradient directions for b=1000
mm 2 90 directions for b=2000 s mm -2 and 90 directions for b=3000 s mm 2 We used 3 scans for training (ID numbers 103818_1 11, 105923_1 _ 11111111_111 1), one scan for validation (114823 1) and one scan for testing (115320_1,, which produced numbers of samples n 708724 + 791369 + 681650 = 2181743, 774149, 674404 for the respective splits. We used only voxels inside the provided brain mask and normalized the data voxelwise with a standard technique in MRI, by dividing all measurements by the mean signal in each voxel's b=0 values. Undefined voxels were then removed.
Diffusion tensor imaging (DTI) Basser et al. (1944), diffusion kurtosis imaging (DKI) Jensen & Helpern (2010), and Mean Signal DKI (MSDK)) Henriques (2018) are widely-used qMRl methods. Like NODDI and VERDICT, they use diffusion MRI to sensitize the image intensity to the Brownian motion of water molecules within the tissue to provide window on tissue microstructure. However, whereas NODDI and VERDICT are designed specifically for application to brain tissue and cancer tumors, respectively, DTI and DKI are more general purpose techniques that provide indices of diffusivity (c.g. mean diffusivity MD), diffusion anisotropy (c.g. fractional anisotropy [Basser &
Pierpaoli (1996)- FA), and the deviation from Gaussianity, or kurtosis, (c.g. mean kurtosis Jensen & Helpern 2010) MK) that can inform on tissue integrity or pathology. Mean signal diffusion kurtosis
imaging (MSDKI) is simplified version of DKI that quantifies kurtosis using simpler model that
is easier to fit Henriques (2018). These techniques show promise for extracting imaging biomarkers for wide variety of medical applications, such mild brain trauma, epilepsy, stroke, and Alzheimer's
disease Jensen & Helpern
2010 Ranzenberger & Snyder (022); Tae et al. (0018).
To fit the DTI, DKI, and MSDKI biophysical models to the data, and obtain the downstream metries (parameter maps), we employ the widely-used, open-source DIPY library Garyfallidis et al. (2014).
We followed standard practice for model fitting in MRI and used the least-squares optimization approach and default fitting settings. To remove outliers, values were clamped where DTI FA E [0. 1). DTI MD.AD.RD E [o, 0.003], DKI MK.AK,KK E [0, 3), MSDKI MSD E [o, 0.003] MSDKI MSK E [0, Code for model fitting is in Code Link
The results in figure 2 and table are all scaled by: DTT-FA x x102 DTI-MD x109 DTI-AD X x10", DTI-RD x109, DKI-MK x x102, DKI-AK x x10?, DKI-RK 10? MSDKI-MSD x x100 MSDKI-MSK x102
F.4 AIRBORNE VISIBLE/ INFRARED IMAGING SPECTROMETER (AVIRIS) DATA AND TASK
This section describes the data and task considered in tables 3 10
The Airbome Visible / Infrared Imaging Spectrometer (AVIRIS) is a highly-specialized hyperspectral device for earth remote sensing commissioned by the Jet Propulsion Laboratory (JPL). It obtains acquisitions from adjacent spectral channels bands between the wavelengths 400nm 2500mm. It is flown from four different aircrafts and has been deployed worldwide, for purposes such as examining the effect and rehabilitation of forests affected by large wildfires, the effect of climate change, and
25

---

Published as a conference paper at ICLR 2024
other applications in atmospheric studies and snow hydrology. More information is available Jet Propulsion Laboratory (JPL) (2023); Simmonds & Green (1996); Thompson et al. (2017) and on the
webpage http:g/ghhttpss:///ir///
jpl. nasa. gov
Data used was obtained in June 1992, when the Purdue University Agronomy Department com- missioned AVIRIS to obtain two ground images of the 'Indian Pine' support soils research Baumgardner et al. (2015) from two flight lines: east-to-west and north-to-south. This is publicly available Baumgardner et al. (2022). The data are two 'image cube' corresponding to 2miles² area of 20m² pixel size with
220 channels.
Data from the north-to-south flight are used for training and validation. This consists of 1644292 pixels of which 90-10 % were used for training-valiiattonn Data from the east-to-west flight were used for test data, which consists of 1336722 pixels. We removed outliers from both images details in Code Link and then normalized the image channel-wise so the 19th-percentil is 255 (the maximum standard images).
The objective is examine whether our supervised feature selection approaches can reconstruct the entire image from a subset wavelengths, typical of the ground obtained over Indiana (the location of "Indian Pine").
F.5 ESTIMATION OF OXYGEN SATURATION DATA AND TASK
This experiment and data follows directly from Waterhouse & Stoyanov 2022). Data was generated from the code presented in Waterhouse & Stoyanov 2022), with assistance from its author.
26

---

Published as a conference paper at ICLR 2024
192
384
576
767
1151
1343
82
123
2403
2203
3982
164
205
246
287
0
57
114
171
250
200
28
85
142
200
150
100
50
Figure 6: 2D brain slices from 3D MRI scans, for different nessureeeentsseeeuues//aamee (values of C) for the Multi-Diffusion (MUDI) challenge (top), HCP (middle) data. Bottom: Different wavelengths for the 'Indian Pine" remote sensing hyperspectral data north-to-south flight.
27


Reviews:
[{"review_id": "SHYrswpogE", "reviewer": "Reviewer_Kzed", "summary": "This paper develops an automated approach for channel subsampling and successfully applies this apporach to several MRI and hyperspectral imaging applications. The approach, TADRED, combines existing feature selection literature with recursive feature elimination (RFE). In particular, TADRED gradually reduces the number of features/channels used. Each time it reduces the feature/channel set it solves an optimization problem (3) that simultaneously trains a network to solve a task, designs a binary mask (optimized as continuous numbers) that subsamples the remaining channels, and learns a weighting matrix that weights the remaining channels fed into the network. TADRED outperforms several baselines on 6 distinct tasks.", "questions": "##  Minor comments\nThe sentence, \"e.g. the standard design for VERDICT model Panagiotaki et al. (2015a) (used as a baseline in table 1) is computed by optimizing the Fisher-matrix for one specific combination of parameter values, despite aiming to highlight contrast in those parameters throughout the entire prostate\" in related work is provided without the context of prostate cancer.\n\nThe statement \"Rather than learning the mask mt end-to-end e.g. using a sparsity term/prior, we modify elements of mt during our training procedure\" could use further clarification\n\nThe baseline methods were all published in medical imaging journals. This work might be better appreciated, and receive more informed reviews, in such a venue.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "Proposed method is validated across many distinct tasks\n\nProposed methods seems to outperform existing baselines by a small margin.\n\nSelecting a subset of informative channels is an interesting problem.", "weaknesses": "The paper could do a better job explaining the proposed method. For instance, algorithm 2 would benefit from additional comments. I have little intuition for what each step is trying to do.\n\nThe figures table captions are generally uninformative which makes the results very difficult to follow: \n-In figure 1, \"Model Free\" is uninformative.\n-Table 1 is very hard to parse. It's essentially two tables, with the top table having a different format than the bottom one.\n-Table 2 doesn't mention MRI anywhere.\n-Figure 2 is covered with arrows, but the caption doesn't state what they're pointing to. \n\n\"Feature Fill\" is not defined in the main text", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AzxQGOZPVS", "reviewer": "Reviewer_FaxL", "summary": "The paper presents a new task-specific scheme for experimental design for imaging data and application. The authors focus on expanding the design to go beyond and aim at user-specific analysis tasks at the same time. The authors proposed the TAsk-DRiven experimental design in imaging, or TADRED, to select the channel-subset while at the same time carry out the tasks.", "questions": "N/A", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The case studies are quite extensive and substantial, showing the performance of the proposed framework against a variety of applications in feature selection.", "weaknesses": "1. In the experiment section, although quite a number of baselines are used for comparison, including Fisher-matrix, the baselines are not clearly explained. Some details are included in the appendix, but it may be better to at least explain one of them in detail.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iwSaQzCrSc", "reviewer": "Reviewer_aeL1", "summary": "This paper proposes a new approach to Experimental Design in imaging tasks, where the goal is to select a restricted set of channels from multi-channel images to solve a given task (e.g. reconstruction of missing channels).\nThe approach relies on 2 networks optimized jointly: a score-network and a task network. Throughout optimization, channels are dropped at regular intervals based on the score-network's average output for these channels.\nExperiments on several datasets are then conducted to show the efficiency of the approach.", "questions": "- I see that random feature selection sometimes performs on par with other methods especially in high number of selected channels regimes. Why is that?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- **Code**: The code is present and allows reproducing the results.\n- **Experiments**: The experimental section shows impressive results and must have required a huge amount of work to benchmark all these datasets and all these methods.\n- **Prior work**: The prior work is abundantly discussed in relation to the proposed approach which makes for a great read and gives a lot of interesting context to understand the relevance of the different implementation choices.\n- **Clarity**: More generally, I would say the paper is well written and clear, with particularly well-worked figures.", "weaknesses": "- **Validation of score network**: I think a nice validation that the score network is actually working as intended would be to set random score values at the beginning of optimization of TADRED for each channel (and keep them fixed). This way, the task network is optimized for the same number of steps and in the same manner, i.e. with gradually less channels. Maybe this gradual aspect is actually making the reconstruction network better and it's why TADRED is performing better. I list this as a weakness and not as a question, because to me the experiments are lacking of proof that the score network is working as intended and not just a smart preprocessor or anything else.\n- **Experiments**: my current problem with the experimental section is that I see it in 2 folds: qMRI/EOS where TADRED outperforms state-of-the-art | others where TADRED outperforms baselines. In the others setting, I gather that there was no previous interest reported in the literature for solving the task with ED (i.e. no published results to compare against). For qMRI/EOS, I have an issue because it seems that these 2 tasks have not gathered a lot of interest either if we simply look at the citation count for the papers introducing the challenges (17 for MUDI and 5 for EOS).\n\nThis last reason is why I am not increasing my score at the moment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4XiJ4ZLn8o", "reviewer": "Reviewer_e6jZ", "summary": "This work suggests a task-driven paradigm for experimental design for real-world imaging applications, that does not require a-priori model specification and replaces high-dimensional continuous search with a subsampling problem.", "questions": "- To me it is still not so clear the difference between feature selection and experimental design optimisation.In paper you mention the difference about features being irrelevant and channels being repetitive. How do you ensure or balance this in your work? Is there a specific loss you use?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ Well written paper. \n+ Good structure for Intro, related work and methods. \n+ The problem setting and contributions are clearly communicated. \n+ Comprehensive previous work and comparison to baselines.", "weaknesses": "- Hard to follow the results and it feels like they are rashly presented. I would reiterate table 1,2 and fig 2 and explain better what are the columns rows etc. Consider taking some information from appendix to the main text. \n- Also you repeat some of the results from the beginning of section 4 afterwards in different paragraphs, which makes it hard to follow. I would reiterate here.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)