An Experiment Design Paradigm using Joint Feature Selection and Task Optimization
Authors: Stefano B Blumberg
Hongxiang Lin
Yukun Zhou
Paddy John Slator
Daniel C. Alexander
Venue: Submitted to ICLR 2023 (2023)
Keywords: Experiment Design
Populationwide Supervised Feature Selection
Quantitative Magnetic Resonance Imaging
Deep Learning

Abstract: This paper presents a subsampling-task paradigm for data-driven task-specific experiment design (ED) and a novel method in populationwide supervised feature selection (FS).  Optimal ED, the choice of sampling points under constraints of limited acquisition-time, arises in a wide variety of scientific and engineering contexts. However the continuous optimization used in classical approaches depend on a-priori parameter choices and challenging non-convex optimization landscapes.  This paper proposes to replace this strategy with a subsampling-task paradigm, analogous to populationwide supervised FS.  In particular, we introduce JOFSTO, which performs JOint Feature Selection and Task Optimization.  JOFSTO jointly optimizes two coupled networks: one for feature scoring, which provides the ED, the other for execution of a downstream task or process.  Unlike most FS problems, e.g. selecting protein expressions for classification, ED problems typically select from highly correlated globally informative candidates rather than seeking a small number of highly informative features among many uninformative features.  JOFSTO's construction efficiently identifies potentially correlated, but effective subsets and returns a trained task network.  We demonstrate the approach using parameter estimation and mapping problems in quantitative MRI, where economical ED is crucial for clinical application.  Results from simulations and empirical data show the subsampling-task paradigm strongly outperforms classical ED, and within our paradigm, JOFSTO outperforms state-of-the-art supervised FS techniques.  JOFSTO extends immediately to wider image-based ED problems and other scenarios where the design must be specified globally across large numbers of acquisitions.  Our code is available for reviewers https://www.dropbox.com/scl/fo/qe6vb1w6fuf869hx4ht0k/h?dl=0&rlkey=og8czcorurl57jbiixio7hcjt 

Under review as a conference paper at ICLR 2023
AN EXPERIMENT DESIGN PARADIGM USING JOINT FEA- TURE SELECTION AND TASK OPTIMIZATION
Anonymous authors Paper under double-blind review
ABSTRACT
This paper presents a subsampling-tast paradigm for data-driven task-specific experiment design (ED) and novel method in populationwide supervised feature selection (FS). Optimal ED, the choice of sampling points under constraints of limited acquisition-time, arises in wide variety of scientific and engineering con- texts. However the continuous optimization used in classical approaches depend on a-priori parameter choices and challenging non-convex optimization landscapes. This paper proposes to replace this strategy with a subsampling-tard Daaliinggtskkk paradigm, analogous to populationwide supervised FS. In particular, which performs JOint Feature Selection and Task Optimization. JOFSTO jointly optimizes two coupled networks: one for feature scoring, which provides the ED, the other for execution of a downstream task or process. Unlike most FS problems, e.g. selecting protein expressions for classification, ED problems typically select from highly correlated globally informative candidates rather than seeking small number of highly informative features among many uninformative features. JOF- STO's construction efficiently identifies potentially correlated, but effective subsets and returns a trained task network. Changed: We demonstrate the approach using parameter estimation and mapping problems in clinically-relevant applications in quantitative MRI and in hyperspectral imaging. Results from simulations and empirical data show the subsampling-tad paradigm strongly outperforms classical ED, and within our paradigm. JOFSTO outperforms state-of-the-aad supervised FS techniques. JOFSTO extends immediately to wider image-based based ED problems and other scenarios where the design must be specified globally across large numbers of acquisitions. Our code is available for reviewers Code 2022
introduce JOFSTO,
INTRODUCTION
Experiment design (ED) seeks optimally informative sampling scheme within budget of mea- surement time Antony (2003). The problem arises across a wide range of scientific disciplines and applications wherever mathematical models are fitted desal-world noisy measurements to estimate quantities that cannot be measured directly e.g. agriculture CanaltturrrGuptaa et al. (2015). civil engincering Lye (2002), economics Jacquemet & l'Haridon (2019), microbiology Vanot & Sergent (2005). Classical approaches [Frieden (2004); Montgomery (2001) optimize the design to minimize the uncertainty of parameter values of a prespecified model, often derived from the Fisher information matrix. For any non-linear model, these approaches require a-priori specification of model parameter values to optimize the design for, which leads to circularity, as parameter values by definition unknown at application stage. Moreover the optimization itself is usually cumbersome over a high- niigh--imessionll and highly non-convex space.
For example, quantitative imaging techniques estimate and map model parameters pixel by pixel from multi-channel images. Multiple acquisition parameters often control the contrast each channel. The ED challenge is to identify the combination of acquisition parameters that best inform the estimation of the model parameters, which vary substantially over the image. The popular MRI brain-imaging technique NODDI exemplifies the challenges: five acquisition parameters can vary for each of around 100 channels, thus the ED optimization 500 dimensional. The standard acquisition protocol was designed by optimizing the Fisher-man matrix for specific combination of parameter values, although the aim of the technique is to highlight contrast in those parameters over the extent of the brain the acquisition protocol is therefore by definition suboptimal.

---

Under review as a conference paper at ICLR 2023
This paper considers a new paradigm for ED, which cast a populationwide feature selection (FS) problem, instead of one of continuous optimization. The paradigm requires training data densely sampled over the space of possible measurements from a representative population of test cases. It identifies i) a subset of locations in the measurement space that allows high performance of a downstream task, and ii) trained network to support this task. The problem differs from most FS problems addressed by recent approaches Lee et al. (2022); Wojtas & Chen (2020) which use e.g. in protein-coding genes or noisy two-moons data, which typically aim to 'identify a small, highly discriminative subset' Kuncheva et al.] (2020). Typically in ED, each measurement individually offers similar amounts of information overall to support task performance, i.e. ED measurements are all informative, but informs different aspects of the task; ED seeks a combination that covers all important aspects. Accordingly, again unlike most FS problems, measurements are highly correlated (see figure
Thus we propose JOFSTO: an approach for joint F$ and task optimization, applied to task-driven ED. JOFSTO's 's novel architecture simultaneously trains two neural networks end-to-end; the first scores and ranks the features by relevance to optimize subsampling and the second uses a subset of measurements to perform prespecified task, such as estimating ED model parameters. JOFSTO's s subsampling coupled with downstream-suppoon network trained for optimal task performance given that design. A simple scoring mechanism enables JOFSTO to identify effective subsets of potentially highly correlated features efficiently, and joint training during subsampling maintains high task performance. Furthermore we put JOFSTO's 's novel approach to scoring and feature subsampling in recursive feature climination (RFE) framework, to reduce the full set of samples aids convergence to strong solutions.
task training outputs both an optimized ED and a
small subsample stepwise, which improves the optimization and
We demonstrate the benefits of JOFSTO within applications in quantitative (MR and hyperspectral) imaging, where the standard aim is to fit a model to multichannel measurements in each image voxel obtain informative parameters that provide e.g. biological information. MRI, acquisition time is limited by factors of cost and the ability of subjects to remain motionless in the noisy and claustrophobic environment of the scanner, thus ED is crucial to support the most accurate image-driven diagnosis, prognosis, treatment choices. In hyperspectral imaging, recovering high- quality information from the few wavelengths chosen by ED, increases acquisition speed, avoids misalignment, reduces storage requirements, and speeds up clinical adoption. Experiments, using both simulations and real-world data, show that JOFSTO outperforms classical ED and produces state-of-the-art performance in subsampling-reconstnuu estimation. Moreover, within the subsampling paradigm, JOFSTO outperforms mate-of-thhearr FS algorithms Wojtas & Chen 2020): Lee et al. (2022) on four datasets/tasks.
MRI challenge, and oxygen saturation
2 RELATED WORK
Experiment Design (ED) The design of an experiment is a set {aa,.... a²) where each a' A is a combination of under the control of the experimenter and is the number of measurements acquired. Each data acquisition under ED A provides set of measurements elements of A; x is a sample under design A. ED optimization seeks the A that maximally supports a task, such estimating parameters 0 of model fx: 9), that relates measurements, x, of system to underlying properties of interest encapsulated in 0. Classical ED approaches typically aim to minimize the expected variance of model parameters e.g. encoded in the Fisher information matrix Pukelsheim (2006), using Bayesian techniques Chaloner & Verdinelli (1995); Kaddour et al. (2020) or indirectly via an operating characteristic curve Montgomery (2001). However, for non-linen non-linear models, parameter uncertainty depends on parameter value, so the resign-optimization requires pre- specification of parameter values of interest, leading to circularity. Moreover, the optimization often becomes cumbersome particularly as model complexity increases and C' becomes large Alexander (2008). Our subsampling-tan paradigm for ED avoids this circularity and replaces the challenging continuous optimization problem with neural network training. Recent work e.g. Pizzolato et al. (2020); ; Blumberg et al. (2022): ; Waterhouse & Stoyanov (2022) raise the possibility of a subsampling approach to ED, but consider it only in specific scenarios, whilst Grussu et al. (2021) does not densely-sample the measurement and spatial domains, instead arguing that this leads to a model--rree ED. Here we uniquely couple the subsampling with specific downstream tasks optimize jointly the
settings, i.e. choices of independent variables
(a ...., ²). corresponding to the
2

---

Under review as a conference paper at ICLR 2023
execution of parameter mapping or other downstream task with identification of the design that optimally supports it.
Supervised Feature Selection (FS) FS is the well-studied problem of identifying informative subset of features Guyon et al.] 2004): : Liu & Motoda (1998), either at the instance level e.g. identifying different salient parts of different images; at the population/global level selecting across all the instances. Populationwide supervised FS is of primary relevance here, since each combination of acquisition parameters, i.e. elements of A, global many samples (image voxels).
State-of-the-art approaches Wojtas & Chen (2020); Lee et al. (2022) outperform classical approaches
e.g. (original) RFE Guyon et al. (2002), BAHSIC Song et al. (2007 2012). mRMR Peng al.
(2005), CCM Chen et al. (2017), RF Breiman (2001), DFS Li et al. (2016). LASSO Tibshirani
1996... L-Score He et al. (2005); and recently, DI.-based Abid et al. (2019): Yamada et al. (2020):
Lindenbaum et al. (2021).
Strategies for Supervised FS There are two strategies for populationwide supervised FS, which aim to 'identify a small, highly discriminative subset' Kuncheva et al. al.] (2020). The first strategy e.g. Lee et al. (2022) first exploits elements of sesk-peffrrmanc, self-supervision. additional unlabeled data, and correlated feature subsets, to score all features, then perform FS for chosen subset size C. In a second step, a network is trained on feature subset for the task. However, this step does not exploit information from all of the features assumed to be unnecessary for e.g. classification on DNA/RNA/pooeiins, or even counterproductive when a large number of nuisance features masks the structured features e.g. noisy two-moons dataset. This differs from our ED paradigm, where such a discriminative subset may not exist, as data are globally informative and e.g. has large global correlation (figure 3). A second strategy combines training for FS and task optimization, and returns an optimal feature subset of pre-given size C' and a trained task network. For example, [Wojtas & Chen (2020) has unconnected dual feature scoring and task mesk-prediction networks and a stochastic local search. However in the ED paradigm, this unnecessarily searches for the discriminative subset, which involves the NP hard problem assessing input-target functional dependence Weston et al.) (2003) and performing multiple task evaluations on different feature combinations. This may result in overfitting and long training times. Here we follow the second strategy with jointly optimized and connected dual networks, but avoid searching for the -ighl-diisrimmnative subset.
METHODS
Figure oottines JOFSTO's computational graph with the example tasks we demonstrate later. This section gives a conceptual overview of the procedure. Scction in particular Algorithm defines the details of the implementation. The operation is element-wiss and follows broadcasting rules.
Overview JOFSTO presents novel approach to supervised FS, by performing concurrent feature scoring, feature subsampling, and task pask-prediction. In an outer loop, JOFSTO considers decreasing feature set sizes, instead of e.g. subsampling all required features once, improving the optimization procedure. It also transfers information from larger feature sets to smaller ones. For a fixed step in the outer loop, JOFSTO trains a novel dual-network in an inner loop, where the networks score the features and performing the task. As training is performed alongside a novel approach in progressive feature climination, this maintains high predictive performance during subsampling.
Problem Specification Our subsampling ED paradigm requires an input data set X {x1,,
8n E
RBxC which consists of n samples and C features acquired under super-design À (a- ...., that densely-samples the measurement/featuue space. We also require a target data set V = (y1,.... Nn) E
g=*M which provides gold-standaal output of the task driving the ED; each y; is a vector of M targets (c.g. MRI model parameters) for the corresponding sample x, JOFSTO aims to subsample C' <<< C acquisition parameeer combinations, corresponding to C features, obtain an economical design A = {al ..... a) C A, which corresponds to acquiring X = (x]...,,x,] x E R where the
same C elements of each x; are in x;-
Outer Loop JOFSTO considers multiple candidates for C' across steps t = 1,... T and gradually samples from the full data set of C features to a minimum candidate of Cr i.e. the user selects hyperparameters C., t= = .... T, C C₁ > C2... > Cr. At each step f, JOFSTO aims to construct mask to subsample the features m, E {0, 1] ||me]l = Ct so X where f E C is user-chosen and, score for the features 8, E RCC which is also used to
m, X (1 m²)- X /ill
3

---

Under review as a conference paper at ICLR 2023
Computational Graph
SP
S
X
m
Tasks
X
Rax Input Data with Features
Subsampled Data
Scoring Network
T 0
T
Task Network
Train
Sample Dependent Feature Score
X
RR
limple-independent Feature Score
X
Final Feature Score
Test
St
t
T
0
00
1J
MRI Model Parameters
Mask Subsample Features
T
X
X Y
0
Backpropagation
Estimates
X
M
s
No Backpropagation
T
Figure 1: JOFSTO trains two neural networks S, T end-to-end where gradients passed through the black lines. The score s for each measurement is a combination learnt score s", and an averaged score that is not directly learnt. During training in step i, JOFSTO progressively sets to to construct 8,, and progressively constructs the mask m, to subsample the data, X, based on the lowest scores. We consider three types of tasks with distinct target output Y: mapping each sample to set of model parameters sections 4.1.1 4.2.2] reconstructing the full data set in sections 4.1.2 estimating model parameters from the reconstructed full data set in section 4.1.3.
normalize the features, and trained task network Tr, minimize Y Tx(X 8t)ll. As the trained network To is used construct T(+1. information is passed from larger feature sets to smaller ones. JOFSTO gradually reduces the feature set sizes across = 1,... (instead of c.g. subsampling all the features once), which also improve the optimization procedure. JOFTO's novel approach to scoring and feature subsampling is novel method in the RFE framework Guyon et al. (2002), itself in the backward elimination wrapper framework Kohavi & John (1997).
Computational Graph and Network JOFSTO has two neural networks: a Scoring Network S to score the features and Task Network T to perform the task, and they are trained consecutively and end-to-end, so feature scoring is concurrent to task prediction. First JOFSTO learns data-dependent score 8 (SX) which practise is computed across batches, where activation function to ensure positive scores. The final feature score s is linear combination of the learnt score s", and sample-independent score which an average of s across the entire data and a candidate for 5, This is expressed as s a² s" + (1 a²) - 8, where a E [0, 1] is set by the user. [0, 1 to subsample the measurements with X = m X (1 - m) X
E + [0, oo) is an
JOFSTO uses mask where Xfill fill E Eº is user-chosen (e.g. zeros or data median) and fills the locations of un-subsampled measurements. The Task Network T takes the weighted, subsampled measurements and estimates the target Φ = T(s X), where multiplication here allows gradients to flow end-to-end. A supervised loss is calculated L(Ý, Y), for user-chosen L, allowing computation of gradients of S,T.
Progressive Network Modification JOFSTO's construction of m,, 8, is motivated by previous work Karras al. (2018); Blumberg et al. (2019). These approaches progressively modified neural network topology across training, from learning a simpler task to a more complicated task.
Inner Loop Training In step JOFTO performs standard deep learning training in the inner loop. As DL training is on batches, but we want to learn and then train on sample-independee score, we first learn sample-depender
score s" R which is progressively set its average across the batches. This is done by by progressively setting a from 1 to 0 which sets s = s" to s = 3. Then, 3 E CC ranks the Ö features and the Ct-1 - C, lowest-scored features are chosen to be removed. m is progressively modified to have C,-1 ones and Č CC zeros, to having C, ones and Ō C, zeros during training, the dual networks adapt subsampling and maintain high task performance with
4

---

Under review as a conference paper at ICLR 2023
Table I: LeeanSSqaree--nnee Frror x 10 between estimated model parameters and ground truth model parameters used to simulate data, comparing JOFSTO with classical Fisher Matrix ED from the literature.
VERDICT
NODDI
Classic ED JOFSTO
15.0
C'=
C 99
2.04
C = 20
4.51 C 99, C = 3612
Table 2: Mean-Squeer Seaa--u-uuuurrrr -rroor x 10 between estimated model parameters and ground truth model parameters used to simulate data, comparing JOFSTO against approaches in supervised feature selection, selecting C' from C measurements
VERDICT C
220
NODDI C
3612
110
55
28
14
1806
903
452
226
Random FS + DL.
1.54
2.24
3.25
6.10
2.99
3.34
3.88
4.31
Lee et al. (2022)
1.06
1.28
1.89
4.58
2.95
3.39
3.73
4.39
Wojtas Chen (2020)
2.22
2.14
3.09
4.05
4.21
4.61
4.96
5.14
JOFSTO
1.03
1.18
1.80
2.64
2.59
2.92
3.33
3.85
continual training. Further details are in algorithm 2 At the completion of step t, we take my 5t: Te. m, 8, T i.e. figure Test.
EXPERIMENTS AND RESULTS
This section presents the experiments and results in the example applications of qMRI in section 4.] and hyperspectral imaging in section 4.2 Table F33 analyzes the impact of removing JOFSTO's components performance and examines how the random seed impacts performance.
Data Data contains densely-sampled measurements in the acquisition parameter space, produced either by scanning in-vivo human brains, via simulations using qMRI models, or from real and simulated hyperspectral images, and are visualized in section F Here, each element a of the super- design ED A produces 2D/3D grayscale image, Á correspond to 2D/3D images with C' channels. The econonomical design A then corresponds to a subset the image channels. Following standard practice in MR parameter estimation and mapping Cercignani et al. (2018); Alexander al. (2019). and hyperspectral image filter design Waterhouse Stoyanov (022)). the data samples x, RC individual voxels, thus the C features are the mcasurements/Chaamee of voxels in each measureeeentchhmmel is different, we normalize the data channel- srinnel-wise at the input and output of all neural networks for more efficient training.
As the statistical distribution
Baselines Experiments evaluate against two levels of baseline. First, to evaluate the subsampling ED paradigm in general, we compare JOFSTO to classical qMRI ED techniques using the Fisher information matrix Alexander (2008). Then, within the subsampling paradigm, we consider three baseline supervised FS methods. Firstly, random FS then deep learning (DL. training an off the- shelf, simple approach; and two baselines described in section Lee et al. (2022) - the state-of-the-an approach in supervised FS, and WWojas & Chen (2020) which, like JOFSTO has dual feature scoring and task prediction networks. Each approach conducts a brief hyperparameter search, and for fairness, the same number of evaluations used for each feature subset. We used the official code repository for the baselines when required. Further details are in supplementary section B
4.1 APPLICATION TO QUANTITATIVE MAGNETIC RESONANCE IMAGING (QMRI)
This section presents results on four experiments, to show that JOFSTO has enhanced capacity, compared to alternative approaches, for the ED problem of designing an efficient MRI acquisition scheme A underpinning clinically useful tasks. Section 4.1.1 examines learning ground truth model parameters from simulations of two widely used qMRl models, that map histologic features of tumors and complexity of neural dendrites and axons. Section 4.1.2 MRI subsampling-
5

---

Under review as a conference paper at ICLR 2023
Table 3: Mean-squared-errow between = 1344 reconstructed measurements and Ö ground-truth measurements, on same experimental settings Blumberg al. (2022).
500
250
100
40
20
Blumberg et al. 2022 Table State-of of Art
0.49
0.61
0.89
1.35
1,53
1.87
2,50
3,48
JOFSTO
0.22
0.43
0.88
1.34
1.52
1.76
2.12
2.88
Table Mean-squared-eree between = 1344 reconstructed measurements and Ò ground-truth measurements, on MUltiDifffsiin (MUDI) challenge subjects.
C 500
250
100
50
Random + DI.
0.93
1.41
2.12
5.23
Lee et al. (2022)
0.63
0.86
1.24
1.61
Wojtas Chen 2020
1.67
1.72
2.17
2.34
JOFSTO
0.21
0.44
0.94
1.34
reconstruction challenge. Section 4.1.3 focusses on estimating multiple clinically relevant downstream metries, computed by fitting qMRI models on reconstructed data.
4.1.1 JOFSTO PRODUCES BETTER PARAMETER ESTIMATES THLAN BASELINES ON SIMULATIONS
This section shows JOFSTO outperforms classical approaches in ED and in supervised FS, in setting where only simulated data accessible, to best estimate tissue-related model parameters. This demonstrates that JOFSTO can learn time-efficient qMRI protocols without any MRI scans
Recent trends qMRI Palombo et al. (2020); Gyori et (2022) use DL estimate ground truth (biophysical) model parameters. We adopt that idea with two qMRI models: VERDICT which aids early detection and classification prostate cancer Panagiotaki et al. (0115), and NODDI, which has cancerPPangiiote
been applied in Alzheimer's 's disease Kamiya et al. (2020) and multiple sclerosis Grussu et al. (2177). Both models are described in section[C] We use simulations with known ground truth parameters to evaluate the different ED approaches via accuracy of model-parameter estimation. For the classical ED approach, we use two published Fisher matrix-optimized acquisition schemes for VERDICT from Panagiotaki al. (2015b) where C 20; and NODDI from ZZhag al. (2012) where C 99. For the supervised FS approaches. we select features from -peec-sp densely-sample acquisition schemes for VERDICT (C 220) Panagiotaki al. 2015a); and NODDI (C 3612) 2) Ferizi et al. (017)). Simulated input qRI data is created with x; = Jo + noise, for MRI model J. with parameters 0: sample/voxel = 1,..., n, generated from biologically plausible range and fixed across different experiments. described in section F and Rician noise Gudbjartsson & Patz (1955)
representative of clinical qMRI. Target data are the ground truth model parameters Y =UU,....
Performance is the MSE between ground truth and learnt parameters across the voxels.
Table I compares the classical Fisher information matrix ED Alexander (2008) against the JOFSTO. where JOFSTO FS size C' is set to the number of measurements produced by the classical ED acquisition scheme. Table JOFSTO against alternative approaches in supervised FS, where
C' C C C Further details regarding the models, experimental settings, and data are in 1 T6 sections In both cases, JOFSTO outperforms the baselines.
4.1.2 JOFSTO PRODUCES SAAT--FF-TEE-ARE PERFORMANCE IN THE MUDI RECONSTRUCTION CHALLENGE
This section shows that JOFSTO obtains state-of-the-att performance, and outperforms approaches in supervised FS, in subsampling- MRI challenge.
The MICCAI
(MUDI) challenge Pizzolato et al. (020)) aimed to identify an informative subset of C' {500, 250, 100, 50} measurements from densely sampled qMRI data set with C 1344 measurements, then use the subset to reconstruct the full data set i.e. : X. This reconstruction task first provides generic challenge that tests the ability of an ED FS algorithm to identify a subset with maximal information content. The MUDI data detailed in section [F] comes from state-of-the qMRI technique that acquires multiple MRI modalities simultancously in

---

Under review as a conference paper at ICLR 2023
Table 5: Mean-squared-error between parametric maps (downstream metrics) estimated from
C' = 288 ground truth measurements and C' from C' reconstructed measurements.
DTI
DKI
MSDKI
== 18
FA
MD
AD
RD
MK
AK
RK
MSD
MSK
Random FS DL
2.23
6.09
22.7
6.97
9.03
7.83
15.3
6.82
7.59
Lee et al. (002))
2.86
12.9
31.2
14.9
12.0
9.26
20.3
8.96
8.17
Wojtas & Chen 20220)
9.83
23.2
77.7
26.8
11.9
10.9
21.3
10.8
6.03
JOFSTO
1.29
2.55
13,4
2.60
7.67
6.73
13.9
6.37
4.94
Qualitative comparison of 2D slices from 3D parametric maps (downstream metrics), where gold standard is estimated from C = 288 ground truth measurements, and baseline/ours is from C 18 reconstructed measurements. JOFSTO's parametric maps visually closer to the gold standard than those from the best performing baseline. Baseline
Figure 2:
Gold Standard
Ours
FA
64
MK
RK
high-dimensional acquisition parameter space where R6, underscoring the importance ED, as adequately sampling this high-dimensional measurement space in time budget that is realistic in clinical settings is difficult Slator et al. (2021). We consider two experiments that highlight the benefits of JOFSTO in this paradigm.
The first experiment follows the setting in Blumberg et al. (002)), which has slate-of-the-ee art perfor- mance the data, table 3shows results, confirming that JOFSTO improves over the state-of-the-an The second experiment compares JOFSTO with the tfate-of-the-es techniques supervised FS in the settings of the original challenge, and table shows that JOFSTO outperforms these techniques.
4.1.3 JOFSTO IMPROVES ESTIMATION OF TISSUE MICROSTRUCTURE FROM QMRI SCANS
This section shows JOFSTO outperforms approaches in supervised FS for the ED task of designing an efficient acquisition scheme to support reconstruction of a densely-sampler dataset and downstream in-vivo quantification of human tissue microstructure.
7

---

Under review as a conference paper at ICLR 2023
Table 6: Mean-squared-error between C = 220 reconstructed measurements and C ground-truth measurements, on Indian Pine AVIRIS hyperspeetral data.
C 110
55
28
14
Random FS + DL
1.22
1.76
2.91
5.61
Lee et al. (2022)
1.36
3.11
3.77
7.61
Wojtas Chen 2020)
6.34
6.68
7.16
7.78
JOFSTO
0.60
1.42
2.33
4.49
Table 7: -ar-M-aann-uueeeeeeeeee 100 between the predicted and ground truth oxygenation (abundance maps), experimental settings Waterhouse & Stoyanov (2022)
C==6
2
Waterhouse & Stoyanov (2022) Fig. 5a Best
4.54
4.91
5.33
6.17
7.55
JOFSTO
3.22
3.61
4.64
4.94
5.18
DTI Basser al. (1994). Jensen & Helpern (2010). and MSDKI are widely-used qMRI methods, described in detail in section D that can quantify tissue microstructure, and show promise for extracting imaging biomarkers for many medical applications, such mild brain trauma, epilepsy, stroke, and Alzheimer's 's disease |Ranzenberger & Snyder (2022); Tae et al. (2018) Jensen & Helpern (2010). This experiment uses rich, high-resolution HCP data detailed section with C = 288. The
C task is to subsample feature set sizes C = E. 16 then reconstruct the data, where the models are then fitted using standard techniques detailed in section D Therefore the aim is reduce the acquisition requirements for these imaging techniques, thereby enabling their usage in a wider range of clinical application areas. Quantitative and qualitative results are in table suupllemmntarris-tain where the JOFSTO outperforms the baselines quantitatively in 17/18 comparisons on clinically useful downstream metries (parameter maps) and JOFSTO's parametric maps are visually closer to the gold standard than those from the best baseline.
figure 2
4.2 APPLICATION TO HYPERSPECTRAL. IMAGING
Hyperspectral imaging (a.k.a. imaging spectroscopy) obtains pixel-wise information across multi- ple wavelengths of the electromagnetic spectrum from specialized hardware, recovering detailed information of the examined abject-of-intent Manolakis et al. (2016). For example, the JPL's Airborne Visible / Infrared Imaging Spectrometer (AVIRIS) Thompssnn et al. (2017) remotely senses elements of the Earth's 's atmosphere and surface from airplanes, and has been used to examine the effect and rehabilitation of forests affected by large wildfires. This produces an 'image cube' 2D spatial image with multiple channels, where, similar to qMRI, the image channels correspond to the measurements. In this paradigm, ED consists of choosing the wavelengths and/or filters Arad & Ben-Shahar ()017); Waterhouse & Stoyanov (2022); Wu et al. (2019), which controls the number of channels. The aim is to obtain high-quality information from only few measurements (correspond- ing image channels), which can reduce the cost of current acquisitions or spur the development of more economical hardware Stuart et al. (2020). We conduct two experiments with hyperspectral data, subsampling then reconstructing AVIRIS images, and estimating oxygen saturation (a model parameter-estimation task).
4.2. JOFSTO IMPROVES ON
HYPERSPECTRAL IMAGES
This experiment follows the approach in section 4.1.2 and examines a
task on mublicy--vvilable high-quality AVIRIS ground images. To support soils research, Purdue University Agronomy obtained AVIRIS data from north-to-south flight, dubbed the 'Indian Pine' dataset Baumggrdner et al. (2015), where C' 220 corresponds to the number of wavelengths. Results in table 6 show that JOFSTO improves over state-of-the-art approaches in supervised FS, for the C C G C
subsample sizes C
8

---

Under review as a conference paper at ICLR 2023
4.2.2 JOFSTO IMPROVES THE ESTIMATION OF OXYGEN SATURATION
This experiment in the setting of Waterhouse Stoyanov 2022). This examines how to design spectral filter sets to allow the estimation of oxygen saturation via obtaining non-invasive and non-destructive hyperspectral images, via the model in Can & Ülgen (011)) In this setting, the measurements are obtained from filter applied to wavelength (center), producing C 4 87 measurements, and the target is 2-dimensional i.c. M 2. Results in table 7 show that JOFSTO outperforms the six approaches in Waterhouse & Stoyanov (2022) for feature sizes C = 6,5,4,
5 DISCUSSION
This paper proposes to replace classic approaches in ED with a subsampling-task paradigm analogous to populationwide supervised FS, thereby avoiding complications in continuous optimization and dependence of the design a-priori model parameter choices. We introduce JOFSTO, which performs JOint Feature Selection and Task Optimization, that takes advantage of the structure of ED data. Experiments address the clinically relevant problem of obtaining high-quality microstructural information in the limited time available in MRI scans, and hyperspectral imaging. Results show JOFSTO outperforms classical ED and ofte-of-the-art approaches in populationwide supervised FS. JOFSTO also obtains state-of-the-ee performance in a subsampling-recomee reconstruction MRI challenge and in oxygen saturation. The code is available to reviewers Code 2022).
Our paradigm assumes we can obtain densely-sampler data in the measurement space. This is possible in MRI, as only few subjects are required to provide densely-sampless training data. They may be e.g. sedated, and remain motionless for the long acquisition times required. Whilst such data acquirable in z 1 hour research studies, in extreme cases motivated researchers endured longer, e.g. 8 hours in Ferizi et al. (2017). Similarly, in hyperspectral imaging, expensive devices can acquire large numbers of images with different spectral sensitivity simultaneously to provide training data for the design of much cheaper deployable devices with a small but informative set of sensitivities for particular applications. The same paradigm is applicable in other imaging problems and beyond, e.g. studies of cell populations Sinkoe & Hahn| (2017), or questionnaire design [Bllio et al. (2020). In neuropsychological evaluation for example, multiple questions have highly correlated answers and a key challenge is to identify multiple small combinations of questions that can evaluate c.g. dementia patients quickly and variably to avoid practice effects. Small dense data sets are also achievable in other ED scenarios, although the strategy may not extend to all. Future work will study the size and diversity of densely sampled data needed to ensure strong generalizable design.
Unlike alternative approaches in supervised FS e.g. [ee et al. (2022); Wojtas & Chen (2020). JOFSTO does not purport to consistently identify a small, highly discriminative feature subset Kuncheva et al. (2020); it is not designed to ignore large number of nuisance features masking the variability of structured features Lindenbaum et al. (202T) or to explicitly integrate the correlation structure between feature subsets for efficiency| Lee et al. (2022). This is an advantage in the ED paradigm, but means that JOFSTO may perform less well supervised FS tasks with these requirements, c.g. classification on some (but not all) proteins/DNA/RNN datasets, or identifying structured features of the synthetic noisy two-moon dataset. As JOFSTO uses DL, it is likely to overfit when the number of samples n is limited, or TI is less than the number of selected features C' Kuncheva et al. (2020); Wojtas & Chen (2000). This scenario is frequent in e.g. genomics and could potentially occur even in imaging tasks if the diversity of signals over the image is low or the image is small e.g. in Ferizi ct al. (2017). JOFSTO's use of RFE is to aid the challenging optimization by reducing to stepwise consideration of nested feature subsets across steps f L..., T. However, this decreases the upper bound on performance the optimal feature set for C. may not be subset of the optimal set for Cz-1; future work will consider alternative optimization strategies to seek globally optimal designs.
ETHICS STATEMENT
This paper highlights direct impact of our novel approach to ED applied to MRI and hyperspectral imaging. By selecting few acquisition parameters which may be obtained in routine MRI scans, and performing a task that provides improved visual or microstructural information, clinician may better analyze and diagnose a variety of pathologies, potentially improving public health. This of course depends the specific acquisition platforms and site, which varies sean

---

Under review as a conference paper at ICLR 2023
performance. Thus results should be thoroughly verified by clinical experts before deployment. This paper also used vivo human brain MRI scans and followed the respective guidelines. where the authors of the cited, respective papers completed appropriate ethical reviews.
REPRODUCIBILITY
The code for our method in this paper will be publicly available, and is submitted to the reviewers Code 2022). All data are publicly available. Data acquired from MRI are described in section F including processing, splits, and download links. Code to generate simulated data is in the code repository. Settings the JOFSTO and baseline methods are in section Details on MRI model fitting is in section D and uses publicly available libraries.
REFERENCES
Abubakar Abid, Muhammed Fatih Balin, and James Zou. Concrete autoencoders: Differentiable feature selection and reconstruction. In: International Conference on Machine Learning (ICML), 2019.
Daniel C. Alexander. A general framework for experiment design in diffusion MRI and its application in measuring direct cussue-microstruutuur features. Magnetic resonance in medicine, 60(2):439 448, 2008.
Daniel C. Alexander, Tim B. Dyrby, Markus Nilsson, and Hui Zhang. Imaging brain microstructure with diffusion MRI: practicality and applications. NMR in Biomedicine, 32(4):c3841, 2019.
Fidel Alfaro-Almagro, Mark Jenkinson, Neal K. Bangerter, Jesper L. R. Andersson, Ludovica Grif- fanti, Gwenaëlle Douaud, Stamatios N. Sotiropoulos, Saad Jbabdi, Moises Bernandez-Fermandee, Emmanuel Vallee, Diego Vidaurre, Matthew Webster, Paul McCarthy, Christopher Rorden, Alessan- dro Daducci, Daniel C. Alexander, Hui Zhang, Iulius Dragonu, Paul M. Matthews, Karla L. Miller, and Stephen M. Smith. Image processing and quality control for the first 10,000 brain imaging datasets from UK biobank. Neurolmage, 166:400-422 424,,444445 2018.
Jiju Antony. Design of Experiments for Engineers and Scientists. Oxford: Hew.errmtn--ennmmannnn. 2003.
Boaz Arad and Ohad Ben-Shahar. Filter selection for hyperspectral estimation. In: International Conference on Computer Vision (ICCV), 2017.
P.J. Basser, Mattiello, and D. LeBihan. MR diffusion tensor spectroscopy and imaging, Biophysical journal, 66(1):259-267, 1994.
Peter J. Basser and Carlo Pierpaoli. Microstructural and physiological features of tissues elucidated by citiiiatiii-diffuiion tensor MRI. Journal of Magnetic Resonance, 11 11:209-219, 1996.
M. F. Baumgardner, L. L. Bichl, and D. A. Landgrebe. 220 band aviris hyperspectral image data set: June 12, 1992 indian pine test site (north-south image). Purdue University Research Repository doi: /1..443///72299/1/ 2015.
Maura Bellio, Neil P Oxtoby, Zuzana Walker, Susie Henley, Annemie Ribbens, Ann Blandford, Daniel C Alexander, and Keir X X Yong. Analyzing large alzheimer's ' disease cognitive datasets: Considerations and challenges. Alzheimer's Dementia: Diagnosis, Assessment Disease Monitor- ing. 12(1), 2020.
Stefano B. Blumberg, Marco Palombo, Can Son Khoo, Chantal M. W. Tax, Ryutaro Tanno, and Daniel C. Alexander. Multi-stage prediction networks for data harmonization. In: Medical Image Computing and Computer Assisted Intervention (MICCAI), 2019.
Stefano B. Blumberg, Hongxiang Lin, Francesco Grussu, Yukun Zhou, Matteo Figini, and Daniel C. Alexander. Progressive subsampling for oversampled data - application to quantitative MRI. In: Medical Image Computing and Computer Assisted Intervention (MICCAI), 2022.
Leo Breiman. Random forests. Machine Learning, 45:5-32, 2001.
10

---

Under review as a conference paper at ICLR 2023
Osman Melih Can and Yekta Ülgen. Modeling diffuse reflectance spectra of donated blood with their hematological parameters. Clinical and Preclinical Optical Diagnostics I1, 2019.
Mara Cercignani, Nicholas G. Dowell, and Paul S. Tofts. Quantitative MRI of the Brain: Principles of Physical Measurement. CRC Press, second edition, 2018.
Kathryn Chaloner and Isabella Verdinelli. Bayesian experimental design: A review. Statistical Science, 10(3):273-304, 1993.
Jianbo Chen, Mitchell Stern, Martin J. Wainwright, and Michael I. Jordan. Kernel feature selection conditional covariance minimization. In: Neural Information Processing Systems (NIPS), 2017.
Paper Code. Code for this paper. 2022.
David C. Van Essen, Stephen M. Smith, Deanna M. Barch, Timothy E. Behrens, Essa Yacoub, Kamil Ugurbil, and WU-Minn HCP Consortium. The WU-Minn Human Connectome Project: an overview. Neuroimage, 80:62-79, 2013.
Uran Ferizi, Benoit Scherrer, Torben Schneider, Mohammad Alipoor, Odin Eufracio, Rutger H. Fick, Rachid Deriche, Markus Nilsson, Ana K. Loya-Olivas, Mariano Rivera, Dirk H. J. Poot, Alonso Ramirez-Manzanares, Jose L. Marroquin, Ariel Rokem, Christian Pötter, Robert F. Dougherty, Ken Sakaie, Claudia Wheeler-Kingee Kingshott, Simon K. Warfield, Thomas Witzel, Lawrence L. Wald, José G Raya, and Daniel C. Alexander. Diffusion MRI microstructure models with in vivo human brain connectome data: results from multi-group comparison. NMR in biomedicine, 30(9), 2017.
Rutger Fick, Demian Wassermann, and Rachid Deriche. The Dmipy toolbox: Diffusion MRI multi- compartment modeling and microstructure recovery made easy. Frontiers in Neuroinformatics, 13 (64), 2019.
B. Roy Frieden. Science from Fisher information: unification. Cambridge University Press, 2004.
Eleftherios Garyfallidis, Matthew Brett, Bagrat Amirbekian, Ariel Rokem, Stefan van der Walt, Maxime Descoteaux, Ian Nimmo-Smith, and Dipy Contributors. DIPY, library for the analysis of diffusion MRI data. Frontiers Neuroinformaatis, 8(8), 2014.
Francesco Grussu, Torben Schneider, Carmen Tur, Richard L. Yates, Mohamed Tachrount, Andrada lanuş, Marios C. Yiannakas, Jia Newcombe, Hui Zhang. Daniel C. Alexander, Gabriele C. DeLuca, and Claudia A. M. Gandini Wheeler- Kingshott. Neurite dispersion: new marker of multiple sclerosis spinal cord pathology? Annals of Clinical and Translational Neurology, 4(9), 2017.
Francesco Grussu, Stefano B. Blumberg, Marco Battiston, Lebina Lebina S. Kakkar, Hongxiang Lin, Andrada lanuş, Torben Schneider, Saurabh Singh, Roger Bourne, Shonit Punwani, David Atkinson, Claudia A. M. Gandini Wheeler-Kingg Kingshott, Eleftheria Panagiotaki, Thomy Mertzanidou, and Daniel C. Alexander. Feasibility of data-driven, model-free quantitative MRI protocol design: Application to brain and prostate iiffusiionrelaxattin imaging. Frontiers in Physics, 9:615, 2021.
Hákon Gudbjartsson and Samuel Patz. The rician distribution of noisy MRI data. Magnetic Resonance in Medicine, 34(6):910-9) 91441,, 1995.
V. K. Gupta, Rajender Parsad, and Baidya N. Mandal. Significance of Experimental Designs in Agricultural Research. Indian Agricultural Statistics Research Institute (IASRI), 2015.
Isabelle Guyon, Jason Weston, Stephen Barnhill, and Vladimir Vapnik. Gene selection for cancer classification using support vector machines. Journal of Machine Learning Research, 46(1): 389-422, 2002.
Isabelle Guyon, Steve Gunn, Asa Ben-Hur, and Gideon Dror. Result analysis of the NIPS 2003 feature selection challenge. In: Neural Information Processing System (NIPS), 2004.
Noemi G. Gyori, Mareo Palombo, Christopher A. Clark, Hui Zhang, and Daniel C. Alexander. Training data distribution significantly impacts the estimation tissue microstructure with machine learning. Magnetic Resonance in Medicine, 87(2):932-944 2022.
11

---

Under review as a conference paper at ICLR 2023
Xiaofei He, Deng Cai, and Partha Niyogi. Laplacian for feature selection. In: Neural Information Processing Systems (NIPS), 2005.
Rafael Neto Henriques. Advanced methods for diffusion MRI data analysis and their application to the healthy ageing brain. Ph.D Thesis, 2018.
Jana Hutter, Paddy J. Slator, Daan Christiaens, Rui Pedro Teixeira, Thomas Roberts, Laurence Jackson, Anthony N. Price, Shaihan Malik, and Joseph Hajnal. Integrated and efficient diffusion- relaxometry using zebra. Scientific reports, 8(1):1-13, 2018.
Nicolas Jacquemet and Olivier L'Haridon. Experimental Economics Method and Applications. Cambridge University Press, 2019.
Jens H. Jensen and Joseph A. Helpemn MRI quantification of non-Gaussian water diffusion by kurtosis analysis. NMR in Biomedicine, 23(7):698-710, 2010.
Edward W. Johnston, Elisenda Bonet-Came, Uran Ferizi, Ben Yvernault, Hayley Pye, Dominic Patel, Joey Clemente, Wivijin Piga, Susan Heavey, Harbir Sidhu, Francesco Giganti, James O'Callaghan, Mrishta Brizmohun Appayya, Alistair Grey, Alexandra Saborowska, Sebastien Ourselin, David Hawkes, Caroline M. Moore, Mark Emberton, Hashim U. Ahmed, Hayley Whitaker, Manuel Rodriguez-Justo, Alexander Freeman, David Atkinson, Daniel Alexander, Eleftheria Panagiotaki, and Shonit Punwani. VERDICT MRI for prostate cancer: Intracellular volume fraction versus apparent diffusion coefficient. Radiology, 291(2):391-399 2019.
Jean Kaddour, Steindór Sæemundsoon, and Mare Peter Deisenroth. Probabilistic active meta-learning. In: Neural Information Processing Systems (NeurIPS), 2020.
Kouhei Kamiya, Masaaki Hori, and Shigeki Aoki. NODDI in clinical research. Journal of Neuro- science Methods, 346: 108908, 2020. ISSN 0165-0270.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of GANs for im- proved quality, stability, and variation. In: International Conference on Learning Representations (ICLR), 2018.
Ron Kohavi and George H. John. Wrappers for feature subset selection. Artificial Intelligence, 97(1): 273-324, 1997.
Ludmila I. Kuncheva, Clare E. Matthews, Álvar Arnaiz-González, and Juan José Rodríguez Diez. Feature selection from high-dimensional data with very low sample size: A cautionary tale. arXiv preprint arXiv:2008. /2025, 2020.
Changhee Lec.
Code
for self-supervision enhanced feature https: thub. com/ch18856//EF3 2022.
selection git
with commit
correlated gates.
Changhee Lee, Fergus Imrie, and Mihaela van der Schaar. Self-supervision enhanced feature selection with correlated gates. International Conference on Learning Representations (ICLR), 2022.
Yifeng Li, Chih Yu Chen, and Wyeth W. Wasserman. Deep feature selection: Theory and application to identify enhancers and promoters. Journal of Computational Biology, 23(5):322-336, 2016.
Ofir Lindenbaum, Uri Shaham, Erez Peterfreund, Jonathan Svirsky, Nicolas Casey, and Yuval Kluger. Differentiable unsupervised feature selection based gated laplacian. In: Neural Information Processing Systems (NeurIPS), 2021.
Huan Liu and Hiroshi Motoda. Feature Selection for Knowledge Discovery and Data Mining. The Springer International Series in Engincering and Computer Science. Springer, 1998.
Leonard Lye. Design of experiments in civil engineering: are we still in the 1920's ? Dekker New York, 2002.
Dimitris G. Manolakis, Ronald B. Lockwood, and Thomas W. Cooley. Hyperspectral Imaging Remote Sensing: Physics, Sensors, and Algorithmss. Cambridge University Press, 2016.
12

---

Under review as a conference paper at ICLR 2023
Douglas C. Montgomery. Design and analysis of experiments. John Wiley & Sons, fifth edition, 2001.
MUDI Organizers. MUlti-dimensional DIffusion (MUDI) MRI challenge 2019 data. https 7 7/www. www..eeeeloonnnggaiii.coo htt.:p/...g..paaa........aaaaaa. 2022.
Marco Palombo, Andrada lanuş, Michele Guerreri, Daniel Nunes, Daniel C. Alexander, Noam Shemesh, and Hui Zhang. SANDI: A compartment- based model for non-invasive apparent soma and neurite imaging by diffusion MRI. Neurolmage, 215: 16835, 2020.
Eleftheria Panagiotaki, Rachel W. Chan, Nikolaos Dikaios, Hashim U. Ahmed, James O' O'Callaghan. Alex Freeman, David Atkinson, Shonit Punwani, David J. Hawkes, and Daniel C. Alexander. Microstructural characterization of normal and malignant human prostate tissue with vascular, extracellular, and restricted diffusion for cytometry in tumours magnetic resonance imaging. Investigative Radiology. 50(4):218-227, 2015a.
Eleftheria Panagiotaki, Andrada Ianuş, Edward Johnston, Rachel W. Chan, Nicola Stevens, David Atkinson, Shonit Punwani, David J. Hawkes, and Daniel C. Alexander. Optimised VERDICT MRI protocol for prostate cancer characterisation. In: International Society for Magnetic Resonance in Medicine (ISMRM), 2015b.
Eletheria Panagiotaki. Simon Walker-Samuel, Bernard Siow, Peter Johnson, Vineeth Rajkumar, Barbara R. Pedley, Mark F. Lythgoe, and Daniel C. Alexander. Noninvasive quantification of solid tumor microstructure using VERDICT MRI. Cancer research, 74(7):1902-1912, 2014.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach De Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. PyTorch: An imperative style, peefformmfm deep learning library. In: Neural Information Processing Systems (NeurIPS), 2019.
Hanchuan Peng, Fuhui Long, and Chris Ding. Feature selection based on mutual information: criteria of max-dependency, max-relevance, and min-redundancy. IEEE Transactions on Pattern Analysis and Machine Intelligence 27(6):1226-123 2005.
Marco Pizzolato, Marco Palombo, Elisenda Bonet-Carne, Francesco Grussu, Andrada lanuş, Fabian Bogusz, Tomasz Pieciak, Lipeng Ning. Stefano Blumberg, Thomy Mertzanidou, Daniel C. Alexander. Maryam Afzali, Santiago Aja-Feráández, Derek K. Jones, Carl-Fredrik Westin, Yogesh Rathi, Steven H. Baete, Lucilio Handero-Grande Thilo Ladner, Paddy J. Slator, Daan Christi- aens, Jhil-Philippe Thiran, Anthony N. Price, Farshid Sepehrband, Fan Zhang, and Jana Hutter. Acquiring and predicting MUlti-dimensional DIffusion (MUDI) data: an open challenge. In: International Society for Magnetic Resonance in Medicine (ISMRM), 2020.
Friedrich Pukelsheim. Optimal Design of Experiments. Society for Industrial and Applied Mathemat- ics, 2006.
Logan R. Ranzenberger and Travis Snyder. Diffusion Tensor Imaging. StatPearls Publishing, fifth edition, 2022.
Saurabh Singh, Harriet Rogers, Baris Kanber, Joey Clemente, Hayley Pye, Edward W. Johnston, Tom Parry, Alistair Grey, Eoin Dinneen, Greg Shaw, Susan Heavey, Urszula Stopka-Farooqui, Aiman Haider, Alex Freeman, Francesco Giganti, David Atkinson, Caroline M. Moore, Hayley C. Whitaker, Daniel C. Alexander, Eleftheria Panagiotaki. and Shonit Punwani. Avoiding unnecessary biopsy after multiparametrie prostate MRI with VERDICT analysis: The INNOVATE study. Radiology, pp. 21 212536, 2022.
Andrew Sinkoe and Juergen Hahn. Optimal experimental design for parameter estimation of an il-6 signaling model. Processes, 5(3), 2017.
Paddy J. Slator, Marco Palombo, Karla L. Miller, Furl-Fredrik Westin, Frederik Laun, Daeun Kim, Justin P. Haldar, Dan Benjamini, Gregory Lemberskiy, Joao P. de Almeida Martins, and Jana Hutter. Combined iiffusionn-elaxomeett microstructure imaging: Current status and future prospects. Magnetic Resonance in Medicine, 86(6))29873011, 2021.
13

---

Under review as a conference paper at ICLR 2023
Le Song, Alexander J. Smola, Arthur Gretton, Justin Bedo, and Karsten M. Borgwardt. Supervised feature selection via dependence estimation, In: International Conference on Machine learning (ICML), 2007.
Le Song, Alexander J. Smola, Arthur Gretton, Justin Bedo, and Karsten M. Borgwardt. Feature selection via dependence maximization. Journal of Machine Learning Research, 13(5):13331144, 2012.
Mary B. Stuart, Leigh Stanger, Matthew Hobbs, Tom D. Pering, Daniel Thio, Andrew J.S. McGonigle, and Jon R. Willmott. Low-cost hyperspectral imaging system: Design and testing for laboratory-based environmental applications. Sensors, 20(1 1):3239, 2020.
Woo Suk Tae, Byung Joo Ham, Sung Bom Pyun, Shin Hyuk Kang, and Byung Jo Kim. Current clinical applications of diffusion-tensor imaging in neurological disorders. Journal of Clinical Neurology, 14(2):129-140, 2018.
David R. Thompson, Joseph W. Boardman, Michael L. Eastwood, and Robert O. Green. A large airborne survey of earth's visible-infrared spectral dimensionality. Optics Express, 25(8):9186 9195, 2017.
Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistics Society. Series B (Methodological), pp. 267-288, 1996.
Guillaume E. Vanot and Michelle Sergent. Experimental Design in Microbiology. Humana Press, 2005.
Dale Waterhouse and Danail Stoyanov. Optimized spectral filter design enables more accurate estimation of oxygen saturation in spectral imaging. Biomedical Optics Express, 13(4):2156-217) 2022.
Jason Weston, André Elisseeff, Bernhard Schölkopf, and Michael E. Tipping. Use of the zero-norm with linear models and kernel methods. Journal of Machine Learning Research, 2003.
Maksymilian A. Wojtas. Code for feature importance ranking for deep learning, git com- mit http://cceeeoeeeeeeeeaaeececeee1 Feature 2021.
https: com com/maksym33
Maksymilian A. Wojtas and Ke Chen. Feature importance ranking for deep learning. In: Neural Information Processing System (NeurIPS), 2020.
Renjie Wu, Yuqi Li, Xijiong Xie, and Zhijie Lin. Optimized multi-spectral filter arrays for spectral reconstruction. Sensors, 19(13), 2019.
Yutaro Yamada, Ofir Lindenbaum, Sahand Negahban, and Yuval Kluger. Feature selection using stochastic gates. In: International Conference on Machine Learning (ICML), 2020.
Hui Zhang, Torben Schneider, Claudia A. Wheeler- Wheeler-Kingshott Kingshott, and Daniel C Alexander. NODDI: practical vivo neurite orientation dispersion and density imaging of the human brain. Neurolmage, 6000):10001111, 6, 2012.
14

---

Under review as a conference paper at ICLR 2023
A JOFSTO's S OPTIMIZATION ALGORITHM
This section presents further details concerning JOFSTO's optimization algorithm, outlined in section 3] We first describe the computational graph, illustrated in figure [ We then describe the optimization procedure in full, described in algorithm 2 For steps t 1, I.... 7 this procedure obtains m4; 84,81, Th which are m, T at the completion of step I. Multiplication follows standard array broadcasting rules e.g. Z1 E IRd Z2 E Raxd, then 2₁ Zz E R is element-wise multiplication broadcasted to the first dimension.
Computational Graph This describes JOFSTO's computational graph during training, illustrated in figure toop and described in algorithm | JOFSTO has two neural networks: a Scoring Network S and a Task Network T. The final feature score is linear combination of learnt score s = (X)) and sample-independent (averaged) score S, expressed as s = a s"
-a))=
where s E R=xC_= E R²,, and a* E [o, 1] is not learnt but used to progressively set s = 8, and a: R + [0, oo) is an activation function to ensure positive scores. JOFSTO uses a mask m E [o, 1j F
X + (1 (1-m)- - fill where X fill E RC is user-chosen (e.g. zeros or data median) and fills the locations of un-subsampled measurements. The second neural network, the Task Network T takes the weighted, subsampled measurements and estimates the target T= X), where multiplication here allows gradients to flow end-to-end. A supervised loss is calculated L(Y, Y), for user-chosen allowing computation of S, T gradients.
to subsample the measurements with X m
Recursive Feature Elimination (RFE) The user prespecifies the number of features to consider across steps I.... T: C1 .... Cr where C C1 ... Cr, so C,-1 - C. features are removed in step t > 2. In step t JOFSTO uses its novel approach to score features with 8,, and uses m, to subsample the features, within a RFE framework.
Step t= On the first step JOFSTO trains on full information and learns a sample-independe score. = []]c, x* 1 and trains across epochs 1,... E. At completion, set the first
It sets sample-independent score 81 = where 5" the mean of 8 across samples/batches.
Steps t = 2,...,T For subsequent steps, JOFSTO splits the inner loop training procedure <= FA Eg E3 < E for hyperparameters epochs Et, E2, Ez.
1. Across e = 1,..., F JOFSTO learns a cumple-independent score for each measurement. trains on combination of (previous) nample-independent score and current learnt score, and at Eg it sets the cumple-independer score as the mean of learnt scores. This is 5 = 5 a 1.4 m
m4-1, at e Ej set s = (5 + s*) where s is the mean of s" across samples/batches.
Across e E1 + 1, .... Eg JOFSTO progressively sets the score to be sample-independent, by progressively setting a" from to 0 successively these epochs, allowing training to go from sample-dependent (s = sP) to sample-independent scores (s = 8).
3. Across e Fog + 1,, 1,... F3 JOFSTO chooses the lowest-scored measurements subsample, then progressively modifies mask elements to perform this subsampling. At epoch e = Fg choose the measurements subsample D. arg minc,_.-c { m [i] 1}. with mask target mi = m D. Construct the mask progressively with m t -m-1 + (1 1 am)) m where at [0, 1] is set from 1 to 0 across these epochs.
Across E = Eg + -,,..,, E the Task Network T trained. At the completion of step f, this process returns (trained) T and m, 8, denoted as Te, ma 8t. Then use as ÿ = Te(X 5.) as figure Test.
Using JOFSTO For C, acquisition -arrrittoon-paaameter combinations, use A = {ar : m 1, .... C') to obtain new data, where a" e A produces the measurement 2f z Then one fills
1. c
un-acquired measurements with X fill by setting X = (x) if m[c] = 1, else X fi
[c) producing
X = Then estimate Y with Ý Tx(X 5z), and compute downstream metric on Y.
Implementation JOFSTO's code is in Torch Paszke et al. (2019 and is submitted anonymously
Code 2022). We describe its hyperparameters in section
15

---

Under review as a conference paper at ICLR 2023
Algorithm JOFSTO Forward and Backwards Pass: FPBP(
Input Input Data X & RaxC, Target Data Y, Subsampled Data Fill X Scoring Network S, Task
Network Final Feature Score Multiplier a* E [0, 1], Mask [0. 1
1: 8º (x)(
Learn Gample-Dependent Feature Score
2: s = a² -"
- + (1 a²) 3: X m m-X+
Final Feature Score
m)
a Measurement Subsampling with Mask
4: Y = T(s.X)
D Perform the Task Calculate Supervised Loss 6: Backpropagate on I,, compute gradients of S, weights, update weights of S, T
5: I. Y(Y.Y)
Algorithm 2 JOFSTO Optimization
Input Input Data X E _ Target Data Y. Subsampled Data Fill Xfill Feature Size Candidates
and RFE Steps C = C >
> Cr, Training Steps <= Fạ
E2
Eg E
1: for + do
4- m1 [1]0 a² +-
2:
3:
for - ...... E do
D. Train on Full Information
4: FPBP(
5: end for
6: Calculate 3" the mean learnt score of across data
7:
$1 + 52
b First Sample-Independen Score
8:
end for
9:
for t
+ 2, .... T do
4 81 a² f- 2: m +-
m4-1
11: for 4-1 1, I.... Ei do
Joint Optimization
12:
FPBP(
13: end for
14:
Calculate 3" the mean learnt score of s across data
15:
5t 2 (5 8*)
Sample-Independent Score for step t
16:
for e - E + 1,.., Eg do
a + maxfa" ET-EI:01 0}
Progressively Set the Score be Sample-Independent
18:
FPBP( end for
19:
D. + arg min
[ 1--11 13
Features Subsample Mask Target
21:
m. m4-1 LieD
22:
am 1
23:
for
Es + 1,,,, Eg do
24:
an + max(a"
E3 Eg-Ey 0}
25:
+- th mp-1 (1 mm) mi FPBP(
Progressively Construct the Mask
26:
27:
end for
28:
for e t-
Eg + 1,... do FPBP()
29:
30:
end for
31:
Cache Tr, my St, then
& T((X 8t)
See figure Test
32:
end for
16

---

10, Under review as a conference paper at ICLR 2023
B SETTINGS FOR DIFFERENT APPROACHES
This section describes the settings for the different approaches used in this paper.
We used the official code for all baselines when required, following the implementation carefully. For every approach, for fairness, we considered the same number of evaluations on the validation data, to choose the best model apply to the test data, for each different feature set size C. Data are split in training, validation and test sets, described in section F We normalize the data following Grussu al. (2021), which divides each channel/measureeeet by 99th percentile value calculated from the training set, the input and output of the neural network are normalized by this value. Other general hyperparameters are batch size 1500, learning rate 10 (10 in section 4.1.1) and ADAM optimizer, default network weight initialization.
OURS: JOFSTO
We conducted brief search for the hyperparameters, setting E1 = 25, E2 +0, = Es
we do not set E, as like other approaches. we train on Ez,... .... until early stopping on the validation set. We linearly modify the values of a in epochs e e= En + 1,... E2 and linearly modily m epochs e = E2 + ..., ., Ez. The activation function for the score is a 2 x sigmoid, so the feature scores are initialized to approximately 1. The subsampled data fill X E RC the median of the data across each measurement/channel. These hyperparameters were constant across all experiments. For each experiment we chose C₁ to be the subset sizes required for the task e.g. in C
table we set C1,2,3,455 = C, 2 1 16'
We perform grid search to find the optimal network architecture hyperparameters for each task. The Scoring Network S and Task Network T have the same number of hidden layers E 21, 3]. number of units {30, 100, 300, 1000, 3000), for each combination we obtain task performance the feature set sizes C1 > C2 The best performing network on the validation set is deployed on the test data.
RANDOM FEATURE SELECTION (FS) DEEP LEARNING (DL.)
For different C' we repeat the following process. We randomly select C' number of measurements on a fixed random seed and perform grid search on the task network (mapping subsampled data to full data), with number of hidden layers {1, 2, 3), number of units (30, 100, 300, 1000, 3000). Training is until early stopping on the validation set. The best trained model is evaluated on the test set.
LEE ET AL. (0022
This approach has three-stage procedure and four neural networks exploit pask-performance, self-supervision, additional unlabeled data, and correlated feature subsets. It scores the features then subsequently train task-based network on subsampled data. We use the official repository Lee (2022). The optimization procedure is split into i) self-supervision phase, ii) supervision phase, Iii) training on selected features only.
The self-supervision phase finds the optimal encoder network hyperparameters. We perform grid search, the encoder network, feature vector estimator network, gate vector estimator network, all have same number of hidden layers (1,2.3 number of units, including hidden dimen- sion E (30, 100, 300, 1000, 3000), following Lee et al. (2022) tabbeess,, other hyperparameters {0.01, 0.1, 1.0, 10, 100), {0.2, 0.4, 0.6, 0.83. The self superviurry dataset is training set X. On the best validation performance (with early stopping), this returns a trained encoder network, cached for the supervision phase.
The supervision phase scores the features. The pre-trained encoder is loaded from the previous phase. We then perform grid search, where the predictor network has number of hidden layers E 2, 3). number of units E (30, 100, 300, 1000, 3000). following table S.1 3 E {0.01, 0.1, 1.0, 10, 100). On the best validation performance with early stopping, the process returns score for all features.
17

---

Under review as a conference paper at ICLR 2023
The final phase is repeated for different number of features subset sizes C. We extract the C highest scored features from the previous phase and perform grid search on the task network (map- ping subsampled data to full data), with number of hidden layers E {1, 2,3,, number of units E {30, 100, 300, 1000, 3000}. Training is until early stopping on the validation set. The best trained model is evaluated on the test
WOJTAS & CHEN 2020)
This approach has a complicated three-stage procedure and two neural networks that respectively performs the task, and scores masks, where the two networks are trained alternatively. This returns an optimal mask (analogous to my in our approach) and trained task-network (analogous to Te in our approach). We use the official repository Wojtas (202T). The following process is repeated for different feature subset sizes C.
We perform a grid search to find the optimal hyperparameters. The operator network (analogous to task network T in this paper) and the selector network have the same number of hidden layers E 2, 3), number of units E {30, 100, 300, 1000, 3000], = 5, E₁ = 15000. The joint training uses early stopping the validation set, and returns optimal feature set size, of size C' and trained operator network. The best performing operator network on the validation set is deployed on the test data.
C THE NODDI AND VERDICT MRI MODELS
This section describes the VERDICT and NODDI models used in section 4.1.1
1. The VERDICT (Vascular, Extracellular and Restricted Diffusion for Cytometry in Tumors) model Panagiotaki et al.) (2014), which maps histological features of solid-cancer tumors particularly for early detection and classification of prostate cancer Panagiotaki et al.] JJohnston et al. (2019); Singh et al. 2022). The VERDICT model includes parameters: // the intra-cellular volume fraction, fv the vascular volume fraction, D, the vascular perpendieular diffusivity,, R the mean cell radius, and n a 3D vector defining mean local vascular orientation.
The NODDI (Neurite Orientation Dispersion and Density Imaging) model Zhang et (0012). maps parameters of the cellular composition of brain tissue and is widely used in neuroimaging studies in neuroscience such as the UK Biobank study Alfaro-AAlagre et al. (2188), and neurology e.g. in Alzheimer's 's disease Kamiya et al. (2020) and multiple sclerosis Grussu et al. (2017). The NODDI model includes five tissue parameters: fie the intra-cellular volume fraction, fiso the isotropic volume fraction, the orientation dispersion index (ODI) that reflects the level of variation in neurite orientation, and a 3D vector defining the mean local fibre orientation.
Implementations of both models are publicly available Fick et al. (2019).
D HOw TO FIT DTI, DKI AND MSDKI MODELS AND COMPUTE DOWNSTREAM METRICS (PARAMETER MAPS)
This section provides additional information for the DTI, DKI and MSDKI models, how they are fitted MRI data, and how compute downstream metrics, used in section
Diffusion tensor imaging (DTI) Basser et al. (1994), diffusion kurtosis imaging (DKI) Jensen & Helpern 2010), and Mean Signal DKI (MSDKI) are widely-used qMRI methods. Like NODDI and VERDICT, they use diffusion MRI to sensitise the image intensity to the Brownian motion of water molecules within the tissue to provide a window on tissue mierostructure. However, whereas NODDI and VERDICT are designed specifically for application to brain tissue and cancer tumours, respectively, DTI and DKI are more general purpose techniques that provide indices of diffusivity (e.g. mean diffusivity MD), diffusion anisotropy (e.g. fractional anisotropy Basser & Pierpaoli
(1996) FA), and the deviation from Gaussianity, or kurtosis, (e.g. mean kurtosis Jensen & Helpern (2010 MK) that can inform tissue integrity pathology. Mean signal diffusion kurtosis imaging (MSDKI) is a simplified version of DKI that quantifies kurtosis using a simpler model that is casier to fit Henriques 2018). These techniques show promise for extracting imaging biomarkers for wide
18

---

Under review as a conference paper at ICLR 2023
Table 8: Mean--quared-error between parametric maps (downstream metrics) estimated from
C' = 288 ground truth measurements and C' from C reconstructed measurements. Additional results
table 5
DTI
DKI
MSDKI
C = 36
FA
MD
AD
RD
MK
AK
RK
MSD
MSK
Random F$ DL
1.17
3.13
9.3
3.75
6.37
6.28
13.3
2.50
4.15
Lee et al. 2022)
5.96
10.4
43.7
14.2
8.74
7.27
16.9
3.87
10.8
Wojtas & Chen 2020
10.4
68.8
106
81.4
9.39
10.3
18.6
11.1
8.82
JOFSTO
0.94
1.73
8.15
1.50
6.15
5.92
12.6
2.75
3.78
variety of medical applications, such mild brain trauma, epilepsy, stroke, and Alzheimer's disease Ranzenberger Snyder (2022) Tae et al. (2018; Jensen & Helpern (2010).
This section provides an outline of the standard MRI model fitting approaches we used in section 4.1.3 To fit the MISKI MRI models to the data, and obtain the downstream metrics (parameter maps), employ the widely-used. open-source DIPY library Garyfallidis et al. (2014). We followed standard practice for model fitting in MRI and used the least-squares optimization approach and default fitting settings. To remove outliers, values were clamped where DTI FA E [0, 1], DTI MD.AD,RD E [0, 0.003), DKI MK,AK,RK E [0.3], 3), MSDKI MSD E [0, 0.003 MSDKI MSK 3].
E ADDITIONAL RESULTS ESTIMATING TISSUE MICROSTRUCTURE FROM WATER DIFFUSIVITY
This section provides additional results to section 4.1.3 in table
DATA
This section describes the data used this paper. We provide summary in table slices are in figure
Visualizations of
Table 9:
Summary of data.
No. Measurements
Resolution mm
Acquisition Parameter
No. voxels n (K)
Name
Type
C
Dimension
Train
Val
Test
MUDI
gMR scan
1344
2.5
A R
321
132
105
HCP VERDICT
gMRI can
288
1.25
A AC R
2182
774
674
Simulated qMRI
220
A C R
1000
100
100
NODDI-WMC
Simulated qMRI
3612
AC RT
100
10
10
VERDICT AND NODDI MRI MODELS SIMULATED DATA
Table 10: Parameter ranges for simulating synthetic VERDICT and NODDI model data,
VERDICT
NODDI
Parameter
Minimum
Maximum
Parameter
Minimum
Maximum
fi Jv
0.01
0.99
fie 0.01
0.99
0.01
0.99
Jwo
0.01
0.99
Do (jans² 1
3.05
10
ODI
0.01
0.99
R (μm)
0.01
20
n
[-1-1-1]
[1 1 1
n
[-1 -1]
[1 1 1]
To simulate synthetic data for the VERDICT and NODDI models we use a known, fixed, acquisition scheme in combination with set of ground truth model parameters. We choose the ground truth model parameters by uniformly sampling parameter combinations from the bounds given in table TO We choose these bounds as they approximate the physically feasible limits of the parameters.
19

---

Under review as a conference paper at ICLR 2023
The VERDICT data has number of samples 1000 100K, 100K in the train, validation, test split, with target data Y E R= 0 E RR, 1,...,n. n. The classical ED approach yields an acquisition scheme derived from the Fisher information matrix Panagiotaki et al. (2015b) and here X E |R C = 20. The approaches in supervised feature selection use a densely-sampled empirical acquisition scheme, designed specifically for the VERDICT protocol from Panagiotakii ct al. (2015a) and here X E 1222220 C = 220.
The NODDI data has number of samples
100K, 10K, 10K in the train, validation, split,
with target data Y E R 0; E IK, 1,.... n. The classical ED approach yields an acquisition scheme derived from the Fisher information matrix Zhang et al. (2012) and so X E *=*990 C 99. The approaches in supervised feature selection use a densely-samples empirical acquisition scheme from an extremely rich acquisition from Ferizi et al. (017). This was designed for the ISBI 2015 White Matter Challenge, which aimed to collect the richest possible data to rank MRI models, and required a single subject to remain motionless for two uncomfortable back-to-back 4h scans, Here X E 36132 C 3612.
We used Rician noise, as appropriate for MRI data Gudbjartsson & Patz 1995). The signal to noise ratio of the unweighted signal is 50, which is representative of clinical qMRI.
To conduct the simulations we employ the widely-used, open-source dmipy toolbox Fick et al. (2019). and the code for the simulations is anonymously submitted. The results in table 5 are all scaled by: DTI-FA x 10?, DTI-MD x109, DTI-AD 110", DTI-RD x10", DKI-MK x x10?, DKI-AK x x10", DKI-RK 10?, MSDKI-MSD x10 MSDKI-MSK x10?
F.1 MULTI-DIFFUSINN (MUDI) CHALLENGE DATA
Data used in section 4.1.2 are images from 5 vivo human subjects, and publicly available MUDI Organizers 22022) and was acquired with the tfte-of-thee-rr ZEBRA sequence Hutter et al. (2018). Diffusion-relaxation MRI has 6l) acquisition parameter space A RR. echo time (TE), inversion time (TT), b-value, and b-vector directions in 3 dimensions: bz, by. bz. Data has 2.5mm isotropic resolution and field-of-veew 220 x 230 x 140mm and resulted in 5 3D bran images with = 1344 measurements/channels which here unique diffusion- 72" and 7'- weighting contrasts. More information is in Hutter et al. (2018): Pizzolato et al. (020)). Each subject has an associated brain mask, after removing outlier voxels resulted in 104520, 110420, 105743, 132470, 105045 voxels for respective subjects 11, 12, 13, 14, 15. For the experiment in table 4 we followed the original MUDI challenge Pizzolato et al. (2020) and took subjects 11, 12, 13 as the training and validation set, and subjects 14, 15 as the unseen test set, where 90% - 10% of the training/validattin set voxels are respectively, for training and validation.
F.2 HUMAN CONNECTOME PROJECT (HCP) TEST RETEST DATA
we utilise WU-Minn Human Connectome Project (HCP) diffusion data, which is publicly available at www.ac/cnnnnctttooo..c..e.e
In section 4.1.3
org (Test Retest Data Release, release date: Mar 01, 2017) Essen et al. (003)). The data comprises C 288 volumes (i.e. measuremeens//caamells,, with 18 b = O (i.e. non-diffusion weighted) volumes, 90 gradient directions for b 1000 mm 90 directions for b=2000 s mm 2 and 90 directions for b=3000 S mm -2, We used scans for training 103818_1, 105923_1, 111312.1, one scan for validation 114823_1 and one scan for testing 115320_1. which produced number of samples n = 708724 + 791369 + 681650 = 2181743, 774149, 674404 for the respective splits. We only used voxels inside the provided brain mask and normalized the data voxelwise with a standard technique in MRI, by dividing all measurements by the mean signal in each voxel's b- 0 values. Undefined voxels were then removed.
20

---

Under review as a conference paper at ICLR 2023
MUDI
VERDICT simulations
0
1.0
0.5
1343
219
0
HCP TestRetest
1343
NODDI simulations
219
0.0
-0.5
-1.0
287
3611
0
287
0
3611
Figure 3:
Correlation coefficient between the measurementt/features of the data, entries correspond to b0-values.
21

---

Under review as a conference paper at ICLR 2023
0
192
384
576
30
767
959
1151
1343
10
5
0
0
41
82
123
1600
1400
1200
1000
164
205
246
287
800
600
400
200
Figure 4: 2D brain slices from 3D MRI scans, for different measuremennt/eeauur//ccaamaa for the MUDI (above) HCP (below) data.
22

---

Under review as a conference paper at ICLR 2023
Table 11: Ablation study JOFSTO's components, experimental settings in table
C JOFSTO w/o Scoring Network S
110
55
28
14
7.23
10.7
11.5
11.5
JOFSTO w/o RFE
1.03
1.19
1.83
2.80
JOFSTO
1.03
1.18
1.80
2.64
Table 12: Standard deviation of performance x x102 and mean Jaccard Index between chosen measurements, across 10 random seeds, experimental settings in table | VERDICT simulations.
STD performance nerves seeds
Mean Jaccard Chosen Measurements %
C
500
250
100
50
500
250
100
50
Random FS + DL
0.11
0.23
0.37
0.76
32.8
15.2
6.82
2.87
Lee al. (2022)
0.01
0.02
0.05
0.18
81.2
71.4
62.2
75.9
Wojtas & Chen 2020)
0.44
0.34
0.37
0.44
34.3
18.1
48.8
41.0
JOFSTO
0.01
0.02
0.01
0.12
74.3
82.1
84.6
59.1
F.3
ABLATION STUDY AND THE EFFECT ON RANDOMNESS PERFORMANCE
Table II examines the impact of removing JOFSTO's s components on performance. First it considers JOFSTO without RFE i.e. without iteratively removing features our optimization algorithm, fixing t = 2 and C1, C2 = C, C, showing that iterative subsampling has better performance than subsampling all the features in a single iteration. As our approach to feature scoring is a key element of JOSFTO, we also consider the impact of removing the scoring network S, which results in extremely poor performance as training destabilized when progressively setting the score from cenpleedependence to sample-independence Table [2ble|T2]examine how the random seed affects network initialization and data shuffling. impacts performance. Results show JOFSTO performs favourably compared to alternative approaches, and JOFSTO is mostly robust to randomness inherent DL.
23


Reviews:
[{"review_id": "wGnTVGlgXN", "reviewer": "Reviewer_S5n8", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new method for \"experiment design (ED)\" by jointly performing feature selection and task optimization.\nThe performance of the proposed method, called JOFSTO, has been evaluated based on applications in Quantitative Magnetic Resonance Imaging (qMRI), where the evaluation results show that JOFSTO outperforms other existing schemes/baselines under a number of scenarios.\n\n", "summary_of_the_review": "In this work, they propose a new method called JOFSTO for experiment design (ED) (mainly in qMRI), which results in improved parameter estimates and reconstruction results through joint feature selection and task optimization. \nThe proposed method is simple yet reasonable, and although the work is somewhat limited in terms of the novel methodological contributions it is making, performance assessment results for several qMRI applications show that JOFSTO outperforms other existing schemes/baselines, and hence may provide practical advantages over alternatives.\n", "strength_and_weaknesses": "OVERALL COMMENTS\n\nThe main strength of the work lies in the overall performance gain achieved by JOFSTO in qMRI applications - in terms of improved parameter estimation, enhanced reconstruction performance, and more accurate quantification of tissue microstructure.\nWhile the proposed method itself strongly builds on existing work - especially,  Wojtas & Chen (2020) that utilizes dual feature scoring and task prediction networks - and the underlying idea is relatively simple, the performance evaluation results show that it nevertheless has potential benefits in qMRI analysis.\n\nHowever, the literary and technical presentations in the paper leave much room for improvement.\nSeveral important details for understanding the proposed methodology are missing or unclear, and the authors often refer to examples without first explaining their relevance and some notations without providing a clear definition.\n\nFurthermore, the proposed work appears to be motivated by a very specific application (i.e., ED for qMRI acquisition parameter optimization), and it is unclear whether JOFSTO would be applicable to other experiment design problems beyond qMRI applications considered in this study.\n\n\nDETAILED COMMENTS\n\n1. Considering that the proposed work is focused on a very specific application and problem setting, this should be more clearly reflected in the title and abstract.\n\n2. Experimental design is a widely studied topic across various disciplines, while the authors seem to be mostly focused on \"experiment design\" in qMRI applications.\nHowever, \"ED in qMRI\" seems to be quite different from what is typically referred to \"experimental design\" or \"optimal experiment design (OED)\" in various other fields, and this may potentially lead to confusion for readers.\nIt would be important to provide a more general treatment of the OED problem in the introduction or background and then zoom into this more specific ED problem in qMRI to avoid any confusion.\n\nHere are a few relevant references:\n\nLindley, Dennis Victor. Bayesian statistics: A review. Society for industrial and applied mathematics, 1972.\nChaloner, Kathryn, and Isabella Verdinelli. \"Bayesian experimental design: A review.\" Statistical Science (1995): 273-304.\nClyde, Merlise A. \"Experimental design: A Bayesian perspective.\" International Encyclopia Social and Behavioral Sciences 8 (2001): 5075-5081.\nSebastiani, Paola, and Henry P. Wynn. \"Maximum entropy sampling and optimal Bayesian experimental design.\" Journal of the Royal Statistical Society: Series B (Statistical Methodology) 62.1 (2000): 145-157.\nHuan, Xun, and Youssef M. Marzouk. \"Simulation-based optimal Bayesian experimental design for nonlinear systems.\" Journal of Computational Physics 232.1 (2013): 288-317.\nDehghannasiri, Roozbeh, Byung-Jun Yoon, and Edward R. Dougherty. \"Optimal experimental design for gene regulatory networks in the presence of uncertainty.\" IEEE/ACM Transactions on Computational Biology and Bioinformatics 12.4 (2014): 938-950.\nFoster, Adam, et al. \"Variational Bayesian optimal experimental design.\" Advances in Neural Information Processing Systems 32 (2019).\n\n3. The authors mention certain problems/applications - seemingly not closely related to the current work - without providing any context. Some examples include:\n\"e.g. selecting protein expressions for classification\" (in the abstract)\n\"unlike for example protein-coding genes 10x Genomics (2022) or noisy two-moons data Scikit-Learn (2022)\" (page 1/2)\n\"genes 10x Genomics (2022) or noisy two-moons data Scikit-Learn (2022)\" (page 3)\n\" e.g. noisy two-moons dataset.\" (page 3)\nShould the authors want to keep these examples, their relevance should be first clearly explained.\n\n4. In Section 2 \"Related Work\", the cardinality C of the set A need to be clearly defined.\nFor example, later in the paper C is used to refer to measurements or number of channels, but it would be better to provide a clear yet general definition first and then give specific examples in the current setting (e.g., qMRI application)\n\n5. The subsampling strategy for selecting C out of \\bar{C} should be better explained. \nFor example, JOFSTO is said to \"progressively\" construct m_t to have C_t ones, but how is this precisely achieved?\nDoes each step t -> t+1 reduce the C_t by a fixed number? (e.g. removal of a single feature) Or can it be variable?\n\n6. Furthermore, it is not clear how the feature scores quantify the relative importance of the available features and rank them.\nThis is central to JOFSTO to improve ED performance, and this needs to be clearly elaborated.\nEspecially, clearly explanation is needed regarding how the task at hand informs this scoring process to make the joint feature selection and task optimization work.\n\n7. On page 2, it is ambiguous what the authors are referring to as \"circularity\" and how the proposed subsampling-task paradigm for ED avoid this issue.\nPlease clarify.\n\n8. While the authors mention \"studying the size and diversity of densely sampled data needed to ensure strong generalizable design\" as future work, there should be at least some empirical evaluations on the minimum C needed to attain a certain level of performance for the task at hand.\nLet's denote this C as C_min.\nIt would be meaningful to understand the performance of JOFSTO (and any under/overfitting of the optimized network models) when C (<C_min) is much smaller than needed or when C (>C_min) is unnecessarily large, at least empirically.\nBased on this, prescribing a general guideline for deciding the optimal value of C would be practically important.\n\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Jk5JG2K_lR_", "reviewer": "Reviewer_SWxw", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a joint feature selection and model training procedure (JOFSTO) which is suitable for problems where features have certain spatial dependency properties (“densely sampled in a measurement space”). Procedure trains two connected neural networks, one which performs feature selection task, and the other which builds a prediction model on top of selected features. In particular application, the task is regression on quantitative MRI data, where JOFSTO had the smallest error against two recent state of the art joint feature selection and neural network training approaches.", "summary_of_the_review": "Under the impression of limited novelty and some questions around approach motivation and empirical evaluation, I am leaning towards rejection, although very eager to hear arguments that would change my impressions. ", "strength_and_weaknesses": "Strengths\nCompelling performance against strong and recently published baselines. \nThe problem is formulated within the Experimental Design setting, but is also correctly positioned within the feature selection literature.\nThe writing style of the paper stands out, easily readable and comprehensible.\n\nWeaknesses\nIt is unclear, at least to me, how the “selection of highly correlated globally informative (set of) candidates” is suitable to ED setting, while “small number of highly informative features among the many uninformative ones” is more suitable for FS. It all depends on the overall objective, and if both ED and FS share the same objective, they should favor the same/similar set of features. And how are those properties distinct/different? \nWhile it is encouraging that JOFSTO is outperforming Lee at al. (2022) and Wojtas & Chen (2020) approaches, it is peculiar that random FS + DL is also outperforming them (Table 5.). Appears that somehow informed feature selection, during the network training, is actually hurting the performance. Can it be that some sort of overfitting is behind such results?\nAblation study (Table 6.) suggests that Recursive Feature Elimination is not bringing that much performance improvement. Is the RFE step necessary then?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "9G2jFHB6Pw", "reviewer": "Reviewer_SJC9", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose an approach for experiment design in context of qMRI studies. The new approach combines feature selection and the choice of tasks in an integrated solution, using a combination of two neural networks.", "summary_of_the_review": "A novel and potentially interesting model for setting the configuration of qMRI devices, but the paper does not feel like a good fit for ICLR since the method makes assumptions that are not easy to satisfy in general cases and the method presentation is not detailed enough. I cannot see future work building on this.", "strength_and_weaknesses": "Strengths:\n- The approach is very well motivated for the qMRI application where the effect of different acquisition parameters is reflected in a different manner in each voxel\n- Works well in several experiments\n\nWeaknesses:\n- The method explicitly relies on densely sampled super-design and access to ground truth labels for 'the task driving the ED'. While the authors explain why we can satisfy these assumptions in the specific case of qMRI, I cannot identify other common scenarios where we would meet these criteria and the goal would be to find an 'economical design' for the same task. This severely limits the value of the work for general AI audiences\n- The paper is presented strongly from the perspective of the application (as it needs to be because the details rely on the assumptions that hold for this application) and would better fit a more specialised venue.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject