Implicit Deep Adaptive Design: Policy-Based Experimental Design without Likelihoods
Authors: Desi R. Ivanova
Adam Foster
Steven Kleinegesse
Michael U. Gutmann
Tom Rainforth
Venue: NeurIPS 2021 Poster (2021)
Keywords: Experimental Design
Mutual Information
Implicit Models
Bayesian methods
Bayesian Optimal Experimental Design
BOED
Sequential BOED
Adaptive experiments
Likelihood-free methods
Likelihood-free inference
Parameter estimation
Variational methods
Mutual information lower bounds
Stochastic simulator models

Abstract: We introduce implicit Deep Adaptive Design (iDAD), a new method for performing adaptive experiments in real-time with implicit models. iDAD amortizes the cost of Bayesian optimal experimental design (BOED) by learning a design policy network upfront, which can then be deployed quickly at the time of the experiment. The iDAD network can be trained on any model which simulates differentiable samples, unlike previous design policy work that requires a closed form likelihood and conditionally independent experiments. At deployment, iDAD allows design decisions to be made in milliseconds, in contrast to traditional BOED approaches that require heavy computation during the experiment itself. We illustrate the applicability of iDAD on a number of experiments, and show that it provides a fast and effective mechanism for performing adaptive design with implicit models.

Implicit Deep Adaptive Design: Policy-Based Experimental Design without Likelihoods
Desi R. Ivanova' Adam Foster' Steven Kleinegesse* Michael U. Gutmann' Tom Rainforth"
'Department of Statistics, University of Oxford School of Informatics, University of Edinburgh
desi www.ova@statteeoooomm ac,
Abstract
We introduce implicit Deep Adaptive Design (iDAD), a new method for performing adaptive experiments real-time with implicit models. iDAD amortizes the cost of Bayesian optimal experimental design (BOED) by learning design policy network upfront, which can then be deployed quickly at the time of the experiment. The iDAD network can be trained any model which simulates differentiable samples, unlike previous design policy work that requires a closed form likelihood and conditionally independent experiments. At deployment, iDAD allows design decisions to be made in milliseoodds, in contrast to traditional BOED approaches that require heavy computation during the experiment itself. We illustrate the applicability of iDAD on a number experiments, and show that it provides a fast and effective mechanism for performing adaptive design with implicit models.
Introduction
Designing experiments to maximize the information gathered about an underlying process is key challenge in science and engineering. Most such experiments are naturally adaptive- can design later iterations the basis of data already collected, refining our understanding of the process with each step [36, 45, 51]. For example, suppose that a chemical contaminant has accidentally been released and is rapidly spreading; we need to quickly discover its unknown source. To this end, we measure the contaminant concentration level at locations f ... E (our experimental designs), obtaining observations y/..... yr. Provided
perform the necessary computations sufficiently quickly, we can design each &, using data from steps 1, I....,t
to narrow on the source.
Bayesian optimal experimental design (BOED) [7. 32] is principled model-based framework for choosing designs optimally; it has been successfully adopted [52, 58, 60]. In BOED, the unknown quantity of interest (e.g. contaminant location) is encapsulated by parameter 0, and our initial information about it by a prior p(0). A simulator, or likelihood. model 000, & describes the relationship between 0, our controllable design &, and the experimental outcome y. To select designs optimally, the guiding principle is information maximization-we select the design that maximizes the expected (Shannon) information gained about 0 from the data y. or, equivalently, that maximizes the mutual information between 0 and y.
diverse range of scientific fields
This naturally extends to adaptive settings by considering the conditional expected information gain given previously collected data. The traditional approach, depicted in Figure 1a, is to posterior p(0([1=-11, 11:1-1) after each iteration, and then select & in a myopic fashion using the one-step mutual information (see, e.g.. [51] for a review). Unfortunately, this approach necessitates significant computation at each t and does not lend itself to selecting optimal designs quickly and adaptively.
Recently, Foster al. [17] proposed exciting alternative approach, called Deep Adaptive Design (DAD), that is based on learning design policies. DAD provides a way to avoid significant computation
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Live experiment
Offlie
deploy
Policy
Design
Live experiment
Design
Critic
T
iterations
T iterations
Offline
Infer
Observe
Infer
data
Observe
(a) Traditional BOED: costly computa- tions (design optimisation and parameter inference) required each iteration.
(b) Policy-based BOED using iDAD: a design policy and critic are learnt before the live experiment. The policy enables quick and adaptive experiments, the critic assists likelihond-free inference.
Figure 1: Overview of adaptive BOED approaches applicable to implicit models.
deployment-time by, prior the experiment itself, learning a design policy network that takes past design-outcome pairs and near waut--nstttttnoussll. returns the design for the next stage of the experiment. The required training is done using simulated experimental histories, without the need to estimate any posterior or marginal distributions. DAD further only needs single policy network to be trained for multiple experiments, further allowing for amortization of the adaptive design process. Unfortunately, DAD requires conditionally independent experiments and only works for the restricted class of models that have an explicit likelihood model we can simulate from, evaluate the density of, and calculate derivatives for, substantially reducing its applicability.
To address this shortfall, we instead consider far more general class of models where we require only the ability to simulate y|O, & and compute the derivative 0y/OC. c.g. via automatic differentiation [5]. Such models are ubiquitous in scientific modelling and include differentiable implicit models [19], for which the likelihood density P(y]0, {) is intractable. Examples include mixed effects models [15, 18). various models from chemistry and epidemiology [1]. the Lotka Volterra model used in ecology [19], and models specified via stochastic differential equations (such as the SIR model [10]).
To perform rapid adaptive experimentation with this large class of models, introduce implicit Deep Adaptive Design (iDAD), a method for learning adaptive design policy networks using only simulated outcomes (see Figure 1b). To achieve this, introduce likelihood-free lower bounds on the total information gained from a sequence of experiments, which iDAD utilizes to learn deep policy network. This policy network amortizes the cost of experimental design for implicit models and can be run in milliseconds at deployment-time. To train it, we show how the InfoNCE [57] and NWJ [37] bounds, popularized in representation learning, can be applied the policy-based experimental design setting. The optimization of both of these bounds involves simultaneously learning an auxiliary critic network, bringing an important added benefit: it can used to perform likelihood-free posterior inference of the parameters given the data acquired from the experiment.
We also relax DAD's requirement for experiments to be conditionally independent, allowing its application in complex settings like time series data, and, through innovative architecture adaptations, also provide improvements in the conditionally independent setting as well. This further expands the model space for policy-based BOED, and leads to additional performance improvements.
Critically, iDAD forms the first method in the literature that can practically perform real-time adaptive BOED with implicit models: previous approaches are either not fast enough to run in real-time for non-trivial models, require explicit likelihood models. We illustrate the applicability of iDAD range of experimental design problems, highlighting its benefits over existing baselines, even finding that it often outperforms costly non-amortized approaches. Code for iDAD is publicly available https: //t:://gitttt ttt.gttttbb..e..e com/desi. desi- ivanovaa matoooaaiidaa iaad.
2
Background
The BOED framework [32] begins by specifying a Bayesian model of the experimental process, consisting of prior on the unknown parameters p(0), a set of controllable designs ç, and data generating process that depends on them y/0, as usual in BOED, we assume that p(0) does not depend on In this paper, consider the situation where 0/0, & is specified implicitly. This means

---

that it is defined by deterministic transformation, f(E; 0,;; of base (or noise) random variable, that is independent of the parameters and the design: e.g., E NN 0. I). The function f is itself often not known explicitly in closed form, but is implemented as stochastic computer program (i.e. simulator) with input (0, <) and & corresponding to the draws from the underlying random number generator (or equivalently the random seed). Regardless, the resulting induced likelihood density p(y|0, <) is still generally intractable, but sampling y|0, ę is possible.
Having acquired design-outcome pair (E, y), we can quantify the amount of information we have gained about by calculating the reduction entropy from the prior to the posterior. We can further assess the quality of a design before acquiring by computing the expected reduction in entropy with respect the marginal distribution of the outcome, plyl() Epto [p(y] <)]. The resulting quantity, called the expected information gain (EIG), is of central interest in BOED and is defined as
T(e)= = (p(0))(110.5) log p(0 v)
(p(0)(2)1) log (,)
p(0)
p(yl()
Note that (EE is equivalent to the mutual information (MI) between the parameters 0 and data y when performing experiment Ç. The optimal & is then the one that maximises the EIG, i.c. 5* = arg max; I(E). Performing this optimization is a major computational challenge since the information objective is doubly intractable [46]. For implicit models, the cost becomes even greater as the likelihood is also not available closed form, so estimating it, along with the marginal likelihood p(yl(), is already itself major computational problem [11, 20, 33, 54).
Jointly optimizing the design variables for all undertaken experiments at the same time using (1) is called static experimental design. In practice, however, we are often more interested in performing multiple experiments adaptively in a sequence ET, so that the choice of each & can be guided by past experiments, namely the corresponding history h4- == {EE Wi)]imta-11 The typical approach in such settings is to sequentially perform (approximate) posterior inference for @ h followed by one-step look ahead (myopic) BOED optimization that conditions on the observed history. In other words, to determine the designs .i.....Er Er,, we sequentially optimize the objectives
p(ve &, 11-1) log Souls,h,-1
I (E+) = (110)h.-1)pwwlle....
t= 1= T.
However, such approaches incur significant computational cost during the experiment itself, particu- larly for implicit models [16, 21, 30). This has critical consequences: most cases they cannot be run in real-time, undermining one's 's ability to them in practice.
2.1 Policy-based adaptive design with likelihoods
For tractable likelihood models, Foster et al. [17] proposed new framework, called Deep Adaptive Design (DAD), for adaptive experimental design that avoids expensive computations during the experiment. To achieve this, they introduce parameterized deterministic design function, or policy, To that takes the history h as input and returns the design & To (h) to be used for the next experiment as output. This set-up allows them to consider the objective
Ir( = (p(0)p(br100
Σ Imm (Ex) (Ei)
&. = (h
(3)
which crucially depends the policy rather than the individual design & Learning policy up-front, rather than designs, is what allows adaptive experiments to be performed in real-time.
Under the assumption that ye is independent of h4-1 conditional on the parameters 0 and the design & ie. p(yele 0, &e, h1- p(ye 1(y.10,E.) (0, &.), Foster al. [17] showed that the objective can be simplified to
p(hr 10, ka) Ir( = Ep(0rp(br100m)) log
p(hr 0, T6)
II 1lv. E.). 1=1
(4)
p(1 (T))
To deal with the marginal p(hr (xs) in the denominator, they then derived several optimizable lower bounds on [TTT)), such as the sequential Prior Contrastive Estimation (sPCE) bound
p(or To)
Ci CAPCB (T&, L) Ep(ea)n( (0. ()0L))))
log
< IT(TT) VL 2 1. (5)
L+I El p(TTT TT)
3

---

The parameters of the policy o, which takes the form of deep neural network, are now learned prior the experiment(s) using stochastic gradient ascent on this bound with simulated experimental histories. Design decisions can then be made using a single forward pass of T during deployment. Unfortunately, training the DAD network by optimizing (5) requires the likelihood density p p(hrr00, x) to be analytically available- an assumption that is too restrictive in many practical situations. The architecture for DAD also assumes conditionally independent designs, which is unsuitable in some settings like time-series data. Our method lifts both of these restrictions.
3 Implicit Deep Adaptive Design
We have seen that the traditional step-by-step approach to adaptive design for implicit models [16, 21, 30] is too costly to deploy for most applications, whilst the only existing volicy-based approach, DAD [17], makes restrictive assumptions that prevent it being applied to implicit models. We aim to relax the restrictive assumptions of the latter, making policy-based BOED applicable to all models where we can sample from ylo. and compute the derivative 0y/ a strict superset of the class of models that can be handled by DAD. This requires new training objectives for the policy network that do not involve an explicit likelihood and are not based on conditionally independent experiments, along with new architectures that work for non-exchangeable models like time series.
3.1 Information lower bounds for policy-based experimental design without likelihoods
To establish a suitable likelihood-free training objective for the implicit setting, our high-level idea is to leverage recent advances in variational MI [see 42, for an overview), which have shown promise for static BOED [16, 28, 29]. While using these bounds in the traditional framework of (2) would not permit real-time experiments, one could consider a naive application of them to the policy objective of (3) by replacing each Ih, with a suitable variational lower bound that uses 'critic' U, H x θ + I avoid explicit likelihood evaluations, where H' and Q are the spaces of histories and parameters respectively. An effective critic successfully encapsulates the true likelihood, tightening the bound. Although its form depends on the choice of bound, all critics are parametrized and trained in the same way, namely by neural network Ug. which is optimized to tighten the bound. Unfortunately, replacing each Ih.-, involves learning T such critic networks and requires samples from all posteriors p(0) which will typically be impractically costly.
To avoid this issue, we show that we can obtain unified information objective similar to (4), even without conditionally independent experiments. The following proposition therefore marks the first key milestone in eliminating the restrictive assumptions of [17]. by establishing a unified objective without intermediate posteriors that is valid even when the model itself changes between time steps. Proposition 1 (Generalized total expected information gain). Consider the data generating distribu- tion (0, T) = II I.T P(ye (0, &e, hx-1), where & = T (hb-1) are the designs generated by the policy and, unlike in (4), Yt allowed to depend on the history hz-1- Then we can write (3) as
[r(
E (x(0))((T7(8xx)) logg(he (0, x)] E.p(trrss)
[log ((brl
Proofs are presented in Appendix A. The advantage of (6) is that we can draw samples from p(0)p(hrr( T simply by sampling our model and taking forward passes through the design network. However, neither of the densities p(hr 0, π) and p(/rl( are tractable for implicit models.
To side this intractability, we observe that (T() takes analogous form to MI between 0 and hy. For measure- theoretic reasons, namely because the {urr are deterministic given yı.r (see Appendix A for a full discussion), it is not the true MI. However, the following two propositions show that we can treat (=) as if it were this MI. Specifically, we show that the InfoNCE [57] the MI can be adapted to establish tractable lower bounds on our unified objective (Tr). These two bounds both utilize a single auxiliary critic network Up that is trained simultaneously with the design network.
and NWJ [37] bounds
Proposition 2 (NWJ bound for implicit policy-based BOED). For a design policy n and a critic function U: H x θ R, let
CT
NW. (T, U) == Ep(0)p(hr () (Uh, 0)] - - Ep(r) (exp(U (hr 0)).
(7)
then IT [T( 2 LNW (T,, U) holds for any U. Further, the inequality is tight for the optimal critic Uww 0) = p(g x) π)
log bog((hr(m) (T) +
4

---

Algorithm 1: Implicit Deep Adaptive Design with (iDAD)
Input: Differentiable simulator f. sampler for prior p(0), number of experimental steps Output: Design network Td, critic network Up while Computational training budget not exceeded do
Sample 0 ~ p(f) and set ho =
for 1, 1,... T do
Compute & & (h4-1)
Sample E₁ ~ p()) and compute y/: = [(ex; Et, ht-1)
Set h: (1),,)),,,..............,,),), v.),
end
Estimate Vo Uv) as per (10) where Cr is DN (7) or LNCE (8)
Update the parameters (Ø, 4/) using stochastic gradient ascent scheme
end
For deployment, use the deterministic trained design network T obtain a designs & directly.
Proposition 3 (InfoNCE bound for implicit policy-based BOED). Let 0I.L ~ -((0..)==11, p(0.) be a set of contrastive samples where Z 1. For design policy n and critic function U UH x let
exp(U(tt,, 00))
LACE CACE(TT (T, U; (( Ep(0LL))
log I+I exp(U (hT 02))
(8)
then (T) 2 CANC (Tr, U: L.) for any U and I, 2 1. Further, the optimal critic, UNr O) log p( (0 10,)) c((rr) where c(hvr) is any function depending only on the history, recovers the sPCE bound (5); the inequality is tight in the limit as L + oc for this optimal critic.
function
We propose these two alternative bounds due to their complementary properties: the NWJ bound can have large variance, but tends to be less biased. That is, the NWJ bound tends to be tighter for good critics, but is itself more difficult to reliably estimate and thus optimize. While the NWJ critic must learn to self-normalize, the InfoNCE bound avoids this issue but typically will not be tight for finite L even with an optimal critic (note CNCR < log(L + 1) [42]). Consequently, only the NWJ objective recovers the true optimal policy our critic has infinite capacity and our optimization scheme is perfect, i.c. arg max, maxy CNWI (I,, U) '²" * arg max, maxy CNCF (ι, U; L) in general, but it can be more difficult to work with in practice. We present a third bound that provides a potential solution to this, and further discuss the relative merits of the two bounds, in Appendix A.
We note that for both bounds the optimal critic does not depend on the learned policy. The final p(0)(T)/p(0)
trained critic can be used to approximate the density ratio p(hr7(0, r)/p(br(TT) either directly in the case of the NWJ critic, or via self-normalization for the InfoNCE bound. We can use this to help approximate the posterior over 0 given the collected real data from the experiment. This means we can perform likelihood -free inference after training the critic, which extends previous results [28, 29] from the static to the adaptive policy-based setting.
3.2 Parameterization and gradient estimation
In practice, we represent the policy π and the critic U as neural networks, Ne and U, respectively, such that the lower bounds become a function L( Up) of their parameters. By simultaneously optimiz- ing L(ms, Up) with respect to both & and v. we both learn a tight bound that accurately represents the true MI and a design policy network that produces high-quality designs under this metric.
We optimize these bounds using stochastic gradient methods [26, 49]. For this, must account for the fact that the parameter ó affects the probability distributions with respect to which expectations are taken. We deal with this problem by utilizing the reparametrizzaionn trick [35, 48], for which we assume that design space E\t and observation space y are continuous. To this end, we first formalize the notion of differentiable implicit model in the adaptive design setting as
(9) and we assume that we can compute the derivatives a5/ag and 0f/Oh. Interestingly, it is possible to use an implicit prior without access to the density p(0), and we do not need access to 0 /00
y = (Ex: (h, 1), 0, 1) where 0 ~ p(0), E, P(E) Vt E (1....,T]
5

---

hr
History encoder
h4
History encoder
MLP
[1+1
Dot Product
It+1 Ul hr
0 encoder
(a) Critic network, Up
(b) Policy network, n
Figure 2: Overview of network architectures used in iDAD.
Under these conditions, we can express the bounds in terms of expectations that do not depend on ø or et, and hence move the gradient operator inside. For C
Usi L), for example, we have (To. CNC
exp (hr (E1.T, Ro),
NCE 4.
Eppeo. (p((er)
V
log
(10)
E exp(U ($1., mo), 0,))
While each element of the history fr depends é in possibly nested manner. do not need to explicitly keep track of these dependencies thanks to automatic differentiation [5, 41].
Like DAD, our new simulated histories hr [(C,, y)) prior to the actual experiment, allowing design decision to be made using single forward pass during deployment. Unlike DAD, however, it does not require knowledge of the likelihood function, nor the assumption of conditionally independent designs, which significantly broadens its applicability. A summary of the iDAD approach is given in Algorithm 1.
we call implicit Deep Adaptive Design (iDAD) is trained with
3.3 Network architectures
The iDAD approach involves the simultaneous training of the policy Hh and critic Up networks. It is essential to choose the neural architectures of these two components carefully to learn effective policies: poor choices of critic architecture will lead to loose, unrepresentative, bounds, while poor choices of policy architecture will directly lead to ineffective policies. Good choices of architecture need to balance flexibility with ease of training, and will typically require the incorporation of -obbem--peccfic inductive biases. A high-level summary of our architectures is shown in Figure 2.
The critic network, U... takes complete history hr and the parameter input, and outputs scalar. Our suggested architecture first encodes the two inputs separately to representations of the same dimension, using history encoder, Exg,, and a parameter encoder, F respectively. The output of the critic is then simply taken their dot product (hr Esg, (hr) Exg, (0): after training, the two encodings correspond to approximate sufficient statistics [9]. This setup corresponds to a separable critic architecture, as is commonly used in the representation learning literature [3, 8, 57]. While we use a simple MLP for E the setup for Exy, varies with the context as we discuss below.
The policy network, ag. takes the available history, hr, as input, and outputs design. Our suggested architecture makes use of a history encoder, Es,, of the same form as Egg, except that it must now take in varying length inputs; its output remains a fixed dimensional embedding. We then pass this embedding through an MLP to produce the design &++1- At the next iteration of the experiment, the same policy network is then called again with the updated history h4+1 = h2 U {(E+11, ((+++,,(+1)),
We use the same architecture for both history encoders, Eg,, and Eg., but do not share network parameters between them. This architecture first individually embeds each design-outcome pair (Et; We) to a corresponding representation, rt, using a simple MIP that shared across all time steps. The produced history encoding is then aggregation of these representations, with how this is done depending on whether the experiments are conditionally independent, i.c. yt JL h, 10, E,, or not.
Foster et al. [17] proved that if experiments are conditionally independent, then the optimal policy is invariant to the order of the history. We prove that the same true for the critic Proposition 5 in Appendix C. In our setup, we can exploit this result by using a permutation invariant aggregation strategy for (ri..... rr} when conditional independence holds. The simplest approach do this would be to use sum-pooling [62], as was done in DAD. However, to improve on this, we instead propose using more advanced permutation invariant architecture based on self-attention [13, 25, 39, 47, 59], namely that of Parmar et al. [40]; we find this provides notable empirical gains. When conditional independence does not hold, this approach is no longer appropriate and we instead use an LSTM [22] for the aggregation. See Appendix C for further details.

---

4 Related work
Adaptive policy-based BOED has only recently been introduced [17] and has not yet been extended to implicit models the gap that this work addresses. Previous approaches to adaptive experiments usually follow the two-step greedy procedure described in Section 2. Methods for MI/EIG estimation without likelihoods include the use of variational bounds [15, 16, 28] and ratio estimation [27, 30]; approximate Bayesian computation together with kernel density estimation [43]; and approximating the intractable likelihood first, for example via polynomial chaos expansion [24], followed by applying likelihood-basee estimators, such as nested Monte Carlo [46]. The maximization step in more traditional methods tends to rely on gradient-free optimization, including grid-search, evolutionary algorithms [44], Bayesian optimization [15, 30), or Gaussian process surrogates [38]. More recently, gradient-basee approaches have been introduced [15, 28], some of which allow the estimation and optimization simultaneously in single stochastic-gras -raadiiiggginntttttccme scheme [16, 23, 29]. From a posterior estimation perspective, likelihood free inference can be performed via approximate Bayesian computation [33, 54], ratio estimation [56]. conventional MCMC for methods that make tractable approximation to the likelihood [23, 24), or as a byproduct of MI estimation [16, 27, 29, 30].
Experiments
We evaluate the performance of iDAD on a num- ber of real-world experimental design problems and a range of baselines. A summary of all the methods that we consider is given in Table 1. Since we aim to perform adaptive experiments in real-time, we focus mostly on baselines that do not require significant computational time dur- ing the experiment. These include heuristic ap- proaches that require no training, namely equal interval designs (when possible) and random designs, as well as static BOED approaches, where we, non-adaptively, choose all the designs prior to experiment by optimising the mutual information objective of Equation (1) with E = {Ei, (6.,.-..77 and y (yI................... (y1...., .T)): J. The static BOED approaches we consider are the MINEBED method of [28] and the likelihood- free ACE approach of [16], where we use the prior as a proposal distribution, referring to this baseline as SG-BÖED. We also implement the expensive traditional non- non-amortized myopic strategy described in Section 2. for which we use the variational approach of [16], with the Barber-Agakov bound [4, 15], at each experiment step (see Appendix D.3 for details). Finally, where possible, we compare our method with DAD [17], in order to assess the performance gap that would arise if we had analytic likelihood. This comparison is done primarily for evaluation purposes because it has access to the likelihood density, DAD serves as an upper bound on the performance iDAD can achieve; one should use explicit likelihood methods whenever possible.
Table Key properties of considered methods.
Adaptive Real-time Implicit
Random
X
N/A
Equal interval MINEBED SG-BOED Variational DAD
x
N/A
X
N/A
X
N/A
X
X
iDAD
The main performance metric that we focus on is the total EIG, Ir(T), as given in (6). In cases where the likelihood is available, we estimate the (T( using the sPCE lower bound in (5) and its sister upper bound, the sequential Nested Monte Carlo bound [sNMC; 17J. To ensure that the bounds are tight, we evaluate them with large number of contrastive samples, i.e. L 10S. Where the likelihood is truly intractable, we assess the iDAD strategy in a more qualitative manner by looking at the optimal designs and approximate posteriors. For the adaptive experiments, we further consider the deployment time (i.e. the time required to propose a design), which is a critical metric for our aims. All deployment times exclude the time needed to determine the first experiment it can be computed up-front, during the training phase. Timings for training the policy itself are given in Appendix D.
5.1 Location Finding
We first demonstrate our approach the location finding experiment from [17]. Inspired by the acoustic energy attenuation model [53], this experiment involves finding the locations of multiple hidden sources, each emitting signal with intensity that decreases according to the inverse-square law. The total intensity-a superposition of these signals-can be measured noisily at any location. The design problem is choosing where to measure the total signal in order to uncover the sources.
7

---

Table 2: Lower bounds on the total information, [T)(T), for the location finding experiment in Section 5.1. The bounds were estimated using L = 5 10 contrastive samples. Errors indicate #1 standard errors estimated over 4096 histories (128 for variational). Corresponding upper bounds are given in Table 6 in Appendix D.
Method dim.
4D
6D
10D
20D
Random
4.791 + 0.040
3,468 + 0.014
1.889 = 0.011
0.552 = 0.006
MINEBED
5,518 + 0.028
4.221 + 0.028
2,458 0.029
0.801 + 0.019
SG-BOED
5.547 + 0.028
4
0.803 0.019
Variational
4.639 0.144
3.625 0.165
2.181 0.151
0.669 + 0.097
iDAD (NWJ)
7.694 + 0.045
5.765 0.036
3.252 + 0.039
0.877 0.022
iDAD (InfoNCE)
7.750 + 0.039
5.986 0.037
3,251 = 0.039
0.871 = 0.020
DAD
7.967 + 0.034
6.300 + 0.030
3.337 + 0.039
0.937 + 0.022
Table 3: Lower and upper bounds on MI (I) for different network architectures on location finding experiment using the InfoNCE bound. All estimates obtained in Table 2.
Design
Critic
Lower bound
Upper bound
Attention
Attention
7.750 + 0.039
7.863 + 0.043
Attention
Pooling
7.567 0.037
7.632 + 0.039
Pooling
Attention
7.398 + 0.040
7,470 + 0.042
Pooling
Pooling
7.135 0.034
7.192 + 0.041
In Table 2 we can see that iDAD substantially outper- forms all baselines including, perhaps surprisingly, the traditional (non-amortized) adaptive variational approach, despite its large computational cost shown Table The poor performance of the variational ap- proach is likely driven by the inability of the mean- field variational family to capture the highly non- Gaussian true posterior, highlighting the detrimental effect that wrong posteriors can have on determining optimal designs when using the traditional sequen- tial BOED approach.
Table 4: Deployment time of adaptive meth- ods 2D, measured on a CPU. Errors were calculated on the basis of 10 runs.
Method
Deployment time (sec.)
Variational
2256.0
+ 1 %
iDAD (NWJ)
0.0167 + 2%
iDAD (InfoNCE)
0.0168 + 2%
DAD
0.0070 + 6%
Table 2 further shows that the performance gap to the likelihood- based DAD method is small, even as the dimension of the design and parameter space grows. Though the information gained by all methods decreases with the dimensionality, this is to be expected: in higher dimensions it is inherently more difficult to infer the relative direction the sources from observing their intensity. Overall, this experiment demonstrates that iDAD is able to learn near-optimal amortized design policies without likelihoods, while being run milliseconds at deployment.
Ablation: attention to history. We next assess the benefit of utilizing our proposed more sophis- ticated permutation invariant architectures, compared the simple pooling of [62] used in [17]. Our approach incorporates attention layers into both networks that we train. This leads us to four possible combinations of network architectures. Table compares the efficacy the resulting design policies and strongly suggests that incorporating attention mechanisms in either and/or both networks improves performance, with inclusion in the design network particularly important.
We preform further ablation studies to investigate and demonstrate important properties of our method, such its scalability with the number of experiments T, stability between different training runs and performance to errors in the design network (introduced by not training the network to convergence). Results and discussion are provided Appendix D.4.4.
5.2 Pharmacokinetic model
Our next experiment is taken from the pharmacokineties literature and has been studied other recent works on BOED for implicit models [28, 63]. Specifically, we consider the compartmental model

---

C Concentration trajectories
b) iDAD strategy, T 5
cJ MI estimates, =
3:54
8 B
8,
15-
Experimnt rumneettuueer....
Concertration
10-
Randoon
Eoual
Vonarional
Maenep
55-40RD
RND away)
@ao nessice)
DAD
12
16
20
Measurement time
Design E
Figure 3: Plots for pharmacokinetics experiment. a) Visualisation of model showing concentration level as a function of measurement time for 3 values of 0, resulting in a quick (0,). average (0.). or slow (0.) trajectory. b) Designs selected by an iDAD policy trained with InfoNCE. c) MI lower bounds achieved by iDAD and baselines, All estimates obtained as in Table 2.
of [50], for which the distribution of an administered drug through the body is governed by three parameters: absorption rate ka, climination rate ke. and volume V. which form the parameters of interest, i.c. 0 = (ka kee V). Given T' = 5 patients, the design problem is to adaptively choose blood sampling times, 0 S & S 24 hours, for each, measured from the the point the drug was administered (with patient 2 not being administered until after sampling patient I etc). Plausible concentration trajectories shown in Figure 3a). Full details and further results are given in Appendix D.5.
We first qualitatively consider the design policy of iDAD (trained with the InfoNCE objective) in Figure 3b). As we have not y observed any data, the optimal design for the first patient (bottom row) the same for all 0. For the second patient, only guided by & and the outcome 1/1, iDAD is already able to distinguish between quickly and slowly decaying concentration trajectories: it proposes significantly earlier measurement time for the quickly decaying trajectory (purple triangle, 6.)) and later time for the slowly decaying one (yellow diamond, 0.). For the third patient, iDAD always targets the peak of the drug concentration distribution which is quite similar for all B. Measurements for the last two patients made soon after the drug has been administered (~ 15 30 min), when concentration levels increase rapidly, to capture information about how quickly the drug is absorbed.
To provide more quantitative assessment and compare to our base- lines, we again consider the final EIG values as shown in Fig- ure 3c). This reveals that the iDAD strategies perform best among the methods that are applicable to implicit models, confirming that the learnt policies propose superior designs. The performance gap to DAD, which relies on explicit likelihoods, is not statistically significant (at the 5% level) for iDAD trained with InfoNCE, while significant, but still small, for NWJ.
MI lover bound
DAD IDAD (InfoNCE)
Finally, we consider the convergence of the iDAD networks under the different training objectives and compare to DAD for reference. As shown in Figure 4, although all three converge to approximately the same value, they do so at rather different speeds: while DAD requires about 5000 gradient updates, implicit methods need longer training and tend to exhibit higher variance, particularly NWJ.
IDAD (NW))
10000
20000
30000
40d0b
50050
Training step
Figure 4: Convergence of MI lower bounds.
5.3
SIR Model
In this experiment, demonstrate our approach implicit model from epidemiology. Namely, we consider a formulation of the stochastic SIR model [10] that is based on stochastic differential equations (SDEs), done by [29]. Here, individuals in fixed population belong one of three categories: susceptible, infected or recovered. Susceptible people can become infected and then recover, with the dynamics of these two events being governed by two model parameters the infection rate 3 and the recovery rate 7. Our aim is to determine the optimal times T at which to measure the number of infected people, [(r), in order to estimate the two parameters. This implicit model is challenging because data simulation is expensive, since we need to solve many SDEs, and experimental designs have a -inedeepnd.ncc. See Appendix D.6 for full details.

---

a) E pidemic trajectories
b) iDAD strategy, T 5
c) Estimated posterior, R 3.0
300
1.3
R =8.0
Experiment numberr
Number infecttdd
202
0.16
0
Recovey
0.10-
100-
AR
+8.0
0.05-
x True value
50 75
100
0.2
Measurement time
Design E
Infection rate, P
Figure 5: a) Epidemic trajectories for 3 realization of (3, 7) with different reproduction numbers R = B/7. b) Designs selected by an iDAD policy trained with NWJ. c) Example posterior estimates from the critic network given data generated with the ground-truth parameters shown by the red cross.
We train a iDAD networks to perform 7' = 5 experiments and compare against random, equal interval, and static design baselines; DAD cannot be run because the problem corresponds to true implicit model. Table shows lower bound estimates on the MI and demonstrates that iDAD outperforms all compared methods. Note that degree of caution is required when analysing the results, as they are influenced by unavoidable biases the estimation process. Namely, a critic is still required to estimate the MI lower bound, and there may be variations in the effectiveness of these critics, with less effective ones corresponding to looser bounds and therefore underestimating the true MI. Nonetheless, for other models where such checks are possible, we have found the bounds to be relatively tight, while, even if this turns out not to be the case here, the fact that the critics for the static approaches are easier to train should mean our relative evaluations for iDAD (and Random) still conservative compared the other baselines.
Table 5: MI lower bounds (#11 s.e.).
Method
Lower bound
Random
1.915 + 0.032
Equal interval
2.669 0.023
MINEBED
3.400 + 0.001
SG-BOED
3.752 0.020
iDAD (NWJ)
3.869 + 0.001
iDAD (InfoNCE)
3,915 = 0.020
Figure further demonstrates important qualitative results for this model. Figure Sa) shows different epidemic trajectories, i.e. the number of infected I(r) people as function of measurement time whilst 5b) plots their corresponding designs obtained from the learned iDAD policy. Importantly, diseases with a significantly different profile, e.g. a slow (R = 1.3) or fast (R = 8.0) spread result in different sets of optimal designs, highlighting the adaptivity of iDAD. Finally, Figure 5c) shows an example posterior distribution estimate from the learnt iDAD critic network, which we see is consistent with the ground truth parameters.
Discussion
Limitations. The benefit that iDAD can be used live experiments comes at the cost of substantial training that can be computationally expensive. However, this is mitigated by its amortization of the adaptive design process, such that only one network needs training, even if we have multiple experiment instances. The cost-performance trade-off can also be directly controlled by judicious choices architecture and the amount of training performed. Another natural limitation is that the use of gradients naturally restricts the approach to continuous design settings, something which future work might look to address.
Conclusions. In this paper we introduced iDAD-tth the first posecy-based adaptive BOED method that can be applied implicit models. By training a design network without likelihoods upfront, iDAD is thus the first method that allows real-time adaptive experiments for simulator based models. In our experiments, iDAD performed significantly better than all likelihood- frre baselines. Further, by using models where the likelihood is available as a test bed, we found that it was able to almost adaptive approach, which acts as an upper bound on what
match the analogous likelihood-baat might be achieved without access to the likelihood itself. In conclusion, we believe iDAD marks step change in Bayesian experimental design for implicit models, allowing designs to be proposed quickly, adaptively, and non-myopically during the live experiment.
10

---

Acknowledgments and Disclosure of Funding
DRI supported by EPSRC through the Modern Statistics and Statistical Machine Learning (StatML) CDT programme, grant no. EP/S023151/1. AF gratefully acknowledges funding from EPSRC grant no. EP/N509711/1. SK was supported in part by the EPSRC Centre for Doctoral Training in Data Science, funded by the UK Engineering and Physical Sciences Research Council (grant EP/L016427/I) and the University of Edinburgh.
References
[1] Edward J. Allen, Linda J. S. Allen, Armando Arciniega, and Priscilla E. Greenwood. Construc- tion of equivalent stochastic differential equation models. Stochastic Analysis and Applications, 26(2):274-297, 2008
[2] Linda J.S. Allen. primer on stochastic epidemic models: Formulation, numerical simulation, and analysis. Infectious Disease Modelling, 2(2):128-142, 2017. ISSN 2468-0427. doi:
http:://oi.org// 101110066.........m 2017 .00.0011.
[3] Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. arXiv preprint arXiv:1906.000 00910, 2019.
[4] David Barber and Felix Agakov. The im algorithm: A variational approach to information maximization. In Proceedings of the 16th International Conference on Neural Information Processing Systems, NIPS 03, page 201-208, Cambridge, MA, USA, 2003. MIT Press.
[5] Atilim Gunes Baydin, Barak A Pearlmutter, Alexey Andreyevich Radul, and Jeffrey Mark Siskind. Automatic differentiation machine learning: a survey, Journal of machine learning research, 18, 2018
[6] Eli Bingham, Jonathan P Chen, Martin Jankowiak, Fritz Obermeyer, Neeraj Pradhan, Theofanis Karaletsos, Rohit Singh, Paul Szerlip, Paul Horsfall, and Noah Goodman. Pyro: Deep universal probabilistic programming. Journal of Machine Learning Research, 2018.
[7] Kathryn Chaloner and Isabella Verdinelli. Bayesian experimental design: A review. Statistical Science, pages 273-304, 1995.
[8] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 1597-1607. PMLR, 13-18 Jul 2020.
[9] Yanzhi Chen, Dinghuai Zhang, Michael U. Gutmann, Aaron Courville, and Zhanxing Zhu. Neural approximate sufficient statistics for implicit models. In International Conference on Learning Representations, 2021.
[10] Alex R Cook, Gavin J Gibson, and Christopher A Gilligan. Optimal observation times in experimental epidemic processes. Biometrics, 64(3):860-868, 2008
[11] Kyle Cranmer, Johann Brehmer, and Gilles Louppe. The frontier of simulation-based inference. Proceedings of the National Academy of Sciences, 2020.
[12] Mahasen B. Dehideniya, Christopher C. Drovandi, and James M. McGree. Optimal bayesian design for discriminating between models with intractable likelihoods in epidemiology. Computationnl Statistics & Data Analysis, 124:277-2977 2018. ISSN 0167-9473. doi: 1ttp:/.....1/......aa.0 8..03.004.
[13] Jacob Devlin, Ming Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, volume 1, pages 4171-4186, 2019.
[14] Yann Dubois, Jonathan Gordon, and Andrew YK Foong. http://yanndubs.rm wwtp:duuuu..ii.buuuuuge io//eeraal- Faaocess-Faailyy,, September 2020.
Neural Process Family.

---

[15] Adam Foster, Martin Jankowiak, Elias Bingham, Paul Horsfall, Yee Whye Teh, Thomas Rainforth, and Noah Goodman. Variational Bayesian Optimal Experimental Design. In Advances in Neural Information Processing Systems 32, pages 14036-14047. Curran Associates, Inc., 2019.
[16] Adam Foster, Martin Jankowiak, Matthew O' OeMaar, Yee Whye Teh, and Tom Rainforth. A unified stochastic gradient approach to designing bayesian-optimal experiments. In International Conference on Artificial Intelligence and Statistics, pages 2959-2969. PMLR, 2020.
[17] Adam Foster, Desi R Ivanova, llyas Malik, and Tom Rainforth. Deep adaptive design: Amortiz- ing sequential bayesian experimental design. Proceedings of the 38th International Conference on Machine Learning (ICML), PMLR 139, 2021.
[18] Andrew Gelman, John B Carlin, Hal S Stern, David B Dunson, Aki Vehtari, and Donald B Rubin. Bayesian data analysis. Chapman and Hall/CRC, 2013.
[19] Matthew Graham and Amos Storkey. Asymptotically exact inference in differentiable generative models. In Artificial Intelligence and Statistics, pages 499-508. PMLR, 2017.
[20] PeterJ. Green, Krzysztof Latuszynski. Marcelo Pereyra, and Christian P. Robert. Bayesian computation: a summary of the current state, and samples backwards and forwards. Statistics and Computing, 25(4):835-866 862, 2015. doi: 10. 012--115-557-5.
[21] Markus Hainy, Christopher C. Drovandi, and James M. McGree. Likelihood- extensions for Bayesian sequentially designed experiments. Joachim Kunert, Christine H. Müller, and Anthony C. Atkinson, editors, mODa // Advances in Model-Oriented Design and Analysis, pages 153-161. Springer International Publishing, 2016.
[22] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1785-1780, 1997.
[23] Xun Huan and Youssef Marzouk. Gradient- based stochastic optimization methods in bayesian experimental design. International Journal for Uncertainty Quantification, 4(6), 2014.
[24] Xun Huan and Youssef M Marzouk. Simulation- based optimal bayesian experimental design for nonlinear systems. Journal of Computational Physics, 232(1):288-317, 2013.
[25] Cheng Zhi Anna Huang. Ashish Vaswani, Jakob Uszkoreit, Noam Shazeer, Ian Simon, Curtis Hawthorne, Andrew M. Dai, Matthew D. Hoffman, Monica Dinculescu, and Douglas Eck. Music transformer: Generating music with long-term structure, 2019. ISSN 23318422.
[26] Diederík P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 2014.
[27] S. Kleinegesse and M.U. Gutmann. Efficient Bayesian experimental design for implicit models. In Kamalika Chaudhuri and Masashi Sugiyama, editors. Proceedings of the International Conference Artificial Intelligence and Statistics (AISTATS), volume 89 of Proceedings of Machine Learning Research, pages 1592. PMLR, 2019.
[28] Steven Kleinegesse and Michael Gutmann. Bayesian experimental design for implicit models by mutual information neural estimation. In Proceedings of the 37th International Conference on Machine Learning, Proceedings of Machine Learning Research, pages 5316-5326. PMLR,
2020.
[29] Steven Kleinegesse and Michael U. Gutmann. Bradient-based bayesian experimental design for implicit models using mutual information lower bounds. arXiv preprint urXiv:2105.04377 2021.
[30] Steven Kleinegesse, Christopher Drovandi, and Michael U. Gutmann. Sequential Bayesian Experimental Design for Implicit Models via Mutual Information. Bayesian Analysis, pages 30, 2021. doi: 10. 221420-BA122
[31] Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres. Quantifying the carbon emissions of machine learning, arXiv preprint 1rX1v:1910.09700,00 2019.
12

---

[32] Dennis V Lindley. On measure of the information provided by an experiment. The Annals of Mathematical Statistics, pages 986-1005, 1956.
[33] J. Lintusaari, M.U. Gutmann, R. Dutta, S. Kaski, and J. Corander. Fundamentals and recent developments in approximate Bayesian computation. Systematic Biology, 66(1))c66--88) January 2017.
[34] David McAllester and Karl Stratos. Formal limitations on the measurement of mutual informa- tion. In Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learming Research, pages 875-884. PMLR, 2020.
[35] Shakir Mohamed, Mihacla Rosca, Michael Figurnov, and Andriy Mnih. Monte carlo gradient estimation in machine learning. Journal of Machine Learning Research, 21(132):1-62, 2020.
[36] Jay I Myung, Daniel R Cavagnaro, and Mark Pitt. A tutorial on adaptive design optimization. Journal of mathematical psychology, 57(3-4):53-67, 2013.
[37] Xuanlong Nguyen, Martin Wainwright, and Michael I. Jordan. Estimating divergence functionals and the likelihood ratio by convex risk minimization. IEEE Transactions on Information Theory, 56(1 1), 2010. ISSN 00189448. doi: 10.1109/11.22210000663
[38] Antony Overstall and James McGree. Bayesian Design of Experiments for Intractable Likeli- hood Models Using Coupled Auxiliary Models and Multivariate Emulation. Bayesian Analysis, 15(1):103 131, 2020. doi: 10. 10.1224/19-BA. 1444
[39] Emilio Parisotto, Francis Song. Jack Rae, Razvan Pascanu, Caglar Gulcehre, Siddhant Jayaku- mar, Max Jaderberg. Raphaël Lopez Kaufmman, Aidan Clark, Seb Noury, Matthew Botvinick. Nicolas Heess, and Raia Hadsell. Stabilizing transformers for reinforcement learning. In Pro- ceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 7487-7498. PMLR, 2020.
[40] Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In International Conference Machine Learning, pages 4055 4064. PMLR, 2018
[41] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vito,,, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances Neural Information Processing Systems 32, pages 8024-8035. Curran Associates, Inc., 2019.
[42] Ben Poole, Sherjil Ozair, Aäron van den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. In International Conference Machine Learning, pages 5171-5180, 2019.
[43] David J. Price, Nigel Bean, Joshua V. Ross, and Jonathan Tuke. On the efficient determination of optimal bayesian experimental designs using abc: A case study in optimal observation of epidemics. Journal of Statistical Planning and Inference, 172:1-15 May 2016.
[44] David Price, Nigel G Bean, Joshua V Ross, and Jonathan Tuke. An induced natural selection heuristic for finding optimal bayesian experimental designs. Computational Statistics & Data Analysis, 126:1 124, 2018.
[45] Tom Rainforth. Automating Inference, Learning, and Design using Probabilistic PriiiistttiPPPrrmamia Programming. PhD thesis, University of Oxford, 2017.
[46] Tom Rainforth, Rob Cornish, Hongseok Yang, Andrew Warrington, and Frank Wood. On nesting monte carlo estimators. In International Conference on Machine Learning, pages 4267-4276. PMLR, 2018
13

---

[47] Prajit Ramachandran, Niki Parmar, Ashish Vaswani, Irwan Bello, Anselm Levskaya, and Jon Shlens. Stand-alone self-attention vision models. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[48] Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In Proceedings of the 31st International Conference Machine Learning. volume 32, pages 1278-1286, 2014.
[49] Herbert Robbins and Sutton Monro. A stochastic approximation method. The annals of mathematical statistics, pages 400-407, 1951.
[50] Elizabeth G. Ryan, Christopher C. Drovandi, M. Helen Thompson, and Anthony N. Pettitt. Towards bayesian experimental design for nonlinear models that require large number of sampling times. Computational Statistics & Data Analysis, 70:45-60 60, 2014. ISSN 0167-9473. doi: https::/doi.orr/10.. 1016000.111..... 111..00001111....21111111111 cww..ccc............. 0.7.
[51] Elizabeth G Ryan, Christopher C Drovandi, James M McGree, and Anthony N Pettitt. A review of modern computational algorithms for bayesian optimal design. International Statistical Review, 84(1):128-155) 2016.
[52] Ben Shababo, Brooks Paige, Ari Pakman, and Liam Paninski. Bayesian inference and online experimental design for mapping neural microcircuits. In Advances in Neural Information Processing Systems, pages 1304-1312, 2013.
[53] Xiaohong Sheng and Yu Hen Hu. Maximum likelihood multiple-source localization using acoustic energy measurements with wireless sensor networks. IEEE Transactions on Signal Processing, 2005. ISSN 1053587X. doi: 10.1 10.9/TTS.20048888933
[54] S.A. Sisson, Y. Fan, and M. Beaumont. Handbook of Approximate Bayesian Computation. Chapman & Hall/CRC Handbooks of Modern Statistical Methods. CRC Press, 2018 ISBN 9781351643467.
[55] Jiaming Song and Stefano Ermon. Understanding the limitations of variational mutual informa- tion estimators. In International Conference on Learning Representations, 2020.
[56] Owen Thomas, Ritabrata Dutta, Jukka Corander, Samuel Kaski, and Michael U Gutmann. Likelihood-free inference by ratio estimation. arXiv preprint 1rXiv:1611.100 10242, 2016.
[57] Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv preprint arXiv: 003448, 2018
[58] Joep Vanlier, Christian A Tiemann, Peter AJ Hilbers, and Natal AW van Riel. A Bayesian approach to targeted experiment design. Bioinformatics, 28(8):1 (8(8):11361118 136- 2012.
[59] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 9998-6008 2017.
[60] Benjamin T Vincent and Tom Rainforth. The DARC toolbox: automated, flexible, and efficient delayed and risky choice experiments using bayesian adaptive design. PsyArXiv preprini, 2017.
[61] M. Zaharia, Andrew Chen, A. Davidson, A. Ghodsi, S. Hong, A. Konwinski, Siddharth Murch- ing, Tomas Nykodym, Paul Ogilvie, Mani Parkhe, Fen Xie, and Corey Zumar. Accelerating the machine learning lifecycle with MLflow. IEEE Data Eng. Bull., 41:39 45, 2018.
[62] Manzil Zaheer, Satwik Kottur, Siamak Ravanbhakhsh, Barnabás P6czos, Ruslan Salakhutdinov, and Alexander J Smola. Deep sets. In Proceedings of the 31st International Conference Neural Information Processing Systems, NIPS' 17, 2017.
[63] Jiaxin Zhang, Sirui Bi, and Guannan Zhang. A stochastic approximate gradient ascent method for Bayesian experimental design with implicit models. In The 24nd International Conference on Artificial Intelligence and Statistics, 2021.
14


Reviews:
[{"review_id": "cidH7IUGuAt", "reviewer": "Reviewer_k7ct", "summary": "This paper proposes iDAD, an amortized learning framework for implicit (likelihood-free) experimental design. The iDAD framework extends the previously proposed deep adaptive design (DAD) method (Foster et al., 2013), which is an amortized learning framework for experiment design in likelihood-based (i.e. non-implicit) models, which learns a policy network offline that can then be deployed online for fast experimental design. Additionally, the proposed iDAD framework aims to alleviate the constraint in DAD that requires conditionally independent experiments.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is written quite clearly and does a good job in providing context for the proposed iDAD method. As such, the contributions of this paper are easy to follow. However, there are a few places where I think this paper could have provided additional details or empirical study, which I summarize below.\n\nI would appreciate more discussion and experimental evaluation on the cost/difficulty of training the policy network to an accurate level under a greater variety of settings. For example, are there conditions, in terms of dimensionality of the input space, dimensionality of the parameter space, query budget or other factors, in which the cost of training an accurate network becomes prohibitive? It would be ideal to include an ablation study that investigates the settings where training becomes difficult, and show the rate at which performance suffers in these conditions, relative to non-amortized methods.\n\nRelated to this, I think this paper would benefit from including a discussion (or even some experimental evaluation) on situations where there are (slight) errors with the design policy network—i.e. showing how much will minor errors affect the performance of this amortized method in practice, and detailing any strategies to mitigate any issues caused by these errors.\n\nThe iDAD method seems to still have the restriction that one is able to compute the derivative of y with respect to the design point (e.g. via automatic differentiation). While there were a few examples shown of likelihood-free BOED situations where this is possible, it would be great to try and better characterize the scope or types of models that this applies to, and any examples of cases where the framework is limited due to this requirement.\n\nFinally, as the proposed work is motivated by the existing DAD method, I feel that this paper would be improved if there were more specific examples in the introduction of implicit BOED settings where the proposed methods provide advantages over the DAD (and other non-implicit BOED) methods.\n\n—————————————————\n\n**Update:** I thank the authors for their detailed response to my review, and for providing the additional empirical results. I believe these have sufficiently answered all of my questions and concerns that I described above.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This paper includes a section on limitations in the Discussion section, which is a nice aspect of the paper.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "HBX5JcU3Q8i", "reviewer": "Reviewer_8e2K", "summary": "In this work, the authors extend the DAD method by using implicit models to overcome some weaknesses of DAD. The authors further derive several bounds to design re-parametrization gradients. The authors show that the proposed method outperforms existing methods in several Bayesian experimental design tasks. ", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: It is new to use implicit models in the setting of Bayesian experimental design. This work is a novel combination of implicit models and DAD.\n\nQuality: It seems that the proposed method is technically sound and promising. I do not check the correctness of the proofs in the appendix. The experimental results also support the claim.\n\nClarity: This submission is easy to read and well-organized.\n\nSignificance: I think this work extends the scope of DAD and it could be useful for Bayesian experimental design tasks.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "6leqJrVMXAv", "reviewer": "Reviewer_nc49", "summary": "This paper proposes a method for Bayesian Optimal Experiment Design (BOED) which uses a policy network to decide which experiment to perform next based on the history of past experiments and their outcomes. The goal is to enable experiments to be chosen quickly for real-time applications. The method trains a policy network in simulation, and does not require assumptions necessary for a prior method (DAD), which also trains a policy network in simulation but requires additional assumptions such as the availability of a likelihood model of the history given the parameters and policy. The present approach removes this requirement by using variational approximations together with a reparameterization which enables optimization with gradient descent. Experiments are done in a 2D location finding environment, a pharmacokinetics model and an epidemiology simulation. Results show that the proposed method almost matches the performance of an exact method which has access to exact likelihood on the first task, and does better than baselines on all the tasks while being fast to evaluate. ", "questions": "", "limitations": "", "rating": 7, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is very much outside my research area and I’m not familiar with the current state of the art in the area so my review is an educated guess. From what I can tell the paper addresses an important question, which is being able to perform experiments in real time (they give the example of tracking the source of a chemical spill which I found to be a good motivating example) for a general class of problems which only require the ability to simulate from the environment and compute gradients of the measurement wrt design. This is still not totally general since not all processes are differentiable, which should be mentioned somewhere The paper is very well written and did a good job of making the main arguments accessible. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, they mention the substantial computational cost of pretraining.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5"}, "extra_text": {}}, {"review_id": "2qqD8z4TwxM", "reviewer": "Reviewer_B4Q3", "summary": "The paper tackles the problem of adaptive Bayesian experimental design. Specifically, the authors try to relax the conditionally independent experiments and the explicit likelihood of the policy-based adaptive design model proposed by Foster. Code and supplemental materials are provided. \n\nThere are 3 contributions:\n- By leveraging the advance in variational mutual information, the authors can establish a suitable likelihood-free objective in the implicit setting. \n- The authors theoretically show that they can obtain a unified information objective without conditionally independent experiments. \n- Based on those insights, the authors propose the iDAD algorithm, which is much based on the previous DAD work. \n", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Main Strengths:\n- The paper makes a very good connection between the difficulties and limitations of the previous  DAD models and how they derive the idea of new modifications. This shows a very clear motivation of the authors in this paper.\n- The idea of using a critic to derive a lower bound for the objective function and optimize this lower bound at the same time with the objective function is interesting, somewhat similar to the development of actor-critic from Reinforce. This mechanism helps to extend iDAD to various applications.\n- The authors provide a solid theoretical foundation for their proposed method.  The author successfully eliminates conditional independence which is a restriction of the previous DAD model.\n- All the proofs are clear and easy to understand although the idea of the proofs is not new. Overall the paper is well presented and easy to follow; the structure of the paper is similar to DAD (which is not a bad thing). \n\nWeakness: \n- The method requires a lot of pre-training which can be computationally expensive.\n- The performance of iDAD is still worse than that of DAD in many experiments shown. Probably the authors need to reserve more space to do more ablation and analyze why it is the case. For example, when claiming there is a “cost-performance trade-off”, we also want to see when the performances peak, do they beat that of DAD and why?   \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors did not provide limitations and societal impact directly derived from this work. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper is on Bayesian experimental design, building on the recently proposed Deep Adaptive Design (DAD). The paper's contribution is an extension iDAD (I for implicit) that does not require explicit likelihoods, does not require that experiments are conditionally independent, and is fast enough for real-time applications. The paper is interesting and thorough.
Decision: Accept (Poster)