Implicit Deep Adaptive Design: Policy–Based Experimental Design without Likelihoods
Year: 2021

Abstract:
We introduce implicit Deep Adaptive Design (iDAD), a new method for performing adaptive experiments in real-time with implicit models. iDAD amortizes the cost of Bayesian optimal experimental design (BOED) by learning a design policy network upfront, which can then be deployed quickly at the time of the experiment. The iDAD network can be trained on any model which simulates differentiable samples, unlike previous design policy work that requires a closed form likelihood and conditionally independent experiments. At deployment, iDAD allows design decisions to be made in milliseconds, in contrast to traditional BOED approaches that require heavy computation during the experiment itself. We illustrate the applicability of iDAD on a number of experiments, and show that it provides a fast and effective mechanism for performing adaptive design with implicit models.

heading: 1 Introduction
text: Designing experiments to maximize the information gathered about an underlying process is a key challenge in science and engineering. Most such experiments are naturally adaptive—we can design later iterations on the basis of data already collected, refining our understanding of the process with each step [36, 45, 51]. For example, suppose that a chemical contaminant has accidentally been released and is rapidly spreading; we need to quickly discover its unknown source. To this end, we measure the contaminant concentration level at locations ξ1, . . . , ξT (our experimental designs), obtaining observations y1, . . . , yT . Provided we can perform the necessary computations sufficiently quickly, we can design each ξt using data from steps 1, . . . , t− 1 to narrow in on the source. Bayesian optimal experimental design (BOED) [7, 32] is a principled model-based framework for choosing designs optimally; it has been successfully adopted in a diverse range of scientific fields [52, 58, 60]. In BOED, the unknown quantity of interest (e.g. contaminant location) is encapsulated by a parameter θ, and our initial information about it by a prior p(θ). A simulator, or likelihood, model y|θ, ξ describes the relationship between θ, our controllable design ξ, and the experimental outcome y. To select designs optimally, the guiding principle is information maximization—we select the design that maximizes the expected (Shannon) information gained about θ from the data y, or, equivalently, that maximizes the mutual information between θ and y.
This naturally extends to adaptive settings by considering the conditional expected information gain given previously collected data. The traditional approach, depicted in Figure 1a, is to fit a posterior p(θ|ξ1:t−1, y1:t−1) after each iteration, and then select ξt in a myopic fashion using the one-step mutual information (see, e.g., [51] for a review). Unfortunately, this approach necessitates significant computation at each t and does not lend itself to selecting optimal designs quickly and adaptively.
Recently, Foster et al. [17] proposed an exciting alternative approach, called Deep Adaptive Design (DAD), that is based on learning design policies. DAD provides a way to avoid significant computation
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
Design
Observe
T iterations
Live experiment
Infer
(a) Traditional BOED: costly computations (design optimisation and parameter inference) are required at each iteration.
Design
Observe
T iterations L ive experim
ent
O ffl
in e
O ffl
in e
Infer
Policy
Critic
data
deploy
(b) Policy-based BOED using iDAD: a design policy and critic are learnt before the live experiment. The policy enables quick and adaptive experiments, the critic assists likelihood-free inference.
Figure 1: Overview of adaptive BOED approaches applicable to implicit models.
at deployment-time by, prior to the experiment itself, learning a design policy network that takes past design-outcome pairs and near–instantaneously returns the design for the next stage of the experiment. The required training is done using simulated experimental histories, without the need to estimate any posterior or marginal distributions. DAD further only needs a single policy network to be trained for multiple experiments, further allowing for amortization of the adaptive design process. Unfortunately, DAD requires conditionally independent experiments and only works for the restricted class of models that have an explicit likelihood model we can simulate from, evaluate the density of, and calculate derivatives for, substantially reducing its applicability.
To address this shortfall, we instead consider a far more general class of models where we require only the ability to simulate y|θ, ξ and compute the derivative ∂y/∂ξ, e.g. via automatic differentiation [5]. Such models are ubiquitous in scientific modelling and include differentiable implicit models [19], for which the likelihood density p(y|θ, ξ) is intractable. Examples include mixed effects models [15, 18], various models from chemistry and epidemiology [1], the Lotka Volterra model used in ecology [19], and models specified via stochastic differential equations (such as the SIR model [10]).
To perform rapid adaptive experimentation with this large class of models, we introduce implicit Deep Adaptive Design (iDAD), a method for learning adaptive design policy networks using only simulated outcomes (see Figure 1b). To achieve this, we introduce likelihood-free lower bounds on the total information gained from a sequence of experiments, which iDAD utilizes to learn a deep policy network. This policy network amortizes the cost of experimental design for implicit models and can be run in milliseconds at deployment-time. To train it, we show how the InfoNCE [57] and NWJ [37] bounds, popularized in representation learning, can be applied to the policy-based experimental design setting. The optimization of both of these bounds involves simultaneously learning an auxiliary critic network, bringing an important added benefit: it can be used to perform likelihood-free posterior inference of the parameters given the data acquired from the experiment.
We also relax DAD’s requirement for experiments to be conditionally independent, allowing its application in complex settings like time series data, and, through innovative architecture adaptations, also provide improvements in the conditionally independent setting as well. This further expands the model space for policy-based BOED, and leads to additional performance improvements.
Critically, iDAD forms the first method in the literature that can practically perform real-time adaptive BOED with implicit models: previous approaches are either not fast enough to run in real-time for non-trivial models, or require explicit likelihood models. We illustrate the applicability of iDAD on a range of experimental design problems, highlighting its benefits over existing baselines, even finding that it often outperforms costly non-amortized approaches. Code for iDAD is publicly available at https://github.com/desi-ivanova/idad.

heading: 2 Background
text: The BOED framework [32] begins by specifying a Bayesian model of the experimental process, consisting of a prior on the unknown parameters p(θ), a set of controllable designs ξ, and a data generating process that depends on them y|θ, ξ; as usual in BOED, we assume that p(θ) does not depend on ξ. In this paper, we consider the situation where y|θ, ξ is specified implicitly. This means
that it is defined by a deterministic transformation, f(ε; θ, ξ), of a base (or noise) random variable, ε, that is independent of the parameters and the design; e.g., ε ∼ N (ε; 0, I). The function f is itself often not known explicitly in closed form, but is implemented as a stochastic computer program (i.e. simulator) with input (θ, ξ) and ε corresponding to the draws from the underlying random number generator (or equivalently the random seed). Regardless, the resulting induced likelihood density p(y|θ, ξ) is still generally intractable, but sampling y|θ, ξ is possible. Having acquired a design-outcome pair (ξ, y), we can quantify the amount of information we have gained about θ by calculating the reduction in entropy from the prior to the posterior. We can further assess the quality of a design ξ before acquiring y, by computing the expected reduction in entropy with respect to the marginal distribution of the outcome, p(y|ξ) = Ep(θ)[p(y|θ, ξ)]. The resulting quantity, called the expected information gain (EIG), is of central interest in BOED and is defined as
I(ξ) := Ep(θ)p(y|θ,ξ) [ log
p(θ|ξ, y) p(θ)
] = Ep(θ)p(y|θ,ξ) [ log
p(y|θ, ξ) p(y|ξ)
] . (1)
Note that I(ξ) is equivalent to the mutual information (MI) between the parameters θ and data y when performing experiment ξ. The optimal ξ is then the one that maximises the EIG, i.e. ξ∗ = argmaxξ I(ξ). Performing this optimization is a major computational challenge since the information objective is doubly intractable [46]. For implicit models, the cost becomes even greater as the likelihood is also not available in closed form, so estimating it, along with the marginal likelihood p(y|ξ), is already itself a major computational problem [11, 20, 33, 54]. Jointly optimizing the design variables for all undertaken experiments at the same time using (1) is called static experimental design. In practice, however, we are often more interested in performing multiple experiments adaptively in a sequence ξ1, . . . , ξT , so that the choice of each ξt can be guided by past experiments, namely the corresponding history ht−1 := {(ξi, yi)}i=1:t−1. The typical approach in such settings is to sequentially perform (approximate) posterior inference for θ|ht−1, followed by a one-step look ahead (myopic) BOED optimization that conditions on the observed history. In other words, to determine the designs ξ1, . . . , ξT , we sequentially optimize the objectives
Iht−1(ξt) := Ep(θ|ht−1)p(yt|θ,ξ,ht−1) [ log
p(yt|θ, ξ, ht−1) p(yt|ξ, ht−1)
] , t = 1, . . . , T. (2)
However, such approaches incur significant computational cost during the experiment itself, particularly for implicit models [16, 21, 30]. This has critical consequences: in most cases they cannot be run in real-time, undermining one’s ability to use them in practice.

heading: 2.1 Policy-based adaptive design with likelihoods
text: For tractable likelihood models, Foster et al. [17] proposed a new framework, called Deep Adaptive Design (DAD), for adaptive experimental design that avoids expensive computations during the experiment. To achieve this, they introduce a parameterized deterministic design function, or policy, πϕ that takes the history ht−1 as input and returns the design ξt = πϕ(ht−1) to be used for the next experiment as output. This set-up allows them to consider the objective
IT (πϕ) = Ep(θ)p(hT |θ,πϕ) [ T∑ t=1 Iht−1(ξt) ] , ξt = πϕ(ht−1), (3)
which crucially depends on the policy π rather than the individual design ξt. Learning a policy up-front, rather than designs, is what allows adaptive experiments to be performed in real-time.
Under the assumption that yt is independent of ht−1 conditional on the parameters θ and the design ξt, i.e. p(yt|θ, ξt, ht−1) = p(yt|θ, ξt), Foster et al. [17] showed that the objective can be simplified to
IT (πϕ) = Ep(θ)p(hT |θ,πϕ) [ log
p(hT |θ, πϕ) p(hT |πϕ)
] , p(hT |θ, πϕ) = T∏ t=1 p(yt|θ, ξt). (4)
To deal with the marginal p(hT |πϕ) in the denominator, they then derived several optimizable lower bounds on IT (πϕ), such as the sequential Prior Contrastive Estimation (sPCE) bound
LsPCET (πϕ, L) = Ep(θ0)p(hT |θ,πϕ)p(θ1:L)
[ log
p(hT |θ0, πϕ) 1
L+1 ∑L ℓ=0 p(hT |θℓ, πϕ)
] ≤ IT (πϕ) ∀L ≥ 1. (5)
The parameters of the policy ϕ, which takes the form of a deep neural network, are now learned prior to the experiment(s) using stochastic gradient ascent on this bound with simulated experimental histories. Design decisions can then be made using a single forward pass of πϕ during deployment. Unfortunately, training the DAD network by optimizing (5) requires the likelihood density p(hT |θ, π) to be analytically available—an assumption that is too restrictive in many practical situations. The architecture for DAD also assumes conditionally independent designs, which is unsuitable in some settings like time-series data. Our method lifts both of these restrictions.

heading: 3 Implicit Deep Adaptive Design
text: We have seen that the traditional step-by-step approach to adaptive design for implicit models [16, 21, 30] is too costly to deploy for most applications, whilst the only existing policy-based approach, DAD [17], makes restrictive assumptions that prevent it being applied to implicit models. We aim to relax the restrictive assumptions of the latter, making policy-based BOED applicable to all models where we can sample from y|θ, ξ and compute the derivative ∂y/∂ξ, a strict superset of the class of models that can be handled by DAD. This requires new training objectives for the policy network that do not involve an explicit likelihood and are not based on conditionally independent experiments, along with new architectures that work for non-exchangeable models like time series.

heading: 3.1 Information lower bounds for policy-based experimental design without likelihoods
text: To establish a suitable likelihood-free training objective for the implicit setting, our high-level idea is to leverage recent advances in variational MI [see 42, for an overview], which have shown promise for static BOED [16, 28, 29]. While using these bounds in the traditional framework of (2) would not permit real-time experiments, one could consider a naive application of them to the policy objective of (3) by replacing each Iht−1 with a suitable variational lower bound that uses a ‘critic’ Ut : Ht−1 ×Θ → R to avoid explicit likelihood evaluations, where Ht−1 and Θ are the spaces of histories and parameters respectively. An effective critic successfully encapsulates the true likelihood, tightening the bound. Although its form depends on the choice of bound, all critics are parametrized and trained in the same way, namely by a neural network Uϕt which is optimized to tighten the bound. Unfortunately, replacing each Iht−1 involves learning T such critic networks and requires samples from all posteriors p(θ|ht−1), which will typically be impractically costly. To avoid this issue, we show that we can obtain a unified information objective similar to (4), even without conditionally independent experiments. The following proposition therefore marks the first key milestone in eliminating the restrictive assumptions of [17], by establishing a unified objective without intermediate posteriors that is valid even when the model itself changes between time steps. Proposition 1 (Generalized total expected information gain). Consider the data generating distribution p(hT |θ, π) = ∏ t=1:T p(yt|θ, ξt, ht−1), where ξt = π(ht−1) are the designs generated by the policy and, unlike in (4), yt is allowed to depend on the history ht−1. Then we can write (3) as
IT (π) = Ep(θ)p(hT |θ,π) [log p(hT |θ, π)]− Ep(hT |π) [log p(hT |π)] . (6)
Proofs are presented in Appendix A. The advantage of (6) is that we can draw samples from p(θ)p(hT |θ, π) simply by sampling our model and taking forward passes through the design network. However, neither of the densities p(hT |θ, π) and p(hT |π) are tractable for implicit models. To side-step this intractability, we observe that IT (π) takes an analogous form to a MI between θ and hT . For measure-theoretic reasons, namely because the ξ1:T are deterministic given y1:T (see Appendix A for a full discussion), it is not the true MI. However, the following two propositions show that we can treat IT (π) as if it were this MI. Specifically, we show that the InfoNCE [57] and NWJ [37] bounds on the MI can be adapted to establish tractable lower bounds on our unified objective IT (π). These two bounds both utilize a single auxiliary critic network Uψ that is trained simultaneously with the design network. Proposition 2 (NWJ bound for implicit policy-based BOED). For a design policy π and a critic function U : HT ×Θ → R, let
LNWJT (π, U) := Ep(θ)p(hT |θ,π) [U(hT , θ)]− e −1Ep(θ)p(hT |π) [exp(U(hT , θ))] , (7)
then IT (π) ≥ LNWJT (π, U) holds for any U . Further, the inequality is tight for the optimal critic U∗NWJ(hT , θ) = log p(hT |θ, π)− log p(hT |π) + 1.
Algorithm 1: Implicit Deep Adaptive Design with (iDAD) Input: Differentiable simulator f , sampler for prior p(θ), number of experimental steps T Output: Design network πϕ, critic network Uψ while Computational training budget not exceeded do
Sample θ ∼ p(θ) and set h0 = ∅ for t = 1, ..., T do
Compute ξt = πϕ(ht−1) Sample εt ∼ p(ε) and compute yt = f(εt; ξt, θ, ht−1) Set ht = {(ξ1, y1), ..., (ξt, yt)}
end Estimate ∇ϕ,ψLT (πϕ, Uψ) as per (10) where LT is LNWJT (7) or LNCET (8) Update the parameters (ϕ, ψ) using stochastic gradient ascent scheme
end For deployment, use the deterministic trained design network πϕ to obtain a designs ξt directly.
Proposition 3 (InfoNCE bound for implicit policy-based BOED). Let θ1:L ∼ p(θ1:L)= ∏ i p(θi) be a set of contrastive samples where L ≥ 1. For design policy π and critic function U :HT×Θ→R, let
LNCET (π, U ;L) := Ep(θ0)p(hT |θ0,π)Ep(θ1:L)
[ log
exp(U(hT , θ0)) 1
L+1 ∑L i=0 exp(U(hT , θi))
] , (8)
then IT (π) ≥ LNCET (π, U ;L) for any U and L ≥ 1. Further, the optimal critic, U∗NCE(hT , θ) = log p(hT |θ, π)+ c(hT ) where c(hT ) is any arbitrary function depending only on the history, recovers the sPCE bound in (5); the inequality is tight in the limit as L→ ∞ for this optimal critic.
We propose these two alternative bounds due to their complementary properties: the NWJ bound can have large variance, but tends to be less biased. That is, the NWJ bound tends to be tighter for good critics, but is itself more difficult to reliably estimate and thus optimize. While the NWJ critic must learn to self-normalize, the InfoNCE bound avoids this issue but typically will not be tight for finite L even with an optimal critic (note LNCET ≤ log(L+ 1) [42]). Consequently, only the NWJ objective recovers the true optimal policy if our critic has infinite capacity and our optimization scheme is perfect, i.e. argmaxπmaxU LNWJT (π, U) = π∗ ̸= argmaxπmaxU LNCET (π, U ;L) in general, but it can be more difficult to work with in practice. We present a third bound that provides a potential solution to this, and further discuss the relative merits of the two bounds, in Appendix A.
We note that for both bounds the optimal critic does not depend on the learned policy. The final trained critic can be used to approximate the density ratio p(hT |θ, π)/p(hT |π) = p(θ|hT )/p(θ), either directly in the case of the NWJ critic, or via self-normalization for the InfoNCE bound. We can use this to help approximate the posterior over θ given the collected real data from the experiment. This means we can perform likelihood-free inference after training the critic, which extends previous results [28, 29] from the static to the adaptive policy-based setting.

heading: 3.2 Parameterization and gradient estimation
text: In practice, we represent the policy π and the criticU as neural networks, πϕ andUψ respectively, such that the lower bounds become a function L(πϕ, Uψ) of their parameters. By simultaneously optimizing L(πϕ, Uψ) with respect to both ϕ and ψ, we both learn a tight bound that accurately represents the true MI and a design policy network that produces high-quality designs under this metric.
We optimize these bounds using stochastic gradient methods [26, 49]. For this, we must account for the fact that the parameter ϕ affects the probability distributions with respect to which expectations are taken. We deal with this problem by utilizing the reparametrization trick [35, 48], for which we assume that design space Ξ and observation space Y are continuous. To this end, we first formalize the notion of a differentiable implicit model in the adaptive design setting as
yt = f(εt; ξt(ht−1), θ, ht−1), where θ ∼ p(θ), εt ∼ p(ε) ∀t ∈ {1, . . . , T} (9)
and we assume that we can compute the derivatives ∂f/∂ξ and ∂f/∂h. Interestingly, it is possible to use an implicit prior without access to the density p(θ), and we do not need access to ∂f/∂θ.
Under these conditions, we can express the bounds in terms of expectations that do not depend on ϕ or ψ, and hence move the gradient operator inside. For LNCET (πϕ, Uψ;L), for example, we have
∇ϕ,ψLNCET = Ep(θ0:L)p(ε1:T ) [ ∇ϕ,ψ log exp(Uψ(hT (ε1:T , πϕ), θ0)) 1
L+1 ∑L i=0 exp(Uψ(hT (ε1:T , πϕ), θi))
] . (10)
While each element of the history hT depends on ϕ in a possibly nested manner, we do not need to explicitly keep track of these dependencies thanks to automatic differentiation [5, 41].
Like DAD, our new method—which we call implicit Deep Adaptive Design (iDAD)—is trained with simulated histories hT = {(ξi, yi)}i=1:T prior to the actual experiment, allowing design decision to be made using a single forward pass during deployment. Unlike DAD, however, it does not require knowledge of the likelihood function, nor the assumption of conditionally independent designs, which significantly broadens its applicability. A summary of the iDAD approach is given in Algorithm 1.

heading: 3.3 Network architectures
text: The iDAD approach involves the simultaneous training of the policy πϕ and critic Uψ networks. It is essential to choose the neural architectures of these two components carefully to learn effective policies: poor choices of critic architecture will lead to loose, unrepresentative, bounds, while poor choices of policy architecture will directly lead to ineffective policies. Good choices of architecture need to balance flexibility with ease of training, and will typically require the incorporation of problem-specific inductive biases. A high-level summary of our architectures is shown in Figure 2.
The critic network, Uψ , takes a complete history hT and the parameter θ as input, and outputs a scalar. Our suggested architecture first encodes the two inputs separately to representations of the same dimension, using a history encoder, Eψh , and a parameter encoder, Eψθ , respectively. The output of the critic is then simply taken as their dot product Uψ(hT , θ) := Eψh(hT )
⊤Eψθ (θ); after training, the two encodings correspond to approximate sufficient statistics [9]. This setup corresponds to a separable critic architecture, as is commonly used in the representation learning literature [3, 8, 57]. While we use a simple MLP for Eψθ , the setup for Eψh varies with the context as we discuss below.
The policy network, πϕ, takes the available history, ht, as input, and outputs a design. Our suggested architecture makes use of a history encoder, Eϕh , of the same form as Eψh , except that it must now take in varying length inputs; its output remains a fixed dimensional embedding. We then pass this embedding through an MLP to produce the design ξt+1. At the next iteration of the experiment, the same policy network is then called again with the updated history ht+1 = ht ∪ {(ξt+1, yt+1)}. We use the same architecture for both history encoders, Eψh and Eϕh , but do not share network parameters between them. This architecture first individually embeds each design–outcome pair (ξt, yt) to a corresponding representation, rt, using a simple MLP that is shared across all time steps. The produced history encoding is then an aggregation of these representations, with how this is done depending on whether the experiments are conditionally independent, i.e. yt ⊥⊥ ht−1|θ, ξt, or not. Foster et al. [17] proved that if experiments are conditionally independent, then the optimal policy is invariant to the order of the history. We prove that the same is true for the critic in Proposition 5 in Appendix C. In our setup, we can exploit this result by using a permutation invariant aggregation strategy for {r1, . . . , rt} when conditional independence holds. The simplest approach to do this would be to use sum-pooling [62], as was done in DAD. However, to improve on this, we instead propose using a more advanced permutation invariant architecture based on self-attention [13, 25, 39, 47, 59], namely that of Parmar et al. [40]; we find this provides notable empirical gains. When conditional independence does not hold, this approach is no longer appropriate and we instead use an LSTM [22] for the aggregation. See Appendix C for further details.

heading: 4 Related work
text: Adaptive policy-based BOED has only recently been introduced [17] and has not yet been extended to implicit models—the gap that this work addresses. Previous approaches to adaptive experiments usually follow the two-step greedy procedure described in Section 2. Methods for MI/EIG estimation without likelihoods include the use of variational bounds [15, 16, 28] and ratio estimation [27, 30]; approximate Bayesian computation together with kernel density estimation [43]; and approximating the intractable likelihood first, for example via polynomial chaos expansion [24], followed by applying likelihood-based estimators, such as nested Monte Carlo [46]. The maximization step in more traditional methods tends to rely on gradient-free optimization, including grid-search, evolutionary algorithms [44], Bayesian optimization [15, 30], or Gaussian process surrogates [38]. More recently, gradient-based approaches have been introduced [15, 28], some of which allow the estimation and optimization simultaneously in a single stochastic-gradient scheme [16, 23, 29]. From a posterior estimation perspective, likelihood-free inference can be performed via approximate Bayesian computation [33, 54], ratio estimation [56], conventional MCMC for methods that make tractable approximation to the likelihood [23, 24], or as a byproduct of MI estimation [16, 27, 29, 30].
5 Experiments
We evaluate the performance of iDAD on a number of real-world experimental design problems and a range of baselines. A summary of all the methods that we consider is given in Table 1. Since we aim to perform adaptive experiments in real-time, we focus mostly on baselines that do not require significant computational time during the experiment. These include heuristic approaches that require no training, namely equal interval designs (when possible) and random
designs, as well as static BOED approaches, where we, non-adaptively, choose all the designs prior to the experiment by optimising the mutual information objective of Equation (1) with ξ = {ξ1, ..., ξT } and y = {y1, . . . , yT }. The static BOED approaches we consider are the MINEBED method of [28] and the likelihood-free ACE approach of [16], where we use the prior as a proposal distribution, referring to this baseline as SG-BOED. We also implement the expensive traditional non-amortized myopic strategy described in Section 2, for which we use the variational approach of [16], with the Barber-Agakov bound [4, 15], at each experiment step (see Appendix D.3 for details). Finally, where possible, we compare our method with DAD [17], in order to assess the performance gap that would arise if we had an analytic likelihood. This comparison is done primarily for evaluation purposes—because it has access to the likelihood density, DAD serves as an upper bound on the performance iDAD can achieve; one should use explicit likelihood methods whenever possible.
The main performance metric that we focus on is the total EIG, IT (π), as given in (6). In cases where the likelihood is available, we estimate the IT (π) using the sPCE lower bound in (5) and its sister upper bound, the sequential Nested Monte Carlo bound [sNMC; 17]. To ensure that the bounds are tight, we evaluate them with a large number of contrastive samples, i.e. L ≥ 105. Where the likelihood is truly intractable, we assess the iDAD strategy in a more qualitative manner by looking at the optimal designs and approximate posteriors. For the adaptive experiments, we further consider the deployment time (i.e. the time required to propose a design), which is a critical metric for our aims. All deployment times exclude the time needed to determine the first experiment as it can be computed up-front, during the training phase. Timings for training the policy itself are given in Appendix D.

heading: 5.1 Location Finding
text: We first demonstrate our approach on the location finding experiment from [17]. Inspired by the acoustic energy attenuation model [53], this experiment involves finding the locations of multiple hidden sources, each emitting a signal with intensity that decreases according to the inverse-square law. The total intensity—a superposition of these signals—can be measured noisily at any location. The design problem is choosing where to measure the total signal in order to uncover the sources.
In Table 2 we can see that iDAD substantially outperforms all baselines including, perhaps surprisingly, the traditional (non-amortized) adaptive variational approach, despite its large computational cost shown Table 4. The poor performance of the variational approach is likely driven by the inability of the meanfield variational family to capture the highly nonGaussian true posterior, highlighting the detrimental effect that wrong posteriors can have on determining optimal designs when using the traditional sequential BOED approach.
Table 2 further shows that the performance gap to the likelihood-based DAD method is small, even as the dimension of the design and parameter space grows. Though the information gained by all methods decreases with the dimensionality, this is to be expected: in higher dimensions it is inherently more difficult to infer the relative direction of the sources from observing their intensity. Overall, this experiment demonstrates that iDAD is able to learn near-optimal amortized design policies without likelihoods, while being run in milliseconds at deployment.
Ablation: attention to history. We next assess the benefit of utilizing our proposed more sophisticated permutation invariant architectures, compared to the simple pooling of [62] used in [17]. Our approach incorporates attention layers into both networks that we train. This leads us to four possible combinations of network architectures. Table 3 compares the efficacy of the resulting design policies and strongly suggests that incorporating attention mechanisms in either and/or both networks improves performance, with inclusion in the design network particularly important.
We preform further ablation studies to investigate and demonstrate important properties of our method, such as its scalability with the number of experiments T , stability between different training runs and performance to errors in the design network (introduced by not training the network to convergence). Results and discussion are provided in Appendix D.4.4.

heading: 5.2 Pharmacokinetic model
text: Our next experiment is taken from the pharmacokinetics literature and has been studied in other recent works on BOED for implicit models [28, 63]. Specifically, we consider the compartmental model
of [50], for which the distribution of an administered drug through the body is governed by three parameters: absorption rate kα, elimination rate ke, and volume V , which form the parameters of interest, i.e. θ = (kα, ke, V ). Given T = 5 patients, the design problem is to adaptively choose blood sampling times, 0 ≤ ξt ≤ 24 hours, for each, measured from the the point the drug was administered (with patient 2 not being administered until after sampling patient 1 etc). Plausible concentration trajectories are shown in Figure 3a). Full details and further results are given in Appendix D.5.
We first qualitatively consider the design policy of iDAD (trained with the InfoNCE objective) in Figure 3b). As we have not yet observed any data, the optimal design for the first patient (bottom row) is the same for all θ. For the second patient, only guided by ξ1 and the outcome y1, iDAD is already able to distinguish between quickly and slowly decaying concentration trajectories: it proposes a significantly earlier measurement time for the quickly decaying trajectory (purple triangle, θq) and later time for the slowly decaying one (yellow diamond, θs). For the third patient, iDAD always targets the peak of the drug concentration distribution which is quite similar for all θ. Measurements for the last two patients are made soon after the drug has been administered (∼ 15− 30 min), when concentration levels increase rapidly, to capture information about how quickly the drug is absorbed.
To provide more quantitative assessment and compare to our baselines, we again consider the final EIG values as shown in Figure 3c). This reveals that the iDAD strategies perform best among the methods that are applicable to implicit models, confirming that the learnt policies propose superior designs. The performance gap to DAD, which relies on explicit likelihoods, is not statistically significant (at the 5% level) for iDAD trained with InfoNCE, while significant, but still small, for NWJ.
Finally, we consider the convergence of the iDAD networks under the different training objectives and compare to DAD for reference. As shown in Figure 4, although all three converge to approximately the same value, they do so at rather different speeds: while DAD requires about 5000 gradient updates, implicit methods need longer training and tend to exhibit higher variance, particularly NWJ.

heading: 5.3 SIR Model
text: In this experiment, we demonstrate our approach on an implicit model from epidemiology. Namely, we consider a formulation of the stochastic SIR model [10] that is based on stochastic differential equations (SDEs), as done by [29]. Here, individuals in a fixed population belong to one of three categories: susceptible, infected or recovered. Susceptible people can become infected and then recover, with the dynamics of these two events being governed by two model parameters—the infection rate β and the recovery rate γ. Our aim is to determine the optimal times τ at which to measure the number of infected people, I(τ), in order to estimate the two parameters. This implicit model is challenging because data simulation is expensive, since we need to solve many SDEs, and experimental designs have a time-dependency. See Appendix D.6 for full details.
We train a iDAD networks to perform T = 5 experiments and compare against random, equal interval, and static design baselines; DAD cannot be run because the problem corresponds to a true implicit model. Table 5 shows lower bound estimates on the MI and demonstrates that iDAD outperforms all compared methods. Note that a degree of caution is required when analysing the results, as they are influenced by unavoidable biases in the estimation process. Namely, a critic is still required to estimate the MI lower bound, and there may be variations in the effectiveness of these critics, with less effective ones corresponding to
looser bounds and therefore underestimating the true MI. Nonetheless, for other models where such checks are possible, we have found the bounds to be relatively tight, while, even if this turns out not to be the case here, the fact that the critics for the static approaches are easier to train should mean our relative evaluations for iDAD (and Random) are still conservative compared to the other baselines.
Figure 5 further demonstrates important qualitative results for this model. Figure 5a) shows different epidemic trajectories, i.e. the number of infected I(τ) people as a function of measurement time τ , whilst 5b) plots their corresponding designs obtained from the learned iDAD policy. Importantly, diseases with a significantly different profile, e.g. a slow (R = 1.3) or a fast (R = 8.0) spread result in different sets of optimal designs, highlighting the adaptivity of iDAD. Finally, Figure 5c) shows an example posterior distribution estimate from the learnt iDAD critic network, which we see is consistent with the ground truth parameters.

heading: 6 Discussion
text: Limitations. The benefit that iDAD can be used in live experiments comes at the cost of substantial training that can be computationally expensive. However, this is mitigated by its amortization of the adaptive design process, such that only one network needs training, even if we have multiple experiment instances. The cost–performance trade-off can also be directly controlled by judicious choices of architecture and the amount of training performed. Another natural limitation is that the use of gradients naturally restricts the approach to continuous design settings, something which future work might look to address.
Conclusions. In this paper we introduced iDAD—the first policy-based adaptive BOED method that can be applied to implicit models. By training a design network without likelihoods upfront, iDAD is thus the first method that allows real-time adaptive experiments for simulator-based models. In our experiments, iDAD performed significantly better than all likelihood-free baselines. Further, by using models where the likelihood is available as a test bed, we found that it was able to almost match the analogous likelihood-based adaptive approach, which acts as an upper bound on what might be achieved without access to the likelihood itself. In conclusion, we believe iDAD marks a step change in Bayesian experimental design for implicit models, allowing designs to be proposed quickly, adaptively, and non-myopically during the live experiment.
Acknowledgments and Disclosure of Funding
DRI is supported by EPSRC through the Modern Statistics and Statistical Machine Learning (StatML) CDT programme, grant no. EP/S023151/1. AF gratefully acknowledges funding from EPSRC grant no. EP/N509711/1. SK was supported in part by the EPSRC Centre for Doctoral Training in Data Science, funded by the UK Engineering and Physical Sciences Research Council (grant EP/L016427/1) and the University of Edinburgh.

Review summaries:
This paper proposes iDAD, an amortized learning framework for implicit (likelihood-free) experimental design. The iDAD framework extends the previously proposed deep adaptive design (DAD) method (Foster et al., 2013), which is an amortized learning framework for experiment design in likelihood-based (i.e. non-implicit) models, which learns a policy network offline that can then be deployed online for fast experimental design. Additionally, the proposed iDAD framework aims to alleviate the constraint in DAD that requires conditionally independent experiments.

In this work, the authors extend the DAD method by using implicit models to overcome some weaknesses of DAD. The authors further derive several bounds to design re-parametrization gradients. The authors show that the proposed method outperforms existing methods in several Bayesian experimental design tasks. 

The paper tackles the problem of adaptive Bayesian experimental design. Specifically, the authors try to relax the conditionally independent experiments and the explicit likelihood of the policy-based adaptive design model proposed by Foster. Code and supplemental materials are provided.   There are 3 contributions: - By leveraging the advance in variational mutual information, the authors can establish a suitable likelihood-free objective in the implicit setting.  - The authors theoretically show that they can obtain a unified information objective without conditionally independent experiments.  - Based on those insights, the authors propose the iDAD algorithm, which is much based on the previous DAD work.  

This paper proposes a method for Bayesian Optimal Experiment Design (BOED) which uses a policy network to decide which experiment to perform next based on the history of past experiments and their outcomes. The goal is to enable experiments to be chosen quickly for real-time applications. The method trains a policy network in simulation, and does not require assumptions necessary for a prior method (DAD), which also trains a policy network in simulation but requires additional assumptions such as the availability of a likelihood model of the history given the parameters and policy. The present approach removes this requirement by using variational approximations together with a reparameterization which enables optimization with gradient descent. Experiments are done in a 2D location finding environment, a pharmacokinetics model and an epidemiology simulation. Results show that the proposed method almost matches the performance of an exact method which has access to exact likelihood on the first task, and does better than baselines on all the tasks while being fast to evaluate.