Sourcerer: Sample-based Maximum Entropy Source Distribution Estimation
Authors: Julius Vetter
Guy Moss
Cornelius Schröder
Richard Gao
Jakob H. Macke
Venue: NeurIPS 2024 poster (2024)
Keywords: source distribution estimation
maximum entropy
Sliced-Wasserstein distance
empirical Bayes
simulation-based inference
Primary area: probabilistic_methods
TL;DR: We propose a sample-based, maximum entropy approach to the source distribution estimation problem.

Abstract: Scientific modeling applications often require estimating a distribution of parameters consistent with a dataset of observations - an inference task also known as source distribution estimation. This problem can be ill-posed, however, since many different source distributions might produce the same distribution of data-consistent simulations. To make a principled choice among many equally valid sources, we propose an approach which targets the maximum entropy distribution, i.e., prioritizes retaining as much uncertainty as possible. Our method is purely sample-based - leveraging the Sliced-Wasserstein distance to measure the discrepancy between the dataset and simulations - and thus suitable for simulators with intractable likelihoods. We benchmark our method on several tasks, and show that it can recover source distributions with substantially higher entropy than recent source estimation methods, without sacrificing the fidelity of the simulations. Finally, to demonstrate the utility of our approach, we infer source distributions for parameters of the Hodgkin-Huxley model from experimental datasets with hundreds of single-neuron measurements. In summary, we propose a principled method for inferring source distributions of scientific simulator parameters while retaining as much uncertainty as possible.

Sourcerer: Sample-based Maximum Entropy Source
Distribution Estimation
Julius V22
Guy 1o.-
Cornelius Schröder1.
Richard Gao
Jakob H. Mackel23,
Machine Learning in Science, Excellence Cluster Machine Learning, University of Tübingen
TTübingen AI Center
Department Empirical Inference, Max Planck Institute for Intelligent Systems
Tübingen, Germany 'Equal contribution.
Abstract
Scientific modeling applications often require estimating distribution of param- eters consistent with a dataset of observations inference task also known as source distribution estimation. This problem can be ill-posed, however, since many different source distributions might produce the same distribution of data- consistent simulations. To make principled choice among many equally valid sources, we propose approach which targets the maximum entropy distribu- tion, i.e., prioritizes retaining as much uncertainty possible. Our method is purely sample-based leveraging the Sliced- Wasserstein distance to measure the discrepancy between the dataset and simulations and thus suitable for simulators with intractable likelihoods. We benchmark our method several tasks, and show that it can recover source distributions with substantially higher entropy than recent source estimation methods, without sacrificing the fidelity of the simulations. Finally, to demonstrate the utility of our approach, we infer source distributions for parameters of the Hodgkin-Huxley model from experimental datasets with hundreds of single-neuron measurements. summary, we propose a principled method for inferring source distributions of scientific simulator parameters while retaining as much uncertainty as possible.
Introduction
In many scientific and engineering disciplines, mathematical and computational simulators are used to gain mechanistic insights. A common challenge is to identify parameter settings of such simulators that make their outputs compatible with a of empirical observations. For example, by finding distribution of parameters that, when passed through the simulator, produces a distribution of outputs that matches that of the empirical dataset of observations.
Suppose we have a stochastic simulator with input parameters 0 and output which allows to generate samples from the forward model p(x(0) (which is usually intractable). We have acquired a dataset D (.,, } of observations with empirical distribution (z),, and want to identify
'[firstname. www.oddaame www.iiggee.bnnn...eeme Code available at https://gthhb..cmmmaaccccbi/oo
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

a distribution q(O) over parameters that, once passed through the simulator, yields a "pushforward" distribution of simulations { p(x)0))(())))) that is indistinguishable from the empirical distribution. This setting is known by different names in different disciplines, for example as unfolding in high energy physies [10], stochastic inverse problems in various disciplines [7], population of models in electrophysiology [30] and population inference in gravitational wave astronomy [55]. Adopting the terminology of Vandegar et al. [58], we refer to this task as source distribution estimation.
=
common approach to source distribution estimation is empirical Bayes [51, 15]. Empirical Bayes uses hierarchical models in which each observation is modeled arising from different parameters p(x,(0,). The syper-parameters of the prior (and thus the source (o) are found by optimizing the marginal likelihood p( PP) II; J p(a; (o::10) (0)aa(()d over Φ. Empirical Bayes has been successfully applied to a range of applications [31, 32, 55]. However, empirical Bayes is typically not applicable to models with intractable likelihoods, which usually the case for scientific simulators. Using surrogate models for such likelihoods, empirical Bayes has been extended to increasingly more complicated parameterizations @ of the source distribution, including neural networks [59, 58].
A more general issue, however, is that the source distribution problem can often be ill-posed with- out the introduction of a hyper-prior or other regularization principles,
simulator plx(e)
also noted in Van- degar et al. [58]: Distinct source distributions q(0) can give rise to the same data distribu- tion q (z) when pushed through the simula- tor p(z](0) (Fig. 1, illustrative example in Ap- pendix A.7).
dataset
MaxEnt source 1 source 2
M
X
We here propose to use the maximum entropy principle, i.e., choosing the "maximum igno- rance" distribution within a class of distribu- tions to resolve the ill-posedness of the source distribution problem [19, 24]. The maximum entropy principle formalizes the notion that good choice for distributions should "assume less". It has been applied to specific source dis- tribution estimation problems in scientific disci- plines such as cosmology [23] and high-energy physics [10].
Goal: source estimation
Figure 1: Maximum entropy source distribu- tion estimation. Given an observed dataset D = {II....,IIA] from some data distribution P,,(z), ) the source distribution estimation problem is to find the parameter distribution q(0) that reproduces Po Po(r) when passed through the simulator p(xr(0), ie. a (x) = f p(x (() = px(x)) for all x. This problem can be ill-posed, as there might be more than one distinct source distribution. We resolve this by targeting the maximum entropy dis- tribution, which is unique.
Our contributions We introduce Sourcerer, a general method for source distribution estima- tion, providing two key innovations: First, we target the maximum entropy source distribution to obtain well-posed problem, thereby increasing the entropy of the estimated source distributions at no cost to their fidelity. Second, we use general distance metrics between distributions, in particular the Sliced-Wassersted Wasserstein distance, instead of maxi- mizing the marginal likelihood as in empirical Bayes. This allows evaluation of the objective using only samples from differentiable simulators, removing the requirement to have tractable likelihoods. We validate our method on multiple tasks, including tasks with nigh-dimensional observation space, which are challenging for ikelihood-basee methods. Finally, we apply our method estimate the source distribution over the mechanistic parameters of the Hodgkin-Huxley model from a large (~~ 1000 samples) dataset of electrophysiological recordings.
2 Methods
We formulate the source distribution estimation problem in terms of the maximum entropy principle. The (differential) entropy (p) of a distribution p(0) is defined as
/ p(0) log d00)d0.
H(p) --
(1)
2

---

2.1 Data-consistency and regularized objective
For a given distribution q()) and a simulator with (possibly intractable) likelihood p(z[0), the pushforward of q is given by (x) p(z)(0)q(0)dd) The distribution (()) is source distribution its pushfoward matches the observed data distribution Po(x), that is, 9** = Po almost everywhere. Equivalently, given a distance metric (---) -) between probability distributions P(X' over the data space X, a source distribution q is one which satisfies D(q .o)) = 0. general, for a given distribution of observations (x) and likelihood p(ar(0). the source distribution problem is ill-posed as there are possibly many different source distributions. The maximum entropy principle can be employed to resolve this ill-posedness:
Proposition 2.1. Ler Q l }o} be the set of source distributions for a given likelihood p(z (z]0) and data distribution po- Suppose that Q is non-empe non-empty and compact. Then q" = arg max geQ H (4) exists and is unique.
This proposition follows from the fact that the set of source distributions is convex and that the (differential) entropy (g) strictly concave functional. See Appendix A.7 for proof and additional assumptions.
Proposition 2.1 suggests to solve the constrained optimization problem
Maximize entropy H(q), regularize mismatch SWD
max (go) s.t. D(g # Po) O,
simulate
compare
P(x(B)
SWD
where is some parametric family of distribu- tions.
source model q(0)
pushforward distribution q*(x)
dataa distribution p,(0
Practically, however, a solution might not exist, for example due to simulator misspecification. Furthermore, even if solution exists, it diffi- cult to obtain since only have fixed number of samples from Po and can thus only estimate # Po). We therefore propose a regularized approximation of Eq. (2) and solve
Figure 2: Overview of Sourcerer. Given source distribution q(0), we sample ~ q and simulate using p(x(0) to obtain samples from the pushfor- ward distribution y# (x) = p( (O0 We maximize the entropy of the source distribution (()) while regularizing with Sliced-Wasserse Wasserstein distance (SWD) term between the pushforward of 9#h and the data distribution (a(x) (Eq. (3)). O and X' in top right corner of boxes denote parameter space and data/observation space, respectively.
D(a
max (H(g)) - (1 X) log Dig Po)) (3)
"
instead, where À is a parameter determining the strength of the cata-conns term and the logarithm is added for numerical stability. This regularized objective is related to the Lagrangian relaxation of Eq. (2), where now log D ((**, .Po) S log for some e > and the dual variable is -A)A..
For X + the loss in Eq. (3) is dominated by the entropy term, and for À 0 by the data- consistency term. We apply ideas from constrained optimization and reinforcement learning [49, 4, 1] and use dynamical schedule during training. We initialize training with λ==1 1, and decay this value linearly to final value A= À >> O over the course of training. This dynamical schedule encourages the variational source model first explore high-entropy distributions, and later increase consistency with the data between high-entropy distributions. Pscudocode and details of the schedule in Appendix A.3.
2.2 Reference distribution
For many tasks, there is an additional constraint in terms of a reference distribution p(0). For example, the Bayesian inference framework, it common have prior distribution p(0), encoding existing knowledge about the parameters from previous studies. In such cases, a distribution with higher entropy than p(f), even if it is a source distribution, is not always desirable. We therefore adapt our objective function in Eq. (3) to minimize the Kullback-Leibler (KL) divergence between the source q()) and the reference p(0):
min ADRi(gll)) + +1 - A) A))g ,po)).
(4)
3

---

The KL divergence term can be rewritten as (g = -g()) + (), where (q. = - (og()()))q(())
the cross-entropy between q and p. Thus, provided we can evaluate the density p(0), we can obtain a sample-based estimate of the loss in Eq. (4). In our work, we consider p(0) to be the uniform distribution over some bounded domain Be (and hence the maximum entropy distribution on this domain). This "box prior" is often used as the naive estimate from literature observations in inference studies. More specifically, in this case, H p) = -1/| Bel. where [eel is the volume of Be. Therefore, it is independent of q, and hence minimizing the KL divergence is equivalent to maximizing H (q) on Be, In the case where p(0) is non-uniform (e.g.. Gaussian) the cross-entropy term regularizes the loss by penalizing large ((f) when p(0) is small.
2.3 Sliced- Wassersessr as a distance metric
We are free choose any distance metric (): for the loss function Eq. (4). In this work, the fast, sample-based, and differentiable Sliced- Wasserstein distance (SWD) [6, 27, 42] of order two. The SWD is defined as the expected value of the one-dimensional Wasserstein distance between the projections of the distribution onto uniformly random directions u on the unit sphere gd-1 in Rd More precisely, the SWD is defined
(5) where P., is the one mie-dimensional distribution with samples u x for æ p(x), and Wm is the one- dimensional Wasserstein distance of order m. In the empirical setting, where we are given n samples each from Pa and qu respectively, the one-dimensional Wasserstein distance is computed from the order statistics as
SWD (p. q) Bu--((--1) [wm (Pu: [w)];
1/m
W (Pus Qu)
Σ
(0
(1)
(6)
,r(")), ), and where ( denotes the i-th order statistic of the samples from Pu. (and similarly for x.g
Il denotes the L distance on R [47]. The time complexity of computing the sample-based one-dimensional Wasserstein distance thus the time complexity of computing the order statisties, which is On log n) in the number of datapoints [6]. This is significantly faster than computing the multi-dimeesional Wasserstein distance (O(n3) 3) 29), or the commonly used Sinkhorn algorithm for approximating the Wasserstein distance (O(n2) ) 47). While the SWD is not the same as the multi- dimensional Wasserstein distance, it is still a valid metric on the space of probability distributions. In particular, the SWD converges quickly with rate (n)) to its true value [41, 42].
2.4 Differentiable simulators and surrogates
Our method only requires that sampling from the simulator p(x:(0) is differentiable operation. In practice, however, many simulators do not satisfy this property. For such simulators, we first train a surrogate model. In particular, our method can make use of surrogates that model the likelihood only implicitly. Such surrogate models can be easier to train and evaluate in practice. This is a distinct requirement from likelihood- likelihood p(z 0) can be evaluated explicitly and is differentiable. This means that our sample-based approach can be readily applied a larger set of simulators than likelihood breeeeooddbbaed approaches.
approaches such as Vandegar et al. [58], which require that the
2.5 Source model and entropy estimation
In this work we use neural samplers as proposed in Vandegar et al. [58] to parameterize a source model ls- These samplers employ unconstrained neural network architectures (in our case multi-layer perceptron) to transform a random sample from 2 E (O, I) into a sample from qo. While neural samplers do not have a tractable likelihood, they are faster to evaluate than models with tractable likelihoods. Furthermore, by using unconstrained network architectures, neural samplers are flexible and additional constraints (e.g., symmetry, monotonicity) are easy introduce.
To use likelihood-fre frreeddddffeeeeeeeee source parameterizations, we require purely sample- estimator for the entropy H (go)). This can be done using the based on a nearest-neighhor density estimate. We use the Kozachenkko-Leonenlle estimator in this work for its simplicity, but note that sample-based entropy estimation is an active area of research, and other choices are possible [48]. Details about the Kozachenko-Leeoe Lonneko estimator can be found in Appendix A.6.
entropy estimator [28, 31, which

---

Example: Inverse Kinematics (IK)
b
Simulator
Surrogate
1.0
Original source
Observations
Entropy
Estimated source
Simulatioos
TM
0.9
0.5
X
1.0
IK EEnropy
C2ST
0.5
12
1.0
SLCP Entropy
C2ST
0.5
1.0
A
Entropy
GM
0.5
8, 8 8,
8
NA
0.015
A 0.125
1.0
NA
0.015
A 0.125
1.0
Figure 3: Results for the source estimation benchmark. (a) Original and estimated source and corresponding pushforward for the differentiable IK simulator (A 0.35). The estimated source has higher entropy than the original source that was used to generate the data. The observations (simulated with parameters from the original source) and simulations (simulated with parameters from the estimated source) match. (b) Performance of our approach for all four benchmark tasks (TM, IK, SLCP, GM) using both the original (differentiable) simulators. and learned surrogates. Source estimation is performed without (NA) and with entropy regularization for different choices of X. For all cases, mean C2ST accuracy between observations and simulations (lower is better) as well as the mean entropy of estimated sources (higher is better) over five runs are shown together with the standard deviation. The gray line 0.35 (λ 0.062 for GM) indicates our choice of final À for the numerical benchmark results (Table I).
Experiments
To evaluate the data-consistency and entropy of source distributions estimated by Sourcerer, we benchmark our method against Neural Empirical Bayes (NEB) [58], state-of-the-ars approach to source distribution estimation. The benchmark comparison is performed on four source distribution estimation tasks including three presented Vandegar et [58]. We then demonstrate the advantage of Sourcerer in the case of differentiable simulators with high-dimensional data domain, where liklihooddbbaee empirical Bayes approaches would require training a likelihood surrogate. Finally, use Sourcerer to estimate the source distribution for Hodkgin Huxley simulator of singlc-ncuron voltage dynamics from a large dataset of experimental alcctrophysollogiaa recordings. For all tasks except the Hodgkin-Huxley task (where the observed dataset is experimentally measured), generate two datasets of observations of equal size from the same reference source distribution. The first is used train the source model, and the second is used to evaluate the quality of the learned source.
3.1 Source Estimation Benchmark
Benchmark tasks The source estimation benchmark contains four simulators: two moons (TM), inverse kinematics (IK), simple likelihood complex posterior (SLCP), and Gaussian Mixture (GM) (details about simulators and source distributions are in Appendix A.2). Notably, all four simulators are differentiable. Therefore, we can evaluate our method directly the simulator as well trained surrogates. For all four simulators, source estimation performed on synthetic dataset of 10000 observations that were generated by sampling from pre-defined original source distribution and evaluating resulting pushforward distribution using the corresponding simulator. The quality of the estimated source distributions is measured using classifier two sample test (C2ST) [33] between observations and simulations from the source. We also report the entropy of the estimated sources. Given two sources with the same C2ST accuracy, the higher entropy source is preferable. We compare
5

---

Table I: Numerical benchmark results for Sourcerer. We show the mean and standard deviation over five runs for differentiable simulators and surrogates of Sourcerer on the benchmark tasks, and compare to NEB. All approaches achieve C2ST accuracies close to 50%. For the Sliced- Wasserstein- based approach, the entropies of the estimated sources are substantially higher (bold) with the entropy regularization (A 0.35 for TM, IK, SLCP, À 0.062 for GM, gray line in Fig. 3).
Sourcerer Sim. (with reg.)
Sourcerer Sim. (w/ reg.)
Sourcerer Sur. (with reg.)
Sourcerer Sur. (wlo reg.)
Method
NEB
C2ST acc.
0.51 (0.004)
0.5 (0.008)
0.51 (0.003)
0.51 (0.006)
0.53 (0.005)
TM
Entropy
1.26 (0.022)
1.0 (0.198)
1.21 (0.054)
1.02 (0.162)
1.13 (0.093)
C2ST acc.
0.51 (0.002)
0.51 (0.005)
0.51 (0.005)
0.51 (0.01)
0.6 (0.014)
IK
Entropy
3.75 (0.066)
1.59 (0.246)
3.78 (0.022)
1.7 (0.165)
0.82 (0.712)
C2ST acc.
0.53 (0.005)
0.53 (0.006)
0.55 (0.003)
0.59 (0.017)
0.53 (0.006)
SLCP
Entropy
9.81 (0.039)
7.23 (0.052)
9.74 (0.039)
6.76 (0.302)
7.56 (0.097)
C2ST acc.
0.51 (0.005)
0.5 (0.006)
0.54 (0.006)
0.55 (0.005)
0.52 (0.004)
GM
Entropy
-1.12 (0.083)
-1.25 (0.106)
-0.36 (0.095)
-2.19 (0.212)
-1.5 (0.052)
to the NEB estimator with the same parameterization of the source model and 1024 Monte Carlo samples to estimate the marginal likelihood (details in Appendix A.3)
Benchmark performance We first check whether minimizing the Sliced- Wasserstein distance without any entropy regularization finds good source distributions. This corresponds to the case À 0 in Eq. (3) without any decay. In this way, we compare the data-consistency objective in Eq. (4) to the NEB objective of maximizing the marginal likelihood. We find that for the differentiable simulators, the Serted--Waseestein--bee
approach is able to find good source distributions with C2ST accuracies close to 50% for all benchmark tasks (Fig. 3, labeled NA). This also applies when we use surrogate models to generate the pushforward distributions. In particular, the quality of the estimated source distributions matches those found by NEB (Table 1).
We then apply entropy regularization as defined in Eq. (3) for all benchmark tasks. The entropy of the estimated sources is drastically increased without any cost in the quality of the simulations (Fig. 3b). While C2ST accuracy remains close to 50% across all benchmark tasks, the entropy of estimated sources is substantially higher than that of sources estimated with NEB, when minimizing only the data-consistency term (Table 1). We also explore the dependence of the results on the final regularization strength x (Fig. 3b). We observe sharp trade-off: above critical value of X, the SWD term becomes too weak, and the fidelity of the simulations rapidly declines. However, below this critical value of λ, the results are robust relative to X: the estimated sources produce simulations that match the observations, and have comparable entropy.
Additionally, for both IK and SLCP simulators, the entropy of the sources estimated by our method is higher than the entropy of the original source distribution (Fig. 3a and Fig. A7) despite the simulations and observations being indistinguishable from each other (C2ST accuracy: 50%). This does not contradict our approach: The original source distribution just happens not to be the maximum entropy source for these simulators.
We also investigate the robustness of our approach to the choice of the differentiable, sample- based distance by repeating all experiments for these benchmark tasks using the Maximum Mean Discrepancy (MMD, 22) and find comparable results (Fig. A4). Finally, we demonstrate (Fig. A5) the robustness of our approach for small dataset sizes by repeating the Two Moons task with (N 100) observations (as opposed to 10000), and for nigh-dimensional parameter spaces by repeating the Gaussian Mixture task with D= 25 dimensions (as opposed to 2).
3.2 High- digeeniimensooal observations: Lotka- Volterra and SIR
Since our method is sample-based and does not require likelihoods, it is possible to estimate sources by back-propagating through the differentiable simulators directly. This is advantageous especially for simulators with high-dimensional outputs, as we no longer require first train surrogate likelihood model, which can be challenging when faced with high-dimensional data such as time series. Here, we

---

a
Determinisii SIR model dim.)
b
Probabiisitic Lotka-Volterra model dim.)
10"
Entropy
minimum SWD
SWD
Entropy
minimum SWD
10
SWD
10
10
10"
NA
0.015
0.125
1.0
0.015
A 0.125
Figure 4: Source estimation on differentiable simulators. For both the deterministic SIR model (a) and probabilistic Lotka- Volterra model (b), the Shiced-Wasserstee distance (lower is better) between observations and simulations as well as entropy of estimated sources (higher better) for different choices of À and without the entropy regularization (NA) are shown. Mean and standard deviation are computed over five runs.
highlight this capability of our method by estimating source distributions for two high-dimensional, differentiable simulators: The Lotka-Volterry model and the SIR (Susceptible, Infectious, Recovered) model. The Lotka- Volterra model is used to model the density of two populations, predators and prey. model the spread of discase in population
The SIR model is commonly used in epidemiology (details about both models and source distributions in Appendix A.2). Compared to the benchmark tasks in Sec. 3.1, the dimensionality of the data space is much larger: Both the Lotka- Volteraa and the SIR model are simulated for 50 time points resulting in a 100 and 50 dimensional time series, respectively.
Furthermore, to show that unlike NEB (which maximizes the marginal likelihood), our sample-based approach is applicable to deterministic simulators, we use deterministic version of the SIR model with no observation noise. Similarly to the benchmark tasks, we define a source, and simulate 10000 observations using samples from this source to define a synthetic dataset which to perform source distribution estimation. Here, we directly evaluate the quality the estimated source distributions using the Sliced- Wassersttin distance. We compare this distance to the minimum expected distance, which is the distance between simulations of different sets of samples from the same original source. For comparison with NEB, train surrogate models with reduced dimensionality and again compute C2ST accuracies and entropies of the estimated sources (see Appendix A.5 and Fig. A3 for details surrogate training and pushforward plots).
Source estimation for the deterministic SIR model Our method is able to estimate good source distribution for the deterministic SIR model: The BlicedWassersstee distance between simulations and observations is close to the minimum expected distance (Fig. 4a). In contrast to the benchmark tasks, estimating sources with entropy regularization does not lead an increase in entropy for the SIR model, and the quality of the estimated source remains constant for various choices of A possible explanation for this is that there is no degeneracy in the parameter space of the deterministic simulator, and there exists only one source distribution.
Source estimation for the probabilistic Lotka- Volterra model For the probabilistic Lotka- Volterra model, our method also capable estimating source distributions. As for the SIR model, the Sliced-Wii Wassersssnsttiin distance between simulations and observations is close to the minimum expected distance (Fig. 4b). However, unlike the SIR model, estimating the source with entropy regularization yields large increase in entropy compared to when not using the regularization. For the Lotka- Volterra model, our method yields a substantially higher entropy at no additional cost in terms of source quality.
When using the surrogate models with reduced dimensionality to estimate the source distributions, we find that Sourcerer achieves better C2ST accuracies than NEB. Furthermore, for the Lotka- Volterra model, the entropy regularization again leads to substantial increase in the entropy the estimated sources (Table 2). In summary, the experiments the SIR and Lotka- Voltera models show that our approach is able to scale to higher dimensional problems and can use gradients of complex simulators to estimate source distributions directly from set of observations.
7

---

Table 2: Numerical results for the SIR and Lotka-Volterra Volterra model We show the mean and standard deviation over five runs for differentiable simulators and surrogates of Sourcerer on the high-dimensional SIR and Lotka-Vonn Volterra (LV) models, and compare to NEB. For the comparison with NEB, we train the required surrogate models with reduced dimensionality (25 dimensions instead of 50 or 100). Sourcerer achieves C2ST accuracies close 50%. For NEB, the C2ST accuracies are worse. For the LV model, the entropies of the estimated sources are higher with the entropy regularization (A 0.015 for SIR, À = 0.125 for LV).
Sourcerer
Sourcerer Sim. (w/o reg.)
Sourcerer Sur. (with reg.)
Sourcerer
Method
NEB
Sim. (with reg.)
Sur. (w/o reg.)
C2ST acc.
0.56 (0.013)
0.56 (0.015)
0.55 (0.005)
0.55 (0.005)
0.76 (0.024)
STR
Entropy
-2.3 (0.079)
-2.37 (0.169)
-2.29 (0.076)
-2.5 (0.05)
-0.63 (0.174)
C2ST acc.
0.57 (0.009)
0.52 (0.001)
0.56 (0.005)
0.54 (0.009)
0.62 (0.011)
LV
Entropy
0.29 (0.017)
-1.34 (0.087)
0.34 (0.05)
-1.01 (0.13)
-1.28 (0.073)
3.3 Estimating source distributions for single-compartment Hodgkin-Huxxey model
Single-compartment Hodgkin-Huxley simulator and summary statistics The single- compartment Hodgkyn-Huxley model consists of system of coupled ordinary differential equations simulating different ion channels in a neuron. We use the simulator described in Bernaerts et al. [2] with 13 parameters. In data space, we use five commonly used summary statistics of the observed and simulated spike trains. These are the (log of the) number of spikes, the mean of the resting potential, and the mean, variance and skewness of the voltage during external current stimulation. As the internal noise in the simulator has little effect on the summary statistics, we train a simple multi-layer perceptron surrogate
106 simulations. The parameters used to generate these training simulations were sampled from a uniform distribution that was used as the prior in Bernaerts al. [2] (details on simulator, choice of surrogate and the surrogate training in Appendix A.9).
Using this surrogate, we estimate source distributions from a real-world dataset of electrophysiological recordings. The dataset [52] consists of 1033 electrophysiological recordings from the mouse motor cortex. In general, parameter inference for HldgginnHuuxie models can be challenging as models often misspecified [56, 2]. Thus, estimating the source distribution for this task is useful for downstream inference tasks, as the prior knowledge gained can significantly constrain the parameters of interest.
Source estimation for the Hodgkin-Huxley, model On visual inspection, simulations from the estimated source look similar to the original recordings (all observations spike at least once, spikes have similar magnitudes) and show none of the unrealistic properties (e.g.. spiking before the stimulus applied) that can be observed in some of the box uniform prior simulations (Fig. 5a). This match is also confirmed by the distribution of summary statistics, which match closely between simulations and observations (Fig. 5b). Furthermore, our method achieves good C2ST accuracy of 61% for different choices of X (Fig. 5d), as well as a small standardized space of summary statistics (Fig. 5e). While the source estimated without entropy regularization also achieves good fidelity, its entropy is significantly lower than any of the source distributions estimated with entropy regularization (Fig. 5d/c, example source distribution in Fig. Sc, full source in Fig. A1)).
distance of R 0.08 in the
Overall, these results demonstrate the importance of estimating source distributions using the entropy regularization, especially on real-world datasets: Estimating the source distribution without any entropy regularization can introduce severe bias, since the estimated source may ignore entire regions of the parameter space. In this example, the parameter space of the single-compartment Hodgkin- Huxley model is known to be highly degenerate, and a given observation can be generated by multiple parameter configurations [14, 39].
8

---

a
Observations
b
Summary stalistics (3 out of
Parameters (3 out of 13)
20
Simulations from estimated
b
250
Siiuulaioon from unitorm distribution
6.75
-125
-25
20
-90
d
log count
mean rest
skewness
Vt
15
1.0
15
Entropy
C2ST
Entropy
SWD
10
-20
0.5
-20
NA
0.015
0.125
1,0
NA
0.015
0:125
1.0
Figure 5: Source estimation for the single-compartment Mlddgkin-Huxley model. (a) Example voltage traces of the real observations of the motor cortex dataset, simulations from the estimated source (A 0.25), and samples from the uniform distribution used to train the surrogate. (b) ID and 2D marginals for three of the five summary statistics used to perform source estimation. (c) ID and 2D marginal distributions of the estimated source for three of the 13 simulator parameters. (d) and (e) C2ST accuracy and Sliced-Wasserseee Wasserstein distance (lower is better) as well as entropy of estimated sources (higher is better) for different choices of À including without entropy regularization (NA). Mean and standard deviation over five runs are shown.
0.25 (gray line) and
4 Related Work
Neural Empirical Bayes nigh--immessioal source distributions have been estimated through variational approximations to the empirical Bayes problem. Louppe et al. [34] train a generative adversarial network (GAN) [20] qu to approximate the source. The use of a discriminator to compute an implicit distance makes this approach purely sample-based as well. In order to find the optimal e²* of the true data- gata-generating process, they augment the adversarial loss with small entropy penalty on the source qu. This penalty encourages low entropy, point mass distributions, which is the opposite of our approach. Vandegar et al. [58] take an empirical Bayes approach, and use normalizing flows for both the variational approximation of the source and as a surrogate for the likelihood p(xr(0). This allows for direct regression on the marginal likelihood, as all likelihoods can be computed directly. Finally, the empirical Bayes problem is also known as "unfolding" in the particle physics literature [10], "population inference" in gravitational wave astronomy [55], and "population of models" in clectrophysiology [30]. Approaches have been developed to identify the source distribution, including classical approaches that seek to increase the entropy of the learned sources [50].
Simulation-Bain Based Inference The use of variational surrogates of the likelihood of simulator with intractable likelihood is known as Neural Likelihood Estimation in the simulation based inference (SBI) literature [60, 45, 36, 11]. In neural posterior estimation [44, 35, 21J, an amortized posterior density estimate is learned, which can be applied to evaluate the posterior of a single observation J; E DD if a prior distribution p(f) is already known. An intuitive but incorrect approach to source distribution estimation would be to take the average posterior distribution over the observations D,
Go) = Σ p()
(7)
The average posterior does not always (and typically does not) converge to a source distribution the infinite data limit, as shown for simple examples in Appendix A.8. Intuitively, the average posterior becomes a worse approximation of source distribution for simulators that have broader likelihoods. Instead, SBI can be seen as a downstream task of source distribution estimation; once prior has been learned from the dataset observations with source estimation, the posterior can be estimated for each new observation individually.

---

Generalized Bayesian Inference Another field related to source estimation is Generalized Bayesian Inference (GBI) [s, 40, 26]. GBI performs distance-based inference, as opposed targeting the exact Bayesian posterior. Similarly to our work, the distance function used in GBI can be arbitrarily chosen for different tasks. However, GBI is used for single-parameter inference tasks, as opposed to the source distribution estimation task considered in this work. Similarly, Bayesian con-paraettric methods [43, 38, 12] learn a posterior directly on the data space which can then be used to sample from posterior distribution over the parameter space.
5 Summary and Discussion
In this work, we introduced Sourcerer as a method to estimate source distributions of simulator parameters given datasets of observations. This is a common problem setting across range of scientific and engineering disciplines. Our method has several advantages: first, we employ maximum entropy approach, improving reproducibility of the learned source, as the maximum entropy source distribution is unique while the traditional source distribution estimation problem can be ill- ill-posed. Second, our method allows for sample-based optimization. In contrast to previous likelihood-bart baaedd approaches, this scales more readily higher dimensional problems, and can be applied to simulators without a tractable likelihood. We demonstrated the performance of our approach across a diverse suite of tasks, including deterministic and probabilistic simulators, differentiable simulators and surrogate models, low- and high-dimensional observation spaces, and contemporary scientific task of estimating a source distribution for the single-compartment Hodgkin -uxkey model from a dataset of electrophysiologica recordings. Throughout our experiments, we have consistently found that our approach yields higher entropy sources without reducing the fidelity of simulations from the learned source.
Limitations In this work, we used the Sliced- Wasserstein distance (and MMD) for the data- consistency term between simulations and observations. In practice, different distance metrics can lead to different estimated sources, depending on its sensitivity to different features. While our method is compatible with any sample-based differentiable distance metric between two distributions, there is still an onus on the practitioner to carefully select a reasonable distance metric for the data at hand. For example, in some cases, it might be appropriate to use combination of several distance metrics for different modalities of the data. Similarly, there a dependence on the final regularization strength X. Principled methods for defining the regularization strength are desirable, though demonstrate, our results are robust to a large range of X.
In addition, the method requires differentiable simulator, which in practice may require the training of a surrogate model, for example, when dealing with a (partially) discrete simulator. While this is common requirement for simulation-basee based methods, this could present challenge for some applications. Finally, in our work, we enforce the maximum entropy principle on the entire (parameter) source distribution. In practice, for example when constructing prior distributions for Bayesian inference, there are other choices, such as the Jeffrey's 's prior [9].
Acknowledgements
This work was funded by the German Research Foundation (DFG) under Germany's Excellence Strategy - EXC number 2064/1 390727645 and SFB 1233 'Robust Vision' (276693517). This work was co-funded by the German Federal Ministry of Education and Research (BMBF): Tübingen AI Center, FKZ: AUIS18039) and the European Union (ERC, DeepCoMechTome, 101089288). Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Council. Neither the European Union nor the granting authority can be held responsible for them. JV is supported by the A14Med-BW graduate program. JV and GM are members of the International Max Planck Research School for Intelligent Systems (IMPRS-IS). We would like to thank Jonas Beck, Sebastian Bischoff, Michael Deistler, Manuel Glöckler, Jaivardhan Kapoor, Auguste Schulz, and all members of Mackelab for feedback and discussion throughout the project.
10

---

References
[1] Zafarali Ahmed, Nicolas Le Roux, Mohammad Norouzi, and Dale Schuurmans. Understanding the impact of entropy on policy optimization. In International conference machine learning, 2019.
[2] Yves Bernaerts, Michael Deistler, Pedro J Goncalves, Jonas Beck, Marcel Stimberg, Federico Scala, Andreas Tolias, Jakob H Macke, Dmitry Kobak, and Philipp Berens. Combined statistical-mechanistiee modeling links ion channel genes to physiology of cortical neuron types. bioRtvi, 2023.
[3] Thomas B. Berrett, Richard J. Samworth, and Ming Yuan. Efficient multivariate entropy estimation via k-nearest neighbour distances. The Annals of Statistics, 2019.
[4] D.P. Bertsekas and W. Rheinboldt. Constrained Optimization and Lagrange Multiplier Methods. Computer science and applied mathematics. Elsevier Science, 2014.
[5] Pier Giovanni Bissiri, Chris C Holmes, and Stephen Walker. A general framework for updating belief distributions. Journal of the Royal Statistical Society Series B: Statistical Methodology, 2016.
[6] Nicolas Bonneel, Julien Rabin, Gabriel Peyré, and Hanspeter Pfister. Sliced and Radon Wasser- stein barycenters of measures. Journal of Mathematical Imaging and Vision, 2015.
[7] T. Butler, J. Jakeman, and T. Wildey. Combining push-forwart measures and bayes' rule to construct consistent solutions to stochastic inverse problems. SIAM Journal on Scientific Computing, 2018.
[8] E.K.P. Chong, W.S. Lu, and S.H. Zak. An Introduction Optimization: With Applications to Machine Learning. Wiley, 2023.
[9] Guido Consonni, Dimitris Fouskakis, Brunero Liseo, and Ioannis Ntzoufras. Prior Distributions for Objective Bayesian Analysis. Bayesian Analysis, 2018.
[10] G. Cowan. Statistical Data Analysis. Oxford science publications. Clarendon Press, 1998.
[11] Kyle Cranmer, Johann Brehmer, and Gilles Louppe. The frontier of simulation- based inference. Proceedings of the National Academy of Sciences, 2019.
[12] Charita Dellaporta, Jeremias Knoblauch, Theodoros Damoulas, and François- François-Xavier Xavier Briol. Robust Bayesian inference for simulator-based models via the MMD posterior bootstrap. In International Conference Artificial Intelligence and Statistics. PMLR, 2022.
[13] Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using Real NVP. In International Conference on Learning Representations, 2017.
[14] Gerald M Edelman and Joseph A Gally. Degeneracy and complexity in biological systems. Proceedings of the National Academy of Sciences, 2001.
[15] Bradley Efron and Carl Morris. Limiting the risk of Bayes and empirical Bayes estimators, part i:: The empirical Bayes case. Journal of the American Statistical Association, 1972.
[16] Philip E. Gill, Walter Murray, and Margaret H. Wright. Practical Optimization. Society for Industrial and Applied Mathematics, 2019.
[17] Manuel Glöckler, Michael Deistler, and Jakob H. Macke. Adversarial robustness of amortized Bayesian inference. In International Conference on Machine Learning, 2023.
[18] Pedro Gonçalves, Jan-Matthis Lueckmann. Michael Deistler, Marcel Nonnenmacher, Kaan Öcal, Giacomo Bassetto, Chaitanya Chintaluri, William F Podlaski, Sara A Haddad, Tim Vogels, et al. Training deep neural density estimators to identify mechanistic models of neural dynamics. Elife, 2020.
[19] J. Good. Maximum entropy for hypothesis formulation, especially for multidimensionnl contingency tables. Annals of Mathematical Statistics, 1963.
11

---

[20] Ian Goodfellow, Jean Pouget-Abadic Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances Neural Information Processing Systems, 2014.
[21] David S. Greenberg, Marcel Nonnenmacher, and Jakob H. Macke. Automatic posterior trans- formation for likelihood free inference. In International Conference on Machine Learning, 2019.
[22] A Gretton, KMM Borgwardt, MJ. Rasch, B Schölkopf, and Alexander Smola. A kernel two- sample test. Journal of Machine Learning Research, 2012.
[23] Will Handley and Marius Millea. Maximum-entropy priors with derived parameters in a specified distribution. Entropy, 2018.
[24] Edwin T. Jaynes. Prior probabilities. IEEE Transactions on Systems Science and Cybernetics, 1968.
[25] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[26] Jeremias Knoblauch, Jack Jewson, and Theodoros Damoulas. An optinization-centric view on bayes' rule: Reviewing and generalizing variational inference. Journal of Machine Learning Research, 2022.
[27] Soheil Kolouri, Kimia Nadjahi, Umut Simsekli, Roland Badeau, and Gustavo Rohde. Gener- alized Sliced Wasserstein distances. In Advances in Neural Information Processing Systems, 2019.
[28] L. Kozachenko and N. Leonenko. A statistical estimate for the entropy of random vector. Problems of Information Transmission, 1987.
[29] H. W. Kuhn. The Hungarian method for the assignment problem. Naval Research Logistics Quarterly, 1955.
[30] Brodie A. Lawson, Christopher C. Drovandi, Nicole Cusimano, Pamela Burrage. Blanca Rodriguez, and Kevin Burrage. Unlocking data sets by calibrating populations of models to data density: A study in atrial electrophysiology. Science Advances, 2018
[31] Tai Sing Lee and David Mumford. Hierarchical Bayesian inference the visual cortex. Opt.
Soc. Am. A, 2003.
[32] Ning Leng, John A. Dawson, James A. Thomson, Victor Ruotti, Anna I. Rissman, Bart M. G. Smits, Jill D. Haag. Michael N. Gould, Ron M. Stewart, and Christina Kendziorski. EBSeq: an empirical bayes hierarchical model for inference in RNA-seq experiments. Bioinformatics, 2013.
[33] David Lopez-Paz and Maxime Oquab. Revisiting classifier two-sample tests. In International Conference on Learning Representations, 2017.
[34] Gilles Louppe, Joeri Hermans, and Kyle Cranmer. Adversarial variational optimization of non- differentiable simulators. In International Conference on Artificial Intelligence and Statistics, 2019.
[35] Jan-Matthis Lueckmann, Pedro Goncalves, Giacomo Bassetto, Kaan Öcal, Marcel Nonnen- macher, and Jakob H Macke. Flexible statistical inference for mechanistic models of neural dynamics. In Advances in Neural Information Processing Systems, 2017.
[36] Jan-Matthis Lueckmann, Giacomo Bassetto, Theofanis Karaletsos, and Jakob H. Macke. Likelihood inference with emulator networks. In Proceedings of The 1st Symposium on Advances in Approximate Bayesian Inference, 2019.
[37] Jan-Mathis Lueckmann, Jan Boelts, David Greenberg, Pedro Goncalves, and Jakob Macke. Benchmarking simulation-based inference. International Conference on Artificial Intelligence and Statistics, 2021.
12

---

[38] Simon Lyddon, Chris C. Holmes, and Stephen G. Walker. General Bayesian updating and the loss-likelihood bootstrap. Biometrika, 2017.
[39] Eve Marder and Adam L Taylor. Multiple models to capture the variability in biological neurons and networks. Nature neuroscience, 2011.
[40] Takuo Matsubara, Jeremias Knoblauch, François- Xavier Briol, and Chris J Oates. Robust generalised Bayesian inference for intractable likelihoods. Journal of the Royal Statistical Society Series B: Statistical Methodology, 2022.
[41] Kimia Nadjahi, Alain Durmus, Umut Simsekli, and Roland Badeau. Asymptotic guarantees for learning generative models with the sliced-wasserstein distance. In Advances in Neural Information Processing Systems. Curran Associates, Inc., 2019.
[42] Kimia Nadjahi, Alain Durmus, Lénaïc Chizat, Soheil Kolouri, Shahin Shahrampour, and Umut Simsekli. Statistical and topological properties of sliced probability divergences. In Advances in Nearal Information Processing Systems, 2020.
[43] Peter Orbanz and Yee Whye Tch. Bayesian nonparametric models. Encyclopedia of Machine Learning, 2010.
[44] George Papamakarios and lain Murray, Fast e-free inference of simulation models with Bayesian conditional density estimation. Advances in Neural Information Processing Systems, 2016.
[45] George Papamakarios, David C. Sterratt, and lain Murray. Sequential neural likelihood: Fast likelihood-free inference with autoregressive flows. In International Conference Artificial Intelligence and Statistics, 2018
[46] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
[47] Gabriel Peyré and Marco Cuturi. Computational optimal transport. Found. Trends Mach. Learn., 2018.
[48] Georg Pichler, Pierre Colombo, Malik Boudiaf, Günther Koliander, and Pablo Piantanida. A differential entropy estimator for training neural networks. In International Conference on Machine Learning, 2022.
[49] John Platt and Alan Barr. Constrained differential optimization. In Neural Information Process- ing Systems, 1987.
[50] Marcel Reginatto, Paul Goldhagen, and Sonja Neumann. Spectrum unfolding, sensitivity analysis and propagation of uncertainties with the maximum entropy deconvolution code MAXED. Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment, 2002.
[51] Herbert F. Robbins. An empirical bayes approach statisties. In Breakthroughs in Statistics: Foundations and basic theory, 1956.
[52] Federico Scala, Dmitry Kobak, Matteo Bernabucci, Yves Bernaerts, Cathryn René Cadwell, Jesus Ramon Castro, Leonard Hartmanis, Xiaolong Jiang, Sophie Laturnus, Elanine Miranda, et al. Phenotypic variation of transcriptomic cell types in mouse motor cortex. Nature, 2021.
[53] Scott A Sisson, Yanan Fan, and Mark Tanaka. Sequential monte carlo without likelihoods. Proceedings of the National Academy of Sciences, 2007.
[54] Alvaro Cejero-Cantero, Jan Boelts, Michael Deistler, Jan-Matthis Lueckmann, Conor Durkan, Pedro J. Gonçalves, David S. Greenberg, and Jakob H. Macke. sbi: A toolkit for simulation- based inference. Journal of Open Source Software, 2020.
[55] Eric Thrane and Colm Talbot. An introduction to Bayesian inference in gravitational -wave astronomy: Parameter estimation, model selection, and hierarchical models. Publications of the Astronomical Society of Australia, 2019.
13

---

[56] Nicholas Tolley, Pedro LC Rodrigues, Alexandre Gramfort, and Stephanie Jones. Methods and considerations for estimating parameters in biophysically detailed neural models with simulation based inference. bioRxiv, 2023.
[57] Pravin M. Vaidya. An On log n) algorithm for the all-nearest-neeighorss problem. Discrete & Computational Geometry, 1989.
[58] Maxime Vandegar, Michael Kagan, Antoine Wehenkel, and Gilles Louppe. Neural empirical Bayes: Source distribution estimation and its applications to simulation-based inference. In International Conference on Artificial Intelligence and Statistics, 2020.
[59] Yixin Wang, Andrew C. Miller, and David M. Blei. Comment: Variational Autoencoders as Empirical Bayes. Statistical Science, 2019.
[60] Simon N. Wood. Statistical inference for noisy nonlinear ecological dynamic systems. Nature, 2010.
[61] Omry Yadan. Hydra framework for elegantly configuring complex applications. Github, 2019. URL https: ///////ttt//////// hctp///aceebborrreeera//////aa/a
14

---

A
Appendix
A.1 Software and data
We use PyTorch [46] for the source distribution estimation and hydra [61] track all configurations.
Code to reproduce results is available at
A.2 Simulators and sources
Here provide definition of the four benchmark tasks Two Moons (TM), Inverse Kinematics (IK), Simple Likelihood Complex Posterior (SLCP) and Gaussian Mixture (GM), as well as the two high-dimensional simulators, the SIR and Lotka- Lotka-Volterra model. We also deseribe the original source distribution used to generate the synthetic observations, and the bounds the reference uniform distribution the parameters.
A.2.1 Two moons simulator
Dimensionality Bounded domain
E IR2, 0 R [
Original source
U[[ 1, 1]2)
r cos(a a)) + 0.25
101 + 021/v2
Simulator
x 0
rsin (a)
(-0: + 02)/v2
where
122, ~
References
Vandegar et al. [58], Lueckmann et al. [37]
A.2.2 Inverse Kinematics simulator
Dimensionality Bounded domain
I E TR22, E Rd x]
Original source
~ N(, Diag( 21 4' 2)
1
Simulator
Ti + 1, sin 0g + 6) + 12 sin (02 + 0g +c)++++p sin + 0g + 04 + t),
= I cos -e) +2 cos 0g + + 4 (0, 0g + 04 + +e),
where I = la = 0.5, = 1.0 and ~ (00,0.000177)
References
Vandegar et al. [58]
A.2.3 SLCP simulator
Dimensionality
R 0 R -,,5
Bounded domain
Original source
([[[333]) 3)
Simulator
x (Z), .cl......
..),,
NM So),
where me
02
[8]
SB
88182
$
p81$2
$2
81
02
82 = 0.P = -lanh Oz.
References
Vandegar et al. [58), Lueckmann et al. [37]
A.2.4
Gaussian mixture simulator
Dimensionality Bounded domain
RR2 E R [_5,5]2
Original source
~~ ((0.5, 1
Simulator
x/0 ~ 0.5A 0.5N( -1).
References
Sisson et al. [53]
15

---

A.2.5 SIR model
Dimensionality
I. E 50 A E TR2
Bounded domain
30 B ~~ LogNorral(log(0A)) 0.5) 7 [og (20....1..... 0.2)
Original source Simulator
x (2 1,...... F50). where 24 I:/N equally spaced and I is
simulated from ds - 81 df =B% yI, dt N
dR 91
di
with initial values S = N - 1, = 1, R = 0 and N = 10
References
Lueckmann et al. [37]
A.2.6 Lotka- Volterra model
Dimensionality Bounded domain
I E TR 100 E TR:4 [0.1,3] 0' ~ (O,00.55) pushed through 0 5(0) exp(o(0')).
Original source
=xppo)))))
where a is the sigmoid function.
Simulator
x (2)
250: ...... 3
where xy (XX, 0.05"), 24 ~ -V(Y,0005 equally spaced,
and X, Y are simulated from dx aX - BXY, dy with initial values X = ===
-yY 6XY
References
Glöckler et al. [17]
A.3
Pseudocode and details on source estimation for benchmark tasks
Pseudocode for Sourcerer is provided in Algorithm 1.
For both the benchmark tasks and high dimensional simulators, sources were estimated from 10000 synthetic observations that were generated by simulating samples from an original previously defined source.
For the benchmark tasks, we used T' 500 linear decay steps from λ==o to A-=r = À and optimized the source model using the Adam optimizer with a learning rate of 10 and weight decay of 10-5 The two high dimensional simulators were optimized with higher learning rate of 10-3 and T = 50 linear decay steps. In both cases, early stopping performed when the overall loss in Eq. (4) did not improve over a set number of training iterations.
As baseline, we compare to Neural Empirical Bayes (NEB) as described in Vandegar et al. [58]. Specifically, we use the biased estimator with 1024 samples per observation (C1024), which are used to compute the Monte Carlo integral. Unlike our Sliced-Waad wassersttin--aaaaeeeee not operate on the whole dataset of observations directly but attempts to maximize the marginal likelihood per observation and thus uses part of the observations fair comparison, we increased the number of observations to 11112 for all NEB experiments, which results in training dataset of 10000 observations when using 10% as validation set. For training, we again used the Adam optimizer (learning rate 10-4, weight decay 10 10-5, training batch size 128).
approach, NEB does
validation set. To ensure
A.4 Source model
Throughout all our experiments, we use neural samplers as the source models [58]. The sampler architecture is three-layer multi-layer perceptron with dimension of 100, ReLU activations and batch normalization as our source model. Samples are generated by drawing sample s N(,0, I) from the standard multivariate Gaussian and then (non-linearly transforming with the neural network.
A.5 Surrogates for the benchmark tasks
We follow Vandegar et al. [58] and train RealNVP flows [13] as surrogates for the four benchmark tasks. For all benchmark tasks, the RealNVP surrogates have a flow length of 8 layers with hidden dimension of 50.
Surrogates for the benchmark tasks were trained using the Adam optimizer [25] on 15000 samples and simulator evaluations from the uniform distribution over the bounded domain (learning rate 10-1 weight decay 5 10-3, training batch size 256). In addition, 20% of the data was used for validation.
16

---

Algorithm 1: Sourcerer
Source model 4e constrained the bounded domain Bg, observed dataset D = (f1,..., Fn. J ~ Po(z)) differentiable model p( (0) to draw samples from (simulator or surrogate), number of samples estimate entropy, regularization schedule At=1;-r> At=T- Outputs:
Inputs:
Trained source model (f).
t 00;;
while not converged do
01
A ~ ()
# sample parameters for pushforward sample pushforward # sample parameters for entropy estimation schedule lambda
2: p( (
0....., 8' ~ (0(
d if S else A=
C + XH((0'
Or
+ ()) J.). (2/ rn)
# compute loss # update source model
4 Φ Adam vo
f 4- ++1
return
To train surrogate models for the SIR and Lotka-Volteryy model, we first reduce the simulator dimension in observation space 25 in both cases. Additionally, add small amount of independent Gaussian noise (N(X,, 0.012)) to the output of the SIR simulator to avoid training the normalizing flow surrogate with simulations from deterministic likelihood. We then use 10 simulations to train and validate (20% validation set) both surrogate models, again using the Adam optimizer (learning rate 5- 10 weight decay 5- 10-5 training batch size 256).
A.6 Kozachenko--Leonenke entropy estimator
Our use of neural samplers requires us to use a sample- based estimate of (differential) entropy, since no tractable likelihood is available (see Sec. 2.5).
use the Bozzachenko-Leonetie estimator [28, 3] for a set of samples {0:) from a distribution p(0) E P(e), given by
d ldg(
2 Hgg))
- g(()) + g(n) + log(Va),
(8)
m
where d, the distance of 0g from its k-th nearest neighbor in {0,} ) d is the dimensionality of O m is the number of non-zero values of d, is the digamma function, and Va is the volume of the unit ball using the same distance measure as used to compute the distances d,
The Kozachenko-LLconenkee estimator is differentiable and can be used for gradient-based optimization. The all-pairs nearest neighbor problem be efficiently solved in Onn log n) [57]. In practice, we find all nearest neighbors by computing all pairwise distances on fixed number of samples. Throughout all experiments, 512 source distribution samples were used to estimate the entropy during training.
A.7 Uniqueness maximum entropy source distribution
Here, prove the uniqueness of the maximum entropy source distribution (Proposition 2.1). First, however, we demonstrate for a simple example that the source distribution without the maximum entropy condition is not unique.
Example non-uniqueness Consider the (deterministic) simulator == f()) 10). Further assume that our observed distribution is the uniform distribution p(z) = (x; a, b), where 0 < < b.
Due the symmetry of f. the source distribution p(0) for the observed distribution p(x) is not unique. Any convex combination of form ouu (0) (1 ar)u2, where (0) U(0; -b. aa and uz(0) (0; a, b) and a [0, 1] provides source distribution. The maximum entropy source distribution is unique and is attained if both distributions are weighted equally with 0.5.
17

---

Proof of Proposition 2.1 First, let us state Proposition 2.1 in full:
Let ④ C TRde and X C Rdx be the parameter and obxervation spaces, respectively. Suppose that Q is compact. Let P()) C (0) and P(X) C (X) be the set of probability measures on θ and X respectively. Let Q = {g g for a given likelihood p(z and data distribution Po P(X)) Suppose that Q is non-empty and compact (in the L' norm topology). Then q" = arg mäx TEQ (()) exists and is unique.
# = Po almost everywhere } C ((()) be the set of source distributions
First, by the compactness assumption on θ, the (differential) entropy of q E P(0) is bounded above (by the entropy of the uniform distribution on θ), and so in particular it is finite. By the compactness assumption on Q. the entropy achieves its supremum of Q. that is, there exists q" such that (q"") = arg max,Eo (g). To show that q" is unique (up to L''ll sets), it is sufficient to show two results: (1) that the set Q a convex set, and (2) that entropy is strictly concave. In this case, if we have two distinct suprema of and q2. then any convex combination of qi 92 is valid source distribution with higher entropy, causing a contradiction. For the remainder of this proof, we let qi and de be two distinct source distributions. Their convex combination q odj1 + (1 a) [0, 1] is valid probability distribution supported on both of the supports of ( and 42-
(1) Sources distributions are closed under convex combination: is also source distribution, since
/ p(x10) (aq1 + a))2
*# (x)
/ plz10) + (1 (1- a) f plz110)22
(9)
apo(z) - (1- = (z)).
(2) Entropy is (strictly) concave: the entropy of q satisfies
Hg))= - f (aqi(0) + a))2((0)) log(ag) (0) + (=)))
2 / [1m1 (0) log(1) (0)) + (1 - ax (0) log( (0)))A0
(10)
aH (q1) + -1- ((2)))
where used the fact that the function f(x) z log is convex on [0, oo), and hence ff is concave. Furthermore, f(x) is strictly convex on [0, oo). so for any 8 E θ, the equality of the integrands
(q) + (1 (1- (og (1- (0)) ovgi (0) log(a (0) + (1 a)) (og)
(11)
holds if and only if a E [0.1] or (0)) (2) Since 91 and q2 are assumed distinct, that is, it holds q1 (0) * (0) on positive measure set, the integral equality in Eq. (10) only holds if a E 1),
and thus entropy strictly concave, which concludes our proof.
Regularized regression as an approximation to constrained optimization In practice, we ap- proximate the optimization problem in Eq. (2) with the regularized regression objective in Eq. (3). As result, we cannot use the result of Proposition 2.1 to guarantee the uniqueness of our solution. However, the dynamic schedule approach to À we use in our work (see Appendix A.3) is similar to the penalty method of approximating solutions to constrained optimization tasks [16, 8]. Future work could use this connection to apply theoretical knowledge of constrained optimization in the source distribution estimation setting.
A.8 Examples related the average posterior distribution
In general, the average posterior distribution is not source distribution. The average posterior distribu- tion is defined in Eq. (7). The infinite data limit is given by (0) n=000 G(0) (()))))
18

---

Here, we provide two examples, one based on coin flips, and one based on a Gaussian bimodal likelihood illustrate this point.
Coin-flip example Consider the classical coin flip example, where the probability of heads (H) follows Bernoulli distribution with parameter 0. The source distribution estimation problem for this setting would consist of the outcomes of flipping n distinct coins, with potentially different values O,.
Proposition A.1. Suppose we have Beta prior distribution on the Bernoulli parameter 0 ~ Beta(a, 3) with parameters = 3 1, and that the empirical measurements consist of 70% heads, i.e:
(0.7
I H I =
Po(x)
0.3
Then the average posterior G(Ø) = p(6) (z))d is not source distribution.or for (z(z).
Proof: Since the Beta distribution is the conjugate prior for the Bernoulli likelihood, the single- observation posteriors are known to be p(0 x H) = Beta(2, 1) and p(0|æ T) (eta 2). Hence, the average posterior is
G(0) = 0.3 Beta(1, 2) + 0.7 B2t.
(12)
However, the ratio of heads observed when pushing this distribution through the Bernoulli simulator is
G# (I H)
8|0.3 Beta(O; (100; + 0.7
(0; 1)]a0
0
0
0 0.3
B(1,2) + 0.7
d0
B2 1)
(13)
/
2
[0.30(1 0) + 0.70%) d0
0.30 0.40 0.567 + 0.7,
where we have used the fact that the Beta function takes the values B(1 = B(2 = 1/2. Therefore, the pushforward of the average posterior distribution does not recover the correct ratio of heads, and so it is not source distribution.
Gaussian bimodal example As another illustrative example to show the differences between average posterior and estimated source, we consider one-dimensional, bimodal Gaussian likelihood given by x ~ 0.5. (æ) - 1, 0.33) + (z5 + 1, 0.33) and the source A (@)0, 0.252). We use the sbi package [54] and perform neural posterior estimation with the uniform prior 0 ~ ([- 5)) to obtain the average posterior and compare to the source estimated with our approach.
While the estimated source matches the original source closely, the average posterior is visibly different and substantially broader (Fig. A1). As expected, this difference persists when sampling from the average posterior and estimated source to simulate from the likelihood. The pushforward distributions data space of the original and estimated source match, while the one the average posterior is again substantially different (Fig. Al).
Additional average posteriors (in comparison to original and estimated source distributions) for the Two Moons and Gaussian mixture are shown in Fig. A6.
A.9 Details on source estimation for the single-compartment Hodgkin- model
We use the simulators as described in Bernaerts et al. [2] for our source estimation. This work provides a uniform prior over specified box domain, which we use as the reference distribution for source estimation. Since the simulator parameters live on different orders of magnitude, we transform the original m--dimensional box domain to the [_,, cube. Note that this transformation does not affect the maximum entropy source distribution. This is because this scaling results in a constant term added to the (differential) entropy. More specifically, for random variable X (associated with
19

---

Bimodal likeiihood example
Parameter space
Data space
Original source
Estimated source
Average posterior
-5
88
Figure Al: Failure of the average posterior as source distribution for the bimodal likelihood example. Each of the individual posteriors is bimodal, resulting in an average posterior with 3 modes (left), the secondary modes produce observations which are not observed in the data distribution when pushed through the likelihood (right), and should not be part of the source distribution.
its probability density p(z)), the (differential) entropy of X scaled by (diagonal) scaling matrix D and shifted by a vector c is given by
H(DX + = H() + log(det D).
(14)
The surrogate trained on 10th parameter-simulatton pairs produced by sampling parameters from the uniform distribution and simulating with the sampled parameters. We do not use the simulated traces directly, but instead compute 5 commonly used summary statistics [2, 18]. These are the number of spikes k transformed by a log (k + 3) transformation (ensuring it is defined in the case of h = 0), the mean of the resting potential, and the first three moments (mean, variance, and skewness) the voltage during the stimulation.
As surrogate, choose deterministic multi-layer perceptron, because found that the internal noise has almost no noticeable effect on the summary statistics, so that the likelihood ( is essentially a point function. We are able make this choice because the sample based nature of our source distribution estimation approach is less sensitive to sharp likelihood functions, whereas likelihood-basee approaches could struggle with such problems.
The multi-layer perceptron (MLP) surrogate has 3 layers with a hidden dimension of 256. ReLU activations and batch normalization were used. Training of the MLP done with Adam (learning rate 5 10 weight decay 10 15 training batch size 4096). Again, 20% of the data were used for validation.
A.10
Computational Resources
All numerical experiments reported in this work were performed on GPU using an NVIDIA A GPU. A single source estimation run for benchmark task using the Sourcerer approach (for one value of X) took approx. 30 seconds. In comparison, learning the source using NEB for the same task took approx. 2 minutes (see Table A1). A source estimation run for Sourcerer the nigh-dimensional tasks took approx. 10 min. When the observations are high- sigh--imensional, training a surrogate (if required) makes up the majority of the computational cost. For the Hadgkin--uxleyy task, training surrogate took approx. 20 minutes, after which estimating the source distribution with Sourcerer took approx. 30 seconds.
20

---

Table A1: Wall-clock runtime comparison between Sourcerer and NEB. Time in seconds mea- sured on an Nvidia A10 GPU. Average and standard deviation are shown over runs. For all three settings (Sourcerer with and without entropy regularization. NEB), surrogate models for the benchmark simulators were used. Sourcerer converges noticeably faster than the NEB baseline.
Method
Sur. (w/o reg.)
Sur. (with reg.)
NEB
TM
29.4 (8.5)
63.9 (10.1)
145.2 (13.9)
IK
28.5 (6.9)
66.7 (10.0)
116,8 (22.6)
SLCP
71.7 (12.8
53.1 (12.2)
91.6
GM
26.6 (5.4)
46.2 (9.2)
98.5 (15.5)
A.11
Supplementary figures
Simulator
Surrogate
10'
TM
Entropy
C2ST
SWD
0.5
10
1.0
10
C2ST
IK
10"
10
12
1.0
10'
SLCP
Entropy
C2ST
10a
SWD
102
GM
Entrpy
C2ST
10"1
0.5
0.125
Figure A2: Extended results for source distribution estimation on the benchmark tasks (Fig. 3) for different choices of X. In addition to the C2ST accuracy and entropy, here the Sliced- Wasserstein distance (SWD) between the observations and the pushforward distribution the estimated source is shown. Mean and standard deviation computed over five runs.
21

---

SIR model
Lolka-Volterra model
3.0
1.0
10"
Entropy
C2ST
SWD
Entropy
C2ST
10
SWD
10
10
10
-3.5
0.015
0.125
NA
0.015
0.125
Predator
90%
Intectious
paputalion
90%
Population densit
50%
10%
70%
Prey
10
20
Time
Simul ations
Observations
10 20
30
40
50
Time
Figure A3: Extended results for source distribution estimation on the differentiable SIR and Lotka- Volterra models (Fig. 4). In addition the Sliced-VV Sliced-Wassersteh Wasserstein distance (SWD), the C2ST accuracy between the observations and the pushforward distribution of the the estimated source shown. Despite the high-dimensional data space of the simulators (50 and 100 dimensions), the estimated sources achieve good C2ST accuracy (below 60%) for various choices of A. Mean and standard deviation were computed over five runs. Additionally, percentile values all samples computed per time point between simulations (simulated with parameters from the estimated source) and observations (simulated with parameters from the original source) closely match.
1.0
1.0
Entropy
TM
0.9
0.5
0.015
À
0.015
12
1.0
1.0
Entropy
SLCP
GM
0.5
-3
NA
0.015
À 0.125
NA
0.015
0.125
1.0
Figure A4: Sourcerer with Maximum Mean Discrepancy (MMD) the differentiable, sample-based distance. We use MMD with an RBF kernel and the median distance heuristic for selecting the kernel length scale. Source estimation is performed without (NA) and with entropy regularization for different choices X. For these tasks, MMD produces similar results to the previously used SWD (Fig. 3b). These results show that Sourcerer is compatible with other sample-based differentiable distances other than the SWD. For all cases, mean C2ST accuracy between observations and simula- tions (lower is better) well the mean entropy of estimated sources (higher is better) over five runs are shown together with the standard deviation.
22

---

Two Moons with 100 observations
25-dimensional Gaussian Mixture
1.0
Entropy
-30
NA
0.015
0.125
1.0
NA 0.015
0.125
1.0
Figure A5: Experiments with less observations and higher- dimeensoaaa sources. Source estimation without (NA) and with entropy regularization for different choices of A. For the Two Moons task, the number of observations was reduced from 10000 to 100. For the Gaussian Mixture task, the dimensionality was increased from to 25. These results show that Sourcerer robust to small datasets of observations, and can estimate nigh-dimmnsiooal source distributions, For all cases, mean C2ST accuracy between observations and simulations (lower is better) as well as the mean entropy of estimated sources (higher is better) over five runs shown together with the standard deviation.
Two moons
Parameter space
Data
Original Estimated source Average posterior
Gaussian Mixture
-5
Figure A6: Original and estimated sources distributions as well as average posterior distribution for Two Moons and Gaussian Mixture simulator with uniform prior 5, 5) For simulators for which the likelihood is unimodal and narrow, such as the Two Moons simulator, the average posterior can be good approximation of a source distribution. However, for simulators where the likelihood is broader, such as the Gaussian Mixture simulator, the average posterior is too broad, and does not reproduce the data distribution P. well, when compared to estimates of source distributions.
23

---

SLCP
Original source Estimated source
Figure A7: Original and estimated source distributions for the benchmark SLCP simulator. The estimated source has higher entropy than the original source.
24

---

Lotka-Volterra model
SIR model
Original source Estimated
8
Figure A8: Original and estimated source distributions for the SIR and Lotka- Volterra model. For the Lotka- Volterra model, the estimated source has higher entropy than the original source.
25

---

Figure A9: 50 random example traces produced by sampling from the estimated source and simulating with the Hodgkin Huxley model.
26

---

000000000000000
Figure A10: 50 random example traces produced by sampling from the uniform distribution over the box domain and simulating with the Hodgkin Badgkin-Huxley model.
27

---

Hodgkin-Huxley model
Source with entropy regularization
Source entropy regularization
0.1
15
20
1000
0.1
70
0
250
0
100
250
-50 3
-100
4000
-90
0.1 V,
-35
R
9,,
9xr
9u
9wa1
9.
E
T
fas
[uFlom+]
[DO]
[ms]
[mSicm]]
[mStam²] [mSrom²] [mS/om²]
[mSicmi] [mSiem?]
[mV]
[ms]
[mV]
[NA]
Figure A11: Estimated sources using for Hodgkin-Huxley Huxley task with the entropy regularization (A = 0.25) and without the entropy regularization. Without, many viable parameter settings are missed, which would have significant downstream effects if the learned source distribution is used as prior distribution for inference tasks.
28

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: We demonstrate in Table our claim that we achieve source distributions with higher entropy than state-of-the-arr comparison, and show results in Fig. 4 and Fig. 5 that our method recovers source distributions on high dimensional tasks and the electrophysiological data, respectively.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We clearly mark the limitations discussion in Sec. 5.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
29

---

Answer: [Yes]
Justification: Proposition 2.1 is stated with full set of assumptions and a complete proof in Appendix A.7.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide pseudocode of our method in Algorithm 1. We provide full details of the architecture of the source model and surrogates in Appendices A.4 and A.5, respectively.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
30

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We public data from existing work which reference for the elec- trophysiological dataset. The code necessary to reproduce our results available at https//ggthb..ommmmcccllabbbttut Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines Bttpii/c/uuddes//oeeeeuubee www.s/wwwwwsssssoosooooooooooaee onPolicy) for more details.
cc/
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //n http:/uullcggguuuee httpe/ggguiieeeeeeeeubiieeeeeeee (CodeSubmissseeeeeee wassoonnoiccyyy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer:
[Yes]
Justification: We provide full details on training the source model in Appendix A.3, A.4, A.5, A.6 and A.9.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
The numerical results in Table I are reported with estimated standard devia- tions, and the figures include error bars showing the standard deviation over an independent set of runs with different random seeds.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
31

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We specify the computational resources used in our numerical experiments in Appendix A0 We provide breakdown the approximate computation time for each of the experiments performed in this work.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported didn't it make it into the paper).
paper (e.g., preliminary or failed experiments that NeurlPS Code of Ethics https: / //te:///nneuee http//nnurrp..ree hwtpp//www....ieccccccceeeeeeeee
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the
Answer: [Yes]
Justification:
We confirm that this work conform with all aspects of the NeurIPS Code of
Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
32

---

Justification: Our work is fundamental in that we develop new approach to solving the source distribution estimation problem. We do develop new classes of models, nor do we apply our approach to problems with societal implications. We do not foresee any direct or indirect misuse of this work.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This work does not involve models that have high risk of misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We use dataset of electrophysiological recordings from Scala et al. [52]. which we cite in the main text.
Guidelines:
The answer NA means that the paper does not existing assets.
33

---

The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- re-packaged, both the original license and the license of the derived asset (if has changed) should provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The public repository contains the code to reproduce our results, along with necessary documentation. is licensed under the MIT license.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
This work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This work does not involve crowdsourcing research with human subjects.
34

---

Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
35


Reviews:
[{"review_id": "QtLRTYjX5x", "reviewer": "Reviewer_swSP", "summary": "The paper introduces “Sourcerer,” a new method for source distribution estimation, focusing on maximum entropy distributions to effectively handle ill-posed problems common in simulating scientific phenomena. This approach leverages the Sliced-Wasserstein distance for sample-based evaluation, offering a significant advantage for simulators with intractable likelihoods, and is demonstrated to recover high-entropy source distributions without sacrificing simulation fidelity across various tasks.", "questions": "- what is the motivation of using the sliced Wasserstein distance? There are other types of distances can be used for this problem. For example, the maximum-mean discrepancy (MMD) and the kernelized Stein discrepancy (KSD). MMD and KSD also have their sliced versions.\n\n- Is there any principle of how to choose the hyperparameter $\\lambda$?\n\n- I know the sliced Wasserstein distance is not really informative because it is estimated via samples thus the density information is a sort of missing. Is there any empirical result of the convergence speeds for the proposed method vs the benchmark NEB?", "limitations": "see the above.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Utilizing the Sliced-Wasserstein distance for evaluating distribution discrepancies enables the method to operate effectively with simulators that have intractable likelihoods.\n\n- The approach is rigorously tested across multiple scenarios, including high-dimensional observation spaces and complex simulators. The results demonstrate that it not only maintains high fidelity in reproducing the empirical distributions but also achieves higher entropy in estimated source distributions compared to existing methods.", "weaknesses": "- The proposed method only applies to differential simulators. I know a lot of these problems in computational biology have non differentiable simulators, as well as the Ising models. This is the drawback of the proposed method.\n\n- I also wish there is more theoretical results in terms of the statistical consistency and error bounds of using the sliced Wasserstein distance. Because the sliced Wasserstein distance is approximated via empirical measures. The estimation error is highly depending on the sample size.\n\n- I'm not sure if the maximum entropy approach is aligned with the intuition. What if the source distribution is highly concentrated in an area with high prob mass? I know $\\lambda$ can be tuned to control it but will it affect the original problem a lot?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thorough feedback. The reviewer raises some questions and concerns about our approach, which we address below.\n\n**W1: \"The proposed method only applies to differential simulators\"**: In the case where the simulator is neither differentiable nor provides explicit likelihood evaluations, our method can still be used, but then requires training a surrogate model.  On our benchmark tasks (Table 1), we have shown that using a surrogate instead of the simulator led to comparable results in terms of simulation fidelity to data, and entropy. The results for the realistic Hodgkin-Huxley task (Fig. 5) were also obtained using a surrogate model. We emphasize that our method has less requirements than previous approaches: We do not require that the simulator have an explicit and differentiable likelihood, but only that the simulator be differentiable. Thus, the set of simulators to which we can apply our method _without_ training a surrogate is larger.\n\n**W2: \"I also wish there is more theoretical results in terms of the statistical consistency and error bounds of using the sliced Wasserstein distance\"**: The Sliced Wasserstein distance (SWD) is well established and its theoretical properties are actively studied. In particular, the sample complexity of the convergence rate has been studied in several existing works. For example, Nadjahi et al. (2020) showed that the finite sample estimate of the SWD between two distributions converges with rate $\\sqrt{N}$ in the number of samples $N$.  Furthermore, Nadjahi et al. (2019) showed that generative models trained to minimize the SWD using finite sample size $N$ also converge to the true optimizer (in distribution) at rate $\\sqrt{N}$. We will add a summary of these properties in our revisions.\n\nWe agree with the reviewer that our benchmark tasks would benefit from exploring how our method performs in the low data limit. Therefore, we provide an additional empirical result by repeating the Two Moons benchmark task with a smaller observed dataset of only 100 samples (Fig. R1a). We observe similar behavior to the baseline case presented in our work for 10000 observations.\n \n[Nadjahi et al. (2020)] - Statistical and Topological Properties of Sliced Probability Divergences. NeurIPS, 2020.  \n[Nadjahi et al. (2019)] - Asymptotic Guarantees for Learning Generative Models with the Sliced-Wasserstein Distance. NeurIPS, 2019.\n\n**W3: \"I'm not sure if the maximum entropy approach is aligned with the intuition\"**: We apologize for not making the intuition for using the maximum entropy principle sufficiently clear. We provide further clarification here and will reflect this in our revisions. \n\nThe maximum entropy principle does not conflict with the case where “the source distribution is highly concentrated in an area with high probability mass”. If parameters outside this area are not consistent with the data observed, our method will result only in distributions that are concentrated in this area. We observe this empirically in the deterministic SIR experiment (Fig. 4), where the source distribution is concentrated and regularization does not lead to higher entropy sources.\n\nInstead, our use of the maximum entropy principle can be viewed as resolving the non-uniqueness of the source distribution estimation problem. According to the maximum entropy principle, given a choice between two source distributions _that are consistent with the data_, one should conservatively choose the source distribution with the higher entropy (which is more dispersed). This is often desirable when source estimation is used to learn prior distributions, as it ensures that a wide range of possible parameters is considered.\n\n**Q1: \"What is the motivation of using the sliced Wasserstein distance?\"**: We thank the reviewer for their question. We agree that other choices of sample-based, differentiable distances are possible. We now perform a new experiment on the benchmark tasks replacing SWD with MMD with an RBF kernel and the median distance heuristic for selecting the kernel length scale (Fig. R3). We observe that the results are comparable to the SWD results presented in our work in terms of simulation fidelity to data and entropy.\n \nThe constraints on the distance function D are computational; D should be differentiable, fast to compute, and not require probability densities. Our new empirical results show that indeed other distances that satisfy these constraints lead to good performance of our method. Our initial choice of using SWD was motivated by its simplicity and scalability - we did not need to choose kernels or other hyperparameters (except for the number of slices).\n\n**Q2: \"Is there any principle of how to choose the hyperparameter?\"**: We empirically find that a small value of $\\lambda$ (e.g. $\\lambda=0.015$), together with our linear decay schedule, is sufficient to obtain substantially higher entropy source distributions in all our experiments. Thus, starting with a small $\\lambda$ is likely to be sufficient. In addition, a run without any regularization can be performed to verify that the regularization does not negatively affect the quality of the estimated source. We agree with the reviewer that more robust methods for choosing $\\lambda$ are an interesting question for future research, and will reflect this in our revisions.\n\n**Q3: \"Is there any empirical result of the convergence speeds for the proposed method vs the benchmark NEB?\"**: We thank the reviewer for their question, and agree that our work can benefit from explicitly comparing the empirical convergence speed of Sourcerer to the baseline. We provide an empirical measurement of the convergence times of our method (with and without regularization) as compared to NEB, measured on the benchmark tasks (Table R1). The source model architectures $q_\\phi$ are the same for all methods, as reported in Table 1 of our original submission. Sourcerer is faster than NEB on all benchmark tasks.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7fXe5lpsky", "reviewer": "Reviewer_DWPT", "summary": "The authors propose to use the maximum entropy principle (possibly tempered with a prior) in order to reduce the ambiguity in solving the problem of source distribution estimation. They propose to use a sample-based technique that optimizes for the Sliced-Wasserstein distance measuring the discrepancy between the simulated data and the real data. They propose experiments on both synthetic and real-world models.", "questions": "- Q1: In some lines authors frame as the maximum entropy principle as something intrinsically beneficial. As they already discuss, this can be modulated by using a reference distribution. Maybe some statements that make the maximum entropy principle stand out as a sort of gold standard could be toned down a bit?\n- Q2: Would it be possible for the authors to show what could happen with the choice of other or wrong distance types?\n- Q3: An interesting case the authors could discuss is when multiple losses may be required for different subsets of parameters?", "limitations": "The authors already address the limitations, or the limitations are intrinsic to the field and cannot be attributed to the proposed methodology.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper is extremely clear, concepts are presented consequentially and the method and experiments are easy to follow\n- The method is simple, but principled and effective on both synthetic and realistic data \n- The paper represents an interesting contribution to the field of source estimation methods.", "weaknesses": "I do not see any major weakness in the work. A minor one:\n\n- W1: it would be interesting to see a more thorough study for models with even higher dimensionality, and how the proposed method's performance scales.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive assessment of our work. We address the reviewer’s questions below.\n\n**W1: \"It would be interesting to see a more thorough study for models with even higher dimensionality\"**: We thank the reviewer for their suggestion to investigate the performance of our approach in high-dimensional parameter spaces. To do so, we perform an additional experiment on the Gaussian Mixture benchmark task. The model is identical to the previous Gaussian Mixture model described in our benchmark task (Appendix 1.4), but now the dimension has been increased to 25. We again find sources that reproduce the data well, and that when we apply entropy regularization, we obtain higher entropy source distributions without any degradation to the pushforward accuracy (Fig. R1b). We also observe that the threshold value of $\\lambda$, at which the pushforward C2ST degrades is now lower than in the 2-dimensional case. We believe that this is due to the entropy of the higher dimensional sources varying on a larger scale, thus restricting our method to smaller values of $\\lambda$ to avoid the entropy term dominating the mismatch term. \n\n**Q1: \"Maybe some statements that make the maximum entropy principle stand out as a sort of gold standard could be toned down a bit?\"**: We agree with the reviewer that statements about maximum entropy being the \"gold standard\" need to be toned down, as this was not our intention. To clarify, we propose maximum entropy as one possible way to solve the ill-posedness of the source distribution estimation problem, which has a number of advantages such as uniqueness, and coverage of all feasible parameters for downstream inference tasks. These are not \"intrinsically\" optimal (and there might be specific cases where other regularizers would be more appropriate), and we will clarify the relevant statements in the revised version.\n\n\n**Q2: \"Would it be possible for the authors to show what could happen with the choice of other or wrong distance types?\"**: We thank the reviewer for their suggestion to use distance metrics other than the Sliced-Wasserstein distance (SWD). Therefore, we repeat our experiments on the benchmark tasks, replacing the SWD with MMD using an RBF kernel and the median distance heuristic for selecting the kernel length scale (Fig. R3). We observe that the results are comparable in terms to the SWD results presented in our work. \n\nThe main considerations in choosing the SWD in our work were computational. We expect that any sample-based, differentiable, and computationally efficient distance metric to measure the mismatch in our objective function (Eqs. 3, 4) will provide reasonable performance. Our initial choice of using SWD was motivated by its simplicity and scalability - we did not need to choose kernels or other hyperparameters (except for the number of slices).\n\n**Q3: \"An interesting case the authors could discuss is when multiple losses may be required for different subsets of parameters?\"**: We strongly agree with the reviewer that there are applications where a general mismatch function such as the SWD might be insufficient to measure the distance between distributions, and some combination of distance functions would be required for different subsets of the data. For example, for simulators that produce multimodal outputs, the SWD may not be an appropriate metric. Instead, we could consider designing a distance custom distance metric that combines the distances over the different modalities of the data. We believe that this is outside the scope of the work, but believe that this is an important point to add to the discussion in our submission.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "42rACQ86vI", "reviewer": "Reviewer_J2pY", "summary": "This paper proposes an approach to estimating a maximum-entropy source distribution (akin to a prior distribution over simulator parameters) for a given set of observations and simulation model. Their method assumes a differentiable simulator that may be deterministic or stochastic, and it uses neural samplers to approximate the prior and a variational objective that encourages proximity of both the marginal likelihood to the true data distribution and the estimated source distribution to some known prior distribution (which may be uniform, corresponding to entropy regularisation). In particular, proximity to the true data distribution is measured with a sliced Wasserstein distance, due to its fast computation and differentiability (preserving the differentiable pipeline). The authors present experiments on four benchmark tasks, before extending to higher dimensional and more complex examples such as the single-compartment Hodgkin-Huxley model.", "questions": "Could the authors provide some more detail and clarification on how their approach differs/relates to/builds on previous works please? Ideally at least those papers already cited such as Vandegar et al. (2020), but I think some additional related literature is also missing, such as references [1] (relevant since it discusses variational approaches to targeting generalised Bayesian posteriors) and [2] (relevant to the problem of source estimation in complex simulation models) below. Is there anything that we lose by using something like the Sliced Wasserstein Distance instead of a likelihood-based discrepancy?\n\n[1] _Knoblauch, J., Jewson, J., & Damoulas, T. (2022). An optimization-centric view on Bayes' rule: Reviewing and generalizing variational inference. Journal of Machine Learning Research, 23(132), 1-109._\n\n[2] _Dyer, J., Quera-Bofarull, A., Bishop, N., Farmer, J. D., Calinescu, A., & Wooldridge, M. (2024, May). Population Synthesis as Scenario Generation for Simulation-based Planning under Uncertainty. In Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems (pp. 490-498)._", "limitations": "The authors provide a good discussion of the limitations towards the end of the paper. Perhaps something additional to discuss would be limitations associated with simulators that involve discrete randomness, which (as far as I can tell) aren't tested in this paper but which often appears in simulation modelling and presents an additional complication when the requirement is that sampling from the simulator is a differentiable operation (see e.g. [1] below). I also do not see any problems related to the broader social impact of this work. \n\n[1] _Arya, G., Schauer, M., Schäfer, F., & Rackauckas, C. (2022). Automatic differentiation of programs with discrete randomness. Advances in Neural Information Processing Systems, 35, 10435-10447._", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "*Originality*\n\nThe authors consider using a sample based loss to capture mismatch between the simulated and true data distributions, in contrast to using likelihood-based notions of distance. This is useful since simulation models often lack tractable likelihood functions. \n\n*Quality* \n\nThe experimental section presents good and extensive empirical testing on a number of toy benchmark models, in addition to two additional more complex simulation models. Their approach is also well-motivated via Proposition 2.1.\n\n*Clarity*\n\nThe clarity of writing is generally good. There are a few minor errors in the writing that made it not absolutely clear all the way through:\n- The last sentence of the first paragraph of Section 1 isn't a full sentence (or, if it is, then it not well-written because I have read it multiple times and could not turn it into a full sentence in my head).\n- There is a subscript $\\phi$ missing in Equation 4. \n\n*Significance*\n\nI think this paper will be of some interest to the community and the techniques presented used by practitioners.", "weaknesses": "My main concern is that the work presented looks like it is not a large or significant contribution beyond what is already present in the literature. From my perspective, it looks like someone could reasonably argue that the contribution is just to use a different sample-based loss function to capture the discrepancy between the data distribution and simulator's marginal likelihood. It also isn't quite clear to me how exactly the approach that uses the surrogate model (Sur. in Table 1) is different from what is presented in Vandegar et al. (2020).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback and careful reading. We address the main concerns and questions raised by the reviewer below. Thank you for identifying typos, we will correct them.\n\n**W1: \"My main concern is that the work presented looks like it is not a large or significant contribution beyond what is already present in the literature\"**: We summarize the two main contributions of our work:\n\nFirst, we provide an approach to resolve the ill-posedness of the source estimation problem by using the maximum entropy principle, and show that it leads to a unique solution.  Empirically, we demonstrate that our approach can estimate high-entropy sources without sacrificing the quality of the match between simulations (from the estimated source) and observations.\nSecond, our approach is fully sample-based. This provides significant computational advantages over previous methods, since our method can use differentiable simulators directly, without training surrogate models.\n\n**W1.1: \"Detailed comparison to Neural Empirical Bayes\"**: Our work makes conceptual, theoretical, and empirical contributions that go beyond Neural Empirical Bayes (NEB, Vandegar et al. (2020)):\n\nFirst, a limitation of NEB - as directly acknowledged in their publication (Section 2, third paragraph) - is that the source distribution estimation problem is ill-posed. This ill-posedness is not addressed by the NEB method. In our work, we propose to use the maximum entropy principle as a way to resolve this ill-posedness, and make the theoretical contribution of showing that the maximum entropy source distribution is unique (Prop. 2.1).  Furthermore, we demonstrate empirically (Table 1, Figures 3,4,5) that our method finds higher entropy source distributions than NEB, without any decline in the predictive performance of the estimated source distributions.\n\nSecond, our approach is entirely sample-based, i.e. it does not require access to likelihoods (or surrogate models with likelihoods).  We show the advantages of our sample-based approach over NEB in the context of high-dimensional, differentiable simulators. In this setting, our method (unlike NEB) does not require the training of surrogate models. We now additionally show (Fig. R2) that our sample-based objective is more robust than NEB even in the case where a surrogate is available for higher-dimensional tasks. For the SIR task, our method achieves a C2ST accuracy (lower is better) of 55% between data and predictive simulations of the learned source distribution, compared to 76% for NEB; similarly, on Lotka-Volterra, our method achieves a C2ST of 56%, compared to 62% for NEB. Thus, our method is demonstrably superior to NEB on these high-dimensional tasks.\n\n**W1.2: \"Comparison to additional works\"**: The reviewer also asked us to contrast our approach with previous work.\n\nRegarding Knoblauch et al. (2022) [1]:  The idea of using other distance measures between distributions is the key approach of Generalized Bayesian Inference (GBI) methods, as discussed in our related work and in [1]. Our method shares similarities with GBI in that it uses non-likelihood-based objective functions. However, we emphasize that our work is concerned with source distribution estimation. This is a different problem than inference, in that we seek to identify a distribution of parameters that is consistent with a population of independent observations (many to many), rather than to explain a single or multiple observations from a single parameter (i.e., posterior inference, one to one or many to one). These are not technical differences, but rather that these methods simply address different goals.\n\nRegarding Dyer et al. (2024) [2]: This problem setting is different from ours. In this paper, the authors seek to find a distribution that produces simulations “matching closely a target, user-specified scenario”. This is different from the source distribution estimation problem, where we try to find a distribution that reproduces observed data. In addition, the entropy regularization has a very different interpretation. In Dyer et al. (2024), the entropy regularization “exhibits the trade-off between (a) the diversity of synthesized populations and scenarios and (b) the ability to identify the most extreme manifestations of the target scenario”. In our work, we show theoretically (Prop. 2.1) and empirically (Table 1, Figures 3,4,5) that no such trade-off is necessary for source distribution estimation.\n\nIn summary, we believe that our work is a significant step upon previous work and hope that our response addresses the reviewer’s concern that our contribution is limited to the use of a sample-based loss.\n\n\n**Q1: \"Is there anything that we lose by using something like the Sliced Wasserstein Distance instead of a likelihood-based discrepancy?\"**: We thank the reviewer for their question. We believe that the main drawback of using sample-based distances is in the low sample limit, where the number of observations $N$ in the true data is very small. Likelihood-based objectives may perform better in this setting, as the value of the likelihood itself contains more information than the sample alone.\n\nWe also evaluated Sourcerer’s performance in a low sample count regime ($N=100$ instead of $N=10000$) in a new experiment for the Two Moons task (Fig. R1a). We observe that our method, using the sample-based Sliced-Wasserstein distance, is still able to recover good sources in this case.\n\n**Q2: \"Perhaps something additional to discuss would be limitations associated with simulators that involve discrete randomness\"**: We thank the reviewer for pointing out this limitation. While we believe that an investigation of discrete simulators is outside the scope of this work, we agree that it merits acknowledgement in the limitations section.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WN9DWMFFPC", "reviewer": "Reviewer_JA3k", "summary": "This paper deals with the problem of identifying the distribution of a source variable $s$ that generates observations $x$.\n\n## The problem\n\nThe authors propose to minimize a classical objective consisting of two terms: \n\n1. A reconstruction term, that encourages the recovered source distribution to induce a distribution over observations that is close to the data distribution, in a certain divergence (here, the sliced Wasserstein)\n\n2. A regularization term, that encourages the recovered source distribution to be close to a reference distribution, in a certain divergence (here, the reverse KL). \n\nMany different source distributions can minimize the first term alone, which motivates the second term to contrain the solution space. \n\n## The probability model\n\nThe probability model $p(x)$ can be expressed in terms of two other model distributions: the source distribution $p(s)$ and the likelihood $p(x | s)$.  \n\nTerm (1) uses a (sliced) Wasserstein divergence. Computing this divergence requires data samples and samples from the model in two steps:  $s \\sim p(s)$ then $x | s \\sim p(x | s)$. Differentiating through this divergence requires the probability model, and therefore $p(s)$ and $p(x | s)$, to be samplable and the sampling process should be differentiable. \n\nTerm (2) uses a reverse KL divergence. Computing this divergence requires evaluating the densities of the model and reference distributions, as well as samples from the model in two steps:  $s \\sim p(s)$ then $x | s \\sim p(x | s)$. Differentiating through this divergence requires the probability model, and therefore $p(s)$ and $p(x | s)$, to be samplable and the sampling process should be differentiable. Additionally, the model density should be known or else a differentiable approximation of the divergence (the authors use the Kozachenko-Leonenko estimator) should be available. \n\n## Numerics\n\nThe authors validate their method using synthetic data where the source is known, and real data where the observations are neurophyisiological recordings and the likelihood is given by a mechanistic model.", "questions": "Could the authors address the weaknesses part?", "limitations": "Yes.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The presentation is clear and well-explained.", "weaknesses": "Overall, I have two concerns. \n\n1. The computational constraints on the model are heavy\n\nThe choices of divergences (Sliced-Wasserstein and reverse KL) in the cost function impose a number of computational constraints of the model.\n\nTerm 1 in the cost uses a vanilla Sliced-Wasserstein divergence estimator that is sample-based. This means the likelihood model $p(x | s)$ should be samplable and the sampling process should be differentiable. The authors propose to train a differentiable approximator of a black-box simulator. To be fair, the authors acknowledge that this is a general issue beyond their paper. \n\nTerm 2 in the cost uses a vanilla KL divergence estimator which requires knowing the density of the model, but the authors circumvent this by using another estimator named \"Kozachenko-Leonenko\". Given that the authors already used a sample-based divergence in Term 1, wouldn't using another sample-based divergence in the Term 2 (e.g. the Sliced-Wasserstein again, or an MMD) between the model and reference distributions not add any further computational constraints on the model?\n\n2. The evaluation procedure or goal remains unclear to me\n\nThe framework of minimizing a cost made up of a reconstruction and regularization term is well-studied in optimization. Once could imagine two natural evaluation procedures:\n- how well the true sources are recovered? But the authors are clear that designing an identifiable probability model where the sources can be recovered (up to acceptable indeterminacies) is not the goal here. \n- how efficient is their method on a new test dataset, as compared to other similar methods?\n\nIt seems to be that the authors answer the latter, but it is not clear to me what are the other methods to benchmark against.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback. We address the reviewer's concerns below.\n\n**W1: \"The computational constraints on the model are heavy\"**: The reviewer raises concerns about computational constraints on the model, which we clarify here for the two terms in the cost function:\n\nFirst, the entropy/KL-divergence term (term 2) imposes negligible computational constraints on the model because it is computed on samples in parameter space, and not in data space. In particular, given samples $\\theta_i$ from our source model, we only need to compute their nearest-neighbor distances $d_i$ to estimate the entropy (Eq. 8 of our submission). Thus, its computation is completely independent of the simulator or surrogate. Hence, this concern is not a limitation of the method.\nIn addition, while other distances (such as Sliced-Wasserstein or MMD, as suggested by the reviewer) can be used to regularize the source in the second term, we emphasize that the choice of entropy (KL-divergence wrt. uniform distribution) stems from the original motivation of the work (i.e., the theoretical justification in Prop 2.1.)\n\nSecond, we agree with the reviewer that requiring the simulator to be differentiable for term 1 is a computational constraint on our model. However, we emphasize that our approach is an improvement over prior work (in particular, Vandegar et al. (2020)) in this regard. As the reviewer identified, the requirement to train surrogates for non-differentiable models is a broader limitation of the field. However, typical works in the field require training a surrogate that has a tractable _and_ differentiable likelihood. We only require differentiability. We do not require tractable likelihoods, which reduces the computational constraints. This is a relative strength of our method relative to previous approaches, not a limitation.\n\n[Vandegar et al.] - Neural Empirical Bayes: Source distribution estimation and its applications to simulation-based inference. In International Conference on Artificial Intelligence and Statistics, 2020.\n\nFinally, in terms of computational cost, the choices we make for the two terms in the loss function are highly efficient. For the mismatch term (term 1), the Sliced-Wasserstein distance is linearithmic in the number of observations, and thus scales well to millions of observations. Furthermore, the distance computation of individual 1-D slices is embarrassingly parallelizable. Our use of Kozachenko-Leonenko estimators to estimate entropy (term 2) is also scalable, since computing the estimate requires solving the all-nearest-neighbors problem, which can also be solved in linearithmic time. We now perform a runtime comparison between NEB and Sourcerer for the benchmark tasks, and find that Sourcerer is significantly faster (Table R1).\n\n\n**W2: \"The evaluation procedure or goal remains unclear to me\"**: We apologize for not making our evaluation procedure sufficiently clear. For the benchmark tasks as well as the SIR and Lotka-Volterra experiment, we always evaluate the match between simulations and observations on a second, unseen ‘test’ set of observations that was generated independently using samples from the ground truth source distribution. Thus, we already do the evaluation in the way the reviewer asks, and just failed to point this out in the manuscript. We will clarify in the revised version.\n\nMore generally, our evaluation is always based on two criteria:  \n1. How well do the simulations (generated by the estimated source) match the set of observations? We measure this match using Classifier Two Sample Tests (C2ST) and the SWD.  \n2. How high is the entropy of the estimated source? We measure this by estimating the differential entropy with samples from the source using the Kozachenko-Leonenko estimator.\n\nWe benchmark our  method against Neural Empirical Bayes (NEB), a state-of-the-art approach for source distribution estimation. For the benchmark tasks, we showed that Sourcerer obtains sources that are comparable to NEB in terms of simulation fidelity (point 1), while having a higher entropy (point 2). We now also perform an additional comparison on the high-dimensional data tasks of SIR and LV (Fig. 2), and find that our method also significantly outperforms NEB in terms of simulation fidelity (point 1). On SIR, Sourcerer achieves a C2ST accuracy (lower is better) of 55% between data and predictive simulations of learned source distribution, compared to 76% for NEB; similarly, on Lotka-Volterra, Sourcer achieves a C2ST of 56%, compared to 62% for NEB.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BBgVTcMby7", "reviewer": "Reviewer_8Zuo", "summary": "This paper introduces a new algorithm for source estimation, the task of estimating a model's parameters probability distributions consistent with observations. In opposition to previous algorithms, the proposed method is sampling-based and can hence be used when the model is a simulator, implicitly defining the likelihood even if the observations are high-dimensional. The authors highlight that several probability distributions might be consistent with the observations and advocate favoring the one with the highest entropy. Their method provides a way to target high-entropy solutions and shows that this new objective leads to a unique optimum.", "questions": "* Is there something that prevents you from comparing against NEB in Figure 4?", "limitations": "The authors have adequately addressed the limitations.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "### Originality\n* The method is new to me.\n\n### Quality\n* I did not identify flaws in the mathematical derivations.\n* The method is evaluated on a wide variety of benchmarks, and several runs have been made. The experiments showcase that the entropy regularization indeed leads to higher entropy sources.\n\n### Clarity\n* All the necessary details are provided\n* The paper is well articulated.\n* The method is clearly explained\n* It is clearly stated how the method fits in the literature. \n* Figures are clean.\n\n### Significance\n* Source estimation is a task that has received attention in recent years.\n* This paper fills a gap in the literature by providing a method that works with high-dimensional simulators.", "weaknesses": "### Originality\n* I have no concerns regarding originality.\n\n### Quality\n* I find that the Lotka-Volterra and SIR experiments are not totally convincing. First, it lacks comparison with NEB, which is claimed to perform poorly in such settings. I agree that NEB will probably provide poor results, but this should still be shown empirically, in my opinion. Second, the Wasserstein distance is hard to interpret. It is claimed that a low value is observed, but that is not clear to me why this value can be considered a low value as there are no baselines to compare with.\n\n### Clarity\n* I have no concerns regarding clarity.\n\n### Significance\n* I have no concerns regarding significance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive feedback of our work.\n\n**W/Q1: \"Is there something that prevents you from comparing against NEB in Figure 4?\"** We thank the reviewer for their suggestion, and have now compared Neural Empirical Bayes (NEB) with Sourcerer on the high-dimensional simulators (SIR and Lotka-Volterra model). Results for this experiment can be found in the provided rebuttal supplement (Fig. R2). In summary, we find that Sourcerer performs better on both tasks: on SIR, Sourcerer achieves a C2ST accuracy (lower is better) of 55% between data and predictive simulations of learned source distribution, compared to 76% for NEB; similarly, on Lotka-Volterra, Sourcerer achieves a C2ST of 56%, compared to 62% for NEB. Thus, Sourcerer is demonstrably better than NEB on these high-dimensional tasks.\n\nTo perform this experiment, we trained surrogates for each simulator. While this was not necessary for Sourcerer, attempting to perform NEB on the differentiable likelihood of the differential equation simulators was computationally infeasible due to the large minibatches required for NEB. To enable surrogate training, we reduced the observation dimensionality of both models to 25 (instead of 50 and 100 respectively), and added Gaussian noise to the previously deterministic SIR simulator. For a fair comparison, we use the same surrogate model for both Sourcerer and NEB.\n\nWe argue that Sourcerer's improved performance over NEB on these tasks is due to the different ways in which the two approaches use the surrogate model - NEB is sensitive to the exact likelihood, and thus may suffer from a lack of robustness to misspecification as in the case of a surrogate model. Our method, on the other hand, requires only samples, and thus the exact likelihoods produced by the surrogate model do not have a detrimental effect.\n\n**W1.1 \"The Wasserstein distance is hard to interpret. It is claimed that a low value is observed, but that is not clear to me why this value can be considered a low value as there are no baselines to compare with\"**: We agree with the reviewer that the numerical value of the Sliced-Wasserstein Distance (SWD) is hard to interpret in isolation. We had therefore computed the expected SWD between two sets of independently generated observations (simulated from the true source distribution). This minimum achievable distance is indicated by the black dotted lines in Figure 4 of our original submission. We found that for both models, the distance between simulations (from the estimated source) and observations is very close to the expected distance between simulations, and will emphasize this in the revision. Finally, we also provide the more interpretable C2ST accuracies for the SIR and Lotka-Volterra experiments in the Appendix (Fig. 8). In both cases, Sourcerer achieves a C2ST accuracy close to 50%.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank the reviewers for their detailed engagement and constructive and positive feedback. Our paper introduced Sourcerer, a method for estimating maximum-entropy source distributions with sample-based distances. Reviewers found our approach to be “well-motivated” (J2pY) and “simple, but principled and effective” (DWPT). They commended our evaluation “[on] a wide variety of benchmarks” (8Zuo) to be “extensive” (J2pY) and “rigorous” (swSP). Finally, all reviewers found the presentation of our work to be good or excellent, with DWPT calling it “extremely clear”. 

We have responded to each of the reviewers in detail to clarify their questions and address their  concerns. We have performed several additional experiments (see supplement) to support our response. Below, we summarize the main points raised by the reviewers, in addition to the results of the new experiments. We hope that our responses address the reviewers’ concerns and allow them to recommend our work for acceptance.

**Clarifying the use of the maximum entropy principle (swSP, J2py, DWPT)**: The source distribution estimation problem is known to be ill-posed - we here proposed to use the maximum entropy principle to address this issue, and showed (Prop. 2.1) that the resulting source distribution is unique. Maximum entropy approaches are often motivated by the idea of “maximum ignorance” - i.e., finding a distribution that makes the least assumptions about the model while satisfying all the known information. A maximum entropy approach is also arguably desirable when source estimation is used to learn prior distributions, as it ensures that a wide range of possible parameters is considered. We showed empirically that our regularized approach is able to obtain a high entropy source distribution without sacrificing the quality of the match between simulations (from the estimated source) and observations.

Reviewer swSP raised a concern about the case where “the source distribution is highly concentrated in an area with high probability mass”.  However, if the parameters outside this area are not consistent with the observed data, our method will result only in distributions that are concentrated in this area. Nevertheless (and as with any regularization term), there are cases where other properties are desirable and where there might be more appropriate objectives to select the source distribution (as also pointed out by DWPT). We will provide a balanced discussion of when entropy regularization might be desirable.

**Clarifying the requirement of differentiable simulators (JA3k, swSP, J2py)**: Another shared concern was the requirement that the simulators be differentiable. We emphasize that this requirement is less restrictive than the ones from previous work (Vandegar et al. (2020)), which required the simulator _and_ its (log-) likelihood to be differentiable. Furthermore, if the simulator is not differentiable, or if computing its gradients is too expensive, our method can still be used. In this case it is necessary to train a surrogate model of the simulator. For example, in our manuscript, we train a surrogate model in the Hodgkin-Huxley experiment because some of the summary statistics used are not differentiable.

[Vandegar et al.] - Neural Empirical Bayes: Source distribution estimation and its applications to simulation-based inference. AISTATS, 2020.

## New Experiments

For new figures (Fig. Rx) and Table R1, please see the **PDF**.

**1. Estimating source distributions with higher dimensionality and less observations (DWPT, swSP)**: Reviewers asked for additional experiments to study the performance of our method in two cases. We now study the robustness of our method to a small number of observations in the dataset (swSP). We repeat the benchmark Two Moons task with $N=100$ observations, and find that our method still identifies high entropy source distributions that reproduce the observed dataset very well (Fig. R1a). We also add a new experiment studying the robustness of our method in the high-dimensional source distribution limit (DWPT). We repeat the Gaussian Mixture task with the source distribution dimension increased to $D=25$ (Fig. R1b), which is almost twice the dimensionality of the highest-dimensional source estimated in our original submission ($D=13$ in the Hodgkin-Huxley example). Again, we find that our method identifies a high entropy distribution with excellent match to the observed data.

**2. Additional comparison against NEB baseline (8Zuo, swSP, JA3k)**: Reviewers requested comparisons to baseline models in terms of convergence speed. We now show empirically that Sourcerer is significantly faster than the Neural Empirical Bayes (NEB) baseline across all benchmark tasks (Table R1 in supplement). In addition, we now also compare our results for the high-dimensional tasks of the SIR and LV models to the NEB baseline (Fig. R2) and show that Sourcerer performs significantly better on these tasks.

**3. Sourcerer with MMD (swSP, DWPT, JA3k)**: Reviewers asked whether the choice of the Sliced-Wasserstein distance (SWD) to measure the mismatch between simulations from our source model and the data was the only appropriate choice. We have clarified the constraints that our approach requires of the distance function to satisfy, namely that the distance must be sample-based, differentiable, and fast to evaluate. We chose to use the SWD because it meets all of these requirements and also requires few hyperparameter choices. However, other sample-based distance metrics are also possible: We have now performed additional experiments on our benchmark tasks, replacing the SWD with the Maximum Mean Discrepancy (MMD). We observe similar quantitative performance as with SWD (Fig. R3).

Meta-review: Except for Reviewer swSP, all reviewers recommend acceptance (4-7-5-6-7). Reviewers all agree on the soundness of the approach, on the overall clarity of the presentation, as well as on the quality of the empirical evaluation. In my opinion, this contribution also makes a valuable step forward in the field; proposing to follow the maximum entropy principle when solving the source estimation problem is well-principled and the use of a sample-based loss does fit well with the simulation-based inference setting, even if the differentiability of the simulator is required. 

The authors-reviewers discussion has been constructive and has led to a number of clarifications and new results, to the satisfaction of the reviewers. The concerns raised by Reviewer swSP are legitimate, but the authors have, in my opinion, provided convincing answers and new results to address them. Despite the lack of acknowledgment from the reviewer, I therefore consider that the major concerns have been addressed. 

For all these reasons, I recommend acceptance. The authors are asked to implement the clarifications and add the new results discussed with the reviewers in the final version of the paper.
Decision: Accept (poster)