Arbitrary Conditional Distributions with Energy
Authors: Ryan Strauss
Junier Oliva
Venue: NeurIPS 2021 Poster (2021)
Keywords: density estimation
arbitrary conditioning
energy-based models
imputation
unsupervised learning

Abstract: Modeling distributions of covariates, or density estimation, is a core challenge in unsupervised learning. However, the majority of work only considers the joint distribution, which has limited relevance to practical situations. A more general and useful problem is arbitrary conditional density estimation, which aims to model any possible conditional distribution over a set of covariates, reflecting the more realistic setting of inference based on prior knowledge. We propose a novel method, Arbitrary Conditioning with Energy (ACE), that can simultaneously estimate the distribution $p(\mathbf{x}_u \mid \mathbf{x}_o)$ for all possible subsets of unobserved features $\mathbf{x}_u$ and observed features $\mathbf{x}_o$. ACE is designed to avoid unnecessary bias and complexity --- we specify densities with a highly expressive energy function and reduce the problem to only learning one-dimensional conditionals (from which more complex distributions can be recovered during inference). This results in an approach that is both simpler and higher-performing than prior methods. We show that ACE achieves state-of-the-art for arbitrary conditional likelihood estimation and data imputation on standard benchmarks.

Arbitrary Conditional Distributions with Energy
Ryan R. Strauss
Junier B. Oliva
Department of Computer Science UNC at Chapel Hill
Department Computer Science UNC at Chapel Hill Chapel Hill, NC 27514 joliva@cs unc edu
Chapel Hill, NC 27514 rrsêcs. unc. edu
Abstract
Modeling distributions of covariates, or density estimation, is a core challenge in unsupervised learning. However, the majority of work only considers the joint distribution, which has limited utility in practical situations. A more general and useful problem is arbitrary conditional density estimation, which aims model any possible conditional distribution over a set of covariates, reflecting the more realistic setting of inference based on prior knowledge. We propose novel method, Arbitrary Conditioning with Energy (ACE), that can simultaneously estimate the distribution p( xx x,) for all possible subsets of unobserved features x,, and observed features x., ACE is designed to avoid unnecessary bias and complexity
we specify densities with highly expressive energy function and reduce the problem to only learning mne-dimensional conditionals (from which more complex distributions can be recovered during inference). This results in an approach that is both simpler and achieves sute-of-the-arr for arbitrary conditional likelihood estimation and data imputation on standard benchmarks.
than prior methods. We show that ACE
1 Introduction
Density estimation, core challenge in machine learning, attempts learn the probability density of some random variables given samples from their true distribution. The vast majority of work on density estimation focuses on the joint distribution p(x) I0 25 I1 23 22 6), i.e., the distribution of all variables taken together. While the joint distribution can be useful (e.g.. to learn the distribution of pixel configurations that represent human faces), limited in the types of predictions it can make. We are often more interested in conditional probabilities, which communicate the likelihood of an event given some prior information. For example, given a patient's 's medical history and symptoms, doctor determines the likelihoods of different illnesses and other patient attributes. Conditional distributions are often more practical since real-world decisions are nearly always informed by prior information.
However, we often do not know ahead of time which conditional distribution (or distributions) will be of interest. That is, we may not know which features will be known (observed) or which will be inferred (unobserved). For example, not every patient that doctor sees will have had the same tests performed (i.e., have the same observed features). A naïve approach requires building an exponential number of models to cover all possible cases (one for every conditional distribution), which quickly becomes intractable. Thus, intelligent system needs to understand the intricate conditional dependencies between all arbitrary subsets of covariates, and must do so with a single model to be practical.
In this work, we consider the problem of learning the conditional distribution p(x,, xx) for any arbitrary subsets of unobserved variables x, E R|a and observed variables x, RR1, where u, (1.....d d} and on 0. We propose method, Arbitrary Conditioning with Energy (ACE),
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

that can assess any conditional distribution over any subset of random variables, using a single model. ACE is developed with an eye for simplicity we reduce the arbitrary conditioning problem to the estimation of one-dimensional conditional densities (with arbitrary observations), and we represent densities with an energy function, which fully specifies unnormalized distributions and has the freedom to be instantiated as any arbitrary neural network.
We evaluate ACE on benchmark datasets and show that it outperforms current methods for arbitrary conditionall/marginll density estimation. ACE remains effective when trained on data with missing values, making it applicable to real-world datasets that are often incomplete. and we find that ACE scales well to high- dige-dimensional data. Also, unlike some prior methods (e.g., (20),, ACE can naturally model data with both continuous and discrete values.
Our contributions are as follows: 1) We develop the first energy-based approach to arbitrary con- ditional density estimation, which eliminates restrictive biases (e-g- normalizing flows, Gaussian mixtures) imposed by common alternatives. 2) We empirically demonstrate that ACE is state-of-the- art for arbitrary conditional density estimation and data imputation. 3) We find that complicated prior approaches can be easily outperformed with a simple scheme that uses mixtures of Gaussians and fully-connected networks.
2
Previous Work
Several methods have been previously proposed for arbitrary conditioning. Sum-Product Networks are specially designed to only contain sum and product operations and can produce arbitrary conditional or marginal likelihoods (27 3] The Universal Marginalizer trains neural network with cross- entropy loss to approximate the marginal posterior distributions of all unobserved features conditioned on the observed ones (5]. VAEAC is approach that extends conditional variational autoencoder by only considering the latent codes of unobserved dimensions [1), and NeuralConditioner uses adversarial training to learn each conditional distribution D. DMFA uses factor analysis to have a neural network output the parameters of a conditional Gaussian density for the missing features given the observed ones (28). The current Mfate-of-the-arr is ACFlow, which extends normalizing flow
models to handle any subset of observed features (20]
Unlike VAEAC, ACE does not suffer from mode collapse or blurry samples. ACE is also able to provide likelihood estimates, unlike NeuralCondittoner which only produces samples. DMFA is limited to learning multivariate Gaussians, which impose bias and are harder to model than one- dimensional conditionals. While ACFlow can analytically produce normalized likelihoods and samples, it is restricted by requirement that its network consist of bijective transformations with tractable Jacobian determinants. Similarly, Sum-Product Networks have limited expressivity due to their constraints. ACE, on the other hand, exemplifies the appeal of energy- methods it has no constraints on the parameterization of the energy function.
Energy- based methods have wide range of applications within machine learning 17 and recent work has studied their utility for density estimation. Deep energy estimator networks 30 and Autoregressive Energy Machines [22] are both energy-based models that perform density estimation. However, both of these methods are only able to estimate the joint distribution.
Much of the previous work on density estimation relies on an autoregressive decomposition of the joint density according to the chain rule of probability. Often, the model only considers a single Uria et al. (33] proposed a method for assessing joint likelihoods based on any arbitrary ordering, where they use masked network inputs effectively share weights between a combinatorial number of models. Germain et al. [8] also consider shared network for joint density estimation with a constrained architecture that enforces the autoregressive constraint in joint likelihoods. In this work, we construct an arder-agnostic weight-sharing technique not for joint likelihoods, but for arbitrary conditioning. Moreover, we make use of our weiigh-sharrng scheme to estimate likelihoods with an energy based approach, which avoids the limitations of the X Query Training [16] is method for answering probabilistic queries. It also takes the approach of computing one-dimensional conditional likelihoods but does not directly pursue an autoregressive extension of that ability.
(arbitrary) ordering of the features 22 24
parametric families used previously (e.g., mixtures of Gaussians 33,, or Bernoullis
2

---

The problem of imputing missing data has been well studied, and there are several approaches based on classic machine learning techniques such as k-nearest neighbors [32]. random forests [T]] and autoencoders [9 More recent work has turned to deep generative models. GAIN is generative adversarial network (GAN) that produces imputations with the generator and uses the discriminator to discern the imputed features 34. Another GAN-based approach is MisGAN, which learns two generators to model the data and masks separately [9]. MIWAE adapts variational autoencoders by modifying the lower bound for missing data and produces imputations with importance sampling 21. ACFlow can also perform imputation and is state-of-the-art for imputing data that are missing completely at random (MCAR) [2]
While it is not always the case that data are missing random, the opposite case (i.e., missingness that depends on unobserved features' values) can be much more challenging to deal with [7]. Like many data imputation methods, we focus on the scenario where data are MCAR, that is, where the likelihood of being missing is independent of the covariates' values.
3 Background
3.1 Arbitrary Conditional Density Estimation
probability density function (PDF), p(x), outputs a nonnegative scalar for a given vector input x E I2 and satisfies / p(x) dx 1. Given a dataset = [x(*),
]== of i.i.d. samples drawn from an unknown distribution p" (x), the object of density estimation is to find a model that best approximates function p² Modern approaches generally rely on neural networks to directly parameterize the approximated PDF
Arbitrary conditional density estimation is a more general task where we estimate the conditional density (xx xx) for all possible subsets of observed features C. {L,.... d} (i.c., features whose values are known) and corresponding subset of unobserved features
{I, {I...., d} such that o and do not intersect. Here X, E IR and X R| The estimation of joint or marginal likelihoods is recovered when is the empty set. Note that marginalization to obtain p(x.) = J p(xu: X,) dx
(and hence p(x, X₂) p(x....o) is intractable in performant generative models like normalizing
p(x.)
flow based models 44 20 thus, we propose a weight-sharing scheme below.
3.2 Energy-Based Models
Energy-based models capture dependencies between variables by assigning nonnegative scalar energy to given arrangement of those variables, where energies closer to zero indicate more desirable configurations [17). Learning consists of finding an energy function that outputs low energies for correct values. We can frame density estimation as an energy-based problem by writing likelihoods as Boltzmann distribution
-E()
p(x)=
7 z=
E(x) dx -B())
Z
where E is the energy function, e E(() is the unnormalized likelihood, and is the normalizer.
Energy-based models are appealing due to their relative simplicity and high flexibility in the choice of representation for the energy function. This is in contrast to other common approaches to density estimation such normalizing flows 4 20), which require invertible transformations with Jacobian determinants that can be computed efficiently. Energy functions are also naturally capable of representing non-smooth distributions with low-density regions or discontinuities.
4 Arbitrary Conditioning with Energy
We are interested in approximating the probability density p(x, xx) for any arbitrary sets of unobserved features x,, and observed features x,, We approach this by decomposing likelihoods into products of one-dimensional conditionals, which makes the learned distributions much simpler. This concept a basic application of the chain rule of probability, but has yet to be thoroughly exploited for arbitrary conditioning. During training, ACE estimates distributions of the form p(zu, xx)).
3

---

k
Popuud
Enery Setwort
(a) Proposal Network
(b) Energy Network
Figure 1: Overview of the networks used in ACE. The plus symbol refers concatenation.
where zut is a scalar. During inference, more complex distributions can then be recovered with an autoregressive decomposition: p(x, x,) = II p( Xou where w' an arbitrary permutation of u and u'z, = (u'...., u/-17- This approach is appealing because the estimated one-dimensional domain, allowing one a myriad of flexible estimators.
distributions are
We adopt an energy-based approach (similar to AEMs (22) which affords large degree flexibility free to represent the energy function with an arbitrary, and highly expressive, neural network that directly outputs unnormalized likelihoods. This contrasts with the current state-of-ie tore---ftthharrrr for arbitrary conditional density estimation, which is limited by normalizing flow transformations [20]. Energy functions are highly expressive and naturally model complex (i.e., multimodal, non-smooth) densities, they avoid a parametric-famil on the shape of the distribution. Our main contribution a method, Arbitrary Conditioning with Energy (ACE), for computing arbitrary conditional likelihoods with energies, one dimension at time !
in modeling the exponentially many conditional distributions hand
4.1 Decomposing Densities
We decompose the arbitrary conditioning task into 1d-domain arbitrary conditional estimation problems, This follows from the chain rule of probability, which allows us to write
[ul
p(xu xx)
II (Iug Xouu'z)
(2)
where w' is an arbitrary permutation of u, Zut the 7th unobserved feature given by w', and W/z = [u' 1. The one-dimensional conditionals in Equation 2 are themselves arbitrary con- ditionals due to the choice of permutation. Thus, the arbitrary conditioning problem may be reduced to one-dimensional arbitrary conditional estimation. we can compute any likelihood of the form p(z, x,y) for of {1....,d), d), possible conditional distributions, marginal distributions, and the joint distribution. Although the reduction of arbitrary conditioning to 1d conditional estimation is seemingly straightforward, this is, to the best of our knowledge, the first work to leverage this formulation.
compute any distribution over the features. This includes all
We train our model non-autorrerressively, to output the likelihood p(z; xx/) for any and x, where E (I....,
....,,d d} \ o'. During inference, an autoregressive procedure that repeatedly applies the
chain rule can be used to compute likelihoods of the form p(x.. x.). In practice, this consists of simply adding the previously considered unobserved dimension to the observed set before moving to the next step in the chain
4.2 Likelihoods from Energies
We express likelihoods in terms of energies by modifying Equation to write
-E(x,,
p(
x.)
(3)
Zuixo
and we choose to represent the energy function as a neural network. order to avoid learning a
different network for each conditional, we adopt hiigh--hhring scheme (e.g. B3 [4] 20
That is,
An implementation of ACE is available at pesttps://giaan "One could also implement this with masked network architecture (e.g.. MADE [)) that can more efficiently compute autoregressive likelihoods. We choose not to do this for simplicity and to enable the use of arbitrary. and more expressive, architectures.
com/lupalab/ace

---

we use a bitmask b E {0.1}² that indicates which features are observed and define zero-imputing function @ (xx,; b) that returns a d-dimensional vector where unobserved features are replaced with zeros (see Figure 5 in Appendix). These are then used as inputs to the energy network (see (see Figure Thp.
Approximating normalized likelihoods. We use Equation 3 to compute normalized likeli- hoods, but only if the normalizing constant Zunx, is known. Directly computing the normalizer is intractable in general. However, importance sampling can be used to obtaín an estimate [2] Nash and Durkan [22] show that this technique is sufficiently accurate when considering one-dimensional conditionals for joint density estimation, and we adapt the approach for arbitrary conditioning. As-
suming access to proposal distribution q xx) which is similar to the target distribution, we approximate Zu,;x. as
= =ʃ (-E(mu dx = e- (atu, (Xo)
Zu Zucxa
q xxo) g(arve Xx) dxu.
(4)
Els
xx)
23
5
1
(1) (ati Xo)
2/1 a(Tus (x)
(5)
==1
For some problems, we may have access to good proposal distribution ahead time, but in general, we can learn in parallel with the energy network. We note that importance sampling is also more computationally efficient than the simpler alternative of using points from a grid. Since distributions may be non-smooth and span large domain, fewer samples from good proposal distribution will be able to obtain as good of an estimate as grid that has many more points, which in turn will require fewer evaluations of the energy network. Furthermore, generating samples from the proposal is very efficient, as it only requires one neural network evaluation to obtain the parameters tractable parametric distribution, which can then be easily sampled.
4.3 Training ACE
We learn the proposal distribution alongside the energy function by having a neural network output the parameters of tractable parametric distribution. For the proposal network, we again share weights between all conditionals as is done with the energy network. The proposal network accepts concatenation of b and o(x;; b) as input, and it outputs the parameters w (u,;x,) for mixture of Gaussians for each unobserved dimension ut The proposal network also outputs latent vector, r(u; x.). for each unobserved dimension, which is used input to the energy network in order to enable weight sharing between the proposal and energy networks (22]. Using a different latent vector for each feature allows the latent vectors represent the fact that different unobserved features may depend on x, in different ways.
We then estimate the normalizing constants in Equation 3 Jusing importance sampling:
9
E( ()(T(1)))))
Zuiixo S g(
(6)
(8)
((uxx)))
where (s) is sampled from gz w(u; x.)). This in turn leads to the following approximation of the log-like likeeloooood of zu, given X,:
log p(ru | xx) R Ex Xo: (u:; x.)) log Zusp..
where we use abbreviated notation in the previous two equations and omit the bitmask b for greater readability. Refer to Figure 1 for the precise inputs of each network
Since Equation 2 gives us a way to autoregressively compute p(x., xx) as a chain of one-dimensional conditionals, we only concern ourselves with learning p(I,,, xx)) for arbitrary u4 and x.. Thus, for a given data point x, we randomly partition it into x, and x,, and jointly optimize the proposal and energy networks with the
likelihood objective
[u]
[u]
J Jxxo: x.: 0)
log p(.,. xx) +
Σ log q(Eu, xx))
(8)
'It is not strictly necessary for the energy network to include x. input, since the latent vectors can encode that information. However, doing acts like sort of skip ciipcconnection, which often helps with deeper networks.
5

---

where 6 holds the parameters of both the energy and proposal networks. Because we want to optimize the proposal and energy distributions independently, gradients are stopped on proposal samples and proposal likelihood before they are used in |Equation 6) We note that optimizing Equation 8 can be interpreted as a stochastic approximation to optimizing the full autoregressive likelihoods p(x,, x,) (i.e.. Equation 2). For a given data point, we are choosing to optimize individual 1d-conditionals from numerous hypothetical autoregressive chains, rather than all of the ld-conditionals from a single autoregressive chain. That is, each random set of observed values that is encountered during training (in Equation 8) can be seen as a single factor in the product in Equation 2 for some (X,, Xx). Over the course of training though, the same conditionals are ultimately being evaluated in either case.
The negative of Equatiin ijis minimized with Adam [15] over set of training data, where observed and unobserved sets are selected at random for each minibatch (see /Section 5). In some cases, we found it useful include a regularization term in the loss that penalizes the energy distribution for large deviations from the proposal distribution. We use the mean-square error (MSE) between the proposal likelihoods and energy likelihoods as a penalty, with gradients stopped on the proposal likelihoods in the error calculation. The coefficient of this term in the loss is a hyperparameter.
4.4 Inference
4.4.1
Likelihoods
Recall that our model learns one-dimensional conditionals: p( xx). Thus, to obtain a complete likelihood for X., we employ an autoregressive application of the chain rule (see [qquatinn 2). The pseudocode for this procedure is presented in the Appendix (see [Algorithm I}. Since the values of x
are known ahead of time, each one-dimensional conditional can be evaluated in parallel as batch, allowing the likelihood p(x,, xx) to computed efficiently.
Importantly, the order in which each unohserved dimension is evaluated does not matter. As argued in previous work 33 8) this can be considered advantageous, since we can effectively leverage multiple orderings at test time to obtain an ensemble of models. However, this does incur extra computational cost during inference. We note that a model which perfectly captures the true densities would give consistent likelihoods for all possible orderings (thus evaluating only one ordering would suffice). However, current arder-agnostic methods (such as ACE or [33 g)
do not inherently satisfy this constraint. In the Appendix. we study how these inconsistencies can be mitigated in ACE, but we leave a true solution this challenge for future work.
4.4.2 Imputing
Sampling allows us to obtain multiple possible values for the unobserved features that are diverse and realistic !]] However, these are not always the primary goals. For example, in the case of data imputation, we may only want a single imputation that aims to minimize some measure of error (see Section 5.1.2). Thus, rather than imputing true samples, we might prefer to impute the mean of the learned distribution. In this case, we forego autoregression and directly obtain the mean of each distribution p( xx) with single forward pass. Analytically computing the mean of the proposal distribution is straightforward since we working with mixture of Gaussians. We estimate the mean of the energy distribution via importance sampling:
Fs
p(ra, (xx) g( xx)
E |
RC
T.
where (() is sampled from q(In, xx). It is worth noting that imputing the mean ignores depen- dencies between features in the unobserved set, so for some applications, other methods of imputing (such as multiple imputation by drawing samples) may make more sense.
4.4.3 Heterogenous Data
Prior approaches to arbitrary conditioning have to make restrictive assumptions when modeling arbitrary dependencies between continuous and discrete covariates. VAEAC 14, for example, makes an assumption of conditional independence given a latent code. ACFlow 20 is not directly
We describe how to generate samples from ACE in the Appendix.

---

Table 1: Test arbitrary conditional log-likelihoods (in nats) for UCI datasets. Higher is better. Likelihood estimates are computed with 20,00 ,00 importance samples for POWER, GAS, and HEPMASS, 10,000 importance samples for MINIBOONE, and 3,000 importance samples for BSDS. Results for ACFlow and VAEAC are taken from Li et al. [20]. The best model for each dataset and missing rate shown in bold. Results are averaged over 5 observed masks.
POWER
GAS
HEPMASS
MINIBOONE
BSDS
Missing Rate
0.0
0.5
0.0
0.1
0.5
0.1
0.5
0.0
01.
0.631
0.600
9.643
9.526
8,530
-3,859
.4.25
-8.133
0.310
-0,688
-5.701
86.701
86.130
M0.613
ACE Proposal
0.583
0.573
0.542
9.484
9.348
8,183
-4,47
-4.9%
-8.497
-0.241
-1.328
-9.169
85.228
84.234
75.767
ACPlow
0:561
0.557
0.458
8.86
7.568
5.405
-8.197
-7.784
-10.538
-0.972
-5.150
-9,892
81.827
80.783
75,050
ACFloo-BBG BG
0.528
0.510
0.417
7,593
7,212
4.818
-6.83
-9.620
-10.975
-1.098
-3.577
-10.849
81,399
79.745
73.061
VAEAC
-.042
-0.43
2.418
2.823
1952
-00.822
-10.389
-11.415
-3,452
-4,242
-9.051
74,850
74.313
66.628
applicable to discrete data given its use of the change of variable theorem. On the other hand, ACE can naturally model arbitrary dependencies between continuous and discrete covariates without any assumptions. In this setting, the proposal network outputs categorical logits for discrete features, as opposed to Gaussian mixture parameters. These logits can themselves be interpreted energies, and we don't 't need to learn an additional energy function for the discrete features. Similarly, ACE could be applied multimodal data straightforward fashion by conditioning the proposal distributions and energy function on a fused multimodal latent representation (which could be freely learned by arbitrary neural networks).
Experiments
5.1 Real-valued UCI Data
We first evaluate ACE on real-valued tabular data. Specifically, we consider the benchmark UCI repository datasets described by Papamakarios et al. [25 (see Table 6 the Appendix).
Unlike other approaches to density estimation that require particular network architectures
20). ACE has no such restrictions. Thus, we use a simple fully-connected network with residual connections
[12] [3] for both the energy network and proposal network. This architecture is highly expressive, yet simple, and helps avoid adding unnecessary complexity to ACE. The bitmask b, which indicates observed features, is sampled for each training example by first drawing &t ~ 24 d 1}. then choosing k distinct features with uniform probability to be observed. Full experimental details and hyperparameters can be found in the Appendix.
We also consider the scenario in which data features are completely missing, i.e., some features are deemed unavailable for particular instances during training and are never part of the observed or unobserved set 55 This allows us to examine the effectiveness of ACE on incomplete datasets, which common when working with real-world data. When training models with missing data, simply modify the sets of observed and unobserved indices to remove any indices which have been declared missing, This a trivial modification and requires no other change the design or training procedure of ACE. We consider two scenarios where data are missing completely at random at a 10% and 50% rate.
5.1.1 Likelihood Evaluation
Table presents the average arbitrary conditional log-likelihoods on held-out test data from models trained with different levels of missing data. During inference, no data missing and b is drawn from a Bernoulli distribution with 0.5. Likelihoods are calculated autoregressively as described in Section 4.4.1) The order in which the unobserved one-dimensional conditionals are computed is randomly selected for each instance.
We draw two key findings from Table First, see that our proposal distribution outperforms ACFlow in all cases. Even this exceptionally simple approach (just a fully-connected network that produces a mixture of Gaussians) can give rise extremely competitive performance, and we see there are advantages to using decomposed densities and unrestricted network architectures. Second,
Features are missing at the per per-instance level. For example, this does not mean that the 2h feature is never observed for all training instances.
7

---

Table 2: Test marginal log- sigllikeliooods (in nats) for UCI datasets. Higher is better. We evaluate the marginal distributions of the first 3, 5, and 10 dimensions of each dataset (POWER and GAS don't t have 10 features, so the joint likelihood over all features is reported instead). The same number of importance samples are used as in Table I] Results for ACFlow and TAN are taken from Li et al. 220] Bold indicates where ACE outperformed ACFlow. Results are averaged over 5 observed masks.
POWER
GAS
HEPMASS
MINIRGONE
BSDS
Dimensions
8
3
10
3
5
10
ACE
-0.56
1.42
0.58
1.31
4.31
12.20
-4.00
-5.91
-10.72
-2.13
-3.80
-7.94
5.10
9.37
20.31
ACE Proposal
-0.58
1.35
0.49
1.11
3.98
11.84
-4.01
-5.4
-10.82
2.14
-3.79
-7.93
5.06
9.30
20.17
ACFlow
-0.57
1.34
0,42
0.78
3.01
10.13
-4003
-6.19
-1.558
-2.76
-5.31
-10.36
5.06
9.26
19.60
TAN
40.54
1.40
0.57
1.22
4.47
12.09
4.000
-5.92
10.87
-2.13
-3.73
-8.
5.11
9.43
20.44
Hеppass
0.6-
9.6-
044-
Iresupa
02
02-
02
ACF
AdFlo
0.-
0.5
0.1
0:0
6:7
2.1
05
3.0
0.3 0.
6-5
0.0
0.2
15
0.0 0.1
35
Miser Rate
Mintig Rate
Ninsing Rate
Maxing
Missing Rate
Figure 2: Normalized root-mean-square square error (NRMSE) of imputations generated by ACE. Lower is better. NRMSE is computed the aat-mean--squrre error normalized by the standard deviation of each feature and then averaged across all features. Estimates of energy distribution means are computed with 20,000 importance samples for POWER and GAS, 10,000 importance samples for HEPMASS and MINIBOONE, and 3,000 importance samples for BSDS, Results for ACFlow and VAEAC are taken from Li et al. (20).
we find that in every case, the likelihood estimates produced by the energy function are higher than those from the proposal, illustrating the benefits of an anergy-based approach which imposes no biases the shape of the learned distributions.
We also examine the arbitrary marginal distributions learned by ACE, i.e., the unconditional distribu- tion over a subset of features. We again test our model against ACFlow, and we additionally compare to Transformation Autoregressive Networks (TAN) 23], which are designed only for joint likelihood estimation. A separate TAN model has to be trained for each marginal distribution. While a single ACFlow model can estimate all marginal distributions, Li et al. (20) retrained models specifically for arbitrary marginal estimation. Contrarily, marginals as were used for arbitrary conditionals. Results are provided in Table 2] We find that ACE outperforms ACFlow in all cases and even surpasses TAN most of the time, even though ACFlow and TAN both received special training for marginal likelihood estimation and ACE did not.
used the same ACE models when evaluating arbitrary
5.1.2 Imputation
We also evaluate ACE for data imputation, where some elements are missing from a dataset completely at random and we seek to infer their values. ACE is naturally applied to this task - we consider p(x,. x,), where x,, contains the missing features.
Figure 2 shows the normalized root-mean mean-square error (NRMSE) on held-out test data. Again, we consider models trained with three different levels of missing data. During inference. b is drawn from a Bernoulli distribution with p = 0.5 for the 0% and 50% missing rates and p = 0.9 for the 10% missing rate The means of the unobserved distributions used the imputed values (see Section 4.4.2).
As seen in Figure 2 ACE achieves lower NRMSE score than ACFlow in all cases (exact numbers are available in the Appendix). These results further validate ACE's ability to accurately model arbitrary conditionals, leading us to again advocate for simple methods with few biases. It is also
This means that for the 10% missing rate, we are imputing fewer values (only 10% as opposed to 50%) during inference than with the other two missing rates. This explains the lower errors for the 10% missing rate that we see in Figure 2 Even though there is missing data during training, are estimating the values of fewer values based on a larger amount of observed data, which intuitively should result in more accurate imputations.
8

---

Table 3: Conditional log-likelihoods (LL),
Table 4: Log-likelihood and imputation results for the UCI Adult dataset. NRMSE measures imputa- tion performance for continuous features and accu- racy is used for discrete features.
joint bits-per-dimension (BPD), and inpainting peak to-noise-ratii (PSNR) for ACE and ACFlow on MNIST.
Method
LL
BPD
PSNR
Method
LL
NRMSE
Aceuracy
ACE
1043.88
1.42
16.81
ACE
2.38
0.90
0.69
ACE Proposal
828.60
2.11
16.70
ACE Proposal
2.24
0.89
0.69
ACFlow
875.87
3.09
13.75
VAEAC
-7.25
0.91
0.67
22
2 2 2 2
2
2 7
2
2 7 4 0 9
2
6
5
5 9 9
5 4 7
6
7
4
6
7
7 4
7
7 77 4
7
7
5 O
3
G
4
4 4 4
4
4 7
4 9 0
4
8
7
7
0
0
3 8
8
6
8
0
0
6
0
6
0
0
9 3
2
0
7
(a) Inpaintings from ACE and ACFlow. Left: Groundtruth and observed pixels. Middle: ACE samples. Right: ACFlow samples.
(b) Samples from the joint distribution (i.c., all pixels are unobserved). Left: ACE samples. Right: ACFlow samples.
Figure 3: MNIST samples generated by ACE and ACFlow.
worth noting that ACE and ACE Proposal do comparably in this imputation task, which estimates the first-order moment of conditional distributions. However, as evidenced in Table the energy-based likelihood better captures higher-order moments.
5.2 High-dimensional Data
We examine ACE's ability to scale to general high-dimensional data by considering the flattened MNIST dataset [18] (i.e., each example 784-dimensional vector). order make a fair comparison, trained non-convolutional ACFlow model (using the authors' code) the flattened data as well. The only change to ACE's training procedure in the distribution bitmasks that are sampled during training, which we detail in the Appendix.
Table compares ACE and ACFlow in terms of arbitrary conditional log- likeliiikiooos, joint bits-per- dimension (BPD), and peak lignal-tonoise-
for inpaintings, and we see that ACE outperforms ACFlow for all three metrics. We also note that ACE's BPD of 1.42 comparable to prior methods for joint density estimation such as TAN, MADE, and MAF, which have reported BPD of 1.19, 1.41, and 1.52 respectively (lower better), despite the fact that these other methods can only model the joint distribution [23]. Qualitatively, we see that ACE produces more realistic inpaintings (see Figure 3ab and samples from joint (see Figure 3b) than ACFlow. These findings indicate that ACE's performance scales well to high-dimensional data.
5.3 Mixed Continuous-Discreete Data
In order to demonstrate ACE's ability to model data with both continuous and discrete features, we conduct experiments the UCI Adult datase[" which offers relatively even balance between the number of continuous and discrete features (see below). We preprocess the data by standardizing continuous features and dropping instances with missing values. We also only keep instances for which the native-country feature is United-States! The processed data has 6 continuous
https://archive ics
Such instances account for about 91% of the data, and the other 9% take on of 39 possible values. Thus able discard highly class- dass-imbalanced feature while retaining the vast majority of the data.

---

Power
Hepmass
boons
BSDS
Samples
importacce
Impprtance Samples
importance
stimpo Samples
Figure 4: Percent error of normalizing constant estimates obtained with importance sampling when compared to "true" values obtained with numerical integration. Estimates grow more accurate the number importance samples increases. Whiskers indicate 1.5 times the IQR.
features and 8 discrete features and is split into train, validation, and test partitions of size 22003, 5501, and 13788 respectively.
We trained an ACE model and VAEAC model (using the authors' publicly available code) on this dataset and evaluated them in terms of likelihoods and imputation. For discrete features, the mode of the learned distribution is imputed (as opposed to the mean). The results are presented in Table 4 where we see that ACE outperforms VAEAC on all metrics. We also reiterate that due to its use of the change of variable theorem, ACFlow cannot be trained on this dataset.
5.4 Importance Sampling Accuracy
In order to better understand the impact of the number of importance samples used when estimating normalizers, we compare the importance sampling estimates from model to "true" estimates obtained via numerical integration (similar to (22). For each UCT dataset, we randomly select 10,000 one-dimensional conditionals (from a validation set that was not seen during training) and integrate the unnormalized energy likelihoods using trapezoidal rule [26). For each conditional, we then also estimate the normalizer with increasing number of importance samples. Figure 4]shows the percentage errors of the importance sampling estimates. In most cases, small number of samples (e.g., 20) can already produce relatively accurate estimates, and we see that the estimates grow more accurate as the number of samples increases.
Conclusion
In this work, we present a simple approach to modeling all arbitrary conditional distributions p(x, x,) over set of covariates. Our method, Arbitrary Conditioning with Energy (ACE), is the first to wholly reduce arbitrary conditioning to one-dimensional conditionals with arbitrary observations and estimate these with energy functions. By using an energy function to specify densities, ACE can more easily model highly complex distributions, and it can freely use high-capacity networks to model the exponentially many distributions at hand.
Empirically, we find that ACE achieves state-of-the-ar the-art performance for arbitrary conditional/marginal density estimation and for data imputation. For given dataset, all of these results produced with the same trained model. This high performance does not come at the cost of complexity - ACE is much simpler than other common approaches, which often require restrictive complexities such as normalizing flow models or networks with specially masked connections (20 g).
Furthermore, ACE's proposal distribution, a basic mixture of Gaussians, outperforms prior methods, demonstrating that the principle of learning one-dimensional distributions still powerful when decoupled from energy-based learning. These results emphasize that seemingly complex problems do not necessitate highly complex solutions, and we believe future work on arbitrary density estimation will benefit from similar ideas.
Acknowledgments and Disclosure of Funding
This work was supported in part by grants NIH IROIAA02687901AL and NSF IIS2
10

---

References
[1] M. Belghazi, M. Oquab, and D. Lopez-Paz. Learning about an exponential amount of conditional distributions. In Advances in Neural Information Processing Systems, pages 13703-13714 2019.
[2] C. M. Bishop. Pattern recognition and machine learning. springer, 2006.
[3] C. J. Butz, J. S. Oliveira, A. E. dos Santos, and A. L. Teixeira. Deep convolutional sum-product networks. Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 2019.
[4] L. Dinh, J. Sohl-Dickstein, and S. Bengio. Density estimation using real nvp. arXiv preprini 160.00605.08803 2016.
[5] L. Douglas, I. Zarov, K. Gourgoulias, C. Lucas, C. Hart, A. Baker, M. Sahani, Y. Perov, and Johri. A universal marginalizer for amortized inference in generative models. arXiv preprint arXiv:1711.00693 2017.
[6] R. Fakoor, P. Chaudhari, Mueller, and Smola. Trade: Transformers for density estimation. arXiv preprini arXiv:2004.02444 2020.
[7] S. Fielding, P. M. Fayers, A. McDonald, G. McPherson, and M. K. Campbell. Simple imputation methods were inadequate for missing not at random (mnar) quality of life data. Health and Quality of Life Outcomes, 6(1):1-9, 2008.
[8] M. Germain, K. Gregor, I. Murray, and H. Larochelle. Made: Masked autoencoder for distri- bution estimation. In International Conference on Machine Learning, pages 881-889. PMLR, 2015.
[9] I.. Gondara and Wang. Mida: Multiple imputation using demoising autoencoders. In Pacific-Asia Conference on Knowledge Discovery and Data Mining, pages 260-272. Springer, 2018.
[10] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. Advances in neural information processing systems, 27: 2672-2680, 2014.
[11] W. Grathwohl, R. T. Chen, J. Bettencourt, I. Sutskever, and D. Duvenaud. Ffjord: Free-form con- tinuous dynamics for scalable reversible generative models. arXiv preprint arXiv: 1810.01367. 2018.
[12] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770- 778, 2016.
[13] K. He, X. Zhang, S. Ren, and J. Sun. Identity mappings in deep residual networks. In European conference computer vixion, pages 630-645. Springer, 2016.
[14] O. Ivanov, M. Figurnov, and D. Vetrov. Variational autoencoder with arbitrary conditioning. arXiv preprint 10X.0:100..0232.. 2018
[15] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint 1129 2014.
[16] M. Lázaro-Gredilla, W. Lehrach, N. Gothoskar, G. Zhou, A. Dedieu, and D. George. Query training: Learning worse model to infer better marginals in undirected graphical models with hidden variables. arXiv preprint arXiv:2006.06800 06803, 2020.
[17] Y. LeCun, S. Chopra, R. Hadsell, M. Ranzato, and F. Huang. A tutorial on energy-based learning. Predicting structured data, 1(0), 2006.
[18] Y. LeCun, C. Cortes, and C. Burges. Mnist handwritten digit database. ATT Labs [Online]. Available: http:///ww...cc....o//xx//////// 2, 2010.

---

[19] S. C.-X. Li, B. Jiang, and B. Marlin. Misgan: Learning from incomplete data with generative adversarial networks. arXiv preprint arXiv:1902.09599 2019.
[20] Y. Li, S. Akbar, and J. Oliva. ACFlow:
Flow models for arbitrary conditional likelihoods. In H. D. III and A. Singh, editors, Proceedings of the 37th International Conference Machine Learning. volume 119 of Proceedings of Machine Learning Research, pages 5831-5841. PMLR, 13-18 Jul 2020. URL http::/proceeddnggg mlr wwwppppessss... v119 1120a. html
[21] P.-A. Mattei and J. Frellsen. MIWAE:
Deep generative modelling and imputation of incomplete data sets. In K. Chaudhuri and R. Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 4413-4423, PMLR, 09-15 Jun 2019. URL
wtw.ll.pres//voe /
mattei 19t.e1999..lllll1
[22] C. Nash and C. Durkan. Autoregressive energy machines. arXiv preprint arXiv:1904.0500 2019.
[23] J. B. Oliva, A. Dubey, M. Zaheer, B. Poczos, R. Salakhutdinov, E. P. Xing, and J. Schneider. Transformation autoregressive networks. arXiv preprint 18Xi 09899, 2018.
[24] A. v. d. Oord, N. Kalchbrenner, O. Vinyals, L. Espeholt, A. Graves, and K. Kavukcuoglu. Conditional image generation with pixelenn decoders. arXiv preprint arXiv: /6civ:666..mm. 05328, 2016.
[25] G. Papamakarios, T. Pavlakou, and I. Murray. Masked autoregressive flow for density estimation. In Advances in Neural Information Processing Systems, 2338-2347, 2017.
[26] M. Pitkin. lintigrate. httpss//gghhbbooco//aatttttttttp
2017.
[27] H. Poon and P. Domingos. Sum- networks: A new deep architecture. In 201/ IEEE International Conference on Computer Vision Workshops (ICCV Workshops), pages 689-690. IEEE, 2011.
[28] M. Przewicžlikowski, M. Smieja, and L. Struski. Estimating conditional density of missing values using deep gaussian mixture model. In International Conference on Neural Information Processing, pages 220-231 231. Springer, 2020.
[29] D. B. Rubin. Using the sir algorithm to simulate posterior distributions. Bayesian statistics, 3: 395-402, 1988.
[30] S. Saremi, A. Mehrjou, B. Schölkopf. and A. Hyvärinen. Deep energy estimator networks. arXiv preprint 1805.0 083000 2018
[31] D. J. Stekhoven and Bühlmann. ninnpretstt-oon-paaaettti missing value imputation for mixed-type data. Bioinformatics, 28(1):112-114) 2012.
[32] O. Troyanskaya, M. Cantor, G. Sherlock, P. Brown, T. Hastie, R. Tibshirani, D. Botstein, and R. B. Altman. Missing value estimation methods for dna microarrays. Bioinformatics, 17(6): 520-525, 2001.
[33] B. Uria, I. Murray, and H. Larochelle. A deep and tractable density estimator. In International Conference on Machine Learning, pages 467-475, PMLR, 2014.
[34] J. Yoon, J. Jordon, and M. van der Schaar. GAIN: Missing data imputation using generative adversarial nets. In Dy and A. Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 5689-5698, Stockholmsmässann Stockholm Sweden, 10-15 Jul 2018. PMLR. URL http://prroceedingg.. mlr. htt.://ss....rr.eeeee/o///oo
12


Reviews:
[{"review_id": "zjy3VUm1ItI", "reviewer": "Reviewer_ENb9", "summary": "This paper introduces Arbitrary Conditioning with Energy (ACE) to estimate arbitrary conditional densities, using a Boltzmann distribution with an energy function parametrized by a neural network to estimate 1-dimensional conditional densities. An intractable normalizing constant is estimated with importance sampling and a proposal network. The method is then demonstrated on a few examples for density estimation under missingness and for imputation. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\nThe paper is clearly written, and I find the setup interesting and useful especially under missing data. Only requiring the estimation of 1-dimensional densities is also easier to understand and computationally appealing. The work is an interesting combination of the setup of ACFlow [Li et al., 2020] and the method of Autoregressive Energy Machines (AEM) [Nash \\& Durkan, 2019]. The experimental results also improve on ACFlow for density estimation, whilst being simpler.\n\n#####################################################################\n\nWeaknesses/Questions:\n\nI summarize the main weaknesses/questions below:\n\n1.) I am not completely familiar with the energy function literature, but are there any guarantees that the integral $Z = \\int \\exp\\left(-\\mathcal{E} (\\mathbf{x}) \\right) d\\mathbf{x}$ is finite? I do not see how a randomly-initialized network necessarily satisfies this property. \n\n2.) Although the idea is simple, I find the actual implementation of ACE to be potentially over-complicated:\n\na.) As we are only in 1 dimension, is there any issue with using numerical quadrature instead of importance sampling? The proposal adds a lot of complexity to training, and I'm not sure if 20 importance samples (as used in the paper) is more accurate than just estimating $Z$ on a small grid.\n\nb.) I could be missing something, but why does the energy network need the inputs $(\\boldsymbol{\\phi}(\\mathbf{x_o}; \\mathbf{b}),\\mathbf{b})$ in addition to the latent vector $\\boldsymbol{\\gamma}(u_i; \\mathbf{x_o})$? Would the latent vector not suffice, as is done for AEMs?\n\nc.) Why does the proposal network output multiple latent vectors, that is one for each $u_i$? If it contains the information in $\\mathbf{x_{o}}$, then it seems better motivated to just use the same context vector for each $u_i$. \n\n\n3.) What is the motivation for training non-autoregressively, that is with \n$ \\sum_{i=1}^{|u|} \\log p(x_{u_i}\\mid \\mathbf{x_o}) $ instead of $\\sum_{i=1}^{|u|}\\log p(x_{u_i} \\mid \\mathbf{x_{o \\cup u_{<i}}})$?  It seems that $\\log p(\\mathbf{x}_{\\mathbf{u}} \\mid \\mathbf{x}_\\mathbf{o})$ is better motivated, since that is being evaluated in the experiments. Also, is $\\mathbf{o}$ selected at random for each gradient step? It wasn't clear to me from the paper. \n\n#####################################################################\n\nOverall:\n\nI think the idea is good and has potential. However, some of the design choices seem over-complicated given the simple motivation, and either need to be simplified or justified better.  \n\n#####################################################################\n\nReferences:\n\nLi, Y., Akbar, S., \\& Oliva, J. (2020, November). ACFlow: Flow models for arbitrary conditional likelihoods. In International Conference on Machine Learning (pp. 5831-5841). PMLR.\n\nNash, C., \\& Durkan, C. (2019, May). Autoregressive energy machines. In International Conference on Machine Learning (pp. 1735-1744). PMLR.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have described some of the limitations of their method, such as the computational cost of averaging over multiple orderings of the dimensions. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5 hours"}, "extra_text": {}}, {"review_id": "uYFyi5hBuUl", "reviewer": "Reviewer_TGax", "summary": "The authors propose a method for performing conditional density estimation through estimating an energy function and using importance weighting to normalize the density estimate. They focus on estimating single-dimensional distribution and make use of the product rule to estimate multivariate distributions. The authors show the method is empirically promising on data imputation tasks.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Summary of review\n\nThe authors approach seems reasonably well-motivated, and appears to empirically perform well. I am somewhat concerned by the heavy reliance on importance sampling to estimate intermediate quantities, and then taking products of/dividing by the results. I think it would be useful if the authors provided some numerical verification (perhaps in a simple case) that the estimates used are in fact reliable. I think the authors perhaps slightly over-emphasize the approach being order-agnostic, as this is only true during training, and as applied the model may depend heavily on ordering. I didn't see an experiment showing the degree to which estimated densities depend on order in the appendix, and this might be worth illustrating when the authors discuss strategies to mitigate this dependence. However, the authors are upfront in acknowledging that order dpendence at test time is a limitation at several points in the paper, and acknowledge there is more work to be done in this regard. Overall, I thought the paper proposed an interesting and reasonably simple method, and was enjoyable to read. \n\n## Questions for authors\n- In the related work section, the authors claim they construct an 'order-agnostic weight-sharing technique'. However, the final conditional density estimates depend on an arbitrary ordering, as later acknowledged. What is the advantage of the training process not depending on choosing an ordering if applying the model still has this constraint?\n- In line 114 the authors say the *energy* is non-negative. Later the energy is taken to be $\\mathcal{E}$, which is the output of a neural netrowk if I understand correctly. Is it actually intended for $\\mathcal{E}$ to be non-negative, is this a typo, or have I misunderstood?\n- The authors justify the importance sampling estimator by the claim that importance sampling is effective in 1D. However, it seems the authors need to take products of the estimated normalizing constants in order to estimate the overall density. Does this not lead to an accumulation of errors as the dimension of the distribution you are trying to estimate increases? \n- The authors claim that normalizing flows place many restrictions on the types of distrbutions that can be modelled. Is this true even when one-dimensional distributions are estimated with the flow? Can you give concrete examples of what cannot be modelled/is difficult to model with flows?\n- What is the effect of ignoring derivatives with respect to the samples from the proposal distribution and density in 8? Is the estimate of the graident still consistent (as the number of proposal samples goes to infinity)? Is there an effect on the variance of the estimator?\n- In Equation 9 how many samples must be used in practice to obtain an accurare estimate of the mean?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "## Limitations\nThe authors point to several limitations of the model considered. For example:\n- missing information is modelled as uninformative, which is not a good assumption in some cases\n- Estimation of the density of a joint distribution may depend on the order in which the product rule is applied (i.e. the network used to model conditional densities may not be consistent).\n\n## Societal implications\nThe authors do not point to societal implications of their work. As the proposed method might be applicable to a wide range of tasks, this seems reasonable. Any societal implications would depend signficantly on which tasks the proposed method is applied to, and it is difficult to assess what these might be.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6 hours"}, "extra_text": {}}, {"review_id": "NSo0JnycLtb", "reviewer": "Reviewer_4vM4", "summary": "The authors propose a strategy to estimate any arbitrary conditional distribution by using an the chain rule:  they estimate all one-dimensional distributions.  While this would require learning exponentially many distributions, a weight sharing scheme is used to limit the number of parameters that need to be learned from data.  The resulting approach represents each univariate distribution in the form exp(neural network) and the normalizing constant is estimated via importance sampling (a proposal distribution can also be estimated if desired).  They show that this simple approach is flexible and outperforms more sophisticated approaches on real-world data sets.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, I quite like the approach. It's a simple and clever combination of modern techniques that should be easy to replicate.  My only real criticisms are related to the experimental evaluation (see below) and these are minor.\n\nOriginality:  The work appears to be a novel approach for density estimation tasks.\n\nQuality:  1) I don't personally love using likelihoods to evaluate continuous models (despite the prevalence of these types of evaluations).  While the average conditional likelihood is definitely better, it still isn't great.  I've seen plenty of examples with sum-product networks and similarly expressive models where high likelihoods does not correspond to good model performance on real tasks.  Thankfully, the imputation/predictive performance is also better for ACE.  Honestly, I'd suggest moving the likelihood stuff to the appendix and replacing it with larger figures (the bar charts in particular are difficult to read) and more experiments.\n\n2)  I actually find Figure 2 somewhat perplexing.  Why does ACE seem to do so much better in the missingness .1 regime than the other competitors? The performance seems more comparable in the other two settings.  Thoughts?\n\n3) While the current experiments are sufficient, in my opinion, for publication, I think it would make an even stronger case if the authors could include a few more interesting data sets (or even one practical domain) for which this approach succeeds where other approaches fail significantly.\n\n4) No mention is really made of wall-clock comparison between the methods (or I missed it).  Could the authors comment a bit?\n\nClarity:  The paper is well-written and easy to follow with only minor typos.\n\nSignificance: The simplicity of the approach lends itself to a variety of different applications.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "n/a + see above", "review_text": "", "extra_scores": {"ethical_concerns": "n/a", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "-bLP8Yr1UqZ", "reviewer": "Reviewer_AEG1", "summary": "In this paper, the authors propose to compute p(x_u|x_o) for any combination of observed variables (x_o) and unobserved varaibles (x_u) with a single NN. The authors actually propose to compute p(x_u|x_o) = prod p(x_{u_i}|x_o,x_{u_1} … x_{u_{i-1}}). Basically, they train a 1D conditional estimator that that can depend on any of the inputs. \n\nThe proposed mechanism is based on two networks. A Proposal network that takes as input a mask for the observed inputs and a vector with the inputs for the observed variables at the right positions. The output of the proposal network is the components for a mixture of Gaussians ($\\omega$) and a latent vector for each missing component of x. The second network takes the latent vector and estimates the probability. ", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I must say that the algorithm is very clever, and it seems to perform well compared to other methods. I would be in favor of having the paper presenting at the conference. But I have a few questions that I do not understand how the algorithm works at inference time, and I would like the authors to be able to explain how their solution performs. Let me start with a very simple example, this would probably be a good simple example for the readers too. Let us assume that we have 4 variables x_1, x_2, and x_3 coming from a multivariate Gaussian. Let’s assume that we train the proposal network with two missing variables and one observed. Side question: Do we always have to have the same number of missing variables in the proposal network, I would say so, but please let me know if I am wrong about this? \n\nAt test time, we are given x_1 = X1 and we compute:\nP(x_2, x_3|X_1=X1) = p(x_2|x_1=X1) p(x_3|x_1=X1,x_2)\n\nThe first thing that, we need to do is to get N samples from q(x_2|x_1=X1) and Z_{{x_2}|x_1=X1}, using equation 5.\n\nNow, how do I compute, Z_{{x_3}|x_1=X1, x_2}. This partition function is no longer a number but a function of x_2. Do I compute this for a specific value of x_2, such as a sample from p(x_2|x_1=X1) or the maximum or the true value that x_2 takes? I guess it is the last one, using this procedure would only work if we are computing the likelihood for the full sample. So Z_{{x_3}|x_1=X1, x_2} is not a function of x_2 but a constant with x_2=X2.\n\nI think this is more evident when looking at the imputation in which there is no access to X2 and the authors use the following approximation:\nP(x_2, x_3|X_1=X1) = p(x_2|x_1=X1) p(x_3|x_1=X1)\nAnd use the mean p(x_2|x_1=X1) and p(x_3|x_1=X1) to impute the variables. For example, using this to generate the top half of an MNIST image would be horrible and would only provide a blurred image. In other image processing would even be worst. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Not needed", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an approach for performing conditional density estimation of the form $p(x_u|x_o)$ for any arbitrary choice of the observed ($x_o$) and unobserved ($x_u$) variables by using an energy function parametrized by a neural network to estimate the 1D distributions.

Three reviewers recommend acceptance, and one reviewer believes the paper is below the acceptance threshold. After reading the reviews and the rebuttal, I recommend this paper for acceptance. I also encourage the authors to improve the paper by considering the reviewers' comments. In particular:
+ Add an analysis of the number of samples needed for the importance sampling approximations.
+ As noted by reviewer AEG1, the imputation method may not be appropriate for certain applications because the imputation is performed using the marginal means, e.g., the means of $p(X_2|X_1=x_1)$ and $p(X_3|X_1=x_1)$, therefore, ignoring the dependences between $X_2$ and $X_3$. This should be mentioned/discussed in Section 4.4.2 or 5.1.2. 
+ Add a sentence motivating importance sampling over the grid.
+ Add some discussion on the skip-connection and multiple latent vectors.
+ Include a derivation or discussion of the stochastic approximation of the autoregressive score in the Appendix.
Decision: Accept (Poster)