Arbitrary Conditional Distributions with Energy
Authors: Ryan Strauss
Junier Oliva
Venue: NeurIPS 2021 Poster (2021)
Keywords: density estimation
arbitrary conditioning
energy-based models
imputation
unsupervised learning

Abstract: Modeling distributions of covariates, or density estimation, is a core challenge in unsupervised learning. However, the majority of work only considers the joint distribution, which has limited relevance to practical situations. A more general and useful problem is arbitrary conditional density estimation, which aims to model any possible conditional distribution over a set of covariates, reflecting the more realistic setting of inference based on prior knowledge. We propose a novel method, Arbitrary Conditioning with Energy (ACE), that can simultaneously estimate the distribution $p(\mathbf{x}_u \mid \mathbf{x}_o)$ for all possible subsets of unobserved features $\mathbf{x}_u$ and observed features $\mathbf{x}_o$. ACE is designed to avoid unnecessary bias and complexity --- we specify densities with a highly expressive energy function and reduce the problem to only learning one-dimensional conditionals (from which more complex distributions can be recovered during inference). This results in an approach that is both simpler and higher-performing than prior methods. We show that ACE achieves state-of-the-art for arbitrary conditional likelihood estimation and data imputation on standard benchmarks.

Arbitrary Conditional Distributions with Energy
Ryan R. Strauss
Junier B. Oliva
Department of Computer Science
Department Computer Science UNC at Chapel Hill Chapel Hill, NC 27514 joliva@cs unc edu
UNC at Chapel Hill
Chapel Hill, NC 27514 rrsêcs. unc. edu
Abstract
Modeling distributions of covariates, or density estimation, is a core challenge in unsupervised learning. However, the majority of work only considers the joint distribution, which has limited utility in practical situations. A more general and useful problem is arbitrary conditional density estimation, which aims model any possible conditional distribution over a set of covariates, reflecting the more realistic setting of inference based on prior knowledge. We propose novel method, Arbitrary Conditioning with Energy (ACE), that can simultaneously estimate the distribution p( xx x,) for all possible subsets of unobserved features x,, and observed features x., ACE is designed to avoid unnecessary bias and complexity
we specify densities with highly expressive energy function and reduce the problem to only learning mne-dimensional conditionals (from which more complex distributions can be recovered during inference). This results in an approach that is both simpler and achieves sute-of-the-arr for arbitrary conditional likelihood estimation and data imputation on standard benchmarks.
than prior methods. We show that ACE
1 Introduction
Density estimation, core challenge in machine learning, attempts learn the probability density of some random variables given samples from their true distribution. The vast majority of work on density estimation focuses on the joint distribution p(x) I0 25 I1 23 22 6), i.e., the distribution of all variables taken together. While the joint distribution can be useful (e.g.. to learn the distribution of pixel configurations that represent human faces), limited in the types of predictions it can make. We are often more interested in conditional probabilities, which communicate the likelihood of an event given some prior information. For example, given a patient's 's medical history and symptoms, doctor determines the likelihoods of different illnesses and other patient attributes. Conditional distributions are often more practical since real-world decisions are nearly always informed by prior information.
However, we often do not know ahead of time which conditional distribution (or distributions) will be of interest. That is, we may not know which features will be known (observed) or which will be inferred (unobserved). For example, not every patient that doctor sees will have had the same tests performed (i.e., have the same observed features). A naïve approach requires building an exponential number of models to cover all possible cases (one for every conditional distribution), which quickly becomes intractable. Thus, intelligent system needs to understand the intricate conditional dependencies between all arbitrary subsets of covariates, and must do so with a single model to be practical.
In this work, we consider the problem of learning the conditional distribution p(x,, xx) for any arbitrary subsets of unobserved variables x, E R|a and observed variables x, RR1, where u, (1.....d d} and on 0. We propose method, Arbitrary Conditioning with Energy (ACE),
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

that can assess any conditional distribution over any subset of random variables, using a single model. ACE is developed with an eye for simplicity we reduce the arbitrary conditioning problem to the estimation of one-dimensional conditional densities (with arbitrary observations), and we represent densities with an energy function, which fully specifies unnormalized distributions and has the freedom to be instantiated as any arbitrary neural network.
We evaluate ACE on benchmark datasets and show that it outperforms current methods for arbitrary conditionall/marginll density estimation. ACE remains effective when trained on data with missing values, making it applicable to real-world datasets that are often incomplete. and we find that ACE scales well to high- dige-diiensional data. Also, unlike some prior methods (e.g., (20),, ACE can naturally model data with both continuous and discrete values.
Our contributions are as follows: 1) We develop the first energy-based approach to arbitrary con- ditional density estimation, which eliminates restrictive biases (e-g- normalizing flows, Gaussian mixtures) imposed by common alternatives. 2) We empirically demonstrate that ACE is state-of-the- art for arbitrary conditional density estimation and data imputation. 3) We find that complicated prior approaches can be easily outperformed with a simple scheme that uses mixtures of Gaussians and fully-connected networks.
2
Previous Work
Several methods have been previously proposed for arbitrary conditioning. Sum-Product Networks are specially designed to only contain sum and product operations and can produce arbitrary conditional or marginal likelihoods (27 3] The Universal Marginalizer trains neural network with cross- entropy loss to approximate the marginal posterior distributions of all unobserved features conditioned on the observed ones (5]. VAEAC is approach that extends conditional variational autoencoder by only considering the latent codes of unobserved dimensions [1), and NeuralConditioner uses adversarial training to learn each conditional distribution D. DMFA uses factor analysis to have a neural network output the parameters of a conditional Gaussian density for the missing features given the observed ones (28). The current Mfate-of-the-arr is ACFlow, which extends normalizing flow
models to handle any subset of observed features (20]
Unlike VAEAC, ACE does not suffer from mode collapse or blurry samples. ACE is also able to provide likelihood estimates, unlike NeuralCondittoner which only produces samples. DMFA is limited to learning multivariate Gaussians, which impose bias and are harder to model than one- dimensional conditionals. While ACFlow can analytically produce normalized likelihoods and samples, it is restricted by requirement that its network consist of bijective transformations with tractable Jacobian determinants. Similarly, Sum-Product Networks have limited expressivity due to their constraints. ACE, on the other hand, exemplifies the appeal of energy- methods it has no constraints on the parameterization of the energy function.
Energy- based methods have wide range of applications within machine learning 17 and recent work has studied their utility for density estimation. Deep energy estimator networks 30 and Autoregressive Energy Machines [22] are both energy-based models that perform density estimation. However, both of these methods are only able to estimate the joint distribution.
Much of the previous work on density estimation relies on an autoregressive decomposition of the joint density according to the chain rule of probability. Often, the model only considers a single Uria et al. (33] proposed a method for assessing joint likelihoods based on any arbitrary ordering, where they use masked network inputs effectively share weights between a combinatorial number of models. Germain et al. [8] also consider shared network for joint density estimation with a constrained architecture that enforces the autoregressive constraint in joint likelihoods. In this work, we construct an arder-agnostic weight-sharing technique not for joint likelihoods, but for arbitrary conditioning. Moreover, we make use of our weiigh-sharrng scheme to estimate likelihoods with an energy based approach, which avoids the limitations of the X Query Training [16] is method for answering probabilistic queries. It also takes the approach of computing one-dimensional conditional likelihoods but does not directly pursue an autoregressive extension of that ability.
(arbitrary) ordering of the features 22 24
parametric families used previously (e.g., mixtures of Gaussians 33,, or Bernoullis
2

---

The problem of imputing missing data has been well studied, and there are several approaches based on classic machine learning techniques such as k-nearest neighbors [32]. random forests [T]] and autoencoders [9 More recent work has turned to deep generative models. GAIN is generative adversarial network (GAN) that produces imputations with the generator and uses the discriminator to discern the imputed features 34. Another GAN-based approach is MisGAN, which learns two generators to model the data and masks separately [9]. MIWAE adapts variational autoencoders by modifying the lower bound for missing data and produces imputations with importance sampling 21. ACFlow can also perform imputation and is state-of-the-art for imputing data that are missing completely at random (MCAR) [2]
While it is not always the case that data are missing random, the opposite case (i.e., missingness that depends on unobserved features' values) can be much more challenging to deal with [7]. Like many data imputation methods, we focus on the scenario where data are MCAR, that is, where the likelihood of being missing is independent of the covariates' values.
3 Background
3.1 Arbitrary Conditional Density Estimation
probability density function (PDF), p(x), outputs a nonnegative scalar for a given vector input x E Id and satisfies / p(x) dx 1. Given a dataset = [x(*),
]== of i.i.d. samples drawn from an unknown distribution p" (x), the object of density estimation is to find a model that best approximates function p² Modern approaches generally rely on neural networks to directly parameterize the approximated PDF
Arbitrary conditional density estimation is a more general task where we estimate the conditional density (x xx) for all possible subsets of observed features C. {L,.... d} (i.c., features whose values are known) and corresponding subset of unobserved features
{I, {I...., d} such that o and do not intersect. Here X, E IR and X R| The estimation of joint or marginal likelihoods is recovered when is the empty set. Note that marginalization to obtain p(x.) = J p(xu: X,) dx
(and hence p(x, X₂) p(x....o) is intractable in performant generative models like normalizing
p(x.)
flow based models 44 20 thus, we propose a weight-sharing scheme below.
3.2 Energy-Based Models
Energy-based models capture dependencies between variables by assigning nonnegative scalar energy to given arrangement of those variables, where energies closer to zero indicate more desirable configurations [17). Learning consists of finding an energy function that outputs low energies for correct values. We can frame density estimation as an energy-based problem by writing likelihoods as Boltzmann distribution
-E()
p(x)=
7 z=
E(x) dx -B())
Z
where E is the energy function, e E(() is the unnormalized likelihood, and is the normalizer.
Energy-based models are appealing due to their relative simplicity and high flexibility in the choice of representation for the energy function. This is in contrast to other common approaches to density estimation such normalizing flows 4 20), which require invertible transformations with Jacobian determinants that can be computed efficiently. Energy functions are also naturally capable of representing non-smooth distributions with low-density regions or discontinuities.
4 Arbitrary Conditioning with Energy
We are interested in approximating the probability density p(x, xx) for any arbitrary sets of unobserved features x,, and observed features x,, We approach this by decomposing likelihoods into products of one-dimensional conditionals, which makes the learned distributions much simpler. This concept a basic application of the chain rule of probability, but has yet to be thoroughly exploited for arbitrary conditioning. During training, ACE estimates distributions of the form p(zu, xx)).
3

---

k
Popuud
Enery Setwort
(a) Proposal Network
(b) Energy Network
Figure 1: Overview of the networks used in ACE. The plus symbol refers concatenation.
where zut is a scalar. During inference, more complex distributions can then be recovered with an autoregressive decomposition: p(x, x,) = II p( Xou where w' an arbitrary permutation of u and u'z, = (u'...., u/-17- This approach is appealing because the estimated one-dimensional domain, allowing one a myriad of flexible estimators.
distributions are
We adopt an energy-based approach (similar to AEMs (22) which affords large degree flexibility free to represent the energy function with an arbitrary, and highly expressive, neural network that directly outputs unnormalized likelihoods. This contrasts with the current state-of-ie tore---ftthharrrr for arbitrary conditional density estimation, which is limited by normalizing flow transformations [20]. Energy functions are highly expressive and naturally model complex (i.e., multimodal, non-smooth) densities, they avoid a parametric-famil on the shape of the distribution. Our main contribution a method, Arbitrary Conditioning with Energy (ACE), for computing arbitrary conditional likelihoods with energies, one dimension at time !
in modeling the exponentially many conditional distributions hand
4.1 Decomposing Densities
We decompose the arbitrary conditioning task into 1d-domain arbitrary conditional estimation problems, This follows from the chain rule of probability, which allows us to write
[ul
p(xu xx)
II (Iug Xouu'z)
(2)
where w' is an arbitrary permutation of u, Zut the 7th unobserved feature given by w', and W/z = [u' 1. The one-dimensional conditionals in Equation 2 are themselves arbitrary con- ditionals due to the choice of permutation. Thus, the arbitrary conditioning problem may be reduced to one-dimensional arbitrary conditional estimation. we can compute any likelihood of the form p(z, x,y) for of {1....,d), d), possible conditional distributions, marginal distributions, and the joint distribution. Although the reduction of arbitrary conditioning to 1d conditional estimation is seemingly straightforward, this is, to the best of our knowledge, the first work to leverage this formulation.
compute any distribution over the features. This includes all
We train our model non-autorrerressively, to output the likelihood p(z; xx/) for any and x, where E (I....,
....,,d d} \ o'. During inference, an autoregressive procedure that repeatedly applies the
chain rule can be used to compute likelihoods of the form p(x.. x.). In practice, this consists of simply adding the previously considered unobserved dimension to the observed set before moving to the next step in the chain
4.2 Likelihoods from Energies
We express likelihoods in terms of energies by modifying Equation to write
-E(x,,
p(
x.)
(3)
Zuixo
and we choose to represent the energy function as a neural network. order to avoid learning a
different network for each conditional, we adopt hiigh--haring scheme (e.g. B3 [4] 20
That is,
An implementation of ACE is available at pesttps://giaan "One could also implement this with masked network architecture (e.g.. MADE [)) that can more efficiently compute autoregressive likelihoods. We choose not to do this for simplicity and to enable the use of arbitrary. and more expressive, architectures.
com/lupalab/ace

---

we use a bitmask b E {0.1}² that indicates which features are observed and define zero-imputing function @ (xx,; b) that returns a d-dimensional vector where unobserved features are replaced with zeros (see Figure 5 in Appendix). These are then used as inputs to the energy network (see (see Figure Thp.
Approximating normalized likelihoods. We use Equation 3 to compute normalized likeli- hoods, but only if the normalizing constant Zunx, is known. Directly computing the normalizer is intractable in general. However, importance sampling can be used to obtaín an estimate [2] Nash and Durkan [22] show that this technique is sufficiently accurate when considering one-dimensional conditionals for joint density estimation, and we adapt the approach for arbitrary conditioning. As-
suming access to proposal distribution q xx) which is similar to the target distribution, we approximate Zu,;x. as
= =ʃ (-E(mu dx = e- (atu, (Xo)
Zu Zucxa
q xxo) g(arve Xx) dra
(4)
Els
xx)
23
5
1
(1) (ati Xo)
2/1 g(Tus (x)
(5)
==1
For some problems, we may have access to good proposal distribution ahead time, but in general, we can learn in parallel with the energy network. We note that importance sampling is also more computationally efficient than the simpler alternative of using points from a grid. Since distributions may be non-smooth and span large domain, fewer samples from good proposal distribution will be able to obtain as good of an estimate as grid that has many more points, which in turn will require fewer evaluations of the energy network. Furthermore, generating samples from the proposal is very efficient, as it only requires one neural network evaluation to obtain the parameters tractable parametric distribution, which can then be easily sampled.
4.3 Training ACE
We learn the proposal distribution alongside the energy function by having a neural network output the parameters of tractable parametric distribution. For the proposal network, we again share weights between all conditionals as is done with the energy network. The proposal network accepts concatenation of b and o(x;; b) as input, and it outputs the parameters w (u,;x,) for mixture of Gaussians for each unobserved dimension ut The proposal network also outputs latent vector, r(u; xx). for each unobserved dimension, which is used input to the energy network in order to enable weight sharing between the proposal and energy networks (22]. Using a different latent vector for each feature allows the latent vectors represent the fact that different unobserved features may depend on x, in different ways.
We then estimate the normalizing constants in Equation 3 Jusing importance sampling:
9
E( ()(T(1)))))
Zuiixo S g(
(6)
(8)
((uxx)))
where (s) is sampled from gz w(u; x.)). This in turn leads to the following approximation of the log-like likeeloooood of zu, given X,:
log p(ru | xx) R Ex Xo: (u:; x.)) log Zuspx.
where we use abbreviated notation in the previous two equations and omit the bitmask b for greater readability. Refer to Figure 1 for the precise inputs of each network
Since Equation 2 gives us a way to autoregressively compute p(x., xx) as a chain of one-dimensional conditionals, we only concern ourselves with learning p(I,,, xx)) for arbitrary u4 and x.. Thus, for a given data point x, we randomly partition it into x, and x,, and jointly optimize the proposal and energy networks with the
likelihood objective
[u]
[u]
J Jxxo: x.: 0)
log p(.,. xx) +
Σ log q(Eu, xx))
(8)
'It is not strictly necessary for the energy network to include x. input, since the latent vectors can encode that information. However, doing acts like sort of skip ciipcconnection, which often helps with deeper networks.
5

---

where f holds the parameters of both the energy and proposal networks. Because we want to optimize the proposal and energy distributions independently, gradients are stopped on proposal samples and proposal likelihood before they are used in |Equation 6) We note that optimizing Equation 8 can be interpreted as a stochastic approximation to optimizing the full autoregressive likelihoods p(x,, x,) (i.e.. Equation 2). For a given data point, we are choosing to optimize individual 1d-conditionals from numerous hypothetical autoregressive chains, rather than all of the ld-conditionals from a single autoregressive chain. That is, each random set of observed values that is encountered during training (in Equation 8) can be seen as a single factor in the product in Equation 2 for some (X,, Xx). Over the course of training though, the same conditionals are ultimately being evaluated in either case.
The negative of Equatiin ijis minimized with Adam [15] over set of training data, where observed and unobserved sets are selected at random for each minibatch (see /Section 5). In some cases, we found it useful include a regularization term in the loss that penalizes the energy distribution for large deviations from the proposal distribution. We use the mean-square error (MSE) between the proposal likelihoods and energy likelihoods as a penalty, with gradients stopped on the proposal likelihoods in the error calculation. The coefficient of this term in the loss is a hyperparameter.
4.4 Inference
4.4.1
Likelihoods
Recall that our model learns one-dimensional conditionals: p( xx). Thus, to obtain a complete likelihood for X., we employ an autoregressive application of the chain rule (see [Equatinn 2). The pseudocode for this procedure is presented in the Appendix (see [Algorithm I}. Since the values of x
are known ahead of time, each one-dimensional conditional can be evaluated in parallel as batch, allowing the likelihood p(x,, xx) to computed efficiently.
Importantly, the order in which each unohserved dimension is evaluated does not matter. As argued in previous work 33 8) this can be considered advantageous, since we can effectively leverage multiple orderings at test time to obtain an ensemble of models. However, this does incur extra computational cost during inference. We note that a model which perfectly captures the true densities would give consistent likelihoods for all possible orderings (thus evaluating only one ordering would suffice). However, current arder-agnostic methods (such as ACE or [33 g)
do not inherently satisfy this constraint. In the Appendix. we study how these inconsistencies can be mitigated in ACE, but we leave a true solution this challenge for future work.
4.4.2 Imputing
Sampling allows us to obtain multiple possible values for the unobserved features that are diverse and realistic ! However, these are not always the primary goals. For example, in the case of data imputation, we may only want a single imputation that aims to minimize some measure of error (see Section 5.1.2). Thus, rather than imputing true samples, we might prefer to impute the mean of the learned distribution. In this case, we forego autoregression and directly obtain the mean of each distribution p( xx) with single forward pass. Analytically computing the mean of the proposal distribution is straightforward since we working with mixture of Gaussians. We estimate the mean of the energy distribution via importance sampling:
Fs
p(ra, (xx) g( xx)
E |
RC
T.
where (() is sampled from q(In, xx). It is worth noting that imputing the mean ignores depen- dencies between features in the unobserved set, so for some applications, other methods of imputing (such as multiple imputation by drawing samples) may make more sense.
4.4.3 Heterogenous Data
Prior approaches to arbitrary conditioning have to make restrictive assumptions when modeling arbitrary dependencies between continuous and discrete covariates. VAEAC 14, for example, makes an assumption of conditional independence given a latent code. ACFlow 20 is not directly
We describe how to generate samples from ACE in the Appendix.

---

Table 1: Test arbitrary conditional log-likelihoods (in nats) for UCI datasets. Higher is better. Likelihood estimates are computed with 20,00 ,00 importance samples for POWER, GAS, and HEPMASS, 10,000 importance samples for MINIBOONE, and 3,000 importance samples for BSDS. Results for ACFlow and VAEAC are taken from Li et al. [20]. The best model for each dataset and missing rate shown in bold. Results are averaged over 5 observed masks.
POWER
GAS
HEPMASS
MINIBOONE
BSDS
Missing Rate
0.0
0.5
0.0
0.1
0.5
0.1
0.5
0.0
01.
0.631
0.600
9.643
9.526
8,530
-3,859
.4.25
-8.133
0.310
-0.688
-5.701
86.701
86.130
M0.613
ACE Proposal
0.583
0.573
0.542
9.484
9.348
8,183
-4,47
-4.9%
-8.497
-0.241
-1.328
-9.169
85.228
84.234
75.767
ACPlow
0:561
0.557
0.458
8.06
7.568
5.405
-8.197
-7.784
-10.538
-0.972
-5.150
-9,892
81.827
80.783
75,050
ACFloo-BBG BG
0.528
0.510
0.417
7,593
7,212
4.818
-6.83
-9.620
-10.975
-1.098
-3.577
-10.849
81,399
79.745
73.061
VAEAC
-.042
-0.43
2.418
2.823
1952
-00.822
-10.389
-11.415
-3,452
-4,242
-9.051
74,850
74.313
66.628
applicable to discrete data given its use of the change of variable theorem. On the other hand, ACE can naturally model arbitrary dependencies between continuous and discrete covariates without any assumptions. In this setting, the proposal network outputs categorical logits for discrete features, as opposed to Gaussian mixture parameters. These logits can themselves be interpreted energies, and we don't 't need to learn an additional energy function for the discrete features. Similarly, ACE could be applied multimodal data straightforward fashion by conditioning the proposal distributions and energy function on a fused multimodal latent representation (which could be freely learned by arbitrary neural networks).
Experiments
5.1 Real-valued UCI Data
We first evaluate ACE on real-valued tabular data. Specifically, we consider the benchmark UCI repository datasets described by Papamakarios et al. [25 (see Table 6 the Appendix).
Unlike other approaches to density estimation that require particular network architectures
20). ACE has no such restrictions. Thus, we use a simple fully-connected network with residual connections
[12] [3] for both the energy network and proposal network. This architecture is highly expressive, yet simple, and helps avoid adding unnecessary complexity to ACE. The bitmask b, which indicates observed features, is sampled for each training example by first drawing &t ~ 24 d 1}. then choosing k distinct features with uniform probability to be observed. Full experimental details and hyperparameters can be found in the Appendix.
We also consider the scenario in which data features are completely missing, i.e., some features are deemed unavailable for particular instances during training and are never part of the observed or unobserved set 55 This allows us to examine the effectiveness of ACE on incomplete datasets, which common when working with real-world data. When training models with missing data, simply modify the sets of observed and unobserved indices to remove any indices which have been declared missing, This a trivial modification and requires no other change the design or training procedure of ACE. We consider two scenarios where data are missing completely at random at a 10% and 50% rate.
5.1.1 Likelihood Evaluation
Table presents the average arbitrary conditional log-likelihoods on held-out test data from models trained with different levels of missing data. During inference, no data missing and b is drawn from a Bernoulli distribution with 0.5. Likelihoods are calculated autoregressively as described in Section 4.4.1) The order in which the unobserved one-dimensional conditionals are computed is randomly selected for each instance.
We draw two key findings from Table First, see that our proposal distribution outperforms ACFlow in all cases. Even this exceptionally simple approach (just a fully-connected network that produces a mixture of Gaussians) can give rise extremely competitive performance, and we see there are advantages to using decomposed densities and unrestricted network architectures. Second,
Features are missing at the per per-instance level. For example, this does not mean that the 2h feature is never observed for all training instances.
7

---

Table 2: Test marginal log- sigllikeliooods (in nats) for UCI datasets. Higher is better. We evaluate the marginal distributions of the first 3, 5, and 10 dimensions of each dataset (POWER and GAS don't t have 10 features, so the joint likelihood over all features is reported instead). The same number of importance samples are used as in Table I] Results for ACFlow and TAN are taken from Li et al. 220] Bold indicates where ACE outperformed ACFlow. Results are averaged over 5 observed masks.
POWER
GAS
HEPMASS
MINIRGONE
BSDS
Dimensions
8
3
10
3
5
10
ACE
-0.56
1.42
0.58
1.31
4.31
12.20
-4.00
-5.91
-10.72
-2.13
-3.80
-7.94
5.10
9.37
20.31
ACE Proposal
-0.58
1.35
0.49
1.11
3.98
11.84
-4.01
-5.4
-10.82
2.14
-3.79
-7.93
5.06
9.30
20.17
ACFlow
-0.57
1.34
0,42
0.78
3.01
10.13
-4003
-6.19
-1.55
-2.76
-5.31
-10.36
5.06
9.26
19.60
TAN
40.54
1.40
0.57
1.22
4.47
12.09
4.000
-5.92
10.87
-2.13
-3.73
-8.
5.11
9.43
20.44
Hеppass
0.6-
9.6-
044-
Iresupa
02
02-
02
ACF
AdFlo
0.-
0.5
0.1
0:0
6:7
2.1
05
3.0
0.3 0.
6-5
0.0
0.2
15
0.0 0.1
35
Miser Rate
Mintig Rate
Ninsing Rate
Maxing
Missing Rate
Figure 2: Normalized root-mean-square square error (NRMSE) of imputations generated by ACE. Lower is better. NRMSE is computed the aat-mean--squrre error normalized by the standard deviation of each feature and then averaged across all features. Estimates of energy distribution means are computed with 20,000 importance samples for POWER and GAS, 10,000 importance samples for HEPMASS and MINIBOONE, and 3,000 importance samples for BSDS, Results for ACFlow and VAEAC are taken from Li et al. (20).
we find that in every case, the likelihood estimates produced by the energy function are higher than those from the proposal, illustrating the benefits of an anergy-based approach which imposes no biases the shape of the learned distributions.
We also examine the arbitrary marginal distributions learned by ACE, i.e., the unconditional distribu- tion over a subset of features. We again test our model against ACFlow, and we additionally compare to Transformation Autoregressive Networks (TAN) 23], which are designed only for joint likelihood estimation. A separate TAN model has to be trained for each marginal distribution. While a single ACFlow model can estimate all marginal distributions, Li et al. (20) retrained models specifically for arbitrary marginal estimation. Contrarily, marginals as were used for arbitrary conditionals. Results are provided in Table 2] We find that ACE outperforms ACFlow in all cases and even surpasses TAN most of the time, even though ACFlow and TAN both received special training for marginal likelihood estimation and ACE did not.
used the same ACE models when evaluating arbitrary
5.1.2 Imputation
We also evaluate ACE for data imputation, where some elements are missing from a dataset completely at random and we seek to infer their values. ACE is naturally applied to this task - we consider p(x,. x,), where x,, contains the missing features.
Figure 2 shows the normalized root-mean mean-square error (NRMSE) on held-out test data. Again, we consider models trained with three different levels of missing data. During inference. b is drawn from a Bernoulli distribution with p = 0.5 for the 0% and 50% missing rates and p = 0.9 for the 10% missing rate The means of the unobserved distributions used the imputed values (see Section 4.4.2).
As seen in Figure 2 ACE achieves lower NRMSE score than ACFlow in all cases (exact numbers are available in the Appendix). These results further validate ACE's ability to accurately model arbitrary conditionals, leading us to again advocate for simple methods with few biases. It is also
This means that for the 10% missing rate, we are imputing fewer values (only 10% as opposed to 50%) during inference than with the other two missing rates. This explains the lower errors for the 10% missing rate that we see in Figure 2 Even though there is missing data during training, are estimating the values of fewer values based on a larger amount of observed data, which intuitively should result in more accurate imputations.
8

---

Table 3: Conditional log-likelihoods (LL),
Table 4: Log-likelihood and imputation results for the UCI Adult dataset. NRMSE measures imputa- tion performance for continuous features and accu- racy is used for discrete features.
joint bits-per-dimension (BPD), and inpainting peak to-noise-ratii (PSNR) for ACE and ACFlow on MNIST.
Method
LL
BPD
PSNR
Method
LL
NRMSE
Aceuracy
ACE
1043.88
1.42
16.81
ACE
2.38
0.90
0.69
ACE Proposal
828.60
2.11
16.70
ACE Proposal
2.24
0.89
0.69
ACFlow
875.87
3.09
13.75
VAEAC
-7.25
0.91
0.67
22
2 2 2 2
2
2 7
2
2 7 4 0 9
2
6
5
5 9 9
5 4 7
6
7
4
6
7
7 4
7
7 77 4
7
7
5 O
3
G
4
4 4 4
4
4 7
4 9 0
4
8
7
7
0
0
3 8
8
6
8
0
0
0
0
6
0
0
9 3
2
0
7
(a) Inpaintings from ACE and ACFlow. Left: Groundtruth and observed pixels. Middle: ACE samples. Right: ACFlow samples.
(b) Samples from the joint distribution (i.c., all pixels are unobserved). Left: ACE samples. Right: ACFlow samples.
Figure 3: MNIST samples generated by ACE and ACFlow.
worth noting that ACE and ACE Proposal do comparably in this imputation task, which estimates the first-order moment of conditional distributions. However, as evidenced in Table the energy-based likelihood better captures higher-order moments.
5.2 High-dimensional Data
We examine ACE's ability to scale to general high-dimensional data by considering the flattened MNIST dataset [18] (i.e., each example 784-dimensional vector). order make a fair comparison, trained non-convolutional ACFlow model (using the authors' code) the flattened data as well. The only change to ACE's training procedure in the distribution bitmasks that are sampled during training, which we detail in the Appendix.
Table compares ACE and ACFlow in terms of arbitrary conditional log- likeliiikiooos, joint bits-per- dimension (BPD), and peak lignal-tonoise-
for inpaintings, and we see that ACE outperforms ACFlow for all three metrics. We also note that ACE's BPD of 1.42 comparable to prior methods for joint density estimation such as TAN, MADE, and MAF, which have reported BPD of 1.19, 1.41, and 1.52 respectively (lower better), despite the fact that these other methods can only model the joint distribution [23]. Qualitatively, we see that ACE produces more realistic inpaintings (see Figure 3ab and samples from joint (see Figure 3b) than ACFlow. These findings indicate that ACE's performance scales well to high-dimensional data.
5.3 Mixed Continuous-Discreete Data
In order to demonstrate ACE's ability to model data with both continuous and discrete features, we conduct experiments the UCI Adult datase[" which offers relatively even balance between the number of continuous and discrete features (see below). We preprocess the data by standardizing continuous features and dropping instances with missing values. We also only keep instances for which the native-country feature is United-States! The processed data has 6 continuous
https://archive ics
Such instances account for about 91% of the data, and the other 9% take on of 39 possible values. Thus able discard highly class- dass-imbalanced feature while retaining the vast majority of the data.

---

Power
Hepmass
boons
BSDS
Samples
importacce
Impprtance Samples
importance
stimpo Samples
Figure 4: Percent error of normalizing constant estimates obtained with importance sampling when compared to "true" values obtained with numerical integration. Estimates grow more accurate the number importance samples increases. Whiskers indicate 1.5 times the IQR.
features and 8 discrete features and is split into train, validation, and test partitions of size 22003, 5501, and 13788 respectively.
We trained an ACE model and VAEAC model (using the authors' publicly available code) on this dataset and evaluated them in terms of likelihoods and imputation. For discrete features, the mode of the learned distribution is imputed (as opposed to the mean). The results are presented in Table 4 where we see that ACE outperforms VAEAC on all metrics. We also reiterate that due to its use of the change of variable theorem, ACFlow cannot be trained on this dataset.
5.4 Importance Sampling Accuracy
In order to better understand the impact of the number of importance samples used when estimating normalizers, we compare the importance sampling estimates from model to "true" estimates obtained via numerical integration (similar to (22). For each UCT dataset, we randomly select 10,000 one-dimensional conditionals (from a validation set that was not seen during training) and integrate the unnormalized energy likelihoods using trapezoidal rule [26). For each conditional, we then also estimate the normalizer with increasing number of importance samples. Figure 4]shows the percentage errors of the importance sampling estimates. In most cases, small number of samples (e.g., 20) can already produce relatively accurate estimates, and we see that the estimates grow more accurate as the number of samples increases.
Conclusion
In this work, we present a simple approach to modeling all arbitrary conditional distributions p(x, x,) over set of covariates. Our method, Arbitrary Conditioning with Energy (ACE), is the first to wholly reduce arbitrary conditioning to one-dimensional conditionals with arbitrary observations and estimate these with energy functions. By using an energy function to specify densities, ACE can more easily model highly complex distributions, and it can freely use high-capacity networks to model the exponentially many distributions at hand.
Empirically, we find that ACE achieves state-of-the-ar the-art performance for arbitrary conditional/marginal density estimation and for data imputation. For given dataset, all of these results produced with the same trained model. This high performance does not come at the cost of complexity - ACE is much simpler than other common approaches, which often require restrictive complexities such as normalizing flow models or networks with specially masked connections (20 g).
Furthermore, ACE's proposal distribution, a basic mixture of Gaussians, outperforms prior methods, demonstrating that the principle of learning one-dimensional distributions still powerful when decoupled from energy-based learning. These results emphasize that seemingly complex problems do not necessitate highly complex solutions, and we believe future work on arbitrary density estimation will benefit from similar ideas.
Acknowledgments and Disclosure of Funding
This work was supported in part by grants NIH IROIAA02687901AL and NSF IIS2
10

---

References
[1] M. Belghazi, M. Oquab, and D. Lopez-Paz. Learning about an exponential amount of conditional distributions. In Advances in Neural Information Processing Systems, pages 13703-13714 2019.
[2] C. M. Bishop. Pattern recognition and machine learning. springer, 2006.
[3] C. J. Butz, J. S. Oliveira, A. E. dos Santos, and A. L. Teixeira. Deep convolutional sum-product networks. Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 2019.
[4] L. Dinh, J. Sohl-Dickstein, and S. Bengio. Density estimation using real nvp. arXiv preprini 160.00605.08803 2016.
[5] L. Douglas, I. Zarov, K. Gourgoulias, C. Lucas, C. Hart, A. Baker, M. Sahani, Y. Perov, and Johri. A universal marginalizer for amortized inference in generative models. arXiv preprint 2017.
arXiv:
[6] R. Fakoor, P. Chaudhari, Mueller, and Smola. Trade: Transformers for density estimation. arXiv preprini arXiv:2004.02444 2020.
[7] S. Fielding, P. M. Fayers, A. McDonald, G. McPherson, and M. K. Campbell. Simple imputation methods were inadequate for missing not at random (mnar) quality of life data. Health and Quality of Life Outcomes, 6(1):1-9, 2008.
[8] M. Germain, K. Gregor, I. Murray, and H. Larochelle. Made: Masked autoencoder for distri- bution estimation. In International Conference on Machine Learning, pages 881-889. PMLR, 2015.
[9] I.. Gondara and Wang. Mida: Multiple imputation using demoising autoencoders. In Pacific-Asia Conference on Knowledge Discovery and Data Mining, pages 260-272. Springer, 2018.
[10] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. Advances in neural information processing systems, 27: 2672-2680, 2014.
[11] W. Grathwohl, R. T. Chen, J. Bettencourt, I. Sutskever, and D. Duvenaud. Ffjord: Free-form con- tinuous dynamics for scalable reversible generative models. arXiv preprint arXiv: 1810.01367. 2018.
[12] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770- 778, 2016.
[13] K. He, X. Zhang, S. Ren, and J. Sun. Identity mappings in deep residual networks. In European conference computer vixion, pages 630-645. Springer, 2016.
[14] O. Ivanov, M. Figurnov, and D. Vetrov. Variational autoencoder with arbitrary conditioning. arXiv preprint 10X.::100..0232.. 2018
[15] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint 1129 2014.
[16] M. Lázaro-Gredilla, W. Lehrach, N. Gothoskar, G. Zhou, A. Dedieu, and D. George. Query training: Learning worse model to infer better marginals in undirected graphical models with hidden variables. arXiv preprint arXiv:2006.06800 06803, 2020.
[17] Y. LeCun, S. Chopra, R. Hadsell, M. Ranzato, and F. Huang. A tutorial on energy-based learning. Predicting structured data, 1(0), 2006.
[18] Y. LeCun, C. Cortes, and C. Burges. Mnist handwritten digit database. ATT Labs [Online]. Available: http:///ww...cc....o//xx//////// 2, 2010.

---

[19] S. C.-X. Li, B. Jiang, and B. Marlin. Misgan: Learning from incomplete data with generative adversarial networks. arXiv preprint arXiv:1902.09599 2019.
[20] Y. Li, S. Akbar, and J. Oliva. ACFlow:
Flow models for arbitrary conditional likelihoods. In H. D. III and A. Singh, editors, Proceedings of the 37th International Conference Machine Learning. volume 119 of Proceedings of Machine Learning Research, pages 5831-5841. PMLR, 13-18 Jul 2020. URL http::/proceeddnggg mlr wwwppppessss... v119 1120a. html
[21] P.-A. Mattei and J. Frellsen. MIWAE:
Deep generative modelling and imputation of incomplete data sets. In K. Chaudhuri and R. Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 4413-4423, PMLR, 09-15 Jun 2019. URL
wtwall.pres//voe /
mattei 19t.e1999..lllll1
[22] C. Nash and C. Durkan. Autoregressive energy machines. arXiv preprint arXiv:1904.0500 2019.
[23] J. B. Oliva, A. Dubey, M. Zaheer, B. Poczos, R. Salakhutdinov, E. P. Xing, and J. Schneider. Transformation autoregressive networks. arXiv preprint 18Xi 09899, 2018.
[24] A. v. d. Oord, N. Kalchbrenner, O. Vinyals, L. Espeholt, A. Graves, and K. Kavukcuoglu. Conditional image generation with pixelenn decoders. arXiv preprint arXiv: /6civ:666..mm. 05328, 2016.
[25] G. Papamakarios, T. Pavlakou, and I. Murray. Masked autoregressive flow for density estimation. In Advances in Neural Information Processing Systems, 2338-2347, 2017.
[26] M. Pitkin. lintigrate. http:s//gghhhbooco//aatttttttttp
2017.
[27] H. Poon and P. Domingos. Sum- networks: A new deep architecture. In 201/ IEEE International Conference on Computer Vision Workshops (ICCV Workshops), pages 689-690. IEEE, 2011.
[28] M. Przewicžlikowski, M. Smieja, and L. Struski. Estimating conditional density of missing values using deep gaussian mixture model. In International Conference on Neural Information Processing, pages 220-231 231. Springer, 2020.
[29] D. B. Rubin. Using the sir algorithm to simulate posterior distributions. Bayesian statistics, 3: 395-402, 1988.
[30] S. Saremi, A. Mehrjou, B. Schölkopf. and A. Hyvärinen. Deep energy estimator networks. arXiv preprint 1805.0 083000 2018
[31] D. J. Stekhoven and Bühlmann. ninnpretstt-oon-paaaettti missing value imputation for mixed-type data. Bioinformatics, 28(1):112-114) 2012.
[32] O. Troyanskaya, M. Cantor, G. Sherlock, P. Brown, T. Hastie, R. Tibshirani, D. Botstein, and R. B. Altman. Missing value estimation methods for dna microarrays. Bioinformatics, 17(6): 520-525, 2001.
[33] B. Uria, I. Murray, and H. Larochelle. A deep and tractable density estimator. In International Conference on Machine Learning, pages 467-475, PMLR, 2014.
[34] J. Yoon, J. Jordon, and M. van der Schaar. GAIN: Missing data imputation using generative adversarial nets. In Dy and A. Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 5689-5698, Stockholmsmässann Stockholm Sweden, 10-15 Jul 2018. PMLR. URL http://prroceedingg.. mlr. htt.://ss....rr.eee/e/o///oo
12


Reviews:
[{"review_id": "wb7lj4wo8U-", "reviewer": "Reviewer_LU53", "summary": "This paper proposes a novel technique (ACE) for training a density estimator that allows for arbitrary conditioning and marginalization.  The core idea is to decompose the target conditional distribution into a product of autoregressive 1-D conditionals.  Each scalar conditionals are substantially easier to model than their joint counterparts.  The training scheme consists of two components, denoted proposal and energy networks.  Energy network is used to model the energy function that models the target distribution, and the proposal network is to aid in the important sampling procedure used to estimate the normalizing constant for the energy function.  The paper has convincing and thorough experimental results that compare against several representative baselines, where ACE exhibits a strong performance gain over them.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths\n- The paper presents a simple idea that is shown to be effective experimentally.  The technique is very sensible, and it overcomes several disadvantages that existing methods have. \n- Writing is very clear and easy to follow.\n\n### Concerns\n- Estimating marginals: The paper claims that ACE can be used to estimate arbitrary marginal distributions, but the paper doesn't explicitly state how it can be done.  How is this done?  One very naive way to compute the marginal $p(x_o)$ would be to fix some random values for the remaining variables ($x_u$), then computing $p(x_o) = p(x_u, x_o) / p(x_u \\vert x_o)$.  I'm concerned that this marginal density estimator likely has very high variance because. (1) Both $p(x_u, x_o)$ and $p(x_u \\vert x_o)$ suffer from high variance due to model inaccuracy and dependence on the ordering, and (2) The choice of $x_u$ presumably affects both of these quantities heavily.  This is seemingly a much more task than estimating conditional distributions, so it'd be important to include experimental results for marginal density estimation.  Also, additional questions include:\n  - Is this naive method how one would estimate an arbitrary marginal, or is there a better way?\n  - If the naive method is what one should use, how does ACE perform, and how much variance is there?\n\n- Justification for the loss function: The loss function, while reasonable, feels a bit arbitrary.  First, I am curious how important it is for the energy and proposal networks to share weights (as was done in this paper).  An interesting ablation study would be to separately train a proposal network then using it to train the energy network.  Second, how accurate is the estimation of the normalizing constant?  How many importance samples ($S$) were necessary?\n  \n- Baseline: It would have been nice to consider Locally Masked Convolution for Autoregressive Models [1] as an additional baseline, which I believe is a very relevant baseline to ACE. \n\n### Comment\nI enjoyed reading this paper, and I found the idea very simple yet effective.  Experiments and the technical motivation could be a bit stronger, but I believe that the submission is overall good.\n\n#### References\n[1] Ajay Jain, Pieter Abbeel, Deepak Pathak. Locally Masked Convolution for Autoregressive Models. In Conference on Uncertainty in Artificial Intelligence (UAI), 2020.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors briefly discuss potential societal impact of their work.", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "oW7J2zSRRO", "reviewer": "Reviewer_4qhh", "summary": "The main contribution of this paper is a new method for data imputation. The presented method is simpler than existing ones, and shows good results in conditional likelihood estimation and in some cases in data imputation. However, experimental results in data imputation are difficult to compare to existing ones, and the authors use a smaller number of datasets to evaluate their method, compared to preceding works.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Detailed summary\n\nGiven a dataset $D$, this article presents a method for:\n1) estimating the conditional likelihood of a subset A of the coordinates of a vector $x \\in D$, given a subset B disjoint from A. A and B can be chosen arbitrarily;\n2) imputing a missing part of a vector $x \\in D$.\n\nThe proposed method is both simpler and more accurate than the preceding ones. \n\nThe proposed method is broadly based on the paper *Autoregressive energy machines*, Nash 2019. It consists in training jointly two neural networks: \n1) a proposal network, which outputs a conditional distribution of unobserved coordinates $x_u$ of $x$, given observed ones $x_o$; it also outputs a latent vector $\\gamma$; \n2) an energy network, which takes as input $x_o$, the latent vector $\\gamma$, and a proposition $x_{u_i}$ for the $i$-th coordinate of $x_u$, and outputs the *energy* associated to such a configuration.\n\nAfter training, each of these networks can be used independently: the proposal network can be use to impute missing data, the energy network can be used to measure the likelihood of the coordinates of a vector given the others.\n\n## Experiments\n\nThe authors test their method with two preceding ones: VAEAC and ACFlow. Their method performs better. The choice of datasets is the same as in the paper introducing ACFlow.\n\nAccording to the authors, this restricted choice of concurrent methods is due to the fact that GAN methods (GAIN and MisGAIN) are known to perform poorly compared to VAEAC [1]. \n\nHowever, about imputation:\n1) the tested datasets in [1] are not tested here (except MNIST with mask);\n2) the experiment made on MNIST in [1] is not comparable to the one presented in this paper: in Fig. 3a, the hidden part is much smaller than in [1], Fig. 1; anyway, there is no comparison with VAEAC in this setup;\n3) the same remark holds when comparing to ACFlow [2]: how can we be sure that results in Fig. 1 are representative of the results obtained with ACE? It would be fair to compare ACE on the same digit, partly hidden with the same mask as in [1] and [2]. This way, the reader would be sure than the presented results are fair;\n4) contrary to [1] and [2], there is no experiment with CelebA and Omniglot;\n5) GAN methods are assumed to perform poorly on the chosen datasets, but there is no evidence of that (the datasets used here differ from [1], in which such experiments have been made).\n\n## Clarity\n\nOverall, the paper is well written.\n\n## Originality\n\nThe proposed method is an adaptation of an existing one. However, the application to arbitrary conditioning is novel.\n\n## Significance\n\nData imputation is a major question in AI.\n\nThe proposed method is simple, which is the main advantage of the proposed method.\n\nAlthough experimental results are encouraging when estimating the likelihood of vectors, the only clear result in imputation is in Fig. 2, which insufficient.\n\n## References\n\n[1] *Variational Autoencoder with Arbitrary Conditioning*, Ivanov 2019\n\n[2] *ACFlow: Flow Models for Arbitrary Conditional Likelihoods*, Li 2019\n\n## Rebuttal\n\nI have read the authors' rebuttal, and I have several remarks:\n * I understand the issue raised by the authors about image completion, especially for medium size images, such as in CelebA or Omniglot;\n * the supplementary experiments made by the authors are encouraging, and may be put in the final paper (or at least in the Appendix), since they challenge [1] on 3 of the 5 datasets presented in Table 1 of [1]. However, I am curious to see a comparison with [1] on the 2 remaining datasets (Zoo and Phishing).\n\nWith these 3 new experiments, the evidence that the presented method works better than VAEAC (and ACFlow?) is stronger. However, to be completely fair, a comparison should be made with the two remaining datasets. I have checked out Ivanov's code, and I confirm that the code related to these datasets is missing, so the 3 tested datasets have not been \"cherry-picked\".\n\nI tend to be mostly satisfied with the authors' answer. Weak accept.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Main limitation: lack of significant experiments in imputation (see Main Review).", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}, {"review_id": "eA8MTH4vt9H", "reviewer": "Reviewer_qrbW", "summary": "This paper proposes arbitrary conditioning with energy (ACE), a method for density estimation that can access any conditional distribution over any subset of random variables. It presents learning and inference procedures for ACE, and validates ACE's performance on arbitrary conditional/marginal density estimation and data imputation for real-valued, high-dimensional and mixed continuous-discrete data.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: the method is new, but its claim of novelty is misleading. The paper claims that it is the first work to leverage the reduction of arbitrary conditioning to 1d conditional estimations. This claim seems too strong. For example autoregressive methods like [31] can similarly handle arbitrary conditioning, as long as the variables to be conditioned on are at the beginning and the variables to be marginalized are at the end. Similar points are made in *Query Training: Learning a Worse Model to Infer Better Marginals in Undirected Graphical Models with Hidden Variables, Miguel Lázaro-Gredilla, Wolfgang Lehrach, Nishad Gothoskar, Guangyao Zhou, Antoine Dedieu, Dileep George, AAAI 2021* (a missing citation that explores very similar ideas and should be covered at least in related works), section *Queries that Need to Be Answered*. It is true that none of these works pursue this task, but the way the paper is framing the contributions should be weakened. To me the main contribution seems to be the use of an energy-based formulation together with an importance sampling based approach, which gives the model useful flexibility to outperform pervious methods based on parametric distributions.\n\nQuality and clarify: the paper is technically sound, and clearly written for the most part. There are some points that need clarifying:\n1. Equation 8 is quite confusing. How are the x_o and x_u partitioned? Where is the autoregressive component?\n2. For likelihood evaluations in 4.4.1, it's not clear how ordering is handled. Is the method essentially computing the product of all possible 1d conditional distributions? If so, even though these can be computed in parallel, there are still a huge number of conditional distributions and it looks very expensive. If the method is using a particular order, then different orders would give different results and it's not clear how the numbers in table 1 are derived.\n3. For ACE with proposal, since it's using a mixture of Gaussians (i.e. a parametric distribution), it looks similar in idea to [31]. While it's good to see that having the additional flexibility from the energy function gives further performance boost, I'm also curious to see what a method like [31] would perform. \n\nSignificance: the studied problem is an interesting and important problem, and the proposed ACE method can potentially be very useful.\n\nOne other question: ACE uses importance sampling. How many samples are used? Would the Monte Carlo estimates result in high variance, e.g. in gradients estimations? How does that affect the learning dynamics?\n\n===================================\nUpdate after rebuttal:\n\nDuring the discussion period it is brought to my attention that the use of an energy-based formulation in combination with importance sampling is also not new. In that case the methodology seems like a (somewhat trivial) combination of several existing pieces, and I ask the authors to weaken their claims of novelty. The experimental results are still interesting and relevant, so I'm maintaining my current score (with the assumption that claims of novelty will be weakened).\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "9Fi22Vjf-Sv", "reviewer": "Reviewer_BNRH", "summary": "Given D random variables, the number of possible conditional distributions is exponential in D. The authors propose a method for efficiently estimating this exponentially large set of distributions with a compact autoregressive energy-based model. The method is applied to conditional density estimation and imputation tasks on UCI repository datasets and MNIST, showing favourable performance compared to strong baselines.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Overall verdict**\n\nThis is an extremely well-written paper with decent empirical results on an important task. The major drawback is the lack of methodological novelty; readers who have read the prior literature may see this paper as somewhat 'obvious'. Overall, I still tend towards acceptance, so long as the authors alter section 4.2/4.3 to *explicitly* state that the contents are adapted from Nash & Durkan (2019).\n\n\n**Originality**\n\nMethodologically, this paper is not especially novel. The proposed method--ACE--is a relatively straightforward combination of two existing ideas: 1) The use of a single autoregressive model to model all possible conditionals (Uria et al., 2014) and 2) The use of an energy function to specify the univariate conditionals of this autoregressive model (Nash & Durkan, 2019).\n\nHowever, the paper is more original in the task that is solves. Prior work has largely been concerned with joint density estimation, whereas this paper targets conditional density estimation. I say 'largely' because the methods of Uria et al. and Germain et al. (2015) can clearly be used for conditional density-estimation, but their experiments are not focused on this.\n\nIn a sense then, this paper does two things:\n\n1) Reminds the community of an overlooked fact, namely that a single autoregressive model can be used for arbitrary conditional density estimation\n2) Demonstrates that such an approach, when combined with modern energy-based architectures, achieves decent empirical performance\n\n\n**Quality**\n\nSince the paper lacks significant methodological or theoretical contributions, its technical quality largely depends on the experiments.\n\nI think the experiments are of an *acceptable* quality. The authors follow a fairly standard setup of UCI + MNIST datasets and assess conditional log-likelihoods + RMSE of imputations. The proposed method clearly and consistently outperforms a strong baseline of ACFlow. Moreover, its very nice that ACE is applicable to continuous & discrete data.\n\nCould the authors comment on the relative computational cost of ACE versus ACFlow? Both in terms of training costs (iterations & wallclock time until convergence) and cost of inference/sampling?\n\n\n**Relation to prior work**\n\nIn various places, the authors cite Nash & Durkan (2019) who introduced Autoregressive Energy Machines (AEM). However, there is insufficient explanation of how AEM and ACE are related. This is highly problematic, as most of Sections 4.2 and 4.3 are adapted from Nash & Durkan without sufficiently explicit attribution. Specifically, Nash & Durkan introduced the idea of jointly training an autoregressive energy model and proposal using dimension-wise importance sampling. \n\nOne key difference between AEM and ACE is the way in which they ensure that autoregressive constraints are satisfied. AEM uses autoregressive-preserving masking in the hidden layers of the network, whilst ACE avoids this by duplicating the input (one copy for each unobserved variable) and applying different masks to each copy. Both are valid strategies, but the former is more computationally efficient, raising the question of why the authors chose the second strategy. In fact, the ACE-style strategy was introduced very early on in NADE (Larochelle & Murray, 2011), and was later made more computationally efficient with hidden-layer masking in MADE (Germain et al., 2015). Arguably, therefore, this paper is taking a step backwards.\n\n**Clarity**\n\nThe paper is exceedingly well-written.\n\n**Significance**\n\nI think this paper is of moderate significance to the generative-modelling-with-missing-data literature. Autoregressive models have generally been overlooked in this literature, despite the early contributions by Uria et al. (2014).\n\n\n**References**\n\nUria, Benigno, Iain Murray, and Hugo Larochelle. \"A deep and tractable density estimator.\" International Conference on Machine Learning. PMLR, 2014.\n\nLarochelle,  Hugo  and  Murray,  Iain.   The  neural  autoregressive distribution estimator. In Proceedings of the14th International Conference on Artificial Intelligenceand Statistics (AISTATS 2011), volume 15, pp. 29–37, Ft.Lauderdale, USA, 2011. JMLR W&CP.\n\nM. Germain, K. Gregor, I. Murray, and H. Larochelle.  MADE: Masked Autoencoder for DistributionEstimation.Proceedings of the 32nd International Conference on Machine Learning, pages 881–889,2015\n\nNash, C. & Durkan, C.. (2019). Autoregressive Energy Machines. Proceedings of the 36th International Conference on Machine Learning, in Proceedings of Machine Learning Research 97:1735-1744 Available from http://proceedings.mlr.press/v97/durkan19a.html. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The main potential limitation I see is that the method may be more computationally costly than the baselines, and I couldn't see a discussion of this in the main text.\n\nOtherwise, I thought the authors were honest about any limitations, and I agree that there are no obvious negative societal impacts from their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose a method for estimating conditional marginal distributions under arbitrary choices of the conditioning and marginalization sets using a decomposition into single-variable conditionals (autoregressive) using a single (neural network based) energy function to model all possible conditionals.  In general, the reviewers were positive but very reserved about the work; given the existence of several previous related works, the method was viewed as incremental or even obvious to experts in the area, and felt that the experimental validation is not strong.  On the whole reviewers thought the work's ideas were generally worthy of publication, but viewed the current paper as only being right on the border for the NeurIPS conference.
Decision: Accept (Poster)