Spatial Dependency Networks: Neural Layers for Improved Generative Image Modeling
Authors: Đorđe Miladinović
Aleksandar Stanić
Stefan Bauer
Jürgen Schmidhuber
Joachim M. Buhmann
Venue: ICLR 2021 Poster (2021)
Keywords: Neural networks
Deep generative models
Image Modeling
Variational Autoencoders

Abstract: How to improve generative modeling by better exploiting spatial regularities and coherence in images? We introduce a novel neural network for building image generators (decoders) and apply it to variational autoencoders (VAEs). In our spatial dependency networks (SDNs), feature maps at each level of a deep neural net are computed in a spatially coherent way, using a sequential gating-based mechanism that distributes contextual information across 2-D space. We show that augmenting the decoder of a hierarchical VAE by spatial dependency layers considerably improves density estimation over baseline convolutional architectures and the state-of-the-art among the models within the same class. Furthermore, we demonstrate that SDN can be applied to large images by synthesizing samples of high quality and coherence. In a vanilla VAE setting, we find that a powerful SDN decoder also improves learning disentangled representations, indicating that neural architectures play an important role in this task. Our results suggest favoring spatial dependency over convolutional layers in various VAE settings. The accompanying source code is given at https://github.com/djordjemila/sdn.

Published as a conference paper at ICLR 2021
SPATIAL DEPENDENCY NETWORKS: NEURAL LAYERS FOR IMPROVED GENERATIVE IMAGE MODELING
Dorde Miladinovié
Aleksandar Stanié Swiss AI Lab IDSIA, USI Lugano, Switzerland
Stefan Bauer
ETH Zurich Zürich, Switzerland
Max-Planck Institute
Tübingen, Germany
Jürgen Schmidhuber Swiss AI Lab IDSIA, USI Lugano, Switzerland
Joachim M. Buhmann
ETH Zurich Zürich, Switzerland
ABSTRACT
How to improve generative modeling by better exploiting spatial regularities and coherence images? We introduce a novel neural network for building image generators (decoders) and apply it variational autoencoders (VAEs). In our spa- tial dependency networks (SDNs), feature maps at each level of a deep neural net are computed in spatially coherent way, using sequential gating-hased mecha- nism that distrihutes contextual information across 2-D space. We show that aug- menting the decoder of a hierarchical VAE by spatial dependency layers consider- ably improves density estimation over baseline convolutional architectures and the tfae-oofthe--an among the models within the same class. Furthermore, we demon- strate that SDN can be applied to large images by synthesizing samples of high quality and coherence. coder also improves learning disentangled representations, indicating that neural architectures play an important role in this task. Our results suggest favoring spa- tial dependency over convolutional layers in various VAE settings. The accompa- nying source code is given at: htttps://githii com http://ggtaaaa.cc/cccccc//////// /sn
vanilla VAE setting, we find that powerful SDN de-
INTRODUCTION
The abundance of data and computation are often identified facilitators of the deep learning revolution. In addition to this technological lcap, historically speaking, most major algorithmic ad- vancements critically hinged on the existence of inductive biases, incorporating prior knowledge in different ways. Main breakthroughs in image recognition (Cireşan et al.) 2012 Krizhevsky et al.] 2012) were preceded by the long-standing pursuit for shift-invariant pattern recognition (Fukushima & Miyake, 1982) which catalyzed the ideas of weight sharing and convolutions (Waibel| 1987 Le-
Cun et al. 1989). Recurrent networks (exploiting temporal recurrence) and transtormers (modeling the "attention" bias) revolutionized the field of natural language processing (Mikolov et al.) 2011]] Vaswani et al., 2017). Visual representation learning also often based on priors e.g. independence of latent factors (Schmidhuber 1992, Bengio et al.) (013) or invariance to input transformations
(Becker & Hinton| 1992|Chen et al.] 2020). Clearly, one promising strategy to move forward is to introduce more structure into learning algorithms, and more knowledge on the problems and data.
Along this line of thought, we explore a way to improve the architecture of deep neural networks that generate images, here referred to as (deep) image generators, by incorporating prior assump- tions based on topological image structure. More specifically, we aim to integrate the priors on spatial dependencies in images. We would like to enforce these priors all intermediate image representations produced by an image generator, including the last one from which the final image synthesized. To that end, introduce class of neural networks designed specifically for build- ing image generators - spatial dependency network (SDN). Concretely, spatial dependency layers of
"Correspondence tjordjjm@etthz.c.

---

Published as a conference paper at ICLR 2021
Oupput
Encoder
Decoder
CNN layara
2
SDN layers
2 Po (2
Input
(a)
(b)
Figure I: (a) DAG of a spatial dependency layer. The input feature vector (red node) is gradu- ally refined (green nodes) as the computation progresses through the four sub-layers until the output feature vector is produced (blue node). In each sub-layer, the feature vector is corrected based on contextual information. Conditional maps within sub-layers are implemented as learnable determin- istic functions with shared parameters; (b) VAE with SDN decoder reconstructing 'celebrity'.
SDN (SDN layers) incorporate two priors: (i) spatial coherence reflects our assumption that feature vectors should be dependent each other in spatially consistent, smooth way. Thus in SDN, the neighboring feature vectors tend to be more similar than the non-neighboring ones, where the sim- ilarity correlates with the 2-D distance. The graphical model (Figure Th) captures this assumption. Note also that due to their unbounded receptive field, SDN layers model long-range dependencies; (ii) spatial dependencies between feature vectors should not depend on their 2-D coordinates. Math- ematically speaking, SDN should be equivariant to spatial translation, in the same way convolutional networks (CNN) are. This is achieved through parameter (weight) sharing both in SDN and CNN.
The particular focus of this paper is the application of SDN to variational autoencoders (VAEs) (Kingma & Welling, 2013). The main motivation is to improve the performance of VAE generative models by endowing their image decoders with spatial dependency layers (Figure [p). While out of the scope of this work, SDN could also be applied to other generative models, e.g. generative adversarial networks (Goodfellow et al. 2014). More generally, SDN could be potentially used in any task in which image generation is required, such as image-to-image translation, super-resolution, image inpainting, image segmentation, scene labeling.
SDN is experimentally examined in two different settings. In the context of lime-iif--mmee den- sity modeling, SDN-empowered hierarchical VAE is shown to reach considerably higher test log- likelihoods than the baseline CNN-based architectures and can synthesize perceptually appealing and coherent images even high sampling temperatures. In synthetic data setting, observe that enhancing non-hierarchical VAE with an SDN facilitates learning of factorial latent codes, suggesting that unsupervised 'disentanglement" of representations can be bettered by using more powerful neural architectures, where SDN stands out as a good candidate model. The contributions and the contents of this paper can be summarized as follows.
CONTRIBUTIONS AND CONTENTS:
The architecture of SDN is introduced and then contrasted to the related ones such as convolutional networks and self-attention.
The architecture of SDN-VAE is introduced - the result of applying SDN to IAF-VAE (Kingma et al.) 2016). with modest modifications to the original architecture. SDN-VAN VAE is evaluated in terms of: (a) density estimation, where the performance is considerably improved both upon the baseline and related approaches to non-autoregressive modeling, on CIFAR 10 and ImageNet32 data sets; (b) image synthesis, where images of competitively high perceptual quality are generated based on CelebAHQ256 data set.
* By integrating SDN into relatively simple, non-hierarchical VAE, trained on the synthetic 3D-Shapes data set, we demonstrate in another comparative analysis substantial improve- ments upon convolutional networks in terms of: (a) optimization of the evidence lower bound (ELBO); (b) learning of disentangled representations with respect to two popular metrics, when B- VAE (Higgins et al. 2016) is used as an objective function.
2

---

Published as a conference paper at ICLR 2021
Project-in stage
Correction stage
Project-out stage
spelial dependencc layer
convolutional layer
X
14
D DXIIIX +*EII*+
X
X
XXIXXXI
*X**XX* XXXXXX
(b)
(c)
Figure 2: Spatial dependency layer. (a) a 3-stage pipeline: (b) dependencies between the input
- and the output X modeled by a spatial dependency layer Lą. Solid arrows represent direct, and dashed indirect (non-neighboring) dependencies. Note that the sub-layers from Figure 1a are regarded as 'latent' here; (c) dependencies modeled by a 2 x 2 convolutional layer.
SDN ARCHITECTURE
The architectural design of SDN is motivated by our intent to improve modeling of spatial coherence and higher-order statistical correlations between pixels. By SDN, we refer to any neural network that contains least spatial dependency layer (Figure Technically speaking, each layer of an SDN is a differentiable function that transforms input to output feature maps in a 3-stage procedure.
We first describe the SDN layer in detail, then address practical and computational concerns.
Project-in stage. Using simple feature vector-wise affine transformation, the input representa- tion X the layer Lg is transformed into the intermediate representation J!,:
T L (1)) + b(t)
where i and are the corresponding 2-D coordinates in X and _ (2) is vector of learnable biases and (11) is learnable matrix of weights whose dimensions depend on the number of chan- nels of X and Thx, In the absence of overfitting, it advisable increase the number of channels to enhance the memory capacity of _ and enable unimpeded information flow in the correction stage. Notice that described transformation corresponds to the convolutional (CNN) operator with kernel of size 1. If the scales between the input X and the output X X differ (e.g. if con- verting an 8x8 to a 16x16 tensor), in this stage we additionally perform upsampling by applying the corresponding transposed CNN operator, also known as 'deconvolution' (Zeiler & Fergus 2014).
Correction stage. The elements of _ are 'corrected' in 4-step procedure. In each step, unidirectional sweep through _ is performed in one of the possible directions: (i) bottom-to-top: (ii) toght-to-left; (iii) left-to-right, and (iv) top-to-bottom. During a single sweep, the elements of the particular column/row are computed in parallel. The corresponding correction equations implement controlled updates of 4 using a gating mechanism, as done in popular recurrent units (Hochreiter Schmidhuberi 1997] Cho et al. 2014). Specifically, we use multiplicative update akin to the one found in GRUs (Cho al. 2014). but adapted the spatial setting. The procedure for the bottom- to-top sweep is given Algorithm I where W denote learnable weights, B are learnable biases, i is the height (growing bottom-to-top) j is the width and is the channel dimension. Gating elements Tij.c and zi.j.c control the correction contributions of the a priori value of It,j,c and the proposed value ni.,jes respectively. For the borderline elements of I' missing neighbors are initialized to zeros (hence the zero padding). The sweeps in the remaining 3 directions performed analogously.
Project-out stage. Corrected (a posteriori) representation l!a is then mapped to the output x Lut1, which has the same number of channels the input X :c:
x!+1= 11 La +
(3) + (3)
(2)
where (33) is the stage-level learnable weight matrix and (3) is the corresponding bias vector.
3

---

Published as a conference paper at ICLR 2021
Algorithm 1 Bottom--o-top sweep of the correction stage
Input: The intermediate representation; N the scale of Th:;
Output: + corrected intermediate representation; after all sweeps
Complexity: (NN)
_ ( done for the first direction only done for the first direction only
[ zero tto___dddin((114
for 1 = to .N and (j = to .V in parallel) do
Figl ([ Lsgle | 1gle
1,5+ B" (..e)) # reset gate a 1-1---16) \ 1-11++1 B², s..c) #update gate
W 1=-1--11)) W L.
Li..c = ( T. + W 7.1.3
Rijic = tanh(ri...c (W 1 / + [-
[W 4 1-1.41le BB i...))
Lig.e 2i.j.chusie + Ma,ee)ma feature correction
end for
Computational considerations. A valid concern in terms of scaling SDN to large images the computational complexity of O(N). N being the scale of the output. We make two key observations:
a. one can operate with less than 4 directions per layer, alternating them across layers to achieve the 'mixing" effect. This improves runtime with some sacrifice in performance;
b. in practice, we found it sufficient to apply spatial dependency layers only at lower scales (e.g. up 64x64 on CelebAHQ256). with little to no loss in performance. We specu- late that this is due to redundancies in high-resolution images, which allows modelling of relevant spatial structure lower scales;
To provide additional insights on the computational characteristics, we compared SDN to CNN in a unit test-like setting (Appendix (A.3). Our implementation of a 2-directional SDN has roughly the same number of parameters as 5 5 CNN, and is around 10 times slower than 3x3 CNN. However, we also observed that in a VAE context, the overall execution time discrepancy is (about 2-3 times) smaller, partially because SDN converges in fewer iterations (Table 4 in Appendix).
Avoiding vanishing gradients. The problem might arise when SDN layers are plugged in very deep neural network (such the one discussed in Section 44). To remedy this, in our SDN- VAE experiments we used the variant an SDN layer with a 'highway" connection, i.c. gated residual (Sri- vastava et al. 2015). as shown in Figure
XL
SDN layer
X h444
Gate
CNN layer
Figure Residual SDN layer.
3 RELATED WORK
Convolutional networks. Both spatial dependency and convolutional layers exploit locality (a feature vector is corrected based its direct neighbors) and equivariance to translation (achieved through parameter sharing). Additionally, SDN enforces spatial coherence. There are also dif- ferences in the dependency modeling (depicted in Figures 2 and ( Firstly, when conditioned on the input XL+, the output feature vectors of a convolutional layer are statistically independent:
- IL x v(i,)) # (1, m). where i and are feature map coordinates. In spatial depen- dency layers, all feature vectors are dependent: x JK. x x/ V(i.)) * (l, Secondly, the unconditional dependency neighborhood of convolutional layer is bounded by the size of its receptive field conceptual limitation not present in spatial dependency layers. Hence, spatial dependency layers can model long-range spatial dependencies.
Autoregressive models and normalizing flows. SDN design inspired by how autoregressive models capture spatial dependencies due to the pixel-by-pixel generation process (Theis & Bethge 2013). SDN(-VAE) improves on this is by modeling spatial dependencies in multiple directions. Au- toregressive models inherently uni-directional; the generation order of pixels can be changed but fixed for training and sampling. Thus some sense, SDN transfers the ideas from autoregressive to aon-auvoreeressive settings in which there are ordering constraints. Also, most autoregressive
4

---

Published as a conference paper at ICLR 2021
models use teacher forcing during training to avoid sequential computation, but sampling time com- plexity is quadratic. SDN has linear complexity in both cases and can operate at smaller scales only. Parallel computation of SDN is similar to the one found in PixeIRNN (Van Oord et al. 2016), but instantiated in the non-autoregressivee setting with no directionality or conditioning constraints.
One can also draw parallels with how autoregressive models describe normalizing flows, for exam- ple, IAF (Kingma et al.) 2016) and MAF (Papamakarios et al.] 2017). In this case, each flow in a stack of flows described with an autoregressive model that operates in different direction, or more generally, has a different output ordering. In its standard 4-directional form (Figure aa)), SDN cre- ates dependencies between all input and output feature vectors; this renders full, difficult mouulttoohhndlll Jacobian matrix. For this reason, SDN is not directly suitable for parameterizing normalizing flows in the same way. contrast, the main application domain of SDN is rather the parameterization of deterministic, unconstrained mappings.
Self-attention. The attention mechanism has been one of the most prominent models in the do- main of text generation (Vaswani et al. (017)) Recently, it has also been applied to generative image modeling (Parmar et al. 2018 Zhang et al. 2019). Both SDN and self-attention can model rangea dependencies. The key difference is how this is done. In SDN, only neighboring feature vectors are dependent on each other (see Figures | and 2 hence the spatial locality is exploited. Gated units are used to ensure that the information is propagated across large distances. On the other hand, self-attention requires an additional mechanism to incorporate positional information, but the dependencies between non-neighboring feature vectors are direct. We believe that the two models should be treated as complementary; self-attention excels in capturing long-range depen- dencies while SDN is better in enforcing spatial coherence, equivariance translation, and locality. Note finally that standard self-attention is costly, with quadratic complexity in time and space in the number of pixels, which makes O(N+) in the feature map scale, if implemented naively.
Other related work. SDNs are also notably reminiscent of Multi- Di--ensiiiinnl RNNs GGrves al. 2007) which were used for image segmentation (Graves et al. 2007 Stollenga et al. 2015) and recognition (Visin et al.) (215)), but not in the context of generative modeling. One technical advantage of our work is the linear training time complexity in the feature map scale. Another difference is that SDN uses GRU, which is less expressive than LSTM, but more memory efficient.
4 SDN-VAE: AN IMPROVED VARIATIONAL AUTOENCODER FOR IMAGES
As a first use case, SDN is used in the SDN-VAL architecture. The basis of SDN-VAE is the IAF- VAE model (Kingma et al. 2016). Apart from integrating spatial dependency layers, introduce additional modifications for improved performance, training stability, and reduced memory require- ments. We first cover the basies of VAFS, then briefly summarize IAF-VAE and the related work. Finally, we elaborate on the novelties in the SDN-VAE design.
Background on VAEs. VAES assume latent variable generative model pa (X) po (X, ZydZ where 0 are model parameters. When framed in the maximum likelihood setting, the marginal prob- ability of data is intractable due to the integral in the above expression. VAEs take a variational approach, approximating posterior go (Z|X) using a learnable function q with parameters ø. Fol- lowing Kingma Welling (2019), we can derive the following equality:
[X, Z)
log Tas (7|x
log po (X) (z)()
log
(3)
(Z(x)
Po [p(()
C. (x,)())=LBB
Ls.0(X x) is the evidence lower bound (ELBO), since it 'bounds' the marginal log-likelihood term log (X), where the gap is defined by the KL-divergence term on the right. In VAEs, both 4s and Po parametrized by deep neural networks. The model is learned via backpropagation, and a low variance approximation of the gradient V Vo.oLo.0 can be computed via the reparametrization trick (Kingma & Welling, 2019 Rezende et al. 2014).
Background on IAF-VAE. Main advancements of IAF in comparison to the vanilla VAE include: (a) leveraging on hierarchical (ladder) network of stochastic variables (Sønderby al.,
5

---

Published as a conference paper at ICLR 2021
2016) for increased expressiveness and improved training; (b) introducing inverse autoregressive flows to enrich simple isotropic Gaussian- based posterior with a normalizing flow; (c) utilizing a structured bi-directional inference procedure to encode latent variables through encoder-decoder interplay: (d) introducing a ResNet block as a residual-based (Srivastava et al. 2015 He et al. 2016) version of the ladder layer. For more details, please the original paper (Kingma et al. 2016).
Related work. BIVA (Maaoee et al. 2019) is an extension of IAF-VAE which adds skip con- nections across latent layers, and stochastic bottom-up path. In our experiments, however, were not able to improve performance a similar way. Concurrently to our work, NVAE (Vahdat & Kautz 2020) reported significant improvements upon the IAF-VAE baseline, by leveraging on many technical advancements including: batch normalization, depth-wise separable convolutions, mixed- preeddpre,siin, spectral normalization and residual parameterization of the approximate posterior. Both methods use the discretized mixture of logistics (Salimans et al. 2017)
SDN VAE ARCHITECTURE
Our main contribution to the IAF-VAE architecture is the integration of residual spatial dependency layers, ResSDN layers (Figure B) We replace the convolutional layers the top-down (gener- ation) path of the IAF-VAE ResNet block, up to a particular scale: up to 32x32 for images of that resolution, and up 64x64 for 256x256 images. For the convenience of the reader, borrowed the scheme from the original paper and clearly marked the modifications (Figure 4
Nochin Styck
sp-
Nornincorit
Figure 4: SDN-VAE ResNet block as modified IAF-VAE ResNet block. (marked in red) include applying: (a) ResSDN layers instead of convolutional layers the top- down path; (b) gated rather than a sum residual; (figure adapted from Kingma et al. (2016)
Two notable changes
Other notable modifications include: (a) gated residual instead of a sum residual in the ResNet block (Figure for improved training stability: (b) more flexible observation model based on a discretized mixture of logistics instead of discretized logistic distribution, for improved performance; and (c) prxed-precision (Micikevicius et al.) 2018), which reduced memory requirements thus allowing training with larger batches. Without ResSDN layers, we refer to this architecture IAF-VAE+
5 IMAGE DENSITY MODELING
Proposed SDN-VAE and the contributions of spatial dependency layers to its performance were em- pirically evaluated in terms of: (a) density estimation, where SDN-VAE was in quantitative fashion compared to the convolutional network baseline, and to the related state-of-the-ar approaches; (b) image synthesis, where learned distribution was analyzed in qualitative way by sampling from it in different ways. Exact details of our experiments, additional ablation studies and additional results in image synthesis are documented in the Appendix sections A.I A.2 and A.4 respectively.
Density estimation. From a set of i.idd images Dtrain = X1..N, the true probability density function p(X) is estimated via parametric model pe (X), whose parameters 0 are learned using the maximum log- ligliiil objective: arg max, [oo (X) = Σn log ps (X.) The test log- likelihood is computed isolated set of images Diest X1 X1..K, K. evaluate learned po (X).

---

Published as a conference paper at ICLR 2021
SDN- VAE and the competing methods tested CIFAR-10 10 geNet32 Van Oord et al. 2016), and CelebAHQ256 (Karras et al. 2017). Quantitative comparison
(Krizhevsky al. 2009), Ima-
given in Table IAF- VAE+ denotes technically enhanced implementation of IAF- IAF-VAE, but
without the SDN modules (recall Section 4
Type
Method
CIFAR-10
ImageNet32
CelchAHQ256
VAE-based
SDN-VAE (ours)
2.87
3.85
0.70
IAF-VAR (ours)
3.05
4.00
0.71
IAF-VAE Kingma et al.
2016
3.11
X
X
BIVA Maalee al. 2019
3.08
X
NVAE Vahdat Kautz
2020
2,91
3.92
0.70
Flow-based
GLOW Kingma & Dhariwal
2018
3.35
4.09
1.03
FLOW
Ho 2019
3.08
3.86
ANF Huang et al.
2020)
3.05
3.92
0.72
Vur (Nielsen et al.
2020
3.08
4.00
Autoregressive*
PixeIRNN
Van Oord et al.
2016
3.86
PixelCNN Van den Oord et al.
2016
3.03
3.83
X
PixeICNN+
Salimans et
2017)
2.92
PiXeISNAIL
Chen et al.
2018
2.85
3.80
X
SPN Menick Kalchbrenner
2018
0.61
IT Parmar et al.
2018
2.90
3.77
X
Table Density estimation results. Negative test log-likelihood measured bits per dimension (BPD) lower is better. As in previous works, only the most successful runs are reported. Circled are the best runs among aon-autoregressive models and bolded are the best runs overall. by autoregressive we refer to the methods based on p(X) = 1km X(x, X;-1) factorization.
Figure 5: Linear interpolation between test images. Given a pair of images from the test dataset (on the far right and far left), linear interpolation was performed the layer 5 at temperature of 0.9 (the sequence above) and on the layer at a temperature of 1.0 (the sequence below).
Image synthesis. We additionally inspected the trained pa (X) by: (a) unconditionally generating samples from the prior, at different temperatures (Figure @ left); (b) sampling in the neighborhood of image not seen during training time (Figure(rright)) (c) interpolating between two images not seen during training time (Figure All technical details on these experiments are in Appendix A. Additional images are given in Appendix A.4
Our results clearly suggest that SDN-VAB is not only superior to the convolutional network baseline, but also to all previous related approaches for non-autoregressive density model- ing. To further inspect whether the increase in performance is due to applying SDN or due to the implicit increase in the number of parameters, we conducted additional ablation studies (described in Appendix A.2 These results provide additional evidence that SDN is the cause of increased
Discussion.

---

Published as a conference paper at ICLR 2021
Temp=0.9
Temp=0.8 8
Temp=0.7
TopLayersFixed=2 Temp=1.0
Figure 6: (left) Unconditional sampling. Samples were drawn from the SDN-VAE VAE prior varying temperatures; (right) Sampling around test image. Conditioned on the top 2 layers of the latent code of the image in the center, the surrounding images were sampled a temperature of 1.0.
performance, as even convolutional networks with more parameters, increased receptive field or more depth lead to inferior performance. In image generation experiments confirmed that SDN- can be successfully scaled to large images. Generated samples are exceptionally high quality. What is particularly challenging in VAE-based image generation, is sampling at high tem- peratures, however as shown in Figure 10 of our Appendix, SDN-VAE can successfully synthesize coherent images even in this case.
LEARNING DISENTANGLED REPRESENTATIONS
To study its performance impact in a more constrained setting, SDN was paired with a VAE architec- turally much simpler than IAF-VAE. Apart from the implementation simplicity and shorter training time, non- hierarchical VAE is more suitable for disentangled representation learning, at least in the sense of (Higgins et al. 2016) where the aim is to decorrelate the dimensions of a latent vector. In particular, the gains in performance when using SDN were evaluated with respect to: (a) evidence lower bound (ELBO); (b) disentanglement of latent codes based on the corresponding metries, to examine the effeets of SDN decoder to the quality of learned latent representations.
Experiment setting. A relatively simple VAE architecture with stack of convolutional layers in both the encoder and decoder (Higgins et al. 2016) was used as a baseline model. SDN decoder was constructed using single non-residual, direectctio instance of spatial dependency layer placed at the scale 32. The competing methods were evaluated on 3D-Shapes (Burgess & Kim) 2018), a synthetic dataset containing 64 64 images of scenes with rooms and objects of various colors and shapes. Following related literature (Locatello et al.) 2001)), there was no training-test split, so the reported ELBOSs are measured on the training set. The remaining details are in Appendix
A.5 A.5
Training with an unmodified VAE objective. Using the original VAE objective (Kingma & Welling 2013), competing architectures were
VAE
KLD [BPDx10
Neg. ELBO [BPD]
compared in terms of ELBO (Table 2. SDN leads to substantially better estimates which come from better approximation of the condi- tional log-likelihood term, as shown in more detail in Figure 12 in the Appendix A.6
CNN
7.40 0.07
4.44 + 0.04
SDN
7.95 0.07
2.66 + 0.04
Table 2: Density estimation on 3D-Shapes.
8

---

Published as a conference paper at ICLR 2021
Training with 3- VAE objective. The same models were trained using 3-VAE objective (Higgins et al., 2016). In the 3-VAE algorithm, the disentanglement of latent dimensions is controlled by an additional hyperparameter denoted 3. which effectively acts regularizer. Modified ELBO X 3) (2)())))) x |log po (X - - (l Z X)llpo(Z)) Z)). To investigate the effects of SDN decoder the shaping of the latent space, measured disentanglement for different values of B. for two popular disentanglement metries: 3-VAE (Higgins et al. 2016) and FactorVAE
reads as: L
(Kim
& Mnih 2018). The results are shown in Figure 7
1.0
6.7
Requisroer
Figure 7: 3-VAE disentanglement results. The plots compare the CNN and bDN-based VAE architectures, with respect to the B-VAE (left) and FactorVAE (right) disentanglement metrics.
Discussion. In a more constrained VAE setting, the beneficial effects of SDN to image modeling were confirmed. Perhaps surprisingly at first glance, also found that improved VAE decoder facilitates factorization of latent codes. This may seem contradictory to the previous findings (Bow-) man al. 2015) Chen al., 2016) which suggest that powerful decoder hurts representation learning. However, the difference here is that SDN decoders do not leverage on teacher forcing. Posteriorr collapse' can happen only very early in the training, but this in our case easily solved using 3 annealing (warm-up) procedure (Bowman et al. 2015). Our results indicate the importance of good neural architectures learning disentangled representations.
7 CONCLUSIONS
Summary. This paper introduced novel spatial dependency layers suitable for deep neural networks that produce images image generators. The proposed SDN is tailored to image generators that operate in a non-autoregressive way, i.c. synthesize all pixels "at once'. Spatial dependency layers improve upon convolutional layers by modeling spatial coherence and long-range spatial dependen- cies. The main motivation behind introducing SDN is its application to VAE image decoders. SDN was analyzed in the context of: (a) complex hierarchical VAE model, where the of -f--th--rtttt performance was obtained in non-autoregressive density modeling; (b) a vanilla VAE, resulting in improvements in both density modeling and disentangled representation learning.
Implications. Spatial dependency layer is a simple-to-use module that can be easily integrated into any deep neural network, and as demonstrated the paper, it favoralle a convolutional layer in multiple settings. SDN was shown to improve the performance of VAEs immensely, which is rele- vant since VAES can be used in any density modeling task (unlike generative adversarial networks), e.g. outlier detection. VAEs are also favorable to autoregressive models in the settings where an explicit latent representation data is needed, e.g. when the latent interpolation between test sam- ples is necessary. We also provide an insight how VAE decoders can be changed for improved representation learning, suggesting an alternative way, and a concrete solution for this problem.
Limitations and future work. The main downside of SDN remains the computation time. However, we suspect that a more optimized implementation could substantially improve the runtime perfor- mance of SDN. In the future, it would be beneficial to explore the applicability of SDN in other settings. For example, one could apply SDN to other VAE architectures or to generative adversarial networks. SDN could also be applied image processing tasks such image super-resolution or image segmentation. It would also be interesting to explore the applicability of SDN for learning structured and disentangled representations of video sequences (Miladinović et al. 2019).

---

Published as a conference paper at ICLR 2021
ACKNOWLEDGMENTS
We thank Max Paulus and Mihajlo Milenković for helpful comments and fruitful discussions. This research was supported by the Swiss National Science Foundation grant 407540_ 167278 EVAC Employing Video Analytics for Crisis Management, and the Swiss National Science Foundation grant 200021 165675/1 |00021_165675/ (successor project: no: 200021 200021_192356)
REFERENCES
Suzanna Becker and Geoffrey E Hinton. Self-organizing neural network that discovers surfaces in random-dot stereograms. Nature, 355(6356):661-163, 1992.
Yoshua Bengio, Aaron Courville, and Paseal Vincent. Representation learning: A review and new perspectives. IEEE transactions pattern analysis and machine intelligence, 35(8):1798-188) 2013.
Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew M Dai, Rafal Jozefowicz, and Samy Ben- gio. Generating sentences from continuous space. arXiv preprint arXiv:/511.06349, 2015..
Yuri Burda, Roger Grosse, and Ruslan Salakhutdinov. Importance weighted autoencoders. arXiv preprint arXiv:1509.000 2015.
Chris Burgess and Hyunjik Kim. 3d shapes dataset. https///sttthbcooo////mmm//////s 2018.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.05700 0550000020077,,... 2020.
Xi Chen, Diederik P Kingma, Tim Salimans, Yan Duan, Prafulla Dhariwal, John Schulman, Ilya Sutskever, and Pieter Abbeel. Variational lossy autoencoder. arXiv preprint 16Xi::6611.00220 2016.
Xi Chen, Nikhil Mishra, Mostafa Rohaninejad, and Pieter Abbeel. Pixelsnail: An improved toregressive generative model. In International Conference on Machine Learning, pp. 864-872. PMLR, 2018.
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre. Dzmitry Bahdanau, Fethi Bougares, Hol- ger Schwenk, and Yoshua Bengio. Learning phrase representations using encoder-decoder for statistical machine translation. arXiv preprint arXiv: /406. 1078, 2014.
Dan Cireşan, Ueli Meier, and Jürgen Schmidhuber. Multi-column deep neural networks for image classification. In 2012 IEEE conference on computer vision and pattern recognition, pp. 3642- 3649. IEEE, 2012.
Kunihiko Fukushima and Sei Miyake. Neocognitron: A self-orgamizing neural network model for mechanism of visual pattern recognition. In Competition and cooperation in neural nets, pp. 267-285. Springer, 1982.
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems, pp. 2672-2680, 2014.
Alex Graves, Santiago Fernández, and Jürgen Schmidhuber. Multi-dimennional recurrent neural networks. International conference on artificial neural networks, pp. 549-558. Springer, 2007.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. 2016.
10

---

Published as a conference paper at ICLR 2021
Jonathan Ho, Xi Chen, Aravind Srinivas, Yan Duan, and Pieter Abbeel. Flow++: Improving flow- based generative models with variational dequantization and architecture design. arXiv preprint arXiv:1902.00025, 2019.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):
1735-1780, 1997.
Chin- Wei Huang, Laurent Dinh, and Aaron Courville. Augmented normalizing flows: Bridging the gap between generative flows and latent variable models. arXiv preprint arXiv:2002.07100, 2020.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for im- proved quality, stability, and variation. arXiv preprint arXiv: 10196, 2017.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. volume 80 of Proceedings of Machine Learning Research, pp. 2649-2658, Stockholmsmässan, Stockholm Sweden, 10-15 Jul 2018. PMLR. htLhttp: htttt:://proeeedinng: mir http:/nww..r....rss.eeeeeeeeeeee ptw.s//80/eiii1l///.0 html
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint
arXiv:1412.6980 2014.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 2013.
Diederik Kingma and Max Welling. An introduction to variational autoencoders, arXiv preprint arXiv:1906.000 02691, 2019.
Durk P Kingma and Prafulla Dhariwal. Glow:
Generative flow with invertible 1x1 convolutions. In Advances in neural information processing systems, pp. 10215-10224, 2018.
Durk P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. Im- proved variational inference with inverse autoregressive flow. In Advances in neural information processing systems, pp. 4743 2016
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. Advances in neural information processing systems, pp. 1097-1105, 2012.
Alex Krizhevsky et Learning multiple layers of features from tiny images. 2009.
Yann LeCun, Berhard Boser, John S Denker, Donnie Henderson, Richard E. Howard, Wayne Hub- bard, and Lawrence Jackel. Backpropagation applied handwritten zip code recognition. Neural computation, 1(4):541-551, 1989.
Francesco Locatello, Stefan Bauer, Mario Lucie, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. international conference on machine learning, pp. 4114-4124, 2019.
Lars Maaløe, Marco Fraccaro, Valentin Liévin, and Ole Winther. Biva: A very deep hierarchy of latent variables for generative modeling. In Advances neural information processing systems, pp. 6551-6563 2019.
Jacob Menick and Nal Kalchbrenner. Generating high fidelity images with subscale pixel networks and multidimensional upscaling. arXiv preprint arXiv:18/2.011 2018
Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al. Mixed precision training. In International Conference on Learning Representations, 2018.
Tomáš Mikolov, Stefan Kombrink, Lukáš Burget, Jan Černocký, and Sanjeev Khudanpur. Exten- sions of recurrent neural network language model. In 2011 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 5528-5531. IEEE, 2011.

---

Published as a conference paper at ICLR 2021
Dorde Miladinovié, Muhammad Waleed Gondal, Bernhard Schölkopf, Joachim M Buhmann, and Stefan Bauer. Disentangled state space representations. arXiv preprint arXiv:1906.03255, 2019.
Didrik Nielsen, Priyank Jaini, Emiel Hoogeboom, Ole Winther, and Max Welling. Survae flows: Surjections to bridge the gap between vaes and flows. arXiv preprint arXiv:2007.02731, 2020.
George Papamakarios, Theo Pavlakou, and lain Murray. Masked autoregressive flow for density estimation. In Advances in Neural Information Processing Systems, pp. 2338-2347, 2017.
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. arXiv preprint arXiv: arXiv: 1802.0571,, 2018.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. arXiv preprini 4rX1v21401.4082, 2014.
Tim Salimans and Durk P Kingma. Weight normalization: A simple reparameterizatioon to accelerate training of deep neural networks. In Advances neural information processing systems, pp. 901- 909, 2016.
Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P Kingma. Pixelcnn++: Improving the pixelenn with discretized logistic mixture likelihood and other modifications. arXiv preprint arXiv: 17, 2017.
Jürgen Schmidhuber. Learning factorial codes by predictability minimization. Neural computation, 4(6):863-879, 1992.
Casper Kaae Sønderby, Tapani Raiko, Lars Maaløe, Søren Kaae Sønderby, and Ole Winther. Ladder variational autoencoders. In Advances in neural information processing systems, pp. 3738-374844 2016.
Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. Highway networks. arXiv preprint arXiv:1505.00338 2015.
Marijn F Stollenga, Wonmin Byeon, Marcus Liwicki, and Juergen Schmidhuber. Parallel multi- dimensional Istm, with application to fast biomedical volumetric image segmentation. In Ad- vances neural information processing systems, pp. 2998-3006, 2015.
Lucas Theis and Matthias Bethge. Generative image modeling using spatial Istms. In Advances in Neural Information Processing Systems, pp. 1927-1935, 2015.
Arash Vahdat and Jan Kautz. Nvae: A deep hierarchical variational autoencoder. arXiv preprint arXiv:2007.03898, 2020.
Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al. Con- ditional image generation with pixelenn decoders. In Advances in neural information processing systems, pp. 4790-4798, 2016.
Aaron Van Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. In International Conference on Machine Learning, pp. 1747-1756, 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Francesco Visin, Kyle Kastner, Kyunghyun Cho, Matteo Matteucci, Aaron Courville, and Yoshua Bengio. Renet: recurrent neural network based alternative to convolutional networks. arXiv preprint / 003903 2015.
Alex Waibel. Phoneme recognition using time-delay neural networks. Meeting of IEICE, Tokyo, Japan, 1987.
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference on computer vision, pp. 818-833. Springer, 2014.
Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial networks. In International conference on machine learning, pp. 7354-7363. PMLR, 2019.
12

---

Published as a conference paper at ICLR 2021
A APPENDIX
A.1 DENSITY ESTIMATION EXPERIMENT DETAILS
CIFARIO
ImageNet32
CelebAHQ256
32
32x32
32
256
# training samples
50K
1.281M
27K
test samples
10K
50K
3K
Quantization
bits
bits
bits
Encoder/decoder depth
15
16
17
Deterministic #channels per layer
200
200
200
Stochastic #channels per layer Scales
4
4
8
82, 1211,,22 322
4², 82, 162, 323 322
8,,8? 16 32",, 641 128", 256²
SDN #channels per scale
120, 240, 260
424 for all
360, 360, 360, 360, 360,
Layers per scale
5 for all
4 for all 3 Adamax
2,2,2,2,2,3,4
Number directions per SDN
2 Adamax
Optimizer
Adamax 0.001 Exponential
Learning
0.002
0.002
Learning rate annealing
Exponential
Exponential
Batch size per GPU
32
32
Number of GPUs
8
8
8
GPU type
T100
Te0
110
GPU memory
32GB
32GB
32GB
Prior model
Gaussian
Gaussian
Gaussian
Posterior model
Gaussian
Gaussian
Gaussian
Posterior flow
IAF
IAF
IAF
Observation model
DML
DML
DML
DML Mixture components
10
30
Exponential Moving Average (EMA)
0.999
0.9995
0.999
Free bits
0.01
0.01
0.01
Number of importance samples
1024
1024
1024
Mixed-precision
Yes
Yes
Yes
Weight normalization
Yes
Yes
Yes
Horizontal flip data augmentation
Yes
No
Yes
Training time
45h
200h
90h
Table 3: Experimental configurations for the density estimation tests. DML the discretized
mixture of logistics (Salimans et al. 2017). Weight normalization, mixed-precision, free bits and Adamax are documented by Salimans & Kingmal (2016); Micikevicius et al. (2018); (ingmaa et al. (2016); Kingma & Ba (214)) respectively. SDN means that SDN was not applied the correspond- ing scale. The IAF (Kingma et al., (2066) contained layers of masked 3x3 convolutional networks, with the context and number of CNN filters both of size 100. The importance sampling (Burda et al. 2015) was used for obtaining tighter lower bounds.
13

---

Published as a conference paper at ICLR 2021
On configuring hyperparameters. Due to high computational demands, we only modestly opti- mized hyperparameters. In principle, our strategy was to scale up: (a) the number of deterministic and SDN filters; (b) batch size; (c) learning rate; (d) the number of spatial directions and (c) the number of DML components; as long as we found no signs of overfitring, had no memory or sta- bility issues, and the training was not considerably slower. EMA coefficient values were taken from the previous related works (Kingma et al. 2016 Chen et al. (0018) we tested 0.999 and 0.9995. We also swept through the values {2, 4, 8, 15) for the number of latent stochastic filters, but saw no significant difference in the performance. Most extensively we explored the number of layers per scale, and found this to have relevant impact on runtime and over/underfitting for both baseline and our architecture. We found that more downsampling improved runtime and reduced memory consumption, but made the tested models prone to overfitting.
A.2 DENSITY ESTIMATION ABLATION STUDIES
Additional ablation studies EDN-VAE were conducted in order to explore in more detail whether the good performance on the density estimation experiments (Table indeed comes from the pro- posed SDN layer. We used CIFAR-10 data on which both baseline and SDN-VAE models converged faster in comparison to two other data sets. Our main motivation was to understand whether the in- crease in performance comes from our architectural decisions or simply from the increase in the number of parameters. Namely, since the ResSDN layer (from Figure 3) is a network with larger number of parameters in comparison to the baseline 3x3 convolutional layer, would be justified to question if the increase in performance can be indeed attributed to ResSDN layers.
To that end, we designed multiple baseline blocks to replace convolutional layers in the AAB+ architecture, by replicating the design protocol from Section in which ResSDN layer was applied to create the SDN-VAL architecture. Baseline blocks are illustrated in Figure g The main idea is to explore whether possibly increased kernel of CNN would be sufficient to model spatial dependen- cies, or whether deeper network can bring the same performance to the basic VAE architecture.
3x3CNN
5X5CNN
7x7 CNN
3x3 CNN
2dIr-SDN
(B2)
(CNN)
Gate
3x3CNN
5x5CNN
3X3CNN
3X3CNN
3x3CNN
3x3CNN
Gate
Gate
(ResSDN)
3x3CNN
3X3CNN
(B1)
(B3)
Figure 8: ResSDN layer and the baseline blocks.
All baseline blocks were trained in the same setting as SDN-VAE i.c. all the parameters were kept fixed, except from the learning rate, which we halved after every 10 unsuccessful attempts to stabilize the training. IAF-VAE based on the CNN block was stable at the default learning rate of 0.002, B2 was stable at the learning rate of 0.001, while B4 and B1 were stable only at 0.0005. B2 was unstable. We also tested ResSDN at a learning rate of 0.001 to understand if there would be a drop in performance, but there wasn't any except from marginally slower convergence time.
The best runs in terms of negative evidence lower bound (ELBO) for each of the baseline blocks, along with our most successful run for SDN-VAE architecture are reported in Table What we can observe is that the increase in capacity can indeed be correlated with good performance. However, even those VAE models which contained more parameters than the proposed SDN- VAE architecture were not able to converge to a higher negative ELBO. In fact, there exist a considerable performance gap between baseline blocks and the proposed ResSDN layer.
14

---

Published as a conference paper at ICLR 2021
Layer
Number of VAE parameters
Time converge
Best Neg. ELBO in BPD
in millions
in hours
in iterations
Baselines
CNN
42M
17h
140K
3.081
B1
192M
21h
75K
3,080
B2
104M
unstable
B3
126M
23h
90K
2.945
B4
130M
22h
124K
3.120
Ours
ResSDN
121M
45h
60K
2.906
Table 4: The comparison ResSDN layer the baselines from Figure We replaced CNN
layers in the IAF-VAE+ architecture (from Section with baseline blocks and compared the density estimation performance in terms of ELBO. The experiments were conducted on CIFAR-10 dataset.
A.3 ADDITIONAL SDN ANALYSIS
Number of parameters. For a filter size of 200 (a value used in the SDN-VAE experiments), compare the number parameters between CNN and SDN layers. Note that the input scale does not have any effect the resulting numbers. The numbers the directions in SDN. PPooeect phase' denotes the size of project-in and project-out SDN sub-layers which are in this experiment of the size, since upsampling is performed. We observe that the 2-directional SDN is approximately of the same size as 5x5CNN in terms of number of free parameters.
given in Table Here 'dir' denotes
3x3CNN
5x5CNN
Project phase
SDN cell
Idir-SDN
2dir-SDN
parameters
360200
1000200
40200
481200
561600
1042800
Table 5: Number of parameters of different neural layers.
Runtime unit tests. We measured the execution times of forward propagation of CNN and SDN layers, for different input scales. To obtain standard deviation estimates, each forward propagation was repeated 100 times. The batch size SDN layer is considerably slower than the CNN layer, but in the reasonable limits. Note that more efficient implementation will likely improve SDN runtime performance.
128. The results given in Table The
Input scale Layer
3x3CNN
5x5CNN
Idir-SDN
2dir-SDN
0.3210.01
1.26:10.11
1.9310.06
3.52+0.07
0.63+0001
3.25
4.09+00.06
7.61
16
2.33+00.88
11.2+0.08
11.4+0.03
21.0++005
32
9.18
20.4:0.18
45.4
83.8
Table 6: Runtime unit tests. The execution time forward propagation in ms.
15

---

Published as a conference paper at ICLR 2021
A.4
IMAGE SYNTHESIS ADDITIONAL RESULTS
LASS
air
E
O
A
MM
Figure 9: MSE-based nearest neighbors. On the left, shown images from Figure 6 On the right, shown are nearest neighbors in terms mean squarred error (MSE), found in the training data set. There are signs that generated samples replicas of the training ones.
16

---

Published as a conference paper at ICLR 2021
Temp=1.0 1.0
Temp=0.9
Temp=0.8
Temp=0.7
Figure 10: Additional samples synthesized at different temperatures. As we decrease the temper- ature, getting closer to the mean of the prior, the photographs become smoother i.e. more 'generic".
17

---

Published as a conference paper at ICLR 2021
TopL Fi TonLaversFike
3
FR
2
2
6
25
25
TopLayersFixed
Temp=0.8
Temp=0.9
Temp=1
Figure 11: Sampling around test image, for varying temperatures and number of fixed layers. Presented is 3x3 grid. In each grid there is a 3x3 sub-grid in which an image in the center is encoded, and the surrounding images are sampled conditioned on the specified number of layers from the encoded image and at the specified temperature. The text on the left hand side denotes the number of layers taken from an encoded image to condition sampling on. The text on the bottom denotes the temperature at which samples were drawn. Two key observations can be made: (a) lowering the temperature decreases the level of details, making photographs smoother; (b) Most of the information in SDN- VAE encoded in the topmost layer. As we go further down the chain the generator network, there is gradual decrease in information content in latent stochastic variables.
18

---

Published as a conference paper at ICLR 2021
A.5
LEARNING DISENTANGLED REPRESENTATIONS EXPERIMENT DETAILS
The hyperparameter configuration given in Table 7 principle, we followed Higgins et al. 2016;; Locatello et al. (2019) in configuring parameters. Notable changes include using of di- cretized logisties (Kingma et al. 2016) as the observation model and the Adamax optimizer. We also applied 3-VAE annealing avoid posterior collapse early in the training, considered architectures. The base architecture was again taken from the same previous works. The exact description of architectures is given in Table g for the CNN-based VAE, and in Table 9 for the 3DN-based VAE.
issue for both
3D Shapes
6-12
64
# data samples
448K
SDN #channels
200
Number of directions per SDN
Optimizer
Adamax
Learning rate
0.001
Batch size
128
Total number of training iterations
200k
Prior and posterior models
Gaussian
Observation model
Discretized Logistics
Table 7: Configuration of disentangled representation learning experiments.
Encoder
Decoder
conv, 32 ReLUU stride 2 FC, 256 ReLU
4 x conv, 32 ReLU, stride 2
FC, 4 x 4 x 256 ReLU
X conv, 64 ReLU, stride 2
4 upconv, 64 ReLU, stride
x conv, 64 ReLU, stride 2
4 x upconv, 32 ReLU, stride
FC 256
upconv, 32 ReLU, stride
FC 2 x 10
4 x 4 upconv, 3 (numbor of channels), stride
Table 8: CNN-based vanilla VAE architecture.
Encoder
Decoder
x conv, 32 ReLU, stride
FC, ReLU
4 x conv, 32 ReLU, stride 2 X conv, ReLU, stride
FC, 4 x 4 x 256 ReLU
upconv, 64 ReLU, stride
4 x conv, 64 ReLU, stride 2
4 x 4 upconv. 32 ReLU, stride
256
Idir-SDN with channels
FC 2 x 10
4 x 4 upconv, 3 (number of channels), stride
Table SDN-based vanilla VAE architecture.
19

---

Published as a conference paper at ICLR 2021
A.6 LEARNING DISENTANGLED REPRESENTATIONS ADDITIONAL RESULTS
For the sake of completeness, we also provide learning curves for varying values of and for different individual terms of 3-VAE objective function. The plots shown Figure 12
(a)
(b)
(c)
(d)
Figure 12: 3-VAE learning curves for CNN and SDN-based VAE architectures. Presented are the following quantities measured over the course of training: (a) evidence lower bound (ELBO); (b) conditional log-likelihood (reconstruction) term; (c) KL divergence term; (d) 3-scaled KL diver- gence term; Top-to-bottom, the rows related VAE architecture) is trained for 10 different random seeds. Linear 3 3-annealing procedure was per- formed, where 3 was increased from 0 to final value across the span of 100K training iterations (the end of the annealing procedure is denoted by the vertical line in the plots of the column (c)).
{1, 32, 100, 300, 500). Each pair (3 value,
20


Reviews:
[{"review_id": "dPmXNgExjbz", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary**: This paper proposes a network architecture, spatial dependency network (SDN), that attempts to more explicitly model spatial dependencies, as compared with convolutional networks. The network involves three steps: projection (via 1x1 CNN), a spatially (by row/column) autoregressive transform applied in each direction, and another projection. The authors use the architecture in place of CNN layers in the IAF VAE architecture. However, note that this is in the conditional mappings and *not* the flows. The model achieves state-of-the-art performance, as compared with non-autoregressive models. The authors also demonstrate improved disentanglement on a single-level VAE.\n\n**Strong Points**: The technical aspects of the approach are generally well-described and clear. I found the diagrams and algorithm box helpful in understanding SDNs. Likewise, the adaptation of the diagram from Kingma et al., 2016 was helpful to distinguish which components of the architecture were modified in SDN-VAE over the IAF architecture. The basics of variational inference and VAEs are described concisely and correctly.\n\nWhile I did not find the experimental results entirely unsurprising (see below), the experiments do appear to be rigorous. The authors demonstrate their network within a relevant architecture (IAF) on multiple benchmark image datasets. Results are compared with relevant recent approaches, where SDN-VAE outperforms non-autoregressive models. While this might be attributed to an increased number of parameters, the authors also conduct an ablation study to demonstrate that SDNs outperform much larger CNN architectures. This suggests that SDNs truly provide a useful inductive bias toward modeling dependencies between dimensions, analogous to how ResNet architectures typically outperform deeper non-ResNet architectures.\n\nThe results on latent factorization are also rigorous. The authors evaluate performance with multiple $\\beta$ values and report latent factorization using two metrics. Results are reported using 10 random seeds, helping to confirm that the improvement in disentanglement is significant.\n\nThe results generally seem to be reproducible, as algorithmic and architectural details are present in the main paper and the appendix. The main results (SDN-VAE) are demonstrated using an existing model architecture (IAF), so the majority of other modeling design choices are already publicly available.\n\nThe paper is formatted well, with clear titles for sections, figures, tables, etc. This will be helpful for unfamiliar readers.\n\nThe approach is not entirely novel (see below), but admittedly, many current VAEs exclusively use convolutions in the generative and inference models. This paper, while also using convolutions, introduces autoregressive transforms throughout the conditional mappings. This is potentially a worthwhile contribution, moving the generative modeling community toward other forms of functions (beyond feedforward and convolutional) in the mappings.\n\n**Weak Points**: While I agree with the approach, I felt as though it was poorly motivated. The authors attempt to motivate SDN by discussing the importance of modeling spatial dependencies between pixels in images, often appealing to notions of “coherence.”  For instance, the authors claim that SDNs “explicitly model image-level spatial coherence and dependencies.” However, from what I can tell, coherence is never defined. I’m assuming that the authors are referring to modeling the full joint distribution over dimensions, rather than assuming independence between dimensions. (Note that VAEs can already model such dependencies through their latent variables.) The closest that the authors come to explicitly motivating the difference from previous approaches is in the “Comparison to CNNs” in Section 2. However, this discussion incorrectly casts the network input/output as random variables, when they are, in fact, parameterized by deterministic functions.\n\nThe approach is somewhat lacking in novelty. Admittedly, the particular architecture itself is novel, and it is likely a useful improvement. However, it belongs to the larger class of neural autoregressive density estimators (NADE) and autoregressive models. Effectively, this is replacing a direct mapping with an autoregressive mapping, which has been used before, e.g. PixelCNN decoders in VAEs. The authors attempt to distance their work from autoregressive models in Section 2, saying that these are “conceptually very different,” yet autoregressive models can also contain multiple orderings of dependencies. This is the idea behind permutation/reversal operations found in autoregressive normalizing flows (which generalize Gaussian autoregressive models).\n\nGiven the similarity with previous autoregressive modeling approaches, I found it surprising that the authors chose to parameterize the conditional mappings, rather than the distributions, with SDNs. This is particularly surprising given that the authors chose the IAF architecture as their baseline, which includes autoregressive normalizing flows at each level of latent variables. While it’s unclear which form of model should be preferred, it seems like an obvious choice to at least evaluate using SDNs to parameterize the normalizing flows (e.g. IAF used MADE to parameterize the autoregressive flows).\n\nMore generally, I found it surprising that the authors chose one particular model class to demonstrate their network architecture. Convolutional networks are used widely in many classes of generative models, as well as discriminative models. If SDNs are a swap-in replacement for CNNs that are truly better at capturing spatial dependencies, then this should be apparent across multiple settings, not just latent variable models (VAEs). Experiments across multiple tasks and model families would help to improve the impact of the paper.\n\nThe main experimental results of the paper are 1) improved generative modeling on image datasets, and 2) improved factorization in single-level models. Given that SDNs are more powerful than CNNs, neither of these results are unexpected. Assuming one can avoid local optima, an improved conditional mapping should yield better density estimates, as well as map a standard Gaussian to data estimates (factorization). I do not expect every experimental result in a paper to be surprising, but it’s unclear what we learn from the results in this paper (other than the fact that SDNs work in practice).\n\n\n**Accept / Reject**: I found this paper somewhat difficult to assess. While the SDN-VAE model achieves state-of-the-art log-likelihood results on multiple image datasets, the method itself is somewhat lacking in novelty and does not seem to appreciably improve our understanding of these models. Autoregressive transforms are already heavily utilized in generative modeling, so it is generally unsurprising that including more autoregressive computations in the model will improve performance. Similarly, it seems fairly intuitively obvious that an improved conditional generative mapping should improve disentanglement, as it can more easily warp a (disentangled) Gaussian random variable into the data distribution.\n\nFurther, the authors present SDNs as a swap-in replacement for CNNs, however, experiments are only performed in one setting: explicit latent variable models of images. Spatial dependencies are relevant for any image dataset, and other data modalities more broadly. This paper would benefit from exploring these other settings (other generative models and discriminative models). Similarly, I would have liked to see the authors use their autoregressive transform within normalizing flows. This seems like a natural use-case.\n\nThe above points aside, developing improved network architectures is still a worthwhile endeavor. I’m hopeful that SDNs could become a standard network layer within the image modeling community. More broadly, it may help researchers to consider spatial dependencies more explicitly when constructing network architectures. For these reasons, I am slightly in favor of accepting this paper.\n\n**Additional Feedback**:\n\nAbstract:  \nThe first sentence is grammatically incorrect. It is missing a subject.\n\nIntroduction:  \nIt would help to clarify what is meant by the word “coherent.”  \nSimilarly, it would help to briefly describe why CNNs (supposedly) cannot explicitly model spatial dependencies and coherence.  \nFigure 1: should label the convolutional filters. Also, I understand that this is a generic overview of SDN-VAE, but it is somewhat too simplistic to be useful.  \n\nSDN Architecture:  \nI found some of the notation confusing. For example, if $\\mathbf{B}^{(1)}$ is a matrix, then why is being added to a vector in Eq. 1?  \nThe tanh on the input (in Algorithm 1) could be explained further. Is this for stability?  \nIt would be helpful to discuss an order-of-magnitude comparison of the number of parameters between SDNs and CNNs.  \nThe gated residual is referred to as a “highway” connection.  \nIt seems inaccurate to express the output of SDNs as a probability (though I understand the point).  \nIn related work, I would discuss (autoregressive affine) normalizing flows, e.g. IAF, MAF, etc., and autoregressive neural networks, e.g. NADE, MADE, EoNADE, etc. These seem highly related.  \n\nSDN-VAE:  \nShould also cite Rezende et al., 2014 for the reparameterization trick.  \nShould mention that the approximate posterior in top-down inference is structured, i.e. it models dependencies between levels of latent variables.  \n\nImage Density Modeling:  \nI would not necessarily consider density estimation and image synthesis as different “tasks.”  \nIn the maximum likelihood objective, missing a $1 / N$ factor.  \n\nLearning Disentangled Representations:  \nI would disagree that non-hierarchical models are more suitable for representation learning. Hierarchical models generally extract increasingly abstract features, which can be quite useful.  \n\nConclusions:  \nSomewhat misleading to compare SDN-VAE with GANs. There is nothing intrinsic to SDNs that allows them to calculate explicit density estimates. Indeed, SDNs could also be used in GANs.  ", "extra_scores": {}, "extra_text": {}}, {"review_id": "XpIKiZ71eE7", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "1. Summary. This work is in line with the latest attempts[1,2] to improve the VAE quality by employing more powerful decoding architectures. The authors propose a new block that handles (non-local) spatial dependence of pixels at the expense of increased computation time - O(scale). The presented experiments demonstrate that usage of the suggested layer is beneficial for the tasks of density estimation and image generation.\n1. Decision. The described idea is novel, clear, and supported by experiment results. To my mind, the provided evaluation is thorough enough, the ablation study is quite comprehensive. Therefore, I tend to vote for acceptance. However, major issues of this approach are the increased number of parameters and computational time. This can make the applicability to other datasets limited.\n1. Questions. \n    1. Is it sufficient that the sweep from Algorithm 1 is GRU-like, or, more generally, gated? Could it be replaced with a masked-convolutional sweep with the output depending only on the \"past\" (row number <= i) and the pixel (i, j)'s value itself?\n    1. Am I right you employed Monte-Carlo integration to provide NLL in Table 1 for VAE-based models? How many samples did you use?\n1. Remarks. I believe the headings of Tables 2 & 4 are a bit misleading: ELBO is a lower bound (the higher score - the better), however in your notation the less - the better.\n\n[1] https://arxiv.org/abs/2007.03898\n[2] https://openreview.net/forum?id=RLRXCV6DbEJ\n\n#############\n\nAfter reading the rebuttal I confirm the initial rating.", "extra_scores": {}, "extra_text": {}}, {"review_id": "QerydVQyh5d", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: The authors propose a new building block called Spatial Dependency Networks (SDN) that increases the expressiveness of a CNN at the cost of some sequential computation. SDN propagates information across a 2D feature map using LSTM-like updates across the rows (or columns). They use SDN blocks to improve the decoder architecture of IAF-VAE [1], a SOTA VAE-based method, and show a substantial performance boost on generative model tasks. Their method performs much better than other VAE methods, and almost as well as SOTA autoregressive models.\n\nStrengths:\n\nSolid empirical results & thorough experiments\nPaper is mostly well-written & many experimental details are included.\nWeaknesses:\n\nThe motivation seems a bit fuzzy to me: the primary contribution of this work seems to be the SDN building block, but it seems like SDN isn't necessarily limited to generative modeling? The authors clearly discuss the advantages of SDN over CNN, but the experiments focus on generative modeling VAEs, but not any of the other tasks where CNNs are used & more expressive architectures might be helpful.\nThe discussion of related work is quite limited, in particular with respect to autoregressive models. The SDN block is quite reminiscent of the Row LSTM from PixelRNN [2], a pioneering autoregressive model. Autoregressive models have been known to perform the best density estimation (compared to other families of likelihood-based generative models), at the expense of slow sampling speed (since each element must be sampled sequentially). Since SDN introduces some sequential dependencies, it trades off computation during sampling time vs during likelihood evaluation (autoregressive models are slow at the former but fast at the latter; the proposed method is somewhere in the middle but the same speed for both). I would like to see some more discussion along these directions to better place the authors' method into context with related work.\nOn a similar note, the authors allude to the extra computational overhead incurred by adding SDN to VAEs, but I don't think they explicitly state what that is? This would also help frame the contributions of their work.\nNit: the description of SDN in Algorithm 1 is a bit confusing -- it doesn't seem to convey the sequential nature of SDN? From reading Algorithm 1 only, it sounds like SDN could be implemented via masked convolutions + gating, but this seems to disagree with the rest of the paper? Also, the output variable I_+ doesn't seem to be used?\nOverall, I would slightly lean towards acceptance (conditioned on improving the discussion around the points from above), but am curious to see what the other reviewers think.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "GD34gCnp8Sg", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a new neural network layer architecture (spatial dependency network - SDN) that can be used in place or in conjunction with traditional convolutional layers. The SDN layers' main difference to convolutional layers is that in SDN layers extracted features are dependent and influenced by nearby features. SDNs are evaluated with VAEs on density modeling and learning of disentangled representation and show better performance than purely convolutional baseline architectures.\n\nStrengths:\n- The new layer architecture can be used as a drop-in replacement for convolutional layers and shows promising results on the evaluted tasks\n- Evaluation is done on two different tasks and shows that SDNs perform better on both tasks than convolutional VAEs\n- Using SDNs seems to support sampling from VAEs at much higher temperatures than usually possible\n\nQuestions/remarks:\n- What is the relationship (similarities/differences) between SDN and normal (self-)attention? Why would SDN be better than attention (except maybe faster runtime)? What can SDN do that attention can't (in principal or in practice)?\n- Section 2, \"project-in\" stage: why are the project-in and project-out states needed? Can't the corrections stage be performed on the input features directly?\n- Figure 2 (b): \"Solid arrows represent direct dependencies, and dashed indirect ones\" -> what exactly do you mean with \"indirect dependencies\"?\n- Avoiding vanishing gradients: do you actually ever observe vanishing gradients? have you tried training with/without any normalization methods to see if vanishing gradients become a problem?\n- SDN-VAE architecture: why do you only use SDN in the decoder? Have you tried using them also in the encoder?\n- How much overhead does SDN in the SDN-VAE actually introduce in practice - regarding model size/VRAM and training time? How much bigger is the SDN-VAE compared to the baseline and how much longer does it take to train/per iteration? I see some of these answers are given in the tables in the appendix but I think it would be beneficial to be open about this and mention the trade-off in the main part of the paper.\n-Learning disentangled representations: why do you think SDN helps with learning disentangled representations?\n\nOverall I think this is a good paper. It would be interesting to see further results with SDN, e.g. on a baseline DCGAN architecture, to see if the improvements transfer to other architectures.\nI also believe the paper would benefit from a discussion comparing SDNs to traditional attention and when SDNs might be a better choice than attention. It would also be interesting to know a comparison of attention  vs SDN in terms of training and run-time. Attention has a higher complexity in O notation but it is not clear what exactly this means in practice, especially when attention/SDN are only used on \"small\" feature layers early on in the decoder.\n\n###\n###\n\nUpdate after revision: I have looked over the revised paper and believe the authors have addressed most issues that were raised by the reviewers, especially by describing in more depth the relation of their approach with autoregressive models and self-attention and mentioning the runtime differences of their model compared to a normal CNN.\nI have, therefore, raised my score to 7.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work proposes a novel network structure, spatial dependency networks that is introduced as an alternative to convolutional neural networks. This new architecture is used successfully to get state of the art performance for a number of common image generation benchmarks when compared with non-autoregressive approach (even much larger CNNs). There is a lot of useful feedback in the reviews themselves: a thing to consider in the final version is the fact that the authors had motivated SDNs as drop-in replacements for CNNs, but do experiments mostly in VAE-like settings. This is a point that was raised by multiple reviewers and is clearly something that should be dealt with as explicitly as possible.

While there are legitimate reasons to be wary of the increased computation time, I tend to side with the authors that baselines that are being compared with SDNs are likely to have more optimized primitives. From the inference numbers presented in the rebuttal, it doesn't appear like the speed issues are insurmountable. 

Given the high quality of writing, the excellent performance on image density modeling, the various ablations and understanding of the disentangling effects, I think this is an interesting piece of work that the field would benefit from.
Decision: Accept (Poster)