Venue: UAI 2022 Workshop CRL

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary: This paper proposes the Structural Autoencoder (SAE) architecture for self-supervised structure representation learning. Compared to prior work, the key change in the architecture is the introduction of a sequence of independent random variables that are fed into the decoder at different levels of NN. The architecture is claimed to emulate general acyclic causal model. Paper also introduces a hybrid sampling method that relates only on the independence of latent variables rather than fixing the whole latent distribution. The authors of the paper experimentally compare their architecture to vanilla AE, VLAE, $\beta$-VAE, WAE, FactorVAE. They observe improved performance in terms of the FID score. 
The paper targets an important problem of structured representation learning and demonstrates promising experimental results.

Clarity: Even though the paper is reasonably well-written, it was quite hard to follow due to the lack of details. Specifically, the paper clearly lacks explicit formulas describing training loss and architecture. So I would like to clarify some questions:
1) Can you please clarify how do you select U_i in training? Are U_i the output of the encoder?
2) I am not sure I understand the proposed hybrid sampling. Do I understand correctly that in your experiments latent variables U_i are assumed to be known for N=128 training samples and U_i are being sampled from their product distribution after that? Are they selected as an output of the encoder? or are they assumed to be a part of the training dataset (input)?
3) What is your training loss? Do I understand correctly that the loss only measures the distance between the training sample and the decoder's output and does not take U into account?
4) I am not sure if the decoder's structure really emulates general acyclic graphical model. Figure 1 seems to suggest that the dependence is $S_i = f(S_{i-1}, U_i)$, which corresponds to a very sparse causal DAG (a path graph). 
5) Even though the paper claims that the proposed architecture is supposed to emulate causal dependencies, it seems that no experiments that study causality are presented. Can you clarify please how the proposed method can be used to uncover causal dependencies between latent (or observed) variables?  
review_title: review for Structure by Architecture: Disentangled Representations without Regularization

final_score: rating: 6: Marginally above acceptance threshold
review_content: ### Summary:
The authors propose a new autoencoder architecture which they motivate through the lens of causality by attempting to emulate a structural causal model in the decoder. Specifically, they infer a set of latent variables $U_i$, and pass these variables separately at multiple points of the decoder to represent the independent noise variables at each mechanism in an SCM. They imply that this structural bias in the decoder will enourage it to learn disentangled features and demonstrate this with a number of metrics and compare with a number of baselines. They additionally show improved reconstruction performance with their 'hybrid sampling' procedure. 

### Quality:
The quality of the paper overall is passable, but it is certainly not the most polished piece of work. Specifically the figures are hard to interpret and sometimes include unexplained elements (e.g. the 'static variables' of figure 1). Additionally, there are many statements made in the text which appear false at face value, but are then later in the text validated by further explanation of the author's intentions.

### Clarity: 
The clarity of the writing is relatively low. It often takes multiple reads to understand what the authors are trying to say due to weird vocabulary choices. One example (Page two, last paragraph), is the authors discuss 'This variable decoder depth for each of the latent variables', which seems to imply the decoder itself has variable depth -- personally I believe this would be better rephrased as 'Due to the fact that latent variables are passed at different layers of the decoder'. However this is only one of many examples. 

### Strengths:
- The number of experiments performed is very impressive as well as the list of baselines. If anything this paper does perform a rigorous analysis of a range of methods which I believe puts it over the threshold for acceptance as a workshop paper. 
- Furthermore, the idea itself is interesting and original as far as I can tell, and thus a valuable addition to the literature.
- Finally, the related work appears well cited and references are prevalent throughout. 

### Limitations:
- The model architectural choices are not well motivated either by theory or ablation testing. Specifically, why are the Str-Tfm layers only performing elementwise transforms and not something more complicated? Why are the latent variables not simply concatenated at each layer? These choices should at least be discussed in the text.  
- The experimental results are relatively weak and sometimes conflicting (as in the case of figure 2, the AdaAE performing better seems to somewhat contradict the value of the SCM structure of the SAE).
- I did not find the 'static features' pictured in Figure 1 described in the text. Are these derived from an encoder? If so, shouldn't these also be considered to be $U_0$ to fit with the SCM interpretation? 
- The hybrid sampling method introduced in the paper seems relatively simple compared with some of the other approaches in the literature which have been introduced to solve the 'latent holes' problem. Specifically methods which learn to map from a compact distribution to the distribution of the VAE seem to be far more powerful and general than this method. It would be helpful if the authors could compare with such methods.
- The exact reason for why the 'architectural asymmetry' between latent variables should yield independence is not entirely clear to me and should perhaps be further theoretically explored. 

### Minor suggestions:
- Page 1, last paragraph: "The most popular autoencoder based method is" should be "The most popular autoencoder based methods are".  

### Conclusion:
The paper provides an interesting new idea and it is sufficiently empirically explored to satisfy the requirements for a workshop paper. I thus recommend acceptance. 
review_title: Review: Structure by Architecture

final_score: rating: 4: Ok but not good enough - rejection
review_content: ### Summary:

This work revisits the important problem of representation learning and proposes a new model architecture (an autoencoder which has a specific structure) and a new training procedure (hybrid sampling). Given an unlabeled dataset, representation learning is the task of learning the underlying higher level representations that generate them. Some of the more prominent approaches have been based on variational autoencoders and their spinoffs. Existing methods seem to lose either in generation quality or disentanglement guarantees. To this end, the authors propose an alternative model and training procedure. There are no theoretical results but they verify their ideas experimentally and conclude that the newer models learn similar or better representations on various image datasets compared to other models. And while doing so, they achieve similar or better disentanglement guarantees. This is a potentially interesting contribution that diverges a bit from the standard techniques in representation learning via VAEs. However, the paper is not well-written and I have reservations about the conceptual contributions of the work (see weaknesses below).

### Technical contributions:

The work assumes that the data is generated from D (jointly) independent latent variables U_1, .., U_D via arbitrarily complicated functions applied on them. With this view, they fit the observed distribution via standard autoencoder based methods, with two main differences

1) The SAE model architecture: The model architecture they propose is similar to the variational ladder autoencoder (VLAE) but in the decoder, they replace the ladder rungs by layers called structural transformer layers. Moreover, they also feed in the latent variables at different layers. The intuition is that the latent variables fed in the higher layers (resp. lower layers) encode higher level information (resp. lower level information). Because the architecture is hierarchical, the authors view it as a causal model so there is an implicit DAG model underneath. That said, I'm not sure what advantage this interpretation offers in this context.

2) Hybrid Sampling: For sample generation, we need to generate fresh U_1, .., U_D. To do this, they propose hybrid sampling: During training, collect together N different values of U_1, .. U_D. So we have a multidimensional array with N rows and several columns corresponding to all the variables. Now, to generate a new latent sample, simply pick a uniformly random element of each column. In other words, the latent space is discrete with the points being the ones learnt during training, but the independence happens across the different variables. Because the values have already been "seen", they are able to avoid bad latent variables during sample generation, which prevents 'holes' in the latent space when using standard VAE priors.

### Strengths:

They show experimentally that for various datasets (3D shapes, MPI3D, Celeb-A and RFD), their technique achieves similar or better reconstruction quality than their counterparts. They also show that they learn latent variables with better disentanglement, measured via various metrics. Finally, if I understand A.3.1 correctly, there is some experimental evidence to suggest that the decoder plays a bigger role than the encoder in generative modeling (see fig 5), so this may possibly be a useful takeaway.

### Weaknesses:

The paper is not well-written with some confusing logic at various places. For example, in section 2.1, what are X_1, ..., X_d and what's their relation to S_1, .., S_D? It's unfortunate that the first technical section already left me confused. Also, usually in VAEs, there are multiple issues when training the model. Regularization is just one of them. Others include choice of loss function, choice of architecture, noise in data, etc. The authors should explain why they claim regularization is the biggest (or most important) issue.

Most of the arguments in this work hinges on regularization being not reasonable for the model. However, it is indeed the case that for any complicated distribution, there is a function which maps the standard normal distribution to the distribution (via the inverse CDF transform). So, if we take the MLP complicated enough, there is no loss in generality in assuming that the U_i are distributed as N(0, 1). The models in this work use a 3-layer MLP which can map N(0, 1) to fairly complicated distributions, so this undermines the contributions of the work.

Underlying the neural architecture is a causal diagram. However, the connection to causality seems to stop there. I don't think the DAG models are being learnt explicitly in this work and the relations to topological orderings is not necessary to absorb the techniques proposed in this work. 

Finally, there isn't much clarity on how the architecture was chosen. In other words, how much hyperparameter tuning was done?

### Other comments:

It's hard to interpret the faces in A.4.8. Putting them next to each other maybe helpful.

In the end of page 4, there is an unreferenced figure.

In the last paragraph of A.2.1, there is a typo "configuration files of a the attached code"

In the first line of A.4.4, there is a typo "exptrapolation".
review_title: Review of "Structure by Architecture: Disentangled Representations without Regularization"

Meta-review: metareview: There is some disagreement on the reviewers, who praised the experimental results, but at the same time raised concerns about writing clarity and lack of details in the main text. The contribution nevertheless appears to be relevant to the workshop topic. Reviewers asked some questions and provided many suggestions which the authors are encouraged to address in a future version of their submission.
confidence: 3: The area chair is somewhat confident
Decision: Accept (Poster)