Venue: ICLR 2018 conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposed a novel neural net architecture that learns object concepts by combining a beta-VAE and SCAN. The SCAN is actually another beta-VAE with an additional term that minimizes the KL between the distribution of its latent representation and the first beta-VAE’s latent distribution. The authors also explored how this structure could be further expanded to incorporate another neural net that learns operators (and, in common, ignore), and demonstrated that the proposed system is able to generate accurate and diverse scenes given the visual descriptions.

In general, I think this paper is interesting. It’s studying an important problem with a newly proposed neural net structure. The experimental results are good and the model is compared with very recent baselines.

I am, however, still lukewarm on this submission for its limited technical innovation and over-simplified experimental setup.

This paper does have technical innovations: the SCAN architecture and the way they learn “recombination operators” are newly proposed. However, there are in essence very straightforward extensions of VAE and beta-VAE (this is based on the fact that beta-VAE itself is a simple modification of VAE and the effect was discussed in a number of concurrent papers).

This would still be fine, as many small modifications of neural net architecture turn out to reveal fundamental insights that push the field forward. This is, however, not the case in this paper (at least not in the current manuscript) due to its over-simplified experiments. The authors are using images as input, but the images are all synthetic, and further, they are all synthesized to have highly regular structure. This suggests the network is likely to overfit the data and learn a straightforward mapping from input to the code. It’s unclear how well the system is able to generalize to real-world scenarios. Note that even datasets like MNIST has much higher complexity than the dataset used in this paper (though the dataset in this paper is more colorful).

I agree that the proposed method performs better that its recent competitors. However, many of those methods like TripleELBO are not explicitly designed for these ‘recombination operators’. In contrast, they seem to perform well on real datasets. I would strongly suggest the authors perform additional experiments on standard benchmarks for a fair comparison.

review_title: A neural network that learns visual concepts and basic operators over them.

final_score: rating: 7: Good paper, accept
review_content: Summary
---
This paper proposes a new model called SCAN (Symbol-Concept Association Network) for hierarchical concept learning. It trains one VAE on images then another one on symbols and aligns their latent spaces. This allows for symbol2image and image2symbol inference. But it also allows for generalization to new concepts composed from existing concepts using logical operators. Experiments show that SCAN generates images which correspond to provided concept labels and span the space of concepts which match these labels.

The model starts with a beta-VAE trained on images (x) from the relevant domain (in this case, simple scenes generated from DeepMind Lab which vary across a few known dimensions). This is complemented by the SCAN model, which is a beta-VAE trained to reconstruct symbols (y; k-hot encoded concepts like {red, suitcase}) with a slightly modified objective. SCAN optimizes the ELBO plus a KL term which pushes the latent distribution of the y VAE toward the latent distribution of the x (image) VAE. This aligns the latent representations so now a symbol can be encoded into a latent distribution z and decoded as an image.

One nice property of the learned latent representation is that more specific concepts have more specific latent representations. Consider latent distributions z1 and z2 for a more general symbol {red} and a more specific symbol {red, suitcase}. Fewer dimensions of z2 have high variance than dimensions of z1. For example, the latent space could encode red and suitcase in two dimensions (as binary attributes). z1 would have high variance on all dimensions but the one which encodes red and z2 would have high variance on all dimensions but red and suitcase. In the reported experiments some of the dimensions do seem to be interpretable attributes (figure 5 right).

SCAN also pays particular attention to hierarchical concepts. Another very simple model (1d convolution layer) is learned to mimic logical operators. Normally a SCAN encoder takes {red} as input and the decoder reconstructs {red}. Now another model is trained that takes "{red} AND {suitcase}" as input and reconstructs {red, suitcase}. The two input concepts {red} and {suitcase} are each encoded by a pre-trained SCAN encoder and then those two distributions are combined into one by a simple 1d convolution module trained to implement the AND operator (or IGNORE/IN COMMON). This allows images of concepts like {small, red, suitcase} to be generated even if small red suitcases are not in the training data.

Experiments provide some basic verification and analysis of the method:
1) Qualitatively, concept samples are correct and diverse, generating images with all configurations of attributes not specified by the input concept.
2) As SCAN sees more diverse examples of a concept (e.g. suitcases of all colors instead of just red ones) it starts to generate more diverse image samples of that concept.
3) SCAN samples/representations are more accurate (generate images of the right concept) and more diverse (far from a uniform prior in a KL sense) than JMVAE and TELBO baselines.
4) SCAN is also compared to SCAN_U, which uses an image beta-VAE that learned an entangled (Unstructured) representation. SCAN_U performed worse than SCAN
and baselines.
5) Concepts expressed as logical combinations of other concepts generalize well for both the SCAN representation and the baseline representations.


Strengths
---

The idea of concept learning considered here is novel and satisfying. It imposing logical, hierarchical structure on latent representations in a general way. This suggests opportunities for inserting prior information and adds interpretability to the latent space.


Weaknesses
---

I think this paper is missing some important evaluation.

Role/Nature of Disentangled Features not Clear (major):

* Disentangled features seem to be very important for SCAN to work well (SCAN vs SCAN_U). It seems that the only difference between the unstructured (entangled) and the structured (disentangled) visual VAE is the color space of the input (RGB vs HSV). If so, this should be stated more clearly in the main paper. What role did beta-VAE (tuning beta) as opposed to plain VAE play in learning disentangled features?

* What color space was used for the JMVAE and TELBO baselines? Training these with HSV seems especially important for establishing a good comparison, but it would be good to report results for HSV and RGB for all models.

* How specific is the HSV trick to this domain? Would it matter for natural images?

* How would a latent representation learned via supervision perform? (Maybe explicitly align dimensions of z to red/suitcase/small with supervision through some mechanism. c.f. "Discovering Hidden Factors of Variation in Deep Networks" by Cheung et al.)

Evaluation of sample complexity (major):

* One of the main benefits of SCAN is that it works with less training data. There should be a more systematic evaluation of this claim. In particular, I would like to see a Number of Examples vs Performance (Accuracy/Diversity) plot for both SCAN and the baselines.

Minor questions/comments/concerns:

* What do the logical operators learn that the hand-specified versions do not?

* Does training SCAN with the structure provided by the logical operators lead to improved performance?

* There seems to be a mistake in figure 5 unless I interpreted it incorrectly. The right side doesn't match the left side. During the middle stage of training object hues vary on the left, but floor color becomes less specific on the right. Shouldn't object color become less specific?


Prelimary Evaluation
---

This clear and well written paper describes an interesting and novel way of learning a model of hierarchical concepts. It's missing some evaluation that would help establish the sample complexity benefit more precisely (a claimed contribution) and add important details about unsupervised disentangled representations. I would happy to increase my rating if these are addressed.
review_title: Good paper, but some pieces are missing

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper introduces a VAE-based model for translating between images and text. The main way that their model differs from other multimodal methods is that their latent representation is well-suited to applying symbolic operations, such as AND and IGNORE, to the text. This gives them a more expressive language for sampling images from text.

Pros:
- The paper is well written, and it provides useful visualizations and implementation details in the appendix.

- The idea of learning compositional representations inside of a VAE framework is very appealing.

- They provide a modular way of learning recombination operations.

Cons:
- The experimental evaluation is limited. They test their model only on a simple, artificial dataset. It would also be helpful to see a more extensive evaluation of the model's ability to learn logical recombination operators, since this is their main contribution.

- The approach relies on first learning a pretrained visual VAE model, but it is unclear how robust this is. Should we expect visual VAEs to learn features that map closely to the visual concepts that appear in the text? What happens if the visual model doesn't learn such a representation? This again could be addressed with experiments on more challenging datasets.

- The paper should explain the differences and trade offs between other multimodal VAE models (such as their baselines, JMVAE and TrELBO) more clearly. It should also clarify differences between the SCAN_U baseline and SCAN in the main text.

- The paper suggests that using the forward KL-divergence is important, but this does not seem to be tested with experiments.

- The three operators (AND, IN COMMON, and IGNORE) can easily be implemented as simple transformations of a (binary) bag-of-words representation. What about more complex operations, such as OR, which seemingly cannot be encoded this way?

Overall, I am borderline on this paper, due to the limited experimental evaluation, but lean slightly towards acceptance.

review_title: interesting idea, but limited experimental evaluation

Meta-review: This paper initially received borderline reviews. The main concern raised by all reviewers was a limited experimental evaluation (synthetic only). In rebuttal, the authors provided new results on the CelebA dataset, which turned the first reviewer positive. The AC agrees there is merit to this approach, and generally appreciates the idea of compositional concept learning.
Decision: Accept (Poster)