Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: In this paper, the author proposed a variational framework for relational learning that decouples relational property and absolute property among objects based on a pre-defined probabilistic graphical model (PGM). The author also proposed the so called relation-preserving data augmentation (RPDA) strategy to address the challenges for the resulting optimization. Overall, the paper is well written and easy to follow. Below are some of my concerns.

1. It seems that RPDA is crucial for training. However, the relation preserving function D is often difficult to find for real data where the existing relation patterns are often unknown.

2. No ablation study is reported regarding the effect of RPDA.

3. Experiments seem to be too simple. It might be hard to find appropriate application cases though.

4. Can the author clarify the relation between the proposed PGM based relation learning framework and causal learning?
review_title: An interesting variational framework for relational learning

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes a model to infer the relationship between multiple instances in a dataset by inferring a latent variable. The authors accomplish this by defining an optimization problem that optimizes the ELBO of the proposed graphical model. The paper presents a nice solution to some of the identification issues that can arise when inferring the latent variable, in particular the so called “information shortcut” when the model overfits to only learning the “absolute” property of the dataset, rather than inferring the shared latent traits. 

I think that the formulation that the authors present and the proposed  optimization model is quite interesting, however I have additional questions / concerns:

It’s not clear exactly what the authors mean by a relationship.
Why is there a latent variable causing b but not a? Why are we assuming that a causes b? 
It’s unclear to me why we refer to the relational property as ‘z’. According to figure 1 z has no effect on a, so it doesn’t seem like it would describe the relationship between a and b? Wouldn’t theta be serving that purpose?

I’m also confused about the difference between what is proposed here and a latent factor model. It would appear that the authors are proposing a model to integrate a latent factor model into a generative net, which is interesting, but does not come through in the text as it currently reads. It would seem that in order to interpret z as a relational variable we should be able to extract some kind of meaning from it? Also, if there are multiple relationships , say “animal”, “rotation”, and “saturation”, should we expect to be able to disentangle these concepts with the proposed model?

I found the experiments to be a bit underwhelming. It is unclear how the authors decided on the architecture, hyperparameters, and number of latent variables for the proposed model. It would seem that the model would be quite sensitive to these. In addition, it seems that some of these evaluations would benefit from comparison to more traditional methods. For example, the faces example appears in “A Dependence Maximization View of Clustering”, within a very similar context.  

Overall, I think this is an interesting idea, but I would like to see the paper a bit more refined before recommending acceptance. 

review_title: Review of "Relational Learning with Variational Bayes"

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper proposes variational relational learning by learning relations between two inputs via variational inference on a probabilistic graphical model (PGM). The PGM that they use factors as p(a)p(z)p(b|a,z) where a,b are the two inputs and z is the supposed relationship between them. The example shown in the experiments is rotational mnist, where b is a rotated version of a, and z should encode the degree of rotation. The paper learns both the forward network and in inference network in a VAE-like approach; the elbo derivations appear correct to me.

In section 3.3 the paper describes two sources of problems with naively optimizing based on the above approach: 1) deterministic mapping where a completely determines b, and no learning of relation is necessary, and 2) information shortcut where z can completely encode b. To me 1 does not seem like a big drawback since it is more of a limitation with the underlying data. For 2, the paper describes a data-augmentation technique they call RPDA, as well as using an informative prior on z. The experimental results seem reasonable, although having setting beyond mnist would have been nice.

The two methods described to handle the information shortcut issue are not super satisfactory in my opinion. The data augmentation technique relies on designing augmentations g that preserve the relationship, so that (a,b) are related in the same way as (g(a), g(b)). But coming up with such augmentations seems to rely on us knowing properties of the relationship z, which is exactly what we are trying to learn in the first place. The alternative of using an informative prior is not super clear to me, since it seems that with a powerful enough model p(b|a,z), scaling and shifting p(z) to fit a Gaussian will not prevent the model from completely encoding b as a function of z.

In my opinion constraining the expressiveness of z seems like the right path, but instead of regularizing p(z) to be a Gaussian, I would consider using discrete z's that explicitly limits the theoretical # of bits z can store, hence preventing it from encoding b. I think this also intuitively aligns with how many relations seem more discrete than continuous (a is friend of b, a is adjective form of b, ...etc). I would be interested in seeing the experiments done using discrete z's, which seems natural too given the 5 discrete rotations considered in the paper.
review_title: Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: Relational learning is an important capability exhibited by humans of learning relations between objects.  This work considers a fairly general setup for relational learning and addresses learning of the relation through a method based on variational inference.

I am not familiar with the VAE literature; having said that, the method seems novel.  The problem setup is so general that it is unlikely to be novel, but the main technical contribution of the paper appears to be the derivation of the objective (4).  Of course, given that the objective is a lower bound of the probability we want to optimize, and since no theory is provided on the quality of the approximation, it is important to have strong demonstrations of the method in which it is shown to have unique advantages compared to existing approaches.  Unfortunately, the paper falls short in this regard, as I will go into more detail.

The equation (4) does seem plausible and it is rather elegant.  Note that (4) can also be seen as likelihood + regularization on z + entropy of z, hence it encourages latent variables z that are not too large according to the regularization, while the entropy term prevents z from collapsing to the mode.  This makes sense, but at the same time it is difficult to see how independence between z and a would be enforced using this objective function.

The other technical contribution is the identification of optimization issues via the "information shortcut" and "deterministic mapping" issues.  This is very insightful.  The idea of RPDA is interesting, and in many cases it could be applied at least via data-augmentation.

The main demonstration is based on learning rotation angles in MNIST.  The relationship of "A is 30 degrees rotated compared to B" does qualify as a relation, albeit one that seems fairly artificial.  (EDIT: since discussion, I retract the following in brackets).  [More worrisome is that the example used, if I understand correctly, does not follow the causal model adopted by the authors.   The model assumes that a and z cause b.  However, the example has b as the MNIST label.  This is causally incorrect because for handwritten digits, the label is the cause of the image, not vice-versa.  Presumably the causal assumptions should be correct for the model to work well?  It would be helpful if the authors could comment on this.]

The Yale face dataset in the supplement seems a better fit to the causal assumptions.  

However neither example is satisfactory in showing a unique benefit of this approach that cannot be already obtained by domain-specific methods, such as spatial transformer networks in the case of learning image rotations/scaling (https://papers.nips.cc/paper/5854-spatial-transformer-networks.pdf) and emotion learning ML methods for facial images (https://www.sciencedirect.com/science/article/pii/S1877050917327679).  Perhaps the authors could propose an application that would showcase the advantages of the approach, even if it is currently infeasible to obtain results?

If we contrast the relational learning approach to the existing body of work on learning disentangled representations, the main difference is that in existing representation learning work, one learns the latent variable _z_ from _a_ only.  But here, _z_ can be learned from both _a_ and _b_ .  This is obviously an important difference, but one could reduce the problem of learning _z_ from _a_, _b_ to methods that learn _z1_ from _a_ and _z2_ from _b_ separately.  For example, in learning the rotation between _a_ and _b_ (both MNIST digits), one could reduce it to learning the orientation of _a_, learning the orientation of _b_ separately, and then subtracting the orientations.  Of course, this seems to be a property of the examples used rather than a general fact.  If one considers more sophisticated relational learning problems such as properties and objects, then it may no longer be reducible.  Yet this again indicates that the examples used in the paper fail to show the uniqueness of the proposed approach.

### Pros:
 * interesting problem with applications to psychology
 * insightful analysis of the optimization involved and remedies for two pitfalls

### Cons:
 * toy examples inadequate to show novelty and generalizability of the method
 * not obvious to me how the objective function encourages independence of _z_ and _a_.
 * do we have any confidence that this method would work in a more complicated problem?

### Recommendation:
The paper starts an interesting line of development but still seems to need more work in either the theory, the sophistication of demonstrations, or both, before one could make a convincing case that it adds something beyond similar approaches in the literature.  Hence I recommend rejection of this paper.  The authors are encouraged to look into adding more sophisticated and unique examples, and further analyzing their method.

### Modifications since discussion
 * retracted objection about inappropriateness of MNIST example
 * raised score from 5 to 6
review_title: Interesting problem in need of more theory and experiments

Meta-review: This paper presents a variational learning framework for inferring relations between data points. The authors further introduce novel regularizers to avoid unfavorable solutions to their relational learning problem. Qualitative results are provided on rotated versions of MNIST. Additional qualitative results on the Yale face dataset are provided in the appendix.

The reviewers agree that the idea overall is interesting, and the chosen experiments certainly provide some insight into the idea behind the method and demonstrate that the method is learning reasonable representations of relations between data points. I share the sentiment of the reviewers, however, that this paper is not yet ready for publication. The paper in its current form lacks clear positioning against related problems and related research in this community, and the experiments are all qualitative in nature without the attempt to rigorously compare the proposed method against established techniques. The argument brought forward by the authors that the proposed problem is completely novel and therefore no baselines exist is unconvincing as pointed out by the reviewers, as there is related research on e.g. spatial transformer networks [1], neural relational inference [2], and discovery of causal mechanisms [3], which similarly address the problem of discovering relations, interactions, or transformations. Even if these methods don't exactly fit the problem setup presented in this paper, an attempt should be made to design an evaluation that allows one to compare against some of these approaches, especially given that the paper claims to address the general problem of inferring relations between pairs of data points. Overall, I am confident that this would make the paper stronger and more relevant to this community.

[1] Jaderberg et al., Spatial Transformer Networks (NeurIPS 2015)
[2] Kipf et al., Neural Relational Inference for Interacting Systems (ICML 2018)
[3] Parascandolo et al., Learning Independent Causal Mechanisms (ICML 2018)

Decision: Reject