Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a variational method for relational learning. It first introduces relational learning as learning based on relational property instead of absolute property, and introduces conditions (Eq. 1). Then it proposes VRL-PGM with a variational lower bound. To eliminate the information short-cut, it introduces relation-preserving data augmentation (RPDA). Experiments show that the method is able to perform relation discrimination and relation mapping, on a variant of MNIST and Yale face datasets.
main_review: Strengths:

The problem set up is novel, and the paper addresses an important problem. The method is intuitive, and the RPDA is novel. The experiment is well-conducted, and shows that the method outperforms other strong baselines.

Weaknesses:
In Eq. 1, the authors introduces 4 conditions for z to be satisfied as relational property. However, the proposed PGM only satisfies 3 of them where (ii) is not satisfied. This is an important omission. The authors may need to consider ways to make it consistent, or justify such omission.

--
Update:
I have read the reply from the author as well as other reviewers' comments. It is understandable that the 4 conditions are hard to achieve simultaneously, and I'm satisfied with the authors' response. Considering the weaknesses of the experiments as raised by other reviewers, I remain my score of borderline accept.
summary_of_the_review: In summary, the authors addresses an interesting problem and proposes a novel method, and overall an interesting paper. The authors may need to improve its justification of the PGM omitting condition (ii), or find ways to make it consistent with the proposed 4 conditions of relational variable.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: A new approach for relational learning is proposed. The main contributions are
A novel formulation of relational learning as a variational inference problem
A novel augmentation method for relational data
Empirical results that show the ability to learn relationships in images.
main_review: A new relational learning approach is proposed based on variational learning. The main idea is to disentangle absolute and relational properties and force the learner to learn based on relationships rather than absolute properties of instances. The problem is formulated as a variational inference problem and a data augmentation method is developed that preserves relational information. Experiments are shown on MNIST, monoglot and Yale face database for unsupervised learning.

The paper has an interesting idea and I liked the formulation of the variational inference problem for relational learning. The main weakness is perhaps the somewhat forced construction of relationships in the empirical results. Since the paper has a very general aim of learning relationships, it would have been really strong if this could be shown through a natural relational problem rather than the constructed ones. The results do show that the method performs better than even a specialized transformer model for rotations (STN) which is definitely.a plus. On the other hand, the RPDA functions need to encode the rotational aspects so it does require some specialization. I was not sure about the data augmentation RPDA functions for the facial emotions data (was it related to illumination changes, etc.).  Also, how different is the relational data augmentation from the typical types of data augmentation methods that people would use if they had the domain knowledge, maybe the speciality of the relational augmentation methods could be highlighted a bit better.

After author feedback

I thank the authors for their feedback. The idea definitely seems good, maybe with a bit stronger comparison/ empirical studies, this would be a stronger paper.
summary_of_the_review: Overall the formulation seems interesting and results show that the general approach is on par with state of the art specialized methods for the tasks considered. However, the relationships in the experiments seem a bit forced and maybe simpler data augmentation methods could replace the relational augmentation methods used.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes an unsupervised learning method, variational relational learning (VRL), to address the relational learning problem and learn the underlying relationship between a pair of data irrespective of the nature of those data, and demonstrates its performance on variations of the MNIST/Omniglot datasets and the Yale Face datasets.
main_review: ### Strength

- The relational learning problem is clearly formulated in the paper, and seems novel and interesting.
- The paper is well written and easy to follow.
- The experiments are well-designed and demonstrate interesting behavior of VRL.

### Questions/concerns

- In Table 1, why is STN-Rotate so much worse on Omniglot than on MNIST? Omniglot seems pretty similar to MNIST, and the two datasets are similarly constructed. Furthermore, STN-Rotate is essentially cheating (as it knows the transformations are rotations) so I would expect it to perform well. It is quite surprising that it does so badly on Omniglot.
- The comparison with STN seems a bit strange. Why does STN take both a and b as input? I would imagine an STN that takes just a as input and we compare the output with b. This is basically directly learning the transformations. In addition, for STN, clustering on the output of the trained localization network does not seem fair. Depending on how the STN is specified, I can imagine different outputs of the trained localization network that lead to the same transformation. Directly comparing the resulting transformations seems like the better approach.
- The VRL-PGM is not symmetric w.r.t. a and b. Does that affect the model in any way? Is this asymmetry considered in the training (e.g. randomizing the order of the two elements in each training example) and test process?
- For the Omniglot experiments, VRL with discrete no RPDA for 5 relationships is worse than VRL with continuous with no RPDA, so the claim that constraining z to be discrete improves performance does not always hold.
- How does the rotation augmentation work for some of the baseline methods, e.g. for VAEs? It seems by use of RPDA the model is cheating a bit, as it essentially has additional supervision signal coming from access to two pairs of samples that it knows share the same transformation. To make the comparison fair, for VAEs for example, in addition to training VAEs on the given training data, there should be additional supervision that the latent state for a pair of samples and the latent state for the pair of samples transformed by a relation preserving function should match. I don't know if this is the way rotation augmentation is used for VAEs. If not, it doesn't seem to be a fair comparison. Similar issues might exist for other baseline methods (although in terms of performance VAEs seem the most promising). If VAEs perform well with this additional piece of supervision information, then the good performance of VRL is more from the way RPDA is used, and less from the method itself.
- How does relational mapping look like on the Yale Face dataset? Some visualizations would be helpful. I expect it won't be able to generate transformed faces as that is quite complicated, but it would be interesting to see what the model really learned.

summary_of_the_review: I have some questions/concerns about the paper so recommend weak reject, but would be happy to bump up the score if my questions/concerns can be satisfactorily addressed.

------------------------------
The authors have satisfactorily addressed my concerns, and I bumped up the score accordingly.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper the authors pose the problem of relational learning in the form of a Probabilistic graphical model and utilize variational inference for learning the relational property. The experiment section is extensive highlighting the utility of the proposed method.
main_review: I have 2 concerns regarding the presentation/organization of the paper:
1. The authors claim in the introduction that they formulate the problem of relational learning as a PGM which seems misleading. They approximate the problem because the omission of the independence condition is an important omission. In fact the authors themselves shed light on what could go wrong because of the omitted condition and use the RDPA trick to address it.
2. The relational learning problem needs better discussion. In particular, the transition from the definition of relational learning to the probabilistic formulation could use an example.

Other than this, I believe that this is an interesting paper with a novel formulation of the relational learning problem and extensive experimentation.
summary_of_the_review: Good paper help back by the presentation.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a relational learning method based on variational Bayes. The main idea is to learn relations among objects independently of each object's own properties. The model theory is discusses where authors define the problem and discuss about solutions for some limitations of the model when a relation between objects A and B can be found by only looking at one of the objects own properties.
The paper is quite well written. Algorithms are provided, but not the code. As far as I understood, results are obtained on "benchmark" datasets, where some synthetic versions of these datasets are used to create the relations between the instances of the original dataset and of the synthetic.
main_review: Strengths:
- the idea of learning relations solely from existing relations is interesting
- the theoretical background is simple and well defined

Weaknesses:
- datasets are only images
- from the authors' example (happy-sad faces), it seems absolute properties are indirectly being used
- how does this work differ from others that work on complex networks, trying to extract motifs from graphs?


summary_of_the_review: Overall, I liked the idea of formulating the problem as "learning relations from relations". However, there are other approaches that are not mentioned here which try to find "motifs" in graphs or perform link prediction, where the objects (without their properties) are represented as nodes, relations are in the edges and new links are learned from the original graph.

From your example (sad-happy faces), it seems features of each image are being used to learn new relations. Why did you choose images for training? I would think that a tabular dataset (specially in areas such as health) could benefit much from your approach.

References are outdated. The last year for your refs is 2019. There has been a whole body of work regarding relational learning and methods developed in 2020 and 2021.
review_title: null

Meta-review: This papers studies the classical problem of relational learning from a probabilistic perspective. The authors propose four reasonable constraints to encode relational properties, and develop a PGM-based variational method for learning relational properties from data. After extensive discussion with the authors, a majority of the reviewer reviewers agree the approach is interesting, if not without some flaws.

The problem studied is interesting, novel, and could lead to new developments in the area of relational learning. It is expected that the experiments have some limitations given the authors have approached the problem from a fresh new angle, which the reviewers have appreciated.

Please pay attention to the suggestions from the reviewers, and in particular, please add a more detailed discussion with statistical relational learning: This material may not be familiar to the broader ML audience, and therefore it is essential to make these comparisons explicit.
Decision: Accept (Poster)