Relational Learning with Variational Bayes
Authors: Kuang-Hung Liu
Venue: ICLR 2022 Poster (2022)
Keywords: Relational learning
psychology
unsupervised learning
variational inference
probabilistic graphical model.

Abstract: In psychology, relational learning refers to the ability to recognize and respond to relationship among objects irrespective of the nature of those objects. Relational learning has long been recognized as a hallmark of human cognition and a key question in artificial intelligence research. In this work, we propose an unsupervised learning method for addressing the relational learning problem where we learn the underlying relationship between a pair of data irrespective of the nature of those data. The central idea of the proposed method is to encapsulate the relational learning problem with a probabilistic graphical model in which we perform inference to learn about data relationship and other relational processing tasks.

Published as a conference paper at ICLR 2022
RELATIONAL LEARNING WITH VARIATIONAL BAYES
Kuang-Hung Liu"
ExxonMobil Research and Engineering kuang-hung, .1 liueexxonmobi coom
ABSTRACT
In psychology, relational learning refers to the ability to recognize and respond to relationship among objects irrespective of the nature of those objects. Relational learning has long been recognized as a hallmark of human cognition and key question in artificial intelligence research. In this work, we propose unsuper- vised learning method for addressing the relational learning problem where we learn the underlying relationship between pair of data irrespective of the nature of those data. The central idea of the proposed method is to encapsulate the rela- tional learning problem with a probabilistic graphical model in which we perform inference to learn about data relationship and other relational processing tasks.
INTRODUCTION
American Psychological Association defines relational learning as (VandenBos & APA, 2007):
Definition 1.1 (Relational learning). Learning to differentiate among stimuli on the basis of relational properties rather than absolute properties.
In other words, relational learning refers to the ability to recognize and respond to relationship (called relational property) among objects irrespective of the nature of those objects (called absolute property). For example (attributed to Doumas & Hummel (2013), how do we come to understand that two circles are the same-shape in the same way that two squares are? In this example, "same-shape" is the relational property and object shape is the absolute property. Relational learning has long been recognized as a hallmark of human cognition with strong implications for both human- -ike learning capabilities and generalization capacity (Biederman, 1987 Medin et al. 1993, Gentner 2003; |enn et al.] 2008 Holyoak 2012) Gentner & Smith 2012). We refer the interested readers to the provided references for comprehensive discussion on this subject. Contemporaneoously the research on learning data relationships-aue also commonly called "relational learning" has flourished in the machine learning community where the overarching goal is learning in context where there may be relationships between learning examples, or where these examples may have complex internal structure (i.e.. consist of multiple components and there may be relationships between these components) (Getoor & Taskar] 2007 De Raedt et al.] 2016). We argue that the key difference between the two "relational learning" defnitions and their learning objectives is that Definition [.] takes the relationship learning problem one step further by requiring the data relationships be learned only on the basis of relational properties rather than absolute properties. To the best of our knowledge, this important Iestinction-eeannnne been rigorously studied in the unsupervised learning community, where most existing methods either encourage or do not constrain the relationships learning through absolute properties.
relationships irrespective of the absolute properties- -hs not
In this work, we propose an unsupervised learning for addressing the relational learning problem defined by Definition 1.1 At its core, VRL encapsulates the relational learning problem with a probabilistic graphical model (PGM) in which we perform inference learn about relational property and other relational processing tasks. Our contribution in this paper is threefold: First, we propose a probabilistic formulation for the relational learning problem defined by Definition I.1 Second, we encapsulate the relational learning problem with PGM in which we perform learning and inference. Third, we propose an efficient and effective learning algorithm that can be trained end-to-end and completely unsupervised.
relational learning (VRL)-
"The author completed this research while working ExxonMobil. Corresponding author e-mail: 1iu. kuanghungggmai i com

---

Published as a conference paper at ICLR 2022
2 PROBLEM DEFINITION
We focus on a canonical form of the relational learning problem where we observed paired dataset X = (a(*) b(0) [E[...)] } consisting of N i.i.d. samples generated from a joint distribution p( a A, B).. We dissect the information in into absolute property and relational property where absolute property represents specific features that describe individual a and b, and relational property represents the relationship between a and b irrespective of their absolute property. this work, we interpret the absolute property of a and b as any information that characterizes (even if only partially) the marginal distribution p( a) and p( b). We propose to represent the relational property as latent random variable (r.v.) that satisfies the following constraints:
(i) p a, x = ((a))((())))))
b, z p(bb))))))))) (2)),
(1)
(iii) p( a, z b) * p(a b) b)p(z|b), (iv) p())((b) b, a) # p(b (p(2(a),
where in Eq. I(i) and I we interpret the specification of relational property in Definition 1.1 learning relationships irrespective of the absolute properties- as meaning statistical independence. while in Eq. [ and [(vv) we ensure r.v. contains relevant (relationship) information that further informs a and b about one another, i.c., H(a b, z) < (a b). H(b a. z) < (b a) where H is the conditional entropy. It is easy to that the following conditions are necessary for r.v. z to exist: (1) H( a) > 0 and H(a b) > O, i.e., a and cannot be fully determined by each other; (2) r.v. a, b, z are not mutually independent, i.e., p( a, b, ) * pp a )p( )p(2) z Our goal for relational learning is to learn about relational property z that satisfies Eq. T in a completely
unsupervised fashion. A motivating example for Eq. is provided Appendix A.1
In addition, we are interested in two related relational processing tasks: relational discrimination and relational mapping defined as (VandenBos APA, 2007):
Definition 2.1 (Relational discrimination in condition). A discrimination based on the relationship between or among stimuli rather than on absolute features of the stimuli.
Definition 2.2 (Relational mapping). The ability to apply what one knows about one set of elements to different set of elements.
Relational discrimination allows us to differentiate (a b()) from (a b(0) based on their relational properties. And relational mapping allows us apply the relational property of (a(*), b(9) to different set of data, for example, deduce that b(5) is related to a(ا) in the same way that (()) is related to (
METHOD
Learning and inference relational property z that satisfies all four constraints in Eq. i a challenging problem due to the hard independence constraints in Eq. [() and
and ((ii) To overcome this challenge, we first introduce VRL as a tractable learning method that satisfies 3 (out of 4) constraints in Eq. [
Eq. 1i) I (iii) (ivv
We then discuss VRL's unique optimization challenges, which are partially
attributable to its relaxation of the independence requirement in Eq. [(i)
3.1 VARIATIONAL RELATIONAL LEARNING
The proposed VRL method consists of two parts: first, encapsulate the relational learning problem with PGM, called VRL-PGM; we then formulate various relational processing tasks as performing inference and learning in VRL-PGM. The VRL-PGM model, shown in Fig. I samples data a, z, and b from parametric families of distributions ( a). po z ).). Po( b a, z } that are differentiable almost everywhere with respect to (w.r.t.) a, z, and 0. In practice, we observe only set of independent realizations { (a(1) b | i [L..N]] while the true parameter 0* and the corresponding latent variables (()) are unobserved. A well-known property of the PGM shown in Fig. is that r.v. a and z are independent with no variables observed, but not conditionally independent when is observed, i.c., po a, z)) ) po a a))p((
Pe a, b) # Po( a b )po{ z b)) (Bishop, 2006). Consequently, VRL-PGM can be viewed as a parametric relational learning model that satisfie 3 (out of 4) constraints in Fq. 1 Eq. 1(i) [(ii) 1(iv) (note that Eq. ((v) is trivially satisfied in VRI-PGM).
2

---

Published as a conference paper at ICLR 2022
a
Figure 1: VRL-PGM: probabilistic graphical model for representing the relational learning problem; the observed r.v. and b are generated from some random process (parameterized by 0) involving latent r.V. z.
Further discussions the connection between VRL-PGM and the relational learning problem is provided in Appendix A.2
Having established VRL-PGM, primary learning objective is to approximate the unknown true likelihood function p a, z and posterior pe( z a, b). Learning pe z a, b provides us a way to infer (a(),, b00)ss relational property (()) moreover, it serves as a basis for performing relational discrimination where compare relational properties between different pairs of data. Learning pa b a, z allows us to perform relational mapping where we use the relational property of (a(), b(o) to map a to b()), i.e., b ~ "Po(b. [b] a(;), (z where 2 ~ po z a b()) ).
We estimate the parameter for pe( b a. by following the maximum-likelihood (ML) principle, and approximate the true posterior Po ( a, b) with variational Bayesian approach. More specifically, we use variational distribution 9- a. b)) parameterized by ó. to approximate the unknown (and often intractable) true posterior. Both 0 and @ are learned through maximizing a variational lower bound, C(,, d; ( (() (abbreviated as (())), for the conditional log- likeeihoddood log pe ((() a
(derivation is provided in Appendix C
E (Zla log po b .2 + pe( z log go( zz/a b
(2)
Recall that learning independent of central to our relational learning goal. While this in- dependence assumption is built into VRL-PGM, the learning objective ((t) does not explicitly force z to be independent of a nor penalize learning dependent z. In practice, there may be numerous reasons that could break this independence assumption, e.g., insufficient training data, failure to reach the global optimum, non-identifiability of the model, etc., and it may be desirable to explicitly enforce independence between and a. One way to achieve this to introduce non-positive function that measures the dependency between and with maximum attained when they are independent. For example, we can append the negative mutual information between z and a, -I(z: : a) Dxx (po z, a Po( z ) )). to (());
(()) (2)a())))) log po (0) (1 z) + log po z)) log g0 (z)
b(()) (2.()) -I(;
[(z; a). (3)
Since [(z; a) 2 0 and [(z a) = 0 if and only if and are independent, the addition of [[(z) a) to ((o) not only maintain the validity of the lower bound, but also retain its quality (z and are independent in VRL-PGM).
3.2 OPTIMIZATION CHALLENGES
A limitation of the proposed VRL method is its inability to enforce the independence constraint between z and b in Eq. (ii) This may lead to a worst-case scenario of learning a degenerated posterior ( a, b) ls( b where relational property z only depends the absolute property of b. Another limitation of VRL is exposed when the assumption H(b a) > is violated and can be completely determined by a. In this case, a degenerated likelihood pe b la z pe b a may be fitted and relational property plays no role in the learning process. To further explain these limitations and potential mitigation strategies for avoiding the worst-case learning scenario, we first dissect VRL's gradient updating process into the following steps (using single data point (a(1) b))
3

---

Published as a conference paper at ICLR 2022
example): (1) sample z 9s ( b(O) by using the current parameter dR: (2) evaluate ( and calculate gradients g Vo,, .b. L() by using @k. Og: (3) use gradients g to update de, 02 and get new parameters +x+1, 0 This process is depicted with an information flow diagram shown in Fig. 2a where ideally we would like every path to contribute to the evaluation of all the terms in its reachable nodes in order to obtain meaningful gradients for updating its associated parameters. However, there are two situations where this is not the case: The first situation, called information-
((1)
b (1)
(6)
b
(1)
20 ~ 9o (z) a (0) b b(o)
a.
x
qo (z) a (9)
.b (8)
0
Pa(b() (1) la (4)
2 ); pa (1)
Po (0)
)),, 2 ), po pz(
(b)
Figure 2: Information flow diagrams depicting VRL's gradients updating process, where each path uses its associated parameters to propagate information in the forward direction and gradients in the backward direction: (a) Overfitting occurs when the learning of po (b (()) rely only on the dash-dotted path interministicmmapping or the dashed path finfomationnssorttoo (b) Parameter updating process improved with RPDA.
shortcut, occurs when the learning of pe( ((.) a(),,
z rely entirely on the dashed path in Fig. 2a
more specifically, the dashed (shortcut) path directly propagates ((() through ( to po (60 (1)
and, as a result, the relational property (0)) may only learn to encode the absolute property of b (0)
i... 4o 2 ( ())) ) = go z ((o) ). The second situation, called deterministic-mapping occurs when (0) can be fully characterized by a (1) in this case, the learning of po ((() a z( ) may
only rely on the dash-dotted path in Fig. 2a i.e., Po (()))a()) 200) = Po( ((0 [a()) ). While both situations can be viewed as overfitting problem, determinnisic-mappiin is mainly caused by the data itself and is beyond our control. On the other hand, information-shon -ooortit is caused by short-cutting the gradient update process, which we may overcome with additional regularization techniques. Here we propose two approaches for mitigating the ceeomattion-hooteen problem by disrupting the flow of information passing through the shortcut path. the first approach, we restrict the flow of information by constraining the expressiveness of the latent variable z, e.g., by adopting discrete categorical r.v. (assuming we know a priori the underlying relational property is discrete). the second approach, we propose novel data augmentation strateyy-relation- relation--reserving preserving data augmentation (RPDA) that aims to climinate the shortcut path entirely. First, we define a set of relation preserving functions D= { d(a, b;r) / R (some index set), A B Ax } that preserve data relationship in the following sense: po( z a, b) = Po( z a', b') ), A (a',b') b')= d(a, b; r). Assuming we have access to D, the proposed RPDA strategy then seek optimize modified lower bound C()
go (xta b(t)) [oo Po( (0)) a zz) log (z) log 2s (1) Br()))
RPDA
(4)
(a (4) ((())) = d(a!!) b b(0;(())))))) ,000 ~ U(R).
where
Note that due to the relation preserving property of D, have qs (s(2()) (1) a (6)
qo( ( aa()) (()) ) and thus c(U) is equivalent to ((0) in Eq. @ When we optimize with ((0
the gradient update process can be redrawn in Fig. 2b where now the learning of pe ((() a1() ())
can no longer rely solely on the shortcut path to propagate ((()) since it differs from ((0) by
non-deterministic factor (() In practice, it may seem unrealistic to assume that we can construct
set of RPDA functions D without extensive knowledge of the underlying relational property. How-
ever, we can treat data augmentation as form of regularization and construct D that reflects
4

---

Published as a conference paper at ICLR 2022
prior knowledge and belief of the underlying system (Ronneberger et al. 2015 Perez & Wang 2017). For example, if we want the learning to be rotation invariance a common theme in com- puter vision applications- we can construct D that consists of image rotation augmentations, e.g.. d(a, b;r)= (rot,(a, r), rot rot(b,r) where rot r) rotates the image x by r E R [0, 360) degrees (note that both a and b are rotated by the same amount). Another example may be, for time-series data of a linear time-invarian (LTI) system (commonly assumed in signal processing and control theory (Oppenheim & Schafer 2009)) ) we can construct D with time delay and amplitude scaling, e.g., d(alt) bld]; 7) = (oalt 7 abft - -r), a, T E R = R x Z. Additional remarks on the practical applicability of RPDA provided in Appendix A.3 The full training procedure for VRL
with RPDA is summarized in Appendix B
4 RELATED WORK
Existing work on relational learning (Definition 1.] are mostly based on supervised learning ap- proaches (Lu et al. 2012: Puebla et al., 2021). The rapid growing field of statistical relational learning (SRL) concerns domain models with relational structure (Getoor & Taskar 2007 De Raedi et al., 2016). Popular SRL methods for link prediction and classification that are based node attributes similarity (Takkar et al., 2004] Yu et al. 2007) directly contradicts our goal of learning a relational property that is independent of absolute property (node attributes). Other SRL methods based on matrix factorization (Singh & Gordon 2008 Menon & Elkan 2011), probabilistic logic
2015 Manhaeve al. 2018), or probabilistic relational models (Fried- man et al. 1999; Getoor et al.
programming (De Raedt et al.
(007) typically require some supervision for learning (e.g.. labeled
examples, known dependency type, etc.) which we do not assume in our work; moreover, most of these methods allow, even encourage, data relationship to depend on its absolute property. Other recent work focus on high-level cognitive tasks, such as visual Q&A and state prediction for complex- physics systems, and derive their relational processing capabilities from learning with clever designed neural networks (Wu et al. [013] Reed et al. 2015) |Fragkiadak et al. [0066] Chang et al.) 2017] Battaglia et al.) 2016 Santoro et al. |217]] Battaglia et al.) [0188] van Steenkiste et al.) 2018 Kipi ct al. 2018). Our work differ from these methods in two fundamental ways: (I) we focus on learning an independent relational property in completely unsupervised fashion; (2) we derive
relational learning capability from a PGM formulation, which gives us the flexibility to use any compatible inference method or function approximation.
Many existing unsupervised learning methods can also be applied to our problem setting (Song et al.
2007 Mikolov et al. 2203aa b. Kingma & Welling, 2014 Goodfellow et al., [2144] Makhzani et al. 2015
Dilokthanakul et al. (017);; however, most of these methods learn a single representation with
superimposing information about the relational and absolute property. The difficulty of decoupling the relational property from the learned representation constitute a major obstacle to relational learning. closely related method is proposed by Guu et al. (Guu et al., 2018). but their learned latent "edit vector" (loosely related to our relational property) may be coupled with absolute property.
Other related work include methods on learning a disentangled representations with applications in style-transfer, image-to to-e-ttt-inng to-image translation, domain adaptation, etc. (Bousmalis et al. 2016 Mathieu al. 2016 Chen et al., 2016) Higgins et al.) 2017 Denton & Birodkar 2017: 2017]HHuag et al. 2018] Chen et al. 2019). Most of these methods strive to learn disentangler representations of content and style (or pose for video sequence data) where content is generically defined as the underling spatial structure, and style as the rendering of the structure. In comparison, our work can be viewed as learning a disentangled representations of relational and absolute property. We argue that style- content separation is fundamentally different from pllational-asoolan separation; more specifically, we consider both style and content information as absolute property (both describe features of an individual data), while relational property provides additional information about data relationships.
5 EXPERIMENT
In this section, we present experimental results from applying VRL to a variety of relational learning tasks. In our implementation, we parameterize po( b ( ((0) and go z a ( br()) with fully- connected networks (MLPs) f! (a()) 20) and (a(a( br())) respectively. For binary valued data,
let Po bl a()) 200) be a multivariate Bernoulli distribution whose probability parameters
5

---

Published as a conference paper at ICLR 2022
computed from fg (a(), (002),z))) 2 For real-valued data, we let pe b aaa),2) (z()) be multivariate Gaussian distribution with fixed diagonal covariance and the mean is computed from Sg (a((), 22))). We experimented with both continuous and discrete r.v. z. For continuous Z, we let the prior pe 2 be
bivariate normal distribution z .N(O,) I) and qo z | (8) by(() be bivariate Gaussian distribution with diagonal covariance whose mean and covariance are computed from ( (()) br()), ). For discrete
2, we adopted two categorical r.v., 2 [21:2), 22). each having a uniform prior over five categories and let 9 z a b)()) represents two categorical r.V. reparameterized with Gumbel-Softmat
distributions whose class probabilities are computed from (a (1) b(()) (Jang et al. 2017 Maddison
iet al., 2017). For RPDA, we used random image rotation as data augmentation functions. Parameters
0 and Φ were jointly trained to maximize C in Eq. 4 using Adam optimizer (Kingma & Ba 2015). Full implementation details are provided in Appendix E. and source code for reproducible results is available online
5.1 RELATIONAL LEARNING WITH DECOUPLED RELATIONSHIPS
We first present two relational learning tasks designed with the MNIST (LeCun & Cortes 2010) and Omniglot (Lake al. 2015) datasets (dataset information and data preprocessing is described in Appendix D). A paired MNIST (and Omniglot) dataset X M (and X { (a(), b(t) ie [1..N] is constructed where each a is a randomly rotated MNIST (and Omniglot) binary image and b(t) is another random counter-clockwiss rotation of (0) by 09, 72", 144 216". or 288 There are five uniquely defined relative rotational relationships between (a b (9) in X= (and X furthermore, the relative relationships are decoupled from their absolute properties, i.c. the relationship cannot be inferred from (8)) (+) alone. The goal of relational learning is to discover the underlying relative rotational relationship. We first demonstrate VRL's relational discrimination and relational mapping capabilities on the Omniglot relational learning task:
Relational discrimination. We used the trained posterior g z a, b) to infer the relational property of a hold-out dataset constructed from the evaluation alphabets. Figure [a shows scatter plot of the relational property inferred by VRL where we can see that the approximated posterior accurately cluster (discriminate) data with the same (different) relative rotational relationship together (apart).
Relational mapping. We used the trained likelihood Po ( [ a. z )) to generate images from given and z. We chose a from a hold-out dataset and from: (1) direct sampling the latent space; (2) relational property inferred from a source data point (a,, ba). Figure F shows predicted images with z sampled from the latent space shown in Fig. 3a Figure 3c shows examples of relational mappings
from a (c) to b (ne) by applying the relational property inferred from a source data point (a..b!")
Next, we compare VRL with other unsupervised and self-supervised learning methods: Spatial transformer networks (STN) (Jaderberg et al. 2015) (we tested two variations of STN that uses its localization network to learn about affine transformation, denoted as STNN-afinee, and rotation transformation, denoted as STN-rotate): Variational autoencoder (VAE) (Kingma & Welling 2014);
VAE with Gaussian mixture prior (GMVAE) (Dilokthanakul et al., 2017); Adversarial autoencoder (AAE) (Makhzani et al., 2015); VAE with added contrastive latent loss (VAE-contrastive) (Chen et al.,
2020); self-supervised representation learning method B YOL (Grill et al. 2020); Learning linear structure in representation space through vector arithmetic (Vec-arithmetic) (Mikolov et al.
301
Radford et al. 2016); Learning independent causal mechanisms (LICM) (Parascandolo et al. 2018)
Neural relational inference (NRT) with graph neural networks (Kipf al. 2018). VAE, GMVAE, AAE, VAE-contrastive, BYOL, and Vec-arithmetic all trained with dim Each baseline method assessed on both MNIST and Omniglot relational learning tasks through unsupervised clustering. For fair comparison, we adopted image rotation augmentation during the training of all the baseline methods. For VAE-contrastive and BYOL, we used random image rotation augmentations to generate "positive paris" for training. In our comparison, STN-affine and STN-rotate are considered as the benchmark methods as they are architecturally designed to learn about the spatial transformation between ( and ((.) In contrast, VRL does not make any assumption about the underlying relationship. Implementation details for the baseline methods are provided in Appendix E.2
hhaps//github.coo//ke iu/vrll..g

---

Published as a conference paper at ICLR 2022
(3)
23)
(3)
(5)
A
6
2
-400
-400
6:00
K00
(a)
(b)
(c)
Figure 3: Learning decoupled relative rotational relationship with VRL: (a) Scatter plots of 2-D relational property of X inferred by VRL (relative rotational relationship labels: O:0p v 72" +: 1444, :116, & : );8"); (b) Images predicted from sampled latent variables (sampling the centroid of each cluster in (a): "O" "0"+z(") "v" +
(3) "x" +z(4) "Q" 2 +2(");; each image (n.e), < r, S 5, was predicted from a (c) (shown in the top row) and (0) by b(ne) ~
po( b a (c) z ( (c) Relational mappings of top row images by applying relational property inferred (5)
from pairs of source images (a,, b. shown in the left-most column with a,, b.²) (1) ..... b. arranged from top to bottom; each image b(r))), S r,e 5 was generated by b(rse) pa( b a a()) ()))))
where (r) y0( z aa. bb²))
To quantitatively evaluate the clustering performance, we calculate the classification error rate of a hold-out dataset based on simple classifier that was trained on 5% of the data with label. For continuous latent space, we trained a standard multi-class support vector machine classifier with radial basis function kernel (Manning et al.) 2008)) For discrete (categorical) latent space, we follow the evaluation protocol on clustering assignment described in Makhzani et al. (2015). Results are summarized in Table where each entry reports the average and standard deviation over 5 runs with different random seeds and each run consists either 3 (randomly selected out of 5) or 5 relative rotational relationships. We can that the proposed VRL with RPDA achieved high accuracy in recovering the relative relationships for all the tasks and outperformed other baseline methods including the benchmark STN-affine and STN-rotate. To gain insight into the information-shortte problem and the proposed mitigation strategies introduced in Sec. 3.2 we performed an ablation study on the prior selection of z and RPDA. The results are summarized in Table ! and we make the following observations: First, representing z as a discrete categorical (Cat.) r.V. limits its expressiveness and improves VRL's performance over continuous prior (Cont.); Second, RPDA is a critical component necessary for VRL to learn meaningful and independent relational property, especially when flexible function approximations such as deep neural networks are used. Additional experimental results and analysis are provided in Appendix
5.2 RELATIONAL LEARNING WITH COUPLED RELATIONSHIPS
To further test the robustness and generalizability of VRL, we consider a scenario where the relative relationship is coupled with the absolute property. To setup this experiment, we modified the construction of xMM, denoted as Ma, so that the relative rotational relationship between (+) and (6) is completely determined by its digit representation (absolute property of ( and ((t)): a( E r'' ['8', '9"] > 288 (read: (1) digit "0" or '1' then the relative rotational relationship between (a (() is 09). The question then arise: is VRL capable of learning an independent relational property even when the underlying relative relationship is coupled with the absolute property? To test this idea, we trained VRL on Ma but validated it on a hold-out dataset of X.M which has a decoupled relative relationship. The results shown in Fig. 4 where indeed capable of learning an independent relational property irrespective the digit representation. If this were not the case, we would expect to scatter plot Fig. 4a with heavily overlapped
0% (+) E ['2', "3'] -3]+772 72 a E ['4', '5'] + 144", (+)) E (''6', '7"] -77-+7 216", (i))
see that VRL was
7

---

Published as a conference paper at ICLR 2022
Table Unsupervised clustering accuracy (in %6) for MNIST and Omniglot relational learning tasks.
MNIST
Omniglot
Num. relationships
5
3
VRL (Cat., no RPDA)
93.9+6.7
46.9+7.4
99.9+0.1
22.4+0077
VRL (Cat., RPDA)
99.8+0.1
99.9+0.1
99.6+0.3
99.9+0.1
VRL (Cont., no RPDA)
60.117.1
28.3
96.416.7
32.1114.9
VRL (Cont., RPDA)
99.810.2
99.910.1
99.810.1
97.71006
STN-affine
33.71111
19.810.9
34.1100.
19.9
STN-rotate
98.3+1.
97.2=0.8
47.4126.1
30.2#13.1
VAE
93.8
60.642.3
98.2
63.0
GMVAE
73.913.7
57.242.5
47.7114
38.8
AAE 49.1+1.2
31.1+2.
66.5+4.2
33.3+1.2
VAE-contrastive
92.4+-3.8
60.5+2.7
93.4+6.3
59.4+4.4
BYOL 49.1112.7
27.7124
36.711.7
23.3
Vec-arithmetic
64.711119
54.311.6
53.5114.1
50.811.
LICM 36.643.3
19.7=1.1
35.0+2.5
21.6+2.
NRI 34.5+112
20.5+1.0
34.6+009
21.1+00.9
relative relationship labels since X XM was constructed with random relative rotational relationships. Figures (ajures/4Jaarree and and 4c further demonstrate VRL generalizes well to unseen data-VRL. trained on X XMa
learned to rotate any digit by any amount despite not having seen most of the digit-rotation pairs during training.
3
3
5
6
(1)
3
5
2
3
0
7
5
9
2
9
8
33
5
7
5
7
6
$
6
4
2
4
(5)
3
6
3
J
9
+
(a)
(b)
(c)
Figure Learning coupled relative rotational relationship with VRL: (a) Scatter plot of 2-D relational property of XM M inferred by VRL (relative rotational relationship labels: : V 72 + 144 x: x 21116, 288%): ) (b) Images predicted from sampled latent variables (sampling the centroid of each cluster in (a): "O" =1(1), "7" 22) "4" (3), "x" ( "ه" z (5)); (c) Relational
mappings of top row images by applying relational property inferred from pairs of source images (a,, b(²)) in the left-most column.
5.3 RELATIONAL LEARNING WITH HIGH-LEVEL PERCEPTION REASONING
Finally, present results of using VRL learn about relative facial expression changes, facial illumination condition changes, and speech emotion changes. We extracted images of three facial expressions (happy, surprised, sad) of each subject from the Yale Face Database (Belhumeur et al. 1997) form a paired dataset X where each data point (a(1), b()) represents subject with different facial expressions. Next, we extracted images of four illumination conditions (left (L), front (F), right (R), top (T)) of each subject from the Extended Yale Face Database B (Georghiades
al. 2001) to form paired dataset X where each data point (a ((0) represents subject with
8

---

Published as a conference paper at ICLR 2022
different illumination conditions. Lastly, extracted speech waveforms (represented as log-mel spectrogram) of three emotions (calm, angry, fearful) of each voice actor from the Ryerson Audio- Visual Database of Emotional Speech and Song (RAVDESS) (Livingstone & Russo 2018) to form paired dataset X where each data point (a(), b(o) represents a voice actor with different emotions. Dataset information and data preprocessing is described in Appendix D Due to the extremely limited data samples in X (45 unique images), X (112 unique images), and X (144 unique speech recordings), we focus on learning undirected relative relationship. i.e., (a(*) (+)) (()) and (b(),, a(") have the same relative relationship. this case, X X and X consist of 3, 6, and undirected relative relationships. respectively. For RPDA, adopted random image rotation for the VRL training on X Fe and X and random time delay and amplitude scaling for X due to its temporal data characteristic; furthermore, since augment RPDA with random swapping operation (see Appendix E. for implementation details). Due to the limited datasets, both training and validation are performed on the entire dataset. The inference results from a trained posterior do z a, b are shown in Fig. 5 where we can see that VRL is effective in learning a relative relationship that independent of subject's identity, facial expression, illumination condition, speech content, and emotion. Comparing our results with existing unsupervised methods on facial images (Song et al., 2007; Wu
only interested in learning undirected relationships,
2013 Shi et al.) 2018 Tapaswi et al. 2019) and speech emotion recognition (Neumann & Vu 2019 Li et al. 2021), we remark that existing methods cluster data by their absolute property (e.g., subject identity, speech emotion, etc.), while VRL clusters data by their relational property (e.g., relative facial expression, illumination, or
speech emotion changes). Additional relational mapping results are provided Appendix F2
45
-30
13
390
49
16 15 -050
-0.00
-1000
860 0200
100
(b)
(c)
Figure 5: Learning relative relationships between facial images and speech waveforms: (a) Learn- ing relative facial expression changes with 2-D VRL relational property (labels: v:"happy-sad" "thhappp-surprieede
+.""surprised-saa") (b) Learning relative facial illumination condition changes with 2-D VRL relational property (labels: V:"L-R, FFT C: "L-F, L:"LT, F:FR, R:"-TT); (c) Learning relative speech emotion changes with 2-D VRL relational property (labels: V:"calm- fearful", "almm-aangr "earfury-fearrul")
DISCUSSION AND CONCLUSION
The proposed VRL method comes with both advantages and disadvantages: the main advantage of VRL is its relational learning capabilities; however, this may also be one of its disadvantages. More specifically, VRL can learn an independent relational property even when it is coupled with the absolute property (see Sec. 5.. i.e., VRL oblivious to the coupling information between the two properties. Nevertheless. such information may be of interest to the user, and in this regard, VRL only provides partial view the data.
In conclusion, the proposed VRL method is an efficient and effective unsupervised learning method for addressing the relational learning problem where our goal is to learn an independent relational property. By dissecting the data information into decoupled relational and absolute property, hope VRL can bring new insight into everyday data analysis and ultimately find applications for a wide variety of problems.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGMENT
We thank anonymous reviewers for comments that helped improve the paper. We would also like thank Huseyin Denli, Antonio Paiva, Ashutosh Tewari, Myun-Seok Cheon, and Peng Xu (ExxonMobil) for insightful discussion and valuable feedback.
REFERENCES
Peter Battaglia, Razvan Pascanu, Matthew Lai, Danilo Jimenez Rezende, and koray kavukcuoglu. Interaction networks for learning about objects, relations and physics. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016.
Peter W. Battaglia, Jessica B. Hamrick, Victor Bapst, Alvaro
Vinícius Flores Zambaldi, Matcusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, Çaglar Gülçehre. H. Francis Song, Andrew J. Ballard, Justin Gilmer, George E. Dahl, Ashish Vaswani, Kelsey R. Allen, Charles Nash, Victoria Langston, Chris Dyer, Nicolas Heess, Daan Wierstra, Pushmeet Kohli, Matthew Botvinick, Oriol Vinyals, Yujia Li, and Razvan Pascanu. Relational inductive biases, deep learning, and graph networks. CoRR, abs/1 806.01261 2018.
N. Belhumeur, J. P. Hespanha, and J. Kriegman. Eigenfaces vs. Fisherfaces: recognition using class specific linear projection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 19(7):711-720, 1997.
Irving Biederman. Recognitionbyy-comoonnttt A theory of human image understanding. Psycho- logical Review, 94:115-147, 1987.
Christopher M. Bishop. Pattern Recognition and Machine Learning. Springer, 2006.
Konstantinos Bousmalis, George Trigeorgis, Nathan Silberman, Dilip Krishnan, and Dumitru Erhan. Domain separation networks. In D. Lee, M. Sugiyama, U. Luxburg. I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016.
Michael B. Chang, Tomer Ullman, Antonio Torralba. and Joshua B. Tenenbaum. A compositional object-based approach to learning physical dynamics. In Sth International Conference on Learning Representations, ICLR 2017, April 24-26 2017, Toulon, France, 2017.
Mingda Chen, Qingming Tang, Sam Wiseman, and Kevin Gimpel. Controllable paraphrase generation with a syntactic exemplar. In Proceedings of ACL, 2019.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations, In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 1597-1607. PMLR, 13-18 Jul 2020.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Pro- ceedings of the 30th International Conference Neural Information Processing Systems, NIPS' 16, pp. 21 80-2188, Red Hook, NY, USA, 2016. Curran Associates Inc. ISBN 978 10833881..
Luc De Raedt, Anton Dries, Ingo Thon, Guy Van Den Broeck, and Mathias Verbeke. Inducing probabilistic relational rules from probabilistic examples. In Proceedings of the 24th Interna- tional Conference on Artificial Intelligence, IJCAI'15, pp. 1835-1843, AAAI Press, 2015. ISBN 9781577357384.
Lue De Raedt, Kristian Kersting, Sriraam Natarajan, and David Poole. Statistical Relational Artificial Intelligence: Logic, Probability, and Computation. Morgan & Claypool, 2016.
Emily Denton and Vighnesh Birodkar. Unsupervised learning of disentangled representations from video. In I. Guyon, U. V. Luxburg, S. Bengio. H. Wallach, R. Fergus, S. Vishwanathan, and R. Gameet (eds.), Advances in Neural Information Processing Systems 30, pp. 4423-4423. Curran Associates, Inc., 2017.
10

---

Published as a conference paper at ICLR 2022
Nat Dilokthanakul, Pedro A. M. Mediano, Marta Garelo, Matthew C. H. Lee, Hugh Salimbeni, Kai Arulkumaran, and Murray Shanahan. Deep unsupervised clustering with gaussian mixture variational autoencoders, 2017.
Hao Dong, Akara Supratak, Luo Mai, Fangde Liu, Axel Oehmichen, Simiao Yu, and Yike Guo. Tensorlayer. Proceedings of the 25th ACM international conference on Multimedia, Oct 2017. doi: 10.1 10.114551123226.31299
Leonidas A. A. Doumas and John E. Hummel. Comparison and mapping facilitate relation discovery and predication. PLOS ONE, 8(6):1-8, 06 2013, doi: 10.137 (ow..mmma...omec00688889999
Katrina Evtimova. Learning independent causal mechanisms for sepsis. https://githut com/
kevtimova
2017.
Ethan Fetaya. Neural relational inference for interacting systems.
https://github.cr com/
ethanfetaaa/nnri 2019.
Katerina Fragkiadaki, Pulkit Agrawal, Sergey Levine, and Jitendra Malik. Learning visual predictive models of physics for playing billiards. In Yoshua Bengio and Yann LeCun (eds.), 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016.
Nir Friedman, Lise Getoor, Daphne Koller, and Avi Pfeffer. Learning probabilistic relational models. In Proceedings of the 16th International Joint Conference Artificial Intelligence Volume 2, ICCAI'9, pp. 1300-1307, San Francisco, CA, USA, 1999, Morgan Kaufmann Publishers Inc.
D. Gentner and L. Smith. Analogical reasoning. In V. S. Ramachandran (ed.), Encyclopedia of Human Behavior. Elsevier, Oxford, UK, 2012.
Dedre Gentner. Why we 're so smart'. Dedre Getner and Susan Goldin-Meadow (eds.), Language in Mind: Advances in the Study of Language and Thought, pp. 195-235. MIT Press, 2003.
A.S. Georghiades. The Yale Face Database B. /ttp://vision ucsiionnussdeee. wdw.d...ddd....ee.eeeee.. http://iissoonuusd...edd//aaaaaa satyaaaeeattbaassaaallee2aaa..a
htm, 2001. Accessed: 2021-05-27.
A.S. Georghiades, P.N. Belhumeur, and D.J. Kriegman. From few to many: Illumination cone models for face recognition under variable lighting and pose. IEEE Trans. Pattern Anal. Mach. Intelligence, 23(6):643-66) 2001.
Lise Getoor and Ben Taskar. Introduction to Statistical Relational Learning (Adaptive Computation and Machine Learning). The MIT Press, 2007. ISBN 0262072882.
Lise Getoor, Nir Friedman, Daphne Koller, Avi Pfeffer, and Ben Taskar. Introduction to Statisti- cal Relational Learning (Adaptive Computation and Machine Learning), chapter Probabilistic Relational Models. In Getoor & Taskar (0007). 2007. ISBN 0262072882.
Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014.
dean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, koray kavukcuoglu, Remi Munos, and Michal Valko. Bootstrap your own latent a new approach to supervupe learning. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 21271-2122 Curran Associates, Inc., 2020.
Kelvin Guu, Tatsunori Hashimoto, Yonatan Oren, and Percy Liang. Generating sentences by editing prototypes. Transactions of the Association for Computational Linguistics, 6(0))437-400, 2018. ISSN 2307-387X.
11

---

Published as a conference paper at ICLR 2022
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew M Botvinick, Shakir Mohamed, and Alexander Lerchner. beta- VAE: Learning basic visual concepts with a constrained variational framework. In ICLR, 2017.
Robert V. Hogg. Joseph McKean, and Allen T. Craig. Introduction to Mathematical Statistics.
Pearson, 6 edition, 2005.
Keith Holyoak. Analogy and relational reasoning. Keith Holyoak and Robert Morrison (eds.), The Oxford Handbook of Thinking and Reasoning, chapter 13, pp. 234-259 Oxford University Press, New York, 2012.
Xun Huang, Ming- Yu Liu, Serge Belongie, and Jan Kautz. Multimodal unsupervised image-to-image translation. In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss (eds.), Computer Vision ECCV 2018, pp. 179-196. Cham, 2018. Springer International Publishing. ISBN 978-3-030-01219-9.
Max Jaderberg, Karen Simonyan, Andrew Zisserman, and koray kavukcuoglu. Spatial transformer networks. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. In 5th International Conference Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In Yoshua Bengio and Yann LeCun (eds.), 2nd International Conference Learning Representations, ICLR 2014, Banff. AB, Canada, April 16, 2014, Conference Track Proceedings, 2014.
Thomas Kipf, Ethan Fetaya, Kuan-Chieh Wang, Max Welling, and Richard Zemel. Neural relational inference for interacting systems. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 2688-2697, PMLR, 10-15 Jul 2018.
B. M. Lake, R. Salakhutdinov, and J. B. Tenenbaum. Human-level concept learning through proba- bilistic program induction. Science, 350:1332-1338 2015.
Yann LeCun and Corinna Cortes. 2010.
MNIST
handwritten
digit
database.
http://yam.leccu.comm/eeee
Mao Li, Bo Yang, Joshua Levy, Andreas Stolcke, Viktor Rozgic, Spyros Matsoukas, Constantinos Papayiannis, Daniel Bone, and Chao Wang. Contrastive unsupervised learning for speech emotion recognition. In ICASSP 2021 2021 IEEE International Conference Acousties, Speech and Signal Processing (ICASSP), pp. 6329-6333, 2021.
doi: 10.1 1011999CASS333992222221.
Steven R. Livingstone and Frank A. Russo. The Ryerson Audio-Vinn Visual Database of Emotional Speech and Song (RAVDESS): A dynamic, multimodal set of facial and vocal expressions in North American English. PLOS ONE, 13(5):1-35, 05 2018. doi: 10.137 1/jormall pone. 0196391.
Hongjing Lu, Dawn Chen, and Keith Holyoak. Bayesian analogy with relational transformations. Psychological review, 1193))617--648 July 2012. ISSN 0033-295X. doi: 10. 10.1007/a0028719 19.
Chris J. Maddison, Andriy Mnih and Yee Whye Teh. The concrete distribution: A continuous relax- ation of discrete random variables. In 5th International Conference Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, and Ian Goodfellow, Adversarial autoencoders. CoRR, abs/1511 1, 1.05644, 2015.
12

---

Published as a conference paper at ICLR 2022
Robin Manhaeve, Sebastijan Dumancic, Angelika Kimmig, Thomas Demeester, and Lue De Raedt. DeepProbLog: Neural probabilistic logic programming. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianche and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
Christopher D. Manning, Prabhakar Raghavan, and Hinrich Schütze. Introduction to Information Retrieval. Cambridge University Press, USA, 2008. ISBN 0521865719.
Michaël Mathieu, Junbo Zhao, Pablo Sprechmann, Aditya Ramesh, and Yann LeCun. Disentangling factors of variation in deep representations using adversarial training. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS' 16, pp. 5047-5055, Red Hook, NY, USA. 2016. Curran Associates Inc. ISBN 9781510838811
Douglas L. Medin, Robert L. Goldstone, and Dedre Gentner. Respects for similarity. PSYCHOLOGI- CAL REVIEW, 100:254-278, 1993.
Aditya Krishna Menon and Charles Elkan. Link prediction via matrix factorization. In Dimitrios Gunopulos, Thomas Hofmann, Donato Malerba, and Michalis Vazirgiannis (eds.), Machine Learning and Knowledge Discovery in Databases, pp. 437-452, Berlin, Heidelberg, 2011. Springer Berlin Heidelberg. ISBN 978-3-644-233783-6
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. In C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems 26, pp. 3111-3119, Curran Associates, Inc., 2013a.
Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. Linguistic regularities in continuous space word representations. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 746-751, Atlanta, Georgia, June 2013b. Association for Computational Linguistics.
Michael Neumann and Ngoc Thang Vu. Improving speech emotion recognition with unsuper- vised representation learning on unlabeled speech. In ICASSP 2019 2019 JEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7390-7394, 2019. doi: 10.1 N0.1109/ICASSP.200 1.1...441.
Alan V. Oppenheim and Ronald W. Schafer. Discrete-Tine Signal Processing. Prentice Hall Press, USA, edition, 2009. ISBN 0131988425.
John Paisley, David M. Blei, and Michael I. Jordan. Variational bayesian inference with stochastic search. In Proceedings of the 29th International Coference on International Conference on Machine Learning, ICML' 12, pp. 1370-1370, Madison, WI, USA, 2012. Omnipress. ISBN 97814503122511
Giambattista Parascandolo, Niki Kilhertus, Mateo Kojas-Carulla, and Bernhard Schölkopf. Learning independent causal mechanisms. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 4036 4044. PMLR, 10-15 Jul 2018
Derek C. Penn, Keith J. Holyoak, and Daniel J. Povinelli. Darwin's mistake: Explaining the discontinuity between human and nonhuman minds. Behavioral and Brain Sciences, 31(2): 109-130, 2008 doi: 10. 0001017750140555000055533
Luis Perez and Jason Wang. The effectiveness of data augmentation in image classification using deep learning. CORR, 2017.
Guillermo Puebla, Andrea Martin, and Leonidas A. Doumas. The relational processing limits of classic and contemporary neural network models of language processing. Language, Cognition and Neuroscience, 36(2):240-254, 2021. doi: 10.1008/23273798..2 2020. 1821906.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. In Yoshua Bengio and Yann LeCun (eds.), 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016.
13

---

Published as a conference paper at ICLR 2022
Scott E Reed, Yi Zhang, Yuting Zhang, and Honglak Lee. Deep visual analogy analogy-making, In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett (eds.). Advances in Neural Information Processing Systems 28, pp. 1252-1260. Curran Associates, Inc., 2015.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Nassir Navab, Joachim Hornegger, William M. Wells, and Alejandro F. Frangi (eds.), Medical Image Computing and Computer- 234-241 Springer International Publishing. 2015. ISBN 978-3-319-24574-4.
Intervention MICCAI 2015, pp.
Adam Santoro, David Raposo, David G Barrett, Mateusz Malinowski, Razvan Pascanu, Peter Battaglia, and Timothy Lillicrap. simple neural network module for relational reasoning. In Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 4967-497777 2017.
Curran Associates, Inc.,
Yichun Shi, Charles Otto, and Anil K. Jain. Face clustering: Representation and pairwise constraints IEEE Transactions on Information Forensics and Security, (47)::622-16400 Jul 2018. ISSN 1556-6021.
Ajit Singh and Geoffrey J. Gordon. Relational learning via collective matrix factorization. In Proceedings of the 14th ACM SIGKDD International Conference Knowledge Discovery and Data Mining, KDD '08, pp. 650-658, New York, NY. USA. 2008, Association for Computing Machinery. ISBN 978160558119 1934. doi: 10.1145/1401890.11 1401969.
Le Song, Alex Smola, Arthur Gretton, and Karsten M. Borgwardt. A dependence maximization view of clustering. In Proceedings of the 24th International Conference on Machine Learning, pp. 815-822, 2007.
Makarand Tapaswi, Mare Law, and Sanja Fidler. Video face clustering with unknown number of clusters. In 2019 /EEE/CVF International Conference on Computer Vision (ICCV), pp. 5026-5035, 2019. doi: 10. 1119//CC..20190005555
Ben Taskar, fang-fai Wong, Pieter Abbeel, and Daphne Koller. Link prediction in relational data. In S. Thrun, L. Saul, and B. Schölkopf (eds.), Advances in Neural Information Processing Systems, volume 16. MIT Press, 2004.
Sjoerd van Steenkiste, Michael Chang, Klaus Greff, and Jürgen Schmidhuber. Relational neural expectation maximization: Unsupervised discovery of objects and their interactions. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018.
Gary VandenBos and American Psychological Association. APA Dictionary of Psychology. American Psychological Association, Washington, DC, edition, 2007.
Baoyuan Wu, Yifan Zhang, Bao-Gang Hu, and Qiang Ji. Constrained clustering and its application to face clustering in videos. In 2013 IEEE Conference on Computer Vision and Pattern Recognition, pp. 3507-3514, 2013. doi: 10.1 109/CVPR.2013..0.
Jajun Wu, Ilker Yildirim, Joseph Lim, Bill Freeman, and Josh Tenenbaum. Galileo: Perceiving physical object properties by integrating physics engine with deep learning. In C. Cortes, N. D. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems 28, pp. 127-135, Curran Associates, Inc., 2015.
Kai Yu, Wei Chu, Shipeng Yu. Volker Tresp, and Zhao Xu. Stochastic relational models for discriminative link prediction. In B. Schölkopf, J. Platt, and T. Hoffman (eds.), Advances in Neural Information Processing Systems, volume 19. MIT Press, 2007.
14

---

Published as a conference paper at ICLR 2022
A ADDITIONAL REMARKS
A.1 REMARKS ON EQUATION 1
To provide additional insight into Eq. ① we give the following example (attributed to S. Bern- stein (Hogg et al., 2005)): Let a and b be two independent tosses of a fair coin, where we designate I for heads and 0 for tails. Let third random variable be equal to if exactly one of two coin tosses resulted in "heads". ,, and otherwise. It is easy to see that this example r.v. a, b ans z satisfy all four conditions in Eq. ① and can intuitively interpret a relational property since it indicates the relationship between a and (same vs. different) and is independent of and b, i.e., z undertermined when observing b alone. The main difference between this example and our definition in Eq. (T) is that we do not specify what z is; we only assume the existence of and our goal is to learn about
meaning "different tosses" which is
A.2 REMARKS ON VRL-PGM
The proposed VRL-PGM reflects our priority and compromise for using a PGM to represent the relational learning problem: we sacrifice some identifiability of the original abstract problem but obtain a rigorous and tractable model that achieves our objective of learning an independent relational property. Here we discuss aspects of the original relational learning problem (see Section (2) that differ from the proposed VRL-PGM (see Section 3.1.. First, the original problem description in Eq. specifies that the relational property z be independent of both a's and b's absolute properties; however, learning and inferencing relational property z that satisfies both independence constraints is a challenging problem; therefore, as a compromise, VRL-PGM only enforce the independence constraint between and a. Second, the original problem is inherently undirected with no cause-effect relationship between a and b, whereas VRL-PGM is based directed acyclic graph (DAG) that artificially introduces conditional dependency between a and b. We note that this can be viewed as the result of VRL-PGM choosing to satisfy Eq. I0)
(iv) It is equally valid for VRL-PGM to ((vv) and it would lead to the same model but with a and swapped. In other words, the application of VRL does not require the true conditional dependency between (a, b) be known in advance only that it is maintained consistently throughout learning and inference. The mbove-mentioned discrepancies represent the compromises we made with adopting VRL-PGM in exchange for a rigorous and tractable method for learning an independent relational property. Finally, these compromises are not been made without consequences: information-shortear is direct result of not enforcing independence constraint between z and b, and deterministic-mappinn can be viewed as caused by the causal relationship VRL-PGM introduced between and b.
choose to satisfy Eq, I(i) I(ii)
A.3 REMARKS ON RPDA
Here we discuss the practical applicability of the proposed RPDA strategy. More specifically, argue that in many practical problem settings, the RPDA functions D can be designed without any knowledge of the underlying relational property. For example, as we have explained in Sec. [.2 in many computer vision applications, rotation invariant is a desirable property for the learned model; for example, in spectral imaging applications, oftentimes the orientation of the images are preserved or not enforced only that they are consistent between the same paired images (Ronneberger al. 20155.. In such problem setting, we can safely use image rotation function for constructing D. Another example may be: for a discrete time-series data aft), b(t) that represent the input and output of linear time-invariant (LTT) system (commonly assumed signal processing and control theory (Oppenheim & Schafer 2009)), and we want to learn relational property that characterize the system's 's impulse response. We have ab|t- - 7] = aajt 7]. Va E I,, 7 E. and we can construct D with d(a[t), b[d]; a. r) = (oa 7], abjt - -]), a, T E R
= R x Z. In all of the above examples, the construction of RPDA functions reflects our prior knowledge and belief of the underlying system and not based on the underlying relational property. However, just like any data augmentation, the effectiveness of RPDA will depend on the problem setting and we advocate to start without RPDA and only apply it when suspecting information-shorteur occurs.
15

---

Published as a conference paper at ICLR 2022
B VRL ALGORITHM
Algorithm 1 VRL with RPDA
procedure VRL(X, p(c), D) b If RPDA not available, D= (id()) (a. b) id(a, b)
Initialize parameters 0
while not convergence of parameters (0, do
Sample minibatch ((a(9, b(0) [ from X. Run RPDA and obtain (a b (() = d(a b();(()), r(0 10)02( ~(()) i=1,... M.
Compute gradients 9 Vo.efRPDN (see Eq. (4)
Update parameters 0, & using gradients g (e.g., SGD).
end while return 0,0
end procedure
C DERIVATION OF VARIATIONAL LOWER BOUND
To derive a variational lower bound for VRL-PGM, first write the log-evidence logpa( (X) log po( { (a(9) b(0) i [1.NN]] EN logro( (+)), b(9) ), where each term in the summation
can be expressed as:
log (2) b(0) DKL. (go(2 a aa00,b() po ( (()))))))))
+ R log pa( u, (1)), ((() log go a b()) (5)
The first term on the right-hand side (RHS) is the KL-divergence from pa a b()) to go(2 z a b00) ), which provides a measure of dissimilarity between the two distributions; the second term the RHS continues
log po z, a (+))) b()) - log g0 a (())))
go ((
log po b (i) a z)po( z ( a ) - log g0 z a ab.0))))))) }]
E (2
(2(0()))) logpo( pa (
|
+ log po 2) log as 2 a (0) b(o) log po ())) (6)
where in the second line use the fact that r.v. and independent. Substitute Eq. (6) back in (5) and rearrange terms gives us the expression for Eq.
log po ()) (a))))
DKI. (go( a
(()) ())b())) ) pa( 2 aa)),
aa. (b())
g(1
where
c
E (2la [log po (a) ,2) log (z) log go( z/a b (2,()),()))
The term ( serves as a lower bound for the conditional log-likelihood log pa ((()) aa()) since KL-divergence is non-negative. Maximizing ((,) w.r.t. à and 0 gives us both a ML estimate for Po( b a. z)) (by maximizing the first term inside the expectation in (()) and a lower KL-divergence
(the better go( a (+), b()) approximates the true posterior Pa z a ((() )) the conditional log-likelihood log Pa (() a()) does not depend ø. The lower bound (()) can be maximized with gradient ascend methods; however, its gradients w.r.t. ø is difficult to obtain: the expectation in (Paisley et al. 2012). To obtain efficient estimators for both (() and its gradients, we adopt the reparameterization trick developed in Kingma & Welling (2014) where the r.v. is expressed a transformation of another r.v.
(()) is taken w.r.t. the distribution q z a b((), ). which is a function
p(e e) that is independent of b, and ø: g(e, ( ((() Φ) where g is some differentiable and invertible transformation. Given such change of variable, the lower bound (()) can be rewritten
as:
(()) = Ep(c) log (3)
+ log 2 - log gol z b(,)) b0))
(8)
16

---

Published as a conference paper at ICLR 2022
where g(e, ,b(", and e ~ p(E))))))))))))))) Note that the expectation in Eq. 8 is taken w.r.t. p( and we can approximate (() with a Monte Carlo estimator:
L
po Po b (a(0, 20000) ,
+log (oggoo))))) 211))) - log loggo(2) (2(()) |a (,))
where ((.)) - g(E(1.)) a b(,, Φ) and ((.)) ~ p(E). An estimator for Camna in Eq. (4) follows similarly as:
L log pa b(o) (0) (a(1,,2 26,0 ) log (1D)) )- - log gol (a.) (0))) b(())
ERADA
(10)
T.
1=1
where (a (+) b ) = d(a ((0)),))))) ,0( ~ (R),,
and (1..) = g(e . (1) br()) 0), e(id)) p(e). The lower bound for minibatches of data XM Ny
{ (a b(0) [I] can be approximated by (am d; X M
Σ M 2ii And finally,
the gradients V Cxnua (0, o; X N Ei=1 Vo can be computed in straightforward manner and used to update the parameters 0 and ø with stochastic optimization methods, such as SGD.
M
D DATASET AND DATA PREPROCESSING
We used the following datasets in our experiments:
MNIST dataset (available under CC BY-SA 3.0 license) consists of 28 x 28 grayscale images of standard bandwritten digits with labels; the dataset is composed of 60,000 training samples and 10,000 testing samples (LeCun & Cortes 2010).
Omniglot dataset (available under MIT license) contains 1623 different handwritten characters (105 105 grayscale images) from 50 different alphabets: the dataset split into a background (training) set of 30 alphabets and an evaluation (testing) set of 20 alphabets (Lake et al. 2015).
Yale Face Database (see (Geerghiadess 200T) for dataset permission) contains 165 grayscale face images of 15 human subjects under 8 facial expressions and 3 illumination conditions (Belhumeur
ct al. 1997)
Extended Yale Face Database B (see (Georghiades) 2001) for dataset permission) contains 16128 grayscale face images of human subjects under poses and 64 illumination conditions (Georghi ades et al. 2001).
Ryerson Audio-Visui Database of Emotional Speech and Song (RAVDESS) (available under CC BY-NC-SA 4.0 license) is speech and song dataset containing 24 professional actors (12 female, 12 male) vocalizing two exrically-matcher statements in a neutral North American accent with wide range of emotions (Livingstone & Russo 2018).
Each handwritten character image in MNIST and Omniglot dataset is resized to 32 x 32 binary image. Each facial image in Yale and Extended Yale dataset is center-cropped, resized 64 64, and normalized pixel values to be within [0, 5). Each speech waveform in RAVDESS dataset is trimmed remove silences both the start and the end, and then clipped zero- pero-padded to be seconds long. We then represent each waveform by its log-amplitude mel spectrogram with 128 mel bands and frame size of 512 samples. Finally, each log-mel spectrogram image is resized to 64 x 64. Examples of facial images from Yale and Extended Yale, and log- mel spectrograms of RAVDESS speech waveforms are shown in Fig
legm
E IMPLEMENTATION DETAILS
E. VRL IMPLEMENTATION
Here we describe the details of our VRL implementation. We parameterize both pa (bbl) a ((()))
and go( a (1) b(()) using fully-connected networks (MLPs) with rectified linear non-linearities.
17

---

Published as a conference paper at ICLR 2022
(a)
(b)
(c)
Figure 6: Examples facial images and speech waveforms: (a) Subjects (columns) with different facial expressions (rows): happy, surprised, sad; (b) Subjects (rows) with different illumination conditions (columns): left, front, right, top: (c) Log-mel spectrogram of actors vocalizing two statements with different speech emotions (Columns: 1. male actor vocalizing "Kids are talking by the door", 2. female actor vocalizing "Kids are talking by the door", 3. male actor vocalizing "Dogs sitting by the door", 4. female actor vocalizing "Dogs sitting by the door"; Rows: calm, angry. fearful).
For binary valued data such as MNIST and Omniglot, we let pe [ ((fl)) be a multivariate Bernouli distribution whose probability parameters p p(t.) are computed from MLPs fo (a(*) 20:0)::
f (a()) (1)) FC(512) FC(256) + FC(512) - +p (1.)
(11)
The reconstruction eπror in this case is the binary cross-entropy: log Po ((() a z(0) = b log + (1 - b()) log(1 - ((.)). For real-valued data such as Yale facial images and RAVDESS log-mel spectrograms, we let po b a(!), ((()) be multivariate Gaussian dis- tribution with a fixed diagonal covariance Nbb; (t.t)) I² where (i is computed from MLPs fg ()) ());
S/ (a 21109 - FC(512) - FC(256) + FC(512) + (81)
(12)
The reconstruction error in this case is the mean squared error: log po ((()) lb(0 (1.0 + const. We experimented with both continuous and discrete (categorical) latent variable z. For continuous z, we let the prior po( 2 be a bivariate normal distribution and
(())
2 0.0
the approximated posterior 9o( (1()) bz(+) be bivariate Gaussian distribution with diagonal
covariance (2z (1) (a ( where yu (9) and (i) are computed from MLPs (a (() br()):
fg
[a (1) b b FC(512) + FC(256) (A (0)
(13)
In this case the latent variable samples are drawn as: z (8.1 (0) + (i)) @ eli,), c(i.)) (O. I). For discrete we used two categorical r.v., z [2,, z2). each having uniform prior over five
categories and let q z (()) br(() represents two categorical r.v. r.v. reparameterized with Gumbel-
Softmax distributions whose class probabilities π. = TI....., T15) and n. (T21,...,, T25)) are
computed from MLPs (a
b((()) (Jang al. 2017
Maddison
2017):
(0 FC(512) + FC((56) + [M], 22
(14)
In this case the latent variable samples are drawn as: 2 (i.)) softmax (( (1.1) + log //)T)) j 1,2
where (i.1) 5 are i.i.d. samples drawn from a Gumbel(0,) 1) distribution and the softmax temperture 7 controls the "smoothness" of the samples.
For MNIST, Omniglot, and Yale relational learning tasks, we used random image rotation for RPDA. ie., D { (rot(a,7) r), rot(b, r)) re (0,360) For time-series RAVDESS speech waveforms, used random time delay and amplitude scaling for RPDA, i.c., D= (aajt r], abjt - r]) | a e
R, E Z Additionally, since we only interested in learning undirected relative relationship
18

---

Published as a conference paper at ICLR 2022
changes in Yale and RAVDESS relational learning tasks (i.e., po(z a) po(z po(z|b,a))
augment RPDA functions with random swapping operations:
swap(a', b') [ - ((b' a'), = 0.5
((a', b'), p= 0.5
7(0) Finally, the learning objective RPDA in Eq. 10 can be constructed once pe b ( z((()) ). pe (2 z and do( z (() by()) are defined. For example, when we adopt a discrete latent variable = [2], /2] for
MNIST and Omniglot relational learning tasks (Sec. 5.1 and 5.2), we derive L (with L 1)
as:
((
Σ ((0) log p (i.1) + (1-b log(1 -p.ppp..) ((.1)) - Σ T log n + const
(15)
RPDA
where (a ) = (rot(a, (()), (ot(bbr(())) r(0) ~ 4((0,360)) [T] π2l = (a (9) br()).
(i.1)
soflmax((e(11) + log म /)/T), ((11)) Gumbel 1), = 1,2. and ((.1)
2
(a(a), (1.2) 22 1). Similarly, RRDA (with L 1) for continuous latent variable can be
((3
derived as:
20
Σ bb log (.1)) + b()) log(1 - ((.))) + (oz (10),,
log N(z (1.1) (1.)
(0 (1)))
(16)
where (a b b(0)) = (rot(a, p()), rot(b, (b, r(())), r() 4((0 360)), [M (0) (a b(O)), (0.1) 4 ( (1.11), ((i.1) (0, 1), and p (f.11) = (a z (.)) For Yale and RAVDESS
relational learning tasks (Sec. 5.3). we adopted a continuous latent variable z with a 1 and RPDAA (with I. 1) can be derived as:
700 -1
0.02 [b 1 122 + log N(2 (1.1), (.11) log ((z (
(0) + const (17)
where (a)()) br(ily
swap(rot(a, r()), (()),, r()) u(10, 360)), !n!! o())
f (a/(0) br((), ), ((11)) = 100) + @ ((i1), ((i.1) ~ I),,I), and p(bi)) = f8 (al)) (1.1))
All MLPs with parameters 0 and were jointly trained for 300k iterations (without batch- (())
normalization, weight decay, nor dropout) to maximize RODA in Eq. 4 with using Adam optimizer (learning rate=0.0004, 3,,009 8 =09999 (Kingma & Ba 2015). Minibatches of size M=100 were used. We anneal the learning rate (0.0004 base learning rate) with step decay (factor of 0.5 every 100k iterations). When Gumbel-Softme distributions used, we anneal the softmax temperature from 1.0 to 0.5 with exponential decay (decay 0atc=0.)
E.2 BASELINE METHODS IMPLEMENTATION
STN-affine and STN-rotate (Jaderberg et al. 2015 Dong et al. 2017) minimize the difference between (()) and geometric transformation of ( (affine transformation for STN-affine and rotation transformation for STN-rotate) by using spatial transformer with localization network that takes both a (6), b(1 as input; during evaluation, we perform clustering the output of the trained localization network. The localization network in STN-affine takes both a b(() as input and outputs an aftine transformation matrix:
aa b 6 FC(512) + FC((56) + FC(256) + AA =
00
012
013
Oz1
Oz2
023
In STN-rotate, we further restrict STN-affine to only allow rotation transformation and the localization network in this case only outputs the rotation angle A0:
[cos( A0)
sin A0) cos A0) 0
0
aa
(8) FC(512) + FC((56) + FC((56) + A0, Ag = sin AO)
VAE, AAE, GMVAE all trained directly using (a(), b (() and we perform clustering on their learned latent space. We selected dim(z)=10 via cross-validation. We follow the implementation of (Kingma
19

---

Published as a conference paper at ICLR 2022
& Welling 2014 Makhzani et al., 2015 Dilokthanakul et al., 2017) since they also experimented with MNIST or Omniglot dataset. However, for fair comparison, we also experimented with the following encoder/decoder architecture and report the best results:
1 FC(512) - FC((56) + (0) ~ (() - FC(256) - FC((12) - (ao.ybo)
Our VAE-contrastive implementation is similar to Chen et al. (2020) in concept. We added an additional latent loss term to the VAE loss function that aim minimize the difference between the latent representation from different rotation augmented image pairs, i.c., a alz z²(² (also
experimented with cosine similarity) where z Enc([a!) ,b b())), z (8) Enc([a" (0) b ()) and we cross-validate a from 0.1 to 10.
BYOL learns joint image represenation for (a(),b b()), on which the clustering is applied during evaluation. We selected dim(z)=10 via cross-validation.
Vec-arithmetic first trains autoencoder individual a (+), (() and then used the trained encoder to compute a latent vector representation of (a(), b()) through Enc(b()) Enc(a()), on which the clustering is applied during evaluation. We selected dim(z)=10 via cross-validation.
LICM (Evtimova. 2017) trained a committee of 5 experts to learn a set of independent mechanisms that generates (dd) from the canonical (²)), and used the trained discriminator during evaluation to select the winning expert (as categorical variable).
NRI infer the interaction between (*) and ((4) and used the NRI-Encoder to predict the relationship type (as a categorical variable). follow the implementation (Kipf al. 2018: 8 Fetaya 2019)
where we used the NRI-Encoder infer the relationhip between (+) and b(,; the inferred relationship
( together with (a()) b()), are then feed into the NRI-Decoder predict ( and уы which
reconstructions for a and (((), respectively. To account for the high-dimensional input data, used two MLPs (one for each ( and (()) to extract lower dimensional feature vectors Za( R
and 2 (4) 10 that serve as input to NRI- -nccdeno
(4) FC(512) FC(256) X 6 FC(512) FC(256) Xb
Similarly, we insert two MLPs (with the same architecture above) in front of NRI-Decode to extract data feature vectors and append two MLPs after NRI-Decoder to convert the predicted feature vectors back the high-dimensional data space:
( + FC(256) > FC(512) Ybe,,
Xb()) + FC(512) FC(256) Yb6))
All MLPs are trained jointly with NRI encoder and decoder.
For our final baseline comparison, we present results from applying InfoGAN to the MNIST rela- tional learning task in Sec. 5.1 (Chen al., 2016). InfoGAN has demonstrated its ability learn disentangled representations (represented by structured latent codes Cı C2, ..,, Cr.) through generative modelling. Although inferring latent codes for given data point is a non-trivial task for InfoGAN, we can examine the learned latent representation by manipulating the latent codes and visually inspect the generated random samples. We modeled the latent codes with one categorical code Cat(P K = 10, 0.1) and two continuous codes 02, ~ Unif(- 1, Figure 7 shows examples of generated images from manipulating the latent codes and it is clear that none of the latent codes distinctively capture the full range of relative rotational relationships.
F ADDITIONAL EXPERIMENTAL RESULTS AND ANALYSIS
F.1 MNIST AND OMNIGLOT RELATIONAL LEARNING RESULTS ANALYSIS
At first glance, the results in Fig. 3c resemble that of transfrr,,,fer but they fundamentally different: in transfer,,,, the image b (nie) is generated by applying the style of b((r) to the content of a (0), whereas VRL generates image b (r:e) by applying the relational property of (a,, bd²)) to the
image ( It is evident from Fig. 3c that predicted images b(re) do not share similar style b!!). but rather the same relative rotational relationship w.r.t. (e) and a,-
20

---

Published as a conference paper at ICLR 2022
V
7
J
G
G
U
C
G
V
L
V
V
Y
G
G
G
6
6
(a)
(b)
Figure 7: Manipulating latent codes of InfoGAN on MNIST where each row represents random samples from varying continuous latent code 02 in (a) and c3 in (b) while other latent codes and noise are fixed; different rows correspond to different categorical code e₁ = 1 1.... 10.
A close inspection of the results in is in fact very challenging even for specialized methods such as STN-affine and STN-rotate that only learn about spatial transformations. Comparing VRL with VAE, we can see that VRL achieved better performance than VAE with more efficient Darming-VAE were only able to solve the simpler 3-relationship tasks by adopting high dimensional latent space: in contrast, VRL were able to solve all tasks with a compact 2-D latent space. Further comparing VAE with GMVAE and AAE shows that regularizing the latent space, as done in AAE and GMVAE, can degrade the performance of relational learning. For contrastive self-supervised learning methods like VAF-contrastive and BYOL there is no way to enforce the learned data representation be independent of absolute property since different augmented image still share large part of their absolute property (c.g., digit representation). For example, it is entirely possible for VAE-contrastive and BYOL to tightly cluster images based on their digit representation (absolute property): this will lead to a small loss function value but makes the subsequent relational discrimination task more challenging since each digit representation can have all possible relative rotation relationships. LICM failed all relational learning tasks due to the fact that the transformed and canonical distributions overlaps completely (both a and (() rotations of MNIST or Omniglot images) and it is not possible for committee of experts to distinguish the two. NRI's inability to solve the relational learning tasks can be attributed to the fact that it is designed to learn in dynamical systems, but more importantly, GNN is not guaranteed to learn an independent relative relationship.
that the seemingly simple relational learning tasks
In summary, we argue that a major challenge for applying existing methods to relational learning problem is that they learn a single representation that encodes both relational and absolute properties and it difficult dissect the relational property from the learned representation.
F.2 YALE AND RAVDESS RELATIONAL MAPPING RESULTS
Figure shows examples of relational mappings predicted by the same set of trained VRL. models
from Sec. 5.3
F.3 RELATIONAL LEARNING WITH MULTIPLE RELATIONSHIPS
Here, we setup a more complex relational learning task that includes both relative rotational and scaling relationships. We constructed paired MNIST dataset X Mio = { (a(), b(0) | [I..N]] where each (+) is randomly rotated and scaled (by factor x0.66 or x1) MNIST image and b(0) is another random rotation (by 0%, 722, 144%, 216", or 288") and scaling (by x1 or x1.5) of ali) Note that there are a total of 10 decoupled relative relationships between (a b(0)) in X Mi
(combinations of 5 rotational and scaling transformations). We trained VRL on X Mio following the same training procedure described in Appendix E but with larger ML.Ps for (a 20) and
(a(()
21

---

Published as a conference paper at ICLR 2022
(b)
(a)
Figure 8: Examples of VRL relational mapping for facial images and speech waveforms; each pair of images shows VRL relational mapping predictions based on the relational property inferred from the first two images in the same row; (a) Relative facial illumination condition changes; (b) Relative facial expression changes; (c) Relative speech emotion changes.
(a b)()) (cf. Eq.
11
13
to account for the increased problem complexity:
f8 |a
(1,0) + FC(1024) - FC(512) ~ FC(512) - p (8.1)
(18)
S (0)
FC(1024) FC(512) - (0) (1)
The inference result is shown in Fig. 9a where cluster (discriminate) data with the same (different) relative relationship together (apart). Examples of images predicted by direct sampling the latent space shown in Fig. 9b
that the approximated posterior accurately
F.4 RELATIONAL LEARNING WITH CONTINCOUS RELATIONSHIPS
Lastly, we present an example with a continuous relational property. Based on the MNIST dataset, we constructed a paired dataset X M. = (a()) (()) i [1..N] where both ( and b(() are random rotation of the same MNIST image.
this case, there is a continuous (and decoupled) relative rotational relationship between (a (()). We trained VRL on X following the same training procedure as described in Appendix B but used convolutional neural networks (CNNs) to capture the continuous relationship. We approximate po( b a(i), ((d))) ) with an autoencoder-like neural network fg (3) f8 fdee (f (a(*), (i:))):
0 ca (6) Conv(3x3x) - Conv(3x3x32) Conv(3x3x128) + FC((0) h e 2 ddee [h (1) 20 z (1.)
FC Conv (3x3x128) Conv (ox3x(3 Conv Conv (3x3x8)
Sigmoid [0, diim(())
- +Conv(Ixxx1)
where Conv(-) is strided (stride 2) convolutional layer and Conv"(.) is a transposed convolutional layer. We used natch-normalizztion after each layer and rectified linear non-linearities. We represent the approximated posterior qs( z (+) ()))) (with bivariate normal distribution as prior) using:
(a b b00) C3nx(333x8) Conv(3x3x32)
Conv(3x3x128)
FC(() ( (0)
22

---

Published as a conference paper at ICLR 2022
2
3
Q
7
2-
6
7
8
41
12
à
9g
2 (7)
(8)
(9)
(10))
(b)
Figure 9: Learning multiple relative relationship with VRL: (a) Scatter plot of 2-D relational property
of X Mio inferred by VRL (relative relationship labels: O(blue) 00, V 72% + 144 x 216
Q 288", O 0", x x1.5, D 72", x x1.5, < 144", x1.5, A : 216", x x1.5, O(cyan) 288", x1.5); (b) Images predicted from sampled latent variables (sampling the centroid of each cluster in (a): (blue) =((1, "v" (2))) >2 (3) "x" 22(1), "3" z5) "O" (6) "p" +2(7, "4" (9))) "C"(cy))) (10))
scatter plot of the relational property inferred by a trained posterior is shown Fig. T0a and examples of images predicted by direct sampling in the latent space (denoted by markers "x" in we can see that VRL learned an independent relational
Fig. 10a) are shown Fig. 10b From Fig. 10
20
2
3
5
6
7
8
9
300
Z
1)
1
3
3
4
5
6
9
8
9
9
5
6
0
350
3)
9
3
6
5
Z (5)
C
71
(a)
(b)
Figure 10: Learning continuous relative relationship with VRL: (a) Scatter plot of 2-D relational property of Mc inferred by VRL, each point is color-coded (best viewed in color) by the degrees of relative rotation between the corresponding data point; (b) Images predicted from sampled latent variables (denoted by markers "x" in (a)).
property that encodes continuous relative rotational relationship: however, there is a small region in Fig. T0a with overlapping relational property that leads to an ambiguous relationship interpretation (120' 240%). This ambiguity is likely caused by compressing the relative relationship down to 2-D latent space, z E R², and motivates us to adopt higher- higher-dimensional latent space, e.g., z E R3 Figure 11 shows inference result from repeating the previous experiment but with adopting R33 We can see that VRL. learned a three-dimensional relational property that unambiguously represents the underlying continuous relative rotational relationship.
23

---

Published as a conference paper at ICLR 2022
300
180
30
Figure 11: Scatter plot of 3-D relational property of X Ma inferred by VRL (with R²); each plot shows a different vantage point of the 3-D scatter plot, and each point is color-coded (best viewed in color) by the degrees of relative rotation between the corresponding data point.
24


Reviews:
[{"review_id": "vCOYyWrAlz", "reviewer": "Reviewer_7vpw", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\nThe problem set up is novel, and the paper addresses an important problem. The method is intuitive, and the RPDA is novel. The experiment is well-conducted, and shows that the method outperforms other strong baselines.\n\nWeaknesses:\nIn Eq. 1, the authors introduces 4 conditions for z to be satisfied as relational property. However, the proposed PGM only satisfies 3 of them where (ii) is not satisfied. This is an important omission. The authors may need to consider ways to make it consistent, or justify such omission.\n\n--\nUpdate:\nI have read the reply from the author as well as other reviewers' comments. It is understandable that the 4 conditions are hard to achieve simultaneously, and I'm satisfied with the authors' response. Considering the weaknesses of the experiments as raised by other reviewers, I remain my score of borderline accept.", "summary_of_the_paper": "This paper introduces a variational method for relational learning. It first introduces relational learning as learning based on relational property instead of absolute property, and introduces conditions (Eq. 1). Then it proposes VRL-PGM with a variational lower bound. To eliminate the information short-cut, it introduces relation-preserving data augmentation (RPDA). Experiments show that the method is able to perform relation discrimination and relation mapping, on a variant of MNIST and Yale face datasets.", "summary_of_the_review": "In summary, the authors addresses an interesting problem and proposes a novel method, and overall an interesting paper. The authors may need to improve its justification of the PGM omitting condition (ii), or find ways to make it consistent with the proposed 4 conditions of relational variable.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "YJy79wB__fT", "reviewer": "Reviewer_Znwn", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- the idea of learning relations solely from existing relations is interesting\n- the theoretical background is simple and well defined\n\nWeaknesses:\n- datasets are only images\n- from the authors' example (happy-sad faces), it seems absolute properties are indirectly being used\n- how does this work differ from others that work on complex networks, trying to extract motifs from graphs?\n\n", "summary_of_the_paper": "This paper proposes a relational learning method based on variational Bayes. The main idea is to learn relations among objects independently of each object's own properties. The model theory is discusses where authors define the problem and discuss about solutions for some limitations of the model when a relation between objects A and B can be found by only looking at one of the objects own properties.\nThe paper is quite well written. Algorithms are provided, but not the code. As far as I understood, results are obtained on \"benchmark\" datasets, where some synthetic versions of these datasets are used to create the relations between the instances of the original dataset and of the synthetic.", "summary_of_the_review": "Overall, I liked the idea of formulating the problem as \"learning relations from relations\". However, there are other approaches that are not mentioned here which try to find \"motifs\" in graphs or perform link prediction, where the objects (without their properties) are represented as nodes, relations are in the edges and new links are learned from the original graph.\n\nFrom your example (sad-happy faces), it seems features of each image are being used to learn new relations. Why did you choose images for training? I would think that a tabular dataset (specially in areas such as health) could benefit much from your approach.\n\nReferences are outdated. The last year for your refs is 2019. There has been a whole body of work regarding relational learning and methods developed in 2020 and 2021.\n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "VFJWN4GEEXf", "reviewer": "Reviewer_ahUC", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "A new relational learning approach is proposed based on variational learning. The main idea is to disentangle absolute and relational properties and force the learner to learn based on relationships rather than absolute properties of instances. The problem is formulated as a variational inference problem and a data augmentation method is developed that preserves relational information. Experiments are shown on MNIST, monoglot and Yale face database for unsupervised learning.\n\nThe paper has an interesting idea and I liked the formulation of the variational inference problem for relational learning. The main weakness is perhaps the somewhat forced construction of relationships in the empirical results. Since the paper has a very general aim of learning relationships, it would have been really strong if this could be shown through a natural relational problem rather than the constructed ones. The results do show that the method performs better than even a specialized transformer model for rotations (STN) which is definitely.a plus. On the other hand, the RPDA functions need to encode the rotational aspects so it does require some specialization. I was not sure about the data augmentation RPDA functions for the facial emotions data (was it related to illumination changes, etc.).  Also, how different is the relational data augmentation from the typical types of data augmentation methods that people would use if they had the domain knowledge, maybe the speciality of the relational augmentation methods could be highlighted a bit better.\n\nAfter author feedback\n\nI thank the authors for their feedback. The idea definitely seems good, maybe with a bit stronger comparison/ empirical studies, this would be a stronger paper.", "summary_of_the_paper": "A new approach for relational learning is proposed. The main contributions are\nA novel formulation of relational learning as a variational inference problem\nA novel augmentation method for relational data\nEmpirical results that show the ability to learn relationships in images.", "summary_of_the_review": "Overall the formulation seems interesting and results show that the general approach is on par with state of the art specialized methods for the tasks considered. However, the relationships in the experiments seem a bit forced and maybe simpler data augmentation methods could replace the relational augmentation methods used.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "SVbYbRWX1ij", "reviewer": "Reviewer_2C7q", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I have 2 concerns regarding the presentation/organization of the paper:\n1. The authors claim in the introduction that they formulate the problem of relational learning as a PGM which seems misleading. They approximate the problem because the omission of the independence condition is an important omission. In fact the authors themselves shed light on what could go wrong because of the omitted condition and use the RDPA trick to address it.\n2. The relational learning problem needs better discussion. In particular, the transition from the definition of relational learning to the probabilistic formulation could use an example.\n\nOther than this, I believe that this is an interesting paper with a novel formulation of the relational learning problem and extensive experimentation.", "summary_of_the_paper": "In this paper the authors pose the problem of relational learning in the form of a Probabilistic graphical model and utilize variational inference for learning the relational property. The experiment section is extensive highlighting the utility of the proposed method.", "summary_of_the_review": "Good paper help back by the presentation.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "HdRVIgUTth", "reviewer": "Reviewer_Jew5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strength\n\n- The relational learning problem is clearly formulated in the paper, and seems novel and interesting.\n- The paper is well written and easy to follow.\n- The experiments are well-designed and demonstrate interesting behavior of VRL.\n\n### Questions/concerns\n\n- In Table 1, why is STN-Rotate so much worse on Omniglot than on MNIST? Omniglot seems pretty similar to MNIST, and the two datasets are similarly constructed. Furthermore, STN-Rotate is essentially cheating (as it knows the transformations are rotations) so I would expect it to perform well. It is quite surprising that it does so badly on Omniglot.\n- The comparison with STN seems a bit strange. Why does STN take both a and b as input? I would imagine an STN that takes just a as input and we compare the output with b. This is basically directly learning the transformations. In addition, for STN, clustering on the output of the trained localization network does not seem fair. Depending on how the STN is specified, I can imagine different outputs of the trained localization network that lead to the same transformation. Directly comparing the resulting transformations seems like the better approach.\n- The VRL-PGM is not symmetric w.r.t. a and b. Does that affect the model in any way? Is this asymmetry considered in the training (e.g. randomizing the order of the two elements in each training example) and test process?\n- For the Omniglot experiments, VRL with discrete no RPDA for 5 relationships is worse than VRL with continuous with no RPDA, so the claim that constraining z to be discrete improves performance does not always hold.\n- How does the rotation augmentation work for some of the baseline methods, e.g. for VAEs? It seems by use of RPDA the model is cheating a bit, as it essentially has additional supervision signal coming from access to two pairs of samples that it knows share the same transformation. To make the comparison fair, for VAEs for example, in addition to training VAEs on the given training data, there should be additional supervision that the latent state for a pair of samples and the latent state for the pair of samples transformed by a relation preserving function should match. I don't know if this is the way rotation augmentation is used for VAEs. If not, it doesn't seem to be a fair comparison. Similar issues might exist for other baseline methods (although in terms of performance VAEs seem the most promising). If VAEs perform well with this additional piece of supervision information, then the good performance of VRL is more from the way RPDA is used, and less from the method itself.\n- How does relational mapping look like on the Yale Face dataset? Some visualizations would be helpful. I expect it won't be able to generate transformed faces as that is quite complicated, but it would be interesting to see what the model really learned.\n", "summary_of_the_paper": "This paper proposes an unsupervised learning method, variational relational learning (VRL), to address the relational learning problem and learn the underlying relationship between a pair of data irrespective of the nature of those data, and demonstrates its performance on variations of the MNIST/Omniglot datasets and the Yale Face datasets.", "summary_of_the_review": "I have some questions/concerns about the paper so recommend weak reject, but would be happy to bump up the score if my questions/concerns can be satisfactorily addressed.\n\n------------------------------\nThe authors have satisfactorily addressed my concerns, and I bumped up the score accordingly.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This papers studies the classical problem of relational learning from a probabilistic perspective. The authors propose four reasonable constraints to encode relational properties, and develop a PGM-based variational method for learning relational properties from data. After extensive discussion with the authors, a majority of the reviewer reviewers agree the approach is interesting, if not without some flaws.

The problem studied is interesting, novel, and could lead to new developments in the area of relational learning. It is expected that the experiments have some limitations given the authors have approached the problem from a fresh new angle, which the reviewers have appreciated.

Please pay attention to the suggestions from the reviewers, and in particular, please add a more detailed discussion with statistical relational learning: This material may not be familiar to the broader ML audience, and therefore it is essential to make these comparisons explicit.
Decision: Accept (Poster)