Year: 2022

Abstract:
In psychology, relational learning refers to the ability to recognize and respond to relationship among objects irrespective of the nature of those objects. Relational learning has long been recognized as a hallmark of human cognition and a key question in artificial intelligence research. In this work, we propose an unsupervised learning method for addressing the relational learning problem where we learn the underlying relationship between a pair of data irrespective of the nature of those data. The central idea of the proposed method is to encapsulate the relational learning problem with a probabilistic graphical model in which we perform inference to learn about data relationship and other relational processing tasks.

heading: 1 INTRODUCTION
text: American Psychological Association defines relational learning as (VandenBos & APA, 2007): Definition 1.1 (Relational learning). Learning to differentiate among stimuli on the basis of relational properties rather than absolute properties.
In other words, relational learning refers to the ability to recognize and respond to relationship (called relational property) among objects irrespective of the nature of those objects (called absolute property). For example (attributed to Doumas & Hummel (2013)), how do we come to understand that two circles are the same-shape in the same way that two squares are? In this example, “same-shape” is the relational property and object shape is the absolute property. Relational learning has long been recognized as a hallmark of human cognition with strong implications for both human-like learning capabilities and generalization capacity (Biederman, 1987; Medin et al., 1993; Gentner, 2003; Penn et al., 2008; Holyoak, 2012; Gentner & Smith, 2012). We refer the interested readers to the provided references for a comprehensive discussion on this subject. Contemporaneously, the research on learning data relationships—also commonly called “relational learning”—has flourished in the machine learning community where the overarching goal is learning in a context where there may be relationships between learning examples, or where these examples may have a complex internal structure (i.e., consist of multiple components and there may be relationships between these components) (Getoor & Taskar, 2007; De Raedt et al., 2016). We argue that the key difference between the two “relational learning” definitions and their learning objectives is that Definition 1.1 takes the relationship learning problem one step further by requiring the data relationships be learned only on the basis of relational properties rather than absolute properties. To the best of our knowledge, this important distinction—learning relationships irrespective of the absolute properties—has not been rigorously studied in the unsupervised learning community, where most existing methods either encourage or do not constrain the relationships learning through absolute properties.
In this work, we propose an unsupervised learning method—variational relational learning (VRL)— for addressing the relational learning problem as defined by Definition 1.1. At its core, VRL encapsulates the relational learning problem with a probabilistic graphical model (PGM) in which we perform inference to learn about relational property and other relational processing tasks. Our contribution in this paper is threefold: First, we propose a probabilistic formulation for the relational learning problem defined by Definition 1.1. Second, we encapsulate the relational learning problem with a PGM in which we perform learning and inference. Third, we propose an efficient and effective learning algorithm that can be trained end-to-end and completely unsupervised. ∗The author completed this research while working at ExxonMobil. Corresponding author e-mail: liu.kuanghung@gmail.com

heading: 2 PROBLEM DEFINITION
text: We focus on a canonical form of the relational learning problem where we observed a paired dataset X = { (a(i),b(i)) | i∈ [1..N ] } consisting of N i.i.d. samples generated from a joint distribution p(a∈A,b∈B ). We dissect the information in X into absolute property and relational property where absolute property represents specific features that describe individual a and b, and relational property represents the relationship between a and b irrespective of their absolute property. In this work, we interpret the absolute property of a and b as any information that characterizes (even if only partially) the marginal distribution p(a ) and p(b ). We propose to represent the relational property as a latent random variable (r.v.) z that satisfies the following constraints:
(i) p(a, z ) = p(a )p( z ), (ii) p(b, z ) = p(b )p( z ),
(iii) p(a, z | b ) 6= p(a | b )p( z | b ), (iv) p(b, z | a ) 6= p(b | a )p( z | a ), (1)
where in Eq. 1(i) and 1(ii) we interpret the specification of relational property in Definition 1.1— learning relationships irrespective of the absolute properties—as meaning statistical independence, while in Eq. 1(iii) and 1(iv) we ensure r.v. z contains relevant (relationship) information that further informs a and b about one another, i.e., H(a | b, z) < H(a | b), H(b | a, z) < H(b | a) where H(· | ·) is the conditional entropy. It is easy to see that the following conditions are necessary for r.v. z to exist: (1) H(b | a) > 0 and H(a | b) > 0, i.e., a and b cannot be fully determined by each other; (2) r.v. a, b, z are not mutually independent, i.e., p(a,b, z ) 6= p(a )p(b )p( z ). Our goal for relational learning is to learn about relational property z that satisfies Eq. 1 in a completely unsupervised fashion. A motivating example for Eq. 1 is provided in Appendix A.1.
In addition, we are interested in two related relational processing tasks: relational discrimination and relational mapping defined as (VandenBos & APA, 2007):
Definition 2.1 (Relational discrimination in condition). A discrimination based on the relationship between or among stimuli rather than on absolute features of the stimuli.
Definition 2.2 (Relational mapping). The ability to apply what one knows about one set of elements to a different set of elements.
Relational discrimination allows us to differentiate (a(i),b(i)) from (a(j),b(j)) based on their relational properties. And relational mapping allows us to apply the relational property of (a(i),b(i)) to a different set of data, for example, deduce that b(j) is related to a(j) in the same way that b(i) is related to a(i).

heading: 3 METHOD
text: Learning and inference relational property z that satisfies all four constraints in Eq. 1 is a challenging problem due to the hard independence constraints in Eq. 1(i) and 1(ii). To overcome this challenge, we first introduce VRL as a tractable learning method that satisfies 3 (out of 4) constraints in Eq. 1— Eq. 1(i), 1(iii), 1(iv). We then discuss VRL’s unique optimization challenges, which are partially attributable to its relaxation of the independence requirement in Eq. 1(ii).

heading: 3.1 VARIATIONAL RELATIONAL LEARNING
text: The proposed VRL method consists of two parts: first, we encapsulate the relational learning problem with a PGM, called VRL-PGM; we then formulate various relational processing tasks as performing inference and learning in VRL-PGM. The VRL-PGM model, shown in Fig. 1, samples data a, z, and b from parametric families of distributions—pθ(a ), pθ( z ), pθ(b |a, z )—that are differentiable almost everywhere with respect to (w.r.t.) a, z, and θ. In practice, we observe only a set of independent realizations { (a(i),b(i)) | i ∈ [1..N ] } while the true parameter θ∗ and the corresponding latent variables z(i) are unobserved. A well-known property of the PGM shown in Fig. 1 is that r.v. a and z are independent with no variables observed, but not conditionally independent when b is observed, i.e., pθ(a, z ) = pθ(a )pθ( z ), pθ(a, z | b ) 6= pθ(a | b )pθ( z | b ) (Bishop, 2006). Consequently, VRL-PGM can be viewed as a parametric relational learning model that satisfies 3 (out of 4) constraints in Eq. 1—Eq. 1(i), 1(iii), 1(iv) (note that Eq. 1(iv) is trivially satisfied in VRL-PGM).
Further discussions on the connection between VRL-PGM and the relational learning problem is provided in Appendix A.2.
Having established VRL-PGM, our primary learning objective is to approximate the unknown true likelihood function pθ(b | a, z ) and posterior pθ( z | a,b ). Learning pθ( z | a,b ) provides us a way to infer (a(i),b(i))’s relational property z(i); moreover, it serves as a basis for performing relational discrimination where we compare relational properties between different pairs of data. Learning pθ(b | a, z ) allows us to perform relational mapping where we use the relational property of (a(i),b(i)) to map a(j) to b(j), i.e., b(j) ∼ pθ(b | a(j), z(i) ) where z(i) ∼ pθ( z | a(i),b(i) ). We estimate the parameter for pθ(b |a, z ) by following the maximum-likelihood (ML) principle, and approximate the true posterior pθ( z | a,b ) with variational Bayesian approach. More specifically, we use a variational distribution qφ( z | a,b ), parameterized by φ, to approximate the unknown (and often intractable) true posterior. Both θ and φ are learned through maximizing a variational lower bound, L(θ, φ;a(i),b(i)) (abbreviated as L(i)), for the conditional log-likelihood log pθ(b(i) | a(i) ) (derivation is provided in Appendix C):
L(i) = Eqφ(z|a(i),b(i)) [ log pθ(b (i)|a(i), z ) + log pθ( z )− log qφ( z|a(i),b(i) ) ] . (2)
Recall that learning z independent of a is central to our relational learning goal. While this independence assumption is built into VRL-PGM, the learning objective L(i) does not explicitly force z to be independent of a nor penalize learning a dependent z. In practice, there may be numerous reasons that could break this independence assumption, e.g., insufficient training data, failure to reach the global optimum, non-identifiability of the model, etc., and it may be desirable to explicitly enforce independence between z and a. One way to achieve this is to introduce a non-positive function that measures the dependency between a and z with maximum attained when they are independent. For example, we can append the negative mutual information between z and a, −I(z ; a) = −DKL(pθ( z,a ) ‖ pθ( z )pθ(a )), to L(i):
L(i) = Eqφ(z|a(i),b(i)) [ log pθ(b (i)|a(i), z ) + log pθ( z )− log qφ( z|a(i),b(i) ) ] − I(z ; a). (3)
Since I(z ; a) ≥ 0 and I(z ; a) = 0 if and only if z and a are independent, the addition of −I(z ; a) to L(i) not only maintain the validity of the lower bound, but also retain its quality (z and a are independent in VRL-PGM).

heading: 3.2 OPTIMIZATION CHALLENGES
text: A limitation of the proposed VRL method is its inability to enforce the independence constraint between z and b in Eq. 1(ii). This may lead to a worst-case scenario of learning a degenerated posterior qφ( z | a,b ) = qφ( z |b ) where relational property z only depends on the absolute property of b. Another limitation of VRL is exposed when the assumption H(b | a) > 0 is violated and b can be completely determined by a. In this case, a degenerated likelihood pθ(b|a, z ) = pθ(b|a ) may be fitted and relational property z plays no role in the learning process. To further explain these limitations and potential mitigation strategies for avoiding the worst-case learning scenario, we first dissect VRL’s gradient updating process into the following steps (using a single data point (a(i),b(i))
as an example): (1) sample z(i) ∼ qφk( z | a(i),b(i) ) by using the current parameter φk; (2) evaluate L(i) and calculate gradients g = ∇θk,φkL(i) by using φk, θk; (3) use gradients g to update φk, θk and get new parameters φk+1, θk+1. This process is depicted with an information flow diagram shown in Fig. 2a where ideally we would like every path to contribute to the evaluation of all the terms in its reachable nodes in order to obtain meaningful gradients for updating its associated parameters. However, there are two situations where this is not the case: The first situation, called information-
shortcut, occurs when the learning of pθ(b(i) | a(i), z(i) ) rely entirely on the dashed path in Fig. 2a; more specifically, the dashed (shortcut) path directly propagates b(i) through z(i) to pθ(b(i)|a(i), z(i) ) and, as a result, the relational property z(i) may only learn to encode the absolute property of b(i), i.e., qφ( z | a(i),b(i) ) = qφ( z | b(i) ). The second situation, called deterministic-mapping, occurs when b(i) can be fully characterized by a(i); in this case, the learning of pθ(b(i) | a(i), z(i) ) may only rely on the dash-dotted path in Fig. 2a, i.e., pθ(b(i)|a(i), z(i) ) = pθ(b(i)|a(i) ). While both situations can be viewed as overfitting problem, deterministic-mapping is mainly caused by the data itself and is beyond our control. On the other hand, information-shortcut is caused by short-cutting the gradient update process, which we may overcome with additional regularization techniques. Here we propose two approaches for mitigating the information-shortcut problem by disrupting the flow of information passing through the shortcut path. In the first approach, we restrict the flow of information by constraining the expressiveness of the latent variable z, e.g., by adopting a discrete categorical r.v. (assuming we know a priori the underlying relational property is discrete). In the second approach, we propose a novel data augmentation strategy—relation-preserving data augmentation (RPDA)—that aims to eliminate the shortcut path entirely. First, we define a set of relation preserving functions D={ d(a,b; r) | r∈R (some index set), d :A×B→A×B } that preserve data relationship in the following sense: pθ( z | a,b ) = pθ( z | a′,b′ ), ∀ (a′,b′)= d(a,b; r). Assuming we have access to D, the proposed RPDA strategy then seek to optimize a modified lower bound L(i)RPDA:
L(i)RPDA = Eqφ( z|a′(i),b′(i) ) [ log pθ(b (i)|a(i), z ) + log pθ( z )− log qφ( z|a′(i),b′(i) ) ]
(4)
where (a′(i),b′(i)) = d(a(i),b(i); r(i)), r(i) ∼ U(R).
Note that due to the relation preserving property of D, we have qφ( z(i) | a′(i),b′(i) ) = qφ( z
(i) | a(i),b(i) ), and thus L(i)RPDA is equivalent to L(i) in Eq. (2). When we optimize with L(i)RPDA, the gradient update process can be redrawn in Fig. 2b, where now the learning of pθ(b(i) | a(i), z(i) ) can no longer rely solely on the shortcut path to propagate b′(i) since it differs from b(i) by a non-deterministic factor r(i). In practice, it may seem unrealistic to assume that we can construct a set of RPDA functions D without extensive knowledge of the underlying relational property. However, we can treat data augmentation as a form of regularization and construct a D that reflects
our prior knowledge and belief of the underlying system (Ronneberger et al., 2015; Perez & Wang, 2017). For example, if we want the learning to be rotation invariance—a common theme in computer vision applications—we can construct a D that consists of image rotation augmentations, e.g., d(a,b; r) = (rot(a, r), rot(b, r)) where rot(x, r) rotates the image x by r ∈R= [0, 360) degrees (note that both a and b are rotated by the same amount). Another example may be, for time-series data of a linear time-invariant (LTI) system (commonly assumed in signal processing and control theory (Oppenheim & Schafer, 2009)), we can construct D with time delay and amplitude scaling, e.g., d(a[t],b[t];α, τ) = (αa[t − τ ], αb[t − τ ]), α, τ ∈ R = R × Z. Additional remarks on the practical applicability of RPDA is provided in Appendix A.3. The full training procedure for VRL with RPDA is summarized in Appendix B.

heading: 4 RELATED WORK
text: Existing work on relational learning (Definition 1.1) are mostly based on supervised learning approaches (Lu et al., 2012; Puebla et al., 2021). The rapid growing field of statistical relational learning (SRL) concerns domain models with relational structure (Getoor & Taskar, 2007; De Raedt et al., 2016). Popular SRL methods for link prediction and classification that are based on node attributes similarity (Taskar et al., 2004; Yu et al., 2007) directly contradicts our goal of learning a relational property that is independent of absolute property (node attributes). Other SRL methods based on matrix factorization (Singh & Gordon, 2008; Menon & Elkan, 2011), probabilistic logic programming (De Raedt et al., 2015; Manhaeve et al., 2018), or probabilistic relational models (Friedman et al., 1999; Getoor et al., 2007) typically require some supervision for learning (e.g., labeled examples, known dependency type, etc.) which we do not assume in our work; moreover, most of these methods allow, even encourage, data relationship to depend on its absolute property. Other recent work focus on high-level cognitive tasks, such as visual Q&A and state prediction for complexphysics systems, and derive their relational processing capabilities from learning with clever designed neural networks (Wu et al., 2015; Reed et al., 2015; Fragkiadaki et al., 2016; Chang et al., 2017; Battaglia et al., 2016; Santoro et al., 2017; Battaglia et al., 2018; van Steenkiste et al., 2018; Kipf et al., 2018). Our work differ from these methods in two fundamental ways: (1) we focus on learning an independent relational property in a completely unsupervised fashion; (2) we derive our relational learning capability from a PGM formulation, which gives us the flexibility to use any compatible inference method or function approximation.
Many existing unsupervised learning methods can also be applied to our problem setting (Song et al., 2007; Mikolov et al., 2013a;b; Kingma & Welling, 2014; Goodfellow et al., 2014; Makhzani et al., 2015; Dilokthanakul et al., 2017); however, most of these methods learn a single representation with superimposing information about the relational and absolute property. The difficulty of decoupling the relational property from the learned representation constitute a major obstacle to relational learning. A closely related method is proposed by Guu et al. (Guu et al., 2018), but their learned latent “edit vector” (loosely related to our relational property) may be coupled with absolute property.
Other related work include methods on learning a disentangled representations with applications in style-transfer, image-to-image translation, domain adaptation, etc. (Bousmalis et al., 2016; Mathieu et al., 2016; Chen et al., 2016; Higgins et al., 2017; Denton & Birodkar, 2017; Huang et al., 2018; Chen et al., 2019). Most of these methods strive to learn a disentangled representations of content and style (or pose for video sequence data) where content is generically defined as the underling spatial structure, and style as the rendering of the structure. In comparison, our work can be viewed as learning a disentangled representations of relational and absolute property. We argue that stylecontent separation is fundamentally different from relational-absolute separation; more specifically, we consider both style and content information as absolute property (both describe features of an individual data), while relational property provides additional information about data relationships.

heading: 5 EXPERIMENT
text: In this section, we present experimental results from applying VRL to a variety of relational learning tasks. In our implementation, we parameterize pθ(b | a(i), z(i) ) and qφ( z | a′(i),b′(i) ) with fullyconnected networks (MLPs) fpθ (a (i), z(i)) and fqφ(a ′(i),b′ (i) ), respectively. For binary valued data, we let pθ(b | a(i), z(i) ) be a multivariate Bernoulli distribution whose probability parameters are
computed from fpθ (a (i), z(i)). For real-valued data, we let pθ(b|a(i), z(i) ) be a multivariate Gaussian distribution with a fixed diagonal covariance and the mean is computed from fpθ (a (i), z(i)). We experimented with both continuous and discrete r.v. z. For continuous z, we let the prior pθ( z ) be a bivariate normal distribution z ∼ N (0, I) and qφ( z | a′(i),b′(i) ) be a bivariate Gaussian distribution with diagonal covariance whose mean and covariance are computed from fqφ(a ′(i),b′ (i)
). For discrete z, we adopted two categorical r.v., z = [z1, z2], each having a uniform prior over five categories and let qφ( z | a′(i),b′(i) ) represents two categorical r.v. reparameterized with Gumbel-Softmax distributions whose class probabilities are computed from fqφ(a ′(i),b′ (i)
) (Jang et al., 2017; Maddison et al., 2017). For RPDA, we used random image rotation as data augmentation functions. Parameters θ and φ were jointly trained to maximize L̃(i)RPDA in Eq. (4) using Adam optimizer (Kingma & Ba, 2015). Full implementation details are provided in Appendix E.1 and source code for reproducible results is available online1.

heading: 5.1 RELATIONAL LEARNING WITH DECOUPLED RELATIONSHIPS
text: We first present two relational learning tasks designed with the MNIST (LeCun & Cortes, 2010) and Omniglot (Lake et al., 2015) datasets (dataset information and data preprocessing is described in Appendix D). A paired MNIST (and Omniglot) dataset XM (and XO) = { (a(i),b(i)) | i∈ [1..N ] } is constructed where each a(i) is a randomly rotated MNIST (and Omniglot) binary image and b(i) is another random counter-clockwise rotation of a(i) by 0◦, 72◦, 144◦, 216◦, or 288◦. There are five uniquely defined relative rotational relationships between (a(i),b(i)) in XM (and XO); furthermore, the relative relationships are decoupled from their absolute properties, i.e., the relationship cannot be inferred from a(i) or b(i) alone. The goal of relational learning is to discover the underlying relative rotational relationship. We first demonstrate VRL’s relational discrimination and relational mapping capabilities on the Omniglot relational learning task:
Relational discrimination. We used the trained posterior qφ( z | a,b ) to infer the relational property of a hold-out dataset constructed from the evaluation alphabets. Figure 3a shows a scatter plot of the relational property inferred by VRL where we can see that the approximated posterior accurately cluster (discriminate) data with the same (different) relative rotational relationship together (apart).
Relational mapping. We used the trained likelihood pθ(b | a, z ) to generate images from a given a and z. We chose a from a hold-out dataset and z from: (1) direct sampling in the latent space; (2) relational property inferred from a source data point (as,bs). Figure 3b shows predicted images with z sampled from the latent space shown in Fig. 3a. Figure 3c shows examples of relational mappings from a(c) to b(r,c) by applying the relational property inferred from a source data point (as,b (r) s ).
Next, we compare VRL with other unsupervised and self-supervised learning methods: Spatial transformer networks (STN) (Jaderberg et al., 2015) (we tested two variations of STN that uses its localization network to learn about affine transformation, denoted as STN-affine, and rotation transformation, denoted as STN-rotate); Variational autoencoder (VAE) (Kingma & Welling, 2014); VAE with Gaussian mixture prior (GMVAE) (Dilokthanakul et al., 2017); Adversarial autoencoder (AAE) (Makhzani et al., 2015); VAE with added contrastive latent loss (VAE-contrastive) (Chen et al., 2020); self-supervised representation learning method BYOL (Grill et al., 2020); Learning linear structure in representation space through vector arithmetic (Vec-arithmetic) (Mikolov et al., 2013a; Radford et al., 2016); Learning independent causal mechanisms (LICM) (Parascandolo et al., 2018); Neural relational inference (NRI) with graph neural networks (Kipf et al., 2018). VAE, GMVAE, AAE, VAE-contrastive, BYOL, and Vec-arithmetic all trained with dim(z)=10. Each baseline method is assessed on both MNIST and Omniglot relational learning tasks through unsupervised clustering. For fair comparison, we adopted image rotation augmentation during the training of all the baseline methods. For VAE-contrastive and BYOL, we used random image rotation augmentations to generate “positive paris” for training. In our comparison, STN-affine and STN-rotate are considered as the benchmark methods as they are architecturally designed to learn about the spatial transformation between a(i) and b(i). In contrast, VRL does not make any assumption about the underlying relationship. Implementation details for the baseline methods are provided in Appendix E.2
1https://github.com/kh1iu/vrl.git
To quantitatively evaluate the clustering performance, we calculate the classification error rate of a hold-out dataset based on a simple classifier that was trained on 5% of the data with label. For continuous latent space, we trained a standard multi-class support vector machine classifier with radial basis function kernel (Manning et al., 2008). For discrete (categorical) latent space, we follow the evaluation protocol on clustering assignment described in Makhzani et al. (2015). Results are summarized in Table 1 where each entry reports the average and standard deviation over 5 runs with different random seeds and each run consists of either 3 (randomly selected out of 5) or 5 relative rotational relationships. We can see that the proposed VRL with RPDA achieved high accuracy in recovering the relative relationships for all the tasks and outperformed other baseline methods including the benchmark STN-affine and STN-rotate. To gain insight into the information-shortcut problem and the proposed mitigation strategies introduced in Sec. 3.2, we performed an ablation study on the prior selection of z and RPDA. The results are summarized in Table 1 and we make the following observations: First, representing z as a discrete categorical (Cat.) r.v. limits its expressiveness and improves VRL’s performance over a continuous prior (Cont.); Second, RPDA is a critical component necessary for VRL to learn a meaningful and independent relational property, especially when flexible function approximations such as deep neural networks are used. Additional experimental results and analysis are provided in Appendix F.

heading: 5.2 RELATIONAL LEARNING WITH COUPLED RELATIONSHIPS
text: To further test the robustness and generalizability of VRL, we consider a scenario where the relative relationship is coupled with the absolute property. To setup this experiment, we modified the construction of XM , denoted as XMd , so that the relative rotational relationship between a(i) and b(i) is completely determined by its digit representation (absolute property of a(i) and b(i)): a(i) ∈ [’0’, ’1’]→ 0◦,a(i) ∈ [’2’, ’3’]→ 72◦,a(i) ∈ [’4’, ’5’]→ 144◦,a(i) ∈ [’6’, ’7’]→ 216◦,a(i) ∈ [’8’, ’9’]→ 288◦ (read: if a(i) is digit ’0’ or ’1’ then the relative rotational relationship between (a(i),b(i)) is 0◦). The question then arise: is VRL capable of learning an independent relational property even when the underlying relative relationship is coupled with the absolute property? To test this idea, we trained VRL on XMd but validated it on a hold-out dataset of XM which has a decoupled relative relationship. The results are shown in Fig. 4, where we can see that VRL was indeed capable of learning an independent relational property irrespective of the digit representation. If this were not the case, we would expect to see a scatter plot Fig. 4a with heavily overlapped
relative relationship labels since XM was constructed with random relative rotational relationships. Figures 4b and 4c further demonstrate VRL generalizes well to unseen data—VRL trained on XMd learned to rotate any digit by any amount despite not having seen most of the digit-rotation pairs during training.
(as,b (r) s ) in the left-most column.

heading: 5.3 RELATIONAL LEARNING WITH HIGH-LEVEL PERCEPTION REASONING
text: Finally, we present results of using VRL to learn about relative facial expression changes, facial illumination condition changes, and speech emotion changes. We extracted images of three facial expressions (happy, surprised, sad) of each subject from the Yale Face Database (Belhumeur et al., 1997) to form a paired dataset XFe where each data point (a(i),b(i)) represents a subject with different facial expressions. Next, we extracted images of four illumination conditions (left (L), front (F), right (R), top (T)) of each subject from the Extended Yale Face Database B (Georghiades et al., 2001) to form a paired dataset XFl where each data point (a(i),b(i)) represents a subject with
different illumination conditions. Lastly, we extracted speech waveforms (represented as log-mel spectrogram) of three emotions (calm, angry, fearful) of each voice actor from the Ryerson AudioVisual Database of Emotional Speech and Song (RAVDESS) (Livingstone & Russo, 2018) to form a paired dataset XS where each data point (a(i),b(i)) represents a voice actor with different emotions. Dataset information and data preprocessing is described in Appendix D. Due to the extremely limited data samples in XFe (45 unique images), XFl (112 unique images), and XS (144 unique speech recordings), we focus on learning undirected relative relationship, i.e., (a(i),b(i)) and (b(i),a(i)) have the same relative relationship. In this case, XFe , XFl , and XS consist of 3, 6, and 3 undirected relative relationships, respectively. For RPDA, we adopted random image rotation for the VRL training on XFe and XFl , and random time delay and amplitude scaling for XS due to its temporal data characteristic; furthermore, since we are only interested in learning undirected relationships, we augment RPDA with random swapping operation (see Appendix E.1 for implementation details). Due to the limited datasets, both training and validation are performed on the entire dataset. The inference results from a trained posterior qφ( z | a,b ) are shown in Fig. 5 where we can see that VRL is effective in learning a relative relationship that is independent of subject’s identity, facial expression, illumination condition, speech content, and emotion. Comparing our results with existing unsupervised methods on facial images (Song et al., 2007; Wu et al., 2013; Shi et al., 2018; Tapaswi et al., 2019) and speech emotion recognition (Neumann & Vu, 2019; Li et al., 2021), we remark that existing methods cluster data by their absolute property (e.g., subject identity, speech emotion, etc.), while VRL clusters data by their relational property (e.g., relative facial expression, illumination, or speech emotion changes). Additional relational mapping results are provided in Appendix F.2.

heading: 6 DISCUSSION AND CONCLUSION
text: The proposed VRL method comes with both advantages and disadvantages: the main advantage of VRL is its relational learning capabilities; however, this may also be one of its disadvantages. More specifically, VRL can learn an independent relational property even when it is coupled with the absolute property (see Sec. 5.2), i.e., VRL is oblivious to the coupling information between the two properties. Nevertheless, such information may be of interest to the user, and in this regard, VRL only provides a partial view of the data.
In conclusion, the proposed VRL method is an efficient and effective unsupervised learning method for addressing the relational learning problem where our goal is to learn an independent relational property. By dissecting the data information into decoupled relational and absolute property, we hope VRL can bring new insight into everyday data analysis and ultimately find applications for a wide variety of problems.

heading: ACKNOWLEDGMENT
text: We thank anonymous reviewers for comments that helped improve the paper. We would also like to thank Huseyin Denli, Antonio Paiva, Ashutosh Tewari, Myun-Seok Cheon, and Peng Xu (ExxonMobil) for insightful discussion and valuable feedback.

heading: A ADDITIONAL REMARKS

heading: A.1 REMARKS ON EQUATION 1
text: To provide additional insight into Eq. (1), we give the following example (attributed to S. Bernstein (Hogg et al., 2005)): Let a and b be two independent tosses of a fair coin, where we designate 1 for heads and 0 for tails. Let a third random variable z be equal to 1 if exactly one of two coin tosses resulted in "heads", and 0 otherwise. It is easy to see that this example r.v. a, b ans z satisfy all four conditions in Eq. (1) and we can intuitively interpret z as meaning "different tosses" which is a relational property since it indicates the relationship between a and b (same vs. different) and is independent of a and b, i.e., z is undertermined when observing a or b alone. The main difference between this example and our definition in Eq. (1) is that we do not specify what z is; we only assume the existence of z and our goal is to learn about it.

heading: A.2 REMARKS ON VRL-PGM
text: The proposed VRL-PGM reflects our priority and compromise for using a PGM to represent the relational learning problem: we sacrifice some identifiability of the original abstract problem but obtain a rigorous and tractable model that achieves our objective of learning an independent relational property. Here we discuss aspects of the original relational learning problem (see Section 2) that differ from the proposed VRL-PGM (see Section 3.1). First, the original problem description in Eq. 1 specifies that the relational property z be independent of both a’s and b’s absolute properties; however, learning and inferencing relational property z that satisfies both independence constraints is a challenging problem; therefore, as a compromise, VRL-PGM only enforce the independence constraint between z and a. Second, the original problem is inherently undirected with no cause-effect relationship between a and b, whereas VRL-PGM is based a directed acyclic graph (DAG) that artificially introduces conditional dependency between a and b. We note that this can be viewed as the result of VRL-PGM choosing to satisfy Eq. 1(i), 1(iii), 1(iv). It is equally valid for VRL-PGM to choose to satisfy Eq. 1(ii), 1(iii), 1(iv) and it would lead to the same model but with a and b swapped. In other words, the application of VRL does not require the true conditional dependency between (a,b) be known in advance only that it is maintained consistently throughout learning and inference. The above-mentioned discrepancies represent the compromises we made with adopting VRL-PGM in exchange for a rigorous and tractable method for learning an independent relational property. Finally, these compromises are not been made without consequences: information-shortcut is a direct result of not enforcing independence constraint between z and b, and deterministic-mapping can be viewed as caused by the causal relationship VRL-PGM introduced between a and b.

heading: A.3 REMARKS ON RPDA
text: Here we discuss the practical applicability of the proposed RPDA strategy. More specifically, we argue that in many practical problem settings, the RPDA functions D can be designed without any knowledge of the underlying relational property. For example, as we have explained in Sec. 3.2, in many computer vision applications, rotation invariant is a desirable property for the learned model; for example, in spectral imaging applications, oftentimes the orientation of the images are not preserved or not enforced only that they are consistent between the same paired images (Ronneberger et al., 2015). In such problem setting, we can safely use image rotation function for constructing D. Another example may be: for a discrete time-series data a[t],b[t] that represent the input and output of a linear time-invariant (LTI) system (commonly assumed in signal processing and control theory (Oppenheim & Schafer, 2009)), and we want to learn a relational property that characterize the system’s impulse response. We have αb[t− τ ] = αa[t− τ ], ∀α ∈ R, τ ∈ Z, and we can construct D with d(a[t],b[t];α, τ) = (αa[t− τ ], αb[t− τ ]), α, τ ∈ R = R×Z. In all of the above examples, the construction of RPDA functions D reflects our prior knowledge and belief of the underlying system and not based on the underlying relational property. However, just like any data augmentation, the effectiveness of RPDA will depend on the problem setting and we advocate to start without RPDA and only apply it when suspecting information-shortcut occurs.
B VRL ALGORITHM
Algorithm 1 VRL with RPDA procedure VRL(X, p( ), D) . If RPDA not available, D={ id(·) | (a,b) = id(a,b) }
Initialize parameters θ, φ while not convergence of parameters (θ, φ) do
Sample minibatch { (a(i),b(i)) | i∈ [1..M ] } from X. Run RPDA and obtain (a′(i),b′(i)) = d(a(i),b(i); r(i)), r(i) ∼ U(R), i=1, ...,M . Compute gradients g = ∇θ,φL(i)RPDA (see Eq. (4)). Update parameters θ, φ using gradients g (e.g., SGD).
end while return θ, φ end procedure

heading: C DERIVATION OF VARIATIONAL LOWER BOUND
text: To derive a variational lower bound for VRL-PGM, we first write the log-evidence as log pθ(X ) = log pθ( { (a(i),b(i)) | i∈ [1..N ] } ) = ∑N i=1 log pθ(a
(i),b(i) ), where each term in the summation can be expressed as:
log pθ(a (i),b(i) ) = DKL ( qφ( z | a(i),b(i) ) ∥∥∥ pθ( z | a(i),b(i) )) + Eqφ(z|a(i),b(i)) [ log pθ( z,a (i),b(i) )− log qφ( z | a(i),b(i) ) ] . (5)
The first term on the right-hand side (RHS) is the KL-divergence from pθ( z | a(i),b(i) ) to qφ( z | a(i),b(i) ), which provides a measure of dissimilarity between the two distributions; the second term on the RHS continues as:
Eqφ(z|a(i),b(i)) [ log pθ( z,a (i),b(i) )− log qφ( z | a(i),b(i) ) ]
= Eqφ(z|a(i),b(i)) [ log pθ(b (i) | a(i), z )pθ( z )pθ(a(i) )− log qφ( z | a(i),b(i) ) ]
= Eqφ(z|a(i),b(i)) [ log pθ(b (i) | a(i), z ) + log pθ( z )− log qφ( z | a(i),b(i) ) ] + log pθ(a (i) ), (6)
where in the second line we use the fact that r.v. a and z are independent. Substitute Eq. (6) back in (5) and rearrange terms gives us the expression for Eq. 2:
log pθ(b (i) | a(i) ) = DKL ( qφ( z | a(i),b(i) ) ∥∥∥ pθ( z | a(i),b(i) ))+ L(i) (7) where
L(i) = Eqφ(z|a(i),b(i)) [ log pθ(b (i)|a(i), z ) + log pθ( z )− log qφ( z|a(i),b(i) ) ] .
The term L(i) serves as a lower bound for the conditional log-likelihood log pθ(b(i) | a(i) ) since KL-divergence is non-negative. Maximizing L(i) w.r.t. φ and θ gives us both a ML estimate for pθ(b | a, z ) (by maximizing the first term inside the expectation in L(i)) and a lower KL-divergence (the better qφ( z | a(i),b(i) ) approximates the true posterior pθ( z | a(i),b(i) )) as the conditional log-likelihood log pθ(b(i) | a(i) ) does not depend on φ. The lower bound L(i) can be maximized with gradient ascend methods; however, its gradients w.r.t. φ is difficult to obtain: the expectation in L(i) is taken w.r.t. the distribution qφ( z | a(i),b(i) ), which is a function of φ (Paisley et al., 2012). To obtain efficient estimators for both L(i) and its gradients, we adopt the reparameterization trick developed in Kingma & Welling (2014) where the r.v. z is expressed as a transformation of another r.v. ∼ p( ) that is independent of a, b, and φ: z = g( ,a(i),b(i), φ) where g is some differentiable and invertible transformation. Given such a change of variable, the lower bound L(i) can be rewritten as:
L(i) = Ep( ) [ log pθ(b (i) | a(i), z ) + log pθ( z )− log qφ( z | a(i),b(i) ) ] , (8)
where z = g( ,a(i),b(i), φ) and ∼ p( ). Note that the expectation in Eq. (8) is taken w.r.t. p( ) and we can approximate L(i) with a Monte Carlo estimator:
L̃(i) = 1 L L∑ l=1 log pθ(b (i) | a(i), z(i,l) ) + log pθ( z(i,l) )− log qφ( z(i,l) | a(i),b(i) ), (9)
where z(i,l) = g( (i,l),a(i),b(i), φ) and (i,l) ∼ p( ). An estimator for L(i)RPDA in Eq. (4) follows similarly as:
L̃(i)RPDA = 1
L L∑ l=1 log pθ(b (i) | a(i), z(i,l) ) + log pθ( z(i,l) )− log qφ( z(i,l) | a′(i),b′(i) ), (10)
where (a′(i),b′(i)) = d(a(i),b(i); r(i)), r(i) ∼ U(R), and z(i,l) = g( (i,l),a′(i),b′(i), φ), (i,l) ∼ p( ). The lower bound for a minibatches of data XM = { (a(i),b(i)) | i∈ [1..M ] } can be approximated by L̃RPDA(θ, φ;XM ) = NM ∑M i=1 L̃ (i) RPDA. And finally,
the gradients ∇θ,φL̃RPDA(θ, φ;XM ) = NM ∑M i=1∇θ,φL̃ (i) RPDA can be computed in a straightforward manner and used to update the parameters θ and φ with stochastic optimization methods, such as SGD.

heading: D DATASET AND DATA PREPROCESSING
text: We used the following datasets in our experiments:
MNIST dataset (available under CC BY-SA 3.0 license) consists of 28 × 28 grayscale images of standard handwritten digits with labels; the dataset is composed of 60,000 training samples and 10,000 testing samples (LeCun & Cortes, 2010).
Omniglot dataset (available under MIT license) contains 1623 different handwritten characters (105 × 105 grayscale images) from 50 different alphabets; the dataset is split into a background (training) set of 30 alphabets and an evaluation (testing) set of 20 alphabets (Lake et al., 2015).
Yale Face Database (see (Georghiades, 2001) for dataset permission) contains 165 grayscale face images of 15 human subjects under 8 facial expressions and 3 illumination conditions (Belhumeur et al., 1997).
Extended Yale Face Database B (see (Georghiades, 2001) for dataset permission) contains 16128 grayscale face images of 28 human subjects under 9 poses and 64 illumination conditions (Georghiades et al., 2001).
Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS) (available under CC BY-NC-SA 4.0 license) is a speech and song dataset containing 24 professional actors (12 female, 12 male) vocalizing two lexically-matched statements in a neutral North American accent with wide range of emotions (Livingstone & Russo, 2018).
Each handwritten character image in MNIST and Omniglot dataset is resized to 32 × 32 binary image. Each facial image in Yale and Extended Yale dataset is center-cropped, resized to 64× 64, and normalized pixel values to be within [0, 5]. Each speech waveform in RAVDESS dataset is trimmed to remove silences both at the start and at the end, and then clipped or zero-padded to be 3 seconds long. We then represent each waveform by its log-amplitude mel spectrogram with 128 mel bands and frame size of 512 samples. Finally, each log-mel spectrogram image is resized to 64× 64. Examples of facial images from Yale and Extended Yale, and log-mel spectrograms of RAVDESS speech waveforms are shown in Fig. 6.
E IMPLEMENTATION DETAILS
E.1 VRL IMPLEMENTATION
Here we describe the details of our VRL implementation. We parameterize both pθ(b | a(i), z(i,l) ) and qφ( z | a′(i),b′(i) ) using fully-connected networks (MLPs) with rectified linear non-linearities.
For binary valued data such as MNIST and Omniglot, we let pθ(b | a(i), z(i,l) ) be a multivariate Bernoulli distribution whose probability parameters p(i,l) are computed from MLPs fpθ (a (i), z(i,l)):
fpθ : [a (i), z(i,l)]→ FC(512)→ FC(256)→ FC(512)→ p(i,l) (11)
The reconstruction error in this case is the binary cross-entropy: log pθ(b(i) | a(i), z(i,l) ) = −∑b(i) logp(i,l) + (1 − b(i)) log(1 − p(i,l)). For real-valued data such as Yale facial images and RAVDESS log-mel spectrograms, we let pθ(b | a(i), z(i,l) ) be a multivariate Gaussian distribution with a fixed diagonal covariance N (b;µ(i,l),σ2I) where µ(i,l) is computed from MLPs fpθ (a (i), z(i)):
fpθ : [a (i), z(i,l)]→ FC(512)→ FC(256)→ FC(512)→ µ(i,l), (12)
The reconstruction error in this case is the mean squared error: log pθ(b(i) | a(i), z(i,l) ) = −1 σ2 ‖b(i) − µ(i,l)‖2 + const. We experimented with both continuous and discrete (categorical) latent variable z. For a continuous z, we let the prior pθ( z ) be a bivariate normal distribution and the approximated posterior qφ( z | a′(i),b′(i) ) be a bivariate Gaussian distribution with a diagonal covariance N (z;µ(i), (σ(i))2I) where µ(i) and σ(i) are computed from MLPs fqφ(a′ (i) ,b′ (i) ):
fqφ : [a ′(i),b′ (i) ]→ FC(512)→ FC(256)→ [µ(i),σ(i)] (13)
In this case the latent variable samples are drawn as: z(i,l) = µ(i) + σ(i) (i,l), (i,l) ∼ N (0, I). For a discrete z, we used two categorical r.v., z = [z1, z2], each having a uniform prior over five categories and let qφ( z | a′(i),b′(i) ) represents two categorical r.v. reparameterized with GumbelSoftmax distributions whose class probabilities π1 = (π11, . . . , π15) and π2 = (π21, . . . , π25) are computed from MLPs fqφ(a ′(i),b′ (i) ) (Jang et al., 2017; Maddison et al., 2017):
fpθ : [a ′(i),b′ (i) ]→ FC(512)→ FC(256)→ [π1,π2] (14)
In this case the latent variable samples are drawn as: z(i,l)j = softmax ( ( (i,l) + logπj)/τ ) , j = 1, 2 where (i,l) ∈ R5 are i.i.d. samples drawn from a Gumbel(0, 1) distribution and the softmax temperture τ controls the “smoothness” of the samples.
For MNIST, Omniglot, and Yale relational learning tasks, we used random image rotation for RPDA, i.e., D={ (rot(a, r), rot(b, r)) | r∈ [0, 360) }. For time-series RAVDESS speech waveforms, we used random time delay and amplitude scaling for RPDA, i.e., D={ (αa[t− τ ], αb[t− τ ]) | α ∈ R, τ ∈ Z }. Additionally, since we are only interested in learning an undirected relative relationship
changes in Yale and RAVDESS relational learning tasks (i.e., pθ( z | a,b ) = pθ( z | b,a )), we augment RPDA functions with random swapping operations:
swap(a′,b′) =
{ (a′,b′), p = 0.5
(b′,a′), p = 0.5
Finally, the learning objective L̃(i)RPDA in Eq. 10 can be constructed once pθ(b |a(i), z(i,l) ), pθ( z ), and qφ( z |a′(i),b′(i) ) are defined. For example, when we adopt a discrete latent variable z = [z1, z2] for MNIST and Omniglot relational learning tasks (Sec. 5.1 and 5.2), we can derive L̃(i)RPDA (with L = 1) as:
L̃(i)RPDA = − ∑ b(i) logp(i,1) + (1− b(i)) log(1− p(i,1))− ∑ πj logπj + const (15)
where (a′(i),b′(i)) = (rot(a, r(i)), rot(b, r(i))), r(i) ∼ U([0, 360)), [π1,π2] = fqφ(a′ (i) ,b′ (i) ), z (i,1) j = softmax(( (i,1) + logπj)/τ), (i,1) ∼ Gumbel(0, 1), j = 1, 2, and p(i,1) = fpθ (a (i), [z (i,1) 1 , z (i,1) 2 ]). Similarly, L̃ (i) RPDA (with L = 1) for a continuous latent variable z can be derived as:
L̃(i)RPDA =− ∑
b(i) logp(i,1) + (1− b(i)) log(1− p(i,1)) + logN (z(i,1);0, I) − logN (z(i,1);µ(i), (σ(i))2I), (16)
where (a′(i),b′(i)) = (rot(a, r(i)), rot(b, r(i))), r(i) ∼ U([0, 360)), [µ(i),σ(i)] = fqφ(a′ (i) ,b′ (i) ), z(i,1) = µ(i)+σ(i) (i,1), (i,1) ∼ N (0, I), and p(i,1) = fpθ (a(i), z(i,1)). For Yale and RAVDESS relational learning tasks (Sec. 5.3), we adopted a continuous latent variable z with σ = 0.1 and L̃(i)RPDA (with L = 1) can be derived as:
L̃(i)RPDA = −1 0.02 ‖b(i) − µ(i,l)‖2 + logN (z(i,1);0, I)− logN (z(i,1);µ(i), (σ(i))2I) + const (17)
where (a′(i),b′(i)) = swap(rot(a, r(i)), rot(b, r(i))), r(i) ∼ U([0, 360)), [µ(i),σ(i)] = fqφ(a ′(i),b′ (i)
), z(i,1) = µ(i) + σ(i) (i,1), (i,1) ∼ N (0, I), and µ(i,l) = fpθ (a(i), z(i,1)). All MLPs with parameters θ and φ were jointly trained for 300k iterations (without batchnormalization, weight decay, nor dropout) to maximize L̃(i)RPDA in Eq. (4) with using Adam optimizer (learning rate=0.0004, β1=0.9, β1=0.999) (Kingma & Ba, 2015). Minibatches of size M=100 were used. We anneal the learning rate (0.0004 base learning rate) with step decay (factor of 0.5 every 100k iterations). When Gumbel-Softmax distributions is used, we anneal the softmax temperature τ from 1.0 to 0.5 with exponential decay (decay rate=0.00005).

heading: E.2 BASELINE METHODS IMPLEMENTATION
text: STN-affine and STN-rotate (Jaderberg et al., 2015; Dong et al., 2017) minimize the difference between b(i) and a geometric transformation of a(i) (affine transformation for STN-affine and rotation transformation for STN-rotate) by using a spatial transformer with a localization network that takes both a(i),b(i) as input; during evaluation, we perform clustering on the output of the trained localization network. The localization network in STN-affine takes both a(i),b(i) as input and outputs an affine transformation matrix:
[a(i),b(i)]→ FC(512)→ FC(256)→ FC(256)→ Aθ = [ θ11 θ12 θ13 θ21 θ22 θ23 ] .
In STN-rotate, we further restrict STN-affine to only allow rotation transformation and the localization network in this case only outputs the rotation angle ∆θ:
[a(i),b(i)]→ FC(512)→ FC(256)→ FC(256)→ ∆θ, Aθ = [ cos(∆θ) − sin(∆θ) 0 sin(∆θ) cos(∆θ) 0 ] .
VAE, AAE, GMVAE all trained directly using (a(i),b(i)) and we perform clustering on their learned latent space. We selected dim(z)=10 via cross-validation. We follow the implementation of (Kingma
& Welling, 2014; Makhzani et al., 2015; Dilokthanakul et al., 2017) since they also experimented with MNIST or Omniglot dataset. However, for a fair comparison, we also experimented with the following encoder/decoder architecture and report the best results:
[a(i),b(i)]→ FC(512)→ FC(256)→ z(i) ∼ q(z(i))→ FC(256)→ FC(512)→ [ya(i) ,yb(i) ].
Our VAE-contrastive implementation is similar to Chen et al. (2020) in concept. We added an additional latent loss term to the VAE loss function that aim to minimize the difference between the latent representation from different rotation augmented image pairs, i.e., α‖z(i) − z′(i)‖2 (also experimented with cosine similarity) where z(i) = Enc([a(i),b(i)]), z′(i) = Enc([a′(i),b′(i)]) and we cross-validate α from 0.1 to 10.
BYOL learns a joint image represenation z for (a(i),b(i)), on which the clustering is applied during evaluation. We selected dim(z)=10 via cross-validation.
Vec-arithmetic first trains an autoencoder on individual a(i), b(i) and then used the trained encoder to compute a latent vector representation of (a(i),b(i)) through Enc(b(i))− Enc(a(i)), on which the clustering is applied during evaluation. We selected dim(z)=10 via cross-validation.
LICM (Evtimova, 2017) trained a committee of 5 experts to learn a set of independent mechanisms that generates b(i) from the canonical a(i), and used the trained discriminator during evaluation to select the winning expert (as a categorical variable).
NRI infer the interaction between a(i) and b(i) and used the NRI-Encoder to predict the relationship type (as a categorical variable). We follow the implementation of (Kipf et al., 2018; Fetaya, 2019) where we used the NRI-Encoder to infer the relationhip between a(i) and b(i); the inferred relationship z(i), together with (a(i),b(i)), are then feed into the NRI-Decoder to predict ya(i) and yb(i) which are reconstructions for a(i) and b(i), respectively. To account for the high-dimensional input data, we used two MLPs (one for each a(i) and b(i)) to extract lower dimensional feature vectors xa(i) ∈ R10 and xb(i) ∈ R10 that serve as input to NRI-Encoder:
a(i) → FC(512)→ FC(256)→ xa(i) , b(i) → FC(512)→ FC(256)→ xb(i) . Similarly, we insert two MLPs (with the same architecture as above) in front of NRI-Decoder to extract data feature vectors and append two MLPs after NRI-Decoder to convert the predicted feature vectors back to the high-dimensional data space:
xa(i) → FC(256)→ FC(512)→ yb(i) , xb(i) → FC(512)→ FC(256)→ yb(i) . All MLPs are trained jointly with NRI encoder and decoder.
For our final baseline comparison, we present results from applying InfoGAN to the MNIST relational learning task in Sec. 5.1 (Chen et al., 2016). InfoGAN has demonstrated its ability to learn disentangled representations (represented by structured latent codes c1, c2, ..., cL) through generative modelling. Although inferring latent codes for a given data point is a non-trivial task for InfoGAN, we can examine the learned latent representation by manipulating the latent codes and visually inspect the generated random samples. We modeled the latent codes with one categorical code c1 ∼ Cat(K = 10, p = 0.1) and two continuous codes c2, c3 ∼ Unif(−1, 1) . Figure 7 shows examples of generated images from manipulating the latent codes and it is clear that none of the latent codes distinctively capture the full range of relative rotational relationships.

heading: F ADDITIONAL EXPERIMENTAL RESULTS AND ANALYSIS

heading: F.1 MNIST AND OMNIGLOT RELATIONAL LEARNING RESULTS ANALYSIS
text: At first glance, the results in Fig. 3c resemble that of style-transfer, but they are fundamentally different: in style-transfer, the image b(r,c) is generated by applying the style of b(r)s to the content of a(c), whereas VRL generates image b(r,c) by applying the relational property of (as,b (r) s ) to the image a(c). It is evident from Fig. 3c that predicted images b(r,c) do not share similar style to b(r)s , but rather the same relative rotational relationship w.r.t. a(c) and as.
A close inspection of the results in Table 1 shows that the seemingly simple relational learning tasks is in fact very challenging even for specialized methods such as STN-affine and STN-rotate that only learn about spatial transformations. Comparing VRL with VAE, we can see that VRL achieved better performance than VAE with more efficient learning—VAE were only able to solve the simpler 3-relationship tasks by adopting a high dimensional latent space; in contrast, VRL were able to solve all tasks with a compact 2-D latent space. Further comparing VAE with GMVAE and AAE shows that regularizing the latent space, as done in AAE and GMVAE, can degrade the performance of relational learning. For contrastive self-supervised learning methods like VAE-contrastive and BYOL there is no way to enforce the learned data representation be independent of absolute property since different augmented image still share large part of their absolute property (e.g., digit representation). For example, it is entirely possible for VAE-contrastive and BYOL to tightly cluster images based on their digit representation (absolute property); this will lead to a small loss function value but makes the subsequent relational discrimination task more challenging since each digit representation can have all possible relative rotation relationships. LICM failed all relational learning tasks due to the fact that the transformed and canonical distributions overlaps completely (both a(i) and b(i) are rotations of MNIST or Omniglot images) and it is not possible for a committee of experts to distinguish the two. NRI’s inability to solve the relational learning tasks can be attributed to the fact that it is designed to learn in a dynamical systems, but more importantly, GNN is not guaranteed to learn an independent relative relationship.
In summary, we argue that a major challenge for applying existing methods to relational learning problem is that they learn a single representation that encodes both relational and absolute properties and it is difficult to dissect the relational property from the learned representation.

heading: F.2 YALE AND RAVDESS RELATIONAL MAPPING RESULTS
text: Figure 8 shows examples of relational mappings predicted by the same set of trained VRL models from Sec. 5.3.

heading: F.3 RELATIONAL LEARNING WITH MULTIPLE RELATIONSHIPS
text: Here, we setup a more complex relational learning task that includes both relative rotational and scaling relationships. We constructed a paired MNIST dataset XM10 = { (a(i),b(i)) | i∈ [1..N ] } where each a(i) is a randomly rotated and scaled (by a factor ×0.66 or ×1) MNIST image and b(i) is another random rotation (by 0◦, 72◦, 144◦, 216◦, or 288◦) and scaling (by ×1 or ×1.5) of a(i). Note that there are a total of 10 decoupled relative relationships between (a(i),b(i)) in XM10 (combinations of 5 rotational and 2 scaling transformations). We trained VRL on XM10 following the same training procedure as described in Appendix E but with larger MLPs for fpθ (a (i), z(i)) and
fqφ(a ′(i),b′ (i) ) (cf. Eq. 11, 13) to account for the increased problem complexity:
fpθ : [a (i), z(i,l)]→ FC(1024)→ FC(512)→ FC(512)→ p(i,l)
fqφ : [a ′(i),b′ (i) ]→ FC(1024)→ FC(512)→ [µ(i),σ(i)]
(18)
The inference result is shown in Fig. 9a, where we can see that the approximated posterior accurately cluster (discriminate) data with the same (different) relative relationship together (apart). Examples of images predicted by direct sampling in the latent space are shown in Fig. 9b.

heading: F.4 RELATIONAL LEARNING WITH CONTINUOUS RELATIONSHIPS
text: Lastly, we present an example with a continuous relational property. Based on the MNIST dataset, we constructed a paired dataset XMc ={ (a(i),b(i)) | i∈ [1..N ] } where both a(i) and b(i) are random rotation of the same MNIST image. In this case, there is a continuous (and decoupled) relative rotational relationship between (a(i),b(i)). We trained VRL on XMc following the same training procedure as described in Appendix E but used convolutional neural networks (CNNs) to capture the continuous relationship. We approximate pθ(b | a(i), z(i,l) ) with an autoencoder-like neural network fpθ (a (i), z(i)) = f decθ ( f encθ (a (i)), z(i,l) ) :
f encθ :a (i) → Conv(3x3x8)→ Conv(3x3x32)→ Conv(3x3x128)→ FC(20)→ h(i)∈R20
f decθ :[h (i)∈R20, z(i,l)]→ FC→ ConvT(3x3x128)→ ConvT(3x3x32)→ ConvT(3x3x8) → Conv(1x1x1) Sigmoid−−−−→ [0, 1]dim(B),
where Conv(·) is a strided (stride 2) convolutional layer and ConvT (·) is a transposed convolutional layer. We used batch-normalization after each layer and rectified linear non-linearities. We represent the approximated posterior qφ( z | a′(i),b′(i) ) (with bivariate normal distribution as prior) using:
fqφ : [a ′(i),b′ (i) ]→ Conv(3x3x8)→ Conv(3x3x32)→ Conv(3x3x128)→ FC(4)→ [µ(i),σ(i)].
A scatter plot of the relational property inferred by a trained posterior is shown in Fig. 10a, and examples of images predicted by direct sampling in the latent space (denoted by markers “×” in Fig. 10a) are shown in Fig. 10b. From Fig. 10 we can see that VRL learned an independent relational
property that encodes a continuous relative rotational relationship; however, there is a small region in Fig. 10a with overlapping relational property that leads to an ambiguous relationship interpretation (120◦ vs. 240◦). This ambiguity is likely caused by compressing the relative relationship down to a 2-D latent space, z ∈ R2, and motivates us to adopt a higher-dimensional latent space, e.g., z ∈ R3. Figure 11 shows inference result from repeating the previous experiment but with adopting z ∈ R3. We can see that VRL learned a three-dimensional relational property that unambiguously represents the underlying continuous relative rotational relationship.

Review summaries:
This paper proposes a model to infer the relationship between multiple instances in a dataset by inferring a latent variable. The authors accomplish this by defining an optimization problem that optimizes the ELBO of the proposed graphical model. The paper presents a nice solution to some of the identification issues that can arise when inferring the latent variable, in particular the so called “information shortcut” when the model overfits to only learning the “absolute” property of the dataset, rather than inferring the shared latent traits. 

The paper proposes variational relational learning by learning relations between two inputs via variational inference on a probabilistic graphical model (PGM). The PGM that they use factors as p(a)p(z)p(b|a,z) where a,b are the two inputs and z is the supposed relationship between them. The example shown in the experiments is rotational mnist, where b is a rotated version of a, and z should encode the degree of rotation. The paper learns both the forward network and in inference network in a VAE-like approach; the elbo derivations appear correct to me.

This paper introduces a variational method for relational learning. It first introduces relational learning as learning based on relational property instead of absolute property, and introduces conditions (Eq. 1). Then it proposes VRL-PGM with a variational lower bound. To eliminate the information short-cut, it introduces relation-preserving data augmentation (RPDA). Experiments show that the method is able to perform relation discrimination and relation mapping, on a variant of MNIST and Yale face datasets.

This paper proposes a relational learning method based on variational Bayes. The main idea is to learn relations among objects independently of each object's own properties. The model theory is discusses where authors define the problem and discuss about solutions for some limitations of the model when a relation between objects A and B can be found by only looking at one of the objects own properties. The paper is quite well written. Algorithms are provided, but not the code. As far as I understood, results are obtained on "benchmark" datasets, where some synthetic versions of these datasets are used to create the relations between the instances of the original dataset and of the synthetic.