SlowGuess/NAACL_2021_OCR huggingface dataset:
Department of Computer Science,
qianqian.xie@manchester.ac.uk
School of Computer Science,
huangjimin@whu.edu.cn
Pan Du
Department of Computer Science and Operations
Research, University of Montreal
pandu@iro.umontreal.ca
Jian-Yun Nie
Department of Computer Science and Operations Research,
nie@iro.umontreal.ca
Graph convolutional networks (GCNs) have been applied recently to text classification and produced an excellent performance. However, existing GCN-based methods do not assume an explicit latent semantic structure of documents, making learned representations less effective and difficult to interpret. They are also transductive in nature, thus cannot handle out-of-graph documents. To address these issues, we propose a novel model named inductive Topic Variational Graph Auto-Encoder (T-VGAE), which incorporates a topic model into variational graph-auto-encoder (VGAE) to capture the hidden semantic information between documents and words. T-VGAE inherits the interpretability of the topic model and the efficient information propagation mechanism of VGAE. It learns probabilistic representations of words and documents by jointly encoding and reconstructing the global word-level graph and bipartite graphs of documents, where each document is considered individually and decoupled from the global correlation graph so as to enable inductive learning. Our experiments on several benchmark datasets show that our method outperforms the existing competitive models on supervised and semi-supervised text classification, as well as unsupervised text representation learning. In addition, it has higher interpretability and is able to deal with unseen documents.
Recently, graph convolutional networks (GCNs)(Kipf and Welling, 2017; Velicković et al., 2018) have been successfully applied to text classification tasks (Peng et al., 2018a; Yao
et al., 2019; Liu et al., 2020; Wang et al., 2020). In addition to the local information captured by CNN or RNN, GCNs learn word and document representations by taking into account the global correlation information embedded in the corpus-level graph, where words and documents are nodes connected by indexing or citation relations.
However, the hidden semantic structures, such as latent topics in documents (Blei et al., 2003; Yan et al., 2013; Peng et al., 2018b), is still ignored by most of these methods (Yao et al., 2019; Huang et al., 2019; Liu et al., 2020; Zhang et al., 2020), which can improve the text representation and provide extra interpretability (in which the probabilistic generative process and topics make more sense to humans compared to neural networks, i.e. topics can be visually represented by top-10 or 20 most probable word clusters). Although few studies such as (Wang et al., 2020) have proposed incorporating a topic structure into GCNs, the topics are extracted in advance from the set of documents, independently from the graph and information propagation among documents and words. We believe that the topics should be determined in accordance with the connections in the graph. For example, the fact that two words are connected provides extra information that these words are on a similar topic(s). Moreover, existing GCN-based methods are limited by their transductive learning nature, i.e. a document can be classified only if it is already seen in the training phase (Wang et al., 2020; Yao et al., 2019; Liu et al., 2020). The lack of inductive learning ability for unseen documents is a critical issue in practical text classification applications, where we have to deal with new documents. It is
Association for Computational Linguistics: Human Language Technologies, pages 4218-4227
Table 1: Comparison with related work. We compare the manner of model learning, whether incorporate the latent topic structure and the manner of topic learning of these models.
Model	Explainability	Learning	Topics
TextGCN (Yao et al., 2019)	-	transductive	-
TensorGCN (Liu et al., 2020)	-	transductive	-
DHTG (Wang et al., 2020)	✓	transductive	static
T-GCN (Huang et al., 2019)	-	inductive	-
TG-Trans (Zhang and Zhang, 2020)	-	inductive	-
TextING (Zhang et al., 2020)	-	inductive	-
HyperGAT (Ding et al., 2020)	-	inductive	-
Our model	✓	inductive	dynamic
intuitive to decouple documents with the global graph and treat each document as an independent graph (Huang et al., 2019; Zhang et al., 2020; Ding et al., 2020; Zhang and Zhang, 2020; Xie et al., 2021). However, no attempt has been made to address both aforementioned issues.
To address these issues, we incorporate the topic model into variational graph auto-encoder (VGAE), and propose a novel framework named inductive Topic Variational Graph Auto-Encoder (T-VGAE). T-VGAE first learns to represent the words in a latent topic space by embedding and reconstructing the word correlation graph with the GCN probabilistic encoder and probabilistic decoder. Take the learned word representations as input, a GCN-based message passing probabilistic encoder is adopted to generate document representations via information propagation between words and documents in the bipartite graph. We compare our model with existing related work in Table 1. Different from previous approaches, our method unifies topic mining and graph embedding learning with VGAE, thus can fully embed the relations between documents and words into dynamic topics and provide interpretable topic structures into representations. Besides, our model builds a document-independent word correlation graph and a word-document bipartite graph for each document instead of a corpus-level graph to enable inductive learning.
1. We propose a novel model T-VGAE based on topic models and VGAE, which incorporates latent topic structures for inductively document and word representation learning. This makes the model more effective and interpretable.
2. we propose to utilize the auto-encoding vari
ational Bayes (AEVB) method to make efficient black-box inference of our model.
3. Experimental results on benchmark datasets demonstrate that our method outperforms the existing competitive GCN-based methods on supervised and semi-supervised text classification tasks. It also outperforms topic models on unsupervised text representation learning.
2.1 Graph based Text Classification
Recently, GCNs have been applied to various NLP tasks (Zhang et al., 2018; Vashishth et al., 2019). For example, TextGCN (Yao et al., 2019) was proposed for text classification, which enriches the corpus-level graph with the global semantic information to learn word and document embeddings. Inspired by it, Liu et al. (Liu et al., 2020) further considered syntactic and sequential contextual information and proposed TensorGCN. However, none of them utilized the latent semantic structures in the documents to enhance text classification. To address the issue, (Wang et al., 2020) proposed dynamic HTG (DHTG), in an attempt to integrate the topic model into graph construction. DHTG learned latent topics from the document-word correlation information (similar to traditional topic models), which will be used for GCN based document embedding. However, the topics in DHTG were learned independently from the word relation graph and the information propagation process in the graph, in which word relations are ignored.
Moreover, the existing GCN-based methods also require a pre-defined graph with all the documents and cannot handle out-of-graph documents, thus limiting their practical applicability.
To deal with the inductive learning problem, (Huang et al., 2019; Zhang et al., 2020; Ding et al., 2020; Zhang and Zhang, 2020) proposed to consider each document as an independent graph for text classification. However, the latent semantic structure and interpretability are still ignored in these methods. Different from previous approaches, we aim to deal with both issues of dynamic topic structure and inductive learning. We propose to combine the topic model and graph based information propagation in a unified framework with VGAE to learn interpretable representations for words and documents.
2.2 Graph Enhanced Topic Models
There are also studies trying to enhance topic models with efficient message passing in the graph data structure of GCNs. GraphBTM (Zhu et al., 2018) proposed to enrich the biterm topic model (BTM) with the word co-occurrence graph encoded with GCNs. To deal with data streams, (Van Linh et al., 2020) proposed graph convolutional topic model (GCTM), which introduces a knowledge graph modeled with GCNs to the topic model. (Yang et al., 2020) presented Graph Attention TOpic Network (GATON) for correlated topic modeling. It tackles the overfitting issue in topic modeling with a generative stochastic block model (SBM) and GCNs. In contrast with these studies, we focus on integrating the topic model into GCN-based VGAE for supervised learning tasks and derive word-topic and document-topic distributions simultaneously.
2.3 Variational Graph Auto-encoders
Variational Graph Auto-encoders (VGAEs) have been widely used in graph representation learning and graph generation. The earliest study (Kipf and Welling, 2016) proposed VGAE method, which extended variational auto-encoder (VAE) on graph structure data for learning graph embedding. Based on VGAE, (Pan et al., 2018) introduced an adversarial training to regularize the latent variables and further proposed adversarially regularized variational graph autoencoder (ARVGA). (Hasanzadeh et al., 2019) incorporated semi-implicit hierarchical variational distribution into VGAE (SIG-VAE) to improve the representation power of node embeddings. (Grover et al., 2019) proposed Graphite model that integrated an iterative graph refinement strategy into VGAE, inspired by low-rank approximations. However, to the best of our knowledge, our model is the first effort to apply VGAE to unify the topic learning and graph embedding for text classification, thus can provide better interpretability and overall performance.
Formally, we denote a corpus as  $C$ , which contains  $D$  documents and the ground truth labels  $Y \in c = \{1, \dots, M\}$  of documents, where  $M$  is the total number of classes in the corpus. Each document  $t \in C$  is represented by a sequence of words  $t = \{w_1, \dots, w_{n_t}\} (w_i \in v)$ , where  $n_t$  is the number of
words in document  $t$  and  $v$  is the vocabulary of size  $V$ .
From the whole corpus, we build a word correlation graph  $G = (v,e)$  containing word nodes  $v$  and edges  $e$ , to capture the word co-occurrence information. Similar to previous work (Yao et al., 2019), we utilize the positive point mutual information (PPMI) to calculate the correlation between two word nodes. Formally, for two words  $(w_i, w_j)$ , we have
$$
P P M I (w _ {i}, w _ {j}) = \max  \left(\log \frac {p \left(w _ {i} , w _ {j}\right)}{p \left(w _ {i}\right) p \left(w _ {j}\right)}, 0\right) \tag {1}
$$
where  $p(w_i, w_j)$  is the probability that  $(w_i, w_j)$  co-occur in the sliding window and  $p(w_i), p(w_j)$  are the probabilities of words  $w_i$  and  $w_j$  in the sliding window. They can be empirically estimated as  $P(w_i, w_j) = \frac{n(w_i, w_j)}{n}$  and  $P(w_i) = \frac{n(w_i)}{n}$ , where  $n(w_i, w_j)$  is the number of co-occurrences of  $(w_i, w_j)$  in the sliding windows,  $n(w_i)$  is the number of occurrences of  $w_i$  in the sliding windows and  $n$  the total number of sliding windows. For two word nodes  $(w_i, w_j)$ , the weight of the edge between them can be defined as:
$$
A _ {i, j} ^ {v} = \left\{ \begin{array}{l l} P P M I \left(w _ {i}, w _ {j}\right), & i \neq j \\ 1, & i = j \end{array} \right. \tag {2}
$$
where  $A^v \in \mathbb{R}^{V*V}$  is the adjacency matrix which represents the word correlation graph structure  $G$ .
Different from the existing studies (Yao et al., 2019; Liu et al., 2020; Wang et al., 2020) that consider all documents and words in a heterogeneous graph, we propose to build a separate graph for each document to enable inductive learning. Typically, documents can be represented by the document-word matrix  $A^d \in \mathbb{R}^{D \times V}$ , in which the row  $A_i^d = \{x_{i1}, \dots, x_{iv}\} \in \mathbb{R}^{1 \times V}$  represents the document  $i$ , and  $x_{ij}$  is the TF-IDF weight of the word  $j$  in document  $i$ . The decoupling of documents from a global pre-defined graph enables our method to handle new documents.
3.2 Topic Variational Graph Auto-encoder
Based on  $A^v$  and  $A^d$ , we propose the T-VGAE model, as shown in Figure 1. It is a deep generative model with structured latent variables based on GCNs.
3.2.1 Generative Modeling
We consider that the word co-occurrence graph  $A^v$  and the bipartite graph  $A_t^d$  of each document  $t$  are generated from the random process with two latent
Figure 1: The architecture of T-VGAE. As shown in the Figure, for a new test document  $i$ , its latent representation  $z_{i}^{d}$  is generated by the UAMP probabilistic encoder based on its document-word vector  $A_{i}^{d}$  and learned word topic distribution matrix  $z^{v}$ . Then,  $z_{i}^{d}$  is fed into the trained MLP classifier  $f_{y}$  to predict the output label. Therefore, new test documents can be classified do not need to be included in the training process, thus enabling inductive learning of our model.
variables  $z^v\in \mathbb{R}^{V\times K}$  and  $z_{t}^{d}\in \mathbb{R}^{1\times K}$ , where  $K$  denotes the number of latent topics. The generating process for  $A^v$ ,  $A^d$  and  $Y$  are as follows (see Figure 2(a)):
(a) Generative process
(b) Inference process
Figure 2: The generative and inference processes.
1. For each word  $i$  in vocabulary  $v$ , draw the latent variable  $z_{i}^{v}$  from the prior  $p_{\theta}(z_i^v)$
2. For each observed edge  $A_{i,j}^{v}$  between words  $i$  and  $j$ , draw  $A_{i,j}^{v}$  from conditional distribution  $p_{\theta}(A_{i,j}^{v}|z_{i}^{v},z_{j}^{v})$
3. For each document  $t$  in corpus  $C$ :
(a) Draw the latent variable  $z_{t}^{d}$  from the prior  $p_{\theta}(z_{t}^{d})$
(b) Draw  $A_{t}^{d}$  from the conditional distribution  $p_{\theta}(A_t^d |z_t^d,z^v)$
(c) Draw  $Y_{t}$  from the conditional distribution  $p_{\theta}(Y_t|z_t^d)$
where  $\theta$  is the set of parameters for all prior distributions. Here, we consider the centered isotropic multivariate Gaussian priors  $p(z^{v}) = \prod_{i = 1}^{V}p(z_{i}^{v}) = \prod_{i = 1}^{V}\mathcal{N}(z_i^v |0,I)$  and  $p(z^{d}) = \prod_{t = 1}^{D}p(z_{t}^{d}) = \prod_{t = 1}^{D}\mathcal{N}(z_{t}^{d}|0,I)$ .
Notice that the priors  $p(z^v)$  and  $p(z^d)$  are parameter free in this case. According to the above generative process, we can maximize the marginal likelihood of observed graph  $A^v$ ,  $A^d$  and  $Y$  to learn parameters  $\theta$  and latent variables as follows:
$$
p \left(A ^ {v}, A ^ {d}, Y \mid Z ^ {v}, Z ^ {d}, X ^ {v}\right) =
$$
$$
\prod_ {t = 1} ^ {D} p _ {\theta} \left(Y _ {t} \mid z _ {t} ^ {d}\right) p _ {\theta} \left(A _ {t} ^ {d} \mid z _ {t} ^ {d}, z ^ {v}\right) p _ {\theta} \left(z _ {t} ^ {d}\right) \tag {3}
$$
$$
\prod_ {i = 1} ^ {V} \prod_ {j = 1} ^ {V} p _ {\theta} \left(A _ {i, j} ^ {v} \mid z _ {i} ^ {v} \left(z _ {j} ^ {v}\right) ^ {\top}\right) p _ {\theta} \left(z ^ {v}\right)
$$
Because the inference of true posterior of latent variable  $z^v$  and  $z^d$  is intractable, we further introduce the variational posterior distribution  $q_{\phi}(z^{v},z^{d}|A^{d},A^{v},X^{v})$  with parameters  $\phi$  to approximate the true posterior  $p_{\theta}(z^v,z^d) = p_{\theta}(z^v)p_{\theta}(z^d)$ . We make the structured meanfield (SMF) assumption  $q_{\phi}(z^{v},z^{d}|A^{d},A^{v},X^{v}) = q_{\phi}(z^{v}|A^{v},X^{v})q_{\phi}(z^{d}|A^{d},z^{v})$ , where  $X^v\in \mathbb{R}^{V\times M}$  are the feature vectors of words and  $M$  is the dimension of the feature vectors (see Figure 2(b)). We can yield the following tractable stochastic evidence lower bound (ELBO):
$$
\begin{array}{l} \mathcal {L} (\theta , \phi ; A ^ {v}, A ^ {d}, X ^ {v}) \\ = \mathbb {E} _ {q _ {\phi} (z ^ {v} | A ^ {v}, X ^ {v})} [ \log p _ {\theta} (A ^ {v} | z ^ {v}) ] \\ + \mathbb {E} _ {q _ {\phi} (z ^ {d} | A ^ {d}, z ^ {v})} [ \log p _ {\theta} (A ^ {d} | z ^ {d}, z ^ {v}) ] \tag {4} \\ + \mathbb {E} _ {q _ {\phi} (z ^ {d} | A ^ {d}, z ^ {v})} [ \log p _ {\theta} (Y | z ^ {d}) ] \\ - K L \left[ q _ {\phi} \left(z ^ {v} \mid A ^ {v}, X ^ {v}\right) \right| \left| p _ {\theta} \left(z ^ {v}\right) \right| \\ - K L \left[ q _ {\phi} \left(z ^ {d} \mid A ^ {d}, z ^ {v}\right) \right\| p _ {\theta} \left(z ^ {d}\right) ] \\ \end{array}
$$
where the first three terms are the reconstruction terms, and the latter two terms are the Kullback-Leibler (KL) divergences of variational posterior
distributions and true posterior distributions. Using auto-encoding variational Bayes (AVB) approach (Kingma and Welling, 2013), we are able to parametrize the variational posteriors  $q_{\phi}$  and true posteriors  $p_{\theta}$  with the GCN-based probabilistic encoder and decoder, to conduct neural variational inference (NVI).
3.2.2 Graph Convolutional Probabilistic Encoder
For the latent variable  $z^v$ , we make the meanfield approximation that:  $q_{\phi}(z^{v}|A^{v},X^{v}) = \prod_{i = 1}^{V}q_{\phi}(z_{i}^{v}|A^{v},X^{v})$ . For simplify the model inference, we consider the multivariate normal variational posterior with a diagonal covariance matrix as previous neural topic models (Miao et al., 2016; Bai et al., 2018)that:  $q_{\phi}(z_i^v |A^v,X^v) = \mathcal{N}(z_i^v |\mu_i^v,\mathrm{diag}((\sigma_i^v)^2))$  where  $\mu_i^v,(\sigma_i^v)^2$  are the mean and diagonal covariance of the multivariate Gaussian distribution.
We use the graph convolutional neural network to parametrize the above posterior and inference  $z^v$  with the input graph  $A^v$  and feature vectors  $X^v$ :
$$
\begin{array}{l} (H ^ {v}) ^ {l + 1} = \rho (\hat {A} ^ {v} (H ^ {v}) ^ {l} (W ^ {v}) ^ {l}) \\ \mu^ {v} = \rho \left(\hat {A} ^ {v} \left(H ^ {v}\right) ^ {l + 1} \left(W _ {\mu} ^ {v}\right) ^ {l + 1}\right) \tag {5} \\ \log \sigma^ {v} = \rho (\hat {A} ^ {v} (H ^ {v}) ^ {l + 1} (W _ {\sigma} ^ {v}) ^ {l + 1}) \\ \end{array}
$$
where  $\mu^v,\sigma^v$  are matrices of  $\mu_i^v,\sigma_i^v,l$  is the number of GCN layers, we use one layer in our experiments,  $\{W_{\mu}^{v},W_{\sigma}^{v}\} \in \phi$  are weight matrices,  $\rho$  is the ReLU,  $\hat{A}^v = (D^v)^{-\frac{1}{2}}A^v (D^v)^{-\frac{1}{2}}$  is the symmetrically normalized adjacent matrix of the word graph, and  $D^v$  denotes the corresponding degree matrix. The input of GCN is the feature vectors  $X_{v}$  which is initialized as the identity matrix  $I$ , i.e.,  $(H^{v})^{0} = X^{v} = I$ , same as in (Yao et al., 2019). Then,  $z^v$  can be naturally sampled as follows according to the reparameterization trick (Kingma and Welling, 2013):  $z^{v} = \mu^{v} + \sigma^{v}\odot \epsilon$ , where  $\odot$  is the element-wise product, and  $\epsilon \sim \mathcal{N}(0,I)$  is the noise variable. Through the message propagation of the GCN layer, words that co-occur frequently tend to achieve similar representations in the latent topic space.
Similar to  $z^v$ , we also have:
$$
q _ {\phi} \left(z ^ {d} \mid A ^ {d}, z ^ {v}\right) = \prod_ {t = 1} ^ {D} q _ {\phi} \left(z _ {t} ^ {d} \mid A _ {t} ^ {d}, z ^ {v}\right) \tag {6}
$$
$$
q _ {\phi} \left(z _ {t} ^ {d} \mid A _ {t} ^ {d}, z ^ {v}\right) = \mathcal {N} \left(z _ {t} ^ {d} \mid \mu_ {t} ^ {d}, d i a g \left(\left(\sigma_ {t} ^ {d}\right) ^ {2}\right)\right)
$$
where  $\mu_t^d, (\sigma_t^d)^2$  are the mean and diagonal covariance of the multivariate Gaussian distribution. Although there are two types of nodes - word and
document - in the bipartite graph  $A^d$ , we mainly focus on learning representations of document nodes based on the representations of word nodes learned from  $A^v$  in this step. Therefore, we propose the unidirectional message passing (UDMP) process on  $A^d$ , which propagates the information from word nodes to documents:  $H_t^d = \rho(\sum_{i=1}^{V} A_{ti}^d z_i^v W^d)$  where  $\rho$  is the Relu activation function,  $W^d$  is the weight matrix.
Then, we parametrize the posterior and inference  $z^{d}$  based on UDMP:
$$
\begin{array}{l} \mu^ {d} = U D M P \left(A ^ {d}, z ^ {v}, W _ {\mu} ^ {d}\right) \\ \log \sigma^ {d} = U D M P \left(A ^ {d}, z ^ {v}, W _ {\sigma} ^ {d}\right) \tag {7} \\ \end{array}
$$
where  $\mu^d, \sigma^d$  are matrices of  $\mu_t^d, (\sigma_t^d)^2$ ,  $UDMP$  is the message passing as in Equation 4,  $W_{\mu}^{d}, W_{\sigma}^{d}$  are weight matrices. Similarly, we sample  $z^d$  as follows  $z^d = \mu^d + \sigma^d \odot \varepsilon$ , where  $\varepsilon \sim \mathcal{N}(0, I)$  is the noise variable. Through the propagation mechanism of UDMP, documents which share similar words tend to yield similar representations in the latent topic space.
Although T-VGAE can learn topics  $z^v$  and document-topic representations  $z^d$  as in traditional topic models, we do not focus on proposing a novel topic model, but aim to combine the topic model with VGAE, to improve word and document representations with latent topic semantic and provide probabilistic interpretability. Moreover, rather than learning topics and document-topic representations from the document-word feature  $A^d$  as LDA topic models (Blei et al., 2003), we propose to learn word-topic representations  $z^v$  from word cooccurrence matrix  $A^v$ , and then infer document-topic representations  $z^d$  based on the document-word feature  $A^d$  and word-topic representations  $z^v$ , which is similar to the Biterm topic model (Yan et al., 2013).
3.2.3 Probabilistic Decoder
With the learned  $z^v$  and  $z^d$ , ideally, the observed graph  $A^v$  and  $A^d$  can be reconstructed through a decoding process. For  $A^v$ , we assume  $P_{\theta}(A^v | z^v)$  conforms to a multivariate Gaussian distribution, whose mean parameters are generated from the inner product of the latent variable  $z^v$ :
$$
\begin{array}{l} P _ {\theta} \left(A ^ {v} \mid z ^ {v}\right) = \prod_ {i = 1} ^ {V} p _ {\theta} \left(A _ {i} ^ {v} \mid z ^ {v}\right) \tag {8} \\ p _ {\theta} \left(A _ {i} ^ {v} | z ^ {v}\right) = \prod_ {i = 1} ^ {V} \mathcal {N} \left(A _ {i} ^ {v} | \rho \left(z _ {i} ^ {v} \left(z ^ {v}\right) ^ {\top}\right), I\right) \\ \end{array}
$$
where  $\rho$  is the nonlinear activation function.
Similarly, the inner product between  $z^v$  and  $z^d$  is used to generate  $A^d$ , which is sampled from the multivariate Gaussian distribution:
$$
P _ {\theta} \left(A ^ {d} \mid z ^ {d}, z ^ {v}\right) = \prod_ {i = 1} ^ {D} p _ {\theta} \left(A _ {i} ^ {d} \mid z _ {i} ^ {d}, z ^ {v}\right) \tag {9}
$$
$$
P _ {\theta} \left(A _ {i} ^ {d} | z _ {i} ^ {d}, z ^ {v}\right) = \prod_ {i = 1} ^ {D} \mathcal {N} \left(A _ {i} ^ {d} | \rho \left(z _ {i} ^ {d} \left(z ^ {v}\right) ^ {\top}\right), I\right)
$$
For categorical labels  $Y$ , we assume  $p_{\theta}(Y|z^d)$  follows a multinomial distribution  $P_{\theta}(Y|z^{d}) = Mul(Y|f_{y}(z^{d}))$ , whose label probability vectors are generated from  $z^{d}$ , where  $f_{y}$  is the multi-layer neural network. For each document  $t$ , the prediction is given by  $\hat{y}_t = \underset{y \in c}{\text{argmax}} P_{\theta}(y|f_y(z_t^d))$ .
3.2.4 Optimization
We can rewrite Equation 4 to yield the final variational objective function:
$$
\begin{array}{l} \mathcal {L} (\theta , \phi) \approx \sum_ {i = 1} ^ {V} \sum_ {j = 1} ^ {V} \log p _ {\theta} \left(A _ {i, j} ^ {v} \mid z _ {i} ^ {v}, z _ {j} ^ {v}\right) \\ + \sum_ {t = 1} ^ {D} \left(\log p _ {\theta} \left(A _ {t} ^ {d} \mid z _ {t} ^ {d}, z ^ {v}\right) + \log p _ {\theta} \left(Y _ {t} \mid z _ {t} ^ {d}\right)\right) \tag {10} \\ - K L \left[ q _ {\phi (z ^ {v})} | | p _ {\theta} \left(z ^ {v}\right) \right] \\ - K L \left[ q _ {\phi (z ^ {d})} | | p _ {\theta} \left(z ^ {d}\right) \right] \\ \end{array}
$$
with following reconstruction terms and KL divergences:
$$
\log p _ {\theta} \left(A _ {i} ^ {v} | z ^ {v}\right) \approx \left| \left| A _ {i} ^ {v} - \rho \left(z _ {i} ^ {v} \left(z ^ {v}\right) ^ {\mathsf {T}}\right) \right| \right| ^ {2}
$$