SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Shengyuan Chen
Department of Computing  
The Hong Kong Polytechnic University  
Hung Hom, Hong Kong SAR  
shengyuan.chen@connect.polyu.hk
Yunfeng Cai
Cognitive Computing Lab Baidu Research 10 Xibeiwang East Rd., Beijing, China caiyunfeng@baidu.com
Huang Fang
Cognitive Computing Lab Baidu Research 10 Xibewang East Rd., Beijing, China fanghuang@baidu.com
Xiao Huang
Department of Computing  
The Hong Kong Polytechnic University  
Hung Hom, Hong Kong SAR  
xiaohuang@comp.polyu.edu.hk
Mingming Sun
Cognitive Computing Lab
Baidu Research
10 Xibeiwang East Rd., Beijing, China
sunmingming01@baidu.com
Knowledge graph (KG) reasoning utilizes two primary techniques, i.e., rule-based and KG-embedding based. The former provides precise inferences, but inferring via concrete rules is not scalable. The latter enables efficient reasoning at the cost of ambiguous inference accuracy. Neuro-symbolic reasoning seeks to amalgamate the advantages of both techniques. The crux of this approach is replacing the predicted existence of all possible triples (i.e., truth scores inferred from rules) with a suitable approximation grounded in embedding representations. However, constructing an effective approximation of all possible triples' truth scores is a challenging task, because it needs to balance the tradeoff between accuracy and efficiency, while compatible with both the rule-based and KG-embedding models. To this end, we proposed a differentiable framework - DiffLogic. Instead of directly approximating all possible triples, we design a tailored filter to adaptively select essential triples based on the dynamic rules and weights. The truth scores assessed by KG-embedding are continuous, so we employ a continuous Markov logic network named probabilistic soft logic (PSL). It employs the truth scores of essential triples to assess the overall agreement among rules, weights, and observed triples. PSL enables end-to-end differentiable optimization, so we can alternately update embedding and weighted rules. On benchmark datasets, we empirically show that DiffLogic surpasses baselines in both effectiveness and efficiency.
Knowledge graph (KG) reasoning refers to the process of using existing triples in a KG to infer new knowledge that is not explicitly stated in the original KG (Safavi & Koutra, 2020). For instance, if we know that (Paris, is_the_capital_of, France) and (France, is_a_country_in, Europe), we can infer
that (Paris, is_a_city_in, Europe), which is not explicitly included in the original KG. KG reasoning is essential to KG completion (Shi & Weninger, 2017) and KG error detection (Zhang et al., 2022a; Dong et al., 2023b). Moreover, KG reasoning can infer underlying knowledge and improve the quality of learning or predictions on KGs, which benefits various downstream tasks, such as question answering (Huang et al., 2019; Dong et al., 2023a), recommendations (Huang et al., 2023; Xu et al., 2023), and interpretable machine learning (Lecue, 2020; Tiddi & Schlobach, 2022).
There are mainly two lines of research in KG reasoning (Zhang et al., 2022b). First, rule-based reasoning derives new triples from existing ones by applying a set of predefined rules, which are usually expressed in a form of logical statements (Yang et al., 2017; Sadeghian et al., 2019; Fang et al., 2023). For example, if there are two triples  $(A, is\_a\_parent\_of, B)$  and  $(B, is\_a\_parent\_of, C)$ , and a rule saying "the parent of a parent is a grandparent", then we can infer a new triple  $(A, is\_a\_grandparent\_of, C)$ . Rule-based reasoning is particularly effective when the rules are well-defined and the KG is relatively small and static (Bach et al., 2017). Recent studies employ Markov logic network (MLN) (Richardson & Domingos, 2006) to dynamically learn soft rules, which are associated with weight scores indicating their credibility. But it remains not scalable because assessing the truth scores to all possible triples requires  $\mathcal{O}(|\mathcal{E}|^2|\mathcal{R}|)$  parameters, where  $|\mathcal{E}|$  and  $|\mathcal{R}|$  denote the numbers of entities and relations. Thus, rule-based reasoning becomes inefficient when dealing with large KGs. Second, KG-embedding based reasoning projects a KG into a low-dimensional space and infers new relations based on embedding representations (Zhang et al., 2021; Ren et al., 2022). It assumes that KG embedding can preserve most semantic information in the original KGs (Bordes et al., 2013; Wang et al., 2014; Sun et al., 2019; Dong et al., 2014) so that missing relations can be inferred by using the distances or semantic matching scores in the embedding space. Recent studies (Shang et al., 2019; ?; Vashishth et al., 2019; Zhang et al., 2023) also seek to design tailored Graph convolutional networks for learning structural-aware KG embeddings. KG-embedding based reasoning can be scalable (Zheng et al., 2020; Ren et al., 2022). However, there is no explicit rule to ensure its accuracy and logical consistency. Also, it requires a sufficient amount of training data, and its results are difficult to interpret.
Neuro-symbolic reasoning models combine the advantages of both techniques, which is to approximate the predicted truth scores of all possible triples inferred by rules with the normalized output scores of a KG-embedding model. However, it is nontrivial to effectively and efficiently construct such an approximation. Directly employing rules (Guo et al., 2016, 2018) to regularize the embedding learning is efficient, but it cannot update rule weights and is thus sensitive to the initialization of rule weights. Incorporating KG-embedding models into sophisticated rule-based models, such as MLN, enables the handling of rule uncertainty (Qu & Tang, 2019; Zhang et al., 2020). However, directly approximating the distributions of the truth scores are still challenging. First, directly approximating the distribution of truth scores within the MLN framework is unfeasible. This process necessitates the optimization of the joint probability of the approximated distribution, which subsequently requires the computation of integration across all scores. Second, optimizing neuro-symbolic models rely on training with ground formulas (i.e. instantiated rules). Given the large grounding space, the performance of neuro-symbolic models may deteriorate if important ground formulas are missed (Guo et al., 2016; Qu & Tang, 2019), and efficiency issues may arise if too many ground formulas are considered (Zhang et al., 2020).
To this end, we proposed a differentiable framework - DiffLogic. Firstly, a tailored filter is used to adaptively select important ground formulas based on weighted rules and extract connected triples. Second, a KG-embedding model is used to compute a truth score for each triple. Then we employ a continuous MLN named probabilistic soft logic (PSL) (Bach et al., 2017) that takes these truth scores as input, and assesses the overall agreement among rules, weights, and observed triples with a joint probability. The PSL template enables end-to-end differentiable optimization. In this way, DiffLogic is optimized by alternately updating embedding and weighted rules. Contributions of this work are summarized as follows:
(1) We develop a unified neuro-symbolic framework — DiffLogic, that combines the advantages of KG-embedding models and rule-based models: efficiency, effectiveness, capability of leveraging prior knowledge and handling uncertainty.
(2) We enable consistent training of rule-based models and KG-embedding models. By employing PSL, the joint probability of truth scores can be optimized directly rather than optimizing an evidence lower bound (ELBO) instead.
(3) We propose an efficient grounding technique that iteratively identifies important ground formulas required for inference, enabling effective and data-efficient optimization.
(4) We devise a fast estimation technique for the gradient of rule weights, which efficiently estimates the rule weight gradient by exploiting the sparsity of violated ground formulas.
(5) Through experiments, we empirically show that: (i) DiffLogic scales to large knowledge graphs with consistently improved performance, such as YAGO3-10; (ii) our model can leverage human prior knowledge by injecting rule patterns using only a compact set of logic rules; and (iii) our grounding technique is both efficient and effective, it reduces the number of ground formulas required for optimization by orders of  $10^{3} \sim 10^{5}$ , without compromising performance.
In this section, we formally define the problem of knowledge base completion, provide a brief introduction to first-order logic, and how to evaluate its agreement with a knowledge base.
2.1 Problem statement
A knowledge base  $\mathcal{K}$  comprises a set of entities  $\mathcal{E}$  and a set of relations  $\mathcal{R}$ . For any pair of head-tail entities  $(h,t)\in \mathcal{E}\times \mathcal{E}$  and a relation  $r\in \mathcal{R}$ , the relation maps the pair of entities to a binary score, i.e.,  $r:\mathcal{E}\times \mathcal{E}\to \{0,1\},\forall r\in \mathcal{R}$ , indicating that the head entity  $h$  either has the relation  $r$  with the tail entity  $t$  or not. In the knowledge base completion problem, people observe a set of facts  $\mathcal{O} = \{(h_i,r_i,t_i)\}_{i = 1}^n$  along with their true assignment vector  $\pmb {x} = [x_{1},\dots ,x_{|\mathcal{O}|}]\in \mathbb{R}^{|\mathcal{O}|}$  with  $x_{i} = r_{i}(h_{i},t_{i})$ . Denote the unobserved facts as  $\mathcal{H} = \mathcal{E}\times \mathcal{R}\times \mathcal{E}\backslash \mathcal{O}$ , and let  $\{(F_q,W_q)\}_{q = 1}^m$  be a set of weighted rules, where  $F_{q}$  is a rule (see Section 2.2 for details, where a rule is referred to as first-order logic),  $W_{q}$  is the corresponding rule weight. The knowledge graph completion task aims to infer the assignment vector for all unobserved facts  $\pmb {y}\in \mathbb{R}^{|\mathcal{H}|}$  given the observed facts and the rules.
2.2 First-order logic
First-order logic. A first-order logic (also referred to as "logic rule" in this paper) associated with a knowledge base  $\mathcal{K}$  is an expression based on relations in  $\mathcal{K}$ . Formally, a logic rule  $F_{q}$  in clausal form can be represented as a disjunction of atoms and negated atoms:
$$
\left(\vee_ {i \in \mathcal {I} _ {q} ^ {+}} r _ {i} \left(A _ {i}, B _ {i}\right)\right) \vee \left(\vee_ {i \in \mathcal {I} _ {q} ^ {-}} \neg r _ {i} \left(A _ {i}, B _ {i}\right)\right), \tag {1}
$$
where  $\mathcal{I}_q^-$  and  $\mathcal{I}_q^+$  are two index sets containing the indices of atoms that are negated or not, respectively,  $A_{i}$  and  $B_{i}$  are variables. Logic rules in clausal form can be equivalently reorganized as an implication from premises (negated) to conclusions (non-negated):
$$
\wedge_ {i \in \mathcal {I} _ {q}} r _ {i} \left(A _ {i}, B _ {i}\right) \Longrightarrow \vee_ {i \in \mathcal {I} _ {q} ^ {+}} r _ {i} \left(A _ {i}, B _ {i}\right). \tag {2}
$$
The implication Eq. (2) is quite expressive since it includes many commonly used types of logic rule, e.g., symmetry/asymmetry, inversion, sub-relation, composition, etc.
2.3 Rule grounding and distance to satisfaction
Rule grounding. For a logic rule  $F_{q}$  in Eq. (2), by assigning entities  $e \in \mathcal{E}$  to the variables  $A_{i}$  and  $B_{i}$ ,  $F_{q}$  is grounded, producing a set of ground formulas. For example, consider a simple logic rule Father(A,B) ∧ Wife(C,A) ⇒ Mother(C,B). Let  $A = \text{Jack}$ ,  $B = \text{Ross}$  and  $C = \text{Judy}$ , we get a ground formula: Father(Jack,Ross) ∧ Wife(Judy,Jack) ⇒ Mother(Judy,Ross).
Distance-to-satisfaction. Denote all ground formulas created by the  $q$ -th logic rule  $F_{q}$  by  $\{G_{q}^{(j)}\}_{j = 1}^{n_{q}}$  where  $n_q$  is the number of ground formula for the  $q$ -th rule. For any ground formula  $G_{q}^{(j)}$  of  $F_{q}$ , when  $r_i(h_i,t_i)\in \{0,1\}$ , the satisfaction of  $G_{q}^{(j)}$  can be evaluated via Eq. (1). The value is either 1 or 0, meaning that the ground formula is satisfied or violated, respectively. The binary value of  $r_i(h_i,t_i)$  can be relaxed to a continuous value ranging over [0, 1]. In such case, we may define the
Figure 1: Overall framework of DiffLogic.
distance-to-satisfaction for  $G_{q}^{(j)}$  via Lukasiewicz t-norm:
$$
\mathrm {d} \left(G _ {q} ^ {(j)}\right) := \max  \left\{1 - \sum_ {i \in \mathcal {I} _ {q} ^ {+}} r _ {i} \left(h _ {i}, t _ {i}\right) - \sum_ {i \in \mathcal {I} _ {q} ^ {-}} \left(1 - r _ {i} \left(h _ {i}, t _ {i}\right)\right), 0 \right\}. \tag {3}
$$
Note that  $d(G_q^{(j)}) \in [0,1]$ , and the smaller  $d$  is, the better satisfied the ground formula  $G_q^{(j)}$  is.
3 Differentiable neuro-symbolic reasoning
In this section, we propose a neuro-symbolic model, namely, DiffLogic, which unifies KG-embedding and rule-based reasoning. What follows we present the overall framework of DiffLogic, then show how to perform DiffLogic efficiently.
3.1 Overall framework
As shown in Figure 1, the model comprises three components: 1) an efficient grounding technique serving as a filter to identify crucial ground formulas and extract triples connected to them; 2) a KG-embedding model to compute truth scores for the extracted triples; and 3) a tailored continuous MLN framework that takes the truth scores as input and assess the overall probability. The model is optimized using an EM algorithm, alternating between embedding optimization and weight updating. During the E-step, we fix the rule weights and optimize embeddings in an end-to-end fashion, by maximizing the overall probability; while in the M-step, we design an efficient rule weight updating method by leveraging the sparsity of violated rules. It is also worth mentioning here that the model requires a set of rules, which can be obtained from certain rule-mining process or domain experts.
Next, we turn to the abovementioned three components.
Rule-guided iterative grounding. The success of probabilistic logic reasoning heavily depends on the grounding process. However, the number of grounding formulas is overwhelmingly large, under our setting, equals  $\sum_{q=1}^{m} |\mathcal{E}|^{\left|\mathcal{I}_q^-\right| + 1}$ . As a result, one has to sample for approximation. In this work, we propose a grounding technique called Rule-guided Iterative Grounding (RGIG) that incrementally identifies crucial ground formulas, reducing the number of ground formulas needed for optimization without compromising performance.
Inference inherently promotes the agreement between assignments (i.e.,  $\mathbf{y}$ ,  $\mathbf{x}$ ) and the weighted rule set (i.e.,  $\{(F_q, W_q)\}_{q=1}^m$ ) by penalizing violated rules. Consequently, ground formulas with higher distance-to-satisfaction are more valuable in guiding optimization. In light of this, we only need to find the ground formulas whose premise atoms in Eq. (2) have (or potentially have) high scores. Pursuing this idea, RGIG iteratively grounds all logic rules and updates a fact set  $\mathcal{V}$ . The fact set  $\mathcal{V}$  is initialized with the observed facts  $\mathcal{O}$  from the training set. In each iteration, rules are grounded only from the facts in  $\mathcal{V}$  that match the premise parts of the rules. New facts are derived from the conclusion parts of the rules and subsequently added to  $\mathcal{V}$ . In the subsequent iterations, the updated
Figure 2: An illustration of probabilistic soft logic (PSL).
$\nu$  is used to derive more new facts. From our numerical experience, a few iterations of RGIG are sufficient, say 3. In the end, RGIG yields a set of ground formulas.
This grounding technique leverages the sparsity of violated rules (the distances to satisfaction for most ground formulas are approximately zero) to efficiently identify important ground formulas, facilitating data-efficient optimization. The case study in Section 4.2 illustrates that our grounding technique attains comparable reasoning performance and is orders of  $(10^{3} \sim 10^{5})$  more efficient in terms of the number of ground formulas.
KG-embedding model. In the literature of probabilistic logic reasoning, the representation of assignment  $\pmb{y}$  for all unobserved facts is very expensive, which is approximate  $|\mathcal{E}|^2 |\mathcal{R}|$  as the observed fact triples are only a small portion. To this end, we employ a KG-embedding model to parameterize  $\pmb{y}$  and  $\pmb{x}$ . By embedding each entity to continuous representation with  $n_e$  parameters, and each relation with  $n_r$  parameters. The number of parameters is reduced from  $|\mathcal{E}|^2 |\mathcal{R}|$  to  $|\mathcal{E}|n_e + |\mathcal{R}|n_r$ , which is linear with respect to the number of entities/relations.
Although (re)-parameterization improves the representation efficiency, we also need to consider effectiveness. The relation pattern modeling capability is essential for honestly modeling the logic in a KG and performing reasoning. For example, the ability to model symmetric and reverse relation is attributed to capturing logic rules with one premise atom (e.g.  $Husband(A,B)\Rightarrow Wife(B,A))$ , while the ability to model composition relation is attributed to capturing logic rules with two premise atoms (e.g. Father  $(A,B)\land Wife(C,A)\Rightarrow Mother(C,B))$ . Among all candidate KG-embedding models, we choose RotatE (Sun et al., 2019) for its simplicity and capability of modeling the various logic patterns. And of course, one may use also other KGE models instead. Let the KGE model be parameterized by  $\theta$ . Using the sigmoid function, we can transform the score for the fact triple  $(h,r,t)$  produced by the KGE model into [0, 1], which can be taken as the truth value for  $(h,r,t)$ . So the truth value for  $(h,r,t)$  can be parameterized by  $\theta$ . Therefore, the assignment vectors  $\mathbf{x}$  and  $\mathbf{y}$  can be parameterized by  $\theta$ , with their entries being the truth values for the observed and unobserved fact triples, respectively.
Hinge-loss Markov random field. Given the parameterized assignments, we employ PSL (Bach et al., 2017) to build a hinge-loss Markov random field (HL-MRF). Specifically, given a knowledge base  $\mathcal{K}$  with assignments  $\pmb{x},\pmb{y}$ , and a set of weighted logic rules  $\{(F_q,W_q)\}_{q = 1}^m$ . A HL-MRF  $P$  over  $\pmb{y}$  conditioned on  $\pmb{x}$  is a probability density function defined as
$$
P _ {\boldsymbol {w}} (\boldsymbol {y} \mid \boldsymbol {x}) = \frac {1}{Z (\boldsymbol {W} , \boldsymbol {x})} \exp (- f _ {\boldsymbol {w}} (\boldsymbol {y}, \boldsymbol {x})), \quad Z (\boldsymbol {W}, \boldsymbol {x}) = \int_ {\boldsymbol {y}} \exp (- f _ {\boldsymbol {w}} (\boldsymbol {y}, \boldsymbol {x})) \mathrm {d} \boldsymbol {y}, \tag {4}
$$
where  $f_{w}$  is the hinge-loss energy function, defined as the weighted sum of all potentials:
$$
f _ {\boldsymbol {w}} (\boldsymbol {y}, \boldsymbol {x}) = \boldsymbol {W} ^ {\top} \boldsymbol {\Phi} (\boldsymbol {y}, \boldsymbol {x}) = \left[ W _ {1}, \dots , W _ {m} \right] \left[ \Phi_ {1} (\boldsymbol {y}, \boldsymbol {x}), \dots , \Phi_ {m} (\boldsymbol {y}, \boldsymbol {x}) \right] ^ {\top} = \sum_ {q = 1} ^ {m} W _ {q} \Phi_ {q} (\boldsymbol {y}, \boldsymbol {x}), \tag {5}
$$
and  $\Phi_q(\pmb {y},\pmb {x})$  is the sum of potentials of all ground formulas of  $F_{q}$  , i.e.,  $\Phi_q(\pmb {y},\pmb {x}) = \sum_{j = 1}^{n_q}\mathrm{d}(G_q^{(j)})$
HL-MRF optimizes the assignment  $\mathbf{y}$  and the rule weights  $\mathbf{W}$  by alternating between Maximum a posterior (MAP) inference and weight learning. The former step fixes the  $\mathbf{W}$  and optimizes  $\mathbf{y}$ , while the latter step fixes  $\mathbf{y}$  and updates  $\mathbf{W}$ .
3.2 Optimization
Below we elaborate on the optimization details of DiffLogic. We demonstrate how the model optimization is efficiently performed by employing numerical optimization techniques and approximation methods that leverage sparse properties.
**Embedding updating** The task in the inference step is to infer  $P_{\boldsymbol{w}}(\boldsymbol{y}|\boldsymbol{x})$ , i.e., finding the optimum  $\boldsymbol{y}$  given the observed assignment  $\boldsymbol{x}$  and current weights  $\boldsymbol{W}$ , which can be formulated as a maximum a posterior (MAP):
$$
\operatorname {a r g m a x} _ {\boldsymbol {y}} P _ {\boldsymbol {w}} (\boldsymbol {y} | \boldsymbol {x}) \equiv \operatorname * {a r g m i n} _ {\boldsymbol {y} \in [ 0, 1 ] ^ {| \mathcal {H} |}} \boldsymbol {W} ^ {\top} \boldsymbol {\Phi} (\boldsymbol {y}, \boldsymbol {x}).
$$
To be consistent with the observation,  $x(\theta)$  should be as small as possible. Additionally, we also want the truth values for negative samples to be large. So, the overall cost function can be formulated as:
$$
\min  _ {\theta} \boldsymbol {W} ^ {\top} \Phi (\boldsymbol {y} (\theta), \boldsymbol {x} (\theta)) + \lambda \left(\frac {1}{| T ^ {+} |} \sum_ {(h, r, t) \in T ^ {+}} [ 1 - r (h, t; \theta) ] + \frac {1}{| T ^ {-} |} \sum_ {(h, r, t) \in T ^ {-}} r (h, t; \theta)\right), \tag {6}
$$
where  $\pmb{x}$ ,  $\pmb{y}$  are parameterized by  $\theta$ ,  $\lambda > 0$  is a parameter,  $r(h, t; \theta) \in [0,1]$  stands for the truth value of a fact triple  $(h, r, t)$ ,  $T^{+}$  and  $T^{-}$  are positive and negative sample sets, respectively. The optimization problem Eq. (6) can be solved by stochastic gradient descent, where we sample a batch of ground formulas to compute the potentials,  $T^{+}$  is a batch of positive triples from  $\mathcal{O}$ , and  $T^{-}$  is a set of negative samples obtained by corrupting  $T^{+}$ .
Rule weights updating In this step,  $\theta$  is fixed, and rule weights are updated by maximizing  $\log P_{\pmb{w}}(\pmb{y}|\pmb{x})$ . The gradient of  $\log P_{\pmb{w}}(\pmb{y}|\pmb{x})$  with respect to the weight of  $F_{q}$  can be given by
$$
\frac {\partial \log P _ {\boldsymbol {w}} (\boldsymbol {y} \mid \boldsymbol {x})}{\partial W _ {q}} = \mathbb {E} _ {\boldsymbol {W}} \left[ \Phi_ {q} (\boldsymbol {y}, \boldsymbol {x}) \right] - \Phi_ {q} (\boldsymbol {y}, \boldsymbol {x}). \tag {7}
$$
The first term computes the expectation of potential, which involves integration over  $\mathbf{y}$  under the distribution defined by rule weight  $\mathbf{W}$ . Directly computing the integration is impractical, so we instead optimize the pseudo-likelihood of training data:
$$
P _ {\pmb {w}} ^ {*} (\pmb {y} \mid \pmb {x}) = \prod_ {i = 1} ^ {n} \frac {\exp \left[ - f _ {\pmb {w}} ^ {i} (y _ {i} \cup \pmb {y} _ {\setminus i} , \pmb {x}) \right]}{Z _ {i} (\pmb {W} , y _ {i} \cup \pmb {y} _ {\setminus i} , \pmb {x})}, \quad f _ {\pmb {w}} ^ {i} = \sum_ {q = 1} ^ {m} W _ {q} \sum_ {j = 1} ^ {n _ {q}} \mathbf {1} _ {\{y _ {i} \to G _ {q} ^ {(j)} \}} d (G _ {q} ^ {(j)}).
$$
Here  $\mathbf{1}_{\{y_i \to G_q^{(j)}\}} = 1$  if  $y_i$  is connected to ground formula  $G_q^{(j)}$ , otherwise, zero.  $Z_i$  is the integration of  $f_w^i$  over  $y_i$ , similar to the second equation in Eq. (4). The pseudo-likelihood is a mean-field approximation for the exact likelihood in Eq. (4). It approximates the exact likelihood by decomposing it into a product of the probabilities of  $y_i$ 's, conditioned on the Markov Blanket of  $y_i$  denoted by  $\mathrm{MB}(y_i)$ . The gradient of the pseudo-likelihood is as follows (see Appendix A.2 for a detailed derivation):
$$
\frac {\partial \log P _ {\pmb {w}} ^ {*} (\pmb {y} \mid \pmb {x})}{\partial W _ {q}} = \sum_ {i = 1} ^ {n} \left\{\mathbb {E} _ {y _ {i} | \mathrm {M B}} \left[ \Psi_ {q, M B (i)} \right] - \Psi_ {q, M B (i)} \right\}, \Psi_ {q, M B (i)} = \sum_ {j = 1} ^ {n _ {q}} \mathbf {1} _ {\{y _ {i} \to G _ {q} ^ {(j)} \}} d (G _ {q} ^ {(j)}).
$$
The above equality enables us to estimate gradients by minibatch sampling. For each sampled  $y_{i}$ , we need to compute both  $\Psi_{q,MB(i)}$  and its expectation. The integration term can be estimated using Monte Carlo integration by fixing other variables and sampling  $y_{i}$  on the interval [0, 1]. The Monte Carlo integration is parallelizable and converges quickly as the number of samples increases.