SlowGuess/ACL_2020_OCR huggingface dataset:
Wentao Xu $^{1}$ , Shun Zheng $^{2}$ , Liang He $^{2}$ , Bin Shao $^{2}$ , Jian Yin $^{1*}$ , and Tie-Yan Liu $^{2}$ $^{1}$  School of Data and Computer Science, Sun Yat-sen University, Guangzhou, China; Guangdong Key Laboratory of Big Data Analysis and Processing, Guangzhou, China  
 $^{2}$  Microsoft Research Asia, Beijing, China  
{xuwt6@mail2, issjyin@mail1}.sysu.edu.cn  
{Shun.Zheng, Liang.He, binshao, tyliu}@microsoft.com
In recent years, knowledge graph embedding becomes a pretty hot research topic of artificial intelligence and plays increasingly vital roles in various downstream applications, such as recommendation and question answering. However, existing methods for knowledge graph embedding can not make a proper trade-off between the model complexity and the model expressiveness, which makes them still far from satisfactory. To mitigate this problem, we propose a lightweight modeling framework that can achieve highly competitive relational expressiveness without increasing the model complexity. Our framework focuses on the design of scoring functions and highlights two critical characteristics: 1) facilitating sufficient feature interactions; 2) preserving both symmetry and antisymmetry properties of relations. It is noteworthy that owing to the general and elegant design of scoring functions, our framework can incorporate many famous existing methods as special cases. Moreover, extensive experiments on public benchmarks demonstrate the efficiency and effectiveness of our framework. Source codes and data can be found at https://github.com/Wentao-Xu/SEEK.
Learning embeddings for a knowledge graph (KG) is a vital task in artificial intelligence (AI) and can benefit many downstream applications, such as personalized recommendation (Zhang et al., 2016; Wang et al., 2018) and question answering (Huang et al., 2019). In general, a KG stores a large collection of entities and inter-entity relations in a triple format,  $(h,r,t)$ , where  $h$  denotes the head entity,  $t$  represents the tail entity, and  $r$  corresponds to the relationship between  $h$  and  $t$ . The goal of knowledge graph embedding (KGE) is to project massive
interconnected triples into a low-dimensional space and preserve the initial semantic information at the same time.
Although recent years witnessed tremendous research efforts on the KGE problem, existing research did not make a proper trade-off between the model complexity (the number of parameters) and the model expressiveness (the performance in capturing semantic information). To illustrate this issue, we categorize existing research into two categories.
The first category of methods prefers the simple model but suffers from poor expressiveness. Some early KGE methods, such as TransE (Bordes et al., 2013) and DistMult (Yang et al., 2015), fell into this category. It is easy to apply these methods to large-scale real-world KGs, but their performance in capturing semantic information (such as link prediction) is far from satisfactory.
In contrast, the second category pursues the excellent expressiveness but introduces much more model parameters and tensor computations. Typical examples include TransH (Wang et al., 2014), TransR (Lin et al., 2015), TransD (Ji et al., 2015), Single DistMult (Kadlec et al., 2017), ConvE (Dettmers et al., 2018) and InteractE (Vashishth et al., 2019). However, as pointed out by Dettmers et al. (2018), the high model complexity often leads to poor scalability, which is prohibitive in practice because real-world KGs usually contain massive triples.
To address these drawbacks of existing methods, in this paper, we propose a light-weight framework for KGE that achieves highly competitive expressiveness without the sacrifice in the model complexity. Next, we introduce our framework from three aspects: 1) facilitating sufficient feature interactions, 2) preserving various necessary relation properties, 3) designing both efficient and effective scoring functions.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3888-3897 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
First, to pursue high expressiveness with the reasonable model complexity, we need to facilitate more sufficient feature interactions given the same number of parameters. Specifically, we divide the embedding dimension into multiple segments and encourage the interactions among different segments. In this way, we can obtain highly expressive representations without increasing model parameters. Accordingly, we name our framework as Segmented Embedding for KGs (SEEK).
Second, it is crucial to preserve different relation properties, especially the symmetry and the antisymmetry. We note that some previous research did not preserve the symmetry or the antisymmetry and thus obtained inferior performance (Bordes et al., 2013; Lin et al., 2015; Yang et al., 2015). Similar to the recent advanced models (Trouillon et al., 2016; Kazemi and Poole, 2018; Sun et al., 2019; Xu and Li, 2019), we also pay close attention to the modeling support of both symmetric and antisymmetric relationships.
Third, after an exhaustive review of the literature, we find that one critical difference between various KGE methods lies in the design of scoring functions. Therefore, we dive deeply into designing powerful scoring functions for a triple  $(h,r,t)$ . Specifically, we combine the above two aspects (facilitating feature interactions and preserving various relation properties) and develop four kinds of scoring functions progressively. Based on these scoring functions, we can specify many existing KGE methods, including DistMult (Yang et al., 2015), HoIE (Nickel et al., 2016), and ComplEx (Trouillon et al., 2016), as special cases of SEEK. Hence, as a general framework, SEEK can help readers to understand better the pros and cons of existing research as well as the relationship between them. Moreover, extensive experiments demonstrate that SEEK can achieve either state-of-the-art or highly competitive performance on a variety of benchmarks for KGE compared with existing methods.
- We propose a light-weight framework (SEEK) for KGE that achieves highly competitive expressiveness without the sacrifice in the model complexity.
- As a unique framework that focuses on designing scoring functions for KGE, SEEK combines two critical characteristics: facilitating
sufficient feature interactions and preserving fundamental relation properties.
- As a general framework, SEEK can incorporate many previous methods as special cases, which can help readers to understand and compare existing research.
- Extensive experiments demonstrate the effectiveness and efficiency of SEEK. Moreover, sensitivity experiments about the number of segments also verify the robustness of SEEK.
We can categorize most of the existing work into two categories according to the model complexity and the model expressiveness.
The first category of methods is the simple but lack of expressiveness, which can easily scale to large knowledge graphs. This kind of methods includes TransE (Bordes et al., 2013) and DistMult (Yang et al., 2015). TransE uses relation  $r$  as a translation from a head entity  $h$  to a tail entity  $t$  for calculating their embedding vectors of  $(h,r,t)$ ; DistMult utilizes the multi-linear dot product as the scoring function.
The second kind of work introduces more parameters to improve the expressiveness of the simple methods. TransH (Wang et al., 2014), TransR (Lin et al., 2015), TransD (Ji et al., 2015), and ITransF (Xie et al., 2017) are the extensions of TransE, which introduce other parameters to map the entities and relations to different semantic spaces. The Single DistMult (Kadlec et al., 2017) increases the embedding size of the DistMult to obtain more expressive features. Besides, ProjE (Shi and Weninger, 2017), ConvE (Dettmers et al., 2018) and InteractE (Vashishth et al., 2019) leverage neural networks to capture more feature interactions between embeddings and thus improves the expressiveness. However, these neural network-based methods would also lead to more parameters since there are many parameters in the neural network. Although the second kind of methods has a better performance compared with simple methods, they are difficult to apply to real-world KGs due to the high model complexity (a large number of parameters).
Compared with the two types of methods above, our SEEK can achieve high expressiveness without increasing the number of model parameters.
Methods	Scoring Function	Performance	# Parameters	Properties
Sym	Antisym
TransE (Bordes et al., 2013)	||h + r - t||	Low	Small	X	✓
DistMult (Yang et al., 2015)	<h, r, t>	Low	Small	✓	X
ComplEx (Trouillon et al., 2016)	Re(⟨h, r, t⟩)	Low	Small	✓	✓
Single DistMult (Kadlec et al., 2017)	<h, r, t>	High	Large	✓	X
ConvE (Dettmers et al., 2018)	f( vec(f([h, r] * ω))W)t	High	Large	X	✓
SEEK	∑ sx,y ⟨rx, hy, twx,y⟩	High	Small	✓	✓
Table 1: Comparison between our SEEK framework and some representative knowledge graph embedding methods in the aspects of the scoring function, performance, the number of parameters, and the ability to preserve the symmetry and antisymmetry properties of relations.
Table 1 shows the comparison between our framework and some representative KGE methods in different aspects.
Besides, preserving the symmetry and antisymmetry properties of relations is vital for KGE models. Many recent methods devote to preserving these relation properties to improve the expressiveness of embeddings (Trouillon et al., 2016; Nickel et al., 2016; Guo et al., 2018; Ding et al., 2018; Kazemi and Poole, 2018; Sun et al., 2019; Xu and Li, 2019). Motivated by these methods, we also pay attention to preserving symmetry and antisymmetry properties of relations when we design our scoring functions.
3 SEEK
Briefly speaking, we build SEEK by designing scoring functions, which is one of the most critical components of various existing KGE methods, as discussed in the related work. During the procedure of designing scoring functions, we progressively introduce two characteristics that hugely contribute to the model expressiveness: 1) facilitating sufficient feature interactions; 2) supporting both symmetric and antisymmetric relations. In this way, SEEK enables the excellent model expressiveness given a light-weight model with the same number of parameters as some simple KGE counterparts, such as TransE (Bordes et al., 2013) and DistMult (Yang et al., 2015).
3.1 Scoring Functions
In this section, we illustrate our four scoring functions progressively.
3.1.1  $f_{1}$ : Multi-linear Dot Product
First, we start with the scoring function  $f_{1}$  developed by Yang et al. (2015), which computes a multi
linear dot product of three vectors:
$$
f _ {1} (h, r, t) = \langle \mathbf {r}, \mathbf {h}, \mathbf {t} \rangle = \sum_ {i} r _ {i} \cdot h _ {i} \cdot t _ {i}, \tag {1}
$$
where  $\mathbf{r},\mathbf{h}$ , and  $\mathbf{t}$  are low-dimensional representations of the relation  $r$ , the head entity  $h$ , and the tail entity  $t$ , respectively, and  $r_i,h_i$ , and  $t_i$  correspond to the  $i$ -th dimension of  $\mathbf{r},\mathbf{h}$ , and  $\mathbf{t}$ , respectively.
We note that the function  $f_{1}$  is the building block of much previous research (Trouillon et al., 2016; Kadlec et al., 2017; Kazemi and Poole, 2018). Different from these existing research, we focus on designing more advanced scoring functions with better expressiveness.
3.1.2  $f_{2}$ : Multi-linear Dot Product Among Segments
Next, we introduce fine-grained feature interactions to improve the model expressiveness further. To be specific, we develop the scoring function  $f_{2}$  that conducts the multi-linear dot product among different segments of the entity/relation embeddings. First, we uniformly divide the  $d$ -dimensional embedding of the head  $h$ , the relation  $r$ , and the tail  $t$  into  $k$  segments, and the dimension of each segment is  $d / k$ . For example, we can write the embedding of relation  $\mathbf{r}$  as:
$$
\mathbf {r} = [ \mathbf {r} _ {0}, \mathbf {r} _ {1}, \dots , \mathbf {r} _ {k - 1} ], \quad \mathbf {r} _ {x} \in \mathbb {R} ^ {d / k},
$$
where  $\mathbf{r}_x$  is the  $x$ -th segment of the embedding  $\mathbf{r}$ . Then, we define the scoring function  $f_2$  as follows:
$$
f _ {2} (h, r, t) = \sum_ {0 \leq x, y, w <   k} \left\langle \mathbf {r} _ {x}, \mathbf {h} _ {y}, \mathbf {t} _ {w} \right\rangle . \tag {2}
$$
Compared with the scoring function  $f_{1}$ , where the interactions only happen among the same positions of  $\mathbf{h}$ ,  $\mathbf{r}$ , and  $\mathbf{t}$  embeddings, the scoring function  $f_{2}$  can exploit more feature interactions among different segments of embeddings.
Figure 1: Scoring function  $f_{3}$  with  $k = 2$ .
3.1.3  $f_{3}$ : Modeling both Symmetric and Antisymmetric Relations
Although the scoring function  $f_{2}$  can facilitate fine-grained feature interactions, it can only preserve the symmetry property of relations and can not support the modeling of antisymmetric relations. For example, given a symmetric relation  $r$ , we have  $f_{2}(h,r,t) = f_{2}(t,r,h)$ , but for an antisymmetric relation  $r'$ , the value of  $f_{2}(h,r',t)$  is also equal to  $f_{2}(t,r',h)$ , which is unreasonable because  $(t,r',h)$  is a false triple.
To preserve the antisymmetry property of relations, we divide the segments of relation embedding  $\mathbf{r}$  into odd and even parts. Then we define a variable  $s_{x,y}$  to enable the even parts of segments to capture the symmetry property of relations and the odd parts to capture the antisymmetry property. We define the scoring function after adding  $s_{x,y}$  as:
$$
f _ {3} (h, r, t) = \sum_ {0 \leq x, y, w <   k} s _ {x, y} \cdot \left\langle \mathbf {r} _ {x}, \mathbf {h} _ {y}, \mathbf {t} _ {w} \right\rangle , \tag {3}
$$
$$
s _ {x, y} = \left\{ \begin{array}{l l} - 1, & \text {i f x i s o d d a n d x + y \geq k}, \\ 1, & \text {o t h e r w i s e}. \end{array} \right.
$$
In the scoring function  $f_{3}$ ,  $s_{x,y}$  indicates the sign of each dot product term  $\langle \mathbf{r}_x, \mathbf{h}_y, \mathbf{t}_w \rangle$ . Figure 1 depicts an example of the function  $f_{3}$  with  $k = 2$ . When  $\mathbf{r}_x$  is the even part of  $\mathbf{r}$  (the index  $x$  is even),  $s_{x,y}$  is positive, and the summation  $\sum_{s_{x,y} == 1} s_{x,y} \cdot \langle \mathbf{r}_x, \mathbf{h}_y, \mathbf{t}_w \rangle$  of  $f_{3}(h,r,t)$  equals to the corresponding one  $\sum_{s_{x,y} == 1} s_{x,y} \cdot \langle \mathbf{r}_x, \mathbf{t}_y, \mathbf{h}_w \rangle$  of  $f_{3}(t,r,h)$ . Therefore, the function  $f_{3}$  can model symmetric relations via the even segments of  $\mathbf{r}$ . When  $\mathbf{r}_x$  is the odd part of  $r$  (the index  $x$  is odd),  $s_{x,y}$  can be either negative or positive depending on whether  $x + y \geq k$ . Then, the summation of odd parts of  $f_{3}(h,r,t)$  is different from that of  $f_{3}(t,r,h)$ . Accordingly,  $f_{3}(h,r,t)$  can support antisymmetric relations with the odd segments of  $\mathbf{r}$ .
The scoring function  $f_{3}$  can support both symmetric and antisymmetric relations inherently because of the design of segmented embeddings.
Figure 2: Scoring function  $f_{4}$  with  $k = 4$ .
Moreover, the optimization of relation embeddings is entirely data-driven, and thus we focus on providing the proper mechanism to capture common relation properties.
3.1.4  $f_{4}$ : Reducing Computing Overheads
However, though capturing various relation properties, the function  $f_{3}$  suffers from huge computation overheads. The time complexity of function  $f_{3}$  is  $O(k^{2}d)$  because there are  $k^3$  dot product terms  $\langle \mathbf{r}_x,\mathbf{h}_y,\mathbf{t}_w\rangle$  in total. Therefore, the scoring function  $f_{3}$  needs  $k^3$  times of dot product to compute the score of a triple  $(h,r,t)$ . Recall that the dimension of each segment is  $d / k$ , so each multi-linear dot product requires  $O(d / k)$  times of multiplication. As a conclusion, the time complexity of the function  $f_{3}$  is  $O(k^{2}d)$ , which can be calculated by  $O(k^{3}\times d / k)$ . To reduce the computation overheads of the function  $f_{3}$ , we introduce another variable  $w_{x,y}$  for the index of tail entity  $t$ . Accordingly, we define the scoring function  $f_{4}$  as follows.
$$
f _ {4} (h, r, t) = \sum_ {0 \leq x, y <   k} s _ {x, y} \cdot \left\langle \mathbf {r} _ {x}, \mathbf {h} _ {y}, \mathbf {t} _ {w _ {x}, y} \right\rangle , \tag {4}
$$
$$
w_{x,y} = \left\{ \begin{array}{cc}y, & \text{if $x$ is even},\\ (x + y)\% k, & \text{if $x$ is odd}. \end{array} \right.
$$
The scoring function  $f_{4}$  reduces the number of dot product terms to  $k^2$ , so its time complexity is  $O(kd)$  (calculated by  $O(k^2 \times d / k)$ ). Moreover, the scoring function  $f_{4}$  can also preserve symmetry property in the even parts of  $\mathbf{r}$  and preserve antisymmetry property in the odd parts of  $\mathbf{r}$ .
Figure 2 shows the example of the scoring function  $f_{4}$  with  $k = 4$ . The dot product terms in
Figure 2 can be categorized into four groups according to the segment indexes of  $\mathbf{r}$ . In the groups of  $\mathbf{r}_0$  and  $\mathbf{r}_2$ , which are the even parts of  $\mathbf{r}$ , the segment  $\mathbf{t}_{w_x,y}$ 's index  $w_{x,y}$  is same as the segment  $\mathbf{h}_y$ 's index  $y$ , and  $s_{x,y}$  is always positive. Thus, the summation  $\sum s_{x,y} \cdot \left\langle \mathbf{r}_x, \mathbf{h}_y, \mathbf{t}_{w_x,y} \right\rangle$  of the even parts of  $f_4(h,r,t)$  is equal to the corresponding one  $\sum s_{x,y} \cdot \left\langle \mathbf{r}_x, \mathbf{t}_y, \mathbf{h}_{w_x,y} \right\rangle$  of  $f_4(t,r,h)$ . In the groups of  $\mathbf{r}_1$  and  $\mathbf{r}_3$ , which are the odd parts of  $\mathbf{r}$ , the segment indexes of  $\mathbf{t}$  are  $(x + y) \% k$ , where  $x$  and  $y$  are the indexes of  $\mathbf{r}$  and  $\mathbf{h}$ , respectively. When  $x + y \geq k$ , the variable  $s_{x,y}$  will change from positive to negative. So the summation of the odd parts of  $f_4(h,r,t)$  and  $f_4(t,r,h)$  will not be the same. Besides, it is apparent that the number of feature interactions on  $h$ ,  $r$  and  $t$  are increasing  $k$  times since each segment has  $k$  interactions with other segments.
In summary, the scoring function  $f_{4}$  of our SEEK framework has the following characteristics:
- Tunable Computation. The scoring function exactly involves each segment of  $\mathbf{r}$ ,  $\mathbf{h}$ , and  $\mathbf{t}$ $k$  times. Thus the number of feature interactions and the computation cost are fully tunable with a single hyperparameter  $k$ .
- Symmetry and Antisymmetry Preservation. The even parts of  $\mathbf{r}$  can preserve the symmetry property of relations, and the odd parts of  $\mathbf{r}$  can preserve the antisymmetry property.
- Dimension Isolation. The dimensions within the same segment are isolated from each other, which will prevent the embeddings from excessive correlations.
3.2 Discussions
Complexity analysis As described before, the number of dot product terms in scoring function  $f_{4}$  is  $k^{2}$ , and each term requires  $O(d / k)$  times of multiplication. So the time complexity of our SEEK framework is  $O(kd)$  (calculated by  $O(k^2 \times d / k)$ ), where  $k$  is a small constant such as 4 or 8. For the space complexity, the dimension of entity and relation embeddings is  $d$ , and there are no other parameters in our SEEK framework. Thus, the space complexity of SEEK is  $O(d)$ . The low time and space complexity of our framework demonstrate that our SEEK framework has high scalability, which is vital for large-scale real-world knowledge graphs.
Connection with existing methods Our SEEK framework is a generalized framework of some existing methods, such as DistMult (Yang et al., 2015), ComplEx (Trouillon et al., 2016), and HolE (Nickel et al., 2016). In the following, we will prove that these methods are special cases of our framework when we set  $k = 1$  and  $k = 2$ , respectively.
Proposition 1. SEEK  $(k = 1)$  is equivalent to DistMult.
Proof. The proof is trivial. Given  $k = 1$ , we have  $x = 0$  and  $y = 0$  in scoring function  $f_{4}$  and  $\mathbf{r}_0 = \mathbf{r}$ ,  $\mathbf{h}_0 = \mathbf{h}$ , and  $\mathbf{t}_0 = \mathbf{t}$ . Thus the function  $f_{4}$  can be written as  $f_{4}^{k=1}(h, r, t) = \langle \mathbf{r}, \mathbf{h}, \mathbf{t} \rangle$ , which is the same scoring function of DistMult.
Proposition 2. SEEK  $(k = 2)$  is equivalent to the ComplEx and HolE.
Proof. Given  $k = 2$ , function  $f_{4}$  can be written as:
$$
f _ {4} ^ {k = 2} (h, r, t) = \sum_ {x = 0, 1} \sum_ {y = 0, 1} s _ {x, y} \cdot \left\langle \mathbf {r} _ {x}, \mathbf {h} _ {y}, \mathbf {t} _ {w _ {x, y}} \right\rangle ,
$$
then we expand the right part of the equation:
$$
\langle \mathbf {r} _ {0}, \mathbf {h} _ {0}, \mathbf {t} _ {0} \rangle + \langle \mathbf {r} _ {0}, \mathbf {h} _ {1}, \mathbf {t} _ {1} \rangle + \langle \mathbf {r} _ {1}, \mathbf {h} _ {0}, \mathbf {t} _ {1} \rangle - \langle \mathbf {r} _ {1}, \mathbf {h} _ {1}, \mathbf {t} _ {0} \rangle .
$$
If we consider  $\mathbf{r}_0, \mathbf{h}_0, \mathbf{t}_0$  as the real part of  $\mathbf{r}, \mathbf{h}, \mathbf{t}$ , and  $\mathbf{r}_1, \mathbf{h}_1, \mathbf{t}_1$  as the imaginary part, then  $f_4^{k=2}(h, r, t)$  is exactly the scoring function of ComplEx framework. Since (Hayashi and Shimbo, 2017) has already discussed the equivalence of ComplEx and HolE, the SEEK ( $k = 2$ ) is also equivalent to the HolE framework.
SEEK takes the negative log-likelihood loss function with  $L_{2}$  regularization as its objective function to optimize the parameters of entities and relations:
$$
\min  _ {\Theta} \sum_ {(h, r, t) \in \Omega} - \log \left(\sigma \left(Y _ {h r t} f _ {4} (h, r, t)\right)\right) + \frac {\lambda}{2 d} \| \Theta \| _ {2} ^ {2}, \tag {5}
$$
where  $\sigma$  is a sigmoid function defined as  $\sigma (x) = \frac{1}{1 + e^{-x}}$ , and  $\Theta$  represents the parameters in the embeddings of entities and relations in knowledge graphs;  $\Omega$  is the triple set containing the true triples in the knowledge graphs and the false triples generated by negative sampling. In the negative sampling, we generate a false triple  $(h^{\prime},r,t)$  or  $(h,r,t^{\prime})$  by replacing the head or tail entity of a true triple
with a random entity.  $Y_{hrt}$  is the label of  $(h, r, t)$ , which is 1 for the true triples and -1 for the false triples.  $\lambda$  is the  $L_2$  regularization parameter.
The gradients of Equation 5 are then given by:
$$
\frac {\partial \mathcal {L}}{\partial \theta} = \frac {\partial \mathcal {L}}{\partial f _ {4}} \frac {\partial f _ {4}}{\partial \theta} + \frac {\lambda \theta}{d}, \tag {6}
$$
where  $\mathcal{L}$  represents the objective function of SEEK, and  $\theta$  is the parameters in the segments. Specifically, the partial derivatives of function  $f_{4}$  for the  $x$ -th segment of  $\mathbf{r}$  and the  $y$ -th segment of  $\mathbf{h}$  are:
$$
\frac {\partial f _ {4}}{\partial \mathbf {r} _ {x}} = \sum_ {0 \leq y <   k} s _ {x, y} \cdot \left(\mathbf {h} _ {y} \odot \mathbf {t} _ {w _ {x}, y}\right),
$$
$$
\frac {\partial f _ {4}}{\partial \mathbf {h} _ {y}} = \sum_ {0 \leq x <   k} s _ {x, y} \cdot \left(\mathbf {r} _ {x} \odot \mathbf {t} _ {w _ {x}, y}\right),
$$
where  $\odot$  is the entry-wise product of two vectors, e.g.  $\mathbf{c} = \mathbf{a}\odot \mathbf{b}$  results in the  $i$ -th dimension of  $\mathbf{c}$  is  $\mathbf{a}_i\cdot \mathbf{b}_i$ . The derivative of scoring function  $f_{4}$  for  $\mathbf{t}_{w}$  is different from those of the above two:
$$
\frac {\partial f _ {4}}{\partial \mathbf {t} _ {w}} = \sum_ {0 \leq x, y <   k} \mathbb {1} _ {[ w = w _ {x, y} ]} \cdot s _ {x, y} \cdot (\mathbf {r} _ {x} \odot \mathbf {h} _ {y}),
$$
where  $\mathbb{1}_{[w = w_{x,y}]}$  has value 1 if  $w = w_{x,y}$  holds, otherwise it is 0.