SlowGuess/ACL_2020_OCR huggingface dataset:
Chengyu Wang $^{1,2}$ , Xiaofeng He $^{3*}$
<sup>1</sup> School of Software Engineering, East China Normal University <sup>2</sup> Alibaba Group
$^{3}$  School of Computer Science and Technology, East China Normal University
chywang2013@gmail.com, hexf@cs.ecnu.edu.cn
The hypernymy detection task has been addressed under various frameworks. Previously, the design of unsupervised hypernymy scores has been extensively studied. In contrast, supervised classifiers, especially distributional models, leverage the global contexts of terms to make predictions, but are more likely to suffer from "lexical memorization". In this work, we revisit supervised distributional models for hypernymy detection. Rather than taking embeddings of two terms as classification inputs, we introduce a representation learning framework named Bidirectional Residual Relation Embeddings (BiRRE). In this model, a term pair is represented by a BiRRE vector as features for hypernymy classification, which models the possibility of a term being mapped to another in the embedding space by hypernymy relations. A Latent Projection Model with Negative Regularization (LPMNR) is proposed to simulate how hypernyms and hyponyms are generated by neural language models, and to generate BiRRE vectors based on bidirectional residuals of projections. Experiments verify BiRRE outperforms strong baselines over various evaluation frameworks.
As a type of linguistic resources, hypernymy relations refer to "is-a" relations between terms. Such relations are frequently exploited in a wide range of NLP tasks, including taxonomy induction (Mao et al., 2018), lexical entailment (Vulic et al., 2017) and Web query understanding (Wang et al., 2015).
In the NLP community, the task of hypernymy detection has been studied under various frameworks, e.g., unsupervised hypernym discovery (Roller et al., 2018; Chen et al., 2018; Chang et al., 2018), supervised hypernymy classification (Shwartz et al., 2016; Nguyen et al.,
2017), graded lexical entailment (Vulic et al., 2017). To address unsupervised hypernym discovery, pattern-based and distributional approaches are two mainstream types of methods. Pattern-based approaches use Hearst patterns (Hearst, 1992) and their variants to extract hypernymy relations from texts (Kozareva and Hovy, 2010; Roller and Erk, 2016). Distributional methods employ hypernymy measures (or called scores) to predict hypernymy based on distributional vectors (Santus et al., 2014, 2017), alleviating the pattern sparsity issue. Le et al. (2019) combine Hearst patterns and hyperbolic embeddings for unsupervised hypernym detection.
Compared to unsupervised tasks, the supervised hypernymy detection task is formulated more directly, classifying a term pair as hypernymy or non-hypernymy based on two terms' representations (Yu et al., 2015; Anke et al., 2016; Nguyen et al., 2017). Although this task definition is more straightforward, the corresponding methods receive criticism because they may suffer from "lexical memorization" (Levy et al., 2015), referring to the phenomenon that they only learn whether a term is a "prototypical hypernym", rather than the actual relations between two terms. To address the problem, several methods combine other signals as inputs for hypernymy classifiers, such as dependency paths (Shwartz et al., 2016) and the WordNet concept hierarchy (Nguyen et al., 2017). Nonetheless, it is worth studying whether supervised classifiers can learn hypernymy relations purely based on distributional representations.
In this paper, we revisit supervised distributional models for hypernymy detection, and propose a representation learning framework named Bidirectional Residual Relation Embeddings (BiRRE). To handle "lexical memorization" (Levy et al., 2015), we learn a BiRRE vector for each term pair as features for the classifier, avoiding using the two terms' embeddings directly. The BiRRE vector
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3630-3640 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
models the possibility of a term being mapped to another in the embedding space by hypernymy relations, learned via existing neural language models and supervised signals of the training set. Specifically, we introduce the Latent Projection Model with Negative Regularization (LPMNR) to simulate how hypernyms and hyponyms are generated in the the embedding space. The BiRRE vectors are generated based on bidirectional residuals of projection results of LPMNR. Experiments over multiple public datasets and various evaluation frameworks prove that BiRRE outperforms strong baselines.
The rest of this paper is organized as follows. Section 2 summarizes the related work. The BiRRE framework is elaborated in Section 3, with experiments shown in Section 4. Finally, we conclude our paper and discuss the future work in Section 5.
In this section, we overview related work on various tasks related to hypernymy detection. Due to space limitation, we focus on recent advances and refer readers to Wang et al. (2017a) for earlier work.
Pattern-based approaches date back to Hearst (1992), utilizing handcrafted patterns in English for text matching. An example of Hearst patterns is “[...] such as [...]”. They are employed to build large-scale taxonomies (Wu et al., 2012; Faralli et al., 2019). Although Hearst patterns are fairly simple, recent studies show they are highly useful for designing hypernymy measures (Roller et al., 2018; Le et al., 2019). Other approaches aim at improving the coverage of generalized Hearst patterns by automatic pattern expansion (Kozareva and Hovy, 2010; Roller and Erk, 2016), or considering other context-rich representations (such as Heterogeneous Information Networks (Shi et al., 2019)). A potential drawback of pattern-based methods is that the recall of extraction results over specific domains is limited (Alfarone and Davis, 2015), as textual patterns are naturally sparse in the corpus.
To overcome the sparsity issue, distributional hypernymy measures model the degree of hypernymy within a term pair. A majority of these hypernymy measures are based on Distributional Inclusion Hypothesis (DIH) (Weeds et al., 2004), meaning that a hypernym covers a broader spectrum of contexts, compared to its hyponyms. The improvements and variants of DIH include (Santus
et al., 2014; Chen et al., 2018; Chang et al., 2018) and many others. A comprehensive overview of distributional hypernymy measures can be found in Santus et al. (2017). Recently, Le et al. (2019) combine Hearst patterns and distributional vectors for hypernym detection. Additionally, the work of graded lexical entailment (Vulic et al., 2017) and cross-lingual graded lexical entailment (Vulic et al., 2019) aims at computing numerical scores, indicating the degree of hypernymy of a term pair.
For supervised hypernymy classification, traditional approaches employ distributional vectors of two terms as features, such as the Concat model, the Diff model, the SimDiff model (Turney and Mohammad, 2015). Recently, several approaches are proposed to learn hypernymy embeddings, considering the semantic hierarchies of concepts (Yu et al., 2015; Luu et al., 2016; Nguyen et al., 2017; Chang et al., 2018; Nickel and Kiela, 2018; Ganea et al., 2018; Rei et al., 2018; Chen et al., 2018). For example, Yu et al. (2015) learn hypernym and hyponym embeddings for a term by max-margin neural network. Nguyen et al. (2017) propose hierarchical embeddings for hypernymy classification, jointly trained over texts and the WordNet concept hierarchy. Rei et al. (2018) propose a directional similarity neural network based on word embeddings to predict the degree of hypernymy between two terms. Yet a number of models encode terms in the hyperbolic space, such as the hyperbolic Lorentz Model (Nickel and Kiela, 2018), Hyperbolic Entailment Cones (Ganea et al., 2018), and others (Le et al., 2019; Aly et al., 2019). The hyperbolic geometry is more capable of modeling the transitivity property of hypernymy. Additionally, patterns and distributional vectors can also be combined for supervised hypernymy prediction, as in Shwartz et al. (2016); Held and Habash (2019) and several systems submitted to SemEval 2018 Task 9 (Camacho-Collados et al., 2018).
Another type of supervised models can be categorized as projection-based approaches, which model how to map embeddings of a term to those of its hypernyms. Fu et al. (2014) is most influential, followed by a number of variants. Biemann et al. (2017); Wang et al. (2017b, 2019b) improve projection learning by considering explicit negative samples. The usage of orthogonal matrices is exploited in Wang et al. (2019a). One advantage is that they do not perform classification on two terms' embeddings directly, alleviating "lexical memoriza
Figure 1: The BiRRE framework for supervised hypernymy detection.
tion” (Levy et al., 2015). Compared to previous work, BiRRE is supervised, but does not minimize the classification error firstly. It uses LPMNR to learn hypernym/hyponym generation process by projection learning. Hence, it takes advantages of both traditional classification and projection-based approaches.
3 The BiRRE Framework
In this section, we first introduce the task description and the BiRRE framework. The detailed steps and justifications are elaborated subsequently.
3.1 Task Description
Given two sets of term pairs: the training sets of hypernymy  $D^{(+)} = \{(x_i, y_i)\}$  and non-hypernymy relations  $D^{(-)} = \{(x_i, y_i)\}$ , the task is to learn a classifier  $f$  to distinguish hypernymy vs. non-hypernymy relations. Particularly,  $y_i$  is a hypernym of  $x_i$  if  $(x_i, y_i) \in D^{(+)}$ . For non-hypernym relations, the relation types between two terms  $x_i$  and  $y_i$  in  $D^{(-)}$  can be reverse-hypernymy, synonymy, antonymy, or unrelated, depending on the respective task and dataset settings.
3.2 General Framework
The BiRRE framework is shown in Figure 1, consisting of pre-processing and three major modules.
Pre-processing: The pre-processing step of the BiRRE framework requires minimal computation. For each term pair  $(x_{i},y_{i})\in D^{(+)}\cup D^{(-)}$  we retrieve the corresponding embedding vectors
from any neural language models (e.g., Word2Vec, GloVe), without fine-tuning. Denote normalized embeddings of  $x_{i}$  and  $y_{i}$  as  $\mathbf{x}_i$  and  $\mathbf{y}_i$ , respectively.
M1: The hyponym projection module learns how to map embeddings of a hybernym to those of its hyonyms. Consider the example in Figure 2. There are usually one-to-many mappings (in semantics) from hypernyms to hyonyms. Hence, we map a hybernym to its  $N$  semantically diverse hyonyms by LPMNR. We denote the  $N$  hyponym embeddings w.r.t.  $y_{i}$  as  $\text{hypo}^{(1)}(\mathbf{y}_i),\dots,$ $\text{hypo}^{(N)}(\mathbf{y}_i)^1$ . Based on the difference between the true hyponym embeddings  $\mathbf{x}_i$  and the  $N$  predicted hyponym embeddings, we compute the hyponym residual vector  $\text{res}^{\text{hypo}}(\mathbf{x}_i,\mathbf{y}_i)$  to measure the "goodness" of mapping from  $\mathbf{y}_i$  to  $\mathbf{x}_i$ . As shown in Biemann et al. (2017), the explicit usage of negative samples (i.e., non-hypernymy relations) improves the performance of projection learning. In this module, we take  $D^{(+)}$  as the training set and  $D^{(-)}$  for regularization purposes.
M2: The hypernym projection module learns how to map embeddings of a hyponym to those of its hypernyms. Based on Figure 2, such mappings tend to be simpler. Hence, we only learn one mapping model from a hyponym to embeddings of its hypernym. We denote the hypernym embeddings
<sup>1</sup>Because the training process is completed in the embedding space, our model learns to associate low-density hypernym regions with multiple numbers of high-density hyponym regions. Here,  $\mathbf{M}^{(1)}\mathbf{y}_i,\dots ,\mathbf{M}^{(N)}\mathbf{y}_i$  may refer to the distributions of word embeddings of hyponyms, with no guarantee that they refers to actual word embeddings.
(a) Hypernym-to-hyponym mappings
(b) Hypnoym-to-hypernym mappings
Figure 2: Two types of hypernymy relation mappings.
as hyper  $(\mathbf{x}_i)$ . This step is learned by a simplified version of LPMNR. Similarly, we denote the hypernym residual vector as  $reshyper(\mathbf{x}_i,\mathbf{y}_i)$ , measuring the "goodness" of mapping from  $\mathbf{x}_i$  to  $\mathbf{y}_i$ . In this module, we also take  $D^{(+)}$  as the training set and  $D^{(-)}$  for regularization.
M3: Finally, the BiRRE vector (denoted as  $\mathbf{r}_i$ ) w.r.t.  $(x_i, y_i)$  is computed by concatenating  $res^{hypo}(\mathbf{x}_i, \mathbf{y}_i)$  and  $res^{hyper}(\mathbf{x}_i, \mathbf{y}_i)$ . A feedforward neural network is trained over  $D^{(+)}$  and  $D^{(-)}$  for hypernymy relation classification. The parameters of M3 are learned by back propagation, with parameters of M1 and M2 fixed in this step.
3.3 Hyponym Projection (M1)
Previously, several approaches (Fu et al., 2014; Yamane et al., 2016) assume there is a  $d \times d$  projection matrix  $\mathbf{M}$  such that  $\mathbf{M} \mathbf{x}_i \approx \mathbf{y}_i$  where  $d$  is the dimension of word embeddings for  $(x_i, y_i) \in D^{(+)}$ . According to Wang et al. (2019a), the usage of orthogonal matrices has better performance for hypernymy prediction, as the cosine similarity of  $\mathbf{M} \mathbf{x}_i$  and  $\mathbf{y}_i$  can be maximized when  $\mathbf{M} \mathbf{x}_i$  and  $\mathbf{y}_i$  are normalized.
Let  $\mathcal{M} = \{\mathbf{M}^{(1)},\dots ,\mathbf{M}^{(N)}\}$  be the parameter collection of our hyponym projection model (i.e.,  $Nd\times d$  orthogonal projection matrices). For each hypernym  $y_{i}$ , these  $N$  projection matrices map  $\mathbf{y}_i$  to the embeddings of  $N$  semantically diverse hyponyms  $\mathbf{M}^{(1)}\mathbf{y}_i,\dots ,\mathbf{M}^{(N)}\mathbf{y}_i$ . The major challenge is that the explicit semantics of  $N$  projections are unknown, and may vary across different datasets. To derive a unified solution for all sce
narios, we introduce a latent variable  $\theta_{i}^{(p)}\in (0,1)$  to represent the weight of  $(x_{i},y_{i})\in D^{(+)}$  w.r.t. the projection matrix  $\mathbf{M}^{(p)}(p\in \{1,\dots ,N\}$ $\sum_{(x_i,y_i)\in D^{(+)}}\theta_i^{(p)} = 1)$  . The objective of hy-. ponym projection is as follows:2
$$
\min  _ {\mathcal {M}} \sum_ {\left(x _ {i}, y _ {i}\right) \in D (+)} \sum_ {p = 1} ^ {N} \theta_ {i, j} ^ {(p)} \left\| \mathbf {M} ^ {(p)} \mathbf {y} _ {i} - \mathbf {x} _ {i} \right\| ^ {2} \tag {1}
$$
$$
\mathrm {s . t .} \mathbf {M} ^ {(p) ^ {T}} \mathbf {M} ^ {(p)} = \mathbf {I} _ {d}, p \in \{1, \dots , N \}
$$
where  $\mathbf{I}_d$  is the  $d\times d$  identity matrix.
A potential drawback of Eq. (1) is that it only considers hypernymy relations  $D^{(+)}$ . The relation classification objective is not optimized. As Biemann et al. (2017) suggest, negative samples can of help for learning projection regularizers. The regularizers push the projected hyponym embeddings of a term further away from its non-hyponyms, making hypernymy and non-hypernymy relations more separable. Hence, we reformulate Eq. (1) as:
$$
\begin{array}{l} \min _ {\mathcal {M}} \frac {1}{| D ^ {(+)} |} \sum_ {(x _ {i}, y _ {i}) \in D ^ {(+)}} \sum_ {p = 1} ^ {N} \theta_ {i} ^ {(p)} \| \mathbf {M} ^ {(p)} \mathbf {y} _ {i} - \mathbf {x} _ {i} \| ^ {2} \\ + \frac {\lambda}{| D ^ {(-)} |} \sum_ {(x _ {i}, y _ {i}) \in D ^ {(-)}} \sum_ {p = 1} ^ {N} \phi_ {i} ^ {(p)} (\mathbf {M} ^ {(p)} \mathbf {y} _ {i}) ^ {T} \cdot \mathbf {x} _ {i} \\ \begin{array}{l} \text {s . t .} \quad \mathbf {M} ^ {(p) ^ {T}} \mathbf {M} ^ {(p)} = \mathbf {I} _ {d}, p \in \{1, \dots , N \} \end{array} \tag {2} \\ \end{array}
$$
where  $\lambda >0$  is the regularization balancing factor. The latent variable  $\phi_i^{(p)}\in (0,1)$  is the weight of the negative sample  $(x_{i},y_{i})\in D^{(-)}$  w.r.t.  $\mathbf{M}^{(p)}$  The constraint  $\sum_{(x_i,y_i)\in D^{(-)}}\phi_i^{(p)} = 1$  also holds.
To the best of our knowledge, there is no standard off-the-shelf solution to Eq. (2). We slightly change the regularization term of Eq. (2). The objective function is changed as follows, which we refer as the Latent Projection Model with Negative Regularization (LPMNR):
$$
\begin{array}{l} \min _ {\mathcal {M}} \frac {1}{| D ^ {(+)} |} \sum_ {(x _ {i}, y _ {i}) \in D ^ {(+)}} \sum_ {p = 1} ^ {N} \theta_ {i} ^ {(p)} \| \mathbf {M} ^ {(p)} \mathbf {y} _ {i} - \mathbf {x} _ {i} \| ^ {2} \\ - \frac {\lambda}{| D ^ {(-)} |} \sum_ {(x _ {i}, y _ {i}) \in D ^ {(-)}} \sum_ {p = 1} ^ {N} \phi_ {i} ^ {(p)} \| \mathbf {M} ^ {(p)} \mathbf {y} _ {i} - \mathbf {x} _ {i} \| ^ {2} \\ \begin{array}{l} \text {s . t .} \quad \mathbf {M} ^ {(p) ^ {T}} \mathbf {M} ^ {(p)} = \mathbf {I} _ {d}, p \in \{1, \dots , N \} \end{array} \tag {3} \\ \end{array}
$$
For simplicity, we omit the constraints of latent variables in the objective functions in this paper.
3633
Optimizing Eq. (3) is non-trivial due to the existence of the unknown weights  $\theta_{i}^{(p)}$  and  $\phi_i^{(p)}$ . In this paper, we present a dual-iterative algorithm to solve the problem. All values of  $\theta_{i}^{(p)}$  and  $\phi_i^{(p)}$  are randomly initialized (with  $\theta_{i}^{(p)},\phi_{i}^{(p)}\in (0,1)$ ,  $\sum_{(x_i,y_i)\in D^{(+)}}\theta_i^{(p)} = 1$  and  $\sum_{(x_i,y_i)\in D^{(-)}}\phi_i^{(p)} = 1$ ). In each iteration, we update the values of  $\theta_{i}^{(p)}$ ,  $\phi_{i}^{(p)}$  and  $\mathbf{M}^{(p)}$ . When all the values of  $\theta_{i}^{(p)}$  and  $\phi_{i}^{(p)}$  are fixed, Eq. (3) can be regarded as a variant of the Multi-Wahba problem (Wang et al., 2019a). For simplicity, let  $\alpha = \frac{\lambda|D^{(+)}|}{|D^{(-)}|}$ . We extend their work and give an SVD based closed-form solution to Eq. (3) in Algorithm 1.
Algorithm 1 Closed-form Solution to Eq. (3)
$$
- \alpha \cdot \sum_ {\left(x _ {i}, y _ {i}\right) \in D ^ {(-)}} \phi_ {i} ^ {(p)} \mathbf {x} _ {i} \mathbf {y} _ {i} ^ {T};
$$
1: for  $p = 1$  to  $N$  do
2:  $\mathbf{B}^{(p)} = \sum_{(x_i,y_i)\in D^{(+)}}\theta_i^{(p)}\mathbf{x}_i\mathbf{y}_i^T$
3:  $\mathbf{U}^{(p)}\mathbf{\Sigma}^{(p)}\mathbf{V}^{(p)T} = SVD(\mathbf{B}^{(p)})$
4:  $\mathbf{R}^{(p)} = \mathrm{diag}(\underbrace{1,\ldots,1}_{d - 1},\det (\mathbf{U}^{(p)})\det (\mathbf{V}^{(p)}));$
5:  $\mathbf{M}^{(p)} = \mathbf{U}^{(p)}\mathbf{R}^{(p)}\mathbf{V}^{(p)T}$ ;
6: end for
Proof: It is trivial to see that the optimal values of each matrix is independent from each other. Hence, we only need to optimize:
$$
\begin{array}{l} \min _ {\mathcal {M}} \frac {1}{| D ^ {(+)} |} \sum_ {(x _ {i}, y _ {i}) \in D ^ {(+)}} \theta_ {i} ^ {(p)} \| \mathbf {M} ^ {(p)} \mathbf {y} _ {i} - \mathbf {x} _ {i} \| ^ {2} \\ - \frac {\lambda}{| D ^ {(-)} |} \sum_ {(x _ {i}, y _ {i}) \in D ^ {(-)}} \phi_ {i} ^ {(p)} \| \mathbf {M} ^ {(p)} \mathbf {y} _ {i} - \mathbf {x} _ {i} \| ^ {2} \\ \end{array}
$$
s.t.  $\mathbf{M}^{(p)^T}\mathbf{M}^{(p)} = \mathbf{I}_d$
For simplicity, let  $\alpha = \frac{\lambda|D^{(+)}|}{|D^{(-)}|}$ , with the superscript  $(p)$  omitted. The problem can be transformed as:
$$
\begin{array}{l} J (\mathbf {M}) = \sum_ {\left(x _ {i}, y _ {i}\right) \in D ^ {(+)}} \theta_ {i} \| \mathbf {M} \mathbf {y} _ {i} - \mathbf {x} _ {i} \| ^ {2} \\ - \alpha \cdot \sum_ {\left(x _ {i}, y _ {i}\right) \in D ^ {(-)}} \phi_ {i} \| \mathbf {M} \mathbf {y} _ {i} - \mathbf {x} _ {i} \| ^ {2} \\ \end{array}
$$
s.t.  $\mathbf{M}^T\mathbf{M} = \mathbf{I}_d$
Define the matrix  $\mathbf{B} = \sum_{(x_i,y_i)\in D^{(+)}}\theta_i\mathbf{x}_i\mathbf{y}_i^T -$ $\alpha \cdot \sum_{(x_i,y_i)\in D(-)}\phi_i\mathbf{x}_i\mathbf{y}_i^T.$  We re-write the objective function as:  $J(\mathbf{M}) = 1 - \mathrm{tr}(\mathbf{MB}^T)$  . Hence,
we have transformed the problem into the MultiWahba problem (Wang et al., 2019a).  $J(\mathbf{M})$  is minimized when the optimal value of  $\mathbf{M}$  is:
$$
\mathbf{M}^{*} = \mathbf{U}\mathrm{diag}(\underbrace{1,\ldots,1}_{d - 1},\det (\mathbf{U})\det (\mathbf{V}))\mathbf{V}^{T}
$$
with  $\mathbf{U}\pmb{\Sigma}\mathbf{V}^T = SVD(\mathbf{B})$
After optimal values of  $\mathbf{M}^{(p)}$  are computed, the values of all  $\| \mathbf{M}^{(p)}\mathbf{y}_i - \mathbf{x}_i\|^2$  are known. In this condition, we fix the values of  $\mathbf{M}^{(p)}$  and update all  $\theta_i^{(p)}$  and  $\phi_i^{(p)}$ . We turn the problem of minimizing Eq. (3) into the following problems:
$$
\min  _ {\theta_ {i} ^ {(p)}} \sum_ {\left(x _ {i}, y _ {i}\right) \in D (+)} \left\| \mathbf {M} ^ {(p)} \mathbf {y} _ {i} - \mathbf {x} _ {i} \right\| ^ {2} \cdot \theta_ {i} ^ {(p)} \tag {4}
$$
$$
\max  _ {\phi_ {i} ^ {(p)}} \sum_ {\left(x _ {i}, y _ {i}\right) \in D ^ {(-)}} \left\| \mathbf {M} ^ {(p)} \mathbf {y} _ {i} - \mathbf {x} _ {i} \right\| ^ {2} \cdot \phi_ {i} ^ {(p)} \tag {5}
$$
We update  $\theta_{i}^{(p)}$  and  $\phi_i^{(p)}$  by constrained gradient descent where the updating formulas are:
$$
\theta_ {i} ^ {(p) *} = \theta_ {i} ^ {(p)} - \eta \cdot \sum_ {\left(x _ {i}, y _ {i}\right) \in D ^ {(+)}} \left\| \mathbf {M} ^ {(p)} \mathbf {y} _ {i} - \mathbf {x} _ {i} \right\| ^ {2} \tag {6}
$$
$$
\phi_ {i} ^ {(p) *} = \phi_ {i} ^ {(p)} + \eta \cdot \sum_ {\left(x _ {i}, y _ {i}\right) \in D ^ {(-)}} \left\| \mathbf {M} ^ {(p)} \mathbf {y} _ {i} - \mathbf {x} _ {i} \right\| ^ {2} \tag {7}
$$
where  $\eta > 0$  is the learning rate (a small decimal).  $\theta_{i}^{(p)*}$  and  $\phi_{i}^{(p)*}$  are updated values of  $\theta_{i}^{(p)}$  and  $\phi_{i}^{(p)}$  for the new iteration, respectively. After the update of all weights, we normalize the weights to satisfy:  $\sum_{(x_i,y_i)\in D(+)}\theta_i^{(p)} = 1$  and  $\sum_{(x_i,y_i)\in D(-)}\phi_i^{(p)} = 1$ . The iterative procedure continues until convergence, with the algorithm summarized in Algorithm 2.
After training, given  $\mathbf{x}_j$ , M1 outputs  $N$  hyponym embeddings:  $hypo^{(1)}(\mathbf{y}_i) = \mathbf{M}^{(1)}\mathbf{y}_i,\dots,hypo^{(N)}(\mathbf{y}_i) = \mathbf{M}^{(N)}\mathbf{y}_i$ . We define the hyponym residual vector  $res^{hypo}(\mathbf{x}_i,\mathbf{y}_i)$  as follows:
$$
r e s ^ {h y p o} (\mathbf {x} _ {i}, \mathbf {y} _ {i})) = \mathbf {x} _ {i} - \mathbf {M} ^ {(\tilde {p})} \mathbf {y} _ {i}
$$
where  $\tilde{p}$  is the index of the selected projection matrix that best fits for  $(x_i,y_i)\in D^{(+)}$ . We set  $\tilde{p}$  empirically as:  $\tilde{p} = \mathrm{argmin}_p\| \mathbf{x}_i - \mathbf{M}^{(p)}\mathbf{y}_i\| _2$
Based on the objective in Eq. (3), if  $(x_{i},y_{i})\in$ $D^{(+)}$  ,  $\| res^{hypo}(\mathbf{x}_i,\mathbf{y}_i)\| _2$  tends to be small. Otherwise,  $\| res^{hypo}(\mathbf{x}_i,\mathbf{y}_i)\| _2$  would be large. Hence, it is discriminative for hypernymy classification.
3.4 Hypernym Projection (M2)
The hypernym projection module can be regarded as a simplified version of the previous module. Denote  $\mathbf{Q}$  as the  $d\times d$  projection matrix. The objective of hypernym projection is formulated as follows:
$$
\min  _ {\mathbf {Q}} \frac {1}{| D ^ {(-)} |} \sum_ {(x _ {i}, y _ {i}) \in D ^ {(-)}} \| \mathbf {Q x} _ {i} - \mathbf {y} _ {i} \| ^ {2} - \frac {\lambda}{| D ^ {(+)} |}
$$
$$
\sum_ {(x _ {i}, y _ {i}) \in D ^ {(+)}} \| \mathbf {Q} \mathbf {x} _ {i} - \mathbf {y} _ {i} \| ^ {2} \text {s . t .} \mathbf {Q} ^ {T} \mathbf {Q} = \mathbf {I} _ {d}
$$
It can be solved by Algorithm 1 with weights reduced and  $N = 1$ . Similar to hyponym projection, we compute the hypernym residual vector  $res^{hyper}(\mathbf{x}_i,\mathbf{y}_i)$  as follows:
$$
r e s ^ {h y p e r} (\mathbf {x} _ {i}) = \mathbf {Q x} _ {i} - \mathbf {y} _ {i}
$$
3.5 Hypernymy Relation Classification (M3)
For each pair  $(x_{i},y_{i})\in D^{(+)}\cup D^{(-)}$  , we generate the BiRRE vector  $\mathbf{r}_i$  via the concatenation of two residual vectors:
$$
\mathbf {r} _ {i} = r e s ^ {h y p o} \left(\mathbf {x} _ {i}, \mathbf {y} _ {i}\right) \oplus r e s ^ {h y p e r} \left(\mathbf {x} _ {i}, \mathbf {y} _ {i}\right) \tag {8}
$$
A feed forward neural network is trained for hypernymy vs. non-hypernymy classification over