SlowGuess/NAACL_2022_OCR huggingface dataset:
Datasets. Table 1 shows the basic statistics of
Nodes	Edges	Max-in degree	Max-out degree	leaf nodes
Assamese WordNet	8466	8363	1	525	7072
Bengali WordNet	26007	25815	1	924	22847
Hindi Wordnet	28242	28016	1	951	24737
Table 1: Dataset Statistics
three WordNet taxonomies used in this study. The
taxonomy networks are extracted from Assamese, Bengali, and Hindi WordNets, respectively.
Metrics. We use Mean Rank (MR), Hit@k, and Mean Reciprocal Rank (MRR) to evaluate the ranks of the retrieved results obtained from different models, for the test queries. Like Taxo-Expan (Shen et al., 2020) evaluation strategy, we scale the MRR score by a factor of 10 to highlight the discrepancy of the performances among different methods. Further, we use Accuracy, Micro/ Macro F1, Precision, Recall, and F-Scores to evaluate a method's prediction capability to decide which operation among merge (M), attach (A), and no-operation (N) needs to be performed.
Baselines. We choose two most recent benchmark SOTA taxonomy-expansion frameworks TaxoExpan (Shen et al., 2020) and Triplet Matching Network(TMN) (Zhang et al., 2021) as the competing methods. As Taxo-Expan and TMN outperform SemEval-2016 (Shen et al., 2020; Zhang et al., 2021), we have not included SemEval-2016 as baseline in this study. In terms of learning objective, Taxo-Expan is similar to ours. It uses ego-tree-based anchor features for matching query features in a regression-based setting. TMN captures fine-grained relationship dynamics of query and anchor concepts using channel-wise gating mechanism-based attention learning.
All datasets and our model implementations are available at: https://github.com/barnal/TEAM
Here we report the classification and ranking results of the competing methods. We also compare and contrast among the variants of our TEAM framework. Apart from the two versions of the TEAM, namely, TEAM-RG and TEAM-CL, we have task-specific model variants specified as — attach-A, merge-M and merge+attach-MA. Here, (attach+merge) means simultaneously optimizing for both the tasks.
5.1 Ranking Results
In Table [2], we show the performance of the competing methods in terms of (best) ranking scores. We see similar trends for all taxonomies in the sub-tables. When considering only attach operation and the test ranking scores, we see TEAMRG clearly beats Taxo-Expan by a large margin of (196.87, 487.75, 470.87) in MR, by a margin of
Assamese WordNet-Noun	Bengali WordNet-Noun	Hindi WordNet-Noun
Methods	Micro_MR	Hit@1	Hit@3	MRR	Micro_MR	Hit@1	Hit@3	MRR	Micro_MR	Hit@1	Hit@3	MRR
TEAM-RG(A)	144.92	0.27	0.42	0.67	191.51	0.17	0.36	0.86	177.85	0.28	0.43	0.67
TEAM-CL(A)	189.75	0.16	0.28	0.57	277.01	0.05	0.18	0.50	220.98	0.13	0.29	0.54
Taxo-Expan(A)	341.81	0.07	0.11	0.29	679.26	0.03	0.04	0.10	648.72	0.04	0.08	0.14
TMN(A)	203.28	0.28	0.41	0.63	319.36	0.10	0.15	0.69	246.45	0.31	0.25	0.61
TEAM-RG(M)	1.27	0.95	0.98	1.00	2.04	0.92	0.98	1.00	5.38	0.83	0.88	0.95
TEAM-CL(M)	6.44	0.71	0.82	0.93	11.06	0.61	0.75	0.89	9.80	0.64	0.71	0.91
TEAM-RG(MA)	73.34	0.61	0.70	0.83	59.81	0.69	0.80	0.85	91.62	0.63	0.71	0.81
TEAM-CL(MA)	99.00	0.37	0.50	0.74	144.38	0.33	0.46	0.70	113.39	0.32	0.47	0.73
Table 2: Ranking results for test queries
(0.2, 0.14, 0.24) in Hit@1 and (0.31, 0.32, 0.37) in Hit@3 for Assamese, Bengali and Hindi WordNet respectively. We see TEAM-CL though performing competitively but is outperformed by TEAM-RG by a margin of (44.82, 86.01, 43.04) in MR, by a margin of (0.11, 0.12, 0.28) in Hit@1 and (0.14, 0.18, 0.43) in Hit@3 respectively for Assamese, Bengali and Hindi WordNet. In TEAM-RG(M), we obtain near-perfect MRR scores. This is because the definitions are already present in training set for the query concepts with known definitions (test sample drawn from the base taxonomy). The score of 1 indicates the ability of the proposed method TEAM-RG(M) to correctly identify the appropriate anchor nodes for the merge operation. TMN gives better performance than Taxo-Expan owing to its useful attention mechanism. But, Team-RG(A) outperforms TMN in all the metrics except Hits@1. We only compare Taxo-Expan results for the attach since it is originally proposed for the attach operation. In the (merge-M) and (merge+attach-MA) section of the tables also, we see that TEAM-RG outperforms TEAM-CL on all three WordNet taxonomies. We attribute this huge performance improvement of TEAM-RG to InfoNCE based training — as it simultaneously provides pseudo-supervision from the negative examples while optimizing for the task-specific regression layers.
5.2 Classification Results
In Table [3], We observe similar trends on all three WordNet taxonomies. Since Taxo-Expan is a regression-based algorithm proposed for only attach operation in taxonomy expansion task — we could not obtain its classification performance. Therefore, we only consider variants of our frameworks as competing methods. As described in the sub-section 3.4, using a classification layer on top
of the regression layer in TEAM-RG, we obtain classification performances for the attach, merge operations along with both (attach+merge) operations. Whereas obtaining classification performance for TEAM-CL is straightforward since this is already a classification framework.
When comparing TEAM-RG (attach) and (merge) variants — we see, unlike ranking results where ranking results of merge operation were always better than the attach operation, here the classification results of merge operation are inferior to attach operation. It means that the RG variant learns better ranking as compared to CL variants, but they fail to distinguish M and A – the operation to perform. This is expected since we do not provide a scheme here to contrast M and A operations — which is the motivation for our CL variant framework.
When comparing TEAM-RG and TEAM-CL for (merge+attach), we see TEAM-CL gives better classification scores using test queries except for Macro-F1 scores. TEAM-RG gives the best performance for the Macro-F1 score for the test cases. This essentially means that class-wise prediction performances are inferior for TEAM-CL. This is expected behavior since, in each batch of the training sample, we include a substantially large number  $(\mathcal{N})$  of negative examples with class-label (N-No operation). We design our training samples like this so that the contrastive loss is better approximated. Nevertheless, it leads to a class-imbalance issue in our three-way classification setup, i.e., a large number of samples with N class labels as compared to the other M/A class labels. Thus, TEAM-CL biases its prediction towards the N class, leading to poorer Macro-F1 scores than TEAM-RG.
To summarize, we observe that TEAM-RG gives the best ranking performances, whereas TEAM-CL gives the best classification performances. TEAM
Assamese WordNet	Bengali WordNet	Hindi WordNet
Methods	Acc	Mi-F1	Ma-F1	Prec.	Recl	F-Sc	Acc	Mi-F1	Ma-F1	Prec.	Recl	F-Sc	Acc.	Mi-F1	Ma-F1	Prec.	Recl	F-Sc
TEAM-RG(A)	0.97	0.97	0.49	0.95	0.97	0.96	0.98	0.98	0.50	0.97	0.98	0.97	0.90	0.90	0.47	0.81	0.90	0.85
TEAM-RG(M)	0.81	0.81	0.45	0.66	0.82	0.73	0.29	0.29	0.29	0.57	0.29	0.48	0.55	0.55	0.30	0.23	0.15	0.39
TEAM-RG(MA)	0.88	0.88	0.88	0.89	0.88	0.88	0.51	0.51	0.36	0.96	0.71	0.85	0.53	0.53	0.43	0.82	0.53	0.62
TEAM-CL(MA)	1.00	1.00	0.50	1.00	1.00	1.00	1.00	1.00	0.48	1.00	1.00	1.00	1.00	1.00	0.50	1.00	1.00	1.00
CL performs poorly in Macro-F1 since it presumably suffers from class-imbalance issues owing to the style of training sample generation. Frameworks with multi-task learning strategy (TEAM-RG and TEAM-CL) outperform frameworks (Taxo-Expan) designed to perform a single task — which is motivated by the fact that simultaneously optimizing for multiple tasks provides self-supervision to each other, resulting in better performances.
5.3 Expansion of Assamese WordNet Taxonomy with Out-Of-Vocabulary (OOV) words
Table 3: Classification results for test queries
Assamese WordNet-Noun
Methods	Micro_MR	Hit@1	Hit@3	MRR
TEAM-RG(A)	65.30	0.33	0.53	0.80
TEAM-CL(A)	240.47	0.02	0.16	0.06
Taxo_Expan(A)	386.30	0.05	0.05	0.11
TEAM-RG(M)	170.79	0.07	0.14	0.38
TEAM-CL(M)	331.93	0.03	0.12	0.29
Table 4: Ranking result for out-of-vocabulary words
To investigate the effectiveness of the proposed models, we employ the models for expanding a WordNet with OOV words. For this, first, we find out-of-vocabulary words, i.e., words that are not present in Assamese WordNet. Second, we manually identify true anchors of respective out-of-vocabulary words with associated operations (Attach/Merge) in Assamese WordNet. We evaluate the predicted results of the proposed model against the manually identified true anchors. Since we can either perform an A or M operation with OOV words and not both, we do not predict expansion tasks for OOV words using any of the MA variants of our proposed frameworks. Table 4 shows the ranking performance of the model in predicting true anchors for attach and merge expansion operations. We see a similar trend of prediction ranking as seen with the test set in our earlier experiments. TEAM-RG gives the best performance in both expansion operations. The detailed analysis of results is in Section F of the Appendix.
Existing methods for taxonomy expansion can be divided into two categories: relying on alignment between multiple taxonomies [Ruiz-Casado et al. (2005), Toral et al. (2008), Ponzetto and Navigli (2009), and Yamada et al. (2011)] or relying on machine learning-based rating sub-graphs. Further, the latter category can be divided into two sub-categories (1) by expanding synonymy relations/Merge (2) by expanding hypernymy relations/Attach. Synonymy-based taxonomy expansion leverages synonymy relations of the taxonomy. Given a seed taxonomy, the distributional approach discovers synonyms by representing strings with their distributional feature and learning a classifier to predict the relation between strings [(Nakashole et al., 2012), (Wang et al., 2019), (Fei et al., 2019)]. Most of the recent taxonomy expansion approaches are based on hypernymy expansion. These methods attempt to determine the attachment position by scoring between several nodes. Recently numerous methods have been proposed to solve this problem(Shen et al., 2018), (Shen et al., 2020), (Yu et al., 2020b), (Zhang et al., 2021), (Liu et al., 2021). (Shen et al., 2020). Hence, all the existing taxonomy expansion approaches expand a taxonomy either by merge operation(synonymy expansion) or by attach operation(hypernymy expansion). However, particular to WordNet expansion it is an integrated task of Merge and attach operation. We are the first to study the problem of taxonomy expansion using both the Attach and Merge taxonomy expansion operations in a single model. A detailed related study is in Section A of the Appendix.
In this paper, we proposed an integrated framework called Taxonomy Expansion with Attach and Merge (TEAM) for expanding taxonomy with attach and merge operations together. We built two multi-task learning-based variants of TEAM, namely, TEAM-Regression and TEAM-Classification, which solve the taxonomy expansion problem as regression and
classification, respectively. Our proposed methods learned to predict the taxonomy expansion operation (merge, attach, or no-operation) to perform and provided a ranked list of candidates. We evaluated the effectiveness of TEAM on WordNet taxonomies of three distinct languages, viz., Assamese, Bangla, and Hindi. In various experimental setups, the proposed TEAM-RG and TEAM-CL outperformed its state of the art for attach operation and provided a highly encouraging performance on merge operation. We had also investigated the performance of the proposed model with out-of-vocabulary concepts.
In the future, we plan to investigate the response of the proposed model with different types of taxonomies and WordNet of different languages. Another future research possibility can be to explore the response of this model using advanced contextual encoders.
James Allan, Ron Papka, and Victor Lavrenko. 1998. On-line new event detection and tracking. In Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, pages 37-45. ACM.
Pushpak Bhattacharyya. 2010. Indowordnet. In In Proc. of LREC-10. CiteSeer.
Adrian Boteanu, Adam Kiezun, and Shay Artzi. 2018. Synonym expansion for large shopping taxonomies. In Automated Knowledge Base Construction (AKBC).
Hongliang Fei, Shulong Tan, and Ping Li. 2019. Hierarchical multi-task word embedding learning for synonym prediction. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 834-842.
Michael Gutmann and Aapo Hyvarinen. 2010. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pages 297-304. JMLR Workshop and Conference Proceedings.
David Jurgens and Mohammad Taher Pilehvar. 2015. Reserating the awesometastic: An automatic extension of the wordnet taxonomy for novel terms. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1459-1465.
David Jurgens and Mohammad Taher Pilehvar. 2016. Semeval-2016 task 14: Semantic taxonomy enrichment. In Proceedings of the 10th international work
shop on semantic evaluation (SemEval-2016), pages 1092-1102.
Thomas N Kipf and Max Welling. 2016. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907.
Zichen Liu, Hongyuan Xu, Yanlong Wen, Ning Jiang, HaiYing Wu, and Xiaojie Yuan. 2021. Temp: Taxonomy expansion with dynamic margin loss through taxonomy-paths. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3854-3863.
George A Miller. 1998. WordNet: An electronic lexical database. MIT press.
Ndapandula Nakashole, Gerhard Weikum, and Fabian Suchanek. 2012. Patty: A taxonomy of relational patterns with semantic types. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 1135-1145.
Kim Anh Nguyen, Sabine Schulte im Walde, and Ngoc Thang Vu. 2017. Distinguishing antonyms and synonyms in a pattern-based neural network. arXiv preprint arXiv:1701.02962.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748.
Bo Pang, Lillian Lee, et al. 2008. Opinion mining and sentiment analysis. Foundations and Trends® in Information Retrieval, 2(1-2):1-135.
Simone Paolo Ponzetto and Roberto Navigli. 2009. Large-scale taxonomy mapping for restructuring and integrating wikipedia. In *IJCAI*, volume 9, pages 2083-2088.
Michael Poprat, Elena Beisswanger, and Udo Hahn. 2008. Building a biowordnet using wordnet data structures and wordnet's software infrastructure—a failure story. In *Software engineering*, testing, and quality assurance for natural language processing, pages 31-39.
Meng Qu, Xiang Ren, and Jiawei Han. 2017. Automatic synonym discovery with knowledge bases. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 997-1005.
Maria Ruiz-Casado, Enrique Alfonseca, and Pablo Castells. 2005. Automatic assignment of wikipedia encyclopedic entries to wordnet synsets. In International Atlantic Web Intelligence Conference, pages 380-386. Springer.
Michael Schlichtkrull and Héctor Martínez Alonso. 2016. Msejrku at semeval-2016 task 14: Taxonomy enrichment by evidence ranking. In Proceedings of the 10th international workshop on semantic evaluation (SemEval-2016), pages 1337-1341.
Jiaming Shen, Zhihong Shen, Chenyan Xiong, Chi Wang, Kuansan Wang, and Jiawei Han. 2020. Taxoexpan: self-supervised taxonomy expansion with position-enhanced graph neural network. In Proceedings of The Web Conference 2020, pages 486-497.
Jiaming Shen, Zeqiu Wu, Dongming Lei, Chao Zhang, Xiang Ren, Michelle T Vanni, Brian M Sadler, and Jiawei Han. 2018. Hiexpan: Task-guided taxonomy construction by hierarchical tree expansion. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 2180-2189.
Amit Singhal et al. 2001. Modern information retrieval: A brief overview. IEEE Data Eng. Bull., 24(4):35-43.
Rion Snow, Daniel Jurafsky, and Andrew Y. Ng. 2006. Semantic taxonomy induction from heterogenous evidence. In Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, pages 801-808, Sydney, Australia. Association for Computational Linguistics.
Kunihiro Takeoka, Kosuke Akimoto, and Masafumi Oyamada. 2021. Low-resource taxonomy enrichment with pretrained language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2747-2758.
Antonio Toral, Rafael Muñoz, and Monica Monachini. 2008. Named entity WordNet. In Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC'08), Marrakech, Morocco. European Language Resources Association (ELRA).
Nikhita Vedula, Patrick K Nicholson, Deepak Ajwani, Sourav Dutta, Alessandra Sala, and Srinivasan Parthasarathy. 2018. Enriching taxonomies with functional domain knowledge. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, pages 745-754.
Petar Velicković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2017. Graph attention networks. arXiv preprint arXiv:1710.10903.
Jin Wang, Chunbin Lin, Mingda Li, and Carlo Zaniolo. 2019. An efficient sliding window approach for approximate entity extraction with synonyms. In EDBT, pages 109-120.
Suyuchen Wang, Ruihui Zhao, Xi Chen, Yefeng Zheng, and Bang Liu. 2021. Enquire one's parent and child before decision: Fully exploit hierarchical structure for self-supervised taxonomy expansion. In Proceedings of the Web Conference 2021, pages 3291-3304.
Ichiro Yamada, Jong-Hoon Oh, Chikara Hashimoto, Kentaro Torisawa, Jun'ichi Kazama, Stijn De Saeger, and Takuya Kawada. 2011. Extending WordNet with hypernyms and siblings acquired from Wikipedia. In
Proceedings of 5th International Joint Conference on Natural Language Processing, pages 874-882, Chiang Mai, Thailand. Asian Federation of Natural Language Processing.
Jiale Yu, Yongliang Shen, Xinyin Ma, Chenghao Jia, Chen Chen, and Weiming Lu. 2020a. Synet: Synonym expansion using transitivity. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pages 1961-1970.
Yue Yu, Yinghao Li, Jiaming Shen, Hao Feng, Jimeng Sun, and Chao Zhang. 2020b. Steam: Self-supervised taxonomy expansion with mini-paths. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1026-1035.
Jieyu Zhang, Xiangchen Song, Ying Zeng, Jiaming Shen, Yuning Mao, Lei Li, et al. 2021. Taxonomy completion via triplet matching network. arXiv preprint arXiv:2101.01896.
Expansion by resource alignment: In the first category of studies, Poprat et al. (2008) first attempted to automatically expand a WordNet with biomedical terminology; however, they were unable in developing the resource. Ruiz-Casado et al. (2005), Toral et al. (2008), Ponzetto and Navigli (2009), and Yamada et al. (2011) exploit structured information in Wikipedia to expand WordNet with new synsets. Snow et al. (2006) leverage distributional similarity techniques for WordNet expansion. Jurgens and Pilehvar (2015) enrich the existing WordNet taxonomy using an additional resource, Wiktionary, to extract sense data based on information in the term concepts.
Synonymy Expansion: Synonymy expansion in a taxonomy leverages synonymy relations to enrich a taxonomy with new concepts. Approaches for synonymy expansion can be divided in two categories: (1) Distributional based approach (Wang et al., 2019), (Fei et al., 2019) (2) Pattern-based approach (Nguyen et al., 2017), (Nakashore et al., 2012). Given a seed taxonomy, the distributional approach discovers synonyms by representing strings with their distributional feature and learning a classifier to predict the relation between strings. However, in the pattern-based approach, consider the sentences mentioning a pair of synonymous strings and learn some textual patterns from these sentences, which are further used to discover more synonyms. Qu et al. (2017) proposed an approach that integrates both the categories. Boteanu et al. (2018) focus on the problem of expanding taxonomies with synonyms for applications in which entities are complex concepts arranged into taxonomies designed to facilitate browsing the product catalog on amazon.com. They first generate synonymy candidates for each node in the taxonomy and then filter synonymy candidates using a binary classifier. Yu et al. (2020a) study a task of synonym expansion using transitivity named SYNET, which leverages both the contexts of two synonymy pairs.
Hypernymy expansion : Jurgens and Pilehvar (2016) formulated a task of synonymy expansion, where it is proposed to enrich the WordNet taxonomy by performing two operations for each new concept. The first action is Attach, where a new concept is treated as a new synset and is attached as a hyponym of one existing synset in WordNet, and the second action is Merge, where a new concept is merged into an existing synset. The best solu
tion proposed by Schlichtkrull and Alonso (2016) included only the attach operation. Later solutions for attaching, as in Shen et al. (2020), adopted self-supervision and tried to exploit the information of nodes in the seed taxonomy to perform node pair matching. On the other hand, Yu et al. (2020b) resorted to classification along mini paths in the taxonomy. In contrast, in our current approach, we have incorporated both the attach and merge operations.
B Graph Propagation Module (in details)
Graph Neural Network (GNN) allows us to transform and propagate node features as messages to learn structure-aware node representations. The EXTRACT() mechanism extracts the messages from a target node and its neighborhood, which is later on combined based on a chosen ATTENTION() mechanism by the AGGREGATE() operation. Next, the aggregated message is propagated to the rest of the graph. Studies apply various aggregation strategies to combine the propagated and extracted messages from the target node's neighborhood based on the importance of each node in the neighborhood towards that target node. GCN (Kipf and Welling, 2016) and GAT (Veličković et al., 2017) are popular GNN frameworks.
GCN uses  $N(*)$  neighborhood-based normalization constant to calculate the importance  $(att_v \to u)$  of node  $v$  towards the target node  $u$  without considering the participating nodes' features as follows.
$$
H _ {u} ^ {l} = \sigma \left(\sum_ {\forall v \in N (\tilde {u})} a t t _ {v \rightarrow u} ^ {l - 1} W ^ {l - 1} H _ {v} ^ {l - 1}\right) \tag {9}
$$
$$
\begin{array}{l} \text {A T T E N T I O N} _ {\mathbf {G C N}} (v \rightarrow u): \\ a t t _ {v \rightarrow u} ^ {l - 1} = \frac {1}{\sqrt {| N (\tilde {u}) | | N (\tilde {v}) |}} \\ \end{array}
$$
$$
\begin{array}{c c c} \operatorname {E X T R A C T G C N} (v) & : & W ^ {l - 1} H _ {v} ^ {l - 1} \end{array}
$$
$$
\begin{array}{c c c} \text {A G G R E G A T E G C N} (*) & : & \sigma (*) \end{array}
$$
where  $\sigma$  is non-linear activation function,  $W^{l}$  is a projection matrix for a GNN layer  $l$  and  $N(*)$  is a node's extended neighborhood structure including the node itself (i.e., including self-loop edges).
GAT uses the same message extraction and aggregation strategies as above except for the fact that it uses attentive aggregation strategies that consider both the participating nodes' features as well as the
neighborhood information, as follows.
$$
\begin{array}{l} \text {A T T E N T I O N G A T} (v \rightarrow u): a t t _ {v \rightarrow u} ^ {l - 1} = \\ \underset {\forall v \in N (\tilde {u})} {\operatorname {S O F T M A X}} \left(c ^ {l - 1} \left(W ^ {l - 1} H _ {u} ^ {l - 1} \oplus W ^ {l - 1} H _ {v} ^ {l - 1}\right)\right) \\ \end{array}
$$