A Hierarchical Hyper-rectangle Mass Model for Fine-grained Entity Typing
Authors: SHING HU
Chunyu Li
Bibo Hao
Venue: Submitted to ICLR 2023 (2023)
Keywords: entity typing
hierarchical classification
hRMM
geometric embedding

Abstract: Fine-grained entity typing is the task of detecting types of entities inside a given language text. Entity typing models typically transform entities into vectors in high-dimensional space, hyperbolic space, or add additional context information. However, such spaces or feature transformations are not compatible with modeling types' inter-dependencies and diverse scenarios. We study the ability of the hierarchical hyper-rectangle mass model(hRMM), which represents mentions and types into hyper-rectangle mass(hRM) and thus captures the relationships of ontology into a geometric mass view. Natural language contexts are fed into the encoder and then projected to hyper-rectangle mass embedding(hRME). We find that hRM perfectly depicts features of mentions and types. With further research in hypervolume indicators and adaptive thresholds, performance achieves additional improvement. Experiments show that our approach achieves better performance on several entity typing benchmarks and attains state-of-the-art results on two benchmark datasets.

Under review as a conference paper at ICLR 2023
A HIERARCHICAL HYPER-RECTANGLE MASS MODEL FOR FINE-GRAINED ENTITY TYPING
Anonymous authors Paper under double-blind review
ABSTRACT
Fine- Fine-grained entity typing is the task of detecting types of entities inside given language text. Entity typing models typically transform entities into vectors in high-dimensional space, hyperbolic space, add additional context information. However, such spaces or feature transformations are not compatible with mod- eling types' inter-dependencies and diverse scenarios. We study the ability of the hierarchical hyper-rectangle mass model(hRMM)), which represents mentions mass(hRM) and thus captures the relationships
and types into hyper-rectangee of ontology into a geometric mass view. Natural language contexts are fed into encoder and then projected to hyper-rectangle mass embedding(hRME). We find that hRM perfectly depicts features of mentions and types. With further research in hypervolume indicator and adaptive thresholds, performance achieves additional improvement. Experiments show that our approach achieves better performance on several entity typing benchmarks and attains state-of-the-arr results on two benchmark datasets.
INTRODUCTION
Entity typing is the task of assigning types to named entities in language texts. Entity typing has shown to widely used tasks such as entity linking (Dai et al.) 2019), knowledge base learning (Hao et al. 2019). and sentence classification. In recent years, this task has become major focus of NLP research.
Many classification systems or hierarchical models have been proposed and achieved promising results in entity typing tasks. Geometric embedding and multi-classification learning are two main techniques in recent years. Rather than representing objects with vectors, geometric representation models are recently assumed to be more suited to expressing relationships in the domain.
Geometric embedding such as Box embedding (Onoe and types (Vilnis et al.) 2018). Box embedding is an interesting view which has been employed for knowledge graph reasoning (Ren al. 2020), knowledge graph completion (Abboud al. 2020), and joint hierarchical representation (Patel et al. (020). However, it has few drawbacks: (I) representation box embedding is too simple to capture latent features in mention and types (2) box embedding treats entity typing task as multiclass multi-label classification task, which means it cannot learn hierarchical knowledge.
202 uses box space to represent mention
Apart from box embedding, Learning hierarchical knowledge in euclidean space (Chen et al. 2020). (Yogatama et al. 2015)) or hyperbolic space (López Strube 2020) are not perfect and far from enough for representing entities which dive into extremely diverse scenarios. In addition, these methods are not capable to illustrate hierarchical relationships between mention, supertype, and subtype.
To overcome the aforementioned drawbacks, we build an entity typing model named hRMM for the entity typing task. As illustrated in figure2] model builds hRM hierarchical architecture that represents entity types and mentions into hRME (Figure
Compared with the baseline model box embedding (Onoe et al. 2021 add density and size scale parameters to determine the mass and size of types in geometric view. These parameters can further investigate more latent features in the language context.

---

Under review as a conference paper at ICLR 2023
/locatoo,, Alsatioonciit A
Adaptive Threshold
Mass Representation
Hyper-redangle Projection
Encoder: Bert large, et:
Jlocatoon
Nocation/cky
Recationyiiver
She works London each day
/lccatinn/kkyothhrss
Mention
Types
Figure I: hRMM architecture for predicting types of entity mention "London" in a given sentence
Further, we develop hypervolume indicators and adaptive thresholds which bring additional improve- ments. The hypervolume indicator minimizes the combination of losses to achieve better loss results. Adaptive threshold, a greedy threshold method, also outperformed the normal threshold method.
Without any hand-crafted features data processing methods, experiments benchmark datasets including Figer, UFET, and OntoNotes demonstrate that our approach outperforms baseline models and prior works. It also indicates that approach is capable of capturing the latent hierarchical structure and language feature in entity typing tasks. We will publish all source codes and datasets of this work GitHub for further exploration.
Memtion and types representations
Intersection computation
Heraachical clessiter and Adaptive threshold
0.4
0.5
0.6
0.8
0.7
Adaptive threshalds
/location
She works London each
/locatio/citty
0.2
Figure Illustration of hRM generation and intersection computations. (Colors objects represent density value which is linear to RGB color model al. 2012))
RELATED WORKS
Many efforts have been invested in entity typing tasks for years. In addition the related work discussed in the previous section, few works focus on correcting noisy labels to improve metrics. Due to the establishment of large-scale weakly and distantly supervised annotation data, may contain abundant noise and thus severely hinder the performance of the entity typing task.
Co-teaching Han et al. 2018) simultaneously train two collaborating networks filter out potentially noisy labels according to their loss. Dix (Li al. 2020) leveraged one-dimensional and two-component Gaussian Mixture Model (GMM) to model the loss distribution of clean and noisy labels.SELF (N Nuyen al. 2019) selects the clean labels according to the agreement between annotated labels and the network's prediction. However, we found that the loss does not form a bimodal distribution in entity typing tasks and thus it is hard to distinguish clean and noisy labels
2

---

Under review as a conference paper at ICLR 2023
precisely. In addition, for deep learning model, precision is not that higher than restricted rules to find noisy labels.
Our work differs from these works in that we do not rely deep learning model denoise the distamtly-label data. Parameters in our model such as density and size scale are trained to depict hierarchies of types. These parameters restrict the hRM and thus denoise the noisy data (show in 3.3) Compared with the deep learning model, our restricted approach holds higher precision. Further, according the experiment in this paper, hRMM learns hierarchical knowledge relies on prior auxiliary resources and the result outperforms prior works in the loss task. In summary, our work focuses on improving the representation of hierarchical entities for hRMM. To this end, we integrate our model in three novel ways: mass representation of mention span and types. hierarchical learning loss, and adaptive threshold.
3 MODEL
In this section, introduce hRMM, our proposed model for entity typing tasks. We start with the entity typing task definition, followed by the overview of hRMM (shown in Figure Further, we introduce technical details of hRME representations includes mention representation, and hierarchical type representation. Apart from descriptions of representation, hierarchical classifier, hierarchical learning loss, and adaptive thresholds will also be discussed the end of this section.
3.1 PROBLEM DEFINITION
Entity typing is considered a hierarchical multi-label classification task. Entity typing datasets consist of collection of sentence text S with mention span X' entity E, and a set of hierarchical types TT. Entities in sentences are labeled with types drawn from type inventory T.
Type set T is consist of hierarchical structure types. For example, type hierarchies take the form of a forest, where each tree rooted by top-level supertype (e.g../ ppr.sn, pperscn/artiss The assignment of labels can be represented as: y/k, yK if k, K E T and K.
We now introduce notations for referring to aspects of hierarchical types in this paper. Relation "type k is subtype of K" is denoted same root" is denoted y E Sb(y) where Sb(y) C.
K Relation "m one of the siblings of k, which shares the
3.2 OVERVIEW OF MODEL
Our model represents mentions and entity types hRME. (Figure 1 Given sentence text S with marked mention span X' and types T respectively, first we employ language model Bert or RoBERTa (Liu al. 2019) to encode input sentence with mention span into single vector. Second, to generate the hyper-rectangle mass representation from the vector, hyper-rectangle projection and mass representation modules transform representations in vector space to the hRM space which consists of four parameters (Ia, Zg, Tú, Ie) (These parameters mean density, size scale value, initial point, and endpoint on an axis respectively). Once the single vector is transformed into hRM representation, the hierarchical classifier step computes the conditional probability of the type E T given the context and mention (S, X). The last step the adaptive threshold, which allocates different threshold values to each type and thus logits will be compared with the threshold to determine the suitable types for each input sentence and mention.
3.3
HRM REPRESENTATION
In total, hRM is characterized by four parameters (I Te, T2 Ie), where æa E R is the density of hRM, T.. Dz, Fe E RR" are the size scale, initial corner and end corner of the hyper-rectangle (Figure 3).
Density parameter The density parameter za is the density for each hRM. It is restricted that types share same root have same density.
3

---

Under review as a conference paper at ICLR 2023
Density Initial Point
End Point
Size Scale
Projection Layer
Encoder
Mention/ //atioor/Typss
Figure 3: natural language is embedded into hRM
Size scale parameter The size scale parameter it is used to multiply by the edge. is assumed that edge length is linear the type level. For example, root type deserves larger size than other level types.
Initial and end corner The initial corner and end corner determined the size of the rectangle 2D view. Figure shows the illustration of size scale, initial and end corner.
/location
/locationccity
size scale
Xi
Xe
Figure 4: 2D view of hRM with size scale, initial corner and end corner
Projection layer extracts features from the hidden vector. We use highway network Srivastava et al. 2015) which projects single vector into hRM. Besides, we use the softplus function (Zheng et al. 2015) to guarantce hRM is positive. To ensure that gradients flow for all conditions, we use Gumbel distributions (Dasgupta et al.) 2020) rather than Gaussian convolution (Li et al. 2019). Gaussian-box process (Rasmussen & Williams 2005) is the most natural way represent hRM. However, cannot support us to solve the expected hRM in an elegant form. We choose gumbel-box process instead to mitigate learning difficulties arising from model unidentifiability and the attendant large flat regions of parameter space. Following the gaussian-bos process, we use softplus approximation represent the mass of reper-re and train embedding later.
Projection layer
hRM representation In total, hRM is then computed
4

---

Under review as a conference paper at ICLR 2023
Ma(x) = Id [[ xec
(I)
k
3.3.1 MENTION REPRESENTATION
To represent sentence with mention in mass representation, first chunk into tokens with specific format. Given mention span m in sequence of words denoted as s, We format input with tokens [CLS] and[SEP] as:
input [CLS]s][EE]) [SEP]s2
Further, the encoder (BERT-large model) (Devlin al. 2018) encodes chunked tokens into feature vectors. We encode the whole sequence into single vector and [CLS] token is chosen to take as the hidden vector.
Moreover, we split the hidden vector and then project them into mass representations. Similarly, softplus used guarantee the mass of each input positive.
We approximate the mention as the following equation with softplus function:
Ma(x)= = I softplus (I. 2z. = x. 2y) * Dd
(3)
8
3.3.2 HIERARCHICAL TYPE REPRESENTATION
In different, density and size scale parameters restricted compared with section Types that share the same root hold the same density but different size scales (Figure 5 See figur 6 for more details about edge length.
/location
/location/city
Figure 5: /location and /ww./allooccciiiioooooooo share same density(same color), but different size scale
3.4 HIERARCHICAL LEARNING Loss
We aim to minimize the combination of losses for various hierarchies with the proposed model structure.
In previous works, linear combination of losses is normally employed and constant weight is assigned to each loss function (Beltagy et al. 2020). However, it has two drawbacks: (1) It is only possible achieve an optimal solution when the Pareto frontier is convex (Boyd Vandenberghe 2004). (2) Weight assigned to each loss function may not reflect the actual behavior.
We develop another loss function: hypervolume indicator (Miranda Von Zuben 2015). This metric preserves the optimal point in various conditions including non-convex Pareto frontiers. To get rid of Nadir point (Miettinen et al. 2010) in original equation of hyper-volume indicator, we simplify the logarithm equation
logH, (z)
Σ l0. f(()))
Compared with prior work, the Nadir point is simplified to 0.5 and thus computation becomes casier. In addition, it achieves better result than the linear combination of losses.

---

Under review as a conference paper at ICLR 2023
3.5 ADAPTIVE THRESHOLDS
In prior works, fixed threshold (r 0.5) is used for classification of all types (Onoe et al. 2021). For severe type imbalance circumstances, all types share fixed threshold that leads poor performance. To find threshold values for each type, a previous approach employs a threshold searching system that searches 20 evenly spaced numbers over a specified interval [0.1] in each evaluation period. The best threshold is then determined from the best Fl on the dev set (Zhang et al. 2018). However, has a few drawbacks:
(1) the threshold searching system is
(2) the true threshold is in evenly spaced numbers.
To overcome drawbacks in prior works, we instead find a time-saving and more precise threshold searching system. In the initial iteration, the threshold for each type evaluated 10 evenly spaced numbers within [0,1] and the best threshold maximizes the strict F1 on the dev set. Once the best threshold(r,) is found in initial evaluation on the dev set, we then assume the best threshold center and threshold interval to [ra-0.25,r:+0.2555 Then in the following evaluations, the best threshold is selected from 10 randomly sampled points from prior interval r1-00.5,,,,+0.... which maximizes the strict Fl on the dev set.
4 EXPERIMENTS
4.1 BASELINE
Our chief comparison is box embedding modeling of entity types (Onoe et al. 2021). As the primary baseline for all experiments, the same box-based version in both box embedding and our model.
4.2
DATASETS
We conduct experiments on three standard fine-grained entity typing datasets: Figer, UFET, and OntoNotes.
4.2.1
FIGER
Figer dataset (Ling & Weld 2021) is fine-grained set of two hicrarchical levels and 112 types, formulating the tagging problem as multi-class and multi-label classification task. Sentences are sampled from Wikipedia articles and news reports.
4.2.2 UFET
Ultra-Fine Entity typing dataset (Choi et al. 2018) has three types: general, fine-grained, and ultra- fine-grained types. UFET dataset is crowd-sourced and on average, each example in UFET has labels: 0.9 general, 0.6 fine-grained and 3.9 ultra-fine types. It is noticeable that classification in UFET does not provide explicit hierarchies in types, and thus all classes are treated equally during training and evaluation.
4.2.3 ONTONOTES
OntoNotes Weischedel et al. 2011 is large multilingual richly-annotated corpus constructed at 90% inter-annonn antertanntttor agreement. For entity typing task, has three hierarchical levels and 89 types in total.
4.3 EVALUATION METRICS
We follow the standard evaluation protocol and use normal evaluation metrics. For Figer and OntoNotes, adopt the metrics used in (Ling & Weld 2021) where results are evaluated via loose macro and loose micro Fl scores. For UFET dataset, we adopt macro-precision macro-recall, and macro-F1.

---

Under review as a conference paper at ICLR 2023
4.4 IMPLEMENTATIONS
We use bert-base uncased and bert-large uncased as two base encoders for tasks, for a fair comparison with previous work and investigation of baseline models. We also the parameters compared with the baseline box embedding model. For example: All experiments project onto a target space of 109 dimensions, softplus scale=1, type box CenterSigmoidBoxTenaaa Apart from similar parameters, we set typical parameters: size scale=0.5, threshold step = 0.05.
5 RESULTS AND ANALYSIS
We compare the experimental results of our approach with previous models. In addition, we study the contributions of our base model architecture, hypervolume indicator, and adaptive thresholds via ablation. To ensure our findings are reliable, we run each experiment multiple times and report the average performance.
Overall, our approach significantly increases the state-of-the-ard Fl on Figer and parts of UFET datasets. Besides, compared with the chief comparison model, our approach improves all three datasets.
FIGER
Model
Ma-Fi
Mi-Fi
Box Embedding Onoee et al.
2021
79.4
75.0
HETrCChen et (2220)
82.6
80.8
FGET-LTR ( & 2019
83.0
79.8
LITE et al. 2022)
80.1
83.3
Our Model
84.9
83.5
w/o Adaptive thresholds
83.8
80.9
w/o Hyper-volume loss function
80.2
78.5
Table 1: Performance on the Figer dataset
Table T shows the macro-Fl and micro-F1 scores on the Figer test set. Our hRMM outperforms the baseline box model (Onoe al., 2021) on both Ma-FI and Mi-F1 and holds the best Ma-F1 and Mi-FI state-of of-t-e--rttttttt- systems. Our model can clearly distinguish the root type and subset type(see example in Appendix A) We notice that removing adaptive thresholds only causes small performance drop. It holds a similar condition with OntoNotes which might be because of the systematic shifts between training and test distribution.
5.2 UFET
Total
Course
Fine
Ultra-Fine
Model
P
R
[1
P
R
FT
P
R
F1
R
Box(Onon et al 2021
$2.9
39.1
43.0
71.2
82.5
76.4
50.9
55.2
53.0
43.4
24.5
31.9
Uhoee & Dumell
2019
51.6
32.8
40.1
67.4
80.6
73.4
41.6
54.7
47.3
46.3
15.6
23.4
LRN et 2021
53.7
44.9
76.4
77.1
50.6
43.4
26.0
HY XlargeLLo Strube
2020
43.4
34.2
38.2
61.4
73.9
67.1
35.7
46.6
40.4
36.5
19.9
25.7
Our Model
53,9
38.7
45.5
71.2
82.1
76.3
51.5
55,4
53.4
46.9
24.9
32.6
w/o Adaptive thresholds
53.3
38.9
45.0
71.0
81.9
76.1
51.1
55.1
53.0
45.9
24.7
32.1
Table 2: Performance on the UFET dataset
Table 2 shows the main results of all baselines and method the UFET test set. For fair comparison, we implement the baseline and our approach with same encoder in all experiments. We see that:
1) hRM representation still achieves the best performance on non non-hienenn hierarchiaal knowledge scenarios which verifies the calibration of our method.
7

---

Under review as a conference paper at ICLR 2023
2) Both density and size scale two useful parameters for fine- ineegrained label prediction. Com- pared with simple box embedding, hRMM can achieve 0.5% FI improvement by adding these two parameters to the Ultra-Fine type.
Lack of prior hicrarchical knowledge leads to no hypervolume indicator, We believe it will leverage label dependencies better to predict labels once hierarchical knowledge exists.
5.3 ONTONOTES
Model
Ma-F1
Mi-FI
Box Embedding Dasgupta et al. 2020)
77.3
70.9
MLM-ET et al. 2021
85.44
80.35
HYM-ET & Strube
2020)
82.0
80.2
LR-ET et 2021
77.6
71.8
AABELGCN(Xiong et al.
2019
77.8
72.2
PLE Ren et al. 2016)
79.8
77.8
Our Model
83.2
76.5
w/o Adaptive thresholds
82.6
75.9
w/o Hyper-volume loss function
80.9
73.1
Table 3: Performance on the OntoNotes dataset
We further conduct experiments on OntoNotes and report results without augmentation data in Table
We consider that: (1) hRMM achieves better performance on both metrics compared with the baseline simple box model (Onoe et al.] 2021). (2) Our method is not leading the board compared with state-of-the-aar models. We found the main reason is: 52% of mentions were marked as \other in the dataset, which causes difficulty for our model to generate hyper-rectangles that distinguish from other types.
In addition, long-tailed problems and less amount of types severe in OntoNotes compared with Figer and UFET cause hRMM cannot learn types well.
5.4 DISCUSSION OF HIERARCHIES
In our model, hierarchy learning is improved by size scale and density parameters compared with baseline model. Size scale restricts the hRM size and thus keeps subset type inside its root type. For density parameter, same root types share the same density value. It guarantees that the mass of the root type is larger than its subtypes. As result, the following figure shows edges for root type / /pprsssas rson and its subtype /person/ author. In conclusion, restricted hierarchies have significant superior rather than prior investigations.
5.5
ERROR ANALYSIS
5.5.
DATA INCONSISTENT
Data error influences the model metric. In the OntoNotes dataset, notice that some of the examples have inconsistent labels because the distant supervision of data labeling. Here we show a few data errors.
(1) htt////nniiaiioosspooooooooeooae which penalizes models that predict the root type correctly
zeam is present, but its root type /organization is missing,
(2) Present /other with /organi zation/ wttpo//cooooocoannnnoooaomaaaa company and /organization which con- fuses model while training.
Besides, in OntoNotes. 52% of mention spans were marked as /other. There is not a clear definition of / /other type leading confusions to model.
8

---

Under review as a conference paper at ICLR 2023
Person (Ll type) Person/Author (L2 type)
Person
Nethon/Author
Intersection Edge
Hyper-rectange Edge
Embedding Dimension
Figure Edges of /Person VS /Person/Author: Because /Person is the root type of (Person/Autes /Atthor edges of subtype are all covered by root type
5.5.2 DATA IMBALANCE
Data imbalance is an another reason for leading to model prediction error For Figer dataset, /location, /person and / /organization are three most types. Besides, it is calculated that the top 3 types cover over 80% of the evaluation data.
CONCLUSION
this paper, we propose new approach for fine-grained entity typing. There are three contributions:
(1) We propose brand new architecture that represents natural language in hRME.
(2) We adopt hypervolume indicator rather than the linear combination of losses to achieve better loss function.
(3) We utilize adaptive classification thresholds to further boost the performance.
Experiments show the benefits of our model over prior investigations and we achieve new state-of art results on two benchmarks. For future work, we are interested incorporating additional ideas from contrast learning and federated learning. Furthermore, we are also passioned about adapting our model other NL.P domains
REFERENCES
Ralph Abboud, ismail Ílkan Ceylan, Thomas Lukasiewicz, and Tommaso Salvatori. Boxe: A box embedding model for knowledge base completion. CoRR, abs/2007.06261 2020. URL
https://aara /ttp:://axivv... org/ abs 2007.0622
Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint wrXiv:004.0051500 2020.
Stephen Boyd and Lieven Vandenberghe. Convex Optimization. Cambridge University Press, 2004. doi: 10. 10.1017/CB097800 1804441.
Tongfei Chen, Yunmo Chen, and Benjamin Van Durme. Hierarchical entity typing via multi-level learning to rank. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online. July 5-10, 2020, pp. 3465-8473 2020. URL https: /t::://www /www acl web. wrg. antho logy ac1 -main. 7497

---

Under review as a conference paper at ICLR 2023
Eunsol Choi, Omer Levy, Yejin Choi, and Zettlemoyer. Ultra-fine entity typing. In Proceedings of the ACL. Association for Computational Linguistics, 2018.
Hongliang Dai, Donghong Du, Xin Li, and Yangqiu Song. Improving fine-grained entity typing with entity linking. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP. IJCNLP). pp. 6210-6215 Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/D199 1643. URL https:: Att.tttt///clllatoooooooooooo www.:hhoooooy../.1-----11-
Hongliang Dai, Yangqiu Song, and Haixun Wang. Ultra-fine entity typing with weak supervision from masked language model. arXiv preprint arXiv:2106.04096 2021.
Shib Sankar Dasgupta, Michael Boratko, Dongxu Zhang, Luke Vilnis, Xiang Lorraine Li, and Andrew McCallum. Improving local identifiability in probabilistic box embeddings. ArXiv, 1os/2000.04831, 2020.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/18 10s//880.04485, 2018. URL
http://arxiv wttp://////...../.0/// 04805
Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy la- bels. In Bengio, Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Asso- ciates, Inc., 2018. URL 00000266777000000000000000066550
wtwp:/owoooeeiioo..e..eeeeeeeeee neurips cc/ cc/pappe//011//11e. pdf
Junheng Hao, Muhao Chen, Wenchao Yu, Yizhou Sun, and Wei Wang. Universal representation learning of knowledge bases by jointly embedding instances and ontological concepts. Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2019.
Noor A Ibraheem, Mokhtar M Hasan, Rafiqul Z Khan, and Pramod K Mishra. Understanding color models: a review. ARPN Journal of science and technology, 2(3):265-275, 2012.
Bangzheng Li, Wenpeng Yin, and Muhao Chen. Ultra-fine entity typing with indirect supervision from natural language inference. Transactions of the Association for Computational Linguistics, 10:607-622, 2022.
Junnan Li, Richard Socher, and Steven C. H. Hoi. Dividemix: Learning with noisy labels as semi- supervised learning. CoRR, abs/2002.07394, 2020. URL http/t:ss////xtttt wtt.aaiiii.gggaaaaaaaaooooo abs aws/2b000 2002 07394
Xiang Lorraine Li, Luke Vilnis, Dongxu Zhang, Michael Boratko, and Andrew McCallum. Smoothing the geometry of probabilistic box embeddings. In ICIR, 2019.
Ying Lin and Heng Ji. An attentive -raiird entity typing model with latent type representation. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing ( (EMNLP-IJCCLL) pp. 6197-6202, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 18.18663/v 1/D19-
URL
1641
Xiao Ling and Daniel Weld. Fine-grained entity recognition. Proceedings of the AAAI Conference on Artificial Intelligence, 26:94- 100, Sep. 2021. doi: 10.1609/aaai.
URL https
//ojs. www.aaaaaaaaaaaaaaaaaaaaaaaaeeee www./oogg.........oo.oooooooe.ee www.///////a/........a... aara/aaiiii http://aaricceeee///./...
Qing Liu, Hongyu Lin, Xinyan Xiao, Xianpei Han, Le Sun, and Hua Wu. Fine-grained entity typing via label reasoning. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 4611 1-4622, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 18.18653/ 1/2021..mml2pppp.. main.378. URL https: / antho ogg.. emn p-mmllmmmma ma-maa
378
10

---

Under review as a conference paper at ICLR 2023
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer. and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692, 2019. http:::/ org/ abs/ //s/// / 1907 9997.11692
Federico López and Michael Strube. A fully hyperbolic neural model for hierarchical multi-class classification. In Findings of the Association for Computational Linguistics; EMNLP 2020, pp. 460-475, Online, November 2020. Association for Computational Linguistics. URL https
/www.aclwwb.... http://////ooooo////////.///////
findings emnlp 42
Kaisa Miettinen, Petri Eskelinen, Francisco Ruiz, and Mariano Luque. Nautilus method: An inter- active technique in multiobjective optimization based on the nadir point. European Journal of
Operational Research, 206(2):426-434, 2010. ISSN 0377-2217, doi: https://doi.org/1000011//
2010.02.041. URL https: /wwww sciencedi com/ wtt.c///cccinnneeaattieeee///aia
90371221710001785
Conrado Silva Miranda and Fernando José Von Zuben. Multi-objeecive optmiiiii for self-adjusting weighted gradient in machine learning tasks. arXiv preprint arXiv: 2015.
Due Tam Nguyen, Chaithanya Kumar Mummadi, Thi-Phuong-Nhung Ngo, Thi Hoai Phuong Nguyen, Laura Beggel, and Thomas Brox. SELF: learning to filter noisy labels with self-ensembling. CoRR, abs/1910.01842, 2019. URL
org/ abs
Yasumasa Onoe and Greg Durrett. Learning to denoise distantly-laheled data for entity typing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 2407-2417, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v 1/t3://///1111121
Yasumasa Onoe, Michael Boratko, Andrew McCallum, and Greg Durrett. Modeling fine-grained entity types with box embeddings. In Proceedings of the 59th Annual Meeting of the Associ- ation for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pp. 2051-2064, Online, August 2021. Asso- ciation for Computational Linguistics. doi: 10. 12111655/1/2221.... Lacl-oon..1l-66666 URL https //aclanthologe wwtp://l.goooa...oo acl-oon.........._____...o
Dhruvesh Patel, Shib Sankar Dasgupta, Michael Boratko, Xiang Li, Luke Vilnis, and Andrew McCallum. Representing joint hierarchies with box embeddings. In Automated Knowledge Base Construction, 2020. URL
Carl Edward Rasmussen and Christopher K. I. Williams. Gaussian Processes for Machine Learning (Adaptive Computation and Machine Learning). The MIT Press, 2005. ISBN 026218253X.
Hongyu Ren, Weihua Hu, and Jure Leskovec. Query2box: Reasoning over knowledge graphs in vector space using box embeddings. CoRR, abs/2002.05969, 2020. URL httph:t///saiiiii
org//bb/ 2202. 05969
Xiang Ren, Wenqi He, Meng Qu, Clare R. Voss, Heng Ji, and Jiawei Han. Label noise reduction in entity typing by heterogeneous partial-label embedding. CoRR, abs/1602.05377, 2016. URL
Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. Highway networks. CoRR, abs/1505.00387, 2015. URL httttt///a/aiiii org/ abs 00387
Luke Vilnis, Xiang Li, Shikhar Murty, and Andrew McCallum. Probabilistic embedding of knowledge graphs with box lattice measures. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 263-272, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18853/w1/P18-1 1025. URL https
7/ //aclanthology. wwt/polgg--........... -025
Ralph Weischedel, Eduard Hovy, Mitchell Mareus, Martha Palmer, Robert Belvin, Sameer Pradhan, Lance Ramshaw, and Nianwen Xue. Ontonotes: A large training corpus for enhanced processing. Handbook of Natural Language Processing and Machine Translation. Springer, 3(3): 2011.
11

---

Under review as a conference paper at ICLR 2023
Wenhan Xiong, Jiawei Wu, Deren Lei, Mo Yu, Shiyu Chang, Xiaoxiao Guo, and William Yang Wang. Imposing rabatonelational inductive bias for extremely fine grained entity typing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume (Long and Short Papers), pp. 773-784, Minneapolis, Minnesota, June 2019, Association for Computational Linguistics. doi: 10.18653/v1/
N19- 1084.
Dani Yogatama, Daniel Gillick, and Nevena Lazic. Embedding methods for fine grained entity type classification, In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pp. 291-296, Beijing, China, July 2015, Association for Computational Linguistics. doi: 10.3115/v 111///1550488
2048
Sheng Zhang, Kevin Duh, and Benjamin Van Durme. Fine-grained entity typing through increased discourse context and adaptive classification thresholds. In Proceedings of the Seventh Joins Conference on Lexical and Computational Semantics, pp. 173-179, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10. 22118653/1/SS1-20222 2022.. URL https //aclantholooy.oom www.ooooy.ooooggo/ 2022
Hao Zheng, Zhanlei Yang, Wenju Liu, Jizhong Liang, and Yanpeng Li. Improving deep neural networks using softplus units. In 20/5 International Joint Conference on Neural Networks (IJCN), pp. 1-4, 2015. doi: 10.110/IUCNN22015.3344499
A APPENDIX
We plot mention Apple Company, type /Organization and unrelated type AarganizationBBducca htt//ddcaaioonal ional Institute respectively in Figure 7 figures, Intersection of Figure (a) largely more than Figure b))
Compared with two
12

---

Under review as a conference paper at ICLR 2023
Organization type) VS Apple company (mention)
Organization
Apple company
Intersection Edge
Hyper-rectangle Edge
Embedding Dimension
(a) Edges of /Organization VS Apple company
Organization Educational Institution (L2 type) VS Apple company (mention)
Organization Educational Institution
Apple company
Intersection Edge
Hyper-rectangle Edge
Embedding Dimension
(b) Edges of /Organi wwwpraaainnnnnnnttiiooooaa Institute VS Apple company
Figure 7: Comparison same mention (grey line) between different hierarchical types (blue lines)
13


Reviews:
[{"review_id": "mky32N9B2V", "reviewer": "Reviewer_6W9o", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents an entity typing technique that uses geometric embeddings, and evaluates it on several data sets, achieving strong performance.", "summary_of_the_review": "While this paper goes after an interesting task, I found the clarity and novelty to be too limited for publication.", "strength_and_weaknesses": "Strengths\n\nThe results in this paper are strong, over a number of data sets, and better handling of hierarchically-organized types in NER is an important task.\n\nWeaknesses\n\nI found the paper to be so unclear that I did not have confidence in the contribution (as discussed below), and to the extent I understood the method I found the novelty to be limited.  The model appears to use box embeddings but with additional size and density parameters which are a fairly straightforward addition.  If the motivation for including these had been described more clearly, it's possible even with limited novelty this could be a sufficient contribution for publication.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Uit_ei8XuOg", "reviewer": "Reviewer_Q73W", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes hRMM for fine-grained entity typing, which represents mentions and types into the hyper-rectangle mass to capture the relationships of ontology into a geometric mass view. ", "summary_of_the_review": "This paper proposes a new model for fine-grained entity typing, which represents mentions and types into the hyper-rectangle mass to capture the relationships of ontology into a geometric mass view. The novelty seems limited and the empirical results are relative weak.", "strength_and_weaknesses": "\nStrength\n\n- The additional parameter: density and edge scaler make sense for a mass model to represent the hierarchical information in types.\n- The method works well on FIGER\n\nWeakness\n\n- Baselines such as LITE (included in the FIGER datasets, SOTA, 50.6 ma-f1), and MLM-ET (49+ ma-f1) are not included in the UFET benchmarks, authors should clarify why you discard them before claiming that **you are SOTA**.\n- The total model is too close to the Box-embedding for FET (Modeling fine-grained entity types with box embeddings. Onoe et al,. 2021), basically adding a scaler and a vector multiplier for type and mentioned encoding, the contribution is not big. The performance of hRMM is subtle on UFET (0.5 ma-f1 higher than Box embedding with “adaptive threshold”), especially given that the SOTA model is LITE with 50.6 ma-f1.\n- The adaptive threshold part is a simple trick that doesn’t have too much contribution.\n- The density scaler and the size scaler vector is the key difference between the basic box embedding (Onoe et al,. 2021),  however, the authors lack a detailed introduction and analysis of them. Why the density x_d \\in \\mathcal{R}? It should be positive x_d \\in \\mathcal{R^+}?  Is the size scaler the same for all edges in a hyper-rectangle? It should be positive? x_d and x_s are both trainable parameters or fixed during training?\n\nMinors:\n- parenthesis usage: hierarchical hyper-rectangle mass model(hRMM) → hierarchical hyper-rectangle mass model (hRMM), should leave a space.\n- Figure2 → Figure 2, Figure4 → Figure 4", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "7bZMEMRPzEj", "reviewer": "Reviewer_zfGa", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a new method of hierarchical entity typing that represents mentions and types using a hyper-rectangle representation, models the mention-type relation using the intersection of these representations, and classifies per-mention types according to the mass of this intersection. The mass representation differs from related work in box embeddings, with the introduction of size scale and density paramaters. There is also a novel loss formulation to aid learning of hierarchical representations.\n\nThe new approach outperforms previous work on two fine grained entity typing benchmarks (Figer, UFET) and approaches state of the art on OntoNotes. A visualization of the hyper-rectangle representation of the types `/person` and `/person/author` show that the more general type subsumes the more specific one, geometrically.\n", "summary_of_the_review": "This paper presents a method of hierachical entity typing that achieves impressive results, with respect to recent published work. However, the current presentation is unclear and key concepts are not properly motivated or defined. I feel that Section 3 needs to be rewritten to (a) properly motivate the size scale and density parameters, (b) mathematically define the full process of hierarchical entity type classification, (c) fully define the loss used to train the model.\n", "strength_and_weaknesses": "#### Strengths\n- The results are quite positive with respect to recent published work.\n- The claim of building hierarchical representations without complex training procedures, is an interesting one.\n\n#### Weaknesses\n- The current presentation lacks a lot of essential detail and motivation. Neither of the key concepts 'size scale' and 'density' are properly motivated and the full model & losses are not mathematically defined. I have more comments below in the section on clarity & reproducability.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "-DRuHl8FWP", "reviewer": "Reviewer_W1hi", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to introduce two additional parameters – density and size scale in the box embedding for fine-grained entity typing. The experiments on three benchmarks show the proposed approach achieved some marginal improvements over the considered baselines.", "summary_of_the_review": "This paper has poor quality with inadequate empirical or theoretical support. The reported results on some of the baselines may have technical flaws. The writing is poor and unclear. The results may not be easily reproduced.", "strength_and_weaknesses": "Strength:\n\nThe authors introduced density and size scale parameters to improve the representation ability of box embedding.\n\nWeaknesses\n\n1. Weak motivation and novelty: while the authors propose to introduce density and size scale to box embedding, there is no clear explanation why we need these two parameters. The original box embedding has already been able to capture the inter-dependencies among event types.\n\n2. The experiments are not solid. In terms of performance gain, the proposed approach only achieves marginal improvements over the baselines on two of the datasets, and underperforms the previous baseline on OntoNotes. Also, there is no detailed empirical analysis of the effect of the two new parameters.\n\n3. The reported results of LITE in Table 1 are not consistent with the results reported in the original paper. In fact, the results in Table 1 are merged from the two versions of LITE (pre-trained on NLI+UFET + NLI+task-specific training). It seems this is a technical error.\n\n4. Writing needs to be significantly improved. The paper is poorly structured and lacks many details, e.g., the intuition and motivation of the approach, as well as theoretical or empirical analysis.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject