Title: Nibbling at the Hard Core of Word Sense Disambiguation
Summary: The authors introduce new challenging test sets for Word Sense Disambiguation evaluation  specifically designed to evaluate model resilience on rare word senses and present a more rigorous evaluation framework.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Word Sense Disambiguation (WSD): automatically assigning a correct meaning to an ambiguous word in context
Phenomenon defined by authors: Yes
Phenomenon definition: "Word Sense Disambiguation (WSD), the task of automatically assigning a meaning to an ambiguous word in context"
Definition scope: Subset
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Widely-agreed
Task definition: Word Sense Disambiguation (WSD) is the task of automatically assigning the correct meaning to an ambiguous word within a given context, selecting from a predefined sense inventory. 
Task item definition: A single item consists of a context (sentence or paragraph) containing a target ambiguous word.
Task source: Real task examples (e.g. GitHub issues), Modified from another benchmark (e.g. translation into another language)
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Task ecology detail: WSD is a fundamental NLP capability that would be used as a component within larger systems.
Dataset size: 7,253 instances
Dataset metadata available: Yes
Dataset metadata detail: word sense frequency, domain information, presence in training data
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Multiple choice
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: micro-averaged F1 and macro-averaged F1 scores
Metric statistics: F1 scores (micro-averaged and macro-averaged) as the primary statistical method.
Metric subscores: Yes
Metric subscores detail: Presence in training data (SemCor; non-SemCor instances); Word sense frequency (WordNet first sense vs non-first sense); Dataset (ALL, ALLNEW, S10NEW, 42D, hardEN, softEN)
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: Yes
Authors discuss validity: Yes
Taxonomy: NLP / Understanding
New taxonomy: General Capability / Natural Language Understanding