Title: Toward Informal Language Processing: Knowledge of Slang in Large Language Models
Summary: Using movie subtitles, the authors construct a dataset that supports evaluation on a diverse
set of tasks pertaining to the automatic processing of slang. For both evaluation and finetuning, they show the effectiveness of their dataset on two core applications: 1) slang detection, and 2) identification of regional and historical sources of slang from natural sentences.
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: informal language processing (Knowledge of slang in LLMs)
Phenomenon defined by authors: No
Phenomenon definition: They focus on two core tasks for informal language processing. First, they evaluate the extent to which LLMs can reliably detect slang usages in natural sentences. Second,
they assess whether LLMs can be used to identify regional-historical sources of slang via a text classification task.
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: Task1: Given a set of sentences, they evaluate slang detection at both sentence-level and word-level.
Task2: Given a sentence containing a slang usage, they ask the model to classify its source (e.g. region and age).
Task item definition: a sentence of natural language
Task source: Crowd-sourced task examples (e.g. Prolific-created tasks)
Task source detail: The benchmark is build on top of OpenSubtitles corpus.
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: 25,000
Dataset metadata available: Yes
Dataset metadata detail: Annotator confidence, Movie ID, Region, Year
Sampling method: Random sample (creators defined a task space and sampled from it)
Split reviewed: Test, Train
Response format: Multiple choice
Metric definition: Exact Match (accuracy, F1, precision, recall), They also report two metrics to compare an LLM’s predictive confidence in slang usages relative to their literal counterparts.
Metric aggregation: Simple Mean
Metric statistics: simple mean
Metric subscores: No
Metric access: Outputs alone
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: Multilinguality / 
New taxonomy: General Capability / Multilinguality