Title: bgGLUE: A Bulgarian General Language Understanding Evaluation Benchmark
Benchmark: bgGLUE
Summary:  bgGLUE (Bulgarian General Language
Understanding Evaluation), a benchmark
for evaluating language models on Natural Language
Understanding (NLU) tasks in Bulgarian.
The benchmark includes NLU tasks targeting
a variety of NLP problems (e.g., natural language
inference, fact-checking, named entity
recognition, sentiment analysis, question answering,
etc.) and machine learning tasks (sequence
labeling, document-level classification,
and regression). 
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: NLU for the Bulgarian language
Phenomenon defined by authors: Yes
Phenomenon definition: We present bgGLUE (Bulgarian General Language Understanding Evaluation), a benchmark for evaluating language models on Natural Language Understanding (NLU) tasks in Bulgarian. Our benchmark includes NLU tasks targeting a variety of NLP problems (e.g., natural language inference, fact-checking, named entity recognition, sentiment analysis, question answering, etc.) and machine learning tasks (sequence labeling, document-level classification, and regression).
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: The task is defined as the evaluation of language models on a benchmark suite of nine NLU tasks in Bulgarian, covering areas such as token classification, regression/ranking, and text classification. Each task is designed to test specific language understanding capabilities, including named entity recognition, sentiment analysis, fact-checking, natural language inference, and question answering
Task item definition: A single item would consist of a text input (e.g., sentence, paragraph, tweet, or document) along with its associated label or target output, depending on the task type. 
Task source: Human exam questions (e.g. GRE questions), Real task examples (e.g. GitHub issues), Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Modified from another benchmark (e.g. translation into another language)
Task ecology: Partial real task (e.g. answering medical questions collected from real people), Representative task (e.g. answering medical licensing exam questions), Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: total 32,448
Dataset size notes: total train 452,449 , total validation 20,930
Dataset metadata available: No
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test, Train, Validation
Response format: Multiple choice, Short free response (e.g. single word or number), Free response (e.g. summary paragarph)
Metric definition: Exact Match (accuracy, F1, precision, recall), Pear./Spear. Corr , Avg. Precision
Metric aggregation: Simple Mean
Metric statistics: simple mean, for tasks with more than one metric (like Pearson and Spearman correlation for sentiment regression), scores are averaged to get a single task score
Metric subscores: No
Metric access: Outputs alone
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: NLP / Understanding
Alternate taxonomy: Multilinguality
New taxonomy: General Capability / Natural Language Understanding