Title: FETA: A Benchmark for Few-Sample Task Transfer in Open-Domain Dialogue
Benchmark: FETA
Summary: Examines few-sample task transfer across 17 subtasks (e.g., utterance-level classification, dialogue-level classification, span extraction, multiple-choice) in open-domain dialogue with diverse properties (dyadic vs. multi-party, anonymized vs. recurring speaker, varying dialogue lengths).
Contribution:  Claims to be "the the first large-scale benchmark for task transfer in dialogue, with 132 source target-task pairs"
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Task transfer, transferring knowledge contained in related tasks, in few-sample settings (10% of original instance set)
Phenomenon defined by authors: Yes
Phenomenon definition: Task transfer, transferring knowledge contained in related tasks. Definition 3 (Task Transfer). Given a source task TS = {YS, fS(XS)} and target task TT =
{YT , fT (XT )}, task transfer is the use of a learning algorithm, A, to improve the learning of fT by using the knowledge in TS.

They also define Few-Sample: For this reason, we focus on the fewsample setting, defined in FETA as 10% of the original instance set. Out of 10%, 5%, and 1%,
10% was empirically determined to be the smallest percentage that retains labels from all label sets in both the train and development partitions.
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Widely-agreed
Purpose notes: They define seperately: (1) Cross-dataset task transfer, when XS ≠ XT ,
we also have P(XS) ≠ P(XT ) and DS ≠ DT ;
domain shift; vs (2)  intra-dataset task transfer, when XS = XT , there is no domain shift.
Task definition: The tasks are classic NLP tasks subsumed in dialog - e.g., emotional recognition during chit-chat conversations, or character identification from a TV transcript.
Task item definition: Input = a dialogue (from DailyDialog); Subtask = Emotion Recognition; Output = Happiness; OR Input = a transcript from a TV Show (from Friends); Subtask = QA, question = How long did Rachael train for?; Output = 2 weeks.
Task definition detail: They focus on intra-dataset transfer but not cross-domain transfer.
Task source: Modified from another benchmark (e.g. translation into another language), Human TV show; Human chitchat dialogues
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Task ecology detail: Using the model for various tasks contained in dialogue seems a more general ecologically valid use case, than the Friends transcript understanding but this could also be an applied usecase.
Dataset size: 71,212
Dataset size notes: Train=28,261, Dev = 5,132
Dataset metadata available: Yes
Dataset metadata detail: They provide the datasource (dialog, friends), the task name (e.g., emotion recognition, or QA), and the a categorisation of task type (e.g., utterance classification vs span extraction)
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible)
Split reviewed: Test, Train, Validation
Response format: Depends on the subtask category (Utterance Classification, Dialogue Classification, Multiple Choice, Span Extraction)
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric statistics: Mean, and they they show a delta (for change in aggregate sources across all tasks). It is unclear if this is a range or a standard deviation. I think it's a range.
Metric subscores: Yes
Metric subscores detail: They provide results over the task categories - Utterance Classification, Dialogue Classification, Multiple Choice, Span Extraction
Metric metascoring: They calculate a top1-score: " to understand how models and algorithms perform if the best source task is known ahead of time. This score is calculated as the maximum score over source tasks averaged over target tasks"
Metric access: Outputs alone
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Partially
Metric face validity: Partially
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: Language Modelling / Adaptability
New taxonomy: General Capability / Adaptability
Procedural notes: Was originally run as a challenge for a ACL 2023 workshop