Title: What to Say and When to Say it: Live Fitness Coaching as a Testbed for Situated Interaction
Benchmark: QEVD-FIT-COACH
Summary: Open-ended, asynchronous interactions, where an AI model may proactively deliver timely responses or feedback based on the unfolding situation in real-time, are an open challenge. This work presents the QEVD-FIT-COACH benchmark and dataset, which explores human-AI interaction in the challenging, yet controlled, real-world domain of fitness coaching – a task which intrinsically requires monitoring live user activity and providing immediate feedback.
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: Situated interaction
Phenomenon defined by authors: Yes
Phenomenon definition: A notable type of situated interaction is the instructional or coaching scenario, where an instructor guides a user through a complex activity, such as live fitness coaching. 
Definition scope: Subset
Definition integrity: Single cohesive phenomenon
Definition integrity detail: No
Phenomenon contested: Contested
Task definition: Feedback Structure: The feedbacks in the QEVD-FIT-COACH benchmark have the following structure: At the start of each exercise, acknowledging feedback is given once the user has started; otherwise, a reminder to do so is provided. A corrective feedback is provided as soon as a mistake is clearly visible. Similarly, when the user begins to correct their mistake, feedback is provided to acknowledge and guide the user to successfully correct the error. If the user is performing the exercise correctly, feedback focuses on repetition counting.  Finally, at the end of each exercise, a feedback focused on the overall performance during that exercise is provided. 
Task item definition: Video frames, a list of feedback statements that correspond to a specific timestep
Task source: Real task examples (e.g. GitHub issues), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task ecology: Complete real task (e.g. providing medical advice to real people interactively)
Dataset size: 28,326
Dataset size notes: 377,678
Dataset metadata available: No
Sampling method: Random sample (creators defined a task space and sampled from it)
Split reviewed: Test, Train
Response format: Free response (e.g. summary paragraph, executable code), Extended interaction (e.g. conversation, calling an API and processing the response)
Metric definition: n-gram (BLEU, ROUGE, chrF), LLM-as-a-Judge (text quality, preferences, NOT extracting answers for other metrics)
Metric aggregation: Simple Mean
Metric statistics: simple mean
Metric subscores: No
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: The benchmark is itself realistic
Human baseline: Yes
Authors discuss validity: No
Taxonomy: User Interaction / 
New taxonomy: General Capability / User Interaction