claw-eval/Claw-Eval huggingface dataset:
Task category: video_search
There is a video file at /workspace/fixtures/video.mp4.

Watch the video carefully and complete the following task:

1. Identify what device is used to shoot vertical short videos in the video. Save your answer as a single line to /workspace/answer.txt.
2. Extract a frame from the video where this device clearly appears, and save it as /workspace/device_frame.png.

Make sure both output files are saved to their specified paths.
Task category: video_search
There is a video file at /workspace/fixtures/video.mp4.

Watch the video carefully and complete the following task:

1. Identify the specific Bugatti model that was historically the first to feature a closed automobile roof. Save your answer (the model name/number only) as a single line to /workspace/answer.txt.
2. From the video, find a frame that clearly shows the on-screen text label of the Bugatti model you identified in step 1. The model name/number must be visibly readable as text in the frame. Save that frame as /workspace/bugatti_frame.png.

Make sure both output files are saved to their specified paths.
Task category: video_search
容器里有以下文件：
- /workspace/fixtures/video.mp4

请观看视频，完成以下任务：

1. 找出视频中正式介绍的第11个建筑物，告诉我它截止2026年3月有多少米高。将你的回答（建筑物名称和高度）保存为一行文本到 /workspace/answer.txt。
2. 从视频中截取一帧该建筑物清晰出现的画面，保存为 /workspace/building_frame.png。
Task category: video_search
There is a video file at /workspace/fixtures/video.mp4.

Watch the video carefully and complete the following task:

Count the total number of distinct news segments that appear in the video.
For each segment, identify its start and end timestamp.

Save your answer (the count and timestamps for each segment) to /workspace/answer.txt.
Task category: video_search
容器里有以下文件：
- /workspace/fixtures/video.mp4

请观看视频，完成以下任务：

我是一个健身小白，帮我总结一下这个视频中介绍的所有动作，以及能锻炼哪些肌肉。把结果总结成一个markdown文档保存到 /workspace/summary.md。
Task category: video_search
容器里有以下文件：
- /workspace/fixtures/video.mp4

请观看视频，完成以下任务：

我是初中体育老师，帮我看看这些同学中有谁完成了至少一个标准引体向上（下巴超过单杠）。把每位完成了标准引体向上的同学对应的帧截图保存为png，按 /workspace/student1.png、/workspace/student2.png 依次命名。
Task category: video_search
容器里有以下文件：
- /workspace/fixtures/video.mp4

请观看视频，完成以下任务：

统计一下视频中几款手机的型号、屏幕大小、处理器、相机、电池续航、价格等参数，总结成一个markdown文档的表格保存到 /workspace/summary.md。表格行是手机型号，列是每个维度的信息。
Task category: video_search
容器里有以下文件：
- /workspace/fixtures/shopping.mp4

请观看视频，完成以下任务：

统计一下视频中今天买了哪些东西，分别多少钱，把结果保存到 /workspace/summary.txt。
Task category: video_search
容器里有以下文件：
- /workspace/fixtures/video.mp4

这是一段监控视频。我在调查一个校园枪击案件，请仔细观察视频，找出凶手进入教学楼的完整片段，将该片段截取出来保存为 /workspace/clip.mp4。
Task category: webpage_generation
The container has the following file:
- /workspace/fixtures/image.png

Please view the image and complete the following task:

Recreate this webpage in HTML based on the provided image, ensuring the layout, styling, and visual details closely match the original. Render the generated HTML as an image and save as /workspace/output.png.
Task category: video_qa
The container has the following file:
- /workspace/fixtures/video.webm

Please watch the provided badminton match video carefully and answer the following two questions:
1. How many total shots (rallies) were hit during the first point of the match?
2. How many total shots (rallies) were hit during the last point of the match?
Task category: video_qa
The container has the following file:
- /workspace/fixtures/video.webm

This video shows the second game of a badminton match. Based on the video content, how many games were played in total in this match? Then, for the 1st point, the 10th point, and the last point of the second game, please identify: who served, whether they served from the left or right half of the court, and who scored the point.
Task category: video_qa
The container has the following file:
- /workspace/fixtures/video.webm

This video shows the first game of a badminton match. Please watch the video and answer: among the first 10 points of this game, how many points ended because a player's return shot went long past the baseline? Please list the scores before each of these points in order (ANTONSEN's score first).
Task category: video_qa
The container has the following file:
- /workspace/fixtures/video.webm

This video shows the first game of a badminton match. Please watch the video and answer: during the segment of this game where the score went from 6:6 to 12:13 (Korean player's score listed first), how many times did the Korean player and the Japanese player each hit the shuttlecock into the net?

Note: hitting into the net means a player strikes the shuttlecock and it hits the net or goes below it, failing to cross over to the opponent's side of the court.
Task category: video_qa
The container has the following file:
- /workspace/fixtures/video.webm

Please carefully watch this video highlights. For the first point shown in the video, how many total shots did the two players hit? Did the server win that point?
Task category: video_qa
The container has the following file:
- /workspace/fixtures/video.webm

Please carefully watch this video of the first game of a table tennis match. In the first 10 points of the video, how many serve points did the Chinese player and the French player each have? How many serve points did the Chinese player win? How many serve points did the French player win?
Task category: video_qa
The container has the following file:
- /workspace/fixtures/video.webm

Please carefully watch this video of the first game of a table tennis match. In the first 10 points shown in the video, how many times did the Chinese player score directly by smashing the ball such that the opponent could not touch it at all? For each such point, please list the score before that point started (Chinese player's score first).
Task category: video_qa
The container has the following file:
- /workspace/fixtures/video.webm

Please carefully watch this video of the first game of a table tennis match. How many let serves (where the ball clips the net and the serve is replayed) occurred in total during this game? Please list the score at the time of each let serve, with the Brazilian player's score first.
Task category: video_qa
The container has the following file:
- /workspace/fixtures/video.webm

Please carefully watch this video of the first frame of a snooker match. In this frame, one player completed a clearance (potted all remaining balls in succession). Please list the colors of the balls potted by that player in order.
Task category: video_qa
The container has the following file:
- /workspace/fixtures/video.webm

Please carefully watch this video of a frame from a snooker match. At what time (minutes:seconds) in the video did the British player first attempt to pot the brown ball? At what time (minutes:seconds) did the British player successfully pot the brown ball? Please base the timestamps on the moment the cue ball makes contact with the brown ball.
Task category: video_qa
The container has the following file:
- /workspace/fixtures/video.webm

Please carefully watch this soccer match video. How many goals were scored in total during this match? For each goal, please provide the time in the match (minutes:seconds) when it was scored and which country scored it.
Task category: video_qa
The container has the following file:
- /workspace/fixtures/video.webm

Please carefully watch this soccer match video. Which team's goalkeeper made the first successful save shown in the video? At what time (minutes:seconds) in the video does the goalkeeper touch the ball during this save? Note that this save is shown multiple times in the video — please use the timestamp from the first time it appears.
Task category: video_qa
The container has the following file:
- /workspace/fixtures/video.webm

How many times in the video did Lehecka lose a point by hitting the ball into the net during a multi-shot rally? Please first state the total count. Then for each such instance, provide the score before that point (including set score, game score, and point score, with Lehecka's score listed first), as well as the time (minutes:seconds) when that point ended.
Task category: video_qa
The container has the following file:
- /workspace/fixtures/video.webm

How many times in the video did a player hit a defensive lob to the opponent's baseline that resulted in a winner (i.e., the player wins the point and the opponent failed to reach the ball)?
For each such instance, please provide the name of the player who scored, the name of the opposing player, who served that point, and the total number of shots hit during that point.
Task category: video_qa
The container has the following file:
- /workspace/fixtures/video.webm

Find the point in the video where the two players hit 10 or more total shots. What is the start time (minutes:seconds) of this point? How many total shots were hit? Who served? Who won the point? Did the winner score by hitting a winner or did the opponent make an error?

Note: A winner is a shot that lands in the valid area of the opponent's court and the opponent fails to touch the ball. An error is when a player hits the ball out of bounds or into the net.
Task category: video_qa
The container has the following file:
- /workspace/fixtures/video.webm

Watch the video and answer: how many set points in total did the player wearing a white top fail to convert across all three sets? How many set points did he fail to convert in the first set, second set, and third set respectively?

Note: A set point is a score situation where a player needs only one more point to win the current set. Failing to convert a set point means the player had a set point but did not win that point.
Task category: video_qa
The container has the following file:
- /workspace/fixtures/video.webm

Watch the video and answer: how many aces (serve winners) did the doubles pair ERRANI/PAOLINI and the doubles pair SINIAKOVA/TOWNSEND each hit in this video? How many return aces did each pair hit?

Note: An ace is a serve that lands in the valid service box and the returner fails to touch the ball at all. A return ace is when the returner hits the return of serve for a clean winner that the server cannot touch.
Task category: video_qa
The container has the following file:
- /workspace/fixtures/video.webm

Watch the video and answer: in the first set of this highlights video, how many service points are shown for Zverev and how many for Sinner respectively? Of those service points, how many did Zverev win on his own serve, and how many did Sinner win on his own serve?
Task category: video_qa
The container has the following file:
- /workspace/fixtures/video.webm

Watch the video and answer: in this highlights video, how many break points did Alcaraz and Medvedev each face in total? How many break points did each of them save?

Note: A break point occurs when the returner (the player who is NOT serving) is one point away from winning the game. In other words, if the returner wins this point, they "break" the server's serve. The server is said to "face" a break point, and if the server wins that point, they are said to have "saved" the break point.
Task category: video_qa
The container has the following file:
- /workspace/fixtures/video.webm

Watch the highlights video and identify the time (minutes:seconds) when the last point of the first set, second set, and third set each ends. Also provide the name of the overall match winner.
Task category: doc_extraction
I need to compare the computational efficiency of the models mentioned in the
"Attention Is All You Need" paper. Please locate Table 2. Extract the
"Training Cost (FLOPs)" from the EN-DE column for the following four models:
"GNMT + RL", "ConvS2S", "Transformer (base model)", and "Transformer (big)".
Ignore the ensemble versions.

Save this data into a JSON file named /workspace/training_costs.json where
the keys are the exact model names and the values are the numeric FLOPs
evaluated as Python floats (e.g., 3.3 * 10^18 should be saved as 3.3e18).

Then, write a Python script named /workspace/calculate_ratio.py that loads
this JSON and calculates how many times larger the training cost of
"GNMT + RL" is compared to "Transformer (base model)". Run the script and
save this exact numerical ratio to a text file named
/workspace/efficiency_ratio.txt rounded to 2 decimal places.
Task category: doc_extraction
The container has the following file:
- /workspace/fixtures/GroundingME.pdf

Based on the GroundingME paper, I need an analysis of how much the "Thinking"
mode relatively improves model performance.

Please focus exclusively on the open-source models among those evaluated for
this feature in Figure 5. Synthesize their baseline "Total" accuracy (No-Think)
from Table 3 and their "Think" accuracy from Figure 5 to calculate the relative
percentage increase for each model.

Generate two artifacts:
1. A CSV file named /workspace/thinking_relative_impact.csv containing the
   model names, their baseline accuracy, their thinking accuracy, and the
   calculated relative percentage increase (rounded to 2 decimal places).
   Sort the data in descending order of this relative increase.
2. A vertical bar chart named /workspace/relative_gain_bar.png that visualizes
   this sorted relative percentage increase.
Task category: doc_extraction
The container has the following file:
- /workspace/fixtures/2512.17495v2.pdf

I am analyzing the Spatial reasoning capabilities of all open-source models
evaluated in the GroundingME paper. Please synthesize a unified leaderboard
that includes both their baseline average Spatial scores from the main
experimental results table and their "Thinking" Spatial scores from the
appendix table. Differentiate the entries by appending "(Base)" or "(Think)"
to the model names. Once you have calculated the complete sorted ranking in
descending order, extract exactly the top 10 highest-scoring entries and save
them into a CSV file named /workspace/top10_os_spatial.csv. The CSV should
contain only two columns: Model_Condition and Score. Finally, generate a bar
chart from this CSV named /workspace/top10_spatial_chart.png.
Task category: doc_extraction
I am a researcher comparing multimodal large language models across different
capability dimensions. From the paper "How Far Are We to GPT-4V? Closing
the Gap to Commercial Multimodal Models with Open-Source Suites", I need
you to merge data from
Table 2 (which covers both OCR-related benchmarks and General Multimodal
benchmarks) and Table 3 (which includes MMT-Bench results).

Specifically:
1. From Table 2, extract each model's scores on: DocVQA, ChartQA, InfoVQA,
   TextVQA, OCRBench (OCR-related), and MMMU, MMBench-EN, MMVet (General
   Multimodal), and MathVista (Math).
2. From Table 3, extract each model's MMT-Bench Overall score.
3. Join these two tables by model name as the key. Only keep models that
   appear in BOTH tables.
4. After merging, sort the rows by MMT-Bench Overall score in descending order.
5. Save the merged table as /workspace/internvl_cross_table_merged.csv with
   columns: Model, DocVQA, ChartQA, InfoVQA, TextVQA, OCRBench, MMMU,
   MMBench_EN, MMVet, MathVista, MMT_Bench_Overall.
6. Create a grouped horizontal bar chart
   (/workspace/internvl_cross_table_chart.png) showing each model's DocVQA,
   ChartQA, MMMU, and MMT-Bench Overall scores side by side, with models on
   the y-axis sorted by MMT-Bench Overall (highest on top).

Use "-" or empty for missing values in the CSV. The chart should have a clear
legend and title.
Task category: doc_extraction
I am a researcher studying how LLaVA-OneVision performs across single-image,
multi-image, and video scenarios. From the paper "LLaVA-OneVision: Easy
Visual Task Transfer", I need to merge
data from Table 3 (single-image benchmarks) and Table 5 (video benchmarks)
to compare models across both modalities.

Specifically:
1. From Table 3 (single-image benchmarks), extract each model's scores on:
   AI2D (test), ChartQA (test), DocVQA (test), MathVista (testmini),
   MMMU (val), MMVet (test).
2. From Table 5 (video benchmarks), extract each model's scores on:
   ActivityNet-QA (test), EgoSchema (test), MLVU (m-avg), MVBench (test),
   PerceptionTest (val), VideoMME (wo subs).
3. Join these tables by model name. Only include all LLaVA-OV variants
   (0.5B, 7B, 72B, both SI and final) and GPT-4V.
4. Compute an "Average_Single_Image" score as the mean of the 6 single-image
   metrics (skip missing values), and an "Average_Video" score as the mean of
   the 6 video metrics (skip missing values).
5. Sort by Average_Video score in descending order.
6. Save as /workspace/llava_ov_cross_modality.csv with columns: Model, AI2D,
   ChartQA, DocVQA, MathVista, MMMU, MMVet, ActivityNetQA, EgoSchema, MLVU,
   MVBench, PerceptionTest, VideoMME, Average_Single_Image, Average_Video.
7. Create a scatter plot (/workspace/llava_ov_cross_modality_chart.png) with
   Average_Single_Image on x-axis and Average_Video on y-axis. Label each
   point with the model name. Add a diagonal reference line (y=x).
Task category: doc_extraction
I am a researcher analyzing how the Qwen2-VL model family compares against
GPT-4o and Claude 3.5 Sonnet across different capability dimensions. From
the paper "Qwen2-VL: Enhancing Vision-Language Model's Perception of the
World at Any Resolution", I need to merge data from Table 2 (multimodal
understanding benchmarks) and Table 4 (video benchmarks).

Specifically:
1. From Table 2, extract each model's scores on: MMMU (val), DocVQA (test),
   ChartQA (test), TextVQA (val), OCRBench, RealWorldQA, MME (sum),
   MMBench-EN (test), MMStar, MMVet, MathVista (testmini).
2. From Table 4, extract each model's scores on: MVBench, PerceptionTest (test),
   EgoSchema (test), Video-MME (without subs).
3. Join these tables by model name. Include: Qwen2-VL-72B, Qwen2-VL-7B,
   Qwen2-VL-2B, GPT-4o, Claude 3.5 Sonnet.
4. After merging, sort by MMMU (val) score in descending order.
5. Save as /workspace/qwen2vl_cross_benchmark.csv with columns: Model, MMMU,
   DocVQA, ChartQA, TextVQA, OCRBench, RealWorldQA, MME, MMBench_EN, MMStar,
   MMVet, MathVista, MVBench, PerceptionTest, EgoSchema, VideoMME.
6. Create a radar/spider chart (/workspace/qwen2vl_cross_benchmark_chart.png)
   comparing Qwen2-VL-72B, GPT-4o, and Claude 3.5 Sonnet on the following
   normalized metrics: MMMU, DocVQA, ChartQA, MathVista, MVBench, EgoSchema.
   Normalize each metric to [0, 100] based on the max value among the three
   models.

Handle missing values with "-" in CSV.
Task category: doc_extraction
I am a researcher reviewing the GLiNER paper on Named Entity Recognition.
Please download and read this PDF: https://arxiv.org/pdf/2311.08526

The paper reports F1 scores in Table 5 for three different negative sampling
ratios, along with the corresponding Precision and Recall values. I want to
verify whether the reported F1 scores are correctly computed from the given
Precision and Recall using the standard F1 formula. Please extract the data
from Table 5, recalculate the F1 scores, and compare them against the
reported values.

Output requirements:
- A CSV file /workspace/gliner_f1_verification.csv with columns:
  Negative_Sampling_Ratio, Precision, Recall, F1_Reported, F1_Calculated,
  Match (Yes/No)
- A comparison visualization /workspace/gliner_f1_verification.png using a
  grouped bar chart showing the reported F1 vs. calculated F1 for each
  negative sampling ratio
Task category: doc_extraction
I am a researcher comparing language models. Please download and read this
PDF: https://arxiv.org/pdf/2405.04434

Please find Table 2. Extract the scores of DeepSeek-V2 and LLaMA 3 70B on
the following 14 English and code benchmarks:
BBH, MMLU, DROP, ARC-Easy, ARC-Challenge, HellaSwag, PIQA, WinoGrande,
RACE-Middle, RACE-High, TriviaQA, NaturalQuestions, HumanEval, MBPP.

Requirements:
1. Create a CSV file /workspace/deepseek_v2_vs_llama3.csv with columns:
   Benchmark, DeepSeek-V2, LLaMA3_70B, Delta
   (Delta = DeepSeek-V2 - LLaMA3_70B)
2. Generate a delta bar chart /workspace/deepseek_v2_vs_llama3_delta.png with
   benchmark names on the X-axis and Delta values on the Y-axis; use blue for
   positive values and red for negative values
Task category: doc_extraction
I am a researcher analyzing video understanding benchmarks. Please download
and read this PDF: https://arxiv.org/pdf/2406.04264

Please find Table 2. Extract the scores of GPT-4o and LLaVA-Onevision on
7 multiple-choice tasks:
TR (Topic Reasoning), AR (Anomaly Recognition), NQA (Needle QA),
ER (Ego Reasoning), PQA (Plot QA), AO (Action Order), AC (Action Count).

Requirements:
1. Create a CSV file /workspace/mlvu_gpt4o_vs_llava_ov.csv with columns:
   Task, GPT-4o, LLaVA-Onevision, Delta
   (Delta = GPT-4o - LLaVA-Onevision)
2. Generate a heatmap /workspace/mlvu_delta_heatmap.png with rows as the two
   models and columns as the 7 tasks; cell values are scores with color
   intensity representing score magnitude
Task category: doc_extraction
I am researching the capabilities of recent RL-based reasoning models.
Please download and read these two PDFs:
- https://arxiv.org/pdf/2501.12948
- https://arxiv.org/pdf/2505.09388

Extract the "Pass@1" accuracy scores for two specific math benchmarks:
"AIME 2024" and "MATH-500". I need this data for four models (thinking
models): DeepSeek-R1, DeepSeek-R1-Zero, Qwen3-235B-A22B, and OpenAI-o1.

Save the data into a file named /workspace/reasoning_math.csv with three
columns: Model, AIME_2024, and MATH_500.

Then, create a 2D Scatter Plot comparing these models, with AIME 2024 on the
x-axis and MATH-500 on the y-axis. Ensure each data point has a text
annotation next to it displaying the model's name. Save the image as
/workspace/math_reasoning_scatter.png.
Task category: doc_extraction
I am compiling a benchmark report on Document AI. Please download and read
these two PDFs:
- https://arxiv.org/pdf/2502.13923
- https://arxiv.org/pdf/2412.10302

Locate the main evaluation tables and extract the test accuracy scores on
three specific benchmarks: DocVQA, ChartQA, and TextVQA, for the following
two models: Qwen2.5-VL-72B and DeepSeek-VL2.

Save the synthesized data into /workspace/doc_ai_metrics.csv with exactly
four columns: Model, DocVQA, ChartQA, and TextVQA.

Next, generate a Horizontal Grouped Bar Chart (bars extending from left to
right) comparing these two models across the three benchmarks. Use a specific
color palette (Hex codes: #1f77b4 for DocVQA, #ff7f0e for ChartQA, #2ca02c
for TextVQA). Ensure the y-axis contains the model names. Save the image as
/workspace/doc_ai_horizontal.png.
Task category: doc_extraction
The container has the following file:
- /workspace/fixtures/2512.17495v2.pdf

I am preparing a presentation on the GroundingME benchmark and I need to
accurately recreate its subtask distribution chart.

Please locate Figure 3 in the paper. Write a Python script named
/workspace/reproduce_fig3.py that fully reconstructs this pie chart. You must
ensure all details are completely consistent with the original figure.

Execute your script to generate the chart and save the final image as
/workspace/subtask_distribution_reproduce.png.
Task category: doc_extraction
The container has the following file:
- /workspace/fixtures/2512.17495v2.pdf

I am preparing a presentation on the GroundingME benchmark and I need to
accurately recreate its performance gain chart.

Please locate Figure 5 ("Performance gain of different models by enabling
thinking mode") in the paper. Write a Python script named
/workspace/reproduce_fig5.py that fully reconstructs this chart. You must
ensure all details are completely consistent with the original figure.

Execute your script to generate the chart and save the final image as
/workspace/figure5_reproduce.png.
Task category: doc_extraction
The container has the following file:
- /workspace/fixtures/2512.17495v2.pdf

I am preparing a presentation on the GroundingME benchmark and I need to
accurately recreate its fine-tuning performance chart.

Please locate Figure 6 ("Out-of-domain performance of fine-tuned
Qwen3-VL-8B-Instruct...") in the paper. Write a Python script named
/workspace/reproduce_fig6.py that fully reconstructs this chart. Ensure all
details are consistent with the original figure.

Execute the script and save the image as /workspace/figure6_reproduce.png.