Title: Can Large Language Models Analyze Graphs like Professionals? A Benchmark, Datasets and Models
Benchmark: ProGraph, LLM4Graph
Summary: Propose datasets. Show that LLMs cannot process graphs well. Use the dataset to boost LLM performance.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: graph processing
Phenomenon defined by authors: Yes
Phenomenon definition: "can LLMs analyze graphs like professionals?"-p2
Definition scope: Comprehensive
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Widely-agreed
Task definition: given a graph question, a model has to use APIs to solve the problem like human experts
Task item definition: a natural language question and a graph
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Crowd-sourced task examples (e.g. Prolific-created tasks), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: 512
Dataset size notes: train: 29,260
Dataset metadata available: Yes
Dataset metadata detail: task category, answer difficulty, question type
Sampling method: Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test, Train
Response format: Free response (e.g. summary paragraph, executable code)
Metric definition: Exact Match (accuracy, F1, precision, recall), LLM-as-a-Judge (text quality, preferences, NOT extracting answers for other metrics)
Metric subscores: Yes
Metric subscores detail: task category, answer difficulty, question type
Metric access: Outputs alone
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: Yes
Author validity detail: the authors conduct human experiment
Taxonomy: NLP / Extraction
New taxonomy: General Application / Extraction