Title: Can Language Models Solve Graph Problems in Natural Language?
Benchmark: NLGraph
Summary: Propose a dataset. Evaluate LLMs with different prompting approaches. Propose new approaches to boost LLM performance.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: graph processing
Phenomenon defined by authors: Yes
Definition scope: Comprehensive
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Widely-agreed
Task definition: Reasoning with graphs and structures
Task item definition: A graph description and a corresponding question.
Task source: Procedurally-generated task examples (e.g. Creating instances from a template)
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: 5,902 problems in a standard version and 29,370 problems in an extended version
Dataset metadata available: Yes
Dataset metadata detail: difficulty level, question type
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test
Response format: Short free response (e.g. single word or number), Free response (e.g. summary paragraph, executable code)
Metric definition: Exact Match (accuracy, F1, precision, recall), partial credit
Metric subscores: Yes
Metric subscores detail: question type, difficulty level
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: Yes
Author validity detail: the dataset is generated by reliable procedure
Taxonomy: NLP / Extraction
New taxonomy: General Application / Extraction