Title: A Comprehensive Study on Text-attributed Graphs: Benchmarking and Rethinking
Benchmark: CS-TAG
Summary: Propose a dataset. Conduct extensive benchmarking experiments on a wide range of models. Propose topological training.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: text-attribute graph processing
Phenomenon defined by authors: Yes
Phenomenon definition: "In many real-world graphs, nodes are often associated with text attributes, giving rise to the text-attributed graphs (TAGs)"-p1
Definition scope: Comprehensive
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Widely-agreed
Task definition: understanding graph topology of text-attributed graphs
Task item definition: a graph
Task source: Real task examples (e.g. GitHub issues), Modified from another benchmark (e.g. translation into another language)
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: 276,661 nodes, 2,877,927 edges
Dataset size detail: see details in appendix B.3
Dataset size notes: train: 1,993,101 nodes, 16,735,860 edges; val: 120,564 nodes, 1,225,530 edges
Dataset metadata available: Yes
Dataset metadata detail: topic area
Sampling method: Random sample (creators defined a task space and sampled from it), Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test, Train, Validation
Response format: Multiple choice
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric subscores: Yes
Metric subscores detail: topic area
Metric metascoring: Hits@K
Metric access: Outputs alone
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: NLP / Extraction
New taxonomy: General Application / Extraction