Title: DTGB: A Comprehensive Benchmark for Dynamic Text-Attributed Graphs
Benchmark: DTGB
Summary: Propose a dataset. Benchmark popular algorithms with this dataset and showcase the limitations of current models in handling dynamic text-attributed graphs.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: dynamic text-attributed graph processing
Phenomenon defined by authors: Yes
Phenomenon definition: "In those dynamic graphs, nodes and edges are typically associated with text attributes, giving rise to dynamic text-attributed graphs (DyTAGs)."-p1
Definition scope: Comprehensive
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Widely-agreed
Task definition: Given a graph, a model has to answer relevant question about the graph
Task item definition: graph and time stamp
Task source: Real task examples (e.g. GitHub issues), Modified from another benchmark (e.g. translation into another language)
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: edge: 2,637,689, node: 554,432
Dataset size detail: counts are estimated based on Table 1
Dataset size notes: train: edge: 12,309,214, node: 2,587,353; val: edge: 2,637,689, node: 554,432
Dataset metadata available: Yes
Dataset metadata detail: topic area
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test, Train, Validation
Response format: Multiple choice, Free response (e.g. summary paragraph, executable code)
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric subscores: Yes
Metric subscores detail: topic area
Metric metascoring: hits@k
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: NLP / Extraction
New taxonomy: General Application / Extraction