Title: TEG-DB: A Comprehensive Dataset and Benchmark of Textual-Edge Graphs
Benchmark: TEG-DB
Summary: Present a large-scale dataset. Develop a pipeline for relevant research. Benchmark existing models on the dataset.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: textual-edge graph processing
Phenomenon defined by authors: Yes
Phenomenon definition: "A Textual-Edge Graph (TEG) is a graph-structured data format in which both nodes and edges have free-form text descriptions."-p4
Definition scope: Comprehensive
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Widely-agreed
Task definition: Given a textual-edge graph, a model has to process it and answer relevant questions
Task item definition: A graph and a question
Task source: Modified from another benchmark (e.g. translation into another language)
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: Total Nodes: 2,164,239  Total Edges: 10,579,752  Total Nodes-Class: 1,053
Dataset metadata available: Yes
Dataset metadata detail: topic area
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test
Response format: Free response (e.g. summary paragarph)
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric subscores: Yes
Metric subscores detail: subsets of topic areas
Metric access: Outputs alone
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: NLP / Extraction
New taxonomy: General Application / Extraction