Title: Still Not Quite There! Evaluating Large Language Models for Comorbid Mental Health Diagnosis
Benchmark: ANGST (ANxiety-Depression Comorbidity DiaGnosis in Reddit PoST)
Summary: ANGST is a benchmark for depression-anxiety comorbidity classification from social media posts. The dataset has multi-class labeling for anxiety, depression, both, or none, and the samples are neutrally seeded from Reddit and human-annotated by expert psychologists. Additionally, the paper presents ANGST-SILVER, a more extensive and silver-labeled dataset by GPT-3.5-turbo to support few-shot learning or supervised fine-tuning. 
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: depression-anxiety comorbidity classification
Phenomenon defined by authors: Yes
Phenomenon definition: Depression-anxiety comorbidity is the phenomenon of depression and anxiety manifesting concurrently, and requiring a dual diagnosis/multiple labels of depression and anxiety. 
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Widely-agreed
Task definition: The benchmark supports three classification tasks: multi-label classification of a Reddit post as showing anxiety, depression, comorbid (both), or control (none), and two binary classification tasks to identify a post as exhibiting depression or non-depression, and anxiety or non-anxiety. 
Task item definition: A single item would be a Reddit post and its label as anxiety, depression, comorbid (both), or control (none).  
Task source: Real task examples (e.g. GitHub issues)
Task ecology: Partial real task (e.g. answering medical questions collected from real people), Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Task ecology detail: The benchmark utilizes publicly available Reddit data, so the task chooses a "diagnosis" based upon data from real people. However, the data has been heavily filtered from mental-health-related subreddits, so the benchmark is somewhat constructed or artificial. 
Dataset size: ANGST: 2876, ANGST-SILVER: 7667
Dataset metadata available: No
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Multiple choice
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Weighted Mean
Metric statistics: Weighted Precision, Recall, F1 scores, and macro-F1 scores for binary and multi-class classification. Hamming loss is also reported for multi-class classification. 
Metric subscores: Yes
Metric subscores detail: Depression vs Control, Anxiety vs Control
Metric access: Outputs alone
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: No, link is broken
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: Yes
Author validity detail: The authors compared the construction of ANGST against SDCNL, Depression Reddit, Dreaddit, and DATD. They measured the inter-class similarity of each benchmark by Jensen-Shannon Divergence (JSD) and Maximum Mean Discrepancy (MMD), and found that ANGST had the lowest pairwise JSD, indicating that ANGST is more challenging to classify, and thus more representative of the minute but vital differences between anxiety and depression. The authors also compared the data drift of ANGST against the other benchmarks, calculated by accuracy, macro-F1, ROC_AUC scores, and Matthews Correlation Coefficient. The results are between 0.904 and 1.0 for ROC-AUC, and 0.990 and 0.875 for MCC, indicating a distinct and inherent difference from existing datasets, claimed to result from its meticulous data curation and gold annotation scheme. 
Taxonomy: Mental Health / 
New taxonomy: Specific Application / Social Sciences