dataesr/scientific-paragraphs-categorization huggingface dataset:
The lack of comprehensive evaluations and practical tests has inadvertently led to a cautious approach, especially in sectors demanding high reliability, like medicine and healthcare. 14  Motivated by the need to better understand mental health sentiments on Chinese social media platforms, our research embarks on a rigorous evaluation of supervised learning and large language models. We offer the following contributions: • We introduce and publicly release two new expertly-manual annotated social media datasets in the mental health domain, specifically focusing on cognitive distortion and suicide risk classification. These datasets not only serve as valuable resources for the community but also have profound real-world implications, potentially informing strategies for suicide prevention and interventions for cognitive distortions.
• We propose a comprehensive benchmark using both traditional supervised learning and large language models on these datasets. By employing a variety of strategies, including zero-shot, few-shot, and finetuning, we seek to determine the most effective methods for leveraging these models in the context of mental health tasks on Chinese social media. • Lastly, our study pioneers the exploration of fine-tuning capabilities of GPT-3. 5, leveraging real-world data. This endeavor seeks to determine the adaptability and specialized performance enhancements possible with the model currently unexplored in the literature.
Text sentiment analysis In the swiftly evolving digital era, social networking platforms have emerged as pivotal channels for expressing emotions globally. These platforms generate vast amounts of unstructured data every second. Accurately and promptly discerning the emotions embedded within this data presents a formidable challenge to computational algorithms. 8  Fu et al. 16  presented a distant supervision method designed to build systems that classify high and low suicide risk levels using Chinese social media data. This approach minimizes the need for human experts of varying expertise levels to perform annotations. By integrating this model with crucial psychological features extracted from user blogs, they attained an F1 score of 77. 98%. Singh et al. 17  employed a BERTbased model for sentiment analysis on tweets sourced globally and another dataset specifically from India, both focusing on the topic of COVID-19.
They reported achieving an accuracy of 94%. Wan 18 introduced a method for sentiment analysis of comments on Weibo platforms, leveraging deep neural networks. The data undergoes feature extraction through multilevel pooling and convolution layers. Comments are preprocessed and transformed into text representations using the word2vec algorithm. Subsequently, key features are extracted from the feature matrix using a CNN. For the final classification and sentiment analysis, the softmax logistic regression method is employed. Zhang et al. 19  explored the correlations among emotion labels, social interactions, and temporal patterns within an annotated Twitter dataset. They introduced a factor graph-based emotion recognition model that seamlessly integrates these correlations into a unified framework.
This model adeptly identifies multiple emotions by applying a multi-label learning approach to Twitter datasets. Wang et al. 20  introduced a topic modeling technique, termed LDA, to examine the primary concerns expressed on Weibo during the COVID-19 pandemic. They assessed the emotional inclinations of these topics, determined their proportional distributions, and conducted user behavior analysis based on metrics such as likes, comments, and retweets. Furthermore, they explored shifts in user concerns and variations in engagement among residents from different regions of mainland China. Such insights guide public sentiment and actions during health emergencies, emphasizing the importance of vigilant social media monitoring. Although deep learning algorithms typically demonstrate impressive results, they often require a significant volume of labeled data to perform optimally.
The distant supervision approach highlighted in Fu et al. 's research  16  aims to reduce the need for labeling, but it still requires the involvement of three different expert groups at various expertise levels to yield desired results. Nonetheless, when applying these models to new datasets or tasks, domain adaptation issues often arise. These trained models can see a decline in their efficacy, making deep learning algorithms both costly and inflexible. Given these hurdles, there's a growing demand for efficient and user-centric methods to assist individuals in emotion detection on social media platforms. The recent advancements in large language models present a potential solution to this challenge, but their precise impact still warrants examination from multiple perspectives and specialists.
Large language model and its applications in medical domain The advent of Large Language Models (LLMs), such as OpenAI's ChatGPT,  12  has revolutionized the field of natural language processing. 21  These LLMs demonstrate emergent abilities that significantly outperform those of their smaller, pre-trained models. 22  Initially conceived for understanding and generating human-like text, LLMs have found diverse applications ranging from content generation, 23 medical report assistant,  24  coding assistance,  25  education,  26  and answering medical related questions. 27  The sheer scale of these models enables them to generate complex, contextually relevant content. LLMs have garnered significant attention in medical domain. 14  For instance, Jiang et al. 28  developed a clinical LLM named NYUTron to assist physicians and healthcare administrators in making time-sensitive decisions.
The absence of human oversight could lead to biased outcomes, thereby posing potential risks to users. Additionally, if this system were to become a foundational diagnostic tool for future psychological counseling, issues related to user privacy could become a point of concern. Chen et al. 31  developed a tool designed to improve the realism of psychiatristpatient simulations using ChatGPT-based chatbots. Their approach involved using distinct prompts to enable large language models (LLMs) to emulate the roles of both a depressed patient and a psychiatrist. The study confirmed the feasibility of utilizing ChatGPT-driven chatbots in psychiatric contexts. However, the research also acknowledged limitations: individual patients and counselors have unique communication styles, and some patients may be reluctant to engage in conversation.
These nuances present a challenge for achieving truly realistic simulations with ChatGPT. Addressing the simulation of diverse personalities in a meaningful way remains a key area for further investigation. Fu et al. 32  developed a counseling support system designed to augment the capabilities of non-professional counselors. The system provides multiple features, including mental health analysis, evaluation of therapist responses, and suggested interventions. This application serves as a valuable use case for language models in the mental health sector. Ten professional psychologists assessed the system on five critical dimensions, and the findings were favorable, with a 78% expert approval rate indicating that the system can deliver effective treatment strategies. Ayers et al. 33  developed a ChatGPT-based chatbot and compared its responses with those of physicians to patient inquiries on a social media forum.
Xu et al. 13  present a pioneering evaluation of multiple Large Language Models (LLMs) across various mental health prediction tasks using four publicly available online text datasets. Their insights offer guidance to practitioners on optimizing the use of LLMs for specific applications. While their research stands as a monumental verification of LLMs' potential in the mental health domain, it is noteworthy that their datasets are exclusively in English and do not address multi-label classification tasks. Yang et al. 34  assessed ChatGPT's capabilities in mental health analysis and emotional reasoning by evaluating its performance on 11 datasets across five tasks. The study also investigated the impact of different emotion-based prompting strategies. Experimental results indicate that while ChatGPT surpasses traditional neural network-based approaches, it still lags behind more advanced, task-specific methods.
Nevertheless, ChatGPT demonstrates significant potential in the area of explainable mental health analysis. In conclusion, while the integration of LLMs in medicine presents compelling prospects, there's an imperative to ensure privacy and uphold ethical standards. Responses generated may not always be flawless. 35  Particularly in mental health, relying solely on LLM-driven systems for diagnosis or support introduces numerous unpredictable variables. It's crucial to recognize that LLMs warrant meticulous scrutiny and validation. 36  Evaluation should be considered an essential discipline to facilitate the more effective development of large language models (LLMs). 37.
Large language models Given that our data is in Chinese, we explored the open-source models ChatGLM2-6B and GLM-130B,  40  both of which support Chinese language processing. The primary distinction between these two models lies in the number of parameters they possess. GPT-3. 5 41 stands as a flagship large-scale language model. We experimented with various prompt word constructions and sought to integrate prior knowledge from the psychological domain, along with the most recent public fine-tuning functionalities. GPT-4, 42 being the latest iteration, was also included in our assessment. Detailed introduction on these models are provided in the subsequent sections. • ChatGLM2-6B: ChatGLM2-6B is an open-source bilingual language model with 6. 2 billion parameters, optimized for Chinese question-answering and dialogue.
It employs similar technology to ChatGPT and is trained on roughly 1TB of Chinese and English text data. The model can be fine-tuned through various techniques like supervised learning and human feedback. It also features an efficient tuning method based on P-Tuning v2, requiring at least 7GB of GPU memory for customization. Due to quantization techniques, it can run on consumer-grade graphics cards with only 6GB of memory. • GLM-130B: GLM-130B is a bilingual pre-trained language model optimized for both English and Chinese, boasting a substantial 130 billion parameters. This model aims to provide an open-source alternative of a scale comparable to GPT-3, while shedding light on the complexities of training such large-scale models. Impressively, GLM-130B surpasses GPT-3 175B on multiple English benchmarks and outperforms ERNIE TITAN 3.
0 260B,  43  the largest existing Chinese language model, on relevant benchmarks. A distinctive feature of GLM-130B is its capability for INT4 quantization without substantial performance degradation, thus facilitating efficient inference on widely available GPUs. • GPT-3. 5: GPT-3. 5 is a cutting-edge language model developed by OpenAI, designed to offer enhanced conversational capabilities. Building on the foundation of its predecessor, GPT-3, this iteration introduces improvements in both performance and cost-efficiency. OpenAI's commitment to refining and advancing the capabilities of their models is evident in GPT-3. 5, which provides users with a more coherent, contextaware, and responsive conversational experience. As part of OpenAI's mission to ensure that artificial general intelligence benefits all of humanity, GPT-3.
Datasets and Evaluation Metrics We undertook two psychology-related classification tasks: suicide risk and cognitive distortion. The suicide risk task primarily differentiates between high and low suicide risks, while the cognitive distortion task focuses on classifications defined by Burns. 44  We sourced our data by crawling comments from the "Zoufan" blog within the Weibo social platform. Subsequently, a team of qualified psychologists were enlisted to annotate the data. Given that this data is publicly accessible, there are no concerns related to privacy breaches. For the suicide detection data, there were 648 records with low suicide risk and 601 records with high suicide risk. The dataset for cognitive distortion consists of a total of 910 entries. The classification labels employed for this data are as follows: all-or-nothing thinking, over-generalization, mental filter, disqualifying the positive, mind reading, the fortune teller error, magnification, emotional reasoning, should statements, labeling and mislabeling, blaming oneself and blaming others.
For both sets of data, the training set and test set are divided according to the ratio of 4:1. The statistics of these two datasets are listed in Table  1 , where N train and N test denote the number of training and test samples, respectively. L is the total number of classes, L is the average number of labels per sample, and W is the average number of words per sample. We utilize three evaluation metrics to measure the performance of different algorithms for our two tasks: precision, recall, and F 1 score. Precision is the ratio of correctly predicted positive observations to the total predicted positives and recall (or sensitivity) represents the ratio of correctly predicted positive observations to all the actual positives. These two metrics provide a comprehensive view of the algorithm's performance in terms of its positive predictions.
DATASET AND CODE AVAILABILITY The experimental texts for our study are sourced from comments on a Sina Weibo post by the user "Zoufan," which can be viewed at: https://www. weibo. com/xiaofan116?is_all=1. An expert-annotated dataset for the study of cognitive distortions and suicide risk, along with the prompt of the large language model and the supervised learning model code, are now available at: https://github. com/HongzhiQ/SupervisedVsLLM-EfficacyEval. Here are the models mentioned earlier along with their corresponding source code and online demo links: • ChatGLM2-6B: -Source code: https://github. com/thudm/chatglm2-6b -Unofficial demo: https://huggingface. co/spaces/mikeee/chatglm2-6b-4bit • GLM-130B:.
A custom deployment method was therefore adopted using a negatively buoyant frame manually lowered with a horizontal telescoping mast and a block pulley. Deploying the sensors into the direction of the sun in highly turbid waters (photons mean free path of less than 1 m) avoided any optical pollution from the small ship hull. The processing of the radiometric data was conducted according to recent protocols implemented in an open source package in R available at https://github. com/belasi01/Cops  (Antoine et al. , 2013; Bélanger et al. , 2017). Due to the highly turbid waters, the so-called self-shadow correction was not estimated using the well-established Gordon and Ding 1992 method, but was rather estimated using Monte Carlo simulations based on SimulO  (Leymarie et al. , 2010; Doxaran et al.
Inherent optical properties To measure inherent optical properties (IOP) in the water column, an optical package was deployed during all four legs of the expedition. During Leg 1, the optical package included a very sensitive depth sensor (RBR model Virtuoso), a particle size meter (Sequoia Scientific, model LISST-100x), as well as a data logger (Seabird Scientific, model DH4), all fitted in a cylindrical metallic frame that was manually lowered through an auger hole in the ice using a tripod and a hoisting system. During Legs 2, 3 and 4, the optical package included a different secondary CTD (RBR Concerto), the same particle size meter and data logger, 2 back-scattering meters (a HydroScat-2 from HobiLabs and a BB3 from Seabird Scientific), and a fluorescence meter (FLBBCD from Seabird Scientific).
, 2021). Sequences processing was performed following the R package DADA2 pipeline version 1. 16  (Callahan et al. , 2016)  with R software (version 4. 0. 5;  R Core Team, 2014). Taxonomic assignment was performed using UNITE  (version 8. 3; Nilsson et al. 2019)  databases. Metadata related to fungal abundance and diversity can be found in Table  2. DNA sequences were deposited in the National Center for Biotechnology Information (NCBI) Sequence Read Archive (SRA) under accession number PRJNA822885.
As far as we know, the data gathered and currently included in this paper are unparalleled in the breadth of their spatio-temporal coverage of a fluvial-to-marine transitional region in the Arctic. The scope of the dataset presented here offers contingencies towards its further exploitation in research. 412, 443, 490, 510, 551, 555, 560, 565, 620, 645, 667, 673, 683, 709, 745    Detailed metadata are associated with each file including the principal investigator's contact information. For specific questions, please contact the principal investigator associated with the data (see Table  2 ). Information about code availability. The code used to process the figures and tables is publicly available (https://github. com/PMassicotte/nunataryuk_data_paper) under the 535 MIT license. Earth System Science Data Discussions Preprint.
Funding sources Financial support for this project was provided by the  European Research and Innovation programme Horizon 2020  to the  Nunataryuk project  (Grant agreement no. 773421 ), by the  Network of Centers of Excellence of Canada ArcticNet  ( P66 -Nunataryuk), by  Québec-Océan  funded through the  Fonds de Recherche du Québec -Nature et Technologies , by the  Sentinel North initiative  funded through the  Canada First Research Excellence Fund , by the  Aurora Research Institute -Aurora College , and through the leadership of the  Canada Excellence Research Chair in Remote Sensing of Canada 's New  Arctic Frontier. The  Department of Fisheries and Oceans Canada  supported the project through the  Arctic Science Funds program. Fungal and bacterial abundance analyses were supported by the  RESTORE  project funded by the  French National Research Agency  (contract n •  ANR-19-CE32-0013 ).
juhls@awi. de Available https://doi. pangaea. de/10. 1594/PANGAEA. 937587 carbon (DOC) Oxidation (Shimadzu TOC-VCPN) Water Fluorescent dissolved x x x x RU Aqualog fluorescence spectrometer Atsushi Matsuoka atsushi. matsuoka@takuvik. ulaval. ca Available https://doi. pangaea. de/10. 1594/PANGAEA. 937587 organic matter A Coble-peak (FDOM A-peak) Water Fluorescent dissolved x x x x dimensionless Aqualog fluorescence spectrometer Atsushi Matsuoka atsushi. matsuoka@takuvik. ulaval. ca Available https://doi. pangaea. de/10. 1594/PANGAEA. 937587 organic matter Biological index (FDOM BIX) Water Fluorescent dissolved x x x x RU Aqualog fluorescence spectrometer Atsushi Matsuoka atsushi. matsuoka@takuvik. ulaval. ca Available organic matter C Coble-peak (FDOM C-peak) https://doi. pangaea. de/10. 1594/PANGAEA.
pangaea. de/10. 1594/PANGAEA. 937587 organic matter Humification index (FDOM HIX) Water Fluorescent dissolved x x x x RU Aqualog fluorescence spectrometer Atsushi Matsuoka atsushi. matsuoka@takuvik. ulaval. ca Available https://doi. pangaea. de/10. 1594/PANGAEA. 937587 organic matter M Coble-peak (FDOM M-peak) Water Fluorescent dissolved x x x x RU Aqualog fluorescence spectrometer Atsushi Matsuoka atsushi. matsuoka@takuvik. ulaval. ca Available https://doi. pangaea. de/10. 1594/PANGAEA. 937587 organic matter T Coble-peak (FDOM T-peak) Water Intensity fluorescent x x x x RU Aqualog fluorescence spectrometer Atsushi Matsuoka atsushi. matsuoka@takuvik. ulaval. ca Available https://doi. pangaea. de/10. 1594/PANGAEA. 937587 dissolved organic matter component 1 (FDOM C1) Water Intensity fluorescent x x x x RU Aqualog fluorescence spectrometer Atsushi Matsuoka atsushi.
matsuoka@takuvik. ulaval. ca Available https://doi. pangaea. de/10. 1594/PANGAEA. 937587 dissolved organic matter component 2 (FDOM C2) Water Intensity fluorescent x x x x RU Aqualog fluorescence spectrometer Atsushi Matsuoka atsushi. matsuoka@takuvik. ulaval. ca Available https://doi. pangaea. de/10. 1594/PANGAEA. 937587 dissolved organic matter component 3 (FDOM C3) Water Absorption NA NA x x 1/m Measured with an AC9 sensor David Doxaran / Guislain doxaran@obs-vlfr. fr / In progress NA coefficient (a) at 440, (WetLabs/Seabird Scientific) Bécu guislain. becu@takuvik. ulaval. ca 555, 630, 715, 730, installed within an optical frame 750, 767, 820, 870 (profiling) nm Water Beam attenuation NA NA x x 1/m Measured with an AC9 sensor David Doxaran / Guislain doxaran@obs-vlfr. fr / In progress NA coefficient (c) at 440, (WetLabs/Seabird Scientific) Bécu guislain.
matsuoka@takuvik. ulaval. ca Available https://doi. pangaea. de/10. 1594/PANGAEA. 937587 carotenoids chromatography (HPLC) concentration Water Photosynthetic x x x x mg/m 3 High performance liquid Atsushi Matsuoka atsushi. matsuoka@takuvik. ulaval. ca Available https://doi. pangaea. de/10. 1594/PANGAEA. 937587 carotenoids chromatography (HPLC) concentration Water Photsynthetic x x x x mg/m 3 High performance liquid Atsushi Matsuoka atsushi. matsuoka@takuvik. ulaval. ca Available https://doi. pangaea. de/10. 1594/PANGAEA. 937587 pigments chromatography (HPLC) concentration Water Prasinoxanthin x x x x mg/m 3 High performance liquid Atsushi Matsuoka atsushi. matsuoka@takuvik. ulaval. ca Available https://doi. pangaea. de/10. 1594/PANGAEA. 937587 concentration chromatography (HPLC) Water Total accessory x x x x mg/m 3 High performance liquid Atsushi Matsuoka atsushi.
16S rRNA gene sequencing and sequence data pre-processing Extracted DNA samples for 16S rRNA sequencing were prepared following a dual barcoded two-step PCR procedure for amplicon sequencing for Illumina. The V4 region was sequenced by generating 2 x 300 bp on an Illumina MiSeq. Data was demultiplexed and filtered for quality and remaining primers according to recommended practices as described previously  [7].
16S rRNA data analysis Illumina MiSeq forward and reverse reads were processed using the MASQUE pipeline (https://github. com/aghozlane/masque). Briefly, raw reads are filtered and combined followed by dereplication (dereplication is the identification of unique sequences so that only one copy of each sequence is reported) Chimera removal and clustering are followed by taxonomic annotation of the resulting OTUs by comparison to the SILVA database. A BIOM file is generated that combines both OTU taxonomic assignment and the number of matching reads for each sample. Relative abundance levels for bacterial taxa (family/genus level) were obtained and analyzed  [7].
This set of 87 genes constituted what we named the targeted resistome (Supplementary Table  1 ). We also included primers targeting 16S rDNA. Primer design and validation prior to and after Biomark analysis has been done as described earlier  [7, 35]. Real-Time PCR analysis was performed using the 96. 96 BioMark™ Dynamic Array for Real-Time PCR (Fluidigm Corporation, San Francisco, CA, U. S. A) as described previously  (7). Thermal cycling and real-time imaging were performed at the Plateforme Génomique GeT -INRA Transfert (https://get. genotoul. fr/en/), and Ct values were calculated using the BioMark Real-Time PCR analysis software. Calculations for normalized and cumulative abundance of individual genes and allocated gene classes were performed as previously described  [7, 32, 35]. The resistome data displayed here and used for statistical analysis is based on resistome data that was obtained for two biological replicates per sample.