dataesr/scientific-paragraphs-categorization huggingface dataset:
In the current study, we introduce a novel workflow that employs LLMs to prioritize pools of candidate genes, such as those forming the A37 modules (Fig. 1C ). We initiated the development of this prioritization workflow by benchmarking four LLMs: OpenAI's GPT-3. 5 and GPT-4, Google's Bard, and Anthropic's Claude. Two primary benchmarking tasks were emphasized: (1) identifying functional convergences within the candidate pool to elucidate relationships and biological themes, a staple in interpreting gene lists from transcriptomics, and (2) scoring individual candidates against specific relevance criteria, such as potential utility as a blood biomarker or significance in immune cell biology. The overarching goal was to utilize LLMs for the initial prioritization of candidate genes. Those ranking highest would subsequently undergo rigorous evaluation via our previously described gene-centric workflow  [30, 31]  (Fig.
5, GPT-4, Claude); SLC4A1 (GPT3. 5, GPT-4, Claude, DAVID); GYPB (GPT3. 5, GPT-4); GYPE (GPT3. 5, GPT-4); XK (GPT-4, DAVID). PubMed queries using the official gene symbols AND "erythrocytes OR erythroid OR erythropoiesis" confirmed these associations. GPT-4 performed best, correctly associating 8 genes with this central theme, only missing CA1 identified uniquely by GP-T3. 5. GPT-3. 5 made 7 associations, while Claude made 3 associations. We next tested the models' ability to generate simple network visualizations, a common method for representing functional associations among gene lists. Since the models cannot directly generate images, we prompted them to output R scripts to visualize a network with nodes for themes and genes, and edges for functional associations between them. All four models generated scripts, but only GPT-4's ran error-free and produced the requested network visualization (Fig.
However, in most instances supporting statements could be "manually" linked to actual references. Verifying all score justifications would be too time-consuming at this stage, but we did do this systematically in subsequent steps when focusing on the top five candidates for this module, as will be describe in detail below. For the next statement ("b. The gene is currently being used as a biomarker in clinical settings. "), Bard appeared to show greater convergence with the other models relative to the previous statement (Fig. 3B ). However, GPT3. 5 this time produced overall much lower scores than the other three. We performed correlation analyses to quantify withinand across-model consistency for all six statements (Fig. 4 ). Overall, we found good level of agreement between GPT-4 and Claude, with high degree of intramodel consistency.
The prompt used this time was: "Could you fact-check and provide a couple of supporting references for this statement? ". This was repeated for each statement and verifications were performed "manually" to ensure that the references were real and contained the relevant information. The humancurated references were then added to the summary. Implementing this workflow for module M9. 2, our illustrative use case, yielded the following material, which was entirely generated by GPT-4, with supporting references added manually (Step 5): -Function: "The BCL2L1, ALAS2, SLC4A1, CA1, and FECH genes play crucial roles in cellular apoptosis regulation, heme biosynthesis, acid-base balance, and overall erythrocyte physiology. The BCL2L1 gene encodes a critical apoptosisregulating protein in the Bcl-2 family that exists as Bcl-xL and Bcl-xS isoforms with opposing impacts on cell survival  [35, 36].
Selection of a top candidate gene based on summarized justifications We next endeavored to select a top candidate from each module. We initially envisioned this task to be performed by the researchers, based on the information generated and summarized by the LLMs. However, we found that GPT-4 and Claude acquitted themselves well of this task and could provide elaborate arguments to justify their choice. Specifically, we requested from GPT-4 and Claude to each select their top candidate gene for module M9. 2 based on the fact-checked summarized justifications generated in Step 5, taking the six scoring criteria into account (Step 6). For this task, we compared the model's performances when inputting either the fact-checked summaries generated earlier in step 5 or the list of 30 candidate genes forming module M9.
Leveraging reference transcriptome datasets for the final candidate selection For module M9. 2, GPT-4 and Claude selected different candidates from the top 5 ranked genes. Given this discrepancy, we proceeded to examine if the models could utilize transcriptome profiling data to refine the prioritization of leading candidates (Step 7). We focused more specifically on three datasets, which measured transcript abundance in isolated cell populations, thus permitting to assess the restriction of the expression of the top 5 candidates. The first dataset profiling transcript abundance across a wide range of hematopoietic cells and precursors (GSE24759,  [21] ), is available from our developmental immunology GXB collection (http:// devel opmen talim  Being a known drug target: BCL2L1 is significant as a target in cancer therapy, with strategies being developed to inhibit its anti-apoptotic function Being a known drug target: Although not a direct target for drugs, ALAS2 is the target of gene therapy for X-linked sideroblastic anemia, demonstrating therapeutic relevance GPT-4 on criterion 6 Therapeutic relevance for immune mediated diseases: BCL2L1 holds potential therapeutic value for immune-mediated diseases due to its role in regulating apoptosis in lymphocytes Therapeutic relevance for immune mediated diseases: While not directly relevant to immune-mediated diseases, the role of iron and heme metabolism in immune responses and inflammation indirectly ties ALAS2 to immune disease processes.
org/ dm3/ geneB rowser/ show/ 40000 26,  [22] ). The second dataset measuring transcript abundance in monocytes, neutrophils, B-cells, CD4 + T-cells, CD8 + T-cells and natural killer (NK) cells and in whole blood (GSE6042459,  [18] ). It is accessible via the "CD2K" GXB collection (http:// cd2k. gxbsi dra. org/ dm3/ geneB rowser/ show/ 40000 98). For additional perspective, a third transcriptome dataset, which measured transcript abundance in whole blood in patients with a wide range of pathological or physiological states was also accessed (GSE100150). This collection of reference datasets was employed for the development of our BloodGen3 module repertoire and has been described in detail previously  [17]. The data is available via the CD2K GXB instance (http:// cd2k. gxbsi dra. org/ dm3/ geneB rowser/ list): "A Transcriptome Fingerprinting Assay for Clinical Immune Monitoring" datasets] and BloodGen3 app (https:// drinc hai.
shiny apps. io/ Blood Gen3M odule/). We first retrieved the transcriptional profiles of BCL2L1, ALAS2, SLC4A1, CA1 and FECH. Abundance profiles of those five genes across the comprehensive set of hematopoietic cells and precursors assembled by Novershtern et al. indicated a high level of restriction of their expression in CD71 + GYPA + erythroid cell populations (ERY3-5 populations) (Fig. 6A ). As we have reported previously, it is a characteristic that is common to most genes constituting A37 modules. However, notably, BCL2L1 displayed overall much lower abundance levels in comparison to the other four genes. In the second reference dataset from Speake et al. , high levels of expression were detected in whole blood for BCL2L1, SLC4A1 and ALAS2, in comparison to isolated leukocyte populations (Fig.
6B ). This is consistent with these genes being expressed by circulating erythroid cells which are present in whole blood but would be excluded upon isolation of various leukocyte populations. Notably, abundance levels observed in whole blood were considerably higher for ALAS2 and to some extent BCL2L1, when compared to SLC4A1, FECH and CA1 levels. The whole blood transcriptome profiles from the BloodGen3 dataset confirmed at the level of module M9. 2 that the overall abundance of the genes constituting this module was increased most prominently in the whole blood of subjects with metastatic melanoma, in pregnant women, followed by patients with acute respiratory syncytial virus (RSV) infection and liver transplant recipients (Fig. 7A ). Decreased abundance was most prominently observed in patients with multiple sclerosis (MS), HIV infection, Chronic Obstructive Pulmonary Disease (COPD), Juvenile Dermatomyositis (JDM) and Influenza virus infection.
Integrating this information into the LLM-driven prioritization and selection process posed a challenge, especially in ensuring that the models received both the data and the essential contextual metadata for interpretation. Given that the main interface for generative LLMs is text we endeavored to convert expression tables into textual input. For this a concatenating function was utilized to automatically generate sentences such as: "In Human Neutrophils, RNA sequencing detected BCL2L1 RNA at a count of 34", which encapsulates both measurements and parameters such as, measurement method, species, or sample types. Another such generated sentence for the Altman et al. dataset was: "In Human Whole blood, the abundance of BCL2L1 RNA measured by Microarrays differed in patients with acute influenza infection compared to controls by a Log2 fold change of -0.
3697682". Here we adopted a multi-tier prompting strategy which permitted to factor in altogether the fact-checked summarized justifications obtained earlier in step 5 and the textualized expression values from the three reference datasets. Five separate prompts were designed to carry out this task (see methods for details). In response to the first prompt, which provided the fact-checked summarized justifications from step 5 as input, the results obtained were similar to those reported earlier for Step 6, with GPT-4 selecting BCL2L1 as its top pick and Claude selecting ALAS2. These selections were maintained despite additional guiding contextual information being provided in the prompt: "Based on the summary below, please recommend the top candidate gene to include in a targeted blood transcriptional profiling panel.
The intent is for the selected gene to be representative of an erythroid cell signature associated with response to mRNA vaccines, severity of RSV infection and that found to be expressed at high levels in patients with metastatic melanoma and in liver transplant recipients. Please weigh the following criteria equally in your recommendation: [etc. …]". GPT-4 and Claude's full response to this and subsequent prompts can be found in Table  5. The second prompt provided as input the textualized transcriptional profiling data from the Speake et al. dataset. At the end of the prompt the following request was made: "Given this additional RNA-seq expression data, does your recommended gene selection remain the same? Please explain which gene you would recommend at this stage and why, incorporating both the summarized information provided earlier and RNA-seq data in your rationale".
" We in turn attempted to input textualized transcriptional profiling data from the Novershtern et al. and Altman et al. datasets. We were not successful with GPT-4, given limits imposed on the amount of information it can receive as input (8192 tokens), which was exceeded by the Novershtern dataset. Breaking the input down in multiple prompts was unsuccessful as GPT-4 lost the thread of the conversation as a result and failed to factor in the summarized justification and the earlier RNAseq dataset in its selection of a top candidate. This was also the case when the Novershtern et al. dataset was omitted and only the Altman dataset provided (Table  5 ). No such issues were however encountered with Claude, which supports 100,000 tokens of context. We were thus successful in inputting transcriptional profiles for all three datasets and completed all the prompts (Table  5 ).
) Fig. 7  Transcriptional profiles of the top 5 scoring candidate genes across 16 reference blood transcriptome datasets. A The box plot shows the changes in abundance of transcripts comprising the M9. 2 module, expressed within a cohort as the percentage of genes showing differential expression between each patient and the respective set of control subjects for this cohort. Differential expression is based on a fixed cut-off (Log2 Fold Change > 1. 5, difference  > 10). These values are expressed as % module activity and can range from + 100% (all genes are upregulated) to -100% (all genes are down regulated). B Box plots show expression levels for the top 5 scoring M9. 2 genes in patient cohorts relative to respective control subjects matched for demographics variables. The datasets were contributed by Altman, Rinchai et al.
Hence, these interpretations should be considered as hypotheses that need further validation arthritis, and melanoma that were highlighted as being relevant, (4) Established utility as a therapeutic target in erythroid disorders like sideroblastic anemia, (5) Potential as a biomarker for disrupted erythropoiesis based on its tight regulation. The other genes (BCL2L1, SLC4A1, CA1, FECH) are more broadly expressed across immune cells and their fold changes are smaller in magnitude or less consistent across key disease states. Each additional dataset provided further evidence supporting ALAS2 as the top choice over other candidates based on erythroid specificity and associations with conditions of interest. In summary, the collective evidence strongly supports ALAS2 as the top gene for inclusion in the intended blood profiling panel to represent an erythroid signature.
The key factors are its erythroid-restricted expression and marked regulation across relevant conditions. ". The final conclusive statement generated by Claude is notable in the balance it strikes between the information provided that was derived from knowledge about candidate genes significance and from relevant transcriptional profiling datasets. Overall, the findings obtained in this use case demonstrate the capabilities of LLMs in aiding with rationale target prioritization and selection, with minimal human input. The workflow described above has been implemented in other modules by participants in the data mining workshop. As a result, we were able to parallelize this process and proceed with the selection of candidates from the 11 A37 modules (Table  2 ). Prioritization reports are available as supplementary material (Additional File 2, Additional File 3, Additional File 4, Additional File 5, Additional File 6 and Additional File 7, Additional File 8, Additional File 9, Additional File 10, Additional File 11, Additional File 12), and LLM scores have been generated for all 11 modules (Fig.
In doing so, LMMs performed basic inferencing, for instance linking a gene's erythropoiesis role and biomarker potential, and incorporated these inferences when scoring candidates. The process of designing prompts involved some experimentation, and in some instances, it was "collaborative" as we sought feedback from the models to enhance the clarity and scope of the prompts. We also found that the quality and accuracy of the output generated by the Large Language Models (LLMs) were directly proportional to the extent of the task. For instance, LLMs showed reluctance to identify functional convergences for extensive gene lists (~ > 30) or provided limited justifications when tasked with scoring more than 2-3 genes in a single prompt-which was the maximum number suggested by GPT-4. And as a result, we decided to request the scoring of only one gene per prompt.
Perhaps most notably, thesupporting references requested while prompting LLMs for scores and justifications on six different statements were seldom factual. However, they were consistently accurate at a later stage when the LLMs were prompted to provide supporting references for specific statements (Step 5). Notably, the phrasing of the prompt was found to be critical for this task. When requested directly to provide relevant references supporting a given statement GPT-4 often declined and explained that it was "beyond Fig. 8  Overview of LLM-generated scores for A37 module genes. This network represents the scores generated by two LLMs, GPT-4 and Claude, for six prespecified criteria, across a pool of candidate genes distributed across 11 BloodGen3 modules from the aggregate A37. Each pie chart shows the relative magnitude of the averaged score from the two models.
In our view, LLMs are unlikely to replace traditional scientific methods; instead, they serve as valuable augmentations. They streamline the laborious process of sifting through extensive literature and data, as exemplified in our knowledge-driven gene prioritization pipeline. In recognition of the rapidly evolving landscape of Large Language Models (LLMs) and their associated tools, we also acknowledge the limitation of focusing on a select few models in this study. At the time of this research, our choice centered around widely recognized and empirically proven models, specifically OpenAI's GPT-4 and Anthropic's Claude, as these provided a robust baseline for our gene prioritization tasks. While newer models and tools continue to emerge, offering potentially different capabilities or performance metrics, the LLMs chosen for this project were deemed most suitable for this specific application.
Conclusions In conclusion, our findings underscore LLMs potential in enhancing knowledge-driven candidate gene prioritization and selection processes. Through this study, we demonstrated that LLMs could direct gene selection with minimal human input, bringing about significant advancements in efficiency and productivity. Our ongoing efforts in developing scripts for interfacing with the GPT-4 and Claude APIs should further expedite the prioritization of extensive module repertoires such as BloodGen3 and pave the way for the creation of a universal immune profiling TFA panel. Moreover, basic parameter adjustments in such streamlined workflow could also allow for the design of disease-specific panels with minimal effort. As previously demonstrated with our work on COVID-19, this could include favoring the selection of genes specifically relevant to the pathophysiology of the target disease  [14].
Established utility as a therapeutic target in erythroid disorders like sideroblastic anemia 5. Potential as a biomarker for disrupted erythropoiesis based on its tight regulation • The other genes (BCL2L1, SLC4A1, CA1, FECH) are more broadly expressed across immune cells and their fold changes are smaller in magnitude or less consistent across key disease states • Each additional dataset provided further evidence support- ing ALAS2 as the top choice over other candidates based on erythroid specificity and associations with conditions of interest In summary, the collective evidence strongly supports ALAS2 as the top gene for inclusion in the intended blood profiling panel to represent an erythroid signature. The key factors are its erythroid-restricted expression and marked regulation across relevant conditions Prompts PROMPT 5: Could you please summarize the key conclusions you have drawn from the conversation so far?.
After the transfer, it is recommended to let the animal and tissue rest for 5-10min to avoid major shifts of the imaging field in lateral or axial directions. 32. Put pre-warmed distilled water on the coverslip, lower the objective and use the epifluorescence function in the "detection mode" of the ZEN software to focus on the lymph node under visual control. If green fluorescent cells have been transferred, you will probably be able to see them through the eyepiece, otherwise the vascularization of the lymph node will often stand out from the highly autofluorescent fat tissue. 33. When the lymph node is in focus, make sure the incubation chamber is closed and all lights are off. In the ZEN software, switch to "detection mode", set the 2P laser to 930 nm and approx. 90mW power at a zoom of 1.
Data processing. The data processing is performed using CODEX Processor 1. 7. 0. 6. It will recognize the images generated during the data export and the experimental file. The processing of the images includes (i) background subtraction that uses as a reference the blank cycles, (ii) cycle alignment, (iii) deconvolution, (iv) shading correction, and (v) stitching. The parameters for each of these steps are pre-defined and cannot be modified. However, it is important to highlight that the data generated by the system can be used with other software. Three different examples of post-processing images are shown in Fig. 6.
The tyramide-signal amplification (TSA)-based Opal mIHC (Fig. 9 ) using the Vectra 3. 0 microscope allows the simultaneous detection of up to six markers on a single tissue section, in addition to DAPI. This enables the investigation of complex immune cell subsets such as DC, which rely on the combination of multiple markers, as well as a functional characterization and distance measurements between different cell types. The enhanced signal amplification combined with the fluorescent-based detection provides a high sensitivity and allows the detection of weakly expressed markers. The stripping-based fluorescent mIHC additionally mediates an independence from the antibody host species and reduces sterical hindrance when staining epitopes that closely colocalize. Upon whole-tissue scanning and acquisition of multispectral images (MSIs), the spectral unmixing is conducted using the inForm software, which also provides the possibility to mark and subtract strong AF from the actual staining signals.
Data analysis 1. Adjust exposure times for both 100x and 200x resolution for all channels as described in the user manual. Save the protocol and acquire whole scans at 100x resolution of all slides, including the library slides. 2. Define regions of interests on the tissue using the Phenochart software. Choose different representative areas to account for tissue heterogeneity. 3. Acquire multispectral images (MSIs) at 200× resolution of all slides. 4. Create a spectral library in 'inForm' using a representative MSI of each library slide. 5. Unmix fluorophores using the spectral library. Optional: use the AF tool in the AF control slide to remove AF signals. 6. Perform trainable tissue segmentation, adaptive cell segmentation, and phenotyping as described in the user manual (Fig. 11 ). Create a different 'inForm' project for markers that are co-localized, using the same tissue and cell segmentation.
General remarks on confocal imaging of cDC1 • Obtaining high-resolution volumetric image datasets of larger tissues on a scanning confocal microscope requires long acquisition time per sample. The large file sizes of such data further results in a substantial increase in processing and analysis time. Therefore, the intention of each imaging experiment should be evaluated to determine the appropriate parameters before image acquisition. For global visualization of cDC1 localization throughout a whole tissue section, it is possible to image with a 20x objective and lower resolution settings, which significantly reduces microscopy time. However, in our hands, such Gpr141b tm2(iCre)Ciphe  [22]  Partial activity in mast cells. Clec9a Cre Clec9a tm2. 1(icre)Crs  [20]  Additional expression in cDC-precursors, leading to recombination in cDC2.
4. 3. 9 Image processing 4. 3. 9. 1 Tile stitching and dye separation. Following image acquisition, use the LAS X software (Leica) for image tile stitching and (optional) channel dye separation. Depending on the microscope manufacturer similar software can be used. Note: Dye separation can be used to computationally unmix overlapping spectra and eliminate cross-talk in multiparameter images (also known as compensation). Use this approach when it is impossible to choose dyes with distinct excitation/emission spectra.