TITLE: Brain Data Alchemy Project: Meta-Analysis of Re-Analyzed Public Transcriptional Profiling Data in the Gemma Database (v.2024)
AUTHORS: Megan Hagenauer, Duy Manh Nguyen, Elizabeth Flandreau, Eva Geoghegan, Sophie Mensch, Mubashshir Ra'eed Bhuiyan, Lakshmi Thirupatamma Chennupati, Sophia Espinoza, Ashlee Lewis, Anna Drozman, Brandon W. Hughes, M.S., Ph.D.

[DESCRIPTION]
Over the past two decades, transcriptional profiling has become an increasingly common tool for investigating the effects of diseases and experimental manipulations on the nervous system. Within transcriptional profiling experiments, microarray or sequencing technologies are used to measure the relative amount of RNA transcript for each of the thousands of genes expressed in a sample. The primary objective of these experiments is to identify genes that are differentially expressed in response to conditions of interest. However, transcriptional profiling experiments have traditionally been conducted with small sample sizes due to expense (e.g., n=3-10/group), resulting in low statistical power. Due to low power, these experiments are prone to capturing large technical artifacts and false positives rather than smaller biological effects of interest.
To address this issue, we developed a 10-week summer research program (The Brain Data Alchemy Project) that guides participants through the process of performing a meta-analysis of differential expression effect sizes (Log2 Fold Change or Log2FC) extracted from publicly available transcriptional profiling datasets. To conduct our meta-analyses, we leverage the efforts of the Gemma project, which has curated, preprocessed, and re-analyzed over 19,000 publicly available datasets (https://gemma.msl.ubc.ca/home.html). Participants learn the fundamental principles of systematic review and R programming to conduct the dataset search, result extraction, and whole transcriptome meta-analysis.
This protocol outlines the methods used during the third pilot year of the program in 2024. These methods differ from the 2022 methods (dx.doi.org/10.17504/protocols.io.j8nlk84jxl5r/v1 ) in terms of the sophistication of the search strategy and inclusion/exclusion procedure, use of Entrez ID annotation and formal mouse/rat gene orthology, and the functionalization of most necessary coding steps. These methods also differ from the 2023 project methods - the 2023 methods imported the preprocessed gene expression data and metadata from Gemma, whereas the 2022 and 2024 project methods imported the differential expression results.

[GUIDELINES]
Any questions regarding this protocol should be directed to Dr. Megan Hagenauer (University of Michigan: hagenaue@umich.edu).

This protocol outlines the meta-analysis methods used by participants within the Brain Data Alchemy Program during the third pilot year of the program (2024). For updates, see the associated forks for this protocol.

[STEPS]
SECTION: Project Preparation: Environment Set-Up
1. Install R and RStudio
 

SECTION: Project Preparation: Environment Set-Up
1.1. Install R
 

 

SECTION: Project Preparation: Environment Set-Up
1.2. Install R Studio
 

 

SECTION: Project Preparation: Environment Set-Up
2. Follow the Brain Data Alchemy code repository on Github
 

SECTION: Project Preparation: Environment Set-Up
2.1. Sign up for a free account on Github: https://github.com/ 
 


SECTION: Project Preparation: Environment Set-Up
2.3. Optional: Install GitHub Desktop
 

 

SECTION: Project Preparation: Environment Set-Up
2.2. Follow the Brain Data Alchemy code repository

 


SECTION: Dataset Identification: Pre-specification of Search Strategy
3. We will first plan the search strategy that we will use to identify useful datasets for the meta-analysis.
 

 

 

SECTION: Dataset Identification: Pre-specification of Search Strategy
3.4. Pre-specify the search terms for your topic.
 

SECTION: Dataset Identification: Pre-specification of Search Strategy
3.2. The information source will be the same for all meta-analyses: The Gemma Database.
 

 

 

SECTION: Dataset Identification: Pre-specification of Search Strategy
3.1. The scope of our meta-analysis search is pre-specified below. The scope will be the same for each meta-analysis conducted using the Brain Data Alchemy analysis pipeline.
 

SECTION: Dataset Identification: Pre-specification of Search Strategy
3.3. Choose your meta-analysis topic and primary research question.
 

SECTION: Dataset Identification: Coding Steps
4. Dataset Identification: Coding Steps. This coding is all performed within the R environment using Rstudio.
 

SECTION: Dataset Identification: Coding Steps
4.1. Install the R package Devtools using: install.packages("devtools")
 


 

SECTION: Dataset Identification: Coding Steps
4.2. Install the gemma.R package.
 

 


 

SECTION: Dataset Identification: Coding Steps
4.4. Use the function get_datasets() to locate transcriptional profiling datasets within the Gemma database that contain your pre-specified keywords. 
 

 

 

SECTION: Initial Dataset Filtering Using MetaData
5.3. This is a critical step. Following this initial filtering by the individual experimenter, all inclusion/exclusion choices should be reviewed and approved by a second researcher (such as a research mentor). 
SECTION: Initial Dataset Filtering Using MetaData
5.2. The titles, abstracts, and metadata for the datasets should then be scanned to determine which datasets should be excluded based on our pre-specified criteria.  Make notes in the empty columns provided for the common exclusion criteria.  For each dataset that is excluded, also make notes in the "Excluded" and "WhyExcluded" column.
 


SECTION: Initial Dataset Filtering Using MetaData
5.1. Open the file containing the dataset metadata ("MyResults_annotated.csv") using a standard spreadsheet software program, like Excel or Google Sheets. Immediately resave the file in a spreadsheet format (e.g., .xlsx) - otherwise any changes that you make to the formatting (like color) will be lost when you close the file (.csv files do not include formatting).


 

 
SECTION: Initial Dataset Filtering Using MetaData
5. The metadata records should be filtered using pre-specified inclusion and exclusion criteria.
SECTION: Secondary Dataset Filtering Using Detailed Methodological Review
6. The remaining dataset metadata records should now be subjected to a detailed review of accompanying experimental methods and available file formats. 
 

6.1.  When reviewing the methodological information in the publications and metadata for the datasets, studies should be excluded using pre-specified inclusion/exclusion criteria. 

 

6.2. Depending on the research topic, other inclusion/exclusion criteria may also be applied. Ideally, these criteria should be pre-specified as much as possible. If not pre-specified, this deviation from the planned protocol should be documented. 
7.3. Following this secondary filtering by the individual experimenter, the experimental design for the remaining datasets and inclusion/exclusion choices should be reviewed and approved by a second researcher (such as a research mentor). 
 

SECTION: Result Extraction: Gemma&#x27;s Differential Expression Results
9. Result Extraction: Gemma's Differential Expression Results. This coding is all performed within the R environment using Rstudio.
 

SECTION: Result Extraction: Gemma&#x27;s Differential Expression Results
9.5. Within the object containing the differential expression results for a dataset, rows that lack unambiguous EntrezID annotation (including  “”, “null”, '\\|' in the "NCBIid" column) need to be counted and excluded. 
 

 

 

SECTION: Result Extraction: Gemma&#x27;s Differential Expression Results
9.6. Extract the specific differential expression results for the statistical contrasts (group comparisons) of interest (i.e., related to your research question).
 

 

SECTION: Result Extraction: Gemma&#x27;s Differential Expression Results
9.10. The reference group for the statistical comparison should be confirmed to fit expectations (e.g., represent the appropriate control or baseline group). 
 

SECTION: Result Extraction: Gemma&#x27;s Differential Expression Results
9.7. The standard error (SE) can be calculated using the Log2FC and T-statistic (Tstat) in the differential expression output for each row. (This will be performed by the function in step 9.9)
 

SECTION: Result Extraction: Gemma&#x27;s Differential Expression Results
9.9. If there is more than one row of results representing the same gene (same Entrez ID), the Log2FC and SE’s should be averaged for each gene EntrezID using the tapply() function. 
 

 

SECTION: Result Extraction: Gemma&#x27;s Differential Expression Results
9.8. The sampling variance (SV) for each Log2FC can then be calculated by squaring the standard error (SE^2). (This will be performed by the function in step 9.9)
 
 
SECTION: Exploring Meta-Analysis Results
13. To explore our meta-analysis results, we first rank the results by p-value and examine the results surviving our false discovery rate correction (FDR<0.05). We will refer to these results as our "top genes".
SECTION: Exploring Meta-Analysis Results
13.1. Within the results surviving false discovery rate correction, count how many show increased expression ("upregulation") in response to our variable of interest and how many show decreased expression ("down-regulation"). 
 

SECTION: Exploring Meta-Analysis Results
13.2. To explore the meta-analysis results for any particular gene in more detail, use the function forest.rma() from the metafor package to make a forest plot illustrating the effect sizes (Log2FC) and confidence intervals for each study.
 

 

 

SECTION: Exploring Meta-Analysis Results
13.4. Quick ways to learn about the functions associated with our top genes:

Basic functional summary: 
GeneCards: https://www.genecards.org/
Rat Genome Database: https://rgd.mcw.edu/

Cell types that express the gene in the brain:
DropViz: http://dropviz.org/
MouseBrain.Org: http://www.mousebrain.org/

Regional distribution of the expression for the gene in the brain: 
https://mouse.brain-map.org/search/index

 

 

 

 

SECTION: Exploring Meta-Analysis Results
13.5. Optional: To learn about the functions associated with our entire collection of differential expression results, we can use a functional ontology analysis. There are many different varieties of functional ontology analysis tools; I find that these two are an easy place to start out:

GOrilla:
https://cbl-gorilla.cs.technion.ac.il/

EnrichR:
https://maayanlab.cloud/Enrichr/

 

 

 

SECTION: Exploring Meta-Analysis Results
13.3. For the sake of easily including results from the project in future posters, presentations and publications, we have a template for quickly summarizing meta-analysis results.
 

SECTION: Wrapping Up the Project
14. In order to wrap up the meta-analysis project in a manner that can be easily followed up on later by yourself or others, it is important to preserve input, output, code, and workspaces.
SECTION: Wrapping Up the Project
14.2. To preserve and share code, it is useful to upload the final R code and workspaces to a Github repository.
 

 

SECTION: Wrapping Up the Project
14.1. First, make sure that you save both your code and workspace in R. 

 

SECTION: Wrapping Up the Project
14.3. To make your analyses reproducible, make sure that you preserve the folders containing the Gemma differential expression results for each dataset that you used as input for your meta-analysis. 
 

 

SECTION: Wrapping Up the Project
14.5. In addition to your results summary and example forest plots, make sure to save the full meta-analysis results for all genes. The full results can be provided as a supplementary table in a publication.

 

SECTION: Wrapping Up the Project
14.6. A README describing the organization for all of the meta-analysis files can help you (or others) navigate the files later.
SECTION: Dataset Identification: Pre-specification of Search Strategy
3.5. Examine the search terms more carefully and specify the formal search syntax.
 

 

SECTION: Project Preparation: Environment Set-Up
2.4. Create a Github repository for your project. Give the repository a name that clearly signals the goals of your specific project (e.g., "SleepDeprivationMetaAnalysis").

 

SECTION: Project Preparation: Environment Set-Up
2.6. Within R Studio, initiate an R project. 
 

 

SECTION: Project Preparation: Environment Set-Up
2.5. Within R Studio, set your Github credentials. 

 

 

SECTION: Dataset Identification: Coding Steps
4.3. Install and load the plyr package and dplyr package. The dplyr package can be installed as part of the full "tidyverse".
 


 

 

 

SECTION: Dataset Identification: Coding Steps
4.9. If there were very few datasets identified by the search (e.g., <50 datasets), it may be possible to filter the datasets down to those that include the targeted tissue/ROI by just looking at the metadata records. If so, jump to step 4.12.

Otherwise, to narrow the search to a particular brain region, it is best if we leverage the formal ontology used for defining "organism part" annotation within the Gemma database.

 

 

SECTION: Dataset Identification: Coding Steps
4.5. Filter the results by data quality to remove "troubled" datasets.
 

 

SECTION: Dataset Identification: Coding Steps
4.6. Output the results of the search in a format that can be easily opened in a standard spreadsheet program (comma separated variable file format: .csv).
 

SECTION: Dataset Identification: Coding Steps
4.7. Survey the results to determine how many datasets come from each brain region. The datasets in Gemma are reliably annotated with either the "organism parts" or "cell types" that were sampled for transcriptional profiling for each dataset. Brain regions are defined as "organism parts", whereas "cell types" are typically either purified cells (e.g., by flow cytometry) or cell culture.
 

SECTION: Dataset Identification: Coding Steps
4.8. If a particular tissue or ROI was pre-specified during the original conception of the project, jump to step 4.9.

If a particular tissue or brain region of interest (ROI) was not pre-specified during the conception of the project, the research question should now be narrowed to a particular tissue or brain ROI based on dataset availability.
 

SECTION: Dataset Identification: Coding Steps
4.10. Re-run the search, using the formal ontology term for the "organism part".
 

SECTION: Dataset Identification: Coding Steps
4.11. Filter the results again down to high quality data (no "troubled" datasets).
 

SECTION: Dataset Identification: Coding Steps
4.12. To make the dataset records easier to triage using our standardized inclusion/exclusion criteria, let's add additional annotation to the dataset metadata results outputted by our search, including information about the organism parts, developmental stages, experimental factors used in the experiment.
 

SECTION: Dataset Identification: Coding Steps
4.13. Add some empty columns to our dataset metadata data.frame for taking inclusion/exclusion notes and output the data frame as a file format that is easily viewed using a standard spreadsheet program (comma separated variable file: .csv).

 

 

SECTION: Initial Dataset Filtering Using MetaData
5.4. Copy the metadata record worksheet and use it to make a new worksheet in the same file. For this new worksheet, delete all metadata records for datasets that were excluded so that it is easier to easily view/read the remaining potential datasets.

 

SECTION: Tertiary Filtering of Datasets: Availability of Relevant Differential Expression Output
8. Identify the differential expression output (result set and statistical contrast) that is relevant to your research question within the Gemma database for each of the selected datasets.

 

SECTION: Tertiary Filtering of Datasets: Availability of Relevant Differential Expression Output
8.4. When reviewing the available result sets and statistical contrast statistical contrasts should be excluded using pre-specified criteria.
 

SECTION: Tertiary Filtering of Datasets: Availability of Relevant Differential Expression Output
8.2. Use the dataset ids (GSE#s) to access the result ids and contrast ids within the Gemma database for each dataset, and their accompanying metadata. Format this information into a more easily reviewable data frame.
 

 

 

7.4. Copy the remaining metadata record worksheet and use it to make a new (third) worksheet in the same file. For this new worksheet, again delete all metadata records for datasets that were excluded so that it is easier to easily view/read the remaining potential datasets. 
 

7.5. Copy the metadata for this final set of selected datasets into a new .csv file and save it in your working directory with the name "MyDatasets_Screened.csv".
SECTION: Tertiary Filtering of Datasets: Availability of Relevant Differential Expression Output
8.1. Input the spreadsheet containing your final chosen datasets into R and extract their dataset ids (GSE#s).
 

SECTION: Tertiary Filtering of Datasets: Availability of Relevant Differential Expression Output
8.3. Open the file containing the result set and statistical contrast metadata ("ResultSets_toScreen.csv") using a standard spreadsheet software program, like Excel or Google Sheets. Immediately re-save the file in a spreadsheet format (e.g., .xlsx) - otherwise any changes that you make to the formatting (like color) will be lost when you close the file (.csv files do not include formatting).
 

SECTION: Tertiary Filtering of Datasets: Availability of Relevant Differential Expression Output
8.5. For the statistical contrasts that remain, we need to double-check that the contrasts include an appropriate reference group (baseline) for our research question. If not, make a note that we will need to later invert the direction of effect in the differential expression results.

 

SECTION: Tertiary Filtering of Datasets: Availability of Relevant Differential Expression Output
8.6. Copy the result set and contrast id metadata worksheet and use it to make a new (second) worksheet in the same file. For this new worksheet, delete all metadata records for result sets and contrast ids that were excluded so that it is easier to easily view/read the remaining potential datasets. 
SECTION: Tertiary Filtering of Datasets: Availability of Relevant Differential Expression Output
8.7. Copy the result set and contrast id metadata for this final set of selected contrasts into a new .csv file and save it in your working directory with the name "ResultSets_Screened.csv".
SECTION: Tertiary Filtering of Datasets: Availability of Relevant Differential Expression Output
8.8. Following this tertiary filtering of result sets and contrasts by the individual experimenter, the inclusion/exclusion choices should be reviewed and approved by a second researcher (such as a research mentor). 
 

7. Next, double check whether the transcriptional profiling data for each dataset appears to have been imported into Gemma properly (especially datasets marked “external”).
This step is best completed for one dataset at a time.
 

 

7.6. Following all filtering, the full search strategy and inclusion/exclusion procedure should be documented using a PRISMA flow diagram. 

 

7.7. Following all filtering, the essential characteristics of the final datasets that will be included in the meta-analysis should be summarized in a table that can be included with your results.

 

7.1. Download the processed gene expression data from Gemma for the potential dataset.
 

7.2. Double-check the range of the processed gene expression data for the dataset.
For log2 RNA-Seq gene expression data, the range is typically between -5 to 12 
For log2 microarray gene expression datasets, the range is often between 4-15
If you see a highly restricted range (e.g., 2-4), the dataset should be excluded (note in column: "WhyExcluded").
 

SECTION: Tertiary Filtering of Datasets: Availability of Relevant Differential Expression Output
8.9. Following all filtering, the full search strategy and inclusion/exclusion procedure should be documented using a PRISMA flow diagram. 

 

SECTION: Tertiary Filtering of Datasets: Availability of Relevant Differential Expression Output
8.10. Following all filtering, the essential characteristics of the final datasets that will be included in the meta-analysis should be summarized in a table that can be included with your results.

 

SECTION: Result Extraction: Gemma&#x27;s Differential Expression Results
9.1. Import the file containing the final selected result sets and contrast ids ("ResultSets_Screened.csv") into R.
 

SECTION: Result Extraction: Gemma&#x27;s Differential Expression Results
9.2. Download the differential expression results from Gemma for the selected result sets and statistical contrasts.

 

 

 

SECTION: Result Extraction: Gemma&#x27;s Differential Expression Results
9.3. For record-keeping purposes, output and save the differential expression results for each result set.
 

 

 

SECTION: Result Extraction: Gemma&#x27;s Differential Expression Results
9.4. The next few processing steps (steps #9.4-9.10) need to be applied to a single result set at a time.

To begin applying these processing steps, we will need to begin by pulling the result set out from the differentials object (which contains all of the result sets, listed numerically) and temporarily making it a new object (DE_Results).  
 

SECTION: Aligning Differential Expression Results Across Datasets and Species
10. Align the differential expression results (Log2FC, SV) from the different datasets and result sets to make a single data frame, with each row representing one gene and each column representing the differential expression results from one statistical contrast.

 

SECTION: Aligning Differential Expression Results Across Datasets and Species
10.2. If there are datasets derived from rats, align them by rat Entrez ID (“NCBI_ID”). 
 

 

 

SECTION: Result Extraction: Gemma&#x27;s Differential Expression Results
9.11. ... and then move on to repeat steps #9.4-#9.10 for each of the other result sets.
SECTION: Result Extraction: Gemma&#x27;s Differential Expression Results
9.12. Make sure you save the R code and R workspace! 
 

SECTION: Aligning Differential Expression Results Across Datasets and Species
10.1. If there are datasets derived from mice, align them using mouse Entrez ID (“NCBI_ID”). 

 

 

 

SECTION: Aligning Differential Expression Results Across Datasets and Species
10.3. Next differential expression results derived from different species (rats vs. mice) need to be aligned (joined) using a gene ortholog database.
We need to read this ortholog information into R.

 


 


 

SECTION: Aligning Differential Expression Results Across Datasets and Species
10.4. Join the gene ortholog database to the two data frames containing all of the aligned mouse differential expression results (the Log2FC dataframe and the SV dataframe).
 

SECTION: Aligning Differential Expression Results Across Datasets and Species
10.5. If there are rat datasets, we then want to join our mouse Log2FC and SV data frames to the rat Log2FC and SV dataframes using the rat/mouse gene ortholog information

 

SECTION: Aligning Differential Expression Results Across Datasets and Species
10.6. If there aren't any rat datasets, we just rename the mouse Log2FC and SV dataframes so that our downstream code works.
 

SECTION: Aligning Differential Expression Results Across Datasets and Species
10.7. The current Mouse-Rat Entrez annotation is missing entries for any genes in the datasets that don't have orthologs. Replace it with a fixed version.
 

SECTION: Aligning Differential Expression Results Across Datasets and Species
10.8. Optional: Compare the effects (Log2FCs) across the different datasets and statistical contrasts using the data from all represented genes. 
Three main ways to do this are:
1) Pairwise: Compare the Log2FCs from two statistical contrasts. Here, I provide example code for making a scatterplot, but rank-rank hypergeometric overlap (RRHO) plots are also popular.
2) Correlation Matrix: Compare the Log2FCs from all statistical contrasts by outputting a matrix containing the correlation coefficients (in this case, Spearman) quantifying the relationship between each possible pair of statistical contrasts.
3) Clustered Heatmap: Make a hierarchically clustered heatmap to illustrate which contrasts/datasets have the most similar Log2FCs for all represented genes.
 

SECTION: Meta-Analysis of Differential Expression Results
11. Meta-Analysis of Differential Expression Results: This coding is all performed within the R environment using Rstudio.
 

SECTION: Meta-Analysis of Differential Expression Results
11.1. Because we are considering differential expression results that are derived from a variety of different transcriptional profiling platforms with varying transcript/probe representation and sensitivity, each dataset will include differential expression results for a slightly different set of genes.

Decide on a minimum number of differential expression results that need to be present to run a meta-analysis for each gene (i.e., a minimum number of Log2FC values that are not NAs). 

 

SECTION: Meta-Analysis of Differential Expression Results
11.2. Install the R package metafor
 

 
 
 

SECTION: Meta-Analysis of Differential Expression Results
11.3. To run a meta-analysis for every gene meeting our criteria for a minimum number of differential expression results, we use a for loop to run the same series of calculations for each row of our Log2FC data frame and accompanying SV data frame.  

 

SECTION: Meta-Analysis of Differential Expression Results
11.4. For each row (gene), we first determine whether the gene meets the criteria of having our pre-specified minimum number of differential expression results (Log2FC values). This is done using a conditional if-else statement.  

 

SECTION: Meta-Analysis of Differential Expression Results
11.5. If the row (gene) meets our criteria, we use the function rma() from the metafor package to fit a random effects meta-analysis model to the Log2FC values (yi) and accompanying SV (vi).
 

 

 

SECTION: Meta-Analysis of Differential Expression Results
11.6. If a random effects model for the Log2FC values for a gene (row) produces a convergence error (i.e., there is not a stable model fit), that row of results will be recorded as NA. 
SECTION: Applying a False Discovery Rate (FDR) Correction to the Meta-Analysis Results
12. False Discovery Rate Correction: Since we have run meta-analyses for thousands of genes, we correct the p-values for false discovery rate (FDR) using the the Benjamini-Hochberg method. This coding is all performed within the R environment using Rstudio.

 

SECTION: Applying a False Discovery Rate (FDR) Correction to the Meta-Analysis Results
12.1. Install the R package multtest 
 

 

 

 

SECTION: Applying a False Discovery Rate (FDR) Correction to the Meta-Analysis Results
12.2. We can add some additional gene annotation to our results during this step too.
 

 

 

SECTION: Applying a False Discovery Rate (FDR) Correction to the Meta-Analysis Results
12.3. We correct the p-values for false discovery rate (FDR) using the the Benjamini-Hochberg method as applied by the mt.rawp2adjp() function within the multtest package.
 

 

SECTION: Applying a False Discovery Rate (FDR) Correction to the Meta-Analysis Results
12.4. We will consider results to be statistically significant if they survive a threshold of 5% false discovery (FDR<0.05). If our meta-analyses do not produce any results surviving this threshold, we may consider results at a weaker threshold (FDR<0.10) with the understanding that a higher percent of these results are likely to be false discovery (10%).
SECTION: Wrapping Up the Project
14.4. In addition to your PRISMA search diagram and summary table of final selected datasets, make sure that you preserve the files containing your intermediate notes documenting the process of triaging datasets and result sets.