TITLE: Brain Data Alchemy Project: Meta-Analysis of Re-Analyzed Public Transcriptional Profiling Data in the Gemma Database
AUTHORS: Megan Hagenauer, Cosette Rhoads, Jinglin Xiong, Duy Manh Nguyen, Erin Hernandez, Annaka Saffron, Amrita Kondur, Elizabeth Flandreau

[DESCRIPTION]
Over the past two decades, transcriptional profiling has become an increasingly common tool for investigating the effects of diseases and experimental manipulations on the nervous system. Within transcriptional profiling experiments, microarray or sequencing technologies are used to measure the relative amount of RNA transcript for each of the thousands of genes expressed in a sample. The primary objective of these experiments is to identify genes that are differentially expressed in response to conditions of interest. However, transcriptional profiling experiments have traditionally been conducted with small sample sizes due to expense (e.g., n=3-10/group), resulting in low statistical power. Due to low power, these experiments are prone to capturing large technical artifacts and false positives rather than smaller biological effects of interest.
To address this issue, we developed a 10-week summer research program (The Brain Data Alchemy Project) that guides participants through the process of performing a meta-analysis of differential expression effect sizes (Log2 Fold Change or Log2FC) extracted from publicly available transcriptional profiling datasets. To conduct our meta-analyses, we leverage the efforts of the Gemma project, which has curated, preprocessed, and re-analyzed over 19,000 publicly available datasets (https://gemma.msl.ubc.ca/home.html). Participants learn the fundamental principles of systematic review and R programming to conduct the dataset search, result extraction, and whole transcriptome meta-analysis.
This protocol outlines the methods used during the first pilot year of the program in 2022.

[GUIDELINES]
Any questions regarding this protocol should be directed to Dr. Megan Hagenauer (University of Michigan: hagenaue@umich.edu).

This protocol outlines the meta-analysis methods used by participants within the Brain Data Alchemy Program during the first pilot year of the program (2022). These methods were updated for later years - for updates, see the associated forks for this protocol.

[STEPS]
SECTION: Project Preparation: Environment Set-Up
1. Install R and RStudio
 

SECTION: Project Preparation: Environment Set-Up
1.1. Install R
 

 

SECTION: Project Preparation: Environment Set-Up
1.2. Install R Studio
 

 

SECTION: Project Preparation: Environment Set-Up
2. Follow the Brain Data Alchemy code repository on Github
 

SECTION: Project Preparation: Environment Set-Up
2.1. Sign up for a free account on Github
 


SECTION: Project Preparation: Environment Set-Up
2.3. Optional: Install GitHub Desktop
 

 

SECTION: Project Preparation: Environment Set-Up
2.2. Follow the Brain Data Alchemy code repository

 


SECTION: Dataset Identification: Pre-specification of Search Strategy
3. We will first plan the search strategy that we will use to identify useful datasets for the meta-analysis.
 

 

 

SECTION: Dataset Identification: Pre-specification of Search Strategy
3.4. Pre-specify the search terms for your topic.
 

SECTION: Dataset Identification: Pre-specification of Search Strategy
3.2. The information source will be the same for all meta-analyses: The Gemma Database.
 

 

 

SECTION: Dataset Identification: Pre-specification of Search Strategy
3.1. The scope of our meta-analysis search is pre-specified below. The scope will be the same for each meta-analysis conducted using the Brain Data Alchemy analysis pipeline.
 

SECTION: Dataset Identification: Pre-specification of Search Strategy
3.3. Choose your meta-analysis topic and primary research question.
 

SECTION: Dataset Identification: Coding Steps
4. Dataset Identification: Coding Steps. This coding is all performed within the R environment using Rstudio.
 

SECTION: Dataset Identification: Coding Steps
4.1. Install the R package Devtools using: install.packages("devtools")
 


 

SECTION: Dataset Identification: Coding Steps
4.2. Install the gemma.R package.
 

 


 

SECTION: Dataset Identification: Coding Steps
4.3. Use the function search_datasets() to locate transcriptional profiling datasets within the Gemma database that contain your prespecified keywords. 
 

 

SECTION: Dataset Identification: Coding Steps
4.4. Compile the metadata records from these identified datasets into a single data frame.
 

SECTION: Dataset Identification: Coding Steps
4.5. Eliminate any duplicated records in the data frame.
 

SECTION: Specification of Tissue of Interest (If Not Pre-Specified)
6. If a particular tissue or brain region of interest (ROI) had not been pre-specified in the search criteria, the research question should now be narrowed to a particular tissue or brain ROI based on dataset availability, and the dataset records filtered accordingly. 
 

SECTION: Initial Dataset Filtering Using MetaData
5.3. This is a critical step. Following this initial filtering by the individual experimenter, all inclusion/exclusion choices should be reviewed and approved by a second researcher (such as a research mentor). 
SECTION: Initial Dataset Filtering Using MetaData
5.2. The titles, abstracts, and metadata for the datasets should then be scanned to determine which datasets should be excluded based on our pre-specified criteria. 
 


SECTION: Initial Dataset Filtering Using MetaData
5.1. Export the data frame containing the remaining metadata records from R as a .csv file. This data frame of metadata can then be more easily explored and annotated in a spreadsheet software program like Microsoft Excel or Google Sheets.

 
 
SECTION: Initial Dataset Filtering Using MetaData
5. The metadata records should be filtered using pre-specified inclusion and exclusion criteria.
SECTION: Secondary Dataset Filtering Using Detailed Methodological Review
7. The remaining dataset metadata records should now be subjected to a detailed review of accompanying experimental methods and available file formats. 
 

7.1. When reviewing the methodological information in the publications and metadata for the datasets, studies should be excluded using pre-specified inclusion/exclusion criteria. 

 

7.2. Depending on the research topic, other inclusion/exclusion criteria may also be applied. Ideally, these criteria should be pre-specified as much as possible. If not pre-specified, this deviation from the planned protocol should be reported. 
7.3. Following this secondary filtering by the individual experimenter, the experimental design for the remaining datasets and inclusion/exclusion choices should be reviewed and approved by a second researcher (such as a research mentor). 
 

7.4. When reviewing the available file formats, studies should be excluded using pre-specified criteria.
 

SECTION: Post-hoc Filtering:  Removing Problematic Externally-Derived Datasets
8. After we ran all of the meta-analyses for the Summer 2022 cohort of the Brain Data Alchemy Project, we realized that datasets that were marked “external” in the Gemma database had sometimes been imported into Gemma in a format that was incompatible with Gemma’s analysis pipeline. 
 

 

SECTION: Post-hoc Filtering:  Removing Problematic Externally-Derived Datasets
8.1. Due to these irregularities, all "external" datasets should be re-reviewed for evidence of irregular units at import or accidental double log2 transformation (as signaled by units with a highly restricted range - such as only ranging from 2-4). Any questionable externally-derived datasets should be excluded. 
 

SECTION: Result Extraction: Gemma&#x27;s Differential Expression Results
9. Result Extraction: Gemma's Differential Expression Results. This coding is all performed within the R environment using Rstudio.
 


SECTION: Result Extraction: Gemma&#x27;s Differential Expression Results
9.3. The files in each dataset folder should then be read into R. After reading in the files from a particular dataset folder, all of the result files for that dataset should be joined together by Element_Name using the join_all() function from the plyr package.

 


 

 

SECTION: Result Extraction: Gemma&#x27;s Differential Expression Results
9.2. Install the plyr package.
 


 

 

SECTION: Result Extraction: Gemma&#x27;s Differential Expression Results
9.4. If gene symbol annotation is missing in the result files, additional annotation information for the transcriptional profiling platform can be downloaded from Gene Expression Omnibus. This information will then need to be read into R, and joined to the differential expression result file.
 
 
SECTION: Result Extraction: Gemma&#x27;s Differential Expression Results
9.5. Within the joined file containing the differential expression results for a dataset, rows that lack unambiguous gene symbol annotation (including  “”, “null”, '\\|' in the gene symbol) need to be counted and excluded. 
 

 

SECTION: Result Extraction: Gemma&#x27;s Differential Expression Results
9.6. The specific statistical contrasts of interest (group comparisons) that are related to your research question should be identified within the differential expression results.
SECTION: Result Extraction: Gemma&#x27;s Differential Expression Results
9.7. The reference group for the statistical comparison should be confirmed to fit expectations (e.g., represent the appropriate control or baseline group). 
 

SECTION: Result Extraction: Gemma&#x27;s Differential Expression Results
9.8. The standard error (SE) can be calculated using the Log2FC and T-statistic (Tstat) in the differential expression output for each row. (This will be performed by the function in step 9.10)
 

SECTION: Result Extraction: Gemma&#x27;s Differential Expression Results
9.10. If there is more than one row of results representing the same gene symbol, the Log2FC and SE’s should be averaged for each gene symbol using the tapply() function. 
 

 

SECTION: Result Extraction: Gemma&#x27;s Differential Expression Results
9.9. The sampling variance (SV) for each Log2FC can then be calculated by squaring the standard error (SE^2). (This will be performed by the function in step 9.10)
 
 
SECTION: Result Extraction: Gemma&#x27;s Differential Expression Results
9.11. The results (Log2FC, SV) from the different statistical contrasts of interest and datasets should then be compiled into a single data frame. To do this, results can be aligned (joined) using official gene symbol.
 

 

 

SECTION: Meta-Analysis of Differential Expression Results
10.1. Because we are considering differential expression results that are derived from a variety of different transcriptional profiling platforms with varying transcript/probe representation and sensitivity, each dataset will include differential expression results for a slightly different set of genes.

Decide on a minimum number of differential expression results that need to be present to run a meta-analysis for each gene (i.e., a minimum number of Log2FC values that are not NAs). 

 

SECTION: Meta-Analysis of Differential Expression Results
10.2. Install the R package metafor
 

 
 
 

SECTION: Meta-Analysis of Differential Expression Results
10.5. If the row (gene) meets our criteria, we use the function rma() from the metafor package to fit a random effects meta-analysis model to the Log2FC values (yi) and accompanying SV (vi).
 

 

 

SECTION: Meta-Analysis of Differential Expression Results
10.3. To run a meta-analysis for every gene meeting our criteria for a minimum number of differential expression results, we use a for loop to run the same series of calculations for each row of our Log2FC data frame and accompanying SV data frame.  

 

SECTION: Meta-Analysis of Differential Expression Results
10.4. For each row (gene), we first determine whether the gene meets the criteria of having our pre-specified minimum number of differential expression results (Log2FC values). This is done using a conditional if-else statement.  

 

SECTION: Meta-Analysis of Differential Expression Results
10.6. If a random effects model for the Log2FC values for a gene (row) produces a convergence error (i.e., there is not a stable model fit), that row of results will be recorded as NA. 
SECTION: Applying a False Discovery Rate (FDR) Correction to the Meta-Analysis Results
11.1. Install the R package multtest 
 

 

 

SECTION: Applying a False Discovery Rate (FDR) Correction to the Meta-Analysis Results
11.2. We correct the p-values for false discovery rate (FDR) using the the Benjamini-Hochberg method as applied by the mt.rawp2adjp() function within the multtest package.
 

 

SECTION: Applying a False Discovery Rate (FDR) Correction to the Meta-Analysis Results
11.3. We will consider results to be statistically significant if they survive a threshold of 5% false discovery (FDR<0.05). If our meta-analyses do not produce any results surviving this threshold, we may consider results at a weaker threshold (FDR<0.10) with the understanding that a higher percent of these results are likely to be false discovery (10%).
SECTION: Exploring Meta-Analysis Results
12. To explore our meta-analysis results, we first rank the results by p-value and examine the results surviving our false discovery rate correction (FDR<0.05). We will refer to these results as our "top genes".
SECTION: Exploring Meta-Analysis Results
12.1. Within the results surviving false discovery rate correction, count how many show increased expression ("upregulation") in response to our variable of interest and how many show decreased expression ("down-regulation"). 
 

SECTION: Exploring Meta-Analysis Results
12.2. To explore the meta-analysis results for any particular gene in more detail, use the function forest.rma() from the metafor package to make a forest plot illustrating the effect sizes (Log2FC) and confidence intervals for each study.
 

 

SECTION: Exploring Meta-Analysis Results
12.4. Quick ways to learn about the functions associated with our top genes:

Basic functional summary: 
GeneCards: https://www.genecards.org/
Rat Genome Database: https://rgd.mcw.edu/

Cell types that express the gene in the brain:
DropViz: http://dropviz.org/
MouseBrain.Org: http://www.mousebrain.org/

Regional distribution of the expression for the gene in the brain: 
https://mouse.brain-map.org/search/index

 

 

 

 

SECTION: Exploring Meta-Analysis Results
12.5. Optional: To learn about the functions associated with our entire collection of differential expression results, we can use a functional ontology analysis. There are many different varieties of functional ontology analysis tools; I find that these two are an easy place to start out:

GOrilla:
https://cbl-gorilla.cs.technion.ac.il/

EnrichR:
https://maayanlab.cloud/Enrichr/

 

 

 

SECTION: Result Extraction: Gemma&#x27;s Differential Expression Results
9.1. The Gemma differential expression output relevant to the meta-analysis (e.g., correct brain ROI) for each of the selected datasets should be downloaded and placed into its own folder. 
 

SECTION: Meta-Analysis of Differential Expression Results
10. Meta-Analysis of Differential Expression Results: This coding is all performed within the R environment using Rstudio.
 

SECTION: Post-hoc Filtering:  Removing Problematic Externally-Derived Datasets
8.2. Following all filtering, the full search strategy and inclusion/exclusion procedure should be documented using a PRISMA flow diagram. 

 

SECTION: Post-hoc Filtering:  Removing Problematic Externally-Derived Datasets
8.3. Following all filtering, the essential characteristics of the final datasets that will be included in the meta-analysis should be summarized in a table that can be included with your results.

 

SECTION: Exploring Meta-Analysis Results
12.3. For the sake of easily including results from the project in future posters, presentations and publications, we have a template for quickly summarizing meta-analysis results.
 

SECTION: Wrapping Up the Project
13. In order to wrap up the meta-analysis project in a manner that can be easily followed up on later by yourself or others, it is important to preserve input, output, code, and workspaces.
SECTION: Wrapping Up the Project
13.2. To preserve and share code, it is useful to upload the final R code and workspaces to a Github repository.
 

SECTION: Wrapping Up the Project
13.1. First, make sure that you save both your code and workspace in R. 

 

SECTION: Wrapping Up the Project
13.3. To make your analyses reproducible, make sure that you save the folders containing the Gemma differential expression results for each dataset that you used as input for your meta-analysis. 
 

SECTION: Wrapping Up the Project
13.4. In addition to your PRISMA search diagram, table of final datasets, and results summary, make sure to save the full meta-analysis results for all genes. The full results can be provided as a supplementary table in a publication.
SECTION: Wrapping Up the Project
13.5. A README describing the organization for all of the meta-analysis files can help you (or others) navigate the files later.
SECTION: Dataset Identification: Pre-specification of Search Strategy
3.5. Examine the search terms more carefully and specify the formal search syntax.
 

SECTION: Applying a False Discovery Rate (FDR) Correction to the Meta-Analysis Results
11. False Discovery Rate Correction: Since we have run meta-analyses for thousands of genes, we correct the p-values for false discovery rate (FDR) using the the Benjamini-Hochberg method. This coding is all performed within the R environment using Rstudio.