TITLE: Brain Data Alchemy Project: Meta-Analysis of Preprocessed Public Transcriptional Profiling Data in the Gemma Database (v.2023)
AUTHORS: Megan Hagenauer, Elizabeth Flandreau, Toni Duan, Anne Bader, Christabel Mclain, Trevonn Gyles, Ashlee Lewis, Duy Manh Nguyen

[DESCRIPTION]
Background: Over the past two decades, transcriptional profiling has become an increasingly common tool for investigating the effects of diseases and experimental manipulations on the nervous system. Within transcriptional profiling experiments, microarray or sequencing technologies are used to measure the relative amount of RNA transcript for each of the thousands of genes expressed in a sample. The primary objective of these experiments is to identify genes that are differentially expressed in response to conditions of interest. However, transcriptional profiling experiments have traditionally been conducted with small sample sizes due to expense (e.g., n=3-10/group), resulting in low statistical power. Due to low power, these experiments are prone to capturing large technical artifacts and false positives rather than smaller biological effects of interest.
Methods: To address this issue, we developed a 10-week summer research program (The Brain Data Alchemy Project) that guides participants through the process of performing a meta-analysis of differential expression effect sizes (Log2 Fold Change or Log2FC) extracted from publicly available transcriptional profiling datasets. To conduct our meta-analyses, we leverage the efforts of the Gemma project, which has curated, preprocessed, and re-analyzed over 19,000 publicly available datasets (https://gemma.msl.ubc.ca/home.html). Participants learn the fundamental principles of systematic review and R programming to conduct the dataset search, result extraction, and whole transcriptome meta-analysis.
Version Information: This protocol outlines the methods used during the second pilot year of the program in 2023. These methods differ from the 2022 methods (dx.doi.org/10.17504/protocols.io.j8nlk84jxl5r/v1): The 2023 project methods import the preprocessed gene expression data and metadata from Gemma, whereas the 2022 project methods import the differential expression results.

[GUIDELINES]
Any questions regarding this protocol should be directed to Dr. Megan Hagenauer (University of Michigan: hagenaue@umich.edu).

This protocol outlines the meta-analysis methods recommended to participants within the Brain Data Alchemy Program during the second pilot year of the program (2023). These methods were updated for later years - for updates, see the associated forks for this protocol and for the protocol from the first pilot year of the program (2022: dx.doi.org/10.17504/protocols.io.j8nlk84jxl5r/v1).

[STEPS]
SECTION: Project Preparation: Environment Set-Up
1. Install R and RStudio
 

SECTION: Project Preparation: Environment Set-Up
1.1. Install R
 

 

SECTION: Project Preparation: Environment Set-Up
1.2. Install R Studio
 

 

SECTION: Project Preparation: Environment Set-Up
2. Follow the Brain Data Alchemy code repository on Github
 

SECTION: Project Preparation: Environment Set-Up
2.1. Sign up for a free account on Github
 


SECTION: Project Preparation: Environment Set-Up
2.3. Optional: Install GitHub Desktop
 

 

SECTION: Project Preparation: Environment Set-Up
2.2. Follow the Brain Data Alchemy code repository

 


SECTION: Dataset Identification: Pre-specification of Search Strategy
3. We will first plan the search strategy that we will use to identify useful datasets for the meta-analysis.
 

 

 

SECTION: Dataset Identification: Pre-specification of Search Strategy
3.4. Pre-specify the search terms for your topic.
 

SECTION: Dataset Identification: Pre-specification of Search Strategy
3.2. The information source will be the same for all meta-analyses: The Gemma Database.
 

 

 

SECTION: Dataset Identification: Pre-specification of Search Strategy
3.1. The scope of our meta-analysis search is pre-specified below. The scope will be the same for each meta-analysis conducted using the Brain Data Alchemy analysis pipeline.
 

SECTION: Dataset Identification: Pre-specification of Search Strategy
3.3. Choose your meta-analysis topic and primary research question.
 

SECTION: Dataset Identification: Coding Steps
4. Dataset Identification: Coding Steps. This coding is all performed within the R environment using Rstudio.
 

SECTION: Dataset Identification: Coding Steps
4.1. Install the R package Devtools using: install.packages("devtools")
 


 

SECTION: Dataset Identification: Coding Steps
4.2. Install the gemma.R package.
 

 

 

SECTION: Dataset Identification: Coding Steps
4.4. Use the function search_datasets() to locate transcriptional profiling datasets within the Gemma database that contain your prespecified keywords. 
 

 

SECTION: Dataset Identification: Coding Steps
4.5. Compile the metadata records from these identified datasets into a single data frame.
 

SECTION: Dataset Identification: Coding Steps
4.6. Eliminate any duplicated records in the data frame.
 

SECTION: Specification of Tissue of Interest (If Not Pre-Specified)
6. If a particular tissue or brain region of interest (ROI) had not been pre-specified in the search criteria, the research question should now be narrowed to a particular tissue or brain ROI based on dataset availability, and the dataset records filtered accordingly. 
 

SECTION: Initial Dataset Filtering Using MetaData
5.4. This is a critical step. Following this initial filtering by the individual experimenter, all inclusion/exclusion choices should be reviewed and approved by a second researcher (such as a research mentor). 
SECTION: Initial Dataset Filtering Using MetaData
5.3. The titles, abstracts, and metadata for the datasets should then be scanned to determine which datasets should be excluded based on our pre-specified criteria. 
 


SECTION: Initial Dataset Filtering Using MetaData
5.2. Export the data frame containing the remaining metadata records from R as a .csv file. This data frame of metadata can then be more easily explored and annotated in a spreadsheet software program like Microsoft Excel or Google Sheets.

 
 
SECTION: Initial Dataset Filtering Using MetaData
5. The metadata records should be filtered using pre-specified inclusion and exclusion criteria.
SECTION: Secondary Dataset Filtering Using Detailed Methodological Review
7. The remaining dataset metadata records should now be subjected to a detailed review of accompanying experimental methods and available file formats. 
 

7.1. When reviewing the methodological information in the publications and metadata for the datasets, studies should be excluded using pre-specified inclusion/exclusion criteria. 

 

7.2. Depending on the research topic, other inclusion/exclusion criteria may also be applied. Ideally, these criteria should be pre-specified as much as possible. If not pre-specified, this deviation from the planned protocol should be reported. 
7.3. Following this secondary filtering by the individual experimenter, the experimental design for the remaining datasets and inclusion/exclusion choices should be reviewed and approved by a second researcher (such as a research mentor). 
 

SECTION: Double Checking Data Normalization: Externally-Derived Datasets
9. After we ran all of the meta-analyses for the Summer 2022 cohort of the Brain Data Alchemy Project, we realized that datasets that were marked “external” in the Gemma database had sometimes been imported into Gemma in a format that was incompatible with Gemma’s analysis pipeline. 
 

 

SECTION: Double Checking Data Normalization: Externally-Derived Datasets
9.1. Due to these irregularities, all datasets - especially "external" datasets - should be re-reviewed for evidence of irregular units at import or accidental double log2 transformation (as signaled by units with a highly restricted range - such as gene expression values only ranging from 2-4). 
 

SECTION: Result Extraction: Differential Expression Results from Each Dataset
17. Extract the differential expression results for each dataset and prepare the results for the meta-analysis. The following code is intended to be applied to the differential expression results from one dataset at a time. 
It may be easiest to conduct these steps within a new R workspace.
 

SECTION: Result Extraction: Differential Expression Results from Each Dataset
17.1. The differential expression results for the dataset should be read into R from the working directory. 
 

SECTION: Result Extraction: Differential Expression Results from Each Dataset
17.2. Within the joined file containing the differential expression results for a dataset, rows that lack unambiguous gene symbol annotation (including  “”, “null”, '\\|' in the gene symbol) need to be counted and excluded. 
 

 

SECTION: Result Extraction: Differential Expression Results from Each Dataset
17.3. The specific statistical contrasts of interest (group comparisons) that are related to your research question should be identified within the differential expression results. The Log2 Fold Changes (Log2FC - i.e., group differences in log2 gene expression) and t-statistics for these statistical contrasts need to be extracted into their own matrix. The columns in these two matrices are then given descriptive names that include the dataset ids.

 

SECTION: Result Extraction: Differential Expression Results from Each Dataset
17.4. The standard error (SE) can be calculated using the Log2FC and T-statistic (Tstat) in the differential expression output for each row. (This will be performed by the function in step 17.6)
 

SECTION: Result Extraction: Differential Expression Results from Each Dataset
17.5. The sampling variance (SV) for each Log2FC can then be calculated by squaring the standard error (SE^2). (This will be performed by the function in step 17.6)
 
 
SECTION: Meta-Analysis of Differential Expression Results
19.1. Because we are considering differential expression results that are derived from a variety of different transcriptional profiling platforms with varying transcript/probe representation and sensitivity, each dataset will include differential expression results for a slightly different set of genes.

Decide on a minimum number of differential expression results that need to be present to run a meta-analysis for each gene (i.e., a minimum number of Log2FC values that are not NAs). 

 

SECTION: Meta-Analysis of Differential Expression Results
19.2. Install the R package metafor
 

 
 
 

SECTION: Meta-Analysis of Differential Expression Results
19.5. If the row (gene) meets our criteria, we use the function rma() from the metafor package to fit a random effects meta-analysis model to the Log2FC values (yi) and accompanying SV (vi).
 

 

 

SECTION: Meta-Analysis of Differential Expression Results
19.3. To run a meta-analysis for every gene meeting our criteria for a minimum number of differential expression results, we use a for loop to run the same series of calculations for each row of our Log2FC data frame and accompanying SV data frame.  

 

SECTION: Meta-Analysis of Differential Expression Results
19.4. For each row (gene), we first determine whether the gene meets the criteria of having our pre-specified minimum number of differential expression results (Log2FC values). This is done using a conditional if-else statement.  

 

SECTION: Meta-Analysis of Differential Expression Results
19.6. If a random effects model for the Log2FC values for a gene (row) produces a convergence error (i.e., there is not a stable model fit), that row of results will be recorded as NA. 
SECTION: Applying a False Discovery Rate (FDR) Correction to the Meta-Analysis Results
20.1. Install the R package multtest 
 

 

 

 

SECTION: Applying a False Discovery Rate (FDR) Correction to the Meta-Analysis Results
20.3. We correct the p-values for false discovery rate (FDR) using the the Benjamini-Hochberg method as applied by the mt.rawp2adjp() function within the multtest package.
 

 

SECTION: Applying a False Discovery Rate (FDR) Correction to the Meta-Analysis Results
20.4. We will consider results to be statistically significant if they survive a threshold of 5% false discovery (FDR<0.05). If our meta-analyses do not produce any results surviving this threshold, we may consider results at a weaker threshold (FDR<0.10) with the understanding that a higher percent of these results are likely to be false discovery (10%).
SECTION: Exploring Meta-Analysis Results
21. To explore our meta-analysis results, we first rank the results by p-value and examine the results surviving our false discovery rate correction (FDR<0.05). We will refer to these results as our "top genes".
SECTION: Exploring Meta-Analysis Results
21.1. Within the results surviving false discovery rate correction, count how many show increased expression ("upregulation") in response to our variable of interest and how many show decreased expression ("down-regulation"). 
 

SECTION: Exploring Meta-Analysis Results
21.2. To explore the meta-analysis results for any particular gene in more detail, use the function forest.rma() from the metafor package to make a forest plot illustrating the effect sizes (Log2FC) and confidence intervals for each study.
 

 

 

SECTION: Exploring Meta-Analysis Results
21.4. Quick ways to learn about the functions associated with our top genes:

Basic functional summary: 
GeneCards: https://www.genecards.org/
Rat Genome Database: https://rgd.mcw.edu/

Cell types that express the gene in the brain:
DropViz: http://dropviz.org/
MouseBrain.Org: http://www.mousebrain.org/

Regional distribution of the expression for the gene in the brain: 
https://mouse.brain-map.org/search/index

 

 

 

 

SECTION: Exploring Meta-Analysis Results
21.5. Optional: To learn about the functions associated with our entire collection of differential expression results, we can use a functional ontology analysis. There are many different varieties of functional ontology analysis tools; I find that these two are an easy place to start out:

GOrilla:
https://cbl-gorilla.cs.technion.ac.il/

EnrichR:
https://maayanlab.cloud/Enrichr/

 

 

 

SECTION: Meta-Analysis of Differential Expression Results
19. Meta-Analysis of Differential Expression Results: This coding is all performed within the R environment using Rstudio.
 

SECTION: Exploring Meta-Analysis Results
21.3. For the sake of easily including results from the project in future posters, presentations and publications, we have a template for quickly summarizing meta-analysis results.
 

SECTION: Wrapping Up the Project
22. In order to wrap up the meta-analysis project in a manner that can be easily followed up on later by yourself or others, it is important to preserve input, output, code, and workspaces.
SECTION: Wrapping Up the Project
22.2. To preserve and share code, it is useful to upload the final R code and workspaces to a Github repository.
 

SECTION: Wrapping Up the Project
22.1. First, make sure that you save both your code and workspace in R. 

 

SECTION: Wrapping Up the Project
22.3. To make your analyses reproducible, make sure that you save the folders containing the Gemma differential expression results for each dataset that you used as input for your meta-analysis. 
 

SECTION: Wrapping Up the Project
22.4. In addition to your PRISMA search diagram, table of final datasets, and results summary, make sure to save the full meta-analysis results for all genes. The full results can be provided as a supplementary table in a publication.
SECTION: Wrapping Up the Project
22.5. A README describing the organization for all of the meta-analysis files can help you (or others) navigate the files later.
SECTION: Dataset Identification: Pre-specification of Search Strategy
3.5. Examine the search terms more carefully and specify the formal search syntax.
 

 

 

SECTION: Applying a False Discovery Rate (FDR) Correction to the Meta-Analysis Results
20. False Discovery Rate Correction: Since we have run meta-analyses for thousands of genes, we correct the p-values for false discovery rate (FDR) using the the Benjamini-Hochberg method. This coding is all performed within the R environment using Rstudio.

 

SECTION: Dataset Identification: Coding Steps
4.3. Use the function search_datasets() to determine how many transcriptional profiling datasets within the Gemma database contain your prespecified keywords. 
 

 

SECTION: Initial Dataset Filtering Using MetaData
5.1. Exclude any datasets that Gemma has marked as "troubled". 

 

SECTION: Dataset Import
8. The gene expression data, gene (row) annotation, and sample metadata for each dataset will need to be read into R.  This will need to be done sequentially - i.e., complete all of the dataset import and analysis steps for one dataset, and then move on to the next.
It may be easiest to conduct these processing steps for each dataset within a its own dedicated R workspace and working directory.

Ideally, this would mean that we import a "tidy" summarized experiment object for the filtered gene expression data, but we found that not all datasets had summarized experiment objects available. We have provided the code for both scenarios below.  

 
SECTION: Dataset Import
8.2. If there is a summarized experiment object for a dataset, this code can be adapted to import the summarized experiment object for the dataset from Gemma into our R environment:
 

 

 

SECTION: Dataset Import
8.3. If there is *not* a summarized experiment object for a dataset, we can make one by adapting the code in this example:
 

 

SECTION: Dataset Import
8.1. This code will make use of several R code packages which may need to be installed before being loaded.  One package is plyr, the other packages are part of a series called "the tidyverse" which can be installed and loaded together using a single command.

 

 

 

 

 

SECTION: Dataset Import
8.4. If the dataset was originally collected using RNA-Seq (vs. microarray), we will need some additional metadata about the number of read counts collected for each sample (library size). 

 


 

 


 

 

 

 

 

 

 

SECTION: Dataset Subsetting
10. Subset the dataset down to the samples that you plan to use. 
 

SECTION: Dataset Subsetting
10.1. Determine the characteristics of the samples in the dataset and how many samples have each characteristic of interest.
 

SECTION: Double Checking Data Normalization: Externally-Derived Datasets
9.2. If the dataset appears to have been clearly accidentally log2 transformed twice over (as signaled by gene expression units with a highly restricted range, e.g., from 2-4), the second log2 transformation can be reversed using y=2x.

7.4. Following all filtering, the full search strategy and inclusion/exclusion procedure should be documented using a PRISMA flow diagram. 

 

7.5. Following all filtering, the essential characteristics of the final datasets that will be included in the meta-analysis should be summarized in a table that can be included with your results.

 

SECTION: Dataset Subsetting
10.2. Filter out the samples that have unwanted characteristics.
 

 

SECTION: Removing outlier samples from the dataset
11. Check for outlier samples and remove them if they are present.

 at
SECTION: Removing outlier samples from the dataset
11.1. Using the full expression data, examine the pattern of sample-sample correlations to identify any outlier samples.

 

 
 
SECTION: Removing outlier samples from the dataset
11.2. If there is an outlier sample, remove it from the summarized experiment object.
 

SECTION: Removing outlier samples from the dataset
11.3. If there isn't an outlier sample, just rename the summarized experiment object so that the downstream code works.

 

SECTION: Filtering out genes lacking variability in expression measurements in experimental subgroups
12. Filter out genes (rows) in the dataset that lack variability in expression measurements in any of the experimental groups.
 


SECTION: Filtering out genes lacking variability in expression measurements in experimental subgroups
12.1. To filter out genes (rows) that lack variability in the experimental subgroups, we will need to know how to define our experimental subgroups of interest.  
If only one variable is of interest (for example, antidepressant treatment), this will be relatively easy.  
If more than one variable is of interest (for example, the interaction of antidepressant treatment with treatment responder vs. treatment non-responder status), then we will need to identify genes that lack variability in any subgroup created by any combination of those two variables (treatment*phenotype).
 

SECTION: Checking for batch confounds
13. Check for confounding variability due to technical processing batches. 

 

SECTION: Checking for batch confounds
13.1. View the distribution of the batch variable across samples and determine if it needs to be divided into subcomponents (e.g., device, run, flow cell, lane, etc). 
 

 

SECTION: Checking for batch confounds
13.2. If the batch variable (block) has subcomponents (e.g., device, run, flow cell, lane, etc), split them apart into individual variables.

 

SECTION: Checking for batch confounds
13.3. Determine whether any of the batch-related variables are distributed in an unbalanced manner across your variables of interest. 
 

SECTION: Principal Components Analysis (PCA)
14. Use Principal Components Analysis (PCA) to identify biological or technical co-variates that may be highly impacting the gene expression data.

 

SECTION: Principal Components Analysis (PCA)
14.1. Load the R stats package: library(stats)
 

 

SECTION: Principal Components Analysis (PCA)
14.2. Run PCA on the transposed gene expression matrix.
Transposing a matrix means to switch the orientation of the matrix so that the original rows are now columns and the original columns are now rows.

 

SECTION: Principal Components Analysis (PCA)
14.4. Extract out the first four principal components of variation (PC1-PC4) and make them named R objects for graphing purposes. These values show you the weighted contribution of each of your samples to the principal components of variation in the dataset.

 

SECTION: Principal Components Analysis (PCA)
14.5. Extract out the first four eigenvectors and export them as a .csv file for later reference. The eigenvectors show the weighted contribution of each of the genes in your gene expression matrix to each of the principal components of variation in the dataset.

 

SECTION: Principal Components Analysis (PCA)
14.3. Create a scree plot for the PCA results. This plot illustrates the proportion of variance explained by each principal component of variation in the dataset.
 

SECTION: Principal Components Analysis (PCA)
14.6. Output a scatterplot illustrating the relationship between Principal components 1 & 2 (PC1 & PC2). 
 

 


SECTION: Principal Components Analysis (PCA)
14.7. Examine the relationship between the top PCs (PC1-4) and each of the potential biological and technical co-variates.

 

SECTION: Double checking the model: Examining the expression data from a single gene vs. the variable of interest
15. Before progressing forward with the differential expression analysis, it can be helpful to examine the data from a single gene in comparison to the variable of interest. This can be used to confirm that the intended differential expression model makes sense and isn't causing error messages.
SECTION: Double checking the model: Examining the expression data from a single gene vs. the variable of interest
15.1. If needed, rename the conditions for the variable(s) of interest so that the names are concise and informative.

 

SECTION: Double checking the model: Examining the expression data from a single gene vs. the variable of interest
15.3. Illustrate the relationship between the expression of a single gene and the variable of interest.
 

SECTION: Double checking the model: Examining the expression data from a single gene vs. the variable of interest
15.2. Set up the categorical variables of interest so that they have an intuitive reference group

 

SECTION: Double checking the model: Examining the expression data from a single gene vs. the variable of interest
15.5. Test out the differential expression model using a simple Type III linear regression analysis.
Within the output:
- The Intercept should be the reference group (in this example, saline)
- The "Estimate" is the Log2FC (the difference between the group of interest and the reference group in Log2 expression units). A positive value means that the group of interest has greater expression than the reference group, a negative value means that the group of interest has lower expression than the reference group. This should match what you see in your boxplot.
- The "Pr(>|t|)" is the p-value showing the probability of this effect arising under conditions of random chance. Ignore this for now - a real differential expression analysis performed on gene expression data normally has additional analysis steps to control for extreme values or mean-variance trends in the data.
 

SECTION: Differential Expression Analysis
16. To run the differential expression analysis, we will use the limma differential expression analysis pipeline (for microarray data) or limma-trend differential expression analysis pipeline (for RNA-Seq data).
 

 

SECTION: Differential Expression Analysis
16.1. Install and load the R package limma:
 

 

 

SECTION: Double checking the model: Examining the expression data from a single gene vs. the variable of interest
15.4. Design the statistical model that will be used for running the differential expression analyses. This model should include the variable of interest (e.g., experimental treatment) as well as any desired biological or technical co-variates. 
 

SECTION: Differential Expression Analysis
16.2. Fit the differential expression model to the entire log2 gene expression dataset and apply an empirical Bayes correction using the eBayes() function to reduce the impact of outliers in small samples

The code for this step will depend on whether your data is from a microarray or RNA-Seq experiment. For the RNA-Seq datasets, we include a correction for the mean-variance relationship present in the dataset (trend=TRUE). 
 

 

SECTION: Differential Expression Analysis
16.3. The limma results need to be written out into the working directory, along with their accompanying gene annotation. 
Note that these results and annotation should either be written out into a directory specific to this dataset or given dataset-specific filenames - otherwise the files will be overwritten when another dataset is analyzed!

 

 

SECTION: Result Extraction: Differential Expression Results from Each Dataset
17.6. The differential expression results for the dataset are then placed into an object in the environment that is named "DEResults_" followed by the dataset name.
 

 

 

SECTION: Aligning Results Across Datasets
18. The results (Log2FC, SV) from the different datasets are then compiled into a single data frame. 
SECTION: Aligning Results Across Datasets
18.1. Install the plyr package.
 


 

 

SECTION: Aligning Results Across Datasets
18.2. The results (Log2FC, SV) from the different statistical contrasts of interest and datasets should then be compiled into a single data frame. To do this, results from studies performed on the same species are first aligned (joined) using Entrez ID (“NCBI_ID”). 
 

 

SECTION: Aligning Results Across Datasets
18.3. Next studies performed on different species (rats vs. mice) need to be aligned (joined) using a gene orthology database.

 

 

SECTION: Applying a False Discovery Rate (FDR) Correction to the Meta-Analysis Results
20.2. We can add some additional gene annotation to our results during this step too.
 

 

 

SECTION: Dataset Import
8.5. To increase the reproducibility of our results, save a copy of the Gemma Summarized Experiment data in an Excel-friendly file format (.csv).