TITLE: Acquiring and integrating data from multiple resources for meta-analysis
AUTHORS: Wu Huang, Alex Twyford, Peter Hollingsworth

[DESCRIPTION]
Acquiring sequence data generated for various purposes and repurposing it to evaluate the nature of genomic differences between plant species requires thorough dataset assessments, efficient data processing, and careful management. In this chapter, I summarise the methodology for searching appropriate datasets, data collection and filtering, and data management. I provide examples of the data formats to guide future downloading and acquiring of the datasets, and the links to curated publicly available bioinformatic tools and self-written scripts to allow reproduction and reuse of this data acquiring and processing pipeline. 

There is now a significant number of studies that have sequenced multiple loci from the nuclear genomes of plants and these are available in public databases or in private data repositories (if the study hasn’t yet been published). Collectively these datasets have great potential for understanding the nature of genomic differences between plant species. To harness this existing information I have developed a set of workflows, with the logic behind my approach being to
●  Develop key criteria for selecting suitable datasets focusing only on datasets which have sampled multiple individuals from multiple congeneric species
●  Search the literature and public data repositories for datasets that match these criteria
●  Obtain and organise the selected datasets
●  Use these datasets to estimate the proportion of species that resolve as monophyletic units as one measure of species discrimination success
●  Establish the frequency distribution of taxonomically informative nucleotide substitutions among taxa to better understand the genomic nature of differences between plant species
●  Subsample the data to better understand the effectiveness of smaller numbers of loci in recovering maximal species 
discrimination, as well as evaluating the attributes of the gene regions that are most effective at telling species apart.

This protocol will focus on the first three bullet points. The protocol for the rest of the bullet points please refer to another protocol "NucBarcoder - a bioinformatic pipeline to characterise the genetic basis of plant species differences". 

[STEPS]
SECTION: Developing data request form
3. Collecting information about the file formats frequently used in the data repositories that are popular in this particular field. (Normally read >20 related publications and the picture will be clear.) Choose the most frequently used ones as input formats for the following analytical pipelines. Three important files are:
SECTION: Developing data request form
1. Sourcing the literature and compiling publications which sequenced multiple nuclear loci for multiple individuals from multiple congeneric species.

SECTION: Acquiring data
5. In addition to mining the published literature, I contacted potential collaborators to request access to unpublished datasets. This involved designing a data request form and developing a list of metadata entries. This is for standardising metadata and agree with a data-sharing scheme.    

SECTION: Data storage and backup
6. The shared data were uploaded to Google Drive (https://www.google.co.uk/intl/en-GB/drive/), and then downloaded and managed on the UK crop diversity bioinformatics HPC platform (https://www.cropdiversity.ac.uk). 
Two copies of the backup are continually updated. A full version of data is backed up on the local hard drive and an abridged version ison the laptop.
SECTION: Developing data request form
2. Reading the methods and concluding the two popular NGS analysis pipelines are HybPiper and iPyRAD. Analyse the files used in each step and evaluate the pros and cons of acquiring data from that step.
SECTION: Developing data request form
2.1. 
 

SECTION: Acquiring data
5.1. Identify potential working groups that have unpublished data.   

SECTION: Acquiring data
4. Through the literature review, identify published data that could be reused.
SECTION: Acquiring data
4.2. Extract metadata from the publication manually.
SECTION: Acquiring data
5.2. Ask people to fill in the metadata entries.
SECTION: Acquiring data
4.1.  Email the corresponding author of published data asking for their consent of using their data (though it's always free to use published data, it helps to build rapport and potentially will gain additional insights into the data used). Attached is an example of the email I used for this request.   

SECTION: Acquiring data
4.3. Download data from the data repositories, such as Zenodo and Dryad. Email the correspondent author for required data or metadata if the files are not self-explanatory. The chance of having a helpful reply is low though.
SECTION: Acquiring data
5.3. Send them the guidance for sharing files, including what files to share, where to share, and how the data will be used and authored.   

SECTION: Developing data request form
1.1. To compile data suitable for meta-analysis, We first searched journal publications from 2013 onwards for studies that sequenced multiple loci from the nuclear genome and which sampled multiple individuals of multiple congeneric species. The cut-off of 2013 was selected as this reflects the initiation of widespread use of next-generation sequence platforms for the recovery of nuclear sequence data from plants. We used ambiguous matching patterns to search in the Web of Science and University of Edinburgh literature search engines. The matching patterns are listed in Table 1.1 
 
SECTION: Developing data request form
1.2. With the full-text publications downloaded, the next step was selecting publications manually.
I only kept studies if they satisfied the following criteria.
    1) Three or more un-linked nuclear loci were sequenced.
    2) More than two species had multiple individuals successfully sequenced and retained.
    3) A phylogeny is available either in visual format or in a machine readable text format (newick, phylip, or nexus formats).
    4) The species identities on the phylogeny could be interpreted and related to the sampled species.
SECTION: Developing data request form
3.1. Metadata. This includes information that corresponds to the sample IDs in the consensus sequence files which links sequences of individuals to their scientific names (species identities). This information is also useful for tracking changes of names wherever this has been applied, as well as monitoring and understanding the inclusion and exclusion of individuals and loci.
 
SECTION: Developing data request form
3.2. Sequence alignment file. This alignment covers the sequence variation in all individuals per dataset, and depending on how the original raw reads were processed, takes the form of either multiple-aligned sequences in .fasta, .phylip, or .nex formats, or SNP matrix in .vcf or .fasta format.
 
SECTION: Developing data request form
3.3. Phylogenetic trees. Phylogenetic trees were recovered from each study to enable easy estimation of the proportion of species that resolve as monophyletic. The preferred format is for this to be a machine-readable text format such as the Newick format. However, where only a graphical representation of the tree was available, this was also retained and used, to maximise the number of studies analysed.