Package: metadatamapping
metadatamappingA python library to fetch metadata from NCBI and MetaSRA for a list of NCBI accessions and data extraction from ARCHS4InstallationSimply run the followingpip install metadatamappingor clone the repositorygit clone git@github.com:dmalzl/metadatamapping.gitand runcd metadatamapping
pip install .you should then be able to import the package as usualExample usageNCBI sample, experiment, biosample or geo accessions can be mapped to SRA uids using themap_accessions_to_srauidsfunction from themetadatamodule of the package. The call to the function as shown below invokes two processes that concurrently fetch the SRA UIDs for the accessions in batches and write the results to the outputfile"/path/to/outputfile".frommetadatamappingimportmetadatasra_uids=metadata.map_accessions_to_srauids(accessions,"/path/to/outputfile",n_processes=2)The resulting SRA UIDs can then either be used to retrieve all associated accessions from the SRA with thesrauids_to_accessionsfunction from themetadatamodule like sofrommetadatamappingimportmetadatancbi_accessions=metadata.srauids_to_accessions(sra_uids)or link them to BioSample UIDs and then retrieve the associated metadata with thelink_sra_to_biosamplefunction from thelinkmodule and thebiosampleuids_to_metadatafunction from the metadata modulefrommetadatamappingimportmetadata,linksrauids_to_biosampleuids=link.link_sra_to_biosample(sra_uids.uid)biosample_metadata=metadata.biosample_uids_to_metadata(srauids_to_biosampleuids.biosample)Finally we can retrieve normalized metadata for the samples fromMetaSRAusing themetasra_from_study_idfunction of themetadatamodule (note that this database might not contain data for all your samples so the function may only returns normalized metadata for some of your samples)frommetadatamappingimportmetadatametasra_metadata=metadata.metasra_from_study_id(ncbi_accessions.study.unique())Additionally, the package provides an interface for parsing theARCHS4 HDF5 formatwhich is located in thearchs4module and handles parsing of associated metadata with theget_filtered_sample_metadatafunction as well as extraction of expression data in theAnnDataformat with thesamplesfunctionarchs4_file="/path/to/archs4.h5"retain_keys=['geo_accession','characteristics_ch1','molecule_ch1','readsaligned','relation','series_id','singlecellprobability','source_name_ch1','title']archs4_metadata=archs4.get_filtered_sample_metadata(archs4_file,retain_keys)archs4_adata=archs4.samples(archs4_file,dataframe_indexed_by_geo_accessions,n_processes=2)For a full demonstration of usage please refer to theSnakefilein theexamplesdirectory which gives an overview of how the intended usage looks like.Entrez credentialsmetadatamappingretrieves data from the Entrez eUtilities using thebiopythoninterface. By default the Entrez API only allows 3 requests per second ifEntrez.emailandEntrez.api_keyare not set. This can be increased by setting these properties accordingly which also speeds up the most timeconsuming part of the pipeline which is the accession -> SRA UID mapping as this relies on eSearch which only allows for one accession at a time (maybe it also takes several but I did not test this as I expect it to be cumbersome to pull apart then). So please make sure to set theEntrezproperties accordingly like sofromBioimportEntrezEntrez.email="<user>@<provider>.<domain>"Entrez.api_key="<NCBI API key>The email typically is the email associated to your NCBI account. The API key can be generated as describedhere