Package: pubget
pubgetpubgetis a command-line tool for collecting data for biomedical text-mining, and in particular large-scale coordinate-based neuroimaging meta-analysis.
It exposes some of the machinery that was used to create theneuroquery dataset, which powersneuroquery.org.pubgetdownloads full-text articles fromPubMed Centraland extracts their text, metadata and stereotactic coordinates.
It can also computeTFIDF featuresfor the extracted text, fitNeuroQueryorNeuroSynth, and format its output for use withNiMAREorlabelbuddy.
It can be extended with plugins.Besides the command-line interface,pubget's functionality is also exposed through itsPython API.InstallationYou can installpubgetby running:pip install pubgetThis will install thepubgetPython package, as well as thepubgetcommand.Quick StartOncepubgetis installed, we can download and process biomedical publications so that we can later use them for text-mining or meta-analysis.pubget run ./pubget_data -q "fMRI[title]"Seepubget run --helpfor a description of this command.
For example, the--n_jobsoption allows running some of the steps in parallel.UsageThe creation of a dataset happens in 3 steps:Downloading the articles in bulk from thePMCAPI.Extracting the articles from the bulk downloadExtracting text, stereotactic coordinates and metadata from the articles, and storing this information in CSV files.Afterwards, some optional steps can also be run, such as:Vectorizing the text: transforming it into vectors ofTFIDFfeatures.Running the same analyses as NeuroSynth or NeuroQuery.Preparing the data for use with labelbuddy or NiMARE.Each of these steps stores its output in a separate directory.
Normally, you will run the whole procedure in one command by invokingpubget run.
However, separate commands are also provided to run each step separately.
Below, we describe each step and its output.
Usepubget -hto see a list of all available commands andpubget run -hto see all the options of the main command.All articles downloaded bypubgetcome fromPubMed Central, and are therefore identified by their PubMed Central ID (pmcid).
Note this is not the same as the PubMed ID (pmid).
Not all articles in PMC have apmid.pubgetonly downloads articles from theOpen Access subsetof PMC.
The open-access papers are the papers whose license allows downloading their text for text-mining or other reuse (Creative Commons or similar licenses).
To restrict search results to the open-access subset on the PMC website (and see the papers that would be downloaded bypubget), select "Open access" in the "article attributes" list.Step 1: Downloading articles from PMCThis step is executed by thepubget downloadcommand.
Articles to download can be selected in 2 different ways: by using a query to search the PMC database, or by providing an explicit list of article PMCIDs.
To use a list of PMCIDs, we must pass the path to a file containing the IDs as the--pmcids_fileparameter.
It must contain one ID per line, for example:8217889
7518235
7500239
7287136
7395771
7154153Note these must be PubMedCentral IDs,notPubMed IDs.
Moreover, Some articles can be viewed on the PubMedCentral website, but are not in the Open Access subset.
The publisher of these articles forbids downloading their full text in XML form.pubgetfilters the list of PMCIDs and only downloads those that are in the Open Access subset.
When we use a query instead of a PMCID list, only articles in the Open Access subset are considered.If we use a query instead, we do not use the--pmcids_fileoption, but either--queryor--query_file.
Everything else works in the same way, and the rest of this documentation relies on an example that uses a query.We must first define our query, with which Pubmed Central will be searched for articles.
It can be simple such asfMRI, or more specific such asfMRI[Abstract] AND (2000[PubDate] : 2022[PubDate]).
You can build the query using thePMC advanced search interface.
For more information seethe E-Utilities help.
Some examples are provided in thepubgetgit repository, indocs/example_queries.The query can be passed either as a string on the command-line with-qor--queryor by passing the path of a text file containing the query with-for--query_file.If we have an NCBI API key (see details in theE-utilities documentation), we can provide it through theNCBI_API_KEYenvironment variable or through the--api_keycommand line argument (the latter has higher precedence).We must also specify the directory in which allpubgetdata will be stored.
It can be provided either as a command-line argument (as in the examples below), or by exporting thePUBGET_DATA_DIRenvironment variable.
Subdirectories will be created for each different query.
In the following we suppose we are storing our data in a directory calledpubget_data.We can thus download all articles with "fMRI" in their title published in 2019 by running:pubget download -q "fMRI[Title] AND (2019[PubDate] : 2019[PubDate])" pubget_dataNote:writing the query in a file rather than passing it as an argument is more convenient for complex queries, for example those that contain whitespace, newlines or quotes.
By storing it in a file we do not need to take care to quote or escape characters that would be interpreted by the shell.
In this case we would store our query in a file, sayquery.txt:fMRI[Title] AND (2019[PubDate] : 2019[PubDate])and runpubget download -f query.txt pubget_dataAfter running this command, these are the contents of our data directory:· pubget_data
  └── query_3c0556e22a59e7d200f00ac8219dfd6c
      ├── articlesets
      │   ├── articleset_00000.xml
      │   └── info.json
      └── query.txtpubgethas created a directory for this query,query_3c0556e22a59e7d200f00ac8219dfd6c— in the following we will call it "the query directory".
Its name contains the md5 checksum of the query (or PMCID list), which is useful forpubgetto reuse the same directory if we run the same query again, but not very helpful for us humans.
Therefore, we can use the--aliascommand-line argument to give this query an alternative name, andpubgetwill create a symbolic link for us.
For example if we run the query above with the added option--alias "fMRI-2019", ourpubget_datadirectory will look like this:· pubget_data
  ├── fMRI-2019 -> query_3c0556e22a59e7d200f00ac8219dd6c
  └── query_3c0556e22a59e7d200f00ac8219dd6cIf we had used a PMCID list instead of a query, the directory name would start withpmcidList_instead ofquery_.If we used a query it will be stored inquery.txt, and if we used a list of PMCIDs, inrequested_pmcids.txt, in the query directory.Inside the query directory, the results of the bulk download are stored in thearticlesetssubdirectory.
The articles themselves are in XML files bundling up to 500 articles calledarticleset_*.xml.
Here there is only one because the search returned less than 500 articles.Some information about the download is stored ininfo.json.
In particular,is_completeindicates if all articles matching the search have been downloaded.
If the download was interrupted, some batches failed to download, or the number of results was limited by using the--n_docsparameter,is_completewill befalseand the exit status of the program will be 1.
You may want to re-run the command before moving on to the next step if the download is incomplete.If we run the same query again, only missing batches will be downloaded.
If we want to force re-running the search and downloading the whole data we need to remove thearticlesetsdirectory.Step 2: extracting articles from bulk downloadThis step is executed by thepubget extract_articlescommand.Once our download is complete, we extract articles and store each of them in a separate directory.
To do so, we pass thearticlesetsdirectory created by thepubget downloadcommand in step 1:pubget extract_articles pubget_data/query_3c0556e22a59e7d200f00ac8219dfd6c/articlesetsThis creates anarticlessubdirectory in the query directory, containing the articles.
To avoid having a large number of files in a single directory when there are many articles, which can be problematic on some filesystems, the articles are spread over many subdirectories.
The names of these subdirectories range from000tofffand an article goes in the subdirectory that matches the first 3 hexidecimal digits of the md5 hash of itspmcid.Our data directory now looks like this (with many articles omitted for conciseness):· pubget_data
  └── query_3c0556e22a59e7d200f00ac8219dfd6c
      ├── articles
      │   ├── 019
      │   │   └── pmcid_6759467
      │   │       ├── article.xml
      │   │       └── tables
      │   │           └── tables.xml
      │   ├── 01f
      │   │   └── pmcid_6781806
      │   │       ├── article.xml
      │   │       └── tables
      │   │           ├── table_000.csv
      │   │           ├── table_000_info.json
      │   │           ├── table_001.csv
      │   │           ├── table_001_info.json
      │   │           └── tables.xml
      │   ├── ...
      │   └── info.json
      └── articlesetsNote that the subdirectories such asarticles/01fcan contain one or more articles, even though the examples that appear here only contain one.Each article directory, such asarticles/01f/pmcid_6781806, contains:article.xml: the XML file containing the full article in its original format.atablessubdirectory, containing:tables.xml: all the article's tables, each provided in 2 formats: its original version, and converted to XHTML using theDocBookstylesheets.For each table, a CSV file containing the extracted data and a JSON file providing information such as the table label, id, caption, andn_header_rows, the number of rows at the start of the CSV that should be treated as part of the table header.If the download and article extraction were successfully run and we run the same query again, the article extraction is skipped.
If we want to force re-running the article extraction we need to remove thearticlesdirectory (or theinfo.jsonfile it contains).Step 3: extracting data from articlesThis step is executed by thepubget extract_datacommand.It creates another directory that contains CSV files, containing the text, metadata and coordinates extracted from all the articles.If we use the--articles_with_coords_onlyoption, only articles in whichpubgetfinds stereotactic coordinates are kept.
The name of the resulting directory will reflect that choice.We pass the path of thearticlesdirectory created bypubget extract_articlesin the previous step to thepubget extract_datacommand:pubget extract_data --articles_with_coords_only pubget_data/query_3c0556e22a59e7d200f00ac8219dfd6c/articles/Our data directory now contains (ommitting the contents of the previous steps):· pubget_data
  └── query_3c0556e22a59e7d200f00ac8219dfd6c
      ├── articles
      ├── articlesets
      └── subset_articlesWithCoords_extractedData
          ├── authors.csv
          ├── coordinates.csv
          ├── coordinate_space.csv
          ├── info.json
          ├── links.csv
          ├── neurovault_collections.csv
          ├── neurovault_images.csv
          ├── metadata.csv
          └── text.csvIf we had not used--articles_with_coords_only, the new subdirectory would be namedsubset_allArticles_extractedDatainstead.metadata.csvcontains one row per article, with some metadata:pmcid(PubMed Central ID),pmid(PubMed ID),doi,title,journal,publication_yearandlicense. Note some values may be missing (for example not all articles have apmidordoi).authors.csvcontains one row per article per author.
Fields arepmcid,surname,given-names.text.csvcontains one row per article.
The first field is thepmcid, and the other fields aretitle,keywords,abstract, andbody, and contain the text extracted from these parts of the article.links.csvcontains the external links found in the articles.
The fields arepmcid,ext-link-type(the type of link, for example "uri", "doi"), andhref(usually an URL).neurovault_collections.csvandneurovault_images.csv:NeuroVaultcollection and image IDs that could be extracted from links in the articles, if any.coordinates.csvcontains one row for each(x, y, z)stereotactic coordinate found in any article.
Its fields are thepmcidof the article, the table label and id the coordinates came from, andx,y,z.coordinate_space.csvhas fieldspmcidandcoordinate_space.
It contains a guess about the stereotactic space coordinates are reported in, based on a heuristic derived fromneurosynth.
Possible values for the space are the terms used byneurosynth: "MNI", "TAL" (for Talairach space), and "UNKNOWN".The different files can be joined on thepmcidfield.If all steps up to data extraction were successfully run and we run the same query again, the data extraction is skipped.
If we want to force re-running the data extraction we need to remove the corresponding directory (or theinfo.jsonfile it contains).Optional step: extracting a new vocabularyThis step is executed by thepubget extract_vocabularycommand.
When running the full pipeline this step is optional: we must use the--extract_vocabularyoption for it to be executed.It builds a vocabulary of all the words and 2-grams (groups of 2 words) that appear in the downloaded text, and computes their document frequency (the proportion of documents in which a term appears).pubget extract_vocabulary pubget_data/query_3c0556e22a59e7d200f00ac8219dfd6c/subset_articlesWithCoords_extractedDataThe vocabulary is stored in a csv file in a new directory.
There is no header and the 2 columns are the term and its document frequency.· pubget_data
  └── query_3c0556e22a59e7d200f00ac8219dfd6c
      ├── articles
      ├── articlesets
      ├── subset_articlesWithCoords_extractedData
      └── subset_articlesWithCoords_extractedVocabulary
          ├── info.json
          └── vocabulary.csvWhen running the whole pipeline (pubget run), if we use the--extract_vocabularyoption and do not provide an explicit value for--vocabulary_file, the freshly-extracted vocabulary is used instead of the defaultneuroqueryone for computing TFIDF features (see next step).Optional step: vectorizing (computing TFIDF features)This step is executed by thepubget vectorizecommand.
When running the full pipeline this step is optional: we must use the--vectorize_textoption for it to be executed.
However, if any of the subsequent steps that rely on TFIDF features (NeuroQuery, NeuroSynth or NiMARE steps, see below) are requested, this step is always run and--vectorize_textis ignored.
This step is also run whenever we use the--vocabulary_fileoption.Some large-scale meta-analysis methods such asNeuroSynthandNeuroQueryrely onTFIDF featuresto represent articles' text.
The last step before we can apply these methods is therefore to extract TFIDF features from the text we obtained in the previous step.TFIDF features rely on a predefined vocabulary (set of terms or phrases).
Each dimension of the feature vector corresponds to a term in the vocabulary and represents the importance of that term in the encoded text.
This importance is an increasing function of theterm frequency(the number of time the term occurs in the text divided by the length of the text) and a decreasing function of thedocument frequency(the total number of times the term occurs in the whole corpus or dataset).To extract the TFIDF features we must therefore choose a vocabulary.By default,pubgetwill download and use the vocabulary used byneuroquery.org.If we use the--extract_vocabularyoption, a new vocabulary is created from the downloaded text and used for computing TFIDF features (see "extracting a new vocabulary" below).If we want to use a different vocabulary we can specify it with the--vocabulary_fileoption.
This file will be parsed as a CSV file with no header, whose first column contains the terms.
Other columns are ignored.We also pass topubget vectorizethe directory containing the text we want to vectorize, created bypubget extract_datain step 3 (here we are using the default vocabulary):pubget vectorize pubget_data/query_3c0556e22a59e7d200f00ac8219dfd6c/subset_articlesWithCoords_extractedData/This creates a new directory whose name reflects the data source (whether all articles are kept or only those with coordinates) and the chosen vocabulary (e6f7a7e9c6ebc4fb81118ccabfee8bd7is the md5 checksum of the contents of the vocabulary file, concatenated with those of the vocabulary mapping file, see "vocabulary mapping" below):· pubget_data
  └── query_3c0556e22a59e7d200f00ac8219dfd6c
      ├── articles
      ├── articlesets
      ├── subset_articlesWithCoords_extractedData
      └── subset_articlesWithCoords-voc_e6f7a7e9c6ebc4fb81118ccabfee8bd7_vectorizedText
          ├── abstract_counts.npz
          ├── abstract_tfidf.npz
          ├── body_counts.npz
          ├── body_tfidf.npz
          ├── feature_names.csv
          ├── info.json
          ├── keywords_counts.npz
          ├── keywords_tfidf.npz
          ├── merged_tfidf.npz
          ├── pmcid.txt
          ├── title_counts.npz
          ├── title_tfidf.npz
          ├── vocabulary.csv
          └── vocabulary.csv_voc_mapping_identity.jsonThe extracted features are stored in.npzfiles that can be read for example withscipy.sparse.load_npz.These files contain matrices of shape(n_docs, n_features), wheren_docsis the number of documents andn_featuresthe number of terms in the vocabulary.
Thepmcidcorresponding to each row is found inpmcid.txt, and the term corresponding to each column is found in the first column offeature_names.csv.feature_names.csvhas no header; the first column contains terms and the second one contains their document frequency.For each article part ("title", "keywords", "abstract" and "body"), we get thecountswhich hold the raw counts (the number of times each word occurs in that section), and thetfidfwhich hold the TFIDF features (the counts divided by article length and log document frequency).
Moreover,merged_tfidfcontains the mean TFIDF computed across all article parts.If all steps up to vectorization were successfully run and we run the same query again, the vectorization is skipped.
If we want to force re-running the vectorization we need to remove the corresponding directory (or theinfo.jsonfile it contains).Vocabulary mapping: collapsing synonymsIt is possible to instruct the tokenizer (that extracts words from text) to collapse some pairs of terms that have the same meaning but different spellings, such as "brainstem" and "brain stem".This is done through a JSON file that contains a mapping of the form{term: replacement}.
For example if it contains{"brain stem": "brainstem"}, "brain stem" will be discarded from the vocabulary and every occurrence of "brain stem" will be counted as an occurrence of "brainstem" instead.
To be found bypubget, this vocabulary mapping file must be in the same directory as the vocabulary file, and its name must be the vocabulary file's name with_voc_mapping_identity.jsonappended: for examplevocabulary.csv,vocabulary.csv_voc_mapping_identity.json.When a vocabulary mapping is provided, a shorter vocabulary is therefore created by removing redundant words.
The TFIDF and word counts computed bypubgetcorrespond to the shorter vocabulary, which is stored along with its document frequencies infeature_names.csv.vocabulary.csvcontains the document frequencies of the original (full, longer) vocabulary.
Avocabulary.csv_voc_mapping_identity.jsonfile is always created bypubget, but if no vocabulary mapping was used, that file contains an empty mapping ({}) andvocabulary.csvandfeature_names.csvare identical.The vocabulary mapping is primarily used by theneuroquerypackage and its tokenization pipeline, and you can safely ignore this – just remember that the file providing the terms corresponding to the TFIDFfeaturesisfeature_names.csv.Optional step: fitting a NeuroQuery encoding modelThis step is executed by thepubget fit_neuroquerycommand.
When running the full pipeline it is optional: we must use the--fit_neuroqueryoption for it to be executed.In this step, once the TFIDF features and the coordinates have been extracted from downloaded articles, they are used to train a NeuroQuery encoding model — the same type of model that is exposed atneuroquery.org.
Details about this model are provided inthe NeuroQuery paperand the documentation for theneuroquery package.Note: for this model to give good results a large dataset is needed, ideally close to 10,000 articles (with coordinates).We pass the_vectorizedTextdirectory created bypubget vectorize:pubget fit_neuroquery pubget_data/query_3c0556e22a59e7d200f00ac8219dfd6c/subset_articlesWithCoords-voc_e6f7a7e9c6ebc4fb81118ccabfee8bd7_vectorizedTextThis creates a directory whose name ends with_neuroqueryModel:· pubget_data
  └── query_3c0556e22a59e7d200f00ac8219dfd6c
      ├── articles
      ├── articlesets
      ├── subset_articlesWithCoords_extractedData
      ├── subset_articlesWithCoords-voc_e6f7a7e9c6ebc4fb81118ccabfee8bd7_neuroqueryModel
      │   ├── app.py
      │   ├── info.json
      │   ├── neuroquery_model
      │   │   ├── corpus_metadata.csv
      │   │   ├── corpus_tfidf.npz
      │   │   ├── mask_img.nii.gz
      │   │   ├── regression
      │   │   │   ├── coef.npy
      │   │   │   ├── intercept.npy
      │   │   │   ├── M.npy
      │   │   │   ├── original_n_features.npy
      │   │   │   ├── residual_var.npy
      │   │   │   └── selected_features.npy
      │   │   ├── smoothing
      │   │   │   ├── smoothing_weight.npy
      │   │   │   └── V.npy
      │   │   ├── vocabulary.csv
      │   │   └── vocabulary.csv_voc_mapping_identity.json
      │   ├── README.md
      │   └── requirements.txt
      └── subset_articlesWithCoords-voc_e6f7a7e9c6ebc4fb81118ccabfee8bd7_vectorizedTextYou do not need to care about the contents of theneuroquery_modelsubdirectory, that is data used by theneuroquerypackage.
Just know that it can be used to initialize aneuroquery.NeuroQueryModelwith:fromneuroqueryimportNeuroQueryModelmodel=NeuroQueryModel.from_data_dir("neuroquery_model")Theneuroquerydocumentation provides information and examples on how to use this model.Visualizing the newly trained model in an interactive web pageIt is easy to interact with the model through a small web (Flask) application.
From inside the[...]_neuroqueryModeldirectory, just runpip install -r requirements.txtto installflask,nilearnandneuroquery.
Then runflask runand point your web browser tohttps://localhost:5000: you can play with a local, simplified version ofneuroquery.orgbuilt with the data we just downloaded.Optional step: running a NeuroSynth meta-analysisThis step is executed by thepubget fit_neurosynthcommand.
When running the full pipeline it is optional: we must use the--fit_neurosynthoption for it to be executed.In this step, once the TFIDF features and the coordinates have been extracted from downloaded articles, they are used to run meta-analyses using NeuroSynth's "association test" method: a Chi-squared test of independence between voxel activation and term occurrences.
Seethe NeuroSynth paperandneurosynth.org, as well as theneurosynthandNiMAREdocumentation pages for more information.We pass the_vectorizedTextdirectory created bypubget vectorize:pubget fit_neurosynth pubget_data/query_3c0556e22a59e7d200f00ac8219dfd6c/subset_articlesWithCoords-voc_e6f7a7e9c6ebc4fb81118ccabfee8bd7_vectorizedTextThis creates a directory whose name ends with_neurosynthResults:· pubget_data
  └── query_3c0556e22a59e7d200f00ac8219dfd6c
      ├── articles
      ├── articlesets
      ├── subset_articlesWithCoords_extractedData
      ├── subset_articlesWithCoords-voc_e6f7a7e9c6ebc4fb81118ccabfee8bd7_neurosynthResults
      │   ├── app.py
      │   ├── info.json
      │   ├── metadata.csv
      │   ├── neurosynth_maps
      │   │   ├── aberrant.nii.gz
      │   │   ├── abilities.nii.gz
      │   │   ├── ability.nii.gz
      │   │   └── ...
      │   ├── README.md
      │   ├── requirements.txt
      │   ├── terms.csv
      │   └── tfidf.npz
      └── subset_articlesWithCoords-voc_e6f7a7e9c6ebc4fb81118ccabfee8bd7_vectorizedTextThe meta-analytic maps for all the terms in the vocabulary can be found in theneurosynth_mapssubdirectory.Visualizing the meta-analytic maps in an interactive web pageIt is easy to interact with the NeuroSynth maps through a small web (Flask) application.
From inside the[...]_neurosynthResultsdirectory, just runpip install -r requirements.txtto installflaskand other dependencies.
Then runflask runand point your web browser tohttps://localhost:5000: you can search for a term and see the corresponding brain map and the documents that mention it.Optional step: preparing articles for annotation withlabelbuddyThis step is executed by thepubget extract_labelbuddy_datacommand.
When running the full pipeline this step is optional: we must use the--labelbuddyor--labelbuddy_batch_sizeoption for it to be executed.It prepares the articles whose data was extracted for annotation withlabelbuddy.We pass the_extractedDatadirectory created bypubget extract_data:pubget extract_labelbuddy_data pubget_data/query_3c0556e22a59e7d200f00ac8219dfd6c/subset_articlesWithCoords_extractedDataThis creates a directory whose name ends withlabelbuddyDatacontaining the batches of documents in JSONL format (in this case there is a single batch):· pubget_data
  └── query_3c0556e22a59e7d200f00ac8219dfd6c
      ├── articles
      ├── articlesets
      ├── subset_articlesWithCoords_extractedData
      ├── subset_articlesWithCoords_labelbuddyData
      │   ├── batch_info.csv
      │   ├── documents_00001.jsonl
      │   └── info.json
      └── subset_articlesWithCoords-voc_e6f7a7e9c6ebc4fb81118ccabfee8bd7_vectorizedTextThe documents can be imported intolabelbuddyusing the GUI or with:labelbuddy mydb.labelbuddy --import-docs documents_00001.jsonlSee thelabelbuddy documentationfor details.The CSV filebatch_info.csvprovides the location of each article in the.jsonlfiles: its columns arepmcid,file_name(the name of the.jsonlfile containing that article) andline(the line number that contains that article, first line is 0).Optional step: creating a NiMARE datasetThis step is executed by thepubget extract_nimare_datacommand.
When running the full pipeline this step is optional: we must use the--nimareoption for it to be executed.It creates aNiMAREdataset for the extracted data in JSON format.
See the NiMAREdocumentationfor details.We pass the_vectorizedTextdirectory created bypubget vectorize:pubget extract_nimare_data pubget_data/query_3c0556e22a59e7d200f00ac8219dfd6c/subset_articlesWithCoords-voc_e6f7a7e9c6ebc4fb81118ccabfee8bd7_vectorizedTextThe resulting directory contains animare_dataset.jsonfile that can be used to initialize animare.Dataset.· pubget_data
  └── query_3c0556e22a59e7d200f00ac8219dfd6c
      ├── articles
      ├── articlesets
      ├── subset_articlesWithCoords_extractedData
      ├── subset_articlesWithCoords-voc_e6f7a7e9c6ebc4fb81118ccabfee8bd7_nimareDataset
      │   ├── info.json
      │   └── nimare_dataset.json
      └── subset_articlesWithCoords-voc_e6f7a7e9c6ebc4fb81118ccabfee8bd7_vectorizedTextUsing this option requires installing NiMARE, which is not installed by default withpubget.
To use this option, install NiMARE separately withpip install nimareor installpubgetwithpip install "pubget[nimare]"Full pipelineWe can run all steps in one command by usingpubget run.The full procedure described above could be run by executing:pubget run -q "fMRI[Title] AND (2019[PubDate] : 2019[PubDate])" \
    --articles_with_coords_only                               \
    pubget_data(The output directory,pubget_data, could also be provided by exporting thePUBGET_DATA_DIRenvironment variable instead of passing it on the command line.)If we also want to apply the optional steps:pubget run -q "fMRI[Title] AND (2019[PubDate] : 2019[PubDate])" \
    --articles_with_coords_only                               \
    --fit_neuroquery                                          \
    --labelbuddy                                              \
    --nimare                                                  \
    pubget_data(remember that--nimarerequires NiMARE to be installed).Here also, steps that had already been completed are skipped; we need to remove the corresponding directories if we want to force running these steps again.Seepubget run --helpfor a description of all options.LoggingBy defaultpubgetcommands report their progress by writing to the standard streams.
In addition, they can write log files if we provide the--log_dircommand-line argument, or if we define thePUBGET_LOG_DIRenvironment variable (the command-line argument has higher precedence).
If this log directory is specified, a new log file with a timestamp is created and all the output is written there as well.Writing pluginsIt is possible to write plugins and defineentry pointsto add functionality that is automatically executed whenpubgetis run.The name of the entry point should bepubget.plugin_actions.
It must be a function taking no arguments and returning a dictionary with keyspipeline_stepsandcommands.
The corresponding values must be lists of processing step objects, that must implement the interface defined bypubget.PipelineStepandpubget.Commandrespectively (their types do not need to inherit from these classes).All steps inpipeline_stepswill be run whenpubget runis used.
All steps instandalone_stepswill be added as additional pubget commands; for example if thenameof a standalone step ismy_plugin, thepubget my_plugincommand will become available.An example plugin that can be used as a template, and more details, are provided in thepubgetgit repository, indocs/example_plugin.ContributingFeedback and contributions are welcome.
Development happens at thepubget GitHub repositiory.
To install the dependencies required for development, from the directory where you clonedpubget, run:pip install -e ".[dev]"The tests can be run withmake test_all, ormake test_coverageto report test coverage.
The documentation can be rendered withmake doc.make run_full_pipelineruns the fullpubgetpipeline on a query returning a realistic number of results (fMRI[title]).Python APIpubgetis mostly intended for use as a command-line tool.
However, it is also a Python package and its functionality can be used in Python programs.
The Python API closely reflects the command-line programs described above.The Python API is described on thepubgetwebsite.pubgetreleases0.0.8NeuroVault image and collection IDs are extracted and stored in theextractedDatadirectory.When a request fails, the request and response (if there is one) are dumped in thearticlesets/failed_requests_dumps/directory for easier diagnosis of issues with Entrez.The download step is more robust to malformed XML files sent by efetch.fcgiAn--aliascommand-line option has been added to create a symlink with a human-readable name to a query's output directory.0.0.7query.txtandrequested_pmcids.txthave moved from thearticlesets/subdirectory to the root of the query or pmcid list's directory.labelbuddyoutput now contains abatch_info.csvfile providing the.jsonlfile and line position of eachpmcid.The environment variable to provide the NCBI API key has been renamed fromPUBGET_API_KEYtoNCBI_API_KEY.When downloading a list of PMCIDs (with--pmcids_fileoption), the list is now filtered to keep only articles that are in the PMC Open Access subset.0.0.6The text vectorization (TFIDF) step is now optional.
It is run when using the options--vectorize_textor--vocabulary_file, or when subsequent steps that need the TFIDF (neurosynth, neuroquery, nimare) are requested.0.0.5nqdcrenamedpubget; all symbols, paths, environment variables etc. have been adapted accordingly.nqdcreleasespubgetused to be callednqdc.
Thenqdcpackage is deprecated and should no longer be used.0.0.3downloading batches of articles now a bit more robust because response content is check and nqdc retries up to 4 times if a request fails.query output directories renamed from query- to query_ to follow a bids-like convention.It is now possible to download an explicit list of PMCIDs rather than a PMC query to select the articles to download.
See the--pmcids_fileparameter or the "Usage/Step 1" part of the documentation.Articles inquery_dir/articlesare now each stored in a separate subdirectory; that also contains atablessubdirectory with the tables extracted from the article.External links are now extracted from articles and stored inlinks.csvduring theextract_datastep.0.0.2Changes to the command-line interface; now all in one commandnqdc;nqdc_full_pipelinebecomesnqdc run.Add several commands/steps:Creating a NiMARE dataset.Preparing documents for annotation with labelbuddy.Extracting a new vocabulary.Fitting neuroquery.Running neurosynth analysis.Possibility to create plugins.Parallelize data extraction; several improvements to text & coordinate extraction0.0.1First release; tentative API for downloading PMC data, extracting articles,
extracting data, and vectorizing text.MIT LicenseCopyright (c) 2022 Jérôme DockèsPermission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.