Blocklist expansion with deep learning-based models
Creators: name: Nastou, Katerina
name: Koutrouli, Mikaela
name: Pyysalo, Sampo
name: Jensen, Lars Juhl
Publisher: name: Zenodo
Year: 2023

description: Files in this repository
Tagger related files
dictionary-files-tagger-STRINGv12.zip: this directory contains all the dictionary files used for the tagger runs for all Jensenlab resources (including STRINGv12). It also contains files only relevant for this work, namely: curated_local.tsv & curated_global.tsv which are used for running tagger only with curated blocklists, blacklist_terms_over_10M.txt & blacklist_terms_over_10M+auto_only_list.txt which are used for the automatic blocklist runs only, and empty_global.tsv which is used for the run without blocklists and regex.
tagger-no-regex.tar.gz : this directory contains the Jensenlab tagger with the tagger_core.h header updated so as not to use a regex to block things. This version of the tagger has been used only for the no-blocklist run of the paper. In order to set it up one needs to do the following (for more details check the original repo):

wget https://zenodo.org/api/records/10008720/files/tagger-no-regex.tar.gz
tar -xzvf tagger-no-regex.tar.gz
cd tagger-no-regex
make tagcorpus
 
Training, development and test sets
125k-w100_grid_search_set.tar.gz: A small dataset with 125,000 training and 62,500 development examples used to perform a grid search to detect the best set of hyperparameters
12.5M-w100_train_test_set.tar.gz: A large dataset of 12.5 million training and 62,500 testing examples to train the model used for prediction with the set of best hyperparameters identified above 
 
Transformer-based model
bert-base-finetuned-large-set.tar.gz: this is the TensorFlow model fine-tuned on the large dataset (12.5M) that is used for all the prediction runs. The model is finetuned starting from the BioBERT base v1.1 model. 
 
Prediction runs input generation
The command used to run tagger before running predictions using the bert-base-finetuned-large-set model:

gzip -cd `ls -1 pmc/*.en.merged.filtered.tsv.gz` `ls -1r pubmed/*.tsv.gz` | cat dictionary/excluded_documents.txt - | tagger/tagcorpus --threads=16 --autodetect --types=dictionary/curated_types.tsv --entities=dictionary/all_entities.tsv --names=dictionary/all_names_textmining.tsv --groups=dictionary/all_groups.tsv --stopwords=dictionary/curated_global.tsv --local-stopwords=dictionary/curated_local.tsv --type-pairs=dictionary/all_type_pairs.tsv --out-matches=all_matches.tsv
Input documents for large-scale execution: all PubMed abstracts (as of August 2022) and all full-texts available in the PubmedCentral BioC text mining collection (as of April 2022). The files are converted to a tab-delimited format in order to convert the output to a format compatible with the RE system (see below).
Input dictionary files: all the files necessary to execute the command above are available in dictionary-files-tagger-STRINGv12.zip
Tagger output: we filter the results of the tagger run down to gene or gene products, species, diseases, and chemicals with the process described in our GitHub repository 
 
Blocklist files
curated+auto-blocklists.tar.gz: a combined automatically generated and manually curated blocklist used for the curated+auto runs in the paper. This is the list currently used for the tagger runs for all Jensenlab resources
curated-only-blocklists.tar.gz: the manually curated blocklist used for the curated_only runs in the paper
auto-only-blocklists.tar.gz: the automatically generated blocklist used for the auto_only runs in the paper