Package: temporal-taggers
BERT Got a Date: Introducing Transformers to Temporal TaggingSatya Almasian*, Dennis Aumiller*, and Michael GertzHeidelberg UniversityContact us via:<lastname>@informatik.uni-heidelberg.deCode and data for the paperBERT Got a Date: Introducing Transformers to Temporal Tagging.Check out our models on Huggingface!Temporal tagging is the task of identification of temporal mentions in text; these expressions can be further divided into different type categories, which is what we refer to as expression (type) classification.
This repository describes two different types of transformer-based temporal taggers, which are both additionally capable of expression classification.
We follow the TIMEX3 schema definitions in their styling and expression classes (notably, the latter are one ofTIME, DATE, SET, DURATION). The available data sources for temporal tagging are in the TimeML format, which is essentially a form of XML with tags encapsulating temporal expressions.An example can be seen below:Due to lockdown restrictions, <TIMEX3 tid="t1" type="DATE" value="2020">2020</TIMEX3> 
might go down as the worst economic <TIMEX3 tid="t2" type="DURATION" value="P1Y">year</TIMEX3>
in over <TIMEX3 tid="t3" type="DURATION" value="P1DE">a decade</TIMEX3>.For more sample instances, look at the content ofdata.zip. Refer to the README file in the respective unzipped folder for more information.InstallationYou can now install the underlying models by simply runningpython3-mpipinstall.after cloning this repository; this will automatically install all necessary dependencies. We're working on making the installation even easier by providing a package on PyPI, stay tuned for more!This repository contains code for data preparation and training of a seq2seq model (encoder-decoder architectured initialized from encoder-only architectures, specifically BERT or RoBERTa), as well as three token classification encoders (BERT-based).The output of the models discussed in the paper is in theresultsfolder. Refer to the README file in the folder for more information.The zip files containing data & results are uploaded using Git LFS and require it as an additional library to work properly.To install Git LFS on Ubuntu:Download/Install Git LFS (sudo apt-get install git-lfs, or download fromhttps://git-lfs.github.com/)Rungit lfs installIf you want to generate data with Heideltime, you will additionally have to set uppython_heideltimeas a wrapper.
Due to the project nature of Heideltime, this installation has to be performed manually.Data PreparationThe scripts to generate training data is in the subfolderdata_preparation.
For more usage information, refer to the README file in the subfolder.
The data used for training and evaluation are provided in zipped form indata.zip.EvaluationFor evaluation, we use a slightly modified version of the TempEval-3 evaluation toolkit (original source here).
We refactored the code to be compatible with Python3, and incorporated additional evaluation metrics, such as a confusion matrix for type classification.
We cross-referenced results to ensure full backward-compatibility and all runs result in the exact same results for both versions.
Our adjusted code, as well as scripts to convert the output of transformer-based tagging models are in theevaluationsubfolder.
For more usage information, refer to the README file in the respective subfolder.Temporal modelsWe train and evaluate two types of setups for joint temporal tagging and classification:Token Classification:We define three variants of simple token classifiers; all of them are based on Huggingface'sBertForTokenClassification. We adapt their "token classification for named entity recognition script" to train these models. All the models are trained usingbert-base-uncasedas their pre-trained checkpoint.Text-to-Text Generation (Seq2Seq):These models are encoder-decoder architectures using BERT or RoBERTa for initial weights. We use Huggingface'sEncoderDecoderclass for initialization of weights, starting frombert-base-uncasedandroberta-base, respectively.Seq2seqTo train the seq2seq models, userun_seq2seq_bert_roberta.py. Example usage is as follows:python3run_seq2seq_bert_roberta.py--model_nameroberta-base--pre_trainTrue\--model_dir./test--train_data./data/seq2seq/train/tempeval_train.json\--eval_data./data/seq2seq/test/tempeval_test.json--num_gpu2--num_train_epochs1\warmup_steps100--seed0--eval_steps200Which trains a roberta2roberta model defined bymodel_namefornum_train_epochsepochs on the gpu with IDnum_gpu.
The random seed is set byseedand the number of warmup steps bywarmup_steps.
Train data should be specified intrain_dataandmodel_dirdefines where the model is saved.
seteval_dataif you want intermediate evaluation defined byeval_steps.
If thepre_trainflag is set to true it will load the checkpoints from the Huggingface hub and fine-tune on the dataset given.
If thepre_trainis false, we are in the fine-tuning mode, and you can provide the path to the pre-trained model withpretrain_path.
We used thepre_trainmode to train on weakly labeled data provided by the rule-based system of HeidelTime and set thepre_trainto false
for fine-tuning on the benchmark datasets. If you wish to simply fine-tune the benchmark datasets using the Huggingface checkpoints
you can set thepre_trainto ture, as displayed in the example above.
For additional arguments such as length penalty, the number of beams, early stopping, and other model specifications, please refer to the script.Token ClassifiersAs mentioned above all token classifiers are trained using an adaptation of the NER script from Huggingface. To train these models userun_token_classifier.pylike the following example:python3run_token_classifier.py--data_dir/data/temporal/BIO/wikiwars\--labels./data/temporal/BIO/train_staging/labels.txt\--model_name_or_pathbert-base-uncased\--output_dir./fine_tune_wikiwars/bert_tagging_with_date_no_pretrain_8epochs/bert_tagging_with_date_layer_seed_19--max_seq_length512\--num_train_epochs8--per_device_train_batch_size34--save_steps3000--logging_steps300--eval_steps3000\--do_train--do_eval--overwrite_output_dir--seed19--model_date_extra_layerWe usedbert-base-uncasedas the base of all our token classification models for pre-training as defined bymodel_name_or_path.
For fine-tuning on the datasetsmodel_name_or_pathshould point to the path of the pre-trained model.labelsfile is created during data preparation for more information refer to thesubfolder.data_dirpoints to a folder that containstrain.txt,test.txtanddev.txtandoutput_dirpoints to the saving location.
You can define the number of epochs bynum_train_epochs, set the seed withseedand batch size on each GPU withper_device_train_batch_size.
For more information on the parameters refer to theHuggingface script.
In our paper, we introduce 3 variants of token classification, which are defined by flags in the script.
If no flag is set the model trains the vanilla BERT for token classification.
The flagmodel_date_extra_layertrains the model with an extra date layer andmodel_crfadds the extra crf layer.
To train the extra date embedding you need to download the vocabulary file and specify its path in thedate_vocabargument.
The description and model definition of the BERT variants are in foldertemporal_models.
Please refer to the README file for further information. For training different model types on the same data, make sure to remove
the cached dataset, since the feature generation is different for each model type.Load directly from the Huggingface Model HubWe uploaded our best-performing version of each architecture to the Huggingface Model Hub.
The weights for the other four seeding runs are available upon request.
We upload the variants that were fine-tuned on the concatenation ofall threeevaluation sets for better generalization to various domains.
Token classification models are variants without pre-training.
Both seq2seq models are pre-trained on the weakly labelled corpus and fine-tuned on the mixed data.Overall we upload the following five models. For other model configurations and checkpoints please get in contact with us:satyaalmasian/temporal_tagger_roberta2roberta: Our best perfoming model from the paper, an encoder-decoder architecture using RoBERTa.
The model is pre-trained on weakly labeled news articles, tagged with HeidelTime, and fined-tuned on the train set of TempEval-3, Tweets, and Wikiwars.satyaalmasian/temporal_tagger_bert2bert: Our second seq2seq model , an encoder-decoder architecture using BERT.
The model is pre-trained on weakly labeled news articles, tagged with HeidelTime, and fined-tuned on the train set of TempEval-3, Tweets, and Wikiwars.satyaalmasian/temporal_tagger_BERT_tokenclassifier: BERT for token classification model or vanilla BERT model from the paper.
This model is only trained on the train set of TempEval-3, Tweets, and Wikiwars.satyaalmasian/temporal_tagger_DATEBERT_tokenclassifier: BERT for token classification with an extra date embedding, that encodes the reference date of the
document. If the document does not have a reference date, it is best to avoid this model. Moreover, since the architecture
is a modification of a default Huggingface model, the usage is not as straightforward and requires the classes defined in thetemporal_modelmodule. This model is only trained on the train set of TempEval-3, Tweets, and Wikiwars.satyaalmasian/temporal_tagger_BERTCRF_tokenclassifier:BERT for token classification with a CRF layer on the output. Moreover, since the architecture
is a modification of a default Huggingface model, the usage is not as straightforward and requires the classes defined in thetemporal_modelmodule. This model is only trained on the train set of TempEval-3, Tweets, and Wikiwars.In theexamplesmodule, you find two scriptsmodel_hub_seq2seq_examples.pyandmodel_hub_tokenclassifiers_examples.pyfor
seq2seq and token classification examples using the Huggingface model hub. The examples load the models and use them on example sentences
for tagging. The seq2seq example uses the pre-defined post-processing from the tempeval evaluation and contains rules for the cases we came across in the benchmark dataset.
If you plan to use these models on new data, it is best to observe the raw output of the first few samples to detect possible format problems that are easily fixable.
Further fine-tuning of the models is also possible.
For seq2seq models you can simply load the models withtokenizer=AutoTokenizer.from_pretrained("satyaalmasian/temporal_tagger_roberta2roberta")model=EncoderDecoderModel.from_pretrained("satyaalmasian/temporal_tagger_roberta2roberta")and use theDataProcessorfromtemporal_models.seq2seq_utilsto preprocess thejsondataset. The model
can be fine-tuned usingSeq2SeqTrainer(same as inrun_seq2seq_bert_roberta.py).
For token classifiers the model and the tokenizers are loaded as follows:tokenizer=AutoTokenizer.from_pretrained("satyaalmasian/temporal_tagger_BERT_tokenclassifier",use_fast=False)model=BertForTokenClassification.from_pretrained("satyaalmasian/temporal_tagger_BERT_tokenclassifier")Classifiers need a BIO-tagged file that can be loaded usingTokenClassificationDatasetand fine-tuned with the HuggingfaceTrainer.
For more information on the usage of these models refer to their model hub page.CitationIf you use our models in your work, we would appreciate attribution with the following citation:@article{almasian2021bert,
  title={{BERT got a Date: Introducing Transformers to Temporal Tagging}},
  author={Almasian, Satya and Aumiller, Dennis and Gertz, Michael},
  journal={arXiv preprint arXiv:2109.14927},
  url={https://arxiv.org/abs/2109.14927},
  year={2021}
}