Package: a2t
Ask2TransformersA Framework for Textual Entailment based Zero Shot text classificationThis repository contains the code for out of the box ready to use zero-shot classifiers among different tasks, such as Topic Labelling or Relation Extraction. It is built on top of 🤗 HuggingFaceTransformerslibrary, so you are free to choose among hundreds of models. You can either, use a dataset specific classifier or define one yourself with just labels descriptions or templates! The repository contains the code for the following publications:📄Ask2Transformers - Zero Shot Domain Labelling with Pretrained Transformersaccepted inGWC2021.📄Label Verbalization and Entailment for Effective Zero- and Few-Shot Relation Extractionaccepted inEMNLP2021📄Textual Entailment for Event Argument Extraction: Zero- and Few-Shot with Multi-Source Learningaccepted as Findings inNAACL2022To get started with the repository consider reading thenewdocumentation!InstallationBy using Pip (check the last release)pipinstalla2tOr by clonning the repositorygitclonehttps://github.com/osainz59/Ask2Transformers.gitcdAsk2Transformers
python-mpipinstall.Demo 🕹️We have realeased a demo on Zero-Shot Information Extraction using Textual Entailment (ZS4IE: A toolkit for Zero-Shot Information Extraction with simple Verbalizations) accepted in theDemo Track of NAACL 2022. The code is publicly availabe on its own GitHub repository:ZS4IE.ModelsAvailable modelsBy default,roberta-large-mnlicheckpoint is used to perform the inference. You can try different models to perform the zero-shot classification, but they need to be finetuned on a NLI task and be compatible with theAutoModelForSequenceClassificationclass from Transformers. For example:roberta-large-mnlijoeddav/xlm-roberta-large-xnlifacebook/bart-large-mnlimicrosoft/deberta-v2-xlarge-mnliComing soon:t5-largelike generative models support.Pre-trained models 🆕We now provide (task specific) pre-trained entailment models to: (1)reproducethe results of the papers and (2)reusethem for new schemas of the same tasks. The models are publicly available on the 🤗 HuggingFace Models Hub.The model name describes the configuration used for training as follows:HiTZ/A2T_[pretrained_model]_[NLI_datasets]_[finetune_datasets]pretrained_model: The checkpoint used for initialization. For example: RoBERTalarge.NLI_datasets: The NLI datasets used for pivot training.S: Standford Natural Language Inference (SNLI) dataset.M: Multi Natural Language Inference (MNLI) dataset.F: Fever-nli dataset.A: Adversarial Natural Language Inference (ANLI) dataset.finetune_datasets: The datasets used for fine tuning the entailment model. Note that for more than 1 dataset the training was performed sequentially. For example: ACE-arg.Some models likeHiTZ/A2T_RoBERTa_SMFA_ACE-arghave been trained marking some information between square brackets ('[['and']]') like the event trigger span. Make sure you follow the same preprocessing in order to obtain the best results.Training your own modelsThere is no special script for fine-tuning your own entailment based models. In our experiments, we have used the publicly availablerun_glue.pypython script (from HuggingFace Transformers). To train your own model, first, you will need to convert your actual dataset in some sort of NLI data, we recommend you to have a look totacred2mnli.pyscript that serves as an example.Tutorials (Notebooks)Coming soon!Results and evaluationTo obtain the results reported in the papers run theevaluation.pyscript with the corresponding configurationfiles. A configuration file containing the task and evaluation information should look like this:{"name":"BabelDomains","task_name":"topic-classification","features_class":"a2t.tasks.text_classification.TopicClassificationFeatures","hypothesis_template":"The domain of the sentence is about {label}.","nli_models":["roberta-large-mnli"],"labels":["Animals","Art, architecture, and archaeology","Biology","Business, economics, and finance","Chemistry and mineralogy","Computing","Culture and society",..."Royalty and nobility","Sport and recreation","Textile and clothing","Transport and travel","Warfare and defense"],"preprocess_labels":true,"dataset":"babeldomains","test_path":"data/babeldomains.domain.gloss.tsv","use_cuda":true,"half":true}Consider reading the papers to access the results.About legacy codeThe old code of this repository has been moved toa2t.legacymodule and is only intended to be use for experimental reproducibility. Please, consider moving to the new code. If you need help read the newdocumentationor post an Issue on GitHub.CitationCite this paper if you want to cite stuff related to Relation Extraction, etc.@inproceedings{sainz-etal-2021-label,title="Label Verbalization and Entailment for Effective Zero and Few-Shot Relation Extraction",author="Sainz, Oscar  andLopez de Lacalle, Oier  andLabaka, Gorka  andBarrena, Ander  andAgirre, Eneko",booktitle="Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing",month=nov,year="2021",address="Online and Punta Cana, Dominican Republic",publisher="Association for Computational Linguistics",url="https://aclanthology.org/2021.emnlp-main.92",pages="1199--1212",abstract="Relation extraction systems require large amounts of labeled examples which are costly to annotate. In this work we reformulate relation extraction as an entailment task, with simple, hand-made, verbalizations of relations produced in less than 15 min per relation. The system relies on a pretrained textual entailment engine which is run as-is (no training examples, zero-shot) or further fine-tuned on labeled examples (few-shot or fully trained). In our experiments on TACRED we attain 63{\%} F1 zero-shot, 69{\%} with 16 examples per relation (17{\%} points better than the best supervised system on the same conditions), and only 4 points short to the state-of-the-art (which uses 20 times more training data). We also show that the performance can be improved significantly with larger entailment models, up to 12 points in zero-shot, allowing to report the best results to date on TACRED when fully trained. The analysis shows that our few-shot systems are specially effective when discriminating between relations, and that the performance difference in low data regimes comes mainly from identifying no-relation cases.",}Cite this paper if you want to cite stuff related with topic labelling (A2TDomains or our paper results).@inproceedings{sainz-rigau-2021-ask2transformers,title="{A}sk2{T}ransformers: Zero-Shot Domain labelling with Pretrained Language Models",author="Sainz, Oscar  andRigau, German",booktitle="Proceedings of the 11th Global Wordnet Conference",month=jan,year="2021",address="University of South Africa (UNISA)",publisher="Global Wordnet Association",url="https://www.aclweb.org/anthology/2021.gwc-1.6",pages="44--52",abstract="In this paper we present a system that exploits different pre-trained Language Models for assigning domain labels to WordNet synsets without any kind of supervision. Furthermore, the system is not restricted to use a particular set of domain labels. We exploit the knowledge encoded within different off-the-shelf pre-trained Language Models and task formulations to infer the domain label of a particular WordNet definition. The proposed zero-shot system achieves a new state-of-the-art on the English dataset used in the evaluation.",}