Package: gpl
Generative Pseudo Labeling (GPL)GPL is an unsupervised domain adaptation method for training dense retrievers. It is based on query generation and pseudo labeling with powerful cross-encoders. To train a domain-adapted model, it needs only the unlabeled target corpus and can achieve significant improvement over zero-shot models.For more information, checkout our publication:GPL: Generative Pseudo Labeling for Unsupervised Domain Adaptation of Dense Retrieval(NAACL 2022)For reproduction, please refer to thissnapshot branch.InstallationOne can either install GPL viapippipinstallgplor viagit clonegitclonehttps://github.com/UKPLab/gpl.git&&cdgpl
pipinstall-e.Meanwhile, please make sure thecorrect version of PyTorchhas been installed according to your CUDA version.UsageGPL accepts data in theBeIR-format. For example, we can download theFiQAdataset hosted by BeIR:wgethttps://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/fiqa.zip
unzipfiqa.zip
head-n2fiqa/corpus.jsonl# One can check this data format. Actually GPL only need this `corpus.jsonl` as data input for training.Then we can either use thepython -mfunction to run GPL training directly:exportdataset="fiqa"python-mgpl.train\--path_to_generated_data"generated/$dataset"\--base_ckpt"distilbert-base-uncased"\--gpl_score_function"dot"\--batch_size_gpl32\--gpl_steps140000\--new_size-1\--queries_per_passage-1\--output_dir"output/$dataset"\--evaluation_data"./$dataset"\--evaluation_output"evaluation/$dataset"\--generator"BeIR/query-gen-msmarco-t5-base-v1"\--retrievers"msmarco-distilbert-base-v3""msmarco-MiniLM-L-6-v3"\--retriever_score_functions"cos_sim""cos_sim"\--cross_encoder"cross-encoder/ms-marco-MiniLM-L-6-v2"\--qgen_prefix"qgen"\--do_evaluation\# --use_amp   # Use this for efficient training if the machine supports AMP# One can run `python -m gpl.train --help` for the information of all the arguments# To reproduce the experiments in the paper, set `base_ckpt` to "GPL/msmarco-distilbert-margin-mse" (https://huggingface.co/GPL/msmarco-distilbert-margin-mse)or import GPL's trainining method in a python script:importgpldataset='fiqa'gpl.train(path_to_generated_data=f"generated/{dataset}",base_ckpt="distilbert-base-uncased",# base_ckpt='GPL/msmarco-distilbert-margin-mse',# The starting checkpoint of the experiments in the papergpl_score_function="dot",# Note that GPL uses MarginMSE loss, which works with dot-productbatch_size_gpl=32,gpl_steps=140000,new_size=-1,# Resize the corpus to `new_size` (|corpus|) if needed. When set to None (by default), the |corpus| will be the full size. When set to -1, the |corpus| will be set automatically: If QPP * |corpus| <= 250K, |corpus| will be the full size; else QPP will be set 3 and |corpus| will be set to 250K / 3queries_per_passage=-1,# Number of Queries Per Passage (QPP) in the query generation step. When set to -1 (by default), the QPP will be chosen automatically: If QPP * |corpus| <= 250K, then QPP will be set to 250K / |corpus|; else QPP will be set 3 and |corpus| will be set to 250K / 3output_dir=f"output/{dataset}",evaluation_data=f"./{dataset}",evaluation_output=f"evaluation/{dataset}",generator="BeIR/query-gen-msmarco-t5-base-v1",retrievers=["msmarco-distilbert-base-v3","msmarco-MiniLM-L-6-v3"],retriever_score_functions=["cos_sim","cos_sim"],# Note that these two retriever model work with cosine-similaritycross_encoder="cross-encoder/ms-marco-MiniLM-L-6-v2",qgen_prefix="qgen",# This prefix will appear as part of the (folder/file) names for query-generation results: For example, we will have "qgen-qrels/" and "qgen-queries.jsonl" by default.do_evaluation=True,# use_amp=True   # One can use this flag for enabling the efficient float16 precision)One can also refer tothis toy exampleon Google Colab for better understanding how the code works.How does GPL work?The workflow of GPL is shown as follows:GPL first use a seq2seq (we useBeIR/query-gen-msmarco-t5-base-v1by default) model to generatequeries_per_passagequeries for each passage in the unlabeled corpus. The query-passage pairs are viewed aspositive examplesfor training.Result files (under path$path_to_generated_data): (1)${qgen}-qrels/train.tsv, (2)${qgen}-queries.jsonland also (3)corpus.jsonl(copied from$evaluation_data/);Then, it runs negative mining with the generated queries as input on the target corpus. The mined passages will be viewed asnegative examplesfor training. One can specify any dense retrievers (SBERTorHuggingface/transformerscheckpoints, we usemsmarco-distilbert-base-v3+msmarco-MiniLM-L-6-v3by default) or BM25 to the argumentretrieversas the negative miner.Result file (under path$path_to_generated_data): hard-negatives.jsonl;Finally, it does pseudo labeling with the powerful cross-encoders (we usecross-encoder/ms-marco-MiniLM-L-6-v2by default.) on the query-passage pairs that we have so far (for both positive and negative examples).Result file (under path$path_to_generated_data):gpl-training-data.tsv. It contains (gpl_steps*batch_size_gpl) tuples in total.Up to now, we have the actual training data ready. One can look atsample-data/generated/fiqafor a quick example about the data format. The very last step is to apply theMarginMSE lossto teach the student retriever to mimic the margin scores, CE(query, positive) - CE(query, negative) labeled by the teacher model (Cross-Encoder, CE). And of course,the MarginMSE stepis included in GPL and will be doneautomatically:). Note that MarginMSE works with dot-product and thus the final models trained withGPL works with dot-product.PS: The--retrieversare for negative mining. They can be any dense retrievers trained on the general domain (e.g. MS MARCO) and donot need to be strong for the target task/domain. Please refer to thepaperfor more details (cf. Table 7).Customized dataOne can also replace/put the customized data for any intermediate step under the path$path_to_generated_datawith the same name fashion. GPL will skip the intermediate steps by using these provided data.As a typical workflow, one might only have the (English) unlabeld corpus and want a good model performing well for this corpus. To run GPL training under such condition, one just needs these steps:Prepare your corpus in the same format as thedata sample;Put yourcorpus.jsonlunder a folder, e.g. named as "generated" for data loading and data generation by GPL;Call gpl.train with the folder path as an input argument: (other arguments work as usual)python-mgpl.train\--path_to_generated_data"generated"\--output_dir"output"\--new_size-1\--queries_per_passage-1Pre-trained checkpoints and generated dataPre-trained checkpointsWe now release the pre-trained GPL models via thehttps://huggingface.co/GPL. There are currently five types of models:GPL/${dataset}-msmarco-distilbert-gpl: Model with training order of (1) MarginMSE on MSMARCO -> (2) GPL on${dataset};GPL/${dataset}-tsdae-msmarco-distilbert-gpl: Model with training order of (1) TSDAE on${dataset}-> (2) MarginMSE on MSMARCO -> (3) GPL on${dataset};GPL/msmarco-distilbert-margin-mse: Model trained on MSMARCO with MarginMSE;GPL/${dataset}-tsdae-msmarco-distilbert-margin-mse: Model with training order of (1) TSDAE on ${dataset} -> (2) MarginMSE on MSMARCO;GPL/${dataset}-distilbert-tas-b-gpl-self_miner: Starting from thetas-b model, the models were trained with GPL on the target corpus${dataset}with the base model itself as the negative miner (here noted as "self_miner").Models 1. and 2. were actually trained on top of models 3. and 4. resp. All GPL models were trained the automatic setting ofnew_sizeandqueries_per_passage(by setting them to-1). This automatic setting can keep the performance while being efficient. For more details, please refer to the section 4.1 in thepaper.Among these models,GPL/${dataset}-distilbert-tas-b-gpl-self_minerones works the best on theBeIRbenchmark:For reproducing the results with the same package versions used in the experiments, please refer to the conda environment file,environment.yml.Generated dataWe now release the generated data used in the experiments of theGPL paper:The generated data for the main experiments on the 6 BeIR datasets:https://public.ukp.informatik.tu-darmstadt.de/kwang/gpl/generated-data/main/;The generated data for the experiments on the full 18 BeIR datasets:https://public.ukp.informatik.tu-darmstadt.de/kwang/gpl/generated-data/beir.Please note that the 4 datasets ofbioasq,robust04,trec-newsandsignal1mare only available after registration with the original official authorities. We only release the document IDs for these corpora with the file namecorpus.doc_ids.txt. For more details, please refer to theBeIRrepository.CitationIf you use the code for evaluation, feel free to cite our publicationGPL: Generative Pseudo Labeling for Unsupervised Domain Adaptation of Dense Retrieval:@article{wang2021gpl,title="GPL: Generative Pseudo Labeling for Unsupervised Domain Adaptation of Dense Retrieval",author="Kexin Wang and Nandan Thakur and Nils Reimers and Iryna Gurevych",journal="arXiv preprint arXiv:2112.07577",month="4",year="2021",url="https://arxiv.org/abs/2112.07577",}Contact person and main contributor:Kexin Wang,kexin.wang.2049@gmail.comhttps://www.ukp.tu-darmstadt.de/https://www.tu-darmstadt.de/Don't hesitate to send us an e-mail or report an issue, if something is broken (and it shouldn't be) or if you have further questions.This repository contains experimental software and is published for the sole purpose of giving additional background details on the respective publication.