Package: asyncval
AsyncvalAsyncval: A toolkit for asynchronously validating dense retriever checkpoints during training.Validating dense retriever checkpoints during training is time-consuming. Asyncval is a toolkit that aims to accelerate this process. Asyncval decouples the validation loop from the training loop, uses another GPU to automatically validate new DR checkpoints generated during training, and thus permits to perform validation asynchronously from training.InstallationFor customized dense retriever encoders, clone this repo and install as editable,git clone https://github.com/ielab/asyncval.git
cd asyncval
pip install --editable .Note: The current code base has been tested with,torch==1.10.0,transformers==4.5.1,datasets==1.16.1,faiss-cpu==1.7.1,python==3.7,ir_measures==0.2.3PreparationTo be able to use Asycval to validate your dense retreiver checkpoints, there are only two things you need to prepare:(1) Prepare the corpus and the validation query filesCorpus and validation query files are to be inJSONformat; specifically each line in the file is a JSON object that represents a passage or a query:{"text_id": str, "text": List[int]}wheretext_idis a unique id for a passage or query in the corpus or query file, andtextis a list of integers representing the token ids of the passage or query.Note thetextfield can also be the raw string of passage or query, however, we suggest Asyncval’s users to pre-tokenize their passages and queries and thus supply only token ids because Pre-tokenizing all text at once can speed up validation as there is no need to tokenize the same query and passage for each model checkpoint.(2) Rewrite DenseModel Class insrc/asyncval/modeling.py. (Tevatron users can skip this step)After the corpus and query file are prepared, a python class calledDenseModelneeds to be defined;Rewrite the__init__constructor method to intital your dense retriever encoder model. This method should take the checkpoint path and the pre-definedAsyncvalArgumentsas inputs. InAsyncvalArgumentswe pre-defined several useful arguments that can be passed via command line.Rewriteencode_passageandencode_querymethods to encode queries and passages into dense vectors.
These two methods should take a dictionary of key and values as input, where keys are Huggingface transformers input keys such asinput_idsandattention_maskand values are batched token ids and an attention mask matrix. Researchers need to implement and override these two methods by (i) taking the input dictionary and feed it to the DR model that they built in the constructor method, and (ii) returning batch dense vectors.Runing Asyncval validationAfter you get the corpus and query files and DenseModel class prepared. You can simply run Asyncval with the following command:python -m asyncval \
	--query_file List[str] \
	--candidate_file str \
	--ckpts_dir str \
	--tokenizer_name_or_path str \
	--qrel_file str \
	--output_dir strArguments Descriptionnamedesciptiontypedefaultquery_fileThe path to the pre-tokenized query JSON file. Multiple files can be provided.List[str]requiredcandidate_dirThe path to the folder that saves the pre-tokenized corpus JSON files.strrequiredckpts_dirThe path to the folder that saves DR checkpoints.strrequiredtokenizer_name_or_pathThe path or name to the Huggingface tokenizer. (for padding and attention masking)strrequiredq_max_lenThe maximum number of query token.int32p_max_lenThe maximum number of document token.int128qrel_fileThe path to the TREC format qrel file.strrequiredrun_nameA descriptor for the run. Typically used forwandb.strNonewrite_runWhether to write run files to disk.boolTrueoutput_dirThe path to save checkpoint run files.strrequiredlogging_dirTensorboard log dir.strNonemetricsThe list ofir_mesuresmetrics for validation.List[str]['RR@10','nDCG@10']per_device_eval_batch_sizeThe batch size for encoding.int8retrieve_batch_sizeThe batch size for FAISS retrieval.int64fp16Whether to use fp16 16-bit (mixed) precision encoding instead of 32-bit encoding.boolFalsereport_toThe list of integrations to report the results and logs to.strNonedeviceSet which GPU device for running asyncval.str'cuda:0'depthRetrival ranking depth (cutoff).int100cache_dirFolder for saving encoding dataset cache.strNoneUse splitter to sample subset of corpus for fast validationTo further speed up checkpoint validation, we also provide a feature to sample from the full corpus a subset, thus to avoid encoding all corpus passages for each checkpoint. The subset is sampled based on a given run file of the validation query set, such as the BM25 results, and a TREC standard qrel file which provides gold (relevant) passages for each query in the validation set. To do this, the following command should be ran:python -m asyncval.splitter \
		--candidate_dir str \
		--run_file str \
		--qrel_file str \
		--output_dir str \
		--depth intwhere--candidate_diris the path to folder that saves the pre-tokenized full corpus JSON files;--run_fileis the path to the run file;--qrel_fileis the path to the TREC qrel file;--output_diris the path to the folder in which to save the JSON file for the subset; and--depthis the number of top passages to keep for each query in the run file. For example, setting--depthto 100 means that only the top 100 passages for each query are kept. This trades-off validation accuracy for speed. Then you can use the generated subset corpus file for validation. We found that, instead of BM25, using a strong dense retriever baseline to generate the subset can dramatically reduce the validation time of each checkpoint without loss of fidelity.