Package: triple-encoders
🤗Models| 📊Datasets| 📃Papertriple-encodersis a library for contextualizing distributedSentence Transformersrepresentations.
At inference, triple encoders can be used for retrieval-based sequence modeling via sequential modular late-interaction:Representations are encodedseparatelyand the contextualization isweightless:mean-poolingto pairwise contextualize sentence representations (creates a distributed query)cosine similarityto measure the similarity between all query vectors and the retrieval candidates.summationto aggregate the similarity (similar to average-based late interaction ofColBERT).Key Features1️⃣One dense vector vs distributed dense vectors: in our paper we demonstrate that our late interaction-based approach outperforms single-vector representations on long sequences, including zero-shot settings.🏎️💨Relative compute: as every representation is encoded separately, you only need to encode, compute mixtures and similarities for the latest added representation (in dialog: the latest utterance).📚No Limit on context-length: our distributed sentence transformer architecture is not limited to any sequence length. You can use your entire sequence as query!🌎Multilingual support:triple-encoderscan be used with anySentence Transformersmodel. This means that you can model multilingual sequences by simply training on a multilingual model checkpoint.InstallationYou can installtriple-encodersvia pip:pipinstalltriple-encodersNote thattriple-encodersrequires Python 3.6 or higher.Getting StartedOur experiments for sequence modeling and short-term planning conducted in the paper can be found in thenotebooksfolder. The hyperparameter that we used for training are the default parameters in thetrainer.pyfile.Retrieval-based Sequence ModelingWe provide an example of how to use triple-encoders for conversational sequence modeling (response selection) with 2 dialog speakers. If you want to use triple-encoders for other sequence modeling tasks, you can use theTripleEncodersForSequenceModelingclass.Loading the modelfromtriple_encoders.TripleEncodersForConversationalSequenceModelingimportTripleEncodersForConversationalSequenceModelingtriple_path=''# load modelmodel=TripleEncodersForConversationalSequenceModeling(triple_path)Inference# load candidates for response selectioncandidates=['I am doing great too!','Where did you go?','ACL is an interesting conference']# load candidates and store indexmodel.load_candidates_from_strings(candidates,output_directory_candidates_dump='output/path/to/save/candidates')# create a sequencesequence=model.contextualize_sequence(["Hi!",'Hey, how are you?'],k_last_rows=2)# model sequence (compute scores for candidates)sequence=model.sequence_modeling(sequence)# retrieve utterance from dialog partnernew_utterance="I'm fine, thanks. How are you?"# pass it to the model with dialog_partner=Truesequence=model.contextualize_utterance(new_utterance,sequence,dialog_partner=True)# model sequence (compute scores for candidates)sequence=model.sequence_modeling(sequence)# retrieve candidates to provide a responseresponse=model.retrieve_candidates(sequence,3)response#(['I am doing great too!','Where did you go?', 'ACL is an interesting conference'],# tensor([0.4944, 0.2392, 0.0483]))Speed:Time to load candidates: 31.815 msTime to contextualize sequence: 18.078 msTime to model sequence: 0.256 msTime to contextualize new utterance: 15.858 msTime to model new utterance: 0.213 msTime to retrieve candidates: 0.093 msEvaluationfromdatasetsimportload_datasetdataset=load_dataset("daily_dialog")test=dataset['test']['dialog']df=model.evaluate_seq_dataset(test,k_last_rows=2)df# pandas dataframe with the average rank for each history lengthShort-Term Planning (STP)Short-term planning enables you to re-rank candidate replies from LLMs to reach a goal utterance over multiple turns.Inferencefromtriple_encoders.TripleEncodersForSTPimportTripleEncodersForSTPmodel=TripleEncodersForSTP(triple_path)context=['Hey, how are you ?','I am good, how about you ?','I am good too.']candidates=['Want to eat something out ?','Want to go for a walk ?']goal=' I am hungry.'result=model.short_term_planning(candidates,goal,context)result# 'Want to eat something out ?'Evaluationfromdatasetsimportload_datasetfromtriple_encoders.TripleEncodersForSTPimportTripleEncodersForSTPdataset=load_dataset("daily_dialog")test=dataset['test']['dialog']model=TripleEncodersForSTP(triple_path,llm_model_name_or_path='your favorite large language model')df=model.evaluate_stp_dataset(test)# pandas dataframe with the average rank and Hits@k for each history length, goal_distanceTraining Triple EncodersYou can train your own triple encoders with Contextualized Curved Contrastive Learning (C3L) using our trainer.
The hyperparameters that we used for training are the default parameters in thetrainer.pyfile.
Note that we pre-trained our best model with Curved Contrastive Learning (CCL) (fromimaginaryNLP) before training with C3L.fromtriple_encoders.trainerimportTripleEncoderTrainerfromdatasetsimportload_datasetdataset=load_dataset("daily_dialog")trainer=TripleEncoderTrainer(base_model_name_or_path=,batch_size=48,observation_window=5,speaker_token=True,# used for conversational sequence modelingnum_epochs=3,warmup_steps=10000)trainer.generate_datasets(dataset["train"]["dialog"],dataset["validation"]["dialog"],dataset["test"]["dialog"],)trainer.train("output/path/to/save/model")CitationIf you use triple-encoders in your research, please cite the following paper:% todo
@article{anonymous,
  title={Triple Encoders: Represenations That Fire Together, Wire Together},
  author={Justus-Jonas Erker, Florian Mai, Nils Reimers, Gerasimos Spanakis, Iryna Gurevych},
  journal={axiv},
  year={2024}
}ContactContact person: Justus-Jonas Erker,justus-jonas.erker@tu-darmstadt.dehttps://www.ukp.tu-darmstadt.de/https://www.tu-darmstadt.de/Don't hesitate to send us an e-mail or report an issue, if something is broken (and it shouldn't be) or if you have further questions.
This repository contains experimental software and is published for the sole purpose of giving additional background details on the respective publication.Licensetriple-encoders is licensed under the Apache License, Version 2.0. SeeLICENSEfor the full license text.Acknowledgementthis package is based upon theimaginaryNLPandSentence Transformers.