Package: gritlm
GRITThis repository provides all materials for the paperGenerative Representational Instruction Tuning. We continue developing the repository and welcome any contributions. If you want to use the code in the exact same way as in the paper, please use the 1.0.0 release at commit hashTODO.InferenceTrainingDataRunEvaluationEmbeddingGenerativeKnown issuesAcknowledgementsCitationInferencepip install gritlmfromgritlmimportGritLM# Loads the model for both capabilities; If you only need embedding pass `mode="embedding"` to save memory (no lm head)model=GritLM("GritLM/GritLM-7B",torch_dtype="auto")# To load the 8x7B you will likely need multiple GPUs.# All the kwargs are passed to HF from_pretrained so you can just do the below to load on multiple GPUs:# model = GritLM("GritLM/GritLM-8x7B", torch_dtype="auto", device_map="auto")# You can also load other models e.g.# model = GritLM("Muennighoff/SGPT-125M-weightedmean-nli-bitfit", pooling_method="weighted_mean", attn=None)# model = GritLM("hkunlp/instructor-base", pooling_method="mean", attn=None)### Embedding/Representation ###instruction="Given a scientific paper title, retrieve the paper's abstract"queries=['Bitcoin: A Peer-to-Peer Electronic Cash System','Generative Representational Instruction Tuning']documents=["A purely peer-to-peer version of electronic cash would allow online payments to be sent directly from one party to another without going through a financial institution. Digital signatures provide part of the solution, but the main benefits are lost if a trusted third party is still required to prevent double-spending. We propose a solution to the double-spending problem using a peer-to-peer network. The network timestamps transactions by hashing them into an ongoing chain of hash-based proof-of-work, forming a record that cannot be changed without redoing the proof-of-work. The longest chain not only serves as proof of the sequence of events witnessed, but proof that it came from the largest pool of CPU power. As long as a majority of CPU power is controlled by nodes that are not cooperating to attack the network, they'll generate the longest chain and outpace attackers. The network itself requires minimal structure. Messages are broadcast on a best effort basis, and nodes can leave and rejoin the network at will, accepting the longest proof-of-work chain as proof of what happened while they were gone.","All text-based language problems can be reduced to either generation or embedding. Current models only perform well at one or the other. We introduce generative representational instruction tuning (GRIT) whereby a large language model is trained to handle both generative and embedding tasks by distinguishing between them through instructions. Compared to other open models, our resulting GritLM 7B sets a new state of the art on the Massive Text Embedding Benchmark (MTEB) and outperforms all models up to its size on a range of generative tasks. By scaling up further, GritLM 8X7B outperforms all open generative language models that we tried while still being among the best embedding models. Notably, we find that GRIT matches training on only generative or embedding data, thus we can unify both at no performance loss. Among other benefits, the unification via GRIT speeds up Retrieval-Augmented Generation (RAG) by > 60% for long documents, by no longer requiring separate retrieval and generation models. Models, code, etc. are freely available at https://github.com/ContextualAI/gritlm."]defgritlm_instruction(instruction):return"
\n"+instruction+"\n
\n"ifinstructionelse"
\n"# No need to add instruction for retrieval documentsd_rep=model.encode(documents,instruction=gritlm_instruction(""))q_rep=model.encode(queries,instruction=gritlm_instruction(instruction))fromscipy.spatial.distanceimportcosinecosine_sim_q0_d0=1-cosine(q_rep[0],d_rep[0])cosine_sim_q0_d1=1-cosine(q_rep[0],d_rep[1])cosine_sim_q1_d0=1-cosine(q_rep[1],d_rep[0])cosine_sim_q1_d1=1-cosine(q_rep[1],d_rep[1])print("Cosine similarity between\"%s\"and\"%s\"is:%.3f"%(queries[0][:15],documents[0][:15],cosine_sim_q0_d0))# Cosine similarity between "Bitcoin: A Peer" and "A purely peer-t" is: 0.608print("Cosine similarity between\"%s\"and\"%s\"is:%.3f"%(queries[0][:15],documents[1][:15],cosine_sim_q0_d1))# Cosine similarity between "Bitcoin: A Peer" and "All text-based " is: 0.101print("Cosine similarity between\"%s\"and\"%s\"is:%.3f"%(queries[1][:15],documents[0][:15],cosine_sim_q1_d0))# Cosine similarity between "Generative Repr" and "A purely peer-t" is: 0.120print("Cosine similarity between\"%s\"and\"%s\"is:%.3f"%(queries[1][:15],documents[1][:15],cosine_sim_q1_d1))# Cosine similarity between "Generative Repr" and "All text-based " is: 0.533### Generation #### We did not finetune GritLM models with system prompts, as you can just include system-like instructions together with your user instructionmessages=[{"role":"user","content":"Please write me a poem about my recent hike of Mt. Fuji at midnight in the style of Shakespeare."},]encoded=model.tokenizer.apply_chat_template(messages,add_generation_prompt=True,return_tensors="pt")encoded=encoded.to(model.device)gen=model.generate(encoded,max_new_tokens=256,do_sample=False)decoded=model.tokenizer.batch_decode(gen)print(decoded[0])"""
 
Please write me a poem about my recent hike of Mt. Fuji at midnight in the style of Shakespeare.
Oh, Mt. Fuji, mountain grand,A sight to see, a climb to command,At midnight, in the dark of night,I climbed your slopes, with all my might.The stars above, they shone so bright,A beacon in the darkness, guiding light,The wind did blow, with a gentle sigh,As I climbed higher, with a steady eye.The path was steep, the climb was tough,But I pressed on, with a steadfast rough,For the summit, I longed to see,The view from the top, a sight to be.At last, I reached the peak, and stood,With awe and wonder, I gazed aloud,The world below, a sight to see,A view that's worth the climb, you'll agree.Mt. Fuji, mountain grand,A sight to see, a climb to command,At midnight, in the dark of night,I climbed your slopes, with all my might.
"""Pretrained modelsThe weights and logs of all models from the paper are freely available:Weights:https://huggingface.co/GritLMLogs:https://wandb.ai/muennighoff/gritlm/overview?workspace=user-muennighoffThe names will not always match across HF & WandB, but you can usually figure out which belongs to which via the--output_dirin the command. Note that we renamed all models fromsgpt2togritlmat some point, so some names/logs/commands contain the old name.Shortcuts:sq = sequence length; sq2048 is 2048 tokensep = epochs; ep1 is 1 epochst = steps; st100 is 100 stepsm7/m8x7/l7/g6 = base model is Mistral 7B/Mistral 8x7B/Llama 2 7B/GPT-J 6Bemb/gen/gritlm = embedding, generative, unifiedbf16c = embeddings are cast back to bf16 after pooling and similarity computation is also done in bf16 (simulating how cached embeddings would operate)bb/cc/bbcc... = order of bidirectional vs causal attentiongendups = not using--use_unique_indicesduring training. If not used and training is unified, then data is duplicated worsening performanceThe most important ones are:ModelDescriptionEmb performance (MTEB)Gen performanceGritLM-7B7B parameter model that uses bidirectional attention for embedding and causal attention for generation. It is finetuned from Mistral-7B66.855.5GritLM-8x7B7B parameter model that uses bidirectional attention for embedding and causal attention for generation. It is finetuned from Mistral-8x7B65.765.7ForGritLM-7BandGritLM-8x7B, the folder contains a custom modeling file (modeling_gritlm*.py) which adds bidirectional attention via the keyword argumentis_causal, such that if you load them withfrom_pretrainedin transformers, it is automatically available. We did not add this for any other models uploaded to the organization, thus for those, you need to either add it yourself or simply replace themodeling_mistral.py&modeling_mixtral.pyfiles in your transformers installation withscripts/modeling_mistral_gritlm.py&scripts/modeling_mixtral_gritlm.py. Note that for models that do not use bidirectional attention or when you do not intend to use the bidirectional attention (e.g. for generation), you don't need to do anything.TrainingDataThe repo uses the below format. Seetraining/toy_data.jsonlfor an example.Format:Embedding data:{"query": str, "pos": List[str], "neg": List[str]}Embedding data with instructions that are excluded from the embedding & loss:{"query": List[str, str], "pos": List[List[str, str]], "neg": List[List[str, str]]}The 1st element in the inner list is the instruction and the 2nd is the text to embed.Generative data:{"text": str}Generative data with instructions that are excluded from the loss:{"text": List[str]}The 1st/3rd/5th.. element is the instruction and the 2nd/4th/6th.. is the response. If you only want single-turn chat then just put two elements, for multi-turn put more.We release the below datasets:Embedding:MEDI (https://hf.co/datasets/GritLM/MEDI)MEDI2 (https://hf.co/datasets/GritLM/MEDI2)MEDI2BGE (https://hf.co/datasets/GritLM/MEDI2BGE)Generative:https://hf.co/datasets/GritLM/tulu2They are explained in more detail in the paper and its appendix. So to e.g. train a GRIT model on MEDI2 & Tulu2, simply download both viagit clone https...and then place them in the same directory and follow the instructions below to run. Unfortunately, we cannot release the E5S data used for our final models.RunSetup:gitclonehttps://github.com/ContextualAI/gritlm`cdgritlm
pipinstall-e.cdgritlmBelow are easy examples for getting started:Embedding modeltorchrun--nproc_per_node1\-mtraining.run\--output_dirtest_path\--model_name_or_pathopenaccess-ai-collective/tiny-mistral\--train_datatraining/toy_data/toy_data_embedding.jsonl\--learning_rate1e-5\--num_train_epochs5\--per_device_train_batch_size2\--dataloader_drop_lastTrue\--normalizedTrue\--temperature0.02\--query_max_len32\--passage_max_len128\--train_group_size2\--negatives_cross_device\--modeembeddingGenerative modeltorchrun--nproc_per_node1\-mtraining.run\--output_dirtest_path\--model_name_or_pathopenaccess-ai-collective/tiny-mistral\--train_datatraining/toy_data/toy_data_generative.jsonl\--learning_rate1e-5\--num_train_epochs5\--per_device_train_batch_size2\--dataloader_drop_lastTrue\--passage_max_len128\--modegenerativeUnified model (GRIT)torchrun--nproc_per_node1\-mtraining.run\--output_dirtest_path\--model_name_or_pathopenaccess-ai-collective/tiny-mistral\--train_datatraining/toy_data\--learning_rate1e-5\--num_train_epochs5\--per_device_train_batch_size2\--dataloader_drop_lastTrue\--normalizedTrue\--temperature0.02\--query_max_len32\--passage_max_len128\--train_group_size2\--negatives_cross_device\--modeunifiedAll arguments are explained intraining/arguments.pyor theHF TrainingArguments documentationexcept fornproc_per_nodewhich is the number of GPUs per node. For our actual training runs, we use accelerate to easily use multiple nodes and GPUs. The scripts are all inscripts/training, for examplescripts/training/train_gritlm_8x7b.shwas used for GritLM-8x7B. For models from the ablations, you can check their folder on the huggingface hub which contains atraining_args.binfile with the arguments. You can also check all their arguments on the WandB:https://wandb.ai/muennighoff/gritlm.EvaluationEmbeddingcdgritlm
pythonevaluation/eval_mteb.py\--model_name_or_pathGritLM/GritLM-7B\--task_typesClassification,Clustering,PairClassification,Reranking,Retrieval,STS,Summarization\--batch_size32For a faster way, checkscripts/eval_mteb.shwhich submits jobs across multiple GPUs for each dataset.Generative## Setup# Setup eval for MMLU/GSM8K/BBH/TyDi QA/Alpacagitclonehttps://github.com/Muennighoff/open-instruct.gitcdopen-instruct
pipinstall-rrequirements.txt
bash./scripts/prepare_eval_data.shcd..# Setup eval for HumanEvalPackgitclonehttps://github.com/bigcode-project/bigcode-evaluation-harnesscdbigcode-evaluation-harness
pipinstall-e.cd..MODEL_PATH=GritLM/gritlm-7b# Run all evals except for Alpaca; You may have to change some paths etc.bashscripts/generative_eval.sh{pathtomodel}# Run Alpaca 1.0exportOPENAI_API_KEY=YOUR_API_KEY
python-meval.alpaca_farm.run_eval\--use_vllm\--model_name_or_path$MODEL_PATH\--tokenizer_name_or_path$MODEL_PATH\--save_dir./\--use_chat_format\--chat_formatting_functioneval.templates.create_prompt_with_gritlm_chat_format# Alpaca 2.0 (not used in the paper)python-meval.alpaca_farm.run_eval\--use_vllm\--model_name_or_path$MODEL_PATH\--tokenizer_name_or_path$MODEL_PATH\--save_dir$MODEL_PATH\--use_chat_format\--chat_formatting_functioneval.templates.create_prompt_with_gritlm_chat_format\--alpaca2Known issuesIf you train on many nodes + a large model + fsdp, you may encounter timeouts upon checkpoint saving withFULL_STATE_DICT. For example, training Mixtral on 32 nodes with 8 GPUs each, fails with the below. Usually the main node will finish the saving unless they are all in the same job manager who kills it. Unfortunately increasing the timeout limit seems not possibe? (https://discuss.pytorch.org/t/how-to-set-nccl-timeout-to-infinity/146006;https://github.com/huggingface/accelerate/issues/2236#issuecomment-1864809701) So the current solution is just to use less nodes. Please let us know if you have a better solution.[dojo-a3-ghpc-9:1]:what():[Rank1]NCCLwatchdogthreadterminatedwithexception:[Rank1]Watchdogcaughtcollectiveoperationtimeout:WorkNCCL(SeqNum=683,OpType=_ALLGATHER_BASE,NumelIn=32768512,NumelOut=262148096,Timeout(ms)=600000)ranfor600032millisecondsbeforetimingout.Add Packing at least for gen, maybe also possible for emb; Need to be careful with the NextTokenLossQLoRa / LoRa integration is not well-testedRuntimeError:oneofthevariablesneededforgradientcomputationhasbeenmodifiedbyaninplaceoperation:[torch.cuda.LongTensor[20,2048]]isatversion21;expectedversion0instead.Hint:enableanomalydetectiontofindtheoperationthatfailedtocomputeitsgradient,withtorch.autograd.set_detect_anomaly(True).If you run into the below error when doing multi-node training, try thishttps://github.com/huggingface/transformers/issues/26971#issuecomment-1868137087load_state_dict_into_meta_modelset_module_tensor_to_device(model,param_name,param_device,**set_module_kwargs)File"/env/lib/conda/gritlm/lib/python3.9/site-packages/accelerate/utils/modeling.py",line317,inset_module_tensor_to_devicenew_value=value.to(device)NotImplementedError:Cannotcopyoutofmetatensor;nodata!DeepSpeed does not work with--mode unifiedand--gradient_accumulation_stepsbigger than 1 (i.e. GradCache) (FSDP is ~equivalent so this is not high prio)fsdp_use_orig_params: truein the accelerate config is critical for performance, else it may not converge at all (see comparisons in the wandb runs)If you get the below error where training fails on saving, then upgrade accelerate & transformers50801/06/202408:28:40-INFO-accelerate.utils.fsdp_utils-Modelsavedto/data/niklas/gritlm/gritlm_mist_sq2048_medibgetuluv2_tuluformat_8nodes_oldtracc/tmp-checkpoint-500/pytorch_model.bin50901/06/202408:30:24-INFO-accelerate.utils.fsdp_utils-SavingOptimizerstateto/data/niklas/gritlm/gritlm_mist_sq2048_medibgetuluv2_tuluformat_8nodes_oldtracc/tmp-checkpoint-500/optimizer.bin510Traceback(mostrecentcalllast):511File"/env/lib/conda/gritlmold/lib/python3.9/runpy.py",line197,in_run_module_as_main512return_run_code(code,main_globals,None,513File"/env/lib/conda/gritlmold/lib/python3.9/runpy.py",line87,in_run_code514exec(code,run_globals)515File"/home/niklas/gritlm/training/run.py",line421,in<module>516main()517File"/home/niklas/gritlm/training/run.py",line411,inmain518trainer.train()519File"/env/lib/conda/gritlmold/lib/python3.9/site-packages/transformers/trainer.py",line1537,intrain520returninner_training_loop(521File"/home/niklas/gritlm/training/gradcache_trainer.py",line962,in_inner_training_loop522self._maybe_log_save_evaluate(tr_loss,model,trial,epoch,ignore_keys_for_eval)523File"/env/lib/conda/gritlmold/lib/python3.9/site-packages/transformers/trainer.py",line2274,in_maybe_log_save_evaluate524self._save_checkpoint(model,trial,metrics=metrics)525File"/env/lib/conda/gritlmold/lib/python3.9/site-packages/transformers/trainer.py",line2354,in_save_checkpoint526self._save_optimizer_and_scheduler(staging_output_dir)527File"/env/lib/conda/gritlmold/lib/python3.9/site-packages/transformers/trainer.py",line2445,in_save_optimizer_and_scheduler528save_fsdp_optimizer(529File"/env/lib/conda/gritlmold/lib/python3.9/site-packages/accelerate/utils/fsdp_utils.py",line146,insave_fsdp_optimizer530torch.save(optim_state,output_optimizer_file)531File"/env/lib/conda/gritlmold/lib/python3.9/site-packages/torch/serialization.py",line618,insave532with_open_zipfile_writer(f)asopened_zipfile:533File"/env/lib/conda/gritlmold/lib/python3.9/site-packages/torch/serialization.py",line492,in_open_zipfile_writer534returncontainer(name_or_buffer)535File"/env/lib/conda/gritlmold/lib/python3.9/site-packages/torch/serialization.py",line463,in__init__536super().__init__(torch._C.PyTorchFileWriter(self.name))537RuntimeError:Parentdirectory/data/niklas/gritlm/gritlm_mist_sq2048_medibgetuluv2_tuluformat_8nodes_oldtracc/tmp-checkpoint-500doesnotexist.If loss is slightly different when changing the number of gradient accumulation steps, this is expected because torch uses weighted mean pooling in its CrossEntropyLoss by default. As the language modeling objective occasionally has the same token being predicted multiple times in one batch, this will result in a different loss when splitting up batches. Meanwhile, for the embedding loss every class id is only predicted once, thus weighted mean is equivalent to mean for embedding (https://github.com/pytorch/pytorch/issues/72047;https://github.com/pytorch/pytorch/issues/40560;https://github.com/pytorch/pytorch/issues/107680).Another reason loss is different when changing the number of processes is that the data order may be different. While all seeds are being set, accelerate.prepare of the data loader in the trainer, sets up the dataloader such that it iterates one sample ahead of time. Thus, on the first iteration it gets two batches for each process instead of one. Somehow, this causes one sample in the first batch to land in the subsequent batch when going from 0 to 8 gpus. I could not figure out why exactly, but investigations are appreciated.Training with fp32 generally converges much faster than with bf16. Changing the allreduce and buffer dtypes to fp32 does not change this (https://github.com/NVIDIA/Megatron-LM/issues/502;https://github.com/pytorch/pytorch/issues/106395)torch.compile fails with the below in unified mode (also seehttps://github.com/pytorch/pytorch/issues/111317):fromusercode:File"/env/lib/conda/gritlmnew/lib/python3.9/site-packages/transformers/models/mistral/modeling_mistral.py",line757,inforwardhidden_states=self.input_layernorm(hidden_states)File"/env/lib/conda/gritlmnew/lib/python3.9/site-packages/torch/nn/modules/module.py",line1527,in_call_implreturnforward_call(*args,**kwargs)File"/env/lib/conda/gritlmnew/lib/python3.9/site-packages/transformers/models/mistral/modeling_mistral.py",line89,inforwardreturnself.weight*hidden_states.to(input_dtype)SetTORCH_LOGS="+dynamo"andTORCHDYNAMO_VERBOSE=1formoreinformationYoucansuppressthisexceptionandfallbacktoeagerbysetting:importtorch._dynamotorch._dynamo.config.suppress_errors=Trueexample_value=wrap_to_fake_tensor_and_record(File"/env/lib/conda/gritlmnew/lib/python3.9/site-packages/torch/_dynamo/variables/builder.py",line1587,inwrap_to_fake_tensor_and_recordfake_e=wrap_fake_exception(File"/env/lib/conda/gritlmnew/lib/python3.9/site-packages/torch/_dynamo/utils.py",line916,inwrap_fake_exceptionreturnfn()File"/env/lib/conda/gritlmnew/lib/python3.9/site-packages/torch/_dynamo/variables/builder.py",line1588,in<lambda>lambda:tx.fake_mode.from_tensor(File"/env/lib/conda/gritlmnew/lib/python3.9/site-packages/torch/_subclasses/fake_tensor.py",line1721,infrom_tensorreturnself.fake_tensor_converter(File"/env/lib/conda/gritlmnew/lib/python3.9/site-packages/torch/_subclasses/fake_tensor.py",line371,in__call__returnself.from_real_tensor(File"/env/lib/conda/gritlmnew/lib/python3.9/site-packages/torch/_subclasses/fake_tensor.py",line324,infrom_real_tensorout=self.meta_converter(File"/env/lib/conda/gritlmnew/lib/python3.9/site-packages/torch/_subclasses/meta_utils.py",line591,in__call__r=self.meta_tensor(File"/env/lib/conda/gritlmnew/lib/python3.9/site-packages/torch/_subclasses/meta_utils.py",line307,inmeta_tensorbase=self.meta_tensor(File"/env/lib/conda/gritlmnew/lib/python3.9/site-packages/torch/_subclasses/meta_utils.py",line478,inmeta_tensorr.grad=self.meta_tensor(torch._dynamo.exc.InternalTorchDynamoError:attemptingtoassignagradientofsize'[27264000]'toatensorofsize'[218112000]'.PleaseensurethatthegradientandthetensorarethesamesizeIf usingfsdp_state_dict_type: FULL_STATE_DICTin the config with many nodes, it may time out upon checkpointing. E.g. for 32 nodes with Mistral-7B in fp32 it timed out for me.DeepSpeed + FlashAttention2 + Optim & Params offloaded to CPU + DeepSpeed ZeRo3 init fails:s.(Triggeredinternallyat/opt/conda/conda-bld/pytorch_1702400412039/work/torch/csrc/t
ensor/python_tensor.cpp:83.)total_norm_cuda=get_accelerator().FloatTensor([float(total_norm)])Invalidatetracecache@step1:expectedmodule1,butgotmodule2[EProcessGroupNCCL.cpp:475][Rank1]Watchdogcaughtcollectiveoperationtimeout:WorkIf implementing full splitting + GC you may run intoFile"/home/niklas/gritlm/training/gradcache_trainer.py",line630,in_inner_training_loopself.accelerator.backward(loss)File"/env/lib/conda/gritlmnew/lib/python3.9/site-packages/accelerate/accelerator.py",line1964,inbackwardloss.backward(**kwargs)File"/env/lib/conda/gritlmnew/lib/python3.9/site-packages/torch/_tensor.py",line492,inbackwardtorch.autograd.backward(File"/env/lib/conda/gritlmnew/lib/python3.9/site-packages/torch/autograd/__init__.py",line251,inbackwardVariable._execution_engine.run_backward(# Calls into the C++ engine to run the backward passFile"/env/lib/conda/gritlmnew/lib/python3.9/site-packages/torch/utils/checkpoint.py",line1075,inunpack_hookframe.check_recomputed_tensors_match(gid)File"/env/lib/conda/gritlmnew/lib/python3.9/site-packages/torch/utils/checkpoint.py",line812,incheck_recomp
uted_tensors_matchraiseCheckpointError(torch.utils.checkpoint.CheckpointError:torch.utils.checkpoint:Adifferentnumberoftensorswassavedduringth
eoriginalforwardandrecomputation.Numberoftensorssavedduringforward:47Numberoftensorssavedduringrecomputation:45VisualsFigure 1:https://colab.research.google.com/drive/13MY11ZRtkDjBMWYEXoaDF1nxyRxis7o7?usp=sharing& then add in logos viavisuals/performance.drawiothat can be opened withhttps://app.diagrams.net/, then add blurbs in viavisuals/performance.keythat can be opened in Keynote.Figure 2:https://docs.google.com/drawings/d/1ZAzaX4h2JfJR1ahan0R5nk3Xm17SMquGjhshnBNJOzY/edit?usp=sharingFigure 3:https://docs.google.com/drawings/d/1vaSNvDWy6xBBuC70rI22qdOmymksxqoTYiplGPH22ys/edit?usp=sharingFigure 4:https://docs.google.com/drawings/d/1rv916zpYvBbaS6QxpFP4_6fc4gABcPWc2qZC3NUpz8s/edit?usp=sharingFigure 5/6/7/8:https://colab.research.google.com/drive/13MY11ZRtkDjBMWYEXoaDF1nxyRxis7o7?usp=sharingOther figures & tables are manual, but there are helper scripts, such asscripts/mteb_to_tex.pyAcknowledgementsThe code is inspired by:https://github.com/Muennighoff/sgpthttps://github.com/FlagOpen/FlagEmbeddinghttps://github.com/embeddings-benchmark/mtebCitationIf useful please consider citing 😊@misc{muennighoff2024generative,title={Generative Representational Instruction Tuning},author={Niklas Muennighoff and Hongjin Su and Liang Wang and Nan Yang and Furu Wei and Tao Yu and Amanpreet Singh and Douwe Kiela},year={2024},eprint={2402.09906},archivePrefix={arXiv},primaryClass={cs.CL}}