Package: indomain
Domain Adapted Language Modeling ToolkitManifestoA great rift has emerged between general LLMs and the vector stores that are providing them with contextual information. The unification of these systems is an important step in grounding AI systems in efficient, factual domains, where they are utilized not only for their generality, but for their specificity and uniqueness. To this end, we are excited to open source the Arcee Domain Adapted Language Model (DALM) toolkit for developers to build on top of our Arcee open source Domain Pretrained (DPT) LLMs. We believe that our efforts will help as we begin next phase of language modeling, where organizations deeply tailor AI to operate according to their unique intellectual property and worldview.Demo DALMsQuery example DALMs created by the Arcee Team.DALM-PatentDALM-PubMedDALM-SECDALM-YoursResearch ContentsThis repository primarily contains code for fine-tuning afully differentialRetrieval Augmented Generation (RAG-end2end) architecture.For the first time in the literature, we modified the initial RAG-end2end model (TACL paper,HuggingFace implementation) to work with decoder-only language models like Llama, Falcon, or GPT. We also incorporated thein-batch negative conceptalongside the RAG's marginalization to make the entire processefficient.Inside thetrainingfolder, you'll find two codes to train the RAG-end2end and Retriever with contrastive learning.All evaluations related to the Retriever and the Generator are located in theevalfolder.Additionally, we have data processing codes and synthetic data generation code inside thedatasetsfolder.UsageTo perform training and evaluation for both the retriever model and the new rag-e2e model, please adhere to the following steps.InstallationYou can install this repo directly viapip install indomainAlternatively, for development or research, you can clone and install the repo locally:gitclonehttps://github.com/arcee-ai/DALM.git&&cdDALM
pipinstall--upgrade-e.This will install the DALM repo and all necessary dependencies.Make sure things are installed correctly by runningdalm version.  On an non-intel Mac you may need to downgradetransformerslibrary:pip install transformers==4.30.Data setuptl;drYou can rundalm qa-gen <path-to-dataset>to preprocess your dataset for training. Seedalm qa-gen --helpfor more optionsIf you do not have a dataset, you can start with ours# Note - our dataset already has queries and answers, so you don't actually need to run this.# replace `toy_dataset_train.csv` with your dataset of titles and passagesdalmqa-gendalm/datasets/toy_data_train.csvThe setup for training and evaluation can be effortlessly executed provided you possess aCSVfile containing two/three columns:Passage,Query(andAnswerif running e2e). You can utilize the scriptquestion_answer_generation.pyto generate this CSV.It's important to highlight that the retriever-only training method employs solely the passages and queries, whereas the rag-e2e training code utilizes all three columns.In our experiments, we utilizeBAAI/bge-large-enas the default retriever and employmeta-llama/Llama-2-7b-hfas the default generator. The code is designed to be compatible with any embedding model or autoregressive model available in the Hugging Face model repository athttps://huggingface.co/models.TrainingYou can leverage our scripts directly if you'd like, or you can use thedalmcli. The arguments for both are identicalTrain Retriever OnlyTrainBAAI/bge-large-enretriever with contrastive learning.pythondalm/training/retriever_only/train_retriever_only.py\--dataset_path"./dalm/datasets/toy_data_train.csv"\--retriever_name_or_path"BAAI/bge-large-en"\--output_dir"retriever_only_checkpoints"\--use_peft\--with_tracking\--report_toall\--per_device_train_batch_size150ordalmtrain-retriever-only"BAAI/bge-large-en""./dalm/datasets/toy_data_train.csv"\--output-dir"retriever_only_checkpoints"\--use-peft\--with-tracking\--report-toall\--per-device-train-batch-size150For all available arguments and options, seedalm train-retriever-only --helpTrain Retriever and Generator Jointly (RAG-e2e)TrainLlama-2-7bgenerator jointly with the retriever modelBAAI/bge-large-en.pythondalm/training/rag_e2e/train_rage2e.py\--dataset_path"./dalm/datasets/toy_data_train.csv"\--retriever_name_or_path"BAAI/bge-large-en"\--generator_name_or_path"meta-llama/Llama-2-7b-hf"\--output_dir"rag_e2e_checkpoints"\--with_tracking\--report_toall\--per_device_train_batch_size150ordalmtrain-rag-e2e\"./dalm/datasets/toy_data_train.csv"\"BAAI/bge-large-en"\"meta-llama/Llama-2-7b-hf"\--output-dir"rag_e2e_checkpoints"\--with-tracking\--report-toall\--per-device-train-batch-size150For all available arguments and options, seedalm train-rag-e2e --helpEvaluationHere's a summary of evaluation results on evaluating on a 200K line test csv of Patent abstractsType of RetrieverRecallHit ratePlain Retriever0.459840.45984Retriever with contrastive learning0.460370.46038Retriever End2End0.736340.73634To run retriever only eval
(make sure you have the checkpoints in the project root)pythondalm/eval/eval_retriever_only.py\--dataset_pathqa_pairs_test.csv\--retriever_name_or_path"BAAI/bge-large-en"\--passage_column_nameAbstract\--query_column_nameQuestion\--retriever_peft_model_pathretriever_only_checkpointsordalmeval-retrieverqa_pairs_test.csv\--retriever-name-or-path"BAAI/bge-large-en"\--passage-column-nameAbstract\--query-column-nameQuestion\--retriever-peft-model-pathretriever_only_checkpointsSeedalm eval-retriever --helpfor all available argumentsFor the e2e evalpythondalm/eval/eval_rag.py\--dataset_pathqa_pairs_test_2.csv\--retriever_name_or_path"BAAI/bge-large-en"\--generator_name_or_path"meta-llama/Llama-2-7b-hf"\--passage_column_nameAbstract\--query_column_nameQuestion\--answer_column_nameAnswer\--evaluate_generator\--query_batch_size5\--retriever_peft_model_pathrag_e2e_checkpoints/retriever\--generator_peft_model_pathrag_e2e_checkpoints/generatorordalmeval-ragqa_pairs_test.csv\--retriever-name-or-path"BAAI/bge-large-en"\--generator-name-or-path"meta-llama/Llama-2-7b-hf"\--retriever-peft-model-pathrag_e2e_checkpoints/retriever\--generator-peft-model-pathrag_e2e_checkpoints/generator\--passage-column-nameAbstract\--query-column-nameQuestion\--answer-column-nameAnswer\--query-batch-size5Seedalm eval-rag --helpfor all available argumentsContributingSeeCONTRIBUTING