Package: longchat
LongChatThe official repository for LongChat and LongEval, which supports training and evaluating long-context LLM based chatbots. Check out ourpostfor scientific findings!News[2023/08] 🔥 We releasedLongChat v1.5based on Llama 2 with 32K context lengths. Downloadweights.Environment setupcondacreate-nlongevalpython=3.10
condaactivatelongeval
gitclonehttps://github.com/DachengLi1/LongChat/cdLongChat/
pipinstall-e.For users who want to test very long sequence length, please also installFlashAttention.longchatTo train a LongChat model yourself, replace  to the llama checkpoint director, and run:python-mtorch.distributed.run--nproc_per_node=8\longchat/train/fine_tune/train_condense_16K.py\--model_name_or_path<path-to-llama>\--data_pathdata/dummy_conversation.json\--bf16\--output_diroutputs\--num_train_epochs3\--per_device_train_batch_size1\--per_device_eval_batch_size4\--gradient_accumulation_steps1\--evaluation_strategyno\--save_strategysteps\--save_steps1000\--save_total_limit1\--learning_rate2e-5\--weight_decay0.\--warmup_ratio0.03\--lr_scheduler_type"cosine"\--logging_steps1\--fsdp"full_shard auto_wrap"\--fsdp_transformer_layer_cls_to_wrap'LlamaDecoderLayer'\--tf32True\--model_max_length16384\--gradient_checkpointingTrue\--lazy_preprocessTrueThis script assumes 8xA100 GPUs and use the dummy data in the repository for example usage only. Please adapt to your use case.
We provided models trained on conversation data in HuggingFace:LongChat-13b-16kandLongChat-7b-16k.longevalWe provide a simplenotebookto demonstrate following steps. We also provided reproduced results under longeval/evaluation folder.To evaluate the LongChat model on the coarsed-grained topics benchmark:cdlongeval
python3eval.py--model-name-or-pathlmsys/longchat-13b-16k--tasktopics--longchat_flash_attnTo evaluate new models, choose a<task>from ["topics", "lines"], and replace<your-model>with your model path:python3eval.py--model-name-or-path<your-model>--task<task>Some models require memory efficient flash attention to evaluate super long test. Please add an issue if you are running into memory issue on your model. We include the commands we used in the release bloghere.
The output will be stored under evaluation/task/predictions/your-model. The line recall task directly outputs an accuracy. The topics recall task outputs natural languages that are hard to parse. you can manually inspect the model output and calculate an accuracy or use chatgpt-3.5-turbo to automatically calculate it. In the latter case,setOPENAI_API_KEY and run:pythonauto_topic_eval.py--test_file<generated_output>Replace <generated_output> with the generated topic prediction, e.g. evaluation/topics/predictions/longchat_13b_16k/5_response.txt.To generate new testcases:python3generate_testcases.py<path-to-generate-testcases-configuration>Replace <path-to-generate-testcases-configuration> with the path to a yaml file containing the
configurations for generating testcases.longeval/generate_testcases_configs.yamlis a configuration file provides default options. To customize the testcases generated, users can tune the options in
the configuration file.Warning: Please set theoutput_diroptionin in the configuration file to some other location that does
not overlaps withlongeval/evaluation/. Otherwise the original testcases could be overwritten.CitationIf you find this repo to be useful, plese cite:@misc{longchat2023,
    title = {How Long Can Open-Source LLMs Truly Promise on Context Length?},
    url = {https://lmsys.org/blog/2023-06-29-longchat},
    author = {Dacheng Li*, Rulin Shao*, Anze Xie, Ying Sheng, Lianmin Zheng, Joseph E. Gonzalez, Ion Stoica, Xuezhe Ma, and Hao Zhang},
    month = {June},
    year = {2023}
}