Repository: lm-sys/FastChat (language: python)

Issue:
transformers version need to be 4.31.0
Hi,

Thanks for your great job. I think to support Llama 2, the transformer lib version should be 4.31.0.

Thanks.


Discussion / hints:
It works with 4.28.1 for Llama2-13b-chat-hf.

Fix patch:
diff --git a/README.md b/README.md
index a69e77a84..6dd972314 100644
--- a/README.md
+++ b/README.md
@@ -267,7 +267,8 @@ torchrun --nproc_per_node=4 --master_port=20001 fastchat/train/train_mem.py \
     --lazy_preprocess True
 ```
 
-If you meet out-of-memory during model saving, see solutions [here](https://github.com/pytorch/pytorch/issues/98823).
+- If you meet out-of-memory due to "FSDP Warning: When using FSDP, it is efficient and recommended... ", see solutions [here](https://github.com/huggingface/transformers/issues/24724#issuecomment-1645189539).
+- If you meet out-of-memory during model saving, see solutions [here](https://github.com/pytorch/pytorch/issues/98823).
 
 ### Other models and LoRA support
 More instructions to train other models (e.g., FastChat-T5) and use LoRA are in [docs/training.md](docs/training.md).
diff --git a/fastchat/conversation.py b/fastchat/conversation.py
index e33564e97..5752de345 100644
--- a/fastchat/conversation.py
+++ b/fastchat/conversation.py
@@ -1,5 +1,8 @@
 """
 Conversation prompt templates.
+
+We kindly request that you import fastchat instead of copying this file if you want to use it.
+You can contribute back the changes you want to make.
 """
 
 import dataclasses
diff --git a/pyproject.toml b/pyproject.toml
index 4b4f72a39..1a11b03dc 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -13,10 +13,10 @@ classifiers = [
     "License :: OSI Approved :: Apache Software License",
 ]
 dependencies = [
-    "accelerate", "fastapi", "gradio", "httpx", "markdown2[all]", "nh3", "numpy",
+    "accelerate>=0.20.3", "einops", "fastapi", "gradio", "httpx", "markdown2[all]", "nh3", "numpy",
     "peft", "prompt_toolkit>=3.0.0", "pydantic<=2.0", "requests", "rich>=10.0.0", "sentencepiece",
     "shortuuid", "shortuuid", "tiktoken", "tokenizers>=0.12.1", "torch",
-    "transformers>=4.28.0,<4.29.0", "uvicorn", "wandb",
+    "transformers>=4.31.0", "uvicorn", "wandb"
 ]
 
 [project.optional-dependencies]


Test patch:
diff --git a/tests/test_cli.py b/tests/test_cli.py
index ceb7480f1..4b8dbcc19 100644
--- a/tests/test_cli.py
+++ b/tests/test_cli.py
@@ -44,7 +44,7 @@ def test_multi_gpu():
     for model_path in models:
         cmd = (
             f"python3 -m fastchat.serve.cli --model-path {model_path} "
-            f"--style programmatic --num-gpus 2 < test_cli_inputs.txt"
+            f"--style programmatic --num-gpus 2 --max-gpu-memory 14Gib < test_cli_inputs.txt"
         )
         ret = run_cmd(cmd)
         if ret != 0: