Repository: hpcaitech/ColossalAI

Issue:
[BUG]: fine train llama-2-7b-hf prepare data set error , `bos_token` and `eos_token` should be the same with `conversation_template.seps`.
### Is there an existing issue for this bug?

- [X] I have searched the existing issues

### 🐛 Describe the bug

 python prepare_sft_dataset.py --data_input_dirs /models/train-data-dir  --tokenizer_dir /models/llama-2-13b-hf --data_output_dirs /models/train-data-dir/out  --num_spliced_dataset_bins 1




 File "/ColossalAI/applications/Colossal-LLaMA/colossal_llama/dataset/spliced_and_tokenized_dataset.py", line 90, in supervised_tokenize_sft
    tokenizer.bos_token == conversation_template.seps[0] and tokenizer.eos_token == conversation_template.seps[1]
AssertionError: `bos_token` and `eos_token` should be the same with `conversation_template.seps`.
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/ColossalAI/applications/Colossal-LLaMA/prepare_sft_dataset.py", line 146, in <module>
    main()
  File "/ColossalAI/applications/Colossal-LLaMA/prepare_sft_dataset.py", line 102, in main
    dataset = dataset.map(
  File "/usr/local/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 602, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 567, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 3248, in map
    for rank, done, content in iflatmap_unordered(
  File "/usr/local/lib/python3.10/site-packages/datasets/utils/py_utils.py", line 718, in iflatmap_unordered
    [async_result.get(timeout=0.05) for async_result in async_results]
  File "/usr/local/lib/python3.10/site-packages/datasets/utils/py_utils.py", line 718, in <listcomp>
    [async_result.get(timeout=0.05) for async_result in async_results]
  File "/usr/local/lib/python3.10/site-packages/multiprocess/pool.py", line 774, in get
    raise self._value
**AssertionError: `bos_token` and `eos_token` should be the same with `conversation_template.seps`.**

### Environment

------------ Environment ------------
Colossal-AI version: 0.3.7
PyTorch version: 2.1.0
System CUDA version: 11.0
CUDA version required by PyTorch: 11.8

tokenizer_config.json 

  "add_bos_token": true,
  "add_eos_token": false,
  "bos_token": {
    "__type": "AddedToken",
    "content": "
",
    "lstrip": false,
    "normalized": true,
    "rstrip": false,
    "single_word": false
  },
  "clean_up_tokenization_spaces": false,
  "eos_token": {
    "__type": "AddedToken",
    "content": "
",
    "lstrip": false,
    "normalized": true,
    "rstrip": false,
    "single_word": false
  },
[tensor] fix some unittests

[tensor] fix some unittests



Discussion:
but meta-Llama-3-8B model correct 

root@63733705cb83:/ColossalAI/applications/Colossal-LLaMA# python prepare_sft_dataset.py --data_input_dirs /models/train-data-dir  --tokenizer_dir /models/llama-2-13b-hf^Ct_dirs /models/train-data-dir/out  --num_spliced_dataset_bins 1
root@63733705cb83:/ColossalAI/applications/Colossal-LLaMA# python prepare_sft_dataset.py --data_input_dirs /models/train-data-dir  --tokenizer_dir /models/meta-Llama-3-8B  --data_output_dirs /models/train-data-dir/out  --num_spliced_dataset_bins 1
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
[05/14/24 16:51:56] INFO     colossalai - colossalai - INFO: /ColossalAI/applications/Colossal-LLaMA/prepare_sft_dataset.py:101 main       
                    INFO     colossalai - colossalai - INFO: Start to process part-0/1 of all original datasets.                           
Map (num_proc=8): 100%|███████████████████████████████████████████████████████████████████████████████| 8/8 [00:02<00:00,  3.26 examples/s]
Filter: 100%|████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:00<00:00, 469.98 examples/s]
[05/14/24 16:51:59] INFO     colossalai - colossalai - INFO: /ColossalAI/applications/Colossal-LLaMA/prepare_sft_dataset.py:127 main       
                    INFO     colossalai - colossalai - INFO: processing 0 spliced data points for                                          
                             /models/train-data-dir/out/jsonl/part-00000.jsonl                                                             
                    INFO     colossalai - colossalai - INFO: /ColossalAI/applications/Colossal-LLaMA/prepare_sft_dataset.py:133 main       
                    INFO     colossalai - colossalai - INFO: Start to save /models/train-data-dir/out/arrow/part-00000                     
Setting num_proc from 48 back to 1 for the train split to disable multiprocessing as it only contains one shard.
Generating train split: 8 examples [00:00, 967.99 examples/s]
Saving the dataset (8/8 shards): 100%|████████████████████████████████████████████████████████████████| 8/8 [00:00<00:00, 36.59 examples/s]
Hi,

As the `bos`, `eos` tokens are different for llama2 and llama3, so you need to choose which version of llama you want to process.

We have a parameter to set version of llama here: https://github.com/hpcaitech/ColossalAI/blob/43995ee4365e7fcbef0bbe6ee65e98eda0436f02/applications/Colossal-LLaMA/prepare_sft_dataset.py#L38

The default one is llama3. 
Also you will need to change the default conversation template to llama2 at here: https://github.com/hpcaitech/ColossalAI/blob/43995ee4365e7fcbef0bbe6ee65e98eda0436f02/applications/Colossal-LLaMA/colossal_llama/dataset/conversation.py#L86

We will fix the issue. Thanks.




Resolving patch:
diff --git a/applications/Colossal-LLaMA/prepare_sft_dataset.py b/applications/Colossal-LLaMA/prepare_sft_dataset.py
--- a/applications/Colossal-LLaMA/prepare_sft_dataset.py
+++ b/applications/Colossal-LLaMA/prepare_sft_dataset.py
@@ -10,7 +10,7 @@
 import os
 from multiprocessing import cpu_count
 
-from colossal_llama.dataset.conversation import default_conversation
+from colossal_llama.dataset.conversation import LLaMA2_Conv
 from colossal_llama.dataset.spliced_and_tokenized_dataset import supervised_tokenize_sft
 from datasets import dataset_dict, load_dataset
 from transformers import AddedToken, AutoTokenizer
@@ -78,6 +78,7 @@ def main():
     # Fix 
 split issue: https://github.com/huggingface/transformers/issues/23833
     if args.llama_version == 2:
         tokenizer.add_tokens(AddedToken("
", normalized=False, special=True), special_tokens=True)
+        default_conversation = LLaMA2_Conv
 
     tokenizer.add_bos_token = False
     tokenizer.add_eos_token = False


Test patch: