qwen3微调报错：

### Reminder

- [x] I have read the above rules and searched the existing issues.

### System Info
```
您好，qwen3-32b全参微调时，使用文本数据报错如下，麻烦官方协助支持下哈，谢谢！[2025-04-30 13:22:50,181] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[INFO|2025-04-30 13:22:52] llamafactory.cli:143 >> Initializing 8 distributed tasks at: 10.0.1.6:29500
Multi-node training enabled: num nodes: 4, node rank: 1
W0430 13:22:53.726000 856001 site-packages/torch/distributed/run.py:792] 
W0430 13:22:53.726000 856001 site-packages/torch/distributed/run.py:792] *****************************************
W0430 13:22:53.726000 856001 site-packages/torch/distributed/run.py:792] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
W0430 13:22:53.726000 856001 site-packages/torch/distributed/run.py:792] *****************************************
[2025-04-30 13:23:06,840] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-04-30 13:23:06,840] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-04-30 13:23:06,885] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-04-30 13:23:06,897] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-04-30 13:23:06,897] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-04-30 13:23:06,899] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-04-30 13:23:06,901] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-04-30 13:23:06,955] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
/data1/tools/anaconda3/envs/train/lib/python3.10/site-packages/transformers/training_args.py:1611: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead
  warnings.warn(
/data1/tools/anaconda3/envs/train/lib/python3.10/site-packages/transformers/training_args.py:1611: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead
  warnings.warn(
[2025-04-30 13:23:07,915] [INFO] [comm.py:658:init_distributed] cdb=None
[2025-04-30 13:23:07,916] [INFO] [comm.py:658:init_distributed] cdb=None
/data1/tools/anaconda3/envs/train/lib/python3.10/site-packages/transformers/training_args.py:1611: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead
  warnings.warn(
[2025-04-30 13:23:07,965] [INFO] [comm.py:658:init_distributed] cdb=None
/data1/tools/anaconda3/envs/train/lib/python3.10/site-packages/transformers/training_args.py:1611: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead
  warnings.warn(
/data1/tools/anaconda3/envs/train/lib/python3.10/site-packages/transformers/training_args.py:1611: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead
  warnings.warn(
[2025-04-30 13:23:07,974] [INFO] [comm.py:658:init_distributed] cdb=None
[2025-04-30 13:23:07,974] [INFO] [comm.py:658:init_distributed] cdb=None
/data1/tools/anaconda3/envs/train/lib/python3.10/site-packages/transformers/training_args.py:1611: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead
  warnings.warn(
[2025-04-30 13:23:07,977] [INFO] [comm.py:658:init_distributed] cdb=None
/data1/tools/anaconda3/envs/train/lib/python3.10/site-packages/transformers/training_args.py:1611: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead
  warnings.warn(
[2025-04-30 13:23:07,980] [INFO] [comm.py:658:init_distributed] cdb=None
/data1/tools/anaconda3/envs/train/lib/python3.10/site-packages/transformers/training_args.py:1611: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead
  warnings.warn(
[2025-04-30 13:23:08,011] [INFO] [comm.py:658:init_distributed] cdb=None
[INFO|2025-04-30 13:23:08] llamafactory.hparams.parser:401 >> Process rank: 8, world size: 32, device: cuda:0, distributed training: True, compute dtype: torch.bfloat16
[INFO|tokenization_utils_base.py:2058] 2025-04-30 13:23:08,325 >> loading file vocab.json
[INFO|tokenization_utils_base.py:2058] 2025-04-30 13:23:08,325 >> loading file merges.txt
[INFO|tokenization_utils_base.py:2058] 2025-04-30 13:23:08,325 >> loading file tokenizer.json
[INFO|tokenization_utils_base.py:2058] 2025-04-30 13:23:08,325 >> loading file added_tokens.json
[INFO|tokenization_utils_base.py:2058] 2025-04-30 13:23:08,325 >> loading file special_tokens_map.json
[INFO|tokenization_utils_base.py:2058] 2025-04-30 13:23:08,325 >> loading file tokenizer_config.json
[INFO|tokenization_utils_base.py:2058] 2025-04-30 13:23:08,325 >> loading file chat_template.jinja
[INFO|2025-04-30 13:23:08] llamafactory.hparams.parser:401 >> Process rank: 11, world size: 32, device: cuda:3, distributed training: True, compute dtype: torch.bfloat16
[INFO|2025-04-30 13:23:08] llamafactory.hparams.parser:401 >> Process rank: 9, world size: 32, device: cuda:1, distributed training: True, compute dtype: torch.bfloat16
[INFO|2025-04-30 13:23:08] llamafactory.hparams.parser:401 >> Process rank: 12, world size: 32, device: cuda:4, distributed training: True, compute dtype: torch.bfloat16
[INFO|2025-04-30 13:23:08] llamafactory.hparams.parser:401 >> Process rank: 15, world size: 32, device: cuda:7, distributed training: True, compute dtype: torch.bfloat16
[INFO|2025-04-30 13:23:08] llamafactory.hparams.parser:401 >> Process rank: 10, world size: 32, device: cuda:2, distributed training: True, compute dtype: torch.bfloat16
[INFO|2025-04-30 13:23:08] llamafactory.hparams.parser:401 >> Process rank: 13, world size: 32, device: cuda:5, distributed training: True, compute dtype: torch.bfloat16
[INFO|2025-04-30 13:23:08] llamafactory.hparams.parser:401 >> Process rank: 14, world size: 32, device: cuda:6, distributed training: True, compute dtype: torch.bfloat16
[INFO|tokenization_utils_base.py:2323] 2025-04-30 13:23:08,687 >> Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
[INFO|configuration_utils.py:697] 2025-04-30 13:23:08,688 >> loading configuration file /data1/llms/huggingface/hub/models--Qwen--Qwen3-32B/snapshots/30b8421510892303dc5ddd6cd0ac90ca2053478d/config.json
[INFO|2025-04-30 13:23:08] llamafactory.model.loader:143 >> Failed to load processor: The checkpoint you are trying to load has model type `qwen3` but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date.

You can update Transformers with the command `pip install --upgrade transformers`. If this does not work, and the checkpoint is very new, then there may not be a release version that supports this model yet. In this case, you can get the most up-to-date code by installing Transformers from source with the command `pip install git+https://github.com/huggingface/transformers.git`.
[INFO|2025-04-30 13:23:08] llamafactory.data.loader:143 >> Loading dataset /data1/train/data/medical_0430.jsonl...
[rank11]:[W430 13:23:08.630141461 ProcessGroupNCCL.cpp:4561] [PG ID 0 PG GUID 0 Rank 11]  using GPU 3 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id.
[rank12]:[W430 13:23:08.640143807 ProcessGroupNCCL.cpp:4561] [PG ID 0 PG GUID 0 Rank 12]  using GPU 4 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id.
[rank9]:[W430 13:23:08.643703867 ProcessGroupNCCL.cpp:4561] [PG ID 0 PG GUID 0 Rank 9]  using GPU 1 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id.
[rank10]:[W430 13:23:08.647022182 ProcessGroupNCCL.cpp:4561] [PG ID 0 PG GUID 0 Rank 10]  using GPU 2 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id.
[rank15]:[W430 13:23:08.647669136 ProcessGroupNCCL.cpp:4561] [PG ID 0 PG GUID 0 Rank 15]  using GPU 7 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id.
[rank13]:[W430 13:23:08.651389961 ProcessGroupNCCL.cpp:4561] [PG ID 0 PG GUID 0 Rank 13]  using GPU 5 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id.
[rank14]:[W430 13:23:08.655567432 ProcessGroupNCCL.cpp:4561] [PG ID 0 PG GUID 0 Rank 14]  using GPU 6 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id.
Converting format of dataset (num_proc=16): 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████| 80000/80000 [00:00<00:00, 189703.71 examples/s]
[rank8]:[W430 13:23:10.226266804 ProcessGroupNCCL.cpp:4561] [PG ID 0 PG GUID 0 Rank 8]  using GPU 0 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id.
Running tokenizer on dataset (num_proc=16):  68%|████████████████████████████████████████████████████████████████████████▉                                   | 54000/80000 [00:08<00:04, 6464.90 examples/s]
[rank8]: multiprocess.pool.RemoteTraceback: 
[rank8]: """
[rank8]: Traceback (most recent call last):
[rank8]:   File "/data1/tools/anaconda3/envs/train/lib/python3.10/site-packages/multiprocess/pool.py", line 125, in worker
[rank8]:     result = (True, func(*args, **kwds))
[rank8]:   File "/data1/tools/anaconda3/envs/train/lib/python3.10/site-packages/datasets/utils/py_utils.py", line 678, in _write_generator_to_queue
[rank8]:     for i, result in enumerate(func(**kwargs)):
[rank8]:   File "/data1/tools/anaconda3/envs/train/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 3476, in _map_single
[rank8]:     batch = apply_function_on_filtered_inputs(
[rank8]:   File "/data1/tools/anaconda3/envs/train/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 3338, in apply_function_on_filtered_inputs
[rank8]:     processed_inputs = function(*fn_args, *additional_args, **fn_kwargs)
[rank8]:   File "/data1/train/LLaMA-Factory-main/src/llamafactory/data/processor/supervised.py", line 99, in preprocess_dataset
[rank8]:     input_ids, labels = self._encode_data_example(
[rank8]:   File "/data1/train/LLaMA-Factory-main/src/llamafactory/data/processor/supervised.py", line 43, in _encode_data_example
[rank8]:     messages = self.template.mm_plugin.process_messages(prompt + response, images, videos, audios, self.processor)
[rank8]:   File "/data1/train/LLaMA-Factory-main/src/llamafactory/data/mm_plugin.py", line 396, in process_messages
[rank8]:     self._validate_messages(messages, images, videos, audios)
[rank8]:   File "/data1/train/LLaMA-Factory-main/src/llamafactory/data/mm_plugin.py", line 196, in _validate_messages
[rank8]:     raise ValueError(
[rank8]: ValueError: The number of images does not match the number of <image> tokens in [{'content': '我想知道帝国大厦的开放时间，以及游客需要经过哪些步骤才能参观？\n<image>', 'role': 'user'}, {'content': '帝国大厦位于美国纽约市，它的开放时间根据季节会有所不同。通常，在夏季（约6月至8月），帝国大厦的开放时间从早上8:00至晚上9:00，而在冬季（约11月至次年2月），开放时间可能会缩短到早上8:00至晚上7:00。为了参观，游客需要采取以下步骤：\n\n1. **提前在线购票**：因为帝国大厦的参观人数较多，尤其是高峰期，建议提前通过官方网站或指定线上平台购票，以避免现场长时间排队等候。在线购票通常会提供跳过部分排队环节或快速通道的选项。\n\n2. **选择合适的参观时间**：尽量避开人多的旺季或周五、周六这样的旅游高峰日，选择工作日的早中时段可能会人流较少，等待时间较短。\n\n3. **到达大厦**：提前抵达大厦进行安全检查和进行一般参观的导览区。帝国大厦入口位置的具体信息可以在官网进行查询，通常建议由第34或35街与布莱恩特公园（Bryant Park）之间的第五大道（5th Avenue）进入。\n\n4. **通过安检**：参观者需要通过安全检查。为了减少检查时间，避免随身携带大量的液体和非必要的电子设备。\n\n5. **向上层观景台**：通过电梯，到达帝国大厦的观景台。帝国大厦设有两个观景台，分别是位于第86层的室内观景台和位于第102层的室外观景台（需额外付费）。请务必根据您的需求和天气状况选择合适的观景台。\n\n6. **享受景点**：参观完观景台后，您可以选择在大楼内探索其他区域，如礼品店、餐厅等，根据个人兴趣进行自由活动。\n\n请记得随时查看帝国大厦官方公布的最新信息，包括开放时间和可能的临时变更，以确保您的旅行计划顺利进行。', 'role': 'assistant'}].
[rank8]: """

[rank8]: The above exception was the direct cause of the following exception:

[rank8]: Traceback (most recent call last):
[rank8]:   File "/data1/train/LLaMA-Factory-main/src/llamafactory/launcher.py", line 23, in <module>
[rank8]:     launch()
[rank8]:   File "/data1/train/LLaMA-Factory-main/src/llamafactory/launcher.py", line 19, in launch
[rank8]:     run_exp()
[rank8]:   File "/data1/train/LLaMA-Factory-main/src/llamafactory/train/tuner.py", line 110, in run_exp
[rank8]:     _training_function(config={"args": args, "callbacks": callbacks})
[rank8]:   File "/data1/train/LLaMA-Factory-main/src/llamafactory/train/tuner.py", line 72, in _training_function
[rank8]:     run_sft(model_args, data_args, training_args, finetuning_args, generating_args, callbacks)
[rank8]:   File "/data1/train/LLaMA-Factory-main/src/llamafactory/train/sft/workflow.py", line 51, in run_sft
[rank8]:     dataset_module = get_dataset(template, model_args, data_args, training_args, stage="sft", **tokenizer_module)
[rank8]:   File "/data1/train/LLaMA-Factory-main/src/llamafactory/data/loader.py", line 315, in get_dataset
[rank8]:     dataset = _get_preprocessed_dataset(
[rank8]:   File "/data1/train/LLaMA-Factory-main/src/llamafactory/data/loader.py", line 256, in _get_preprocessed_dataset
[rank8]:     dataset = dataset.map(
[rank8]:   File "/data1/tools/anaconda3/envs/train/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 560, in wrapper
[rank8]:     out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
[rank8]:   File "/data1/tools/anaconda3/envs/train/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 3165, in map
[rank8]:     for rank, done, content in iflatmap_unordered(
[rank8]:   File "/data1/tools/anaconda3/envs/train/lib/python3.10/site-packages/datasets/utils/py_utils.py", line 718, in iflatmap_unordered
[rank8]:     [async_result.get(timeout=0.05) for async_result in async_results]
[rank8]:   File "/data1/tools/anaconda3/envs/train/lib/python3.10/site-packages/datasets/utils/py_utils.py", line 718, in <listcomp>
[rank8]:     [async_result.get(timeout=0.05) for async_result in async_results]
[rank8]:   File "/data1/tools/anaconda3/envs/train/lib/python3.10/site-packages/multiprocess/pool.py", line 774, in get
[rank8]:     raise self._value
[rank8]: ValueError: The number of images does not match the number of <image> tokens in [{'content': '我想知道帝国大厦的开放时间，以及游客需要经过哪些步骤才能参观？\n<image>', 'role': 'user'}, {'content': '帝国大厦位于美国纽约市，它的开放时间根据季节会有所不同。通常，在夏季（约6月至8月），帝国大厦的开放时间从早上8:00至晚上9:00，而在冬季（约11月至次年2月），开放时间可能会缩短到早上8:00至晚上7:00。为了参观，游客需要采取以下步骤：\n\n1. **提前在线购票**：因为帝国大厦的参观人数较多，尤其是高峰期，建议提前通过官方网站或指定线上平台购票，以避免现场长时间排队等候。在线购票通常会提供跳过部分排队环节或快速通道的选项。\n\n2. **选择合适的参观时间**：尽量避开人多的旺季或周五、周六这样的旅游高峰日，选择工作日的早中时段可能会人流较少，等待时间较短。\n\n3. **到达大厦**：提前抵达大厦进行安全检查和进行一般参观的导览区。帝国大厦入口位置的具体信息可以在官网进行查询，通常建议由第34或35街与布莱恩特公园（Bryant Park）之间的第五大道（5th Avenue）进入。\n\n4. **通过安检**：参观者需要通过安全检查。为了减少检查时间，避免随身携带大量的液体和非必要的电子设备。\n\n5. **向上层观景台**：通过电梯，到达帝国大厦的观景台。帝国大厦设有两个观景台，分别是位于第86层的室内观景台和位于第102层的室外观景台（需额外付费）。请务必根据您的需求和天气状况选择合适的观景台。\n\n6. **享受景点**：参观完观景台后，您可以选择在大楼内探索其他区域，如礼品店、餐厅等，根据个人兴趣进行自由活动。\n\n请记得随时查看帝国大厦官方公布的最新信息，包括开放时间和可能的临时变更，以确保您的旅行计划顺利进行。', 'role': 'assistant'}].
Running tokenizer on dataset (num_proc=16):   0%|                                                                                                                          | 0/80000 [00:00<?, ? examples/s][rank8]:[W430 13:23:21.572119160 ProcessGroupNCCL.cpp:1496] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
Running tokenizer on dataset (num_proc=16):  18%|██████████████████▉                                                                                         | 14000/80000 [00:07<00:37, 1768.22 examples/s]
[rank14]: multiprocess.pool.RemoteTraceback: 
[rank14]: """
[rank14]: Traceback (most recent call last):
[rank14]:   File "/data1/tools/anaconda3/envs/train/lib/python3.10/site-packages/multiprocess/pool.py", line 125, in worker
[rank14]:     result = (True, func(*args, **kwds))
[rank14]:   File "/data1/tools/anaconda3/envs/train/lib/python3.10/site-packages/datasets/utils/py_utils.py", line 678, in _write_generator_to_queue
[rank14]:     for i, result in enumerate(func(**kwargs)):
[rank14]:   File "/data1/tools/anaconda3/envs/train/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 3476, in _map_single
[rank14]:     batch = apply_function_on_filtered_inputs(
[rank14]:   File "/data1/tools/anaconda3/envs/train/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 3338, in apply_function_on_filtered_inputs
[rank14]:     processed_inputs = function(*fn_args, *additional_args, **fn_kwargs)
[rank14]:   File "/data1/train/LLaMA-Factory-main/src/llamafactory/data/processor/supervised.py", line 99, in preprocess_dataset
[rank14]:     input_ids, labels = self._encode_data_example(
[rank14]:   File "/data1/train/LLaMA-Factory-main/src/llamafactory/data/processor/supervised.py", line 43, in _encode_data_example
[rank14]:     messages = self.template.mm_plugin.process_messages(prompt + response, images, videos, audios, self.processor)
[rank14]:   File "/data1/train/LLaMA-Factory-main/src/llamafactory/data/mm_plugin.py", line 396, in process_messages
[rank14]:     self._validate_messages(messages, images, videos, audios)
[rank14]:   File "/data1/train/LLaMA-Factory-main/src/llamafactory/data/mm_plugin.py", line 196, in _validate_messages
[rank14]:     raise ValueError(
[rank14]: ValueError: The number of images does not match the number of <image> tokens in [{'content': '我想知道帝国大厦的开放时间，以及游客需要经过哪些步骤才能参观？\n<image>', 'role': 'user'}, {'content': '帝国大厦位于美国纽约市，它的开放时间根据季节会有所不同。通常，在夏季（约6月至8月），帝国大厦的开放时间从早上8:00至晚上9:00，而在冬季（约11月至次年2月），开放时间可能会缩短到早上8:00至晚上7:00。为了参观，游客需要采取以下步骤：\n\n1. **提前在线购票**：因为帝国大厦的参观人数较多，尤其是高峰期，建议提前通过官方网站或指定线上平台购票，以避免现场长时间排队等候。在线购票通常会提供跳过部分排队环节或快速通道的选项。\n\n2. **选择合适的参观时间**：尽量避开人多的旺季或周五、周六这样的旅游高峰日，选择工作日的早中时段可能会人流较少，等待时间较短。\n\n3. **到达大厦**：提前抵达大厦进行安全检查和进行一般参观的导览区。帝国大厦入口位置的具体信息可以在官网进行查询，通常建议由第34或35街与布莱恩特公园（Bryant Park）之间的第五大道（5th Avenue）进入。\n\n4. **通过安检**：参观者需要通过安全检查。为了减少检查时间，避免随身携带大量的液体和非必要的电子设备。\n\n5. **向上层观景台**：通过电梯，到达帝国大厦的观景台。帝国大厦设有两个观景台，分别是位于第86层的室内观景台和位于第102层的室外观景台（需额外付费）。请务必根据您的需求和天气状况选择合适的观景台。\n\n6. **享受景点**：参观完观景台后，您可以选择在大楼内探索其他区域，如礼品店、餐厅等，根据个人兴趣进行自由活动。\n\n请记得随时查看帝国大厦官方公布的最新信息，包括开放时间和可能的临时变更，以确保您的旅行计划顺利进行。', 'role': 'assistant'}].
[rank14]: """

[rank14]: The above exception was the direct cause of the following exception:

[rank14]: Traceback (most recent call last):
[rank14]:   File "/data1/train/LLaMA-Factory-main/src/llamafactory/launcher.py", line 23, in <module>
[rank14]:     launch()
[rank14]:   File "/data1/train/LLaMA-Factory-main/src/llamafactory/launcher.py", line 19, in launch
[rank14]:     run_exp()
[rank14]:   File "/data1/train/LLaMA-Factory-main/src/llamafactory/train/tuner.py", line 110, in run_exp
[rank14]:     _training_function(config={"args": args, "callbacks": callbacks})
[rank14]:   File "/data1/train/LLaMA-Factory-main/src/llamafactory/train/tuner.py", line 72, in _training_function
[rank14]:     run_sft(model_args, data_args, training_args, finetuning_args, generating_args, callbacks)
[rank14]:   File "/data1/train/LLaMA-Factory-main/src/llamafactory/train/sft/workflow.py", line 51, in run_sft
[rank14]:     dataset_module = get_dataset(template, model_args, data_args, training_args, stage="sft", **tokenizer_module)
[rank14]:   File "/data1/train/LLaMA-Factory-main/src/llamafactory/data/loader.py", line 315, in get_dataset
[rank14]:     dataset = _get_preprocessed_dataset(
[rank14]:   File "/data1/train/LLaMA-Factory-main/src/llamafactory/data/loader.py", line 256, in _get_preprocessed_dataset
[rank14]:     dataset = dataset.map(
[rank14]:   File "/data1/tools/anaconda3/envs/train/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 560, in wrapper
[rank14]:     out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
[rank14]:   File "/data1/tools/anaconda3/envs/train/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 3165, in map
[rank14]:     for rank, done, content in iflatmap_unordered(
[rank14]:   File "/data1/tools/anaconda3/envs/train/lib/python3.10/site-packages/datasets/utils/py_utils.py", line 718, in iflatmap_unordered
[rank14]:     [async_result.get(timeout=0.05) for async_result in async_results]
[rank14]:   File "/data1/tools/anaconda3/envs/train/lib/python3.10/site-packages/datasets/utils/py_utils.py", line 718, in <listcomp>
[rank14]:     [async_result.get(timeout=0.05) for async_result in async_results]
[rank14]:   File "/data1/tools/anaconda3/envs/train/lib/python3.10/site-packages/multiprocess/pool.py", line 774, in get
[rank14]:     raise self._value
[rank14]: ValueError: The number of images does not match the number of <image> tokens in [{'content': '我想知道帝国大厦的开放时间，以及游客需要经过哪些步骤才能参观？\n<image>', 'role': 'user'}, {'content': '帝国大厦位于美国纽约市，它的开放时间根据季节会有所不同。通常，在夏季（约6月至8月），帝国大厦的开放时间从早上8:00至晚上9:00，而在冬季（约11月至次年2月），开放时间可能会缩短到早上8:00至晚上7:00。为了参观，游客需要采取以下步骤：\n\n1. **提前在线购票**：因为帝国大厦的参观人数较多，尤其是高峰期，建议提前通过官方网站或指定线上平台购票，以避免现场长时间排队等候。在线购票通常会提供跳过部分排队环节或快速通道的选项。\n\n2. **选择合适的参观时间**：尽量避开人多的旺季或周五、周六这样的旅游高峰日，选择工作日的早中时段可能会人流较少，等待时间较短。\n\n3. **到达大厦**：提前抵达大厦进行安全检查和进行一般参观的导览区。帝国大厦入口位置的具体信息可以在官网进行查询，通常建议由第34或35街与布莱恩特公园（Bryant Park）之间的第五大道（5th Avenue）进入。\n\n4. **通过安检**：参观者需要通过安全检查。为了减少检查时间，避免随身携带大量的液体和非必要的电子设备。\n\n5. **向上层观景台**：通过电梯，到达帝国大厦的观景台。帝国大厦设有两个观景台，分别是位于第86层的室内观景台和位于第102层的室外观景台（需额外付费）。请务必根据您的需求和天气状况选择合适的观景台。\n\n6. **享受景点**：参观完观景台后，您可以选择在大楼内探索其他区域，如礼品店、餐厅等，根据个人兴趣进行自由活动。\n\n请记得随时查看帝国大厦官方公布的最新信息，包括开放时间和可能的临时变更，以确保您的旅行计划顺利进行。', 'role': 'assistant'}].
Running tokenizer on dataset (num_proc=16):  18%|██████████████████▉                                                                                         | 14000/80000 [00:07<00:36, 1794.80 examples/s]
Running tokenizer on dataset (num_proc=16):  18%|██████████████████▉                                                                                         | 14000/80000 [00:07<00:37, 1754.38 examples/s]
Running tokenizer on dataset (num_proc=16):  18%|██████████████████▉                                                                                         | 14000/80000 [00:07<00:37, 1781.05 examples/s]
[rank13]: multiprocess.pool.RemoteTraceback: 
[rank13]: """
[rank13]: Traceback (most recent call last):
[rank13]:   File "/data1/tools/anaconda3/envs/train/lib/python3.10/site-packages/multiprocess/pool.py", line 125, in worker
[rank13]:     result = (True, func(*args, **kwds))
[rank13]:   File "/data1/tools/anaconda3/envs/train/lib/python3.10/site-packages/datasets/utils/py_utils.py", line 678, in _write_generator_to_queue
[rank13]:     for i, result in enumerate(func(**kwargs)):
[rank13]:   File "/data1/tools/anaconda3/envs/train/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 3476, in _map_single
[rank13]:     batch = apply_function_on_filtered_inputs(
[rank13]:   File "/data1/tools/anaconda3/envs/train/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 3338, in apply_function_on_filtered_inputs
[rank13]:     processed_inputs = function(*fn_args, *additional_args, **fn_kwargs)
[rank13]:   File "/data1/train/LLaMA-Factory-main/src/llamafactory/data/processor/supervised.py", line 99, in preprocess_dataset
[rank13]:     input_ids, labels = self._encode_data_example(
[rank13]:   File "/data1/train/LLaMA-Factory-main/src/llamafactory/data/processor/supervised.py", line 43, in _encode_data_example
[rank13]:     messages = self.template.mm_plugin.process_messages(prompt + response, images, videos, audios, self.processor)
[rank13]:   File "/data1/train/LLaMA-Factory-main/src/llamafactory/data/mm_plugin.py", line 396, in process_messages
[rank13]:     self._validate_messages(messages, images, videos, audios)
[rank13]:   File "/data1/train/LLaMA-Factory-main/src/llamafactory/data/mm_plugin.py", line 196, in _validate_messages
[rank13]:     raise ValueError(
[rank13]: ValueError: The number of images does not match the number of <image> tokens in [{'content': '我想知道帝国大厦的开放时间，以及游客需要经过哪些步骤才能参观？\n<image>', 'role': 'user'}, {'content': '帝国大厦位于美国纽约市，它的开放时间根据季节会有所不同。通常，在夏季（约6月至8月），帝国大厦的开放时间从早上8:00至晚上9:00，而在冬季（约11月至次年2月），开放时间可能会缩短到早上8:00至晚上7:00。为了参观，游客需要采取以下步骤：\n\n1. **提前在线购票**：因为帝国大厦的参观人数较多，尤其是高峰期，建议提前通过官方网站或指定线上平台购票，以避免现场长时间排队等候。在线购票通常会提供跳过部分排队环节或快速通道的选项。\n\n2. **选择合适的参观时间**：尽量避开人多的旺季或周五、周六这样的旅游高峰日，选择工作日的早中时段可能会人流较少，等待时间较短。\n\n3. **到达大厦**：提前抵达大厦进行安全检查和进行一般参观的导览区。帝国大厦入口位置的具体信息可以在官网进行查询，通常建议由第34或35街与布莱恩特公园（Bryant Park）之间的第五大道（5th Avenue）进入。\n\n4. **通过安检**：参观者需要通过安全检查。为了减少检查时间，避免随身携带大量的液体和非必要的电子设备。\n\n5. **向上层观景台**：通过电梯，到达帝国大厦的观景台。帝国大厦设有两个观景台，分别是位于第86层的室内观景台和位于第102层的室外观景台（需额外付费）。请务必根据您的需求和天气状况选择合适的观景台。\n\n6. **享受景点**：参观完观景台后，您可以选择在大楼内探索其他区域，如礼品店、餐厅等，根据个人兴趣进行自由活动。\n\n请记得随时查看帝国大厦官方公布的最新信息，包括开放时间和可能的临时变更，以确保您的旅行计划顺利进行。', 'role': 'assistant'}].
[rank13]: """

[rank13]: The above exception was the direct cause of the following exception:

[rank13]: Traceback (most recent call last):
[rank13]:   File "/data1/train/LLaMA-Factory-main/src/llamafactory/launcher.py", line 23, in <module>
[rank13]:     launch()
[rank13]:   File "/data1/train/LLaMA-Factory-main/src/llamafactory/launcher.py", line 19, in launch
[rank13]:     run_exp()
[rank13]:   File "/data1/train/LLaMA-Factory-main/src/llamafactory/train/tuner.py", line 110, in run_exp
[rank13]:     _training_function(config={"args": args, "callbacks": callbacks})
[rank13]:   File "/data1/train/LLaMA-Factory-main/src/llamafactory/train/tuner.py", line 72, in _training_function
[rank13]:     run_sft(model_args, data_args, training_args, finetuning_args, generating_args, callbacks)
[rank13]:   File "/data1/train/LLaMA-Factory-main/src/llamafactory/train/sft/workflow.py", line 51, in run_sft
[rank13]:     dataset_module = get_dataset(template, model_args, data_args, training_args, stage="sft", **tokenizer_module)
[rank13]:   File "/data1/train/LLaMA-Factory-main/src/llamafactory/data/loader.py", line 315, in get_dataset
[rank13]:     dataset = _get_preprocessed_dataset(
[rank13]:   File "/data1/train/LLaMA-Factory-main/src/llamafactory/data/loader.py", line 256, in _get_preprocessed_dataset
[rank13]:     dataset = dataset.map(
[rank13]:   File "/data1/tools/anaconda3/envs/train/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 560, in wrapper
[rank13]:     out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
[rank13]:   File "/data1/tools/anaconda3/envs/train/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 3165, in map
[rank13]:     for rank, done, content in iflatmap_unordered(
[rank13]:   File "/data1/tools/anaconda3/envs/train/lib/python3.10/site-packages/datasets/utils/py_utils.py", line 718, in iflatmap_unordered
[rank13]:     [async_result.get(timeout=0.05) for async_result in async_results]
[rank13]:   File "/data1/tools/anaconda3/envs/train/lib/python3.10/site-packages/datasets/utils/py_utils.py", line 718, in <listcomp>
[rank13]:     [async_result.get(timeout=0.05) for async_result in async_results]
[rank13]:   File "/data1/tools/anaconda3/envs/train/lib/python3.10/site-packages/multiprocess/pool.py", line 774, in get
[rank13]:     raise self._value
[rank13]: ValueError: The number of images does not match the number of <image> tokens in [{'content': '我想知道帝国大厦的开放时间，以及游客需要经过哪些步骤才能参观？\n<image>', 'role': 'user'}, {'content': '帝国大厦位于美国纽约市，它的开放时间根据季节会有所不同。通常，在夏季（约6月至8月），帝国大厦的开放时间从早上8:00至晚上9:00，而在冬季（约11月至次年2月），开放时间可能会缩短到早上8:00至晚上7:00。为了参观，游客需要采取以下步骤：\n\n1. **提前在线购票**：因为帝国大厦的参观人数较多，尤其是高峰期，建议提前通过官方网站或指定线上平台购票，以避免现场长时间排队等候。在线购票通常会提供跳过部分排队环节或快速通道的选项。\n\n2. **选择合适的参观时间**：尽量避开人多的旺季或周五、周六这样的旅游高峰日，选择工作日的早中时段可能会人流较少，等待时间较短。\n\n3. **到达大厦**：提前抵达大厦进行安全检查和进行一般参观的导览区。帝国大厦入口位置的具体信息可以在官网进行查询，通常建议由第34或35街与布莱恩特公园（Bryant Park）之间的第五大道（5th Avenue）进入。\n\n4. **通过安检**：参观者需要通过安全检查。为了减少检查时间，避免随身携带大量的液体和非必要的电子设备。\n\n5. **向上层观景台**：通过电梯，到达帝国大厦的观景台。帝国大厦设有两个观景台，分别是位于第86层的室内观景台和位于第102层的室外观景台（需额外付费）。请务必根据您的需求和天气状况选择合适的观景台。\n\n6. **享受景点**：参观完观景台后，您可以选择在大楼内探索其他区域，如礼品店、餐厅等，根据个人兴趣进行自由活动。\n\n请记得随时查看帝国大厦官方公布的最新信息，包括开放时间和可能的临时变更，以确保您的旅行计划顺利进行。', 'role': 'assistant'}].
Running tokenizer on dataset (num_proc=16):  18%|██████████████████▉                                                                                         | 14000/80000 [00:08<00:37, 1745.19 examples/s]
[rank10]: multiprocess.pool.RemoteTraceback: 
[rank10]: """
[rank10]: Traceback (most recent call last):
[rank10]:   File "/data1/tools/anaconda3/envs/train/lib/python3.10/site-packages/multiprocess/pool.py", line 125, in worker
[rank10]:     result = (True, func(*args, **kwds))
[rank10]:   File "/data1/tools/anaconda3/envs/train/lib/python3.10/site-packages/datasets/utils/py_utils.py", line 678, in _write_generator_to_queue
[rank10]:     for i, result in enumerate(func(**kwargs)):
[rank10]:   File "/data1/tools/anaconda3/envs/train/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 3476, in _map_single
[rank10]:     batch = apply_function_on_filtered_inputs(
[rank10]:   File "/data1/tools/anaconda3/envs/train/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 3338, in apply_function_on_filtered_inputs
[rank10]:     processed_inputs = function(*fn_args, *additional_args, **fn_kwargs)
[rank10]:   File "/data1/train/LLaMA-Factory-main/src/llamafactory/data/processor/supervised.py", line 99, in preprocess_dataset
[rank10]:     input_ids, labels = self._encode_data_example(
[rank10]:   File "/data1/train/LLaMA-Factory-main/src/llamafactory/data/processor/supervised.py", line 43, in _encode_data_example
[rank10]:     messages = self.template.mm_plugin.process_messages(prompt + response, images, videos, audios, self.processor)
[rank10]:   File "/data1/train/LLaMA-Factory-main/src/llamafactory/data/mm_plugin.py", line 396, in process_messages
[rank10]:     self._validate_messages(messages, images, videos, audios)
[rank10]:   File "/data1/train/LLaMA-Factory-main/src/llamafactory/data/mm_plugin.py", line 196, in _validate_messages
[rank10]:     raise ValueError(
[rank10]: ValueError: The number of images does not match the number of <image> tokens in [{'content': '我想知道帝国大厦的开放时间，以及游客需要经过哪些步骤才能参观？\n<image>', 'role': 'user'}, {'content': '帝国大厦位于美国纽约市，它的开放时间根据季节会有所不同。通常，在夏季（约6月至8月），帝国大厦的开放时间从早上8:00至晚上9:00，而在冬季（约11月至次年2月），开放时间可能会缩短到早上8:00至晚上7:00。为了参观，游客需要采取以下步骤：\n\n1. **提前在线购票**：因为帝国大厦的参观人数较多，尤其是高峰期，建议提前通过官方网站或指定线上平台购票，以避免现场长时间排队等候。在线购票通常会提供跳过部分排队环节或快速通道的选项。\n\n2. **选择合适的参观时间**：尽量避开人多的旺季或周五、周六这样的旅游高峰日，选择工作日的早中时段可能会人流较少，等待时间较短。\n\n3. **到达大厦**：提前抵达大厦进行安全检查和进行一般参观的导览区。帝国大厦入口位置的具体信息可以在官网进行查询，通常建议由第34或35街与布莱恩特公园（Bryant Park）之间的第五大道（5th Avenue）进入。\n\n4. **通过安检**：参观者需要通过安全检查。为了减少检查时间，避免随身携带大量的液体和非必要的电子设备。\n\n5. **向上层观景台**：通过电梯，到达帝国大厦的观景台。帝国大厦设有两个观景台，分别是位于第86层的室内观景台和位于第102层的室外观景台（需额外付费）。请务必根据您的需求和天气状况选择合适的观景台。\n\n6. **享受景点**：参观完观景台后，您可以选择在大楼内探索其他区域，如礼品店、餐厅等，根据个人兴趣进行自由活动。\n\n请记得随时查看帝国大厦官方公布的最新信息，包括开放时间和可能的临时变更，以确保您的旅行计划顺利进行。', 'role': 'assistant'}].
[rank10]: """

[rank10]: The above exception was the direct cause of the following exception:

[rank10]: Traceback (most recent call last):
[rank10]:   File "/data1/train/LLaMA-Factory-main/src/llamafactory/launcher.py", line 23, in <module>
[rank10]:     launch()
[rank10]:   File "/data1/train/LLaMA-Factory-main/src/llamafactory/launcher.py", line 19, in launch
[rank10]:     run_exp()
[rank10]:   File "/data1/train/LLaMA-Factory-main/src/llamafactory/train/tuner.py", line 110, in run_exp
[rank10]:     _training_function(config={"args": args, "callbacks": callbacks})
[rank10]:   File "/data1/train/LLaMA-Factory-main/src/llamafactory/train/tuner.py", line 72, in _training_function
[rank10]:     run_sft(model_args, data_args, training_args, finetuning_args, generating_args, callbacks)
[rank10]:   File "/data1/train/LLaMA-Factory-main/src/llamafactory/train/sft/workflow.py", line 51, in run_sft
[rank10]:     dataset_module = get_dataset(template, model_args, data_args, training_args, stage="sft", **tokenizer_module)
[rank10]:   File "/data1/train/LLaMA-Factory-main/src/llamafactory/data/loader.py", line 315, in get_dataset
[rank10]:     dataset = _get_preprocessed_dataset(
[rank10]:   File "/data1/train/LLaMA-Factory-main/src/llamafactory/data/loader.py", line 256, in _get_preprocessed_dataset
[rank10]:     dataset = dataset.map(
[rank10]:   File "/data1/tools/anaconda3/envs/train/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 560, in wrapper
[rank10]:     out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
[rank10]:   File "/data1/tools/anaconda3/envs/train/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 3165, in map
[rank10]:     for rank, done, content in iflatmap_unordered(
[rank10]:   File "/data1/tools/anaconda3/envs/train/lib/python3.10/site-packages/datasets/utils/py_utils.py", line 718, in iflatmap_unordered
[rank10]:     [async_result.get(timeout=0.05) for async_result in async_results]
[rank10]:   File "/data1/tools/anaconda3/envs/train/lib/python3.10/site-packages/datasets/utils/py_utils.py", line 718, in <listcomp>
[rank10]:     [async_result.get(timeout=0.05) for async_result in async_results]
[rank10]:   File "/data1/tools/anaconda3/envs/train/lib/python3.10/site-packages/multiprocess/pool.py", line 774, in get
[rank10]:     raise self._value
[rank10]: ValueError: The number of images does not match the number of <image> tokens in [{'content': '我想知道帝国大厦的开放时间，以及游客需要经过哪些步骤才能参观？\n<image>', 'role': 'user'}, {'content': '帝国大厦位于美国纽约市，它的开放时间根据季节会有所不同。通常，在夏季（约6月至8月），帝国大厦的开放时间从早上8:00至晚上9:00，而在冬季（约11月至次年2月），开放时间可能会缩短到早上8:00至晚上7:00。为了参观，游客需要采取以下步骤：\n\n1. **提前在线购票**：因为帝国大厦的参观人数较多，尤其是高峰期，建议提前通过官方网站或指定线上平台购票，以避免现场长时间排队等候。在线购票通常会提供跳过部分排队环节或快速通道的选项。\n\n2. **选择合适的参观时间**：尽量避开人多的旺季或周五、周六这样的旅游高峰日，选择工作日的早中时段可能会人流较少，等待时间较短。\n\n3. **到达大厦**：提前抵达大厦进行安全检查和进行一般参观的导览区。帝国大厦入口位置的具体信息可以在官网进行查询，通常建议由第34或35街与布莱恩特公园（Bryant Park）之间的第五大道（5th Avenue）进入。\n\n4. **通过安检**：参观者需要通过安全检查。为了减少检查时间，避免随身携带大量的液体和非必要的电子设备。\n\n5. **向上层观景台**：通过电梯，到达帝国大厦的观景台。帝国大厦设有两个观景台，分别是位于第86层的室内观景台和位于第102层的室外观景台（需额外付费）。请务必根据您的需求和天气状况选择合适的观景台。\n\n6. **享受景点**：参观完观景台后，您可以选择在大楼内探索其他区域，如礼品店、餐厅等，根据个人兴趣进行自由活动。\n\n请记得随时查看帝国大厦官方公布的最新信息，包括开放时间和可能的临时变更，以确保您的旅行计划顺利进行。', 'role': 'assistant'}].
Running tokenizer on dataset (num_proc=16):  18%|██████████████████▉                                                                                         | 14000/80000 [00:07<00:37, 1769.46 examples/s]
[rank11]: multiprocess.pool.RemoteTraceback: 
[rank11]: """
[rank11]: Traceback (most recent call last):
[rank11]:   File "/data1/tools/anaconda3/envs/train/lib/python3.10/site-packages/multiprocess/pool.py", line 125, in worker
[rank11]:     result = (True, func(*args, **kwds))
[rank11]:   File "/data1/tools/anaconda3/envs/train/lib/python3.10/site-packages/datasets/utils/py_utils.py", line 678, in _write_generator_to_queue
[rank11]:     for i, result in enumerate(func(**kwargs)):
[rank11]:   File "/data1/tools/anaconda3/envs/train/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 3476, in _map_single
[rank11]:     batch = apply_function_on_filtered_inputs(
[rank11]:   File "/data1/tools/anaconda3/envs/train/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 3338, in apply_function_on_filtered_inputs
[rank11]:     processed_inputs = function(*fn_args, *additional_args, **fn_kwargs)
[rank11]:   File "/data1/train/LLaMA-Factory-main/src/llamafactory/data/processor/supervised.py", line 99, in preprocess_dataset
[rank11]:     input_ids, labels = self._encode_data_example(
[rank11]:   File "/data1/train/LLaMA-Factory-main/src/llamafactory/data/processor/supervised.py", line 43, in _encode_data_example
[rank11]:     messages = self.template.mm_plugin.process_messages(prompt + response, images, videos, audios, self.processor)
[rank11]:   File "/data1/train/LLaMA-Factory-main/src/llamafactory/data/mm_plugin.py", line 396, in process_messages
[rank11]:     self._validate_messages(messages, images, videos, audios)
[rank11]:   File "/data1/train/LLaMA-Factory-main/src/llamafactory/data/mm_plugin.py", line 196, in _validate_messages
[rank11]:     raise ValueError(
[rank11]: ValueError: The number of images does not match the number of <image> tokens in [{'content': '我想知道帝国大厦的开放时间，以及游客需要经过哪些步骤才能参观？\n<image>', 'role': 'user'}, {'content': '帝国大厦位于美国纽约市，它的开放时间根据季节会有所不同。通常，在夏季（约6月至8月），帝国大厦的开放时间从早上8:00至晚上9:00，而在冬季（约11月至次年2月），开放时间可能会缩短到早上8:00至晚上7:00。为了参观，游客需要采取以下步骤：\n\n1. **提前在线购票**：因为帝国大厦的参观人数较多，尤其是高峰期，建议提前通过官方网站或指定线上平台购票，以避免现场长时间排队等候。在线购票通常会提供跳过部分排队环节或快速通道的选项。\n\n2. **选择合适的参观时间**：尽量避开人多的旺季或周五、周六这样的旅游高峰日，选择工作日的早中时段可能会人流较少，等待时间较短。\n\n3. **到达大厦**：提前抵达大厦进行安全检查和进行一般参观的导览区。帝国大厦入口位置的具体信息可以在官网进行查询，通常建议由第34或35街与布莱恩特公园（Bryant Park）之间的第五大道（5th Avenue）进入。\n\n4. **通过安检**：参观者需要通过安全检查。为了减少检查时间，避免随身携带大量的液体和非必要的电子设备。\n\n5. **向上层观景台**：通过电梯，到达帝国大厦的观景台。帝国大厦设有两个观景台，分别是位于第86层的室内观景台和位于第102层的室外观景台（需额外付费）。请务必根据您的需求和天气状况选择合适的观景台。\n\n6. **享受景点**：参观完观景台后，您可以选择在大楼内探索其他区域，如礼品店、餐厅等，根据个人兴趣进行自由活动。\n\n请记得随时查看帝国大厦官方公布的最新信息，包括开放时间和可能的临时变更，以确保您的旅行计划顺利进行。', 'role': 'assistant'}].
[rank11]: """

[rank11]: The above exception was the direct cause of the following exception:

[rank11]: Traceback (most recent call last):
[rank11]:   File "/data1/train/LLaMA-Factory-main/src/llamafactory/launcher.py", line 23, in <module>
[rank11]:     launch()
[rank11]:   File "/data1/train/LLaMA-Factory-main/src/llamafactory/launcher.py", line 19, in launch
[rank11]:     run_exp()
[rank11]:   File "/data1/train/LLaMA-Factory-main/src/llamafactory/train/tuner.py", line 110, in run_exp
[rank11]:     _training_function(config={"args": args, "callbacks": callbacks})
[rank11]:   File "/data1/train/LLaMA-Factory-main/src/llamafactory/train/tuner.py", line 72, in _training_function
[rank11]:     run_sft(model_args, data_args, training_args, finetuning_args, generating_args, callbacks)
[rank11]:   File "/data1/train/LLaMA-Factory-main/src/llamafactory/train/sft/workflow.py", line 51, in run_sft
[rank11]:     dataset_module = get_dataset(template, model_args, data_args, training_args, stage="sft", **tokenizer_module)
[rank11]:   File "/data1/train/LLaMA-Factory-main/src/llamafactory/data/loader.py", line 315, in get_dataset
[rank11]:     dataset = _get_preprocessed_dataset(
[rank11]:   File "/data1/train/LLaMA-Factory-main/src/llamafactory/data/loader.py", line 256, in _get_preprocessed_dataset
[rank11]:     dataset = dataset.map(
[rank11]:   File "/data1/tools/anaconda3/envs/train/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 560, in wrapper
[rank11]:     out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
[rank11]:   File "/data1/tools/anaconda3/envs/train/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 3165, in map
[rank11]:     for rank, done, content in iflatmap_unordered(
[rank11]:   File "/data1/tools/anaconda3/envs/train/lib/python3.10/site-packages/datasets/utils/py_utils.py", line 718, in iflatmap_unordered
[rank11]:     [async_result.get(timeout=0.05) for async_result in async_results]
[rank11]:   File "/data1/tools/anaconda3/envs/train/lib/python3.10/site-packages/datasets/utils/py_utils.py", line 718, in <listcomp>
[rank11]:     [async_result.get(timeout=0.05) for async_result in async_results]
[rank11]:   File "/data1/tools/anaconda3/envs/train/lib/python3.10/site-packages/multiprocess/pool.py", line 774, in get
[rank11]:     raise self._value
[rank11]: ValueError: The number of images does not match the number of <image> tokens in [{'content': '我想知道帝国大厦的开放时间，以及游客需要经过哪些步骤才能参观？\n<image>', 'role': 'user'}, {'content': '帝国大厦位于美国纽约市，它的开放时间根据季节会有所不同。通常，在夏季（约6月至8月），帝国大厦的开放时间从早上8:00至晚上9:00，而在冬季（约11月至次年2月），开放时间可能会缩短到早上8:00至晚上7:00。为了参观，游客需要采取以下步骤：\n\n1. **提前在线购票**：因为帝国大厦的参观人数较多，尤其是高峰期，建议提前通过官方网站或指定线上平台购票，以避免现场长时间排队等候。在线购票通常会提供跳过部分排队环节或快速通道的选项。\n\n2. **选择合适的参观时间**：尽量避开人多的旺季或周五、周六这样的旅游高峰日，选择工作日的早中时段可能会人流较少，等待时间较短。\n\n3. **到达大厦**：提前抵达大厦进行安全检查和进行一般参观的导览区。帝国大厦入口位置的具体信息可以在官网进行查询，通常建议由第34或35街与布莱恩特公园（Bryant Park）之间的第五大道（5th Avenue）进入。\n\n4. **通过安检**：参观者需要通过安全检查。为了减少检查时间，避免随身携带大量的液体和非必要的电子设备。\n\n5. **向上层观景台**：通过电梯，到达帝国大厦的观景台。帝国大厦设有两个观景台，分别是位于第86层的室内观景台和位于第102层的室外观景台（需额外付费）。请务必根据您的需求和天气状况选择合适的观景台。\n\n6. **享受景点**：参观完观景台后，您可以选择在大楼内探索其他区域，如礼品店、餐厅等，根据个人兴趣进行自由活动。\n\n请记得随时查看帝国大厦官方公布的最新信息，包括开放时间和可能的临时变更，以确保您的旅行计划顺利进行。', 'role': 'assistant'}].
[rank12]: multiprocess.pool.RemoteTraceback: 
[rank12]: """
[rank12]: Traceback (most recent call last):
[rank12]:   File "/data1/tools/anaconda3/envs/train/lib/python3.10/site-packages/multiprocess/pool.py", line 125, in worker
[rank12]:     result = (True, func(*args, **kwds))
[rank12]:   File "/data1/tools/anaconda3/envs/train/lib/python3.10/site-packages/datasets/utils/py_utils.py", line 678, in _write_generator_to_queue
[rank12]:     for i, result in enumerate(func(**kwargs)):
[rank12]:   File "/data1/tools/anaconda3/envs/train/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 3476, in _map_single
[rank12]:     batch = apply_function_on_filtered_inputs(
[rank12]:   File "/data1/tools/anaconda3/envs/train/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 3338, in apply_function_on_filtered_inputs
[rank12]:     processed_inputs = function(*fn_args, *additional_args, **fn_kwargs)
[rank12]:   File "/data1/train/LLaMA-Factory-main/src/llamafactory/data/processor/supervised.py", line 99, in preprocess_dataset
[rank12]:     input_ids, labels = self._encode_data_example(
[rank12]:   File "/data1/train/LLaMA-Factory-main/src/llamafactory/data/processor/supervised.py", line 43, in _encode_data_example
[rank12]:     messages = self.template.mm_plugin.process_messages(prompt + response, images, videos, audios, self.processor)
[rank12]:   File "/data1/train/LLaMA-Factory-main/src/llamafactory/data/mm_plugin.py", line 396, in process_messages
[rank12]:     self._validate_messages(messages, images, videos, audios)
[rank12]:   File "/data1/train/LLaMA-Factory-main/src/llamafactory/data/mm_plugin.py", line 196, in _validate_messages
[rank12]:     raise ValueError(
[rank12]: ValueError: The number of images does not match the number of <image> tokens in [{'content': '我想知道帝国大厦的开放时间，以及游客需要经过哪些步骤才能参观？\n<image>', 'role': 'user'}, {'content': '帝国大厦位于美国纽约市，它的开放时间根据季节会有所不同。通常，在夏季（约6月至8月），帝国大厦的开放时间从早上8:00至晚上9:00，而在冬季（约11月至次年2月），开放时间可能会缩短到早上8:00至晚上7:00。为了参观，游客需要采取以下步骤：\n\n1. **提前在线购票**：因为帝国大厦的参观人数较多，尤其是高峰期，建议提前通过官方网站或指定线上平台购票，以避免现场长时间排队等候。在线购票通常会提供跳过部分排队环节或快速通道的选项。\n\n2. **选择合适的参观时间**：尽量避开人多的旺季或周五、周六这样的旅游高峰日，选择工作日的早中时段可能会人流较少，等待时间较短。\n\n3. **到达大厦**：提前抵达大厦进行安全检查和进行一般参观的导览区。帝国大厦入口位置的具体信息可以在官网进行查询，通常建议由第34或35街与布莱恩特公园（Bryant Park）之间的第五大道（5th Avenue）进入。\n\n4. **通过安检**：参观者需要通过安全检查。为了减少检查时间，避免随身携带大量的液体和非必要的电子设备。\n\n5. **向上层观景台**：通过电梯，到达帝国大厦的观景台。帝国大厦设有两个观景台，分别是位于第86层的室内观景台和位于第102层的室外观景台（需额外付费）。请务必根据您的需求和天气状况选择合适的观景台。\n\n6. **享受景点**：参观完观景台后，您可以选择在大楼内探索其他区域，如礼品店、餐厅等，根据个人兴趣进行自由活动。\n\n请记得随时查看帝国大厦官方公布的最新信息，包括开放时间和可能的临时变更，以确保您的旅行计划顺利进行。', 'role': 'assistant'}].
[rank12]: """

[rank12]: The above exception was the direct cause of the following exception:
```
=============使用的环境信息如下：========
accelerate               1.2.1
aiofiles                 23.2.1
aiohappyeyeballs         2.4.6
aiohttp                  3.11.12
aiosignal                1.3.2
annotated-types          0.7.0
antlr4-python3-runtime   4.9.3
anyio                    4.8.0
async-timeout            5.0.1
attrs                    25.1.0
audioread                3.0.1
auto_gptq                0.7.1
av                       14.1.0
certifi                  2025.1.31
cffi                     1.17.1
charset-normalizer       3.4.1
click                    8.1.8
contourpy                1.3.1
cos-python-sdk-v5        1.9.34
crcmod                   1.7
cycler                   0.12.1
datasets                 3.2.0
decorator                5.1.1
deepspeed                0.16.4
dill                     0.3.8
docstring_parser         0.16
einops                   0.8.1
exceptiongroup           1.2.2
fastapi                  0.115.8
ffmpy                    0.5.0
filelock                 3.17.0
fire                     0.7.0
fonttools                4.56.0
frozenlist               1.5.0
fsspec                   2024.9.0
gekko                    1.3.0
gradio                   5.12.0
gradio_client            1.5.4
h11                      0.14.0
hjson                    3.1.0
httpcore                 1.0.7
httpx                    0.28.1
huggingface-hub          0.30.2
idna                     3.10
jieba                    0.42.1
Jinja2                   3.1.5
joblib                   1.4.2
kiwisolver               1.4.8
lazy_loader              0.4
librosa                  0.10.2.post1
llamafactory             0.9.3.dev0   /data1/train/LLaMA-Factory-main
llvmlite                 0.44.0
markdown-it-py           3.0.0
MarkupSafe               2.1.5
matplotlib               3.10.0
mdurl                    0.1.2
mpmath                   1.3.0
msgpack                  1.1.0
multidict                6.1.0
multiprocess             0.70.16
networkx                 3.4.2
ninja                    1.11.1.3
nltk                     3.9.1
numba                    0.61.0
numpy                    1.26.4
nvidia-cublas-cu12       12.4.5.8
nvidia-cuda-cupti-cu12   12.4.127
nvidia-cuda-nvrtc-cu12   12.4.127
nvidia-cuda-runtime-cu12 12.4.127
nvidia-cudnn-cu12        9.1.0.70
nvidia-cufft-cu12        11.2.1.3
nvidia-curand-cu12       10.3.5.147
nvidia-cusolver-cu12     11.6.1.9
nvidia-cusparse-cu12     12.3.1.170
nvidia-cusparselt-cu12   0.6.2
nvidia-ml-py             12.570.86
nvidia-nccl-cu12         2.21.5
nvidia-nvjitlink-cu12    12.4.127
nvidia-nvtx-cu12         12.4.127
omegaconf                2.3.0
optimum                  1.24.0
orjson                   3.10.15
packaging                24.2
pandas                   2.2.3
peewee                   3.17.9
peft                     0.15.1
pillow                   11.1.0
pip                      25.0
platformdirs             4.3.6
pooch                    1.8.2
propcache                0.2.1
protobuf                 5.29.3
psutil                   7.0.0
py-cpuinfo               9.0.0
pyarrow                  19.0.1
pycparser                2.22
pycryptodome             3.21.0
pydantic                 2.10.6
pydantic_core            2.27.2
pydub                    0.25.1
Pygments                 2.19.1
pynvml                   12.0.0
pyparsing                3.2.1
python-dateutil          2.9.0.post0
python-multipart         0.0.20
pytz                     2025.1
PyYAML                   6.0.2
regex                    2024.11.6
requests                 2.32.3
rich                     13.9.4
rouge                    1.0.1
rouge-chinese            1.0.3
ruff                     0.9.6
safehttpx                0.1.6
safetensors              0.5.2
scikit-learn             1.6.1
scipy                    1.15.2
semantic-version         2.10.0
sentencepiece            0.2.0
setuptools               75.8.0
shellingham              1.5.4
shtab                    1.7.1
six                      1.17.0
sniffio                  1.3.1
soundfile                0.13.1
soxr                     0.5.0.post1
sse-starlette            2.2.1
starlette                0.45.3
swanboard                0.1.7b1
swankit                  0.1.3
swanlab                  0.4.8
sympy                    1.13.1
termcolor                2.5.0
threadpoolctl            3.5.0
tiktoken                 0.9.0
tokenizers               0.21.0
tomlkit                  0.13.2
torch                    2.6.0
tqdm                     4.67.1
transformers             4.50.0
triton                   3.2.0
trl                      0.9.6
typer                    0.15.1
typing_extensions        4.12.2
tyro                     0.8.14
tzdata                   2025.1
ujson                    5.10.0
urllib3                  2.3.0
uvicorn                  0.34.0
websockets               14.2
wheel                    0.45.1
xmltodict                0.14.2
xxhash                   3.5.0
yarl                     1.18.3
=============数据集示例：========
{"instruction":"","input":"","output":""}

### Reproduction

```text
Put your message here.
```


### Others

_No response_

Labels: solved
State: closed