TypeError: LlavaForConditionalGeneration.forward() got an unexpected keyword argument 'default_system'
Author: sysu19351160
Pull request: no
Labels: name: bug

name: VLM

### System Info

I can use llama factory training normally, but encounter the following problems during inference with order: FORCE_TORCHRUN=1 CUDA_VISIBLE_DEVICES=0,1 llamafactory-cli train eval_llava_lora_sft.yaml
system info:
[rank0]: Traceback (most recent call last):
[rank0]:   File "/data/llamafactory/src/llamafactory/launcher.py", line 23, in <module>
[rank0]:     launch()
[rank0]:   File "/data/llamafactory/src/llamafactory/launcher.py", line 19, in launch
[rank0]:     run_exp()
[rank0]:   File "/data/llamafactory/src/llamafactory/train/tuner.py", line 50, in run_exp
[rank0]:     run_sft(model_args, data_args, training_args, finetuning_args, generating_args, callbacks)
[rank0]:   File "/data/llamafactory/src/llamafactory/train/sft/workflow.py", line 127, in run_sft
[rank0]:     predict_results = trainer.predict(dataset_module["eval_dataset"], metric_key_prefix="predict", **gen_kwargs)
[rank0]:   File "/home/amax/.conda/envs/llamafactory/lib/python3.12/site-packages/transformers/trainer_seq2seq.py", line 259, in predict
[rank0]:     return super().predict(test_dataset, ignore_keys=ignore_keys, metric_key_prefix=metric_key_prefix)   
[rank0]:   File "/home/amax/.conda/envs/llamafactory/lib/python3.12/site-packages/transformers/trainer.py", line 4042, in predict
[rank0]:     output = eval_loop(
[rank0]:              ^^^^^^^^^^
[rank0]:   File "/home/amax/.conda/envs/llamafactory/lib/python3.12/site-packages/transformers/trainer.py", line 4158, in evaluation_loop
[rank0]:     losses, logits, labels = self.prediction_step(model, inputs, prediction_loss_only, ignore_keys=ignore_keys)
[rank0]:   File "/data/llamafactory/src/llamafactory/train/sft/trainer.py", line 122, in prediction_step
[rank0]:     loss, generated_tokens, _ = super().prediction_step(  # ignore the returned labels (may be truncated)
[rank0]:   File "/home/amax/.conda/envs/llamafactory/lib/python3.12/site-packages/transformers/trainer_seq2seq.py", line 331, in prediction_step
[rank0]:     generated_tokens = self.model.generate(**generation_inputs, **gen_kwargs)
[rank0]:   File "/home/amax/.conda/envs/llamafactory/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
[rank0]:     return func(*args, **kwargs)
[rank0]:   File "/home/amax/.conda/envs/llamafactory/lib/python3.12/site-packages/transformers/generation/utils.py", line 2215, in generate
[rank0]:     result = self._sample(
[rank0]:   File "/home/amax/.conda/envs/llamafactory/lib/python3.12/site-packages/transformers/generation/utils.py", line 3206, in _sample
[rank0]:     outputs = self(**model_inputs, return_dict=True)
[rank0]:   File "/home/amax/.conda/envs/llamafactory/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
[rank0]:     return self._call_impl(*args, **kwargs)
[rank0]:   File "/home/amax/.conda/envs/llamafactory/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
[rank0]:     return forward_call(*args, **kwargs)
[rank0]: TypeError: LlavaForConditionalGeneration.forward() got an unexpected keyword argument 'default_system'
[rank0]:[W113 18:16:18.306520029 ProcessGroupNCCL.cpp:1250] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present,  but this warning has only been added since PyTorch 2.4 (function operator())
W0113 18:16:19.816000 226009 site-packages/torch/distributed/elastic/multiprocessing/api.py:897] Sending process 226042 closing signal SIGTERM
E0113 18:16:19.850000 226009 site-packages/torch/distributed/elastic/multiprocessing/api.py:869] failed (exitcode: 1) local_rank: 1 (pid: 226043) of binary: /home/amax/.conda/envs/llamafactory/bin/python
Traceback (most recent call last):
  File "/home/amax/.conda/envs/llamafactory/bin/torchrun", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/amax/.conda/envs/llamafactory/lib/python3.12/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 355, in wrapper
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "/home/amax/.conda/envs/llamafactory/lib/python3.12/site-packages/torch/distributed/run.py", line 919, in main
    run(args)
  File "/home/amax/.conda/envs/llamafactory/lib/python3.12/site-packages/torch/distributed/run.py", line 910, in run
    elastic_launch(
  File "/home/amax/.conda/envs/llamafactory/lib/python3.12/site-packages/torch/distributed/launcher/api.py", line 138, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/amax/.conda/envs/llamafactory/lib/python3.12/site-packages/torch/distributed/launcher/api.py", line 269, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
============================================================
/data/llamafactory/src/llamafactory/launcher.py FAILED
------------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2025-01-13_18:16:19
  host      : amax
  rank      : 1 (local_rank: 1)
  exitcode  : 1 (pid: 226043)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

### model
model_name_or_path: model/llava-1.5-7b-hf
adapter_name_or_path: saves/llava-1.5-7b/lora/sft

### method
stage: sft
do_predict: true
finetuning_type: lora

### dataset
eval_dataset: QaTa-COV19-v2
template: llava
cutoff_len: 2048
max_samples: 50
overwrite_cache: true
preprocessing_num_workers: 16

### output
output_dir: saves/llava-1.5-7b-eval
overwrite_output_dir: true

### eval
per_device_eval_batch_size: 1
predict_with_generate: true
ddp_timeout: 180000000


### Expected behavior

successfully running the inference process

Comments:
@sysu19351160 hey!

We usually run generation with the set of kwargs provided in the inputs and from what I see llama-factory is passing a `default_system` as key in `gen_kwargs` when calling `trainer.predict()`. I think the best solution here is to open an issue on llama-factory and remove any kwargs not used directly by `generate()` call

https://github.com/hiyouga/LLaMA-Factory/blob/e3e2c8c689c54ebb2af264de808502e5a8ba0f2b/src/llamafactory/train/sft/workflow.py#L126

# I HOPE THIS RESOLVED YOUR ISSUE

> [@sysu19351160](https://github.com/sysu19351160) hey!
> 
> We usually run generation with the set of kwargs provided in the inputs and from what I see llama-factory is passing a `default_system` as key in `gen_kwargs` when calling `trainer.predict()`. I think the best solution here is to open an issue on llama-factory and remove any kwargs not used directly by `generate()` call
> 
> https://github.com/hiyouga/LLaMA-Factory/blob/e3e2c8c689c54ebb2af264de808502e5a8ba0f2b/src/llamafactory/train/sft/workflow.py#L126

I think your analysis is rational, but I am not sure about how to remove the useless kwargs from gen_kwargs. I would appreciate it if you could help me with more details.

@sysu19351160 I'd recommend opening an issue in llama-factory for that question because the extra kwarg was added by their Argument Parser 🤗 

> [@sysu19351160](https://github.com/sysu19351160) I'd recommend opening an issue in llama-factory for that question because the extra kwarg was added by their Argument Parser 🤗

OK, thanks