GPU Error unspecified launch failure

Sourabh_Yadav: <p>I have 2x 32GB - V100 and when i am training/fine tuning a llm i notice that at some point my gpu memory usage in nvidia-smi was 25 Gb and 18 Gb and after some time it shows 32 Gb and 20 Gb.</p>
<p>So like will it keep on increasing and how to fix this if i am going to train for longer durations?</p>
<h2><a name="p-465163-i-have-loaded-3b-size-model-in-8-bit-and-i-got-this-error-after-72-mins-of-training-the-error-is-this-1" class="anchor" href="#p-465163-i-have-loaded-3b-size-model-in-8-bit-and-i-got-this-error-after-72-mins-of-training-the-error-is-this-1"></a>I have loaded 3B size model in 8-bit and i got this error after 72 mins of training.<br>
The error is this.</h2>
<p>RuntimeError                              Traceback (most recent call last)<br>
Cell In[5], line 38<br>
1 training_args = GRPOConfig(<br>
2     learning_rate=5e-6,<br>
3     adam_beta1=0.9,<br>
(…)<br>
22     output_dir=“outputs”,<br>
23 )<br>
25 trainer = GRPOTrainer(<br>
26     model = model,<br>
27     processing_class = tokenizer,<br>
(…)<br>
36     train_dataset = dataset,<br>
37 )<br>
—&gt; 38 trainer.train()</p>
<p>File c:\ProgramData\anaconda3\envs\llmrl\Lib\site-packages\transformers\trainer.py:2171, in Trainer.train(self, resume_from_checkpoint, trial, ignore_keys_for_eval, **kwargs)<br>
2169         hf_hub_utils.enable_progress_bars()<br>
2170 else:<br>
 → 2171     return inner_training_loop(<br>
2172         args=args,<br>
2173         resume_from_checkpoint=resume_from_checkpoint,<br>
2174         trial=trial,<br>
2175         ignore_keys_for_eval=ignore_keys_for_eval,<br>
2176     )</p>
<p>File c:\ProgramData\anaconda3\envs\llmrl\Lib\site-packages\transformers\trainer.py:2531, in Trainer._inner_training_loop(self, batch_size, args, resume_from_checkpoint, trial, ignore_keys_for_eval)<br>
2524 context = (<br>
2525     functools.partial(self.accelerator.no_sync, model=model)<br>
2526     if i != len(batch_samples) - 1<br>
2527     and self.accelerator.distributed_type != DistributedType.DEEPSPEED<br>
2528     else contextlib.nullcontext<br>
2529 )<br>
2530 with context():<br>
 → 2531     tr_loss_step = self.training_step(model, inputs, num_items_in_batch)<br>
2533 if (<br>
2534     args.logging_nan_inf_filter<br>
2535     and not is_torch_xla_available()<br>
2536     and (torch.isnan(tr_loss_step) or torch.isinf(tr_loss_step))<br>
2537 ):<br>
2538     # if loss is nan or inf simply add the average of previous logged losses<br>
2539     tr_loss = tr_loss + tr_loss / (1 + self.state.global_step - self._globalstep_last_logged)</p>
<p>File c:\ProgramData\anaconda3\envs\llmrl\Lib\site-packages\transformers\trainer.py:3669, in Trainer.training_step(self, model, inputs, num_items_in_batch)<br>
3666 if hasattr(self.optimizer, “train”) and callable(self.optimizer.train):<br>
3667     self.optimizer.train()<br>
 → 3669 inputs = self._prepare_inputs(inputs)<br>
3670 if is_sagemaker_mp_enabled():<br>
3671     loss_mb = smp_forward_backward(model, inputs, self.args.gradient_accumulation_steps)</p>
<p>File c:\ProgramData\anaconda3\envs\llmrl\Lib\site-packages\trl\trainer\grpo_trainer.py:477, in GRPOTrainer._prepare_inputs(self, inputs)<br>
475 with torch.inference_mode():<br>
476     if self.ref_model is not None:<br>
 → 477         ref_per_token_logps = self._get_per_token_logps(<br>
478             self.ref_model, prompt_completion_ids, attention_mask, logits_to_keep<br>
479         )<br>
480     else:<br>
481         with self.accelerator.unwrap_model(self.model).disable_adapter():</p>
<p>File c:\ProgramData\anaconda3\envs\llmrl\Lib\site-packages\trl\trainer\grpo_trainer.py:380, in GRPOTrainer._get_per_token_logps(self, model, input_ids, attention_mask, logits_to_keep)<br>
378 def _get_per_token_logps(self, model, input_ids, attention_mask, logits_to_keep):<br>
379     # We add 1 to <code>logits_to_keep</code> because the last logits of the sequence is later excluded<br>
 → 380     logits = model(<br>
381         input_ids=input_ids, attention_mask=attention_mask, logits_to_keep=logits_to_keep + 1<br>
382     ).logits  # (B, L, V)<br>
383     logits = logits[:, :-1, :]  # (B, L-1, V), exclude the last logit: it corresponds to the next token pred<br>
385     input_ids = input_ids[:, -logits_to_keep:]</p>
<p>File c:\ProgramData\anaconda3\envs\llmrl\Lib\site-packages\torch\nn\modules\module.py:1739, in Module._wrapped_call_impl(self, *args, **kwargs)<br>
1737     return self._compiled_call_impl(*args, **kwargs)  # type: ignore[misc]<br>
1738 else:<br>
 → 1739     return self._call_impl(*args, **kwargs)</p>
<p>File c:\ProgramData\anaconda3\envs\llmrl\Lib\site-packages\torch\nn\modules\module.py:1750, in Module._call_impl(self, *args, **kwargs)<br>
1745 # If we don’t have any hooks, we want to skip the rest of the logic in<br>
1746 # this function, and just call forward.<br>
1747 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks<br>
1748         or _global_backward_pre_hooks or _global_backward_hooks<br>
1749         or _global_forward_hooks or _global_forward_pre_hooks):<br>
 → 1750     return forward_call(*args, **kwargs)<br>
1752 result = None<br>
1753 called_always_called_hooks = set()</p>
<p>File c:\ProgramData\anaconda3\envs\llmrl\Lib\site-packages\accelerate\utils\operations.py:819, in convert_outputs_to_fp32..forward(*args, **kwargs)<br>
818 def forward(*args, **kwargs):<br>
 → 819     return model_forward(*args, **kwargs)</p>
<p>File c:\ProgramData\anaconda3\envs\llmrl\Lib\site-packages\accelerate\utils\operations.py:807, in ConvertOutputsToFp32.<strong>call</strong>(self, *args, **kwargs)<br>
806 def <strong>call</strong>(self, *args, **kwargs):<br>
 → 807     return convert_to_fp32(self.model_forward(*args, **kwargs))</p>
<p>File c:\ProgramData\anaconda3\envs\llmrl\Lib\site-packages\accelerate\utils\operations.py:786, in convert_to_fp32(tensor)<br>
780 def _is_fp16_bf16_tensor(tensor):<br>
781     return (is_torch_tensor(tensor) or hasattr(tensor, “dtype”)) and tensor.dtype in (<br>
782         torch.float16,<br>
783         torch.bfloat16,<br>
784     )<br>
 → 786 return recursively_apply(_convert_to_fp32, tensor, test_type=_is_fp16_bf16_tensor)</p>
<p>File c:\ProgramData\anaconda3\envs\llmrl\Lib\site-packages\accelerate\utils\operations.py:118, in recursively_apply(func, data, test_type, error_on_other_type, *args, **kwargs)<br>
107     return honor_type(<br>
108         data,<br>
109         (<br>
(…)<br>
114         ),<br>
115     )<br>
116 elif isinstance(data, Mapping):<br>
117     return type(data)(<br>
 → 118         {<br>
119             k: recursively_apply(<br>
120                 func, v, *args, test_type=test_type, error_on_other_type=error_on_other_type, **kwargs<br>
121             )<br>
122             for k, v in data.items()<br>
123         }<br>
124     )<br>
125 elif test_type(data):<br>
126     return func(data, *args, **kwargs)</p>
<p>File c:\ProgramData\anaconda3\envs\llmrl\Lib\site-packages\accelerate\utils\operations.py:119, in (.0)<br>
107     return honor_type(<br>
108         data,<br>
109         (<br>
(…)<br>
114         ),<br>
115     )<br>
116 elif isinstance(data, Mapping):<br>
117     return type(data)(<br>
118         {<br>
 → 119             k: recursively_apply(<br>
120                 func, v, *args, test_type=test_type, error_on_other_type=error_on_other_type, **kwargs<br>
121             )<br>
122             for k, v in data.items()<br>
123         }<br>
124     )<br>
125 elif test_type(data):<br>
126     return func(data, *args, **kwargs)</p>
<p>File c:\ProgramData\anaconda3\envs\llmrl\Lib\site-packages\accelerate\utils\operations.py:126, in recursively_apply(func, data, test_type, error_on_other_type, *args, **kwargs)<br>
117     return type(data)(<br>
118         {<br>
119             k: recursively_apply(<br>
(…)<br>
123         }<br>
124     )<br>
125 elif test_type(data):<br>
 → 126     return func(data, *args, **kwargs)<br>
127 elif error_on_other_type:<br>
128     raise TypeError(<br>
129         f"Unsupported types ({type(data)}) passed to <code>{func.__name__}</code>. Only nested list/tuple/dicts of "<br>
130         f"objects that are valid for <code>{test_type.__name__}</code> should be passed."<br>
131     )</p>
<p>File c:\ProgramData\anaconda3\envs\llmrl\Lib\site-packages\accelerate\utils\operations.py:778, in convert_to_fp32.._convert_to_fp32(tensor)<br>
777 def _convert_to_fp32(tensor):<br>
 → 778     return tensor.float()</p>
<p>RuntimeError: CUDA error: unspecified launch failure<br>
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.<br>
For debugging consider passing CUDA_LAUNCH_BLOCKING=1<br>
Compile with <code>TORCH_USE_CUDA_DSA</code> to enable device-side assertions.</p>
<p>My training arguments and settings are</p>
<p>training_args = GRPOConfig(<br>
learning_rate=5e-6,<br>
adam_beta1=0.9,<br>
adam_beta2=0.99,<br>
weight_decay=0.1,<br>
warmup_ratio=0.1,<br>
lr_scheduler_type=“cosine”,<br>
optim=“adamw_torch”,<br>
logging_steps=1,<br>
bf16=torch.cuda.is_bf16_supported(),<br>
fp16=not torch.cuda.is_bf16_supported(),<br>
per_device_train_batch_size=1,<br>
gradient_accumulation_steps=1,<br>
num_generations=10,<br>
max_prompt_length=512,<br>
max_completion_length=250,<br>
num_train_epochs=1,<br>
max_steps=1000,<br>
save_steps=500,<br>
max_grad_norm=0.1,<br>
report_to=“none”,<br>
output_dir=“outputs”,<br>
)</p>
<p>trainer = GRPOTrainer(<br>
model = model,<br>
processing_class = tokenizer,<br>
reward_funcs = [<br>
xmlcount_reward_func,<br>
soft_format_reward_func,<br>
strict_format_reward_func,<br>
int_reward_func,<br>
correctness_reward_func,<br>
],<br>
args = training_args,<br>
train_dataset = dataset,<br>
)<br>
trainer.train()</p>

Sourabh_Yadav: <p>man what is this AI generated suggestions.<br>
first of all model is loaded in 8 bit.<br>
secondly there is no overheating issues.</p>
<p>and where should i use this command. " <strong><code>CUDA_LAUNCH_BLOCKING=1</code></strong>"  ??</p>