akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
RuntimeError: CUDA error: device-side assert triggered in text generation

username: Dovud_Asadov
cooked: <p>I load the model for text generation task and my dataset only contain prompt column which is input to model, and if I want generate output with this code</p>
<pre><code class="lang-auto">def generate_responses(input_texts, max_tokens=2048):
    prompts = [f"&lt;|system|&gt;{SYSTEM_MESSAGE}\n&lt;user&gt;: {input_text}\n&lt;assistant&gt;:" for input_text in input_texts]
    inputs = tokenizer(prompts, return_tensors="pt", padding=True, truncation=True, max_length=max_tokens).to('cuda')
    with torch.inference_mode():  
        outputs = model.generate(
            **inputs,
            max_new_tokens=max_tokens,
            do_sample=False,
            pad_token_id=tokenizer.eos_token_id,  
            num_return_sequences=1
        )
    
    responses = [
        tokenizer.decode(output, skip_special_tokens=True).split("&lt;assistant&gt;:")[-1].strip()
        for output in outputs]
    return responses

def batch_generate_responses(df, batch_size=10, max_tokens=2048):
    all_responses = []
    data_loader = DataLoader(df['prompt'].tolist(), batch_size=batch_size)
    for batch_prompts in data_loader:
        batch_responses = generate_responses(batch_prompts, max_tokens=max_tokens)
        all_responses.extend(batch_responses)
    df['feedback'] = all_responses
    df.to_json('feedbacks.json', orient='records', indent=2)
batch_generate_responses(df)
</code></pre>
<p>after runnig this code this error showing up</p>
<pre><code class="lang-auto">RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
</code></pre>
<p>How can i solve this problem?</p>

username: ptrblck
cooked: <p>Rerun your code with blocking launches as described in the error message to narrow down which line of code fails. Often a wrong indexing operation raises this error e.g. if the input tensor contains indices out of the valid range.</p>
Collect FSDP gradients on CPU

username: nikonikolov
cooked: <p>I am trying to collect gradients of an FSDP module so I can plot them. I am using <code>FSDP.summon_full_params</code>. However, the documentation states that <code>with_grads=True</code> is only supported when using <code>offload_to_cpu=False</code>. This means that gradients and parameters end up being collected on the GPU, which requires extra GPU memory. For large models this easily causes CUDA OOM.</p>
<p>I tried collecting one parameter at a time, but due to the way torch reserves CUDA memory, I need to call <code>torch.cuda.empty_cache()</code> which destroys the computational graph and causes training slowdowns. In some cases, depending on parameter sizes, it still runs out of GPU memory even with <code>torch.cuda.empty_cache()</code> and <code>gc.collect()</code>.</p>
<p>How can I reliably collect gradients of an FSDP module on CPU without running out of CUDA memory and without destroying the computational graph?</p>