Pytorch Nightly with CUDA 12.8 on Nvidia 5070Ti: RuntimeError: CUDA error: too many resources requested for launch

Praveen_Benedict: <p>I recently set up a workstation with a Nvidia 5070. It has been setup with CUDA 12.8. When I try training my model (which runs well on my Nvidia 3090 with CUDA 12.2), it gives me the following error: RuntimeError: CUDA error: too many resources requested for launch</p>
<p>Here is the full error<br>
Traceback (most recent call last):---------------------------------------| 0.00% [0/1414 00:00&lt;?]<br>
File “/home/praveenbenedict/Projects/Authenta/authenta-generated/notebooks/modelling/Gen/train.py”, line 94, in <br>
trainer.train()<br>
File “/home/praveenbenedict/Projects/Authenta/authenta-generated/notebooks/modelling/Gen/trainer/trainer.py”, line 650, in train<br>
self.train_augmenter(epochs=self.config[‘training’][‘settings’][‘epochs’][‘adversarial’][‘augmenter’])<br>
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br>
File “/home/praveenbenedict/Projects/Authenta/authenta-generated/notebooks/modelling/Gen/trainer/trainer.py”, line 535, in train_augmenter<br>
loss.backward()<br>
File “/home/praveenbenedict/miniconda3/envs/authenta-generated/lib/python3.11/site-packages/torch/_tensor.py”, line 648, in backward<br>
torch.autograd.backward(<br>
File “/home/praveenbenedict/miniconda3/envs/authenta-generated/lib/python3.11/site-packages/torch/autograd/<strong>init</strong>.py”, line 353, in backward<br>
_engine_run_backward(<br>
File “/home/praveenbenedict/miniconda3/envs/authenta-generated/lib/python3.11/site-packages/torch/autograd/graph.py”, line 824, in _engine_run_backward<br>
return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass<br>
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br>
RuntimeError: CUDA error: too many resources requested for launch</p>
<p>I’ve installed the Pytorch Nightly version that supports CUDA 12.8. Is this a temporary issue with Pytorch, or is there anything else that I am missing?</p>

ptrblck: <p>Could you try to narrow down which kernel fails to launch via <code>cuda-gdb</code>?</p>

Praveen_Benedict: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a> Here is what I got when I ran with cuda-gdb:</p>
<p>(cuda-gdb) run train.py<br>
Starting program: /home/praveenbenedict/miniconda3/envs/authenta-generated-image-detection/bin/python train.py<br>
[Thread debugging using libthread_db enabled]<br>
Using host libthread_db library “/lib/x86_64-linux-gnu/libthread_db.so.1”.<br>
[New Thread 0x7ffff4fff6c0 (LWP 2712)]<br>
[New Thread 0x7ffff27fe6c0 (LWP 2713)]<br>
[New Thread 0x7fffefffd6c0 (LWP 2714)]<br>
[New Thread 0x7fffed7fc6c0 (LWP 2715)]<br>
[New Thread 0x7fffeaffb6c0 (LWP 2716)]<br>
[New Thread 0x7fffe87fa6c0 (LWP 2717)]<br>
[New Thread 0x7fffe5ff96c0 (LWP 2718)]<br>
[New Thread 0x7fffe57f86c0 (LWP 2719)]<br>
[New Thread 0x7fffe0ff76c0 (LWP 2720)]<br>
[New Thread 0x7fffde7f66c0 (LWP 2721)]<br>
[New Thread 0x7fffddff56c0 (LWP 2722)]<br>
[New Thread 0x7fffdb7f46c0 (LWP 2723)]<br>
[New Thread 0x7fffd8ff36c0 (LWP 2724)]<br>
[New Thread 0x7fffd47f26c0 (LWP 2725)]<br>
[New Thread 0x7fffd3ff16c0 (LWP 2726)]<br>
[New Thread 0x7fffd17f06c0 (LWP 2727)]<br>
[New Thread 0x7fffccfef6c0 (LWP 2728)]<br>
[New Thread 0x7fffcc7ee6c0 (LWP 2729)]<br>
[New Thread 0x7fffc7fed6c0 (LWP 2730)]<br>
[New Thread 0x7fffc57ec6c0 (LWP 2731)]<br>
[New Thread 0x7fffc2feb6c0 (LWP 2732)]<br>
[New Thread 0x7fffc07ea6c0 (LWP 2733)]<br>
[New Thread 0x7fffbdfe96c0 (LWP 2734)]<br>
[New Thread 0x7fffbd7e86c0 (LWP 2735)]<br>
[New Thread 0x7fffb8fe76c0 (LWP 2736)]<br>
[New Thread 0x7fffb67e66c0 (LWP 2737)]<br>
[New Thread 0x7fffb3fe56c0 (LWP 2738)]<br>
[New Thread 0x7ffe674996c0 (LWP 2739)]<br>
[New Thread 0x7ffe66c986c0 (LWP 2740)]<br>
[New Thread 0x7ffe664976c0 (LWP 2741)]<br>
[New Thread 0x7ffe65c966c0 (LWP 2742)]<br>
[New Thread 0x7ffe654956c0 (LWP 2743)]<br>
[New Thread 0x7ffe64c946c0 (LWP 2744)]<br>
[New Thread 0x7ffe644936c0 (LWP 2745)]<br>
[New Thread 0x7ffe63c926c0 (LWP 2746)]<br>
[New Thread 0x7ffe634916c0 (LWP 2747)]<br>
[New Thread 0x7ffe62c906c0 (LWP 2748)]<br>
[New Thread 0x7ffe6248f6c0 (LWP 2749)]<br>
[New Thread 0x7ffe61c8e6c0 (LWP 2750)]<br>
[New Thread 0x7ffe6148d6c0 (LWP 2751)]<br>
[New Thread 0x7ffe60c8c6c0 (LWP 2752)]<br>
[New Thread 0x7ffe6048b6c0 (LWP 2753)]<br>
[New Thread 0x7ffe5fc8a6c0 (LWP 2754)]<br>
[New Thread 0x7ffe5f4896c0 (LWP 2755)]<br>
[New Thread 0x7ffe5ec886c0 (LWP 2756)]<br>
[New Thread 0x7ffe5e4876c0 (LWP 2757)]<br>
[New Thread 0x7ffe5dc866c0 (LWP 2758)]<br>
[New Thread 0x7ffe5d4856c0 (LWP 2759)]<br>
[New Thread 0x7ffe5cc846c0 (LWP 2760)]<br>
[New Thread 0x7ffe5c4836c0 (LWP 2761)]<br>
[New Thread 0x7ffe5bc826c0 (LWP 2762)]<br>
[New Thread 0x7ffe5b4816c0 (LWP 2763)]<br>
[New Thread 0x7ffe5ac806c0 (LWP 2764)]<br>
[New Thread 0x7ffe5a47f6c0 (LWP 2765)]<br>
Number of Fake images; Train: 45223 Valid: 12691<br>
Number of Real images; Train: 95319 Valid: 10993<br>
Real Images – train – Train: 95319 – Fake: 45223<br>
Real Images – valid – Train: 10993 – Fake: 12691<br>
Number of Fake images; Train: 45223 Valid: 12691<br>
Number of Real images; Train: 95319 Valid: 10993<br>
[New Thread 0x7ffe51dff6c0 (LWP 2767)]<br>
[New Thread 0x7ffe50b6e6c0 (LWP 2768)]<br>
[Thread 0x7ffe5d4856c0 (LWP 2759) exited]<br>
[Thread 0x7ffe5b4816c0 (LWP 2763) exited]<br>
[Thread 0x7ffe5ac806c0 (LWP 2764) exited]<br>
[Thread 0x7ffe5a47f6c0 (LWP 2765) exited]<br>
[Thread 0x7ffe5bc826c0 (LWP 2762) exited]<br>
[Thread 0x7ffe5c4836c0 (LWP 2761) exited]<br>
[Thread 0x7ffe5cc846c0 (LWP 2760) exited]<br>
[Thread 0x7ffe5dc866c0 (LWP 2758) exited]<br>
[Thread 0x7ffe5e4876c0 (LWP 2757) exited]<br>
[Thread 0x7ffe5ec886c0 (LWP 2756) exited]<br>
[Thread 0x7ffe5f4896c0 (LWP 2755) exited]<br>
[Thread 0x7ffe5fc8a6c0 (LWP 2754) exited]<br>
[Thread 0x7ffe6048b6c0 (LWP 2753) exited]<br>
[Thread 0x7ffe60c8c6c0 (LWP 2752) exited]<br>
[Thread 0x7ffe6148d6c0 (LWP 2751) exited]<br>
[Thread 0x7ffe61c8e6c0 (LWP 2750) exited]<br>
[Thread 0x7ffe6248f6c0 (LWP 2749) exited]<br>
[Thread 0x7ffe62c906c0 (LWP 2748) exited]<br>
[Thread 0x7ffe634916c0 (LWP 2747) exited]<br>
[Thread 0x7ffe63c926c0 (LWP 2746) exited]<br>
[Thread 0x7ffe644936c0 (LWP 2745) exited]<br>
[Thread 0x7ffe64c946c0 (LWP 2744) exited]<br>
[Thread 0x7ffe654956c0 (LWP 2743) exited]<br>
[Thread 0x7ffe65c966c0 (LWP 2742) exited]<br>
[Thread 0x7ffe664976c0 (LWP 2741) exited]<br>
[Thread 0x7ffe66c986c0 (LWP 2740) exited]<br>
[Thread 0x7ffe674996c0 (LWP 2739) exited]<br>
[Thread 0x7ffff27fe6c0 (LWP 2713) exited]<br>
[Thread 0x7fffefffd6c0 (LWP 2714) exited]<br>
[Thread 0x7fffe57f86c0 (LWP 2719) exited]<br>
[Thread 0x7fffed7fc6c0 (LWP 2715) exited]<br>
[Thread 0x7fffeaffb6c0 (LWP 2716) exited]<br>
[Thread 0x7fffcc7ee6c0 (LWP 2729) exited]<br>
[Thread 0x7fffdb7f46c0 (LWP 2723) exited]<br>
[Thread 0x7fffddff56c0 (LWP 2722) exited]<br>
[Thread 0x7ffff4fff6c0 (LWP 2712) exited]<br>
[Thread 0x7fffd3ff16c0 (LWP 2726) exited]<br>
[Thread 0x7fffbd7e86c0 (LWP 2735) exited]<br>
[Thread 0x7fffd47f26c0 (LWP 2725) exited]<br>
[Thread 0x7fffb8fe76c0 (LWP 2736) exited]<br>
[Thread 0x7fffb67e66c0 (LWP 2737) exited]<br>
[Thread 0x7fffbdfe96c0 (LWP 2734) exited]<br>
[Thread 0x7fffb3fe56c0 (LWP 2738) exited]<br>
[Thread 0x7fffc2feb6c0 (LWP 2732) exited]<br>
[Thread 0x7fffc07ea6c0 (LWP 2733) exited]<br>
[Thread 0x7fffc57ec6c0 (LWP 2731) exited]<br>
[Thread 0x7fffc7fed6c0 (LWP 2730) exited]<br>
[Thread 0x7fffccfef6c0 (LWP 2728) exited]<br>
[Thread 0x7fffd17f06c0 (LWP 2727) exited]<br>
[Thread 0x7fffd8ff36c0 (LWP 2724) exited]<br>
[Thread 0x7fffde7f66c0 (LWP 2721) exited]<br>
[Thread 0x7fffe0ff76c0 (LWP 2720) exited]<br>
[Thread 0x7fffe5ff96c0 (LWP 2718) exited]<br>
[Thread 0x7fffe87fa6c0 (LWP 2717) exited]<br>
[Detaching after fork from child process 2769]<br>
[New Thread 0x7fffb3fe56c0 (LWP 2778)]<br>
[Thread 0x7fffb3fe56c0 (LWP 2778) exited]<br>
[New Thread 0x7fffb3fe56c0 (LWP 2779)]<br>
[New Thread 0x7fffb67e66c0 (LWP 2780)]<br>
[2025-04-15 10:59:32] [INFO] =&gt; Using checkpoint from run id: 13_04_2025_08_14_51 epoch id: 0<br>
[2025-04-15 10:59:32] [INFO] =&gt; Loading checkpoint<br>
[2025-04-15 10:59:32] [INFO]    =&gt; Current Teacher model metrics - Accuracy: {‘accuracy’: 0.9007581159202742, ‘real_accuracy’: 0.9065225290697675, ‘fake_accuracy’: 0.8949937027707808, ‘f1’: 0.894299447464175}<br>
[2025-04-15 10:59:32] [INFO]    =&gt; Best Teacher model metrics- Accuracy: {‘accuracy’: 0.9007581159202742, ‘real_accuracy’: 0.9065225290697675, ‘fake_accuracy’: 0.8949937027707808, ‘f1’: 0.894299447464175}<br>
[2025-04-15 10:59:32] [INFO]    =&gt; Best Pipeline model metrics- Accuracy: {‘accuracy’: tensor(0.8988), ‘real_accuracy’: tensor(0.9122), ‘fake_accuracy’: tensor(0.8853), ‘f1’: 0.8919182083739046}<br>
[2025-04-15 10:59:32] [INFO]    =&gt; Loaded weights for Teacher Network from best checkpoint<br>
[2025-04-15 10:59:32] [INFO]    =&gt; Loaded the weights for student<br>
[2025-04-15 10:59:32] [INFO]    =&gt; Loaded the weights for augmenter<br>
[2025-04-15 10:59:32] [INFO]    =&gt; Loaded weights for Classifier Network from best checkpoint<br>
[2025-04-15 10:59:32] [INFO] =&gt; Using weighted loss for teacher with a weight of 0.7372192144393921<br>
[2025-04-15 10:59:32] [INFO] =&gt; Using weighted loss for Classifier with a weight of 0.7372192144393921<br>
[2025-04-15 10:59:32] [INFO] =&gt; Adversarial Training…<br>
[2025-04-15 10:59:32] [INFO]    =&gt; Freezing CLIPFeatureExtractor<br>
[2025-04-15 10:59:32] [INFO]    =&gt; Freezing TeacherNetworkTransformer<br>
[2025-04-15 10:59:32] [INFO]    =&gt; Freezing StudentNetworkTransformer<br>
[2025-04-15 10:59:32] [INFO]    =&gt; Unfreezing FeatureAugmenterTransformer<br>
[Detaching after fork from child process 2781]---------------| 0.00% [0/1414 00:00&lt;?]<br>
[Detaching after fork from child process 2782]<br>
[Detaching after fork from child process 2783]<br>
[Detaching after fork from child process 2784]<br>
[New Thread 0x7fffb8fe76c0 (LWP 2785)]<br>
[New Thread 0x7fffbd7e86c0 (LWP 2786)]<br>
[New Thread 0x7fffe87fa6c0 (LWP 2787)]<br>
[New Thread 0x7fffe0ff76c0 (LWP 2788)]<br>
[New Thread 0x7fffde7f66c0 (LWP 2789)]<br>
[New Thread 0x7fffddff56c0 (LWP 2790)]<br>
[New Thread 0x7fffdb7f46c0 (LWP 2791)]<br>
[New Thread 0x7fffd8ff36c0 (LWP 2792)]<br>
[New Thread 0x7fffd47f26c0 (LWP 2793)]<br>
[New Thread 0x7fffd3ff16c0 (LWP 2794)]<br>
[New Thread 0x7fffd17f06c0 (LWP 2795)]<br>
[New Thread 0x7fffccfef6c0 (LWP 2796)]<br>
[New Thread 0x7fffcc7ee6c0 (LWP 2797)]<br>
[New Thread 0x7fffc7fed6c0 (LWP 2798)]<br>
[New Thread 0x7fffc57ec6c0 (LWP 2799)]<br>
[New Thread 0x7fffc2feb6c0 (LWP 2800)]<br>
[New Thread 0x7fffc07ea6c0 (LWP 2801)]<br>
[Thread 0x7fffb3fe56c0 (LWP 2779) exited]<br>
[New Thread 0x7fffb3fe56c0 (LWP 2810)]<br>
warning: Cuda API error detected: cudaLaunchKernel returned (0x2bd)</p>
<p>warning: Cuda API error detected: cudaGetLastError returned (0x2bd)</p>
<p>[Thread 0x7fffc2feb6c0 (LWP 2800) exited]<br>
[Thread 0x7fffc57ec6c0 (LWP 2799) exited]<br>
[Thread 0x7fffc07ea6c0 (LWP 2801) exited]<br>
[Thread 0x7fffc7fed6c0 (LWP 2798) exited]<br>
Traceback (most recent call last):<br>
File “/home/praveenbenedict/Projects/Authenta/authenta-generated-image-detection/notebooks/modelling/GenDet/train.py”, line 94, in <br>
trainer.train()<br>
File “/home/praveenbenedict/Projects/Authenta/authenta-generated-image-detection/notebooks/modelling/GenDet/trainer/trainer.py”, line 650, in train<br>
self.train_augmenter(epochs=self.config[‘training’][‘settings’][‘epochs’][‘adversarial’][‘augmenter’])<br>
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br>
File “/home/praveenbenedict/Projects/Authenta/authenta-generated-image-detection/notebooks/modelling/GenDet/trainer/trainer.py”, line 535, in train_augmenter<br>
loss.backward()<br>
File “/home/praveenbenedict/miniconda3/envs/authenta-generated-image-detection/lib/python3.11/site-packages/torch/_tensor.py”, line 648, in backward<br>
torch.autograd.backward(<br>
File “/home/praveenbenedict/miniconda3/envs/authenta-generated-image-detection/lib/python3.11/site-packages/torch/autograd/<em>init</em>.py”, line 353, in backward<br>
_engine_run_backward(<br>
File “/home/praveenbenedict/miniconda3/envs/authenta-generated-image-detection/lib/python3.11/site-packages/torch/autograd/graph.py”, line 824, in _engine_run_backward<br>
return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass<br>
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br>
RuntimeError: CUDA error: too many resources requested for launch<br>
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.<br>
For debugging consider passing CUDA_LAUNCH_BLOCKING=1<br>
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.</p>
<p>[Thread 0x7fffb3fe56c0 (LWP 2810) exited]<br>
[Thread 0x7ffe50b6e6c0 (LWP 2768) exited]<br>
[Thread 0x7fffccfef6c0 (LWP 2796) exited]<br>
[Thread 0x7fffd17f06c0 (LWP 2795) exited]<br>
[Thread 0x7fffd3ff16c0 (LWP 2794) exited]<br>
[Thread 0x7fffd47f26c0 (LWP 2793) exited]<br>
[Thread 0x7fffd8ff36c0 (LWP 2792) exited]<br>
[Thread 0x7fffdb7f46c0 (LWP 2791) exited]<br>
[Thread 0x7fffddff56c0 (LWP 2790) exited]<br>
[Thread 0x7fffde7f66c0 (LWP 2789) exited]<br>
[Thread 0x7fffe0ff76c0 (LWP 2788) exited]<br>
[Thread 0x7fffe87fa6c0 (LWP 2787) exited]<br>
[Thread 0x7fffbd7e86c0 (LWP 2786) exited]<br>
[Thread 0x7fffb8fe76c0 (LWP 2785) exited]<br>
[Thread 0x7fffb67e66c0 (LWP 2780) exited]<br>
[Thread 0x7ffe51dff6c0 (LWP 2767) exited]<br>
[Thread 0x7ffff7ca9740 (LWP 2711) exited]<br>
[Thread 0x7fffcc7ee6c0 (LWP 2797) exited]<br>
[New process 2711]<br>
[Inferior 1 (process 2711) exited with code 01]<br>
(cuda-gdb) run train.py<br>
Starting program: /home/praveenbenedict/miniconda3/envs/authenta-generated-image-detection/bin/python train.py<br>
[Thread debugging using libthread_db enabled]<br>
Using host libthread_db library “/lib/x86_64-linux-gnu/libthread_db.so.1”.<br>
[New Thread 0x7ffff4fff6c0 (LWP 2953)]<br>
[New Thread 0x7ffff47fe6c0 (LWP 2954)]<br>
[New Thread 0x7fffefffd6c0 (LWP 2955)]<br>
[New Thread 0x7fffed7fc6c0 (LWP 2956)]<br>
[New Thread 0x7fffecffb6c0 (LWP 2957)]<br>
[New Thread 0x7fffe87fa6c0 (LWP 2958)]<br>
[New Thread 0x7fffe7ff96c0 (LWP 2959)]<br>
[New Thread 0x7fffe37f86c0 (LWP 2960)]<br>
[New Thread 0x7fffe2ff76c0 (LWP 2961)]<br>
[New Thread 0x7fffde7f66c0 (LWP 2962)]<br>
[New Thread 0x7fffddff56c0 (LWP 2963)]<br>
[New Thread 0x7fffd97f46c0 (LWP 2964)]<br>
[New Thread 0x7fffd6ff36c0 (LWP 2965)]<br>
[New Thread 0x7fffd47f26c0 (LWP 2966)]<br>
[New Thread 0x7fffd3ff16c0 (LWP 2967)]<br>
[New Thread 0x7fffcf7f06c0 (LWP 2968)]<br>
[New Thread 0x7fffccfef6c0 (LWP 2969)]<br>
[New Thread 0x7fffca7ee6c0 (LWP 2970)]<br>
[New Thread 0x7fffc9fed6c0 (LWP 2971)]<br>
[New Thread 0x7fffc57ec6c0 (LWP 2972)]<br>
[New Thread 0x7fffc2feb6c0 (LWP 2973)]<br>
[New Thread 0x7fffc27ea6c0 (LWP 2974)]<br>
[New Thread 0x7fffbdfe96c0 (LWP 2975)]<br>
[New Thread 0x7fffbb7e86c0 (LWP 2976)]<br>
[New Thread 0x7fffbafe76c0 (LWP 2977)]<br>
[New Thread 0x7fffb67e66c0 (LWP 2978)]<br>
[New Thread 0x7fffb3fe56c0 (LWP 2979)]<br>
[New Thread 0x7ffe674996c0 (LWP 2982)]<br>
[New Thread 0x7ffe66c986c0 (LWP 2983)]<br>
[New Thread 0x7ffe664976c0 (LWP 2984)]<br>
[New Thread 0x7ffe65c966c0 (LWP 2985)]<br>
[New Thread 0x7ffe654956c0 (LWP 2986)]<br>
[New Thread 0x7ffe64c946c0 (LWP 2987)]<br>
[New Thread 0x7ffe644936c0 (LWP 2988)]<br>
[New Thread 0x7ffe63c926c0 (LWP 2989)]<br>
[New Thread 0x7ffe634916c0 (LWP 2990)]<br>
[New Thread 0x7ffe62c906c0 (LWP 2991)]<br>
[New Thread 0x7ffe6248f6c0 (LWP 2992)]<br>
[New Thread 0x7ffe61c8e6c0 (LWP 2993)]<br>
[New Thread 0x7ffe6148d6c0 (LWP 2994)]<br>
[New Thread 0x7ffe60c8c6c0 (LWP 2995)]<br>
[New Thread 0x7ffe6048b6c0 (LWP 2996)]<br>
[New Thread 0x7ffe5fc8a6c0 (LWP 2997)]<br>
[New Thread 0x7ffe5f4896c0 (LWP 2998)]<br>
[New Thread 0x7ffe5ec886c0 (LWP 2999)]<br>
[New Thread 0x7ffe5e4876c0 (LWP 3000)]<br>
[New Thread 0x7ffe5dc866c0 (LWP 3001)]<br>
[New Thread 0x7ffe5d4856c0 (LWP 3002)]<br>
[New Thread 0x7ffe5cc846c0 (LWP 3003)]<br>
[New Thread 0x7ffe5c4836c0 (LWP 3004)]<br>
[New Thread 0x7ffe5bc826c0 (LWP 3005)]<br>
[New Thread 0x7ffe5b4816c0 (LWP 3006)]<br>
[New Thread 0x7ffe5ac806c0 (LWP 3007)]<br>
[New Thread 0x7ffe5a47f6c0 (LWP 3008)]<br>
Number of Fake images; Train: 45223 Valid: 12691<br>
Number of Real images; Train: 95319 Valid: 10993<br>
Real Images – train – Train: 95319 – Fake: 45223<br>
Real Images – valid – Train: 10993 – Fake: 12691<br>
Number of Fake images; Train: 45223 Valid: 12691<br>
Number of Real images; Train: 95319 Valid: 10993<br>
[New Thread 0x7ffe51dff6c0 (LWP 3010)]<br>
[New Thread 0x7ffe50b6e6c0 (LWP 3011)]<br>
[Thread 0x7ffe644936c0 (LWP 2988) exited]<br>
[Thread 0x7ffe634916c0 (LWP 2990) exited]<br>
[Thread 0x7ffe64c946c0 (LWP 2987) exited]<br>
[Thread 0x7ffe62c906c0 (LWP 2991) exited]<br>
[Thread 0x7ffe654956c0 (LWP 2986) exited]<br>
[Thread 0x7ffe65c966c0 (LWP 2985) exited]<br>
[Thread 0x7ffe664976c0 (LWP 2984) exited]<br>
[Thread 0x7ffe61c8e6c0 (LWP 2993) exited]<br>
[Thread 0x7ffe6148d6c0 (LWP 2994) exited]<br>
[Thread 0x7ffe66c986c0 (LWP 2983) exited]<br>
[Thread 0x7ffe674996c0 (LWP 2982) exited]<br>
[Thread 0x7ffe5f4896c0 (LWP 2998) exited]<br>
[Thread 0x7ffe5dc866c0 (LWP 3001) exited]<br>
[Thread 0x7ffe5d4856c0 (LWP 3002) exited]<br>
[Thread 0x7ffe5cc846c0 (LWP 3003) exited]<br>
[Thread 0x7ffe5c4836c0 (LWP 3004) exited]<br>
[Thread 0x7ffe5e4876c0 (LWP 3000) exited]<br>
[Thread 0x7ffe5ec886c0 (LWP 2999) exited]<br>
[Thread 0x7ffe5b4816c0 (LWP 3006) exited]<br>
[Thread 0x7ffe5fc8a6c0 (LWP 2997) exited]<br>
[Thread 0x7ffe6048b6c0 (LWP 2996) exited]<br>
[Thread 0x7ffe60c8c6c0 (LWP 2995) exited]<br>
[Thread 0x7ffe6248f6c0 (LWP 2992) exited]<br>
[Thread 0x7ffe63c926c0 (LWP 2989) exited]<br>
[Thread 0x7ffe5a47f6c0 (LWP 3008) exited]<br>
[Thread 0x7ffe5ac806c0 (LWP 3007) exited]<br>
[Thread 0x7ffe5bc826c0 (LWP 3005) exited]<br>
[Thread 0x7ffff47fe6c0 (LWP 2954) exited]<br>
[Thread 0x7fffe37f86c0 (LWP 2960) exited]<br>
[Thread 0x7fffddff56c0 (LWP 2963) exited]<br>
[Thread 0x7fffe2ff76c0 (LWP 2961) exited]<br>
[Thread 0x7fffd47f26c0 (LWP 2966) exited]<br>
[Thread 0x7fffe7ff96c0 (LWP 2959) exited]<br>
[Thread 0x7fffd6ff36c0 (LWP 2965) exited]<br>
[Thread 0x7fffccfef6c0 (LWP 2969) exited]<br>
[Thread 0x7fffe87fa6c0 (LWP 2958) exited]<br>
[Thread 0x7fffd97f46c0 (LWP 2964) exited]<br>
[Thread 0x7fffca7ee6c0 (LWP 2970) exited]<br>
[Thread 0x7fffde7f66c0 (LWP 2962) exited]<br>
[Thread 0x7fffecffb6c0 (LWP 2957) exited]<br>
[Thread 0x7fffd3ff16c0 (LWP 2967) exited]<br>
[Thread 0x7fffbdfe96c0 (LWP 2975) exited]<br>
[Thread 0x7fffc9fed6c0 (LWP 2971) exited]<br>
[Thread 0x7fffed7fc6c0 (LWP 2956) exited]<br>
[Thread 0x7fffefffd6c0 (LWP 2955) exited]<br>
[Thread 0x7ffff4fff6c0 (LWP 2953) exited]<br>
[Thread 0x7fffb67e66c0 (LWP 2978) exited]<br>
[Thread 0x7fffb3fe56c0 (LWP 2979) exited]<br>
[Thread 0x7fffbafe76c0 (LWP 2977) exited]<br>
[Thread 0x7fffbb7e86c0 (LWP 2976) exited]<br>
[Thread 0x7fffc27ea6c0 (LWP 2974) exited]<br>
[Thread 0x7fffc2feb6c0 (LWP 2973) exited]<br>
[Thread 0x7fffc57ec6c0 (LWP 2972) exited]<br>
[Thread 0x7fffcf7f06c0 (LWP 2968) exited]<br>
[Detaching after fork from child process 3012]<br>
[New Thread 0x7fffb3fe56c0 (LWP 3021)]<br>
[Thread 0x7fffb3fe56c0 (LWP 3021) exited]<br>
[New Thread 0x7fffb3fe56c0 (LWP 3022)]<br>
[New Thread 0x7fffb67e66c0 (LWP 3023)]<br>
[2025-04-15 11:03:28] [INFO] =&gt; Using checkpoint from run id: 13_04_2025_08_14_51 epoch id: 0<br>
[2025-04-15 11:03:28] [INFO] =&gt; Loading checkpoint<br>
[2025-04-15 11:03:28] [INFO]    =&gt; Current Teacher model metrics - Accuracy: {‘accuracy’: 0.9007581159202742, ‘real_accuracy’: 0.9065225290697675, ‘fake_accuracy’: 0.8949937027707808, ‘f1’: 0.894299447464175}<br>
[2025-04-15 11:03:28] [INFO]    =&gt; Best Teacher model metrics- Accuracy: {‘accuracy’: 0.9007581159202742, ‘real_accuracy’: 0.9065225290697675, ‘fake_accuracy’: 0.8949937027707808, ‘f1’: 0.894299447464175}<br>
[2025-04-15 11:03:28] [INFO]    =&gt; Best Pipeline model metrics- Accuracy: {‘accuracy’: tensor(0.8988), ‘real_accuracy’: tensor(0.9122), ‘fake_accuracy’: tensor(0.8853), ‘f1’: 0.8919182083739046}<br>
[2025-04-15 11:03:28] [INFO]    =&gt; Loaded weights for Teacher Network from best checkpoint<br>
[2025-04-15 11:03:28] [INFO]    =&gt; Loaded the weights for student<br>
[2025-04-15 11:03:28] [INFO]    =&gt; Loaded the weights for augmenter<br>
[2025-04-15 11:03:28] [INFO]    =&gt; Loaded weights for Classifier Network from best checkpoint<br>
[2025-04-15 11:03:28] [INFO] =&gt; Using weighted loss for teacher with a weight of 0.7372192144393921<br>
[2025-04-15 11:03:28] [INFO] =&gt; Using weighted loss for Classifier with a weight of 0.7372192144393921<br>
[2025-04-15 11:03:28] [INFO] =&gt; Adversarial Training…<br>
[2025-04-15 11:03:28] [INFO]    =&gt; Freezing CLIPFeatureExtractor<br>
[2025-04-15 11:03:28] [INFO]    =&gt; Freezing TeacherNetworkTransformer<br>
[2025-04-15 11:03:28] [INFO]    =&gt; Freezing StudentNetworkTransformer<br>
[2025-04-15 11:03:28] [INFO]    =&gt; Unfreezing FeatureAugmenterTransformer<br>
[Detaching after fork from child process 3024]---------------| 0.00% [0/1414 00:00&lt;?]<br>
[Detaching after fork from child process 3025]<br>
[Detaching after fork from child process 3026]<br>
[Detaching after fork from child process 3027]<br>
[New Thread 0x7fffbafe76c0 (LWP 3028)]<br>
[New Thread 0x7fffbb7e86c0 (LWP 3029)]<br>
[New Thread 0x7fffe85fa6c0 (LWP 3030)]<br>
[New Thread 0x7fffe37f86c0 (LWP 3031)]<br>
[New Thread 0x7fffe2ff76c0 (LWP 3032)]<br>
[New Thread 0x7fffde7f66c0 (LWP 3033)]<br>
[New Thread 0x7fffddff56c0 (LWP 3034)]<br>
[New Thread 0x7fffd97f46c0 (LWP 3035)]<br>
[New Thread 0x7fffd6ff36c0 (LWP 3036)]<br>
[New Thread 0x7fffd47f26c0 (LWP 3037)]<br>
[New Thread 0x7fffd3ff16c0 (LWP 3038)]<br>
[New Thread 0x7fffcf7f06c0 (LWP 3039)]<br>
[New Thread 0x7fffccfef6c0 (LWP 3040)]<br>
[New Thread 0x7fffca7ee6c0 (LWP 3041)]<br>
[New Thread 0x7fffc9fed6c0 (LWP 3042)]<br>
[New Thread 0x7fffc57ec6c0 (LWP 3043)]<br>
[New Thread 0x7fffc2feb6c0 (LWP 3044)]<br>
[Thread 0x7fffb3fe56c0 (LWP 3022) exited]<br>
[New Thread 0x7fffb3fe56c0 (LWP 3053)]<br>
warning: Cuda API error detected: cudaLaunchKernel returned (0x2bd)</p>
<p>warning: Cuda API error detected: cudaGetLastError returned (0x2bd)</p>
<p>[Thread 0x7fffc57ec6c0 (LWP 3043) exited]<br>
[Thread 0x7fffc2feb6c0 (LWP 3044) exited]<br>
[Thread 0x7fffc9fed6c0 (LWP 3042) exited]<br>
[Thread 0x7fffca7ee6c0 (LWP 3041) exited]<br>
Traceback (most recent call last):<br>
File “/home/praveenbenedict/Projects/Authenta/authenta-generated-image-detection/notebooks/modelling/GenDet/train.py”, line 96, in <br>
trainer.train()<br>
File “/home/praveenbenedict/Projects/Authenta/authenta-generated-image-detection/notebooks/modelling/GenDet/trainer/trainer.py”, line 650, in train<br>
self.train_augmenter(epochs=self.config[‘training’][‘settings’][‘epochs’][‘adversarial’][‘augmenter’])<br>
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br>
File “/home/praveenbenedict/Projects/Authenta/authenta-generated-image-detection/notebooks/modelling/GenDet/trainer/trainer.py”, line 535, in train_augmenter<br>
loss.backward()<br>
File “/home/praveenbenedict/miniconda3/envs/authenta-generated-image-detection/lib/python3.11/site-packages/torch/_tensor.py”, line 648, in backward<br>
torch.autograd.backward(<br>
File “/home/praveenbenedict/miniconda3/envs/authenta-generated-image-detection/lib/python3.11/site-packages/torch/autograd/<em>init</em>.py”, line 353, in backward<br>
_engine_run_backward(<br>
File “/home/praveenbenedict/miniconda3/envs/authenta-generated-image-detection/lib/python3.11/site-packages/torch/autograd/graph.py”, line 824, in _engine_run_backward<br>
return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass<br>
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br>
RuntimeError: CUDA error: too many resources requested for launch<br>
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.</p>
<h1><a name="p-469191-thread-0x7fffb3fe56c0-lwp-3053-exited-thread-0x7ffe50b6e6c0-lwp-3011-exited-thread-0x7fffccfef6c0-lwp-3040-exited-thread-0x7fffcf7f06c0-lwp-3039-exited-thread-0x7fffd3ff16c0-lwp-3038-exited-thread-0x7fffd47f26c0-lwp-3037-exited-thread-0x7fffd6ff36c0-lwp-3036-exited-thread-0x7fffd97f46c0-lwp-3035-exited-thread-0x7fffddff56c0-lwp-3034-exited-thread-0x7fffde7f66c0-lwp-3033-exited-thread-0x7fffe2ff76c0-lwp-3032-exited-thread-0x7fffe37f86c0-lwp-3031-exited-thread-0x7fffe85fa6c0-lwp-3030-exited-thread-0x7fffbb7e86c0-lwp-3029-exited-thread-0x7fffbafe76c0-lwp-3028-exited-thread-0x7fffb67e66c0-lwp-3023-exited-thread-0x7ffe51dff6c0-lwp-3010-exited-inferior-1-process-2952-exited-with-code-01-1" class="anchor" href="#p-469191-thread-0x7fffb3fe56c0-lwp-3053-exited-thread-0x7ffe50b6e6c0-lwp-3011-exited-thread-0x7fffccfef6c0-lwp-3040-exited-thread-0x7fffcf7f06c0-lwp-3039-exited-thread-0x7fffd3ff16c0-lwp-3038-exited-thread-0x7fffd47f26c0-lwp-3037-exited-thread-0x7fffd6ff36c0-lwp-3036-exited-thread-0x7fffd97f46c0-lwp-3035-exited-thread-0x7fffddff56c0-lwp-3034-exited-thread-0x7fffde7f66c0-lwp-3033-exited-thread-0x7fffe2ff76c0-lwp-3032-exited-thread-0x7fffe37f86c0-lwp-3031-exited-thread-0x7fffe85fa6c0-lwp-3030-exited-thread-0x7fffbb7e86c0-lwp-3029-exited-thread-0x7fffbafe76c0-lwp-3028-exited-thread-0x7fffb67e66c0-lwp-3023-exited-thread-0x7ffe51dff6c0-lwp-3010-exited-inferior-1-process-2952-exited-with-code-01-1"></a>[Thread 0x7fffb3fe56c0 (LWP 3053) exited]<br>
[Thread 0x7ffe50b6e6c0 (LWP 3011) exited]<br>
[Thread 0x7fffccfef6c0 (LWP 3040) exited]<br>
[Thread 0x7fffcf7f06c0 (LWP 3039) exited]<br>
[Thread 0x7fffd3ff16c0 (LWP 3038) exited]<br>
[Thread 0x7fffd47f26c0 (LWP 3037) exited]<br>
[Thread 0x7fffd6ff36c0 (LWP 3036) exited]<br>
[Thread 0x7fffd97f46c0 (LWP 3035) exited]<br>
[Thread 0x7fffddff56c0 (LWP 3034) exited]<br>
[Thread 0x7fffde7f66c0 (LWP 3033) exited]<br>
[Thread 0x7fffe2ff76c0 (LWP 3032) exited]<br>
[Thread 0x7fffe37f86c0 (LWP 3031) exited]<br>
[Thread 0x7fffe85fa6c0 (LWP 3030) exited]<br>
[Thread 0x7fffbb7e86c0 (LWP 3029) exited]<br>
[Thread 0x7fffbafe76c0 (LWP 3028) exited]<br>
[Thread 0x7fffb67e66c0 (LWP 3023) exited]<br>
[Thread 0x7ffe51dff6c0 (LWP 3010) exited]<br>
[Inferior 1 (process 2952) exited with code 01]</h1>
<p>This is the first time I am using cuda-gdb, so I am not sure if this is what you expected me to do. Please do let me know if I need to provide anything else.</p>
<p>Thanks!</p>