also OOM crash on 3b_int8 and FP16
#8
by KottCh - opened
this seems soft crash, as
PyTorch CUDA memory summary, device ID 0
| CUDA OOMs: 0 | cudaMalloc retries: 0 |
aimdo: /project/src/control.c:177:DEBUG: Aimdo Recorded Usage: 542 MB
aimdo: /project/src/control.c:178:DEBUG: Cuda: 3986 MB / 15885 MB Free
aimdo: /project/src/model-vbar.c:57:DEBUG:---------------- VBAR Usage ---------------
aimdo: /project/src/model-vbar.c:90:DEBUG:Total VRAM for VBARs: 0 MB
aimdo: /project/src/control.c:174:DEBUG:--- VRAM Stats ---
aimdo: /project/src/control.c:177:DEBUG: Aimdo Recorded Usage: 542 MB
aimdo: /project/src/control.c:178:DEBUG: Cuda: 14578 MB / 15885 MB Free
aimdo: /project/src/control.c:174:DEBUG:--- VRAM Stats ---
aimdo: /project/src/control.c:177:DEBUG: Aimdo Recorded Usage: 542 MB
aimdo: /project/src/control.c:178:DEBUG: Cuda: 14578 MB / 15885 MB Free
aimdo: /project/src/control.c:160:DEBUG:cuda_budget_deficit: cuMemGetInfo poll free=14578 MB total=15885 MB deficit_sync=-14322 MB recorded=542 MB
aimdo: /project/src/control.c:164:DEBUG:cuda_budget_deficit: prevailing method cuMemGetInfo
[INFO] Requested to load SeedVR2
[DEBUG] Unloading VideoAutoencoderKLWrapper
[INFO] 0 models unloaded.
aimdo: /project/src/model-vbar.c:219:DEBUG:vbar_allocate (start): size=32979M, device=0
aimdo: /project/src/model-vbar.c:246:DEBUG:vbar_allocate (return): vbar=0x70d5b6548210
aimdo: /project/src/model-vbar.c:330:DEBUG:vbar_get vbar=0x70d5b6548210
aimdo: /project/src/model-vbar.c:301:DEBUG:vbar_prioritize vbar=0x70d5b6548210
[INFO] Model SeedVR2 prepared for dynamic VRAM loading. 3295MB Staged. 0 patches attached. Force pre-loaded 339 weights: 2522 KB.
aimdo: /project/src/model-vbar.c:57:DEBUG:---------------- VBAR Usage ---------------
aimdo: /project/src/model-vbar.c:86:DEBUG:VBAR 0x70d5b6548210: Actual Resident VRAM = 0 MB
aimdo: /project/src/model-vbar.c:90:DEBUG:Total VRAM for VBARs: 0 MB
aimdo: /project/src/plat.h:162:DEBUG:budget_deficit: Prevailing Method: unknown Deficit: 3670 Extra Headroom: 0 Alloc Size 9779
aimdo: /project/src/control.c:174:DEBUG:--- VRAM Stats ---
aimdo: /project/src/control.c:177:DEBUG: Aimdo Recorded Usage: 8557 MB
aimdo: /project/src/control.c:178:DEBUG: Cuda: 16 MB / 15885 MB Free
aimdo: /project/src/model-vbar.c:57:DEBUG:---------------- VBAR Usage ---------------
aimdo: /project/src/model-vbar.c:86:DEBUG:VBAR 0x70d5b6548210: Actual Resident VRAM = 0 MB
aimdo: /project/src/model-vbar.c:90:DEBUG:Total VRAM for VBARs: 0 MB
[ERROR] !!! Exception during processing !!! Allocation on device 0 would exceed allowed memory. (out of memory)
Currently allocated : 8.25 GiB
Requested : 9.55 GiB
Device limit : 15.51 GiB
Free (according to CUDA): 16.25 MiB
PyTorch limit (set by user-supplied memory fraction)
: 17179869184.00 GiB
KottCh changed discussion title from also OOM crash on 3b_int8 to also OOM crash on 3b_int8 and FP16
this should be reworked to be practical. 360p > 1080p, even 5second fails.
congrat' me, 3seconds works. ;;;;
At least you have a different error with the 3b model than I have...( https://huggingface.co/Comfy-Org/SeedVR2/discussions/6 )