Huge memory requirements
#2
by mllearner123 - opened
this function def _build_non_streaming_sdpa_bias() tries to take 4GB of memory even though the model is only 90MB.
at this line: delta = positions.view(1, max_seqlen, 1) - positions.view(1, 1, max_seqlen)
so this breaks GPUs with lower VRAM
maybe xformer masks could be applied or triton kernels