Model is autocasting in BF16 in infer() And the model does not support for multi GPU.
Model is autocasting in BF16 in infer() which is causing problem when I download the model with FP16 Also the model does not support for multi GPU. When I have 2 GPUs but the model uses on one CUDA 0 and when splits it on 2 GPU then while running it throws error saying RuntimeError: Expected all tensors to be on the same device, but got index is on cuda:0, different from other tensors on cuda:1 (when checking argument in method wrapper_CUDA__index_select).
So I manually need to patch the forward function even after all this when I try to run the infer() this BF16 autocast inside the code causing problem uuhhh can you make it support for both BF16 and FP16 however there is no arg in infer() to tell it to use FP16 and do not cast it to BF16 bcoz I have downloaded the model in FP16 now I have to patch it manually again.
Are you really serious???
Uhhh Too much work 😢