Batch Inference
#2
by
krypticmouse
- opened
How do you do batch inference on this? I tried to do it but everything except bsize=1 one is failing. I'm using A100 80GB.
Hi @krypticmouse ,
just fyi I have seen your comments. I am looking into it now, will get back to you asap.
Thanks a lot!
please check https://github.com/texttron/tevatron/tree/main/examples/rankllama for batch inference
MrLight
changed discussion status to
closed
MrLight
changed discussion status to
open