aws-neuron

/

optimum-neuron-cache

like 10

Follow

AWS Inferentia and Trainium 55

Hi @gaussfer , as discussed with you on the optimum-neuron repo. We will need to add tensor parallel support first. Unfortunately, the tp support for t5 is not working for some tracing issue. AWS's side is working on solving it, optimum team will prioritize the t5 tp support as well once it's fixed.

Related: https://github.com/huggingface/optimum-neuron/issues/479

Thanks @Jingya .