aws-neuron

/

optimum-neuron-cache

like 7

Inference cache is only supported for causal lm models. cc @Jingya

Hi @k10 , marian type models are not yet supported by optimum-neuron. To add its cache, we will need to add the export and inference support for it first.

I opened a ticket here, feel free to pick the task up if you want to contribute!