Hugging Face's logo

Intel
/
gpt-j-6B-int8-dynamic-inc

Text Generation Transformers ONNX
English gptj text-generation-inference causal-lm int8 ONNX PostTrainingDynamic Intel® Neural Compressor neural-compressor Inference Endpoints
Model card Files Community
1

Batch inference

#1
by zqcawzs - opened
Discussion
zqcawzs

Hi,

Could you give an example about how to do batch inference using this gpt-j-6B-int8-dynamic model?

Thanks!

MengniWang
 Intel org

Hi,

I have added batch inference code into the evaluation.ipynb, please check it!

Sign up or log in to comment