Update README.md

README.md

@@ -32,6 +32,22 @@ python3 main.py \
 
 
 ### Use the model
+### INT4 Inference with ITREX on CPU
+Install the latest [intel-extension-for-transformers](https://github.com/intel/intel-extension-for-transformers)
+```python
+from intel_extension_for_transformers.transformers import AutoModelForCausalLM
+from transformers import AutoTokenizer
+quantized_model_dir = "Intel/Mistral-7B-v0.1-int4-inc"
+model = AutoModelForCausalLM.from_pretrained(quantized_model_dir,
+                                             device_map="auto",
+                                             trust_remote_code=False,
+                                             use_neural_speed=False,
+                                             )
+tokenizer = AutoTokenizer.from_pretrained(quantized_model_dir, use_fast=True)
+print(tokenizer.decode(model.generate(**tokenizer("There is a girl who likes adventure,", return_tensors="pt").to(model.device),max_new_tokens=50)[0]))
+## <s> There is a girl who likes adventure, and she is a little bit crazy. She is a little bit crazy because she likes to do things that are dangerous. She likes to climb mountains, and she likes to go on long hikes. She also likes to go on long bike rides
+```
+
 
 ### INT4 Inference with AutoGPTQ