Update README.md

README.md

@@ -39,7 +39,29 @@ print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50, do_sample=Fal
 ##There is a girl who likes adventure, and she is always looking for new experiences. She is a bit of a thrill-seeker, and she loves to push herself to the limit. She is also a bit of a free spirit, and she loves to explore new places and try new things
 ```
 
+### INT4 Inference
+```python
+import habana_frameworks.torch.core as htcore
+import habana_frameworks.torch.hpu as hthpu
+
+from auto_round.auto_quantizer import AutoHfQuantizer
 
+from transformers import AutoModelForCausalLM,AutoTokenizer
+quantized_model_dir = "Intel/Qwen2-7B-int4-inc"
+tokenizer = AutoTokenizer.from_pretrained(quantized_model_dir)
+model = AutoModelForCausalLM.from_pretrained(quantized_model_dir).to('hpu').to(bfloat16)
+text = "下面我来介绍一下阿里巴巴公司，"
+text = "9.8和9.11哪个大？"
+text = "Once upon a time,"
+text = "There is a girl who likes adventure,"
+inputs = tokenizer(text, return_tensors="pt").to(model.device)
+print(model.device)
+print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50, do_sample=False)[0]))
+##下面我来介绍一下阿里巴巴公司，阿里巴巴公司是1999年9月8日由马云在杭州创立的，阿里巴巴集团主要经营业务包括：淘宝网、天猫、聚划算、全球速卖通、阿里巴巴国际交易市场、1688、阿里
+##9.8和9.11哪个大？ 9.8和9.11哪个大？ 解：9.8＞9.11； 答：9.8大． 阅读下面的文字，完成下列各题。 ①“中国式现代化”是习近平
+##Once upon a time, there was a little girl named Emily who loved to read books. She would spend hours lost in the pages of her favorite stories, imagining herself in the worlds she read about. One day, Emily stumbled upon a book called "The Enchanted Forest
+##There is a girl who likes adventure, and she is always looking for new experiences. She is a bit of a thrill-seeker, and she loves to push herself to the limit. She is also a bit of a free spirit, and she loves to explore new places and try new things
+```
 
 ### Evaluate the model