Create README.md

README.md

@@ -0,0 +1,39 @@
+---
+license: apache-2.0
+---
+This is the OpenNMT-py converted version of Mistral 7b Instruct v0.2, 4-bit AWQ quantized (gemm version, faster for large batch sizes).
+
+The safetensors file is 4.2GB hence runs smoothly on any RTX card.
+
+Command line to run is:
+```
+python onmt/bin/translate.py --config /pathto/mistral-instruct-inference-awq.yaml --src /pathto/input-vicuna.txt --output /pathto/mistral-output.txt
+```
+Where for instance, input-vicuna.txt contains:
+
+USER:｟newline｠Show me some attractions in Boston.｟newline｠｟newline｠ASSISTANT:｟newline｠
+
+Output will be:
+
+```
+Absolutely, Boston is rich in history and culture. Here are some must-visit attractions in Boston:｟newline｠｟newline｠1. Freedom Trail: This 2.5-mile-long path passes through 16 historical sites, including the Paul Revere House, the Old North Church, and the USS Constitution.｟newline｠｟newline｠2. Boston Common: Established in 1634, Boston Common is the oldest city park in the United States. It covers an area of 50 acres and is home to several monuments, including the Emancipation Monument, the Robert Gould Shaw and the 54th Massachusetts Regiment Memorial, and the Massachusetts Soldiers and Sailors Monument.｟newline｠｟newline｠3. New England Aquarium: Located on the Central Wharf in the Fort Point Channel, the New England Aquarium is one of the premier visitor attractions in Boston. It covers an area of 23 acres and is home to over 20,000 animals, representing more than 1,200 species. The aquarium is divided into several galleries, including the Giant Ocean Tank, the Caribbean Coral Reef Gallery, the Amazon Rainforest Exhibit, the Sh```
+
+
+If you run with a batch size of 60 you can get a nice throughput:
+
+```
+[2023-12-20 13:17:58,837 INFO] Loading checkpoint from /mnt/InternalCrucial4/dataAI/mistral-7B/mistral-instruct-v0.2/Mistral-7B-instruct-onmt-awq-gemm.pt
+[2023-12-20 13:17:58,938 INFO] aawq_gemm compression of layer ['w_1', 'w_2', 'w_3', 'linear_values', 'linear_query', 'linear_keys', 'final_linear']
+[2023-12-20 13:18:02,923 INFO] Loading data into the model
+step0 time:  1.271669864654541
+[2023-12-20 13:18:09,028 INFO] PRED SCORE: -0.2166, PRED PPL: 1.24 NB SENTENCES: 59
+[2023-12-20 13:18:09,028 INFO] Total translation time (s): 5.1
+[2023-12-20 13:18:09,028 INFO] Average translation time (ms): 87.1
+[2023-12-20 13:18:09,028 INFO] Tokens per second: 2494.1
+Time w/o python interpreter load/terminate:  10.200786590576172
+
+```
+
+
+
+