Update README.md

README.md

@@ -13,4 +13,68 @@ thumbnail: "https://cdn-uploads.huggingface.co/production/uploads/6589d7e6586088
 
 The passthrough method differs significantly from the previous ones. By concatenating layers from different LLMs, it can produce models with an exotic number of parameters (e.g., 9B with two 7B parameter models). These models are often referred to as "frankenmerges" or "Frankenstein models" by the community.
 
-Many thanks to [Abacaj](https://huggingface.co/abacaj) for providing the [fine tuned weights](https://huggingface.co/abacaj/phi-2-super) that were used in the creation of this base model. You can find the full script for how the model was merged [here]...thanks to [KatyTheCutie](https://huggingface.co/KatyTheCutie) for helping me figure out how to make the model as big as I possibly could.
+Many thanks to [Abacaj](https://huggingface.co/abacaj) for providing the [fine tuned weights](https://huggingface.co/abacaj/phi-2-super) that were used in the creation of this base model. You can find the full script for how the model was merged [here]...thanks to [KatyTheCutie](https://huggingface.co/KatyTheCutie) for helping me figure out how to make the model as big as I possibly could.
+
+# How to run inference:
+
+```python
+import transformers
+import torch
+
+if __name__ == "__main__":
+  model_name = "abacaj/phi-2-super"
+  tokenizer = transformers.AutoTokenizer.from_pretrained(model_name)
+  
+  model = (
+      transformers.AutoModelForCausalLM.from_pretrained(
+          model_name,
+      )
+      .to("cuda:0")
+      .eval()
+  )
+  
+  messages = [
+      {"role": "user", "content": "Hello, who are you?"}
+  ]
+  inputs = tokenizer.apply_chat_template(messages, return_tensors="pt").to(model.device)
+  input_ids_cutoff = inputs.size(dim=1)
+  
+  with torch.no_grad():
+      generated_ids = model.generate(
+          input_ids=inputs,
+          use_cache=True,
+          max_new_tokens=512,
+          temperature=0.2,
+          top_p=0.95,
+          do_sample=True,
+          eos_token_id=tokenizer.eos_token_id,
+          pad_token_id=tokenizer.pad_token_id,
+      )
+  
+  completion = tokenizer.decode(
+      generated_ids[0][input_ids_cutoff:],
+      skip_special_tokens=True,
+  )
+  
+  print(completion)
+```
+
+# Chat template
+
+The model uses the same chat template as found in Mistral instruct models:
+
+```python
+text = "<|endoftext|>[INST] What is your favourite condiment? [/INST]"
+"Well, I'm quite partial to a good squeeze of fresh lemon juice. It adds just the right amount of zesty flavour to whatever I'm cooking up in the kitchen!<|endoftext|> "
+"[INST] Do you have mayonnaise recipes? [/INST]"
+```
+
+You don't need to do it manually if you use the HF transformers tokenizer:
+
+```python
+  messages = [
+      {"role": "user", "content": "Hello, who are you?"},
+      {"role": "assistant": "content": "I am ..."}
+  ]
+  inputs = tokenizer.apply_chat_template(messages, return_tensors="pt").to(model.device)
+```