Repository: Lightning-AI/litgpt (language: python)

Issue:
Support 128k token version of Phi 3
Right now we only support the Phi 3 version that supports up to 4k tokens, it would be nice to also support the 128k token version: [microsoft/Phi-3-mini-128k-instruct](https://huggingface.co/microsoft/Phi-3-mini-128k-instruct)


Discussion / hints:

Fix patch:
diff --git a/litgpt/config.py b/litgpt/config.py
index 7ff9fc4d25..463aa5514c 100644
--- a/litgpt/config.py
+++ b/litgpt/config.py
@@ -1660,6 +1660,24 @@ def norm_class(self) -> Type:
         sliding_window_size=2048,
         sliding_window_layer_placing="all",
     ),
+    # https://huggingface.co/microsoft/Phi-3-mini-128k-instruct/blob/main/config.json
+    dict(
+        name="Phi-3-mini-128k-instruct",
+        hf_config=dict(org="microsoft", name="Phi-3-mini-128k-instruct"),
+        vocab_size=32000,
+        padded_vocab_size=32064,
+        block_size=131072,
+        n_embd=3072,
+        n_layer=32,
+        rotary_percentage=1.0,
+        bias=False,
+        norm_class_name="RMSNorm",
+        intermediate_size=8192,
+        mlp_class_name="LLaMAMLP",
+        parallel_residual=False,
+        sliding_window_size=262145,
+        sliding_window_layer_placing="all",
+    ),
     # https://huggingface.co/microsoft/Phi-3.5-mini-instruct/blob/main/config.json
     dict(
         name="Phi-3.5-mini-instruct",
diff --git a/tutorials/download_model_weights.md b/tutorials/download_model_weights.md
index 174ebf2700..35263180ce 100644
--- a/tutorials/download_model_weights.md
+++ b/tutorials/download_model_weights.md
@@ -137,6 +137,7 @@ meta-llama/Meta-Llama-3.1-8B
 meta-llama/Meta-Llama-3.1-8B-Instruct
 microsoft/phi-1_5
 microsoft/phi-2
+microsoft/Phi-3-mini-128k-instruct
 microsoft/Phi-3-mini-4k-instruct
 microsoft/Phi-3.5-mini-instruct
 mistralai/mathstral-7B-v0.1


Test patch:
diff --git a/tests/test_model.py b/tests/test_model.py
index c62d9c5cdd..4cff66446b 100644
--- a/tests/test_model.py
+++ b/tests/test_model.py
@@ -324,7 +324,7 @@ def test_against_hf_phi(model_name, device, dtype):
 
 
 @torch.inference_mode()
-@pytest.mark.parametrize("model_name", ("Phi-3-mini-4k-instruct", "Phi-3.5-mini-instruct"))
+@pytest.mark.parametrize("model_name", ("Phi-3-mini-4k-instruct", "Phi-3-mini-128k-instruct", "Phi-3.5-mini-instruct"))
 @pytest.mark.parametrize(
     ("device", "dtype"),
     [