Create README.md

README.md

@@ -0,0 +1,80 @@
+---
+license: llama3.2
+---
+
+# FineLlama-3.2-3B-Instruct-ead-GGUF
+
+GGUF quantized versions of [Geraldine/FineLlama-3.2-3B-Instruct-ead](https://huggingface.co/Geraldine/FineLlama-3.2-3B-Instruct-ead) model, optimized for efficient inference using llama.cpp.
+
+## Model Description
+
+- **Base Model**: FineLlama-3.2-3B-Instruct-ead
+- **Quantization**: Various GGUF formats
+- **Purpose**: EAD tag generation and archival metadata encoding
+- **Framework**: llama.cpp
+
+## Available Variants
+
+The following quantized versions are available:
+
+- Q2_K variant (1.36 GB)
+- Q3_K_M variant (1.69 GB)
+- Q4_K_M variant (2.02 GB)
+- Q5_K_M variant (2.32 GB)
+- Q6_K variant (2.64 GB)
+- Q8_0 variant (3.42 GB)
+- FP16 variant (6.43 GB)
+
+## Installation
+
+1. Download the desired GGUF model variant
+2. Install llama.cpp following the official instructions
+3. Place the model file in your llama.cpp models directory
+
+## Usage
+
+```bash
+# Example using Q4_K_M quantization
+./main -m models/FineLlama-3.2-3B-Instruct-ead-Q4_K_M.gguf -n 1024 --repeat_penalty 1.1
+
+# Example using server mode
+./server -m models/FineLlama-3.2-3B-Instruct-ead-Q4_K_M.gguf -c 4096
+```
+
+```python
+# Example using llama-cpp-python library
+from llama_cpp import Llama
+query = "..."
+llm = Llama.from_pretrained(
+    repo_id="Geraldine/FineLlama-3.2-3B-Instruct-ead-GGUF",
+    filename="*Q8_0.gguf",
+    n_ctx=1024,
+    verbose=False
+)
+output = llm.create_chat_completion(
+        messages = [
+            {"role": "system", "content": "You are an archivist expert in EAD format."},
+            {
+                "role": "user",
+                "content": query
+            }
+        ]
+)
+print(output["choices"][0]["message"]["content"])
+```
+
+## Quantization Details
+
+- Q2_K: 2-bit quantization, optimized for efficiency
+- Q3_K_M: 3-bit quantization with medium precision
+- Q4_K_M: 4-bit quantization with medium precision
+- Q5_K_M: 5-bit quantization with medium precision
+- Q6_K: 6-bit quantization
+- Q8_0: 8-bit quantization, highest precision among quantized versions
+- FP16: Full 16-bit floating point, no quantization
+
+## Performance Considerations
+
+- Lower bit quantizations (Q2_K, Q3_K_M) offer smaller file sizes but may have slightly reduced accuracy
+- Higher bit quantizations (Q6_K, Q8_0) provide better accuracy but require more storage and memory
+- FP16 provides full precision but requires significantly more resources