File size: 3,756 Bytes
eb4e6a1 3209b0d eb4e6a1 3209b0d d321ee8 3209b0d |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 |
---
license: mit
language:
- it
- en
library_name: transformers
tags:
- sft
- it
- mistral
- chatml
---
# Model Information
VolareQuantized is a compact iteration of the model [Volare](https://huggingface.co/MoxoffSpA/Volare), optimized for efficiency.
It is offered in two distinct configurations: a 4-bit version and an 8-bit version, each designed to maintain the model's effectiveness while significantly reducing its size
and computational requirements.
- It's trained both on publicly available datasets, like [SQUAD-it](https://huggingface.co/datasets/squad_it), and datasets we've created in-house.
- it's designed to understand and maintain context, making it ideal for Retrieval Augmented Generation (RAG) tasks and applications requiring contextual awareness.
- It is quantized in a 4-bit version and an 8-bit version following the procedure [here](https://github.com/ggerganov/llama.cpp).
# Evaluation
We evaluated the model using the same test sets as used for the Open Ita LLM Leaderboard
| hellaswag_it acc_norm | arc_it acc_norm | m_mmlu_it 5-shot acc | Average |
|:----------------------| :--------------- | :-------------------- | :------- |
| 0.6474 | 0.4671 | da calcolare | da calcolare|
| f1 | Exact Match |
|:---| :---------- |
| 0.6982 | 0.0 |
## Usage
You need to download the .gguf model first
If you want to use the cpu install these dependencies:
```python
pip install llama-cpp-python huggingface_hub
```
If you want to use the gpu instead:
```python
CMAKE_ARGS="-DLLAMA_CUBLAS=on" pip install huggingface_hub llama-cpp-python --force-reinstall --upgrade --no-cache-dir
```
And then use this code to see a response to the prompt.
```python
from huggingface_hub import hf_hub_download
from llama_cpp import Llama
model_path = hf_hub_download(
repo_id="MoxoffSpA/AzzurroQuantized",
filename="Azzurro-ggml-Q4_K_M.gguf"
)
# Set gpu_layers to the number of layers to offload to GPU. Set to 0 if no GPU acceleration is available on your system.
llm = Llama(
model_path=model_path,
n_ctx=2048, # The max sequence length to use - note that longer sequence lengths require much more resources
n_threads=8, # The number of CPU threads to use, tailor to your system and the resulting performance
n_gpu_layers=0 # The number of layers to offload to GPU, if you have GPU acceleration available
)
# Simple inference example
question = """Quanto è alta la torre di Pisa?"""
context = """
La Torre di Pisa è un campanile del XII secolo, famoso per la sua inclinazione. Alta circa 56 metri.
"""
prompt = f"Domanda: {question}, contesto: {context}"
output = llm(
f"[INST] {prompt} [/INST]", # Prompt
max_tokens=128,
stop=["\n"],
echo=True,
temperature=0.1,
top_p=0.95
)
# Chat Completion API
print(output['choices'][0]['text'])
```
## Bias, Risks and Limitations
VolareQuantized and its original model have not been aligned to human preferences for safety within the RLHF phase or deployed with in-the-loop filtering of
responses like ChatGPT, so the model can produce problematic outputs (especially when prompted to do so). It is also unknown what the size and composition
of the corpus was used to train the base model, however, it is likely to have included a mix of Web data and technical sources
like books and code.
## Links to resources
- SQUAD-it dataset: https://huggingface.co/datasets/squad_it
- Gemma-7b model: https://huggingface.co/google/gemma-7b
- Open Ita LLM Leaderbord: https://huggingface.co/spaces/FinancialSupport/open_ita_llm_leaderboard
## Quantized versions
We have the not quantized version here:
https://huggingface.co/MoxoffSpA/Volare
## The Moxoff Team
Jacopo Abate, Marco D'Ambra, Luigi Simeone, Gianpaolo Francesco Trotta
|