---
license: llama2
base_model: lmsys/vicuna-7b-v1.5
tags:
- generated_from_trainer
model-index:
- name: vanilla-model-spr-packing
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

[<img src="https://raw.githubusercontent.com/OpenAccess-AI-Collective/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/OpenAccess-AI-Collective/axolotl)
# vanilla-model-spr-packing

This model is a fine-tuned version of [lmsys/vicuna-7b-v1.5](https://huggingface.co/lmsys/vicuna-7b-v1.5) on the None dataset.
It achieves the following results on the evaluation set:
- Loss: 0.5932

## Model description

The goal of this model is to compress text using SPR (Sparse Priming Representation), focused on Portuguese language.

Prompt template:

A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. USER: [COMPRESSION - Sua tarefa é comprimir o texto] {bigtext} [/COMPRESSION] ASSISTANT: 


## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 0.0003
- train_batch_size: 1
- eval_batch_size: 1
- seed: 42
- gradient_accumulation_steps: 4
- total_train_batch_size: 4
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- lr_scheduler_warmup_steps: 10
- num_epochs: 3

### Training results

| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| 0.8211        | 0.0   | 1    | 0.7681          |
| 0.4915        | 0.15  | 141  | 0.5718          |
| 0.6263        | 0.3   | 282  | 0.5656          |
| 0.5618        | 0.45  | 423  | 0.5563          |
| 0.5024        | 0.6   | 564  | 0.5509          |
| 0.5353        | 0.75  | 705  | 0.5445          |
| 0.6207        | 0.9   | 846  | 0.5429          |
| 0.3893        | 1.04  | 987  | 0.5532          |
| 0.2821        | 1.19  | 1128 | 0.5521          |
| 0.4027        | 1.34  | 1269 | 0.5508          |
| 0.3965        | 1.5   | 1410 | 0.5480          |
| 0.3884        | 1.65  | 1551 | 0.5449          |
| 0.3307        | 1.8   | 1692 | 0.5409          |
| 0.34          | 1.95  | 1833 | 0.5409          |
| 0.2975        | 2.09  | 1974 | 0.5859          |
| 0.2742        | 2.24  | 2115 | 0.5910          |
| 0.2256        | 2.39  | 2256 | 0.5922          |
| 0.294         | 2.54  | 2397 | 0.5901          |
| 0.2246        | 2.69  | 2538 | 0.5944          |
| 0.2038        | 2.84  | 2679 | 0.5932          |


### Framework versions

- Transformers 4.35.2
- Pytorch 2.0.1+cu117
- Datasets 2.15.0
- Tokenizers 0.15.0