metadata
license: llama2
base_model: lmsys/vicuna-7b-v1.5
tags:
- generated_from_trainer
model-index:
- name: vanilla-model-spr-packing
results: []
vanilla-model-spr-packing
This model is a fine-tuned version of lmsys/vicuna-7b-v1.5 on the None dataset. It achieves the following results on the evaluation set:
- Loss: 0.5932
Model description
The goal of this model is to compress text using SPR (Sparse Priming Representation), focused on Portuguese language.
Prompt template:
A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. USER: [COMPRESSION - Sua tarefa é comprimir o texto] {bigtext} [/COMPRESSION] ASSISTANT:
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 0.0003
- train_batch_size: 1
- eval_batch_size: 1
- seed: 42
- gradient_accumulation_steps: 4
- total_train_batch_size: 4
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- lr_scheduler_warmup_steps: 10
- num_epochs: 3
Training results
| Training Loss | Epoch | Step | Validation Loss |
|---|---|---|---|
| 0.8211 | 0.0 | 1 | 0.7681 |
| 0.4915 | 0.15 | 141 | 0.5718 |
| 0.6263 | 0.3 | 282 | 0.5656 |
| 0.5618 | 0.45 | 423 | 0.5563 |
| 0.5024 | 0.6 | 564 | 0.5509 |
| 0.5353 | 0.75 | 705 | 0.5445 |
| 0.6207 | 0.9 | 846 | 0.5429 |
| 0.3893 | 1.04 | 987 | 0.5532 |
| 0.2821 | 1.19 | 1128 | 0.5521 |
| 0.4027 | 1.34 | 1269 | 0.5508 |
| 0.3965 | 1.5 | 1410 | 0.5480 |
| 0.3884 | 1.65 | 1551 | 0.5449 |
| 0.3307 | 1.8 | 1692 | 0.5409 |
| 0.34 | 1.95 | 1833 | 0.5409 |
| 0.2975 | 2.09 | 1974 | 0.5859 |
| 0.2742 | 2.24 | 2115 | 0.5910 |
| 0.2256 | 2.39 | 2256 | 0.5922 |
| 0.294 | 2.54 | 2397 | 0.5901 |
| 0.2246 | 2.69 | 2538 | 0.5944 |
| 0.2038 | 2.84 | 2679 | 0.5932 |
Framework versions
- Transformers 4.35.2
- Pytorch 2.0.1+cu117
- Datasets 2.15.0
- Tokenizers 0.15.0