vanilla-model-spr-packing
This model is a fine-tuned version of lmsys/vicuna-7b-v1.5 on the None dataset. It achieves the following results on the evaluation set:
- Loss: 0.5932
Model description
The goal of this model is to compress text using SPR (Sparse Priming Representation), focused on Portuguese language.
Prompt template:
A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. USER: [COMPRESSION - Sua tarefa é comprimir o texto] {bigtext} [/COMPRESSION] ASSISTANT:
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 0.0003
- train_batch_size: 1
- eval_batch_size: 1
- seed: 42
- gradient_accumulation_steps: 4
- total_train_batch_size: 4
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- lr_scheduler_warmup_steps: 10
- num_epochs: 3
Training results
Training Loss | Epoch | Step | Validation Loss |
---|---|---|---|
0.8211 | 0.0 | 1 | 0.7681 |
0.4915 | 0.15 | 141 | 0.5718 |
0.6263 | 0.3 | 282 | 0.5656 |
0.5618 | 0.45 | 423 | 0.5563 |
0.5024 | 0.6 | 564 | 0.5509 |
0.5353 | 0.75 | 705 | 0.5445 |
0.6207 | 0.9 | 846 | 0.5429 |
0.3893 | 1.04 | 987 | 0.5532 |
0.2821 | 1.19 | 1128 | 0.5521 |
0.4027 | 1.34 | 1269 | 0.5508 |
0.3965 | 1.5 | 1410 | 0.5480 |
0.3884 | 1.65 | 1551 | 0.5449 |
0.3307 | 1.8 | 1692 | 0.5409 |
0.34 | 1.95 | 1833 | 0.5409 |
0.2975 | 2.09 | 1974 | 0.5859 |
0.2742 | 2.24 | 2115 | 0.5910 |
0.2256 | 2.39 | 2256 | 0.5922 |
0.294 | 2.54 | 2397 | 0.5901 |
0.2246 | 2.69 | 2538 | 0.5944 |
0.2038 | 2.84 | 2679 | 0.5932 |
Framework versions
- Transformers 4.35.2
- Pytorch 2.0.1+cu117
- Datasets 2.15.0
- Tokenizers 0.15.0
- Downloads last month
- 8
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social
visibility and check back later, or deploy to Inference Endpoints (dedicated)
instead.
Model tree for cnmoro/vicuna-7b-v1.5-SPR-Compressor
Base model
lmsys/vicuna-7b-v1.5
Finetuned
this model