cnmoro/vicuna-7b-v1.5-SPR-Compressor

vanilla-model-spr-packing

This model is a fine-tuned version of lmsys/vicuna-7b-v1.5 on the None dataset. It achieves the following results on the evaluation set:

• Loss: 0.5932

Model description

The goal of this model is to compress text using SPR (Sparse Priming Representation), focused on Portuguese language.

Prompt template:

A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. USER: [COMPRESSION - Sua tarefa é comprimir o texto] {bigtext} [/COMPRESSION] ASSISTANT:

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0003
• train_batch_size: 1
• eval_batch_size: 1
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 4
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 10
• num_epochs: 3

Training results

Training Loss	Epoch	Step	Validation Loss
0.8211	0.0	1	0.7681
0.4915	0.15	141	0.5718
0.6263	0.3	282	0.5656
0.5618	0.45	423	0.5563
0.5024	0.6	564	0.5509
0.5353	0.75	705	0.5445
0.6207	0.9	846	0.5429
0.3893	1.04	987	0.5532
0.2821	1.19	1128	0.5521
0.4027	1.34	1269	0.5508
0.3965	1.5	1410	0.5480
0.3884	1.65	1551	0.5449
0.3307	1.8	1692	0.5409
0.34	1.95	1833	0.5409
0.2975	2.09	1974	0.5859
0.2742	2.24	2115	0.5910
0.2256	2.39	2256	0.5922
0.294	2.54	2397	0.5901
0.2246	2.69	2538	0.5944
0.2038	2.84	2679	0.5932

Framework versions

• Transformers 4.35.2
• Pytorch 2.0.1+cu117
• Datasets 2.15.0
• Tokenizers 0.15.0