---
license: mit
---
# 0506_9_8

This model is a fine-tuned version of [../../models/Qwen1.5-7B-sft-0502](https://huggingface.co/../../models/Qwen1.5-7B-sft-0502) on the alpaca_formatted_review_new_data_0505_greater_8 dataset.
It achieves the following results on the evaluation set:

- Loss: 0.5497

## Model description

Qwen1.5 is the beta version of Qwen2, a transformer-based decoder-only language model pretrained on a large amount of data. In comparison with the previous released Qwen, the improvements include:

* 8 model sizes, including 0.5B, 1.8B, 4B, 7B, 14B, 32B and 72B dense models, and an MoE model of 14B with 2.7B activated;
* Significant performance improvement in Chat models;
* Multilingual support of both base and chat models;
* Stable support of 32K context length for models of all sizes
* No need of `trust_remote_code`.

For more details, please refer to the [blog post](https://qwenlm.github.io/blog/qwen1.5/) and [GitHub repo](https://github.com/QwenLM/Qwen1.5).

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:

- learning_rate: 7e-05
- train_batch_size: 2
- eval_batch_size: 1
- seed: 42
- distributed_type: multi-GPU
- num_devices: 2
- gradient_accumulation_steps: 4
- total_train_batch_size: 16
- total_eval_batch_size: 2
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- lr_scheduler_warmup_steps: 13
- num_epochs: 5.0
- mixed_precision_training: Native AMP

### Training results

| Training Loss | Epoch  | Step | Validation Loss |
| :-----------: | :----: | :--: | :-------------: |
|    0.6358     | 0.7619 |  20  |     0.5865      |
|    0.6379     | 1.5238 |  40  |     0.5621      |
|    0.6067     | 2.2857 |  60  |     0.5561      |
|    0.5339     | 3.0476 |  80  |     0.5515      |
|    0.6749     | 3.8095 | 100  |     0.5500      |
|    0.6351     | 4.5714 | 120  |     0.5497      |


### Framework versions

- PEFT 0.10.0
- Transformers 4.40.0
- Pytorch 2.1.0+cu121
- Datasets 2.14.5
- Tokenizers 0.19.1