---
license: mit
---
# 0506_7_7

This model is a fine-tuned version of [../../models/Qwen1.5-7B-sft-0502](https://huggingface.co/../../models/Qwen1.5-7B-sft-0502) on the alpaca_formatted_review_new_data_0505_greater_7 dataset.
It achieves the following results on the evaluation set:

- Loss: 0.7221

## Model description

Qwen1.5 is the beta version of Qwen2, a transformer-based decoder-only language model pretrained on a large amount of data. In comparison with the previous released Qwen, the improvements include:

* 8 model sizes, including 0.5B, 1.8B, 4B, 7B, 14B, 32B and 72B dense models, and an MoE model of 14B with 2.7B activated;
* Significant performance improvement in Chat models;
* Multilingual support of both base and chat models;
* Stable support of 32K context length for models of all sizes
* No need of `trust_remote_code`.

For more details, please refer to the [blog post](https://qwenlm.github.io/blog/qwen1.5/) and [GitHub repo](https://github.com/QwenLM/Qwen1.5).

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:

- learning_rate: 0.0003
- train_batch_size: 2
- eval_batch_size: 1
- seed: 42
- distributed_type: multi-GPU
- num_devices: 2
- gradient_accumulation_steps: 4
- total_train_batch_size: 16
- total_eval_batch_size: 2
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- lr_scheduler_warmup_steps: 20
- num_epochs: 5.0
- mixed_precision_training: Native AMP

### Training results

| Training Loss | Epoch  | Step | Validation Loss |
| :-----------: | :----: | :--: | :-------------: |
|    0.7981     | 0.2768 |  20  |     0.6501      |
|    0.7391     | 0.5536 |  40  |     0.6358      |
|     0.744     | 0.8304 |  60  |     0.6277      |
|    0.6284     | 1.1073 |  80  |     0.6241      |
|    0.7339     | 1.3841 | 100  |     0.6303      |
|    0.8346     | 1.6609 | 120  |     0.6408      |
|    0.6927     | 1.9377 | 140  |     0.6391      |
|    0.4915     | 2.2145 | 160  |     0.6543      |
|    0.7845     | 2.4913 | 180  |     0.6596      |
|    0.6619     | 2.7682 | 200  |     0.6587      |
|    0.4897     | 3.0450 | 220  |     0.6679      |
|    0.5064     | 3.3218 | 240  |     0.6951      |
|    0.6467     | 3.5986 | 260  |     0.6997      |
|    0.6615     | 3.8754 | 280  |     0.6985      |
|    0.4954     | 4.1522 | 300  |     0.7111      |
|    0.5624     | 4.4291 | 320  |     0.7216      |
|    0.5554     | 4.7059 | 340  |     0.7218      |
|    0.6798     | 4.9827 | 360  |     0.7221      |


### Framework versions

- PEFT 0.10.0
- Transformers 4.40.0
- Pytorch 2.1.0+cu121
- Datasets 2.14.5
- Tokenizers 0.19.1