---
base_model:
- Lambent/cosmo-1b-tune-pythontest
- Lambent/cosmo-1b-qlora-pythontest
- Lambent/cosmo-1b-lisa-pythontest
- Lambent/cosmo-1b-galore-pythontest
- HuggingFaceTB/cosmo-1b
library_name: transformers
tags:
- mergekit
- merge
license: apache-2.0
---
# pythontestmerge

This is a merge of pre-trained language models created using [mergekit](https://github.com/cg123/mergekit).

Catastrophic forgetting test results:

Initial evaluation loss on 1k subset of HuggingFaceTB/cosmopedia-100k dataset was 1.038. (I'm impressed.)

100 steps of LISA training isn't strictly reducing this over time, it's reducing but jumping around a bit.
Might be converged to within that method's margin of error; cosmo-1b itself jumped 0.02 points with LISA training.

Comparison to control: cosmo-1b started out with 1.003 loss on (a different subset of) dataset, increasing to 1.024 at 100 steps.

Method by method comparison, initial evaluation loss on Cosmopedia data:

* Full tuning (aka continued pretraining), batch 8: 1.615
* LISA fine-tuning, 4 layers switching every 10 steps, batch 8: 1.217
* QLoRA with Dora (otherwise like below): 1.105
* Qlora fine-tuning, rank 256, scale factor 1, batch 8: 1.102
* Galore tuning, rank 256, scale factor 1, batch 8: 1.182
* This Model Stock merge of all 4 training methods: 1.038
* Model Stock 3/4 Methods (all except full tuning): 1.021
* Control (cosmo-1b): 1.003

Training set validation results:

* Cosmo-1b Starting Eval Loss: ~0.65
* Model Stock 3/4 Loss: 0.451
* Model Stock Loss: 0.40211
* LISA Loss: 0.2534
* GaLore Loss: 0.2426
* QLoRA Loss: 0.2078
* QLoRA with Dora Loss: 0.2055 (almost identical training graph)
* Full Tune Loss: 0.2049

Overall ... not sure what to make of this, beyond that high-rank QLoRA is doing something particularly impressive while using only like 6GB of vRAM.
The Model Stock merge between the 4 different tuning methods clearly recovered a lot of original knowledge, at the cost of something like half the adaptation to new data.
Of course, cosmo-1b was already pretty good at predicting the new data, narrow and task-focused as it was.

## Merge Details
### Merge Method

This model was merged using the [Model Stock](https://arxiv.org/abs/2403.19522) merge method using [HuggingFaceTB/cosmo-1b](https://huggingface.co/HuggingFaceTB/cosmo-1b) as a base.

### Models Merged

The following models were included in the merge:
* [Lambent/cosmo-1b-tune-pythontest](https://huggingface.co/Lambent/cosmo-1b-tune-pythontest)
* [Lambent/cosmo-1b-qlora-pythontest](https://huggingface.co/Lambent/cosmo-1b-qlora-pythontest)
* [Lambent/cosmo-1b-lisa-pythontest](https://huggingface.co/Lambent/cosmo-1b-lisa-pythontest)
* [Lambent/cosmo-1b-galore-pythontest](https://huggingface.co/Lambent/cosmo-1b-galore-pythontest)

### Configuration

The following YAML configuration was used to produce this model:

```yaml
models:
  - model: Lambent/cosmo-1b-lisa-pythontest
  - model: Lambent/cosmo-1b-qlora-pythontest
  - model: Lambent/cosmo-1b-galore-pythontest
  - model: Lambent/cosmo-1b-tune-pythontest
base_model: HuggingFaceTB/cosmo-1b
merge_method: model_stock
parameters:
  filter_wise: false
dtype: float16

```