README.md

metadata

base_model:
  - Lambent/CosmoAlpacaLight-1b
  - HuggingFaceTB/cosmo-1b
  - Lambent/CosmoAlpacaLisa-1b
  - Lambent/CosmoAlpacaLisa-0.2-1b
tags:
  - mergekit
  - merge

Figured the model must have learned the data a somewhat different way for all three training methods. Decided to test if merging them would give generalization benefits. I think it didn't hurt at least!

Model	AGIEval	GPT4All	TruthfulQA	Bigbench	Average
CosmoAlpacaLisa-0.3-1b	23.79	51.61	40.25	29.97	36.41

Model	AGIEval	GPT4All	TruthfulQA	Bigbench	Average
CosmoAlpacaLight-1b	24.28	51.31	40.33	29.47	36.35

lisacosmo

This is a merge of pre-trained language models created using mergekit.

Merge Details

Merge Method

This model was merged using the DARE TIES merge method using HuggingFaceTB/cosmo-1b as a base.

Models Merged

The following models were included in the merge:

• Lambent/CosmoAlpacaLight-1b
• Lambent/CosmoAlpacaLisa-1b
• Lambent/CosmoAlpacaLisa-0.2-1b

Configuration

The following YAML configuration was used to produce this model:

models:
  - model: Lambent/CosmoAlpacaLisa-1b
    parameters:
      density: 1.0
      weight: 1.0
  - model: Lambent/CosmoAlpacaLisa-0.2-1b
    parameters:
      density: 1.0
      weight: 1.0
  - model: Lambent/CosmoAlpacaLight-1b
    parameters:
      density: 1.0
      weight: 1.0
merge_method: dare_ties
base_model: HuggingFaceTB/cosmo-1b
parameters:
  normalize: true
  int8_mask: false
dtype: float16