metadata

base_model:
  - appvoid/arco
  - h2oai/h2o-danube3-500m-base
library_name: transformers
tags:
  - mergekit
  - merge

arco+

This is an untrained passthrough model based on arco and danube as a first effort to train a small enough reasoning language model that generalizes across all kind of reasoning tasks.

Benchmarks

Parameters	Model	MMLU	ARC	HellaSwag	PIQA	Winogrande	Average
488m	arco-lite	23.22	33.45	56.55	69.70	59.19	48.46
773m	arco-plus	23.06	36.43	60.09	72.36	60.46	50.48

Configuration

The following YAML configuration was used to produce this model:

slices:
  - sources:
    - model: appvoid/arco
      layer_range: [0, 14]
  - sources:
    - model: h2oai/h2o-danube3-500m-base
      layer_range: [4, 16]

merge_method: passthrough
dtype: float16