--- base_model: - MrRobotoAI/102 - MrRobotoAI/97 library_name: transformers tags: - mergekit - merge --- # merge This is a merge of pre-trained language models created using [mergekit](https://github.com/cg123/mergekit). ## Merge Details ### Merge Method This model was merged using the [Linear DARE](https://arxiv.org/abs/2311.03099) merge method using [MrRobotoAI/102](https://huggingface.co/MrRobotoAI/102) as a base. ### Models Merged The following models were included in the merge: * [MrRobotoAI/97](https://huggingface.co/MrRobotoAI/97) ### Configuration The following YAML configuration was used to produce this model: ```yaml merge_method: dare_linear models: - model: MrRobotoAI/102 parameters: weight: - filter: v_proj value: [0.8, 0.8, 0.75, 0.55, 0.35, 0.15, 0.35, 0.55, 0.75, 0.8, 0.8] - filter: o_proj value: [0.8, 0.8, 0.75, 0.55, 0.35, 0.15, 0.35, 0.55, 0.75, 0.8, 0.8] - filter: up_proj value: [0.8, 0.8, 0.75, 0.55, 0.35, 0.15, 0.35, 0.55, 0.75, 0.8, 0.8] - filter: gate_proj value: [0.8, 0.8, 0.75, 0.55, 0.35, 0.15, 0.35, 0.55, 0.75, 0.8, 0.8] - filter: down_proj value: [0.8, 0.8, 0.75, 0.55, 0.35, 0.15, 0.35, 0.55, 0.75, 0.8, 0.8] - value: 1 - model: MrRobotoAI/97 parameters: weight: - filter: v_proj value: [0.2, 0.2, 0.25, 0.45, 0.65, 0.85, 0.65, 0.45, 0.25, 0.2, 0.2] - filter: o_proj value: [0.2, 0.2, 0.25, 0.45, 0.65, 0.85, 0.65, 0.45, 0.25, 0.2, 0.2] - filter: up_proj value: [0.2, 0.2, 0.25, 0.45, 0.65, 0.85, 0.65, 0.45, 0.25, 0.2, 0.2] - filter: gate_proj value: [0.2, 0.2, 0.25, 0.45, 0.65, 0.85, 0.65, 0.45, 0.25, 0.2, 0.2] - filter: down_proj value: [0.2, 0.2, 0.25, 0.45, 0.65, 0.85, 0.65, 0.45, 0.25, 0.2, 0.2] - value: 0 base_model: MrRobotoAI/102 tokenizer_source: base dtype: bfloat16 ```