Thziin/mergekit-slerp-ulhxfns

merge

This is a merge of pre-trained language models created using mergekit.

Merge Details

Merge Method

This model was merged using the SLERP merge method.

Models Merged

The following models were included in the merge:

• Qwen/Qwen2.5-Coder-32B-Instruct
• Qwen/QwQ-32B-Preview

Configuration

The following YAML configuration was used to produce this model:

merge_method: slerp
base_model:
  model: Qwen/Qwen2.5-Coder-32B-Instruct  # Define the base model explicitly
slices:
  - sources:
      - model: Qwen/Qwen2.5-Coder-32B-Instruct
        layer_range:
          - 0
          - 32
      - model: Qwen/QwQ-32B-Preview
        layer_range:
          - 0
          - 32
    parameters:
      t:
        - filter: self_attn
          value:
            - 0
            - 0.5
            - 0.3
            - 0.7
            - 1
        - filter: mlp
          value:
            - 1
            - 0.5
            - 0.7
            - 0.3
            - 0
        - value: 0.5
dtype: bfloat16