YOYO-AI/Qwen3.6-35B-A3B-YOYO

Use the combined WUDI and FDAs merging methods to fuse the Qwen3.5-35B-A3B and Qwen3.6-35B-A3B models.

Model Highlights:

• Merge Method: WUDI+ FDAs
• Precision: dtype: bfloat16
• Context Length: 262,144

Parameter Settings:

Non-Thinking Mode: ({%- set enable_thinking = false %})

General Tasks:

temperature=0.7, top_p=0.8, top_k=20, min_p=0.0, presence_penalty=1.5, repetition_penalty=1.0

Reasoning Tasks:

temperature=1.0, top_p=1.0, top_k=40, min_p=0.0, presence_penalty=2.0, repetition_penalty=1.0

Thinking Mode: ({%- set enable_thinking = true %})

General Tasks:

temperature=1.0, top_p=0.95, top_k=20, min_p=0.0, presence_penalty=1.5, repetition_penalty=1.1

Coding Tasks:

temperature=0.6, top_p=0.95, top_k=20, min_p=0.0, presence_penalty=0.0, repetition_penalty=1.1

Papers:

WUDI

https://arxiv.org/abs/2503.08099

FDAs

https://arxiv.org/abs/2510.21223

---

Specific Merging Procedure:

This model was created using a two-stage hybrid merging pipeline. We first utilize WUDI to eliminate parameter-space interference without requiring additional data, and subsequently apply FDAs to refine the Feed-Forward Networks layer-by-layer via input-space knowledge distillation.

Step 1: WUDI Merging (Parameter-Space Alignment)

WUDI theoretically demonstrates that the task vectors of linear layers constitute an approximate linear subspace of their corresponding inputs. By leveraging this property, we minimize interference under the guidance of task vectors directly in the parameter space, eliminating the need for rescaling coefficients or extra datasets.

$${\mathcal{L}}_l=\sum _i\frac{1}{\mathrm{\parallel }{\tau }_{i,l}{\mathrm{\parallel }}_F^2}{\parallel ({\tau }_{m,l}-{\tau }_{i,l}){\tau }_{i,l}^{\mathrm{\top }}\parallel }_F^2\backslash mathcal\{L\}\_l\; =\; \backslash sum\_\{i\}\; \backslash frac\{1\}\{\backslash |\backslash tau\_\{i,l\}\backslash |\_F^2\}\; \backslash left\backslash |\; (\backslash tau\_\{m,l\}\; -\; \backslash tau\_\{i,l\})\; \backslash tau\_\{i,l\}^\backslash top\; \backslash right\backslash |\_F^2$$

Hyperparameters:

• Optimizer: Adam
• Iteration Steps: 300
• Learning Rate: 1e-5

---

Step 2: FDAs Refinement (Input-Space Knowledge Distillation)

Functional Dual Anchors shift the merging process into the input-representation space. Instead of merely manipulating parameter offsets, FDAs generates synthetic inputs whose induced gradients align with task vectors. This refinement is applied layer-wise and strictly scoped to the Feed-Forward Networks to capture richer task-specific functional shifts.

Phase 2.1: FDAs Construction (Data Synthesis in Input Space)

We construct synthetic inputs (FDAs) that effectively simulate the role of task vectors. By optimizing these inputs, we ensure that the gradients they induce on the pretrained model align with the actual parameter shifts of the downstream models.

$$\underset{x_{i1},\dots ,x_{in}}{\min }\text{cos\_dist}({\mathrm{\nabla }}_{\theta }\sum _{j=1}^n\text{Dist}(\varphi (\theta ,x_{ij}),\varphi ({\theta }_i,x_{ij})){\mid }_{\theta ={\theta }_0},{\tau }_i)\backslash min\_\{x\_\{i1\},\; \backslash dots,\; x\_\{in\}\}\; \backslash text\{cos\backslash \_dist\}\; \backslash left(\; \backslash nabla\_\backslash theta\; \backslash sum\_\{j=1\}^n\; \backslash text\{Dist\}(\backslash phi(\backslash theta,\; x\_\{ij\}),\; \backslash phi(\backslash theta\_i,\; x\_\{ij\}))\; \backslash Big|\_\{\backslash theta=\backslash theta\_0\},\; \backslash tau\_i\; \backslash right)$$

Hyperparameters:

• Initialization: Scaled Gaussian Sampling (sigma = 0.01)
• Distance Metric: Cosine Distance
• Optimizer: AdamW
• Iteration Steps: 400
• Learning Rate: 1e-2

Phase 2.2: Parameter Optimization (Knowledge Distillation)

We optimize the layer parameters by minimizing the Mean Squared Error between the merged model's representations and the teachers' representations on the FDAs:

$$\underset{{\theta }^{(l)}}{\min }\sum _{i=1}^m\sum _{j=1}^n\text{MSE}({\varphi }^{(l)}({\theta }^{(l)},x_{ij}),{\varphi }^{(l)}({\theta }_i^{(l)},x_{ij}))\backslash min\_\{\backslash theta^\{(l)\}\}\; \backslash sum\_\{i=1\}^m\; \backslash sum\_\{j=1\}^n\; \backslash text\{MSE\}\; \backslash left(\; \backslash phi^\{(l)\}(\backslash theta^\{(l)\},\; x\_\{ij\}),\; \backslash phi^\{(l)\}(\backslash theta\_i^\{(l)\},\; x\_\{ij\})\; \backslash right)$$

Hyperparameters:

• Distance Metric: MSE
• Optimizer: Adam
• Batch Size: 16384
• Iteration Steps: 100
• Learning Rate: 1e-2