What Limits Bidirectional Model's Generative Capabilities? A Uni-Bi-Directional Mixture-of-Expert Method For Bidirectional Fine-tuning
Authors: Zuchao Li
Yonghua Hei
Qiwei Li
Lefei Zhang
Ping Wang
hai zhao
Baoyuan Qi
Liu Guoming
Venue: ICML 2025 poster (2025)
Keywords: Large Language Model
Bidirectional Modeling
Causal Attention
Primary area: deep_learning->large_language_models
TL;DR: We demonstrate that bidirectional training leads to an increase in subsequent dependency, and propose  A Uni-Bi-Directional Mixture-of-Expert Large Language Model.

Abstract: Large Language Models (LLMs) excel in generation tasks, yet their causal attention mechanisms limit performance in embedding tasks. While bidirectional modeling may enhance embeddings, naively fine-tuning unidirectional models bidirectionally severely degrades generative performance.
To investigate this trade-off, we analyze attention weights as dependence indicators and find that bidirectional fine-tuning increases subsequent dependence, impairing unidirectional generation. Through systematic Transformer module evaluations, we discover the FFN layer is least affected by such dependence.  Leveraging this discovery, we propose UBMoE-LLM, a novel Uni-Bi-directional Mixture-of-Experts LLM, which integrates the original unidirectional FFN with a bidirectionally fine-tuned FFN via unsupervised contrastive learning.  This MoE-based approach enhances embedding performance while preserving robust generation.
Extensive experiments across diverse datasets and model scales validate our attention dependence metric and demonstrate UBMoE-LLM’s superior generative quality and reduced hallucination. Code is available at: https://github.com/heiyonghua/ubmoe_llm.

What Limits Bidirectional Model's Generative Capabilities?
A Uni-Bi-Directional Mixture-of-Expert Method For Bidirectional Fine-tuning
Zuchao Li
Yonghua Hei?
Qiwei Li
Lefei Zhang?
Ping Wang* 4 Hai Zhao
Baoyuan Qi Guoming Liu
Abstract
language processing (Muennighoff et al., 2024; Bao et al., 2020; Dong et al., 2019). Large causal language models have shown strong capabilities across multiple tasks (Tou- vron et al., 2023a;b; Jiang et al., 2023; Bai et al., 2023), making them a promising choice for achieving this goal. Recent research indicates that causal language models are limited by their causal attention mechanisms in embedding tasks, making unidirectional models suboptimal for these tasks (BehnamGhader et al., 2024; Springer et al., 2024; Muennighoff et al., 2024). A natural idea to address this issue is to enhance the bidirectional modeling capabilities of causal language models. However, simply enabling bidirec- tional modeling in Large Language Model (LLMs) results in a significant decline in generative performance (Muen- nighoff et al., 2024). Therefore, critical challenge in build- ing such a unified model is how to enhance its bidirectional modeling capabilities while maintaining robust generative performance.
Large Language Models (LLMs) excel genera- tion tasks, yet their causal attention mechanisms limit performance in embedding tasks. While bidirectional modeling may enhance embeddings, naively fine-tuning unidirectional models bidi- rectionally severely degrades generative perfor- mance. To investigate this trade-off, we analyze attention weights as dependence indicators and find that bidirectional fine-tuning increases subse- quent dependence, impairing unidirectional gen- eration. Through systematic Transformer module evaluations, we discover the FFN layer is least affected by such dependence. Leveraging this dis- covery, we propose UBMOE-LLM, a novel Uni- Bi-directional Mixture-of -f-eeeests LLM, which integrates the original unidirectional FFN with a bidirectionally fine-tuned FFN via unsupervised contrastive learning. This MoE-based approach enhances embedding performance while preserv- ing robust generation. Extensive experiments across diverse datasets and model scales vali- date our attention dependence metric and demon- strate UBMoE-LLM's superior generative qual- ity and reduced hallucination. Code is available https: htt///itttt tht.t.........g//g///gggg..... com/hetie yonghua ubmoe 11m.
Previous studies have explored two main methods to en- hance the bidirectional modeling capabilities of unidirec- tional language models:
(1) Transforming a unidirectional decoder-only language model into a bidirectional encoder model This method initializes the weights using a unidirectional language model and enables bidirectional attention. As shown in the Bidirec- tion in Figure 1, the tokens in the model self-attention matrix are visible to each other. Through contrastive learning (Gao et al., 2022; BehnamGhader et al., 2024; Muennighoft' et al., 2024). this approach maximizes the similarity between the representations of positive samples while minimizing the similarity with negative sample representations, thereby enhancing the model's ability to contextually model bidi- rectional information. Behamghader al. (BehnamGhader et al., 2024) and Li et al. (Li & Li, 2024b) use this method to turn a unidirectional decoder-only language model into a strong bidirectional encoder model. This method can make the model have better text understanding ability, but will bring disastrous effects on the generation tasks (Muen- nighoff al., 2024).
1. Introduction
Creating model capable of both generation and text em- bedding has been long-term goal in the field of natural
'School of Artificial intelligence, Wuhan University This work was conducted during Yonghua Hei's internship the School of Artificial Intelligence, Wuhan University. "School of Computer Science, Wuhan University 'School of Information Management, Wuhan University Department of Computer Science and Engineer- ing, Shanghai Jiao Tong University "Xiaomi, Beijing, China. Cor- respondence to: Lefei Zhang <zzhangleeei@ whu.edu.en?>, Ping Wang <wangping @ whu.edu>.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
(2) Constructing generative models with local bi- directional modeling In the instruction fine-tuning phase,

---

A Uni-Bi- lii-ei-Directional Mixture-of-Expeee Method For Bidirectional Finc-tuning
Causal
Prefix-1
Prefix-H
Bidirection
Instruction
Question
Answer
Figure 1. The image depicts three different attention mask mechanisms: unidirectional attention, prefix attention, and bidirectional attention, prefix attention, bidirectional attention used for the instruction part, while unidirectional attention is maintained the other parts.
local bi-directional attention mechanism is introduced, as shown in Figure 1. There two main implementations, Prefix-I (Rafailov al., 2023) and Prefix-H (Devlin al., 2019; He et al., 2023). Prefix-l uses bi-directional attention in the instruction part and keeps it unidirectional in the rest of the part, while Preifx-H -H enables bi-directional attention for all the history conversations, and keeps the unidirection- ality and calculates the loss only for the reply. However, this approach still has some drawbacks (Muennighoff al., 2024). Prefix-I and Prefix-H maintain unidirectionality in the response part and only compute loss in the response part. Since in the training phase, some of the tokens can see some of the later tokens, which makes it possible to increase the attention to the later tokens in the optimization process to optimize the representation of the hidden state, and this process may generate subsequent dependence. In the gener- ation phase, the subsequent dependence it generates affects the same tokens, which makes these tokens that should be modeled unidirectionally generate an inappropriate depen- dence distribution.
enabling bi-directional attention in the training phase, from comparing the effects of different model scales, evaluation datasets, token lengths, and training methods on subsequent dependence. Through extensive experiments, we verify the correlation between subsequent attention dependence and the generation ability of the model. Besides, we find that the FFN layer is least affected bidirectional fine-tuning.
Based this finding, Directional Mixture-of-Expert Large Language Model, UBMoE-LLM which can reduce the damage of the sub- sequent dependence on generation performance while en- hancing the model's performance in embeddings. We first initialize the weights using a model fine-tuned with instruc- tions, then enable bidirectional attention and use supervised contrastive learning to enhance the model's word embed- ding capability, updating only the FFN parameters during this process. We use extensive comparative experiments to demonstrate the rationality of this approach. Next, we paral- lel the FFN layer of the word embedding fine-tuned model with the FFN layer of the original instruction fine-tuned model and use a gating mechanism to select the parameters activated for each token, constructing mixture of experts model with two experts. Finally, we fine-tune the param- eters of the gating layer using small amount of data to improve the balance and effectiveness of expert selection.
propose novel Uni-Bi-
Recent work has shown that both methods result varying degrees of degradation in generation performance (Muen- nighoff et al., 2024). However, the reasons for this degra- dation remain unexplained. An intuitive explanation is that this degradation is caused by subsequent dependence during the training phase (Wang et al., 2022). To test this interpreta- tion, we propose an attention dependence based explanation method which considers the attention weight of each token on other tokens as a dependence and constructs average preceding/subsequent dependence measures at the hierarchi- cal and global levels. Previous research has shown that all generative tasks can be reduced to generating dialogues fol- lowing user instructions (Raffel et al., 2023; Du et al., 2022); therefore, we use instruction texts to evaluate such depen- dence. We demonstrate that this effect is brought about by
To evaluate the effectiveness of our method across different scales of language models, we used four different scales of Qwen 1-5-hha (Bai et al., 2023) models ranging from 0.5B to 7B the backbone models validate our method's effectiveness. First, we experimentally verified the effec- tiveness of fine-tuning only the FFN layer, which achieved performance comparable to fine-tuning all parameters on ten text similarity (STS) tasks in MTEB (Muennighoff' et al., 2022), consistently demonstrating superior performance. Next, we combined the instruction-tunal model and the
2

---

A Aii-Bi-Directtonal Mixture-of-Expee Expprr Method For Bidirectional Finc-tuning
embedding-ture model construct UBMoE-LLM and fine-tuned the gate layer. We evaluated UBMoF-LLMM using multiple generative criteria, and the results indicated that this approach enhances the model's 's embedding performance while maintaining robust generative performance. We also tested UBMoE-LLM on the TruthfulQA dataset and the results show that UBMoE-LLM incorporating bidirectional attention can effectively resist the hallucinatory problem.
1 Causal
60
Prefix-I
1 Prefix-H
50
2. Related Work
40
Enhanced text embedding performance for LLM Ini- tially, bi-directional pre-trained language models are a ro- bust choice for text embedding tasks (Devlin et al., 2019; Lan al.,, 2020). Recent research has shown that large unidi- rectional language models can also be transformed into pow- erful encoders for embedding tasks (Li & Li, 2023). How- ever, although large causal language models show strong performance on multi tasks (Touvron et al., 2023a;b; Jiang et al., 2023; Bai et al., 2023; Zeng et al., 2023; Brown et al., 2020), but their preference in text embedding has been limited due to its architectural (BehnamGhader al., 2024). Recent research suggests that causal self-attention mechanism is the main reason, with its limitation of models to acquire information from the entire sequence. For this, its text embedding ability can be effectively enhanced by simply enabling bidirectional attention and using contrast learning for training (BehnamGhader et al., 2024; Li & Li, 2024b).
Natural Instructions Dureader
DOQA
Figure 2. Preliminary experiments results.
tras, 2021). These earlier studies analyzed different tasks using different models due to the lack of powerful sin- gle language model. Recent research indicates that various NL.P tasks can unified as instruction followiin tasks (Du et al., 2022), and large language models have become the preferred choice for multiple tasks. This makes it possible to analyze the interpretability of attention weights from unified model and perspective
3. What Causes Generative Performance Decline?
Language model with both embedding and generation capabilities Initially, the encoder-decoder model was a natural choice to unify the generation and embedding tasks (Vaswani al., 2023; Lewis et al., 2019; Raffel et al., 2023). But was replaced by decoder-only unidirectional language models and encoder-only bidirectional language models for generation and embedding tasks (Lan et al., 2020). Recent research has shown that how to unify uni- directional and bidirectional capabilities is key to framing such a unified model (Muennighoff et al., 2024). But the severe generative performance loss problem associated with bi-directional training makes this unification limited. So, discussion of the causes of this damage necessary for the construction of such a unified model.
3.1. Preliminaries
Previous work (Ding et al., 2024) has demonstrated that overcoming the bidirectional attention pure decoder used for text-embedding tasks has several advantages. We present the performance effects of recent work on bidirectional attention in Appendix A. Recent work (BehnamGhader al., 2024) has shown that the supplement of bidirectional attention will lead to the degradation of model generation ability. We conducted preliminary experiments to test the impact of existing bidirectional modeling enhancement efforts
Generation performance. We tested the generation task effects of Cusual, Prefix-l and Prefix-H H methods on the Natural Instructions (Mishra et al., 2021), DOQA (Campos et al., 2020) and Dureader (Contributors, 2021) datasets. Prefix-I and Prefix-H are the bidirectional attention enhance- ment methods mentioned above. Cusual represent unidirec- tional attention calculations. We add a detailed experimental setup for pilot analysis. We set the batch size to 1, loss accu- mulation to 32, learning rate to 0.00003, and others to follow the default Settings of the transformer library (Wolf et al., 2020). We can see from the Figure that from Cuaual to
Interpreting language models using attention weights Based on the self-attention mechanism. language models calculate attention weights through a dot product mecha- nism, allowing each token observe other tokens with varying weights. Previous research has suggested that the interpretability of attention weights may vary significantly across different NLP tasks, and there are critical views on whether attention weights truly provide insights into model predictions (Vashishth et al., 2019; Chrysostomou & Ale-
3

---

Uni-Bi Bi-re---ittti Mixture-of-Expeee Method For Bidirectional Fine-tuning
Table The result of fine-tuning Qwen 1wenl..5-0.5B .5-005B and 1wen1.5-1..8 with the DOQA dataset. "Full test dataset" to evaluate, "Wo no answer" mean no unanswerable questions in the test set.
all office set
Model
Method
Bleu-1
Bleu-2
Bleu-3
Bleu-4
Dist-1
Dist-2
Dist-3
F1
Rouge-L
Full test dataset
Qwen .5-0.5B
causal
40.24
34.29
31.96
30.73
8.46
34.06
49.67
41.03
39.87
Qwen 1.500.5
Prefix-I
37.11
30.79
28.33
27.04
8.44
32.89
47.85
38.80
37.55
Qwen 1,-11.8B
causal
44.39
39.20
37.16
36.11
9.40
37.78
54.80
44.53
43.43
Qwen ..511.8B
Prefix-I
41.89
36.24
34.02
32.83
9.30
36.95
53.72
43.13
41.94
W/o no answer
Owen1.5-0.55
causal
32.62
26.07
23.82
22.84
10.14
39.27
56.00
28.02
26.36
Qwen 1.500.5B
Prefix-I
28.95
22.26
20.01
19.06
9.73
36.39
51.43
25.75
24.05
Qwenl.5-1.8
causal
37.33
31.80
29.85
29.01
10.99
42.76
60.86
33.86
32.38
Qwenl.5-1.8P 1.5-1.8B
Prefix-I
36.85
31.05
29.02
28.14
10.76
42.04
59.95
32.61
31.10
Table 2. Results of mixed training experiments on DOQA datasets
Train Method
Gnenrate Method
F1
Rouge-1
Blcu-1
Blcu-2
Blcu-3
Blcu-4
Dist-1
Dist-2
Dist-3
Qwenl.5-0.5B
-rusxl+Prefix-I
causal
37.20
36.41
30.59
27.33
25.32
24.23
12.68
51.93
72.47
P
Prefix-I
33.10
32.40
20.12
17.62
15.87
14.95
14.33
57.09
76.87
causal+causal
causal
36.58
35.82
30.28
26.98
25.00
23.93
12.57
52.40
73.18
Qwen1.5-1.88
causal+Prefix-I
causal Prefix-I
39.12
38.45
34.07
31.22
29.38
28.34
13.16
54.85
76.67
Prusxl+Prefix-I
37.71
37.08
30.64
27.89
26.06
25.05
13.39
55.42
75.92
causal+causal
causal
38.85
38. 18
34.35
31.52
29.71
28.69
12.92
54.83
76.91
Qwen 1.BB
causal+Prefix-I
causal
40.27
39.81
19.06
18.17
17.00
16.32
15.94
64.60
86.36
c causal+causal
Prefix-I
35.60
35.27
9.32
9.01
8.24
7.81
16.90
69.16
87.46
causal
40.70
40.34
19.28
18.58
17.49
16.87
16.10
64.59
85.59
Prefix-H, the degree of bidirectional attention of the model increasing. We show the experimental results in Figure 2. The experiment results on all tested datasets show that from Cuaual to Prefx-H, the model generation performance de- creases with the increasing degree of bidirectional attention.
resulted in slight increase in unidirectional generation performance, indicating some bidirectional advantage in understanding long text.
3.2. Average Preceding and Subsequent Dependence
To analyze the reason for the decline of productive ability caused by bidirectional attention, we propose an average subsequent and preceding dependence method that uses attention dependence to explain this phenomenon.
Overfitting. As show in Table 1. We used the DOQA dataset to fine-tune Qwen 1.5-0.5b and 1.5B, and we com- pared the effects of using Prefix-I with causal. The results show that on the DOQA data set, Prefix-I produces obvious overfitting and overlearns the unanswerable the train data set.
The specific explanation is shown in Figure 3. We first in- troduce the attention dependence of single layer. For the attention weight is the key, Q is the query, b is head number and / is token sequence length) of the k-th layer in the n self-attention layers, we sum to obtain the attention score A E R!xl between each pair tokens. We then average the A
softmax QKT) Rbxixl (where K
Hybrid mask training. We conducted mixed training on the two methods causal and Preifx-I, and the experimental results are shown in Table 2. We were surprised to find that using Prefxl+ in both the 0.5B and 1.8B models

---

A Mixture-of-Exxeee Method For Bidirectional Fine-tuning
Average
Average
n-th Self-Attention Layer
Dpmain
(nh Self- Self-Attentoon Layer
Attention
Weight
Dpbefore
i-th Self-Attention Layer
Average
Average
2-th Self seff-Attention Layer
Dpafter
1-th Self- Self-Attention Layer
Average
Decoder
Average
Attention Weight
Pigure
The calculation process of average before and after dependence.
self-attention layers to obtain the attention weight for the global model. We assume that A., represents the depen- dence of the i-th token on the j-th token, then the average token
We conduct experiments on the average attention depen- dence metric with different training objectives, different token lengths, and different model sizes. The experiment setting can found in Section 5.1. As shown in Table after training with bi-directional contrastive learning, the model's subsequent dependence shows significant increase. Through the discussion of different token lengths and dif- ferent model scales, it can be considered that the source of this subsequent dependence bidirectional training. Addi- tionally, we observes that increasing the model scale and the number of tokens also leads an increase in subsequent dependence. As the subsequent dependence increases, downward trend in performance the MMLU dataset. This trend demonstrated remarkable consistency across the four model scales. This fully reflects the statisti- cal association between subsequent dependence and model generation ability. We conduct the same experiment on LLAMA3-8B, and the results are shown Table 3. This further validates our hypothesis that bidirectional training leads to the increase of subsequent dependence and thus damages the model generation ability.
dependence of the i-th token the preceding Dpheforee and the average dependence of the following - token Dp can be expressed as:
Dohefore
Auj:
j=1
Dpafter
Ai.j-
Dptefore and fpe/ter reflect the degree of dependence of the i-th token the preceding and the following tokens respectively. In addition, it is easy to understand that Aj,i represents the degree of dependence of the i-th token on itself, which we denote by Dpmain.
In order obtain the dependence of the full text on the preceding text, the following text and itself, carry out average operation each dependence indicator for each token:
4. UBMOE-LLM
Through pre-experiments and attention-dependence analy- sis, we draw the following conclusions: the performance degradation of text generation task caused by the join of bidirectional attention is mainly caused by the model's sub- sequent dependence and the FFN layer is least affected by the subsequent dependence during the bidirectional atten- tion training process. Therefore, a novel Uni-B Hii-BB-diee direccionnl Mixture Expert Large Language Model, UBMoE-LIA is proposed optimize the FFN layer during the bidirec- tional training of the unidirectional generative model. The main architecture of our approach is shown in Figure 4. It joins the advantage of bidirectional attention on the basis of
Dpindicator
Dpindicator
(2)
indicator E {main, fore,, after).
worth noting that = when indicator main while I' - when indicator is fore or after. Because the calculating process of the Dphefore Dpefter will not get fore or ftefter which can be easy to understand
by Figure
5

---

A Uni-Bi-Directional Mixture-of-Exper Expeert Method For Bidirectional Finc-tuning
Table 3. Dependence of four different scales of models in 0-2048 token lengths. DP-M, DP-B, and DP-A indicate Dpmain Dp fore, and
Dpafter respectively. The following tables take the same abbreviation. Uni and Bi indicate unidirectional and bidirectional respectively.
0-256
256-512
512-1024
1024-2048
Model
Attention
MMLU
DP-M
DP-B
DP-A
DP-M
DP-B
DP-A
DP-M
DP-B
DP-A
DP-M
DP-B
DP-A
Uni
4.83
63,48
31.69
3.48
59.57
36,95
3.01
56.62
40.37
2.61
51.45
45,94
32.5
Qwen1.5-0.5B-CMl
Bi
4.21
60.11
35.68
2.78
56.73
40.49
2.35
48.09
49.93
1.98
48.09
49.93
25.8
Uni
4.44
67.23
28.33
3.21
64.15
32.64
2.82
61.51
35.68
2.47
55.32
44.21
44.3
Owen1..5-1.8B-
Bi
4.27
60,97
34.75
2.80
58.93
38.27
2.37
56.32
41.11
1.94
50.51
47.55
25.52
Uni
4.28
68.3
27.43
3.30
66.66
30.04
2,93
65,16
31.9
2.68
61.62
35.69
54.2
Qwen1.5-4B-Chat
Bi
3.60
64.28
32.12
2.64
63.56
33,79
2.40
62,56
35.04
2.18
59.17
38.66
50,93
Uni
4.48
71.10
24.42
3.30
69.34
27.36
2.95
67.81
29.24
2.80
63.65
33.55
60.2
Qwen1.5-7B-CII
Bi
4.16
67.31
28.53
2.84
66.20
30,95
2.54
65.14
32.33
2.33
61.32
36.25
58.12
Uni
3.29
76.81
19.89
2.60
72.49
24.94
2.13
65.55
32.31
1.50
61.58
36.91
68.4
Llama3-8B
Bi
3.05
75.42
21.53
2.51
70.31
27.18
2.02
61.42
36.56
1.48
56.48
41.72
65.4
maintaining the generative ability of unidirectional models, which can resist the hallucination problem faced by large models.
probability of the model predicting that the i sample belongs to the e category.
The gating regularization loss Letting is to ensure uniform use of all experts. It uses negative entropy encourage uniform distribution of the gated network. It is specifically defined as:
4.1. Uni-bi-directional Mixture-of-expert FFN Layer
To combine the context understanding ability of bidirec- tional attention with the text generation ability of unidirec- tional attention, we decided adopt the Mixture of Experts (MoE) method to integrate bidirectional attention FFN layer into the unidirectional generative model. Specifically, We define the bidirectional attention FFN layer as bi-directional embedding expert expert and the FPN layer of the back- bone model is defined as uni-direction generation expert exp
N
M
1 N
LGating
gig log log(9,)).
(4)
where M is the number of experts and Pij" is the probability that the i th sample will have the j th expert selected by the gating network.
As shown in Figure 4 we implement the allocation of tokens through gate control layer, activate only one expert time. Since the gate control layer does not get any training, we use a small amount of data to train the gate control layer while freezing other parameters. Following Jiang et al. (Jiang et al., 2023), we use cross entropy loss and gate control loss L LCE + >LGGating to jointly train our model. Lev is the cross entropy loss and Leating is the gating regularization loss. À is a hyperparameter that adjusts the weight between the cross-entropy loss and the gated regularization loss.
4.2. Contrastive Learning based Bidirectional Expert Training
We use contrastive learning to train the model to obtain the FFN Layer containing the prior knowledge of bidirectional attention. For the input sequence x = {-1, .t), we simultancously input with its similar positive cases t and different negative cases x= After encoding and pooling of the model, we can obtain the embedding representation of the input sequence with its positive and negative cases h, h+ and h- Since the model is the decoder-only model, we choose the vector corresponding to the token at the end of sentence ([EOS] token) position as the pooling operation. After that, we followed Gao et al. (Gao et al., 2021) comparative training objective for training:
The Cross entropy loss is used to supervise the difference between the output of the learning model and the actual label. For a given imput and target label y. the cross-entropy loss ToE is defined as:
LCE N ΣΣΣN yhe log(Ptc);
(3)
loss - loo himm + oim(
him(
(5)
Σ N Σ)=
where N is the number of samples in the batch, C' is the number of categories, Vic is the actual label (one-hot hot coding) of the sample belonging to the e category, and Pic is the
where F is a temperature hyperparameter and sim(h), h2)
4The |all By contrast learning
is the cosine similarity
6

---

A Uni-Bi-Directional Mixture-of-Exper -xxpert Method For Bidirectional Finc-tuning
Dppertrr
Dpmain
Geseraiin Chathiil
Model Scale
Self-Artention Laver
Bidirectional
Self--Attetiin Layer
Unidirectional
Dpbefore
Add Layer Norm
Add Layer Norm
Attention Dependence
PFN,
Router
Add & Layer Norm
FFN,
FFN,
FFF
Grneattnn Cypabilty
Add Laver Norm
Embeelk @
Altentionn
Alle
Contrustive Learning based Bidirectional Expert Training
Uni-bi-directiona Mixture- of-expert FFN Layer
Viebility Aftrr Tokes
Mudel Layars
Figure 4. The model architecture and design ideas of UBMOF-LLM.
with positive and negative examples, the model can better grasp the bidirectional understanding ability of text. It is worth noting that the training process freezes the weights of other modules and trains only the FFN layer of the model. Therefore, we can obtain FFN layers based on bidirectional attention.
Al models on ambiguity resolution tasks (Sakaguchi et al., 2019). (3) TruthfulQA: a dataset designed to challenge AI models on their ability to generate truthful and factual responses in a During evaluation, the few-shot setting for MMLU is set to 5, using accuracy (ace) as the metric; for TruthfulQA, the few-shot setting is set to O, using multiple-choice score (me2) the metrie; for Winogrande, the few-shot setting is set 5, also using accuracy (acc) the metric.
format (Lin et al., 2022).
5. Experimeenss
5.1. Experiments Setup
Datasets. For the training of bidirectional contrast learn- ing, we used the data collected by Li at al. (Li & Li, 2024a) to train our model. There were 480,862 training data in this dataset. Following the settings set by Muennighoff et al. (Muennighoff et al., 2024), we applied the model's default prompt template when training.
Model Scales. To ablate the impact of model scale on dependence, we evaluate models various scales. To mini- mize the influence of other factors, we seleet the Qwen1.5- Chat series models, ranging from 0.5B to 7B scales. These models have identical architecture, tokenizer, and instruc- tion templates.
To test the performance of different fine-tuning layers em- bedding tasks, we conduct experiments on ten text similarity (STS) tasks in MTEB (Muennighoff et al., 2022)
Baselines. For the embedding tasks, previous work has demonstrated that enabling bidirectional attention in uni- directional language models and training them using con- trastive learning methods can effectively improve the embed- ding capability of the models. We followed this approach to train our model. We conducted the following compar- ative experiments: (1) ATT/FTN/EMB: updating only the parameters of the self-attention layer, FFN layer, or embed- ding layer, (2) EMB+ATT//FN: updating the embedding layer while also updating the self-attention layer or FFN layer, (3) FFN+ATT: updating only the FFN layer and the self-attention layer, consistent with the parameter update settings in LLM2VEC (BehnamGhader et al., 2024), (4) ALL: updating all parameters.
For the evaluation of subsequent dependence and the train- ing for gate layer, we use ulu-v2-sff-mixture (Ivison et al., 2023), which a multitask instruction dataset with a wide text interval, with 326.154 training data. We split it into four subsets based the number of tokens recorded in the historical conversation, in other to dissolve the effect of to- ken length on subsequent dependence. For the evaluation of all language models, we use the officially provided prompt template to process the input.
To evaluate the generative performance of UBMOE-LLM. we use the following three datasets: (1) MMLU: dataset to evaluate AI language understanding across various subjects using massive multitask framework (Hendrycks et al., 2021). (2) Winogrande: a large-scale dataset focused on common sense reasoning, designed to improve and evaluate
For the dependence evaluation, the uni-bi-directional mixture-of-expee large language model and bidirectional contrastive learning, we use the backbone model as the baseline.

---

Uni-Bi- Di-rei--irectionll Bixttuee-of-Exper Method For Bidirectional Fine-tuning
Table Experimental results of word embedding ablation. A77, FFN, and EMB indicate freezing other parameters for bidirectional
modeling training on the Attention Layer, FFN Layer, and Embedding Layer respectively. The following tables take the same abbreviation. ALL Indicates the full- parameter bidirectional modeling trained model.
Method
BIOSSES
SICK-R
STS12
STSI3
STS14 STSIS
STSI6
STS17
STS22
STSB
Average
Qwen1 Liven1.5-0.551 15-...5--hhhh
Base
31.56
53.56
24.96
35.86
22.54
21.65
52.30
51.50
12.91
42.76
34.96
FFN
54.01
69.73
72.41
68.16
65.86
75.71
70.89
77.38
42.82
72.61
66.96
ATT
55.07
68.30
70.5
73.67
68.51
77.41
73.56
79.19
43.57
73.15
68.29
EMB
27.77
54.32
34.67
30.18
24.49
23.09
52.19
59.94
20.52
48.02
37.52
PEN+ATT
49.17
66.73
71.85
67.08
65.09
75.82
72.08
76.42
48.59
73.49
66.63
PEN+EMB
58.38
69.89
72.69
71.18
68.86
77.05
73.04
77.62
41.44
73.61
68.38
ATT+EMB
58,32
69.02
70.43
71.55
67.52
76.20
74.27
79.02
41.66
73.55
68.15
ALL
51.98
68.10
72.06
68.38
65.52
74.66
72.38
75.05
49.19
71.98
66.93
Qwen -.8.L..-------CII -hhat
Base
43.87
56.82
48.15
65.80
52.57
62.08
63.35
72.26
33.75
56.84
55.55
FFN
64.95
67.45
73.85
71.61
70.11
78.25
73.33
77.48
48.59
75.50
70.11
ATT
58,42
66.24
72.09
74.28
70.97
77.73
72.67
75.69
47.11
75.15
69.04
EMB
54,90
71.66
63.23
65.82
61.56
72.77
66.72
77.57
23.74
70.71
62.87
FEN+ATT
63.99
67.03
72.69
73.39
71.08
78.03
73.03
76.32
46.64
75.26
69.75
FFN+EMB
66.3
67.6
72.58
72.55
70.57
78.11
73.13
76.82
75.37
70.37
ATT+EMB
58,28
65.86
72.22
73.41
70.69
77.72
72.55
76.15
45.42
75.70
68.80
ALL
61.80
66.48
72.95
74.33
71.23
77.93
72.87
77.85
48.10
75.71
69.93
Qwen1.5..4e -hat
Base
31.00
53.68
24.04
40.75
24.55
25.59
52.40
49.35
10.29
41.34
35.30
FFN
64.97
63.99
72.30
75.62
71.97
78.08
76.87
69.53
54.93
76.51
70.48
ATT
64.96
64.74
71.56
77.54
73.9
77.85
75.54
73.46
57.13
77.35
71.40
EMB
4.44
35.48
27.78
17.13
15.62
22.27
28.05
37.34
10.63
25.60
22.43
FEN+ATT
69.17
63.73
71.86
76.59
73.17
78.00
76.11
71.66
56.44
76.43
71.32
FEN+EMB
63.03
61.16
51.78
35.94
45.57
61.32
51.36
65.33
41.35
54.72
53.16
ATT+EMB
51.16
60.99
53.53
41.90
45.70
59.55
52.68
63.91
39.45
54.60
52.35
ALL
62.16
59.76
51.43
39.6
47.14
59.48
51.01
67.05
39.53
55.97
53.31
Qwen1.5-7B. ChCt
Base
23.79
56.77
11.94
40.85
22.45
26.43
52.39
43.85
12.21
29.59
32.03
FFN
66.20
64.73
73.68
79.74
75.08
79.04
73.87
73.21
49.36
78.55
71.35
ATT
63.87
65.12
73.52
80.25
75.44
80.26
77.62
73.11
58.24
78.23
72.57
EMB
29.94
53.82
37.90
25.33
26.95
32.22
42.70
48.39
7.42
32.46
33.71
ATN+ATT
65.52
65.17
73.20
79.80
74.61
79.59
77.73
71.45
57.27
77.92
72.23
FFN+EMB
60.69
62.18
51.06
36.70
45.57
60.39
52.45
66.65
32.52
54.62
52.28
ATT+EMB
61.55
60.31
51.87
31.64
43.92
60.25
51.70
68.81
31.00
55.94
51.70
ALL.
52.05
60.04
50.72
36.14
46.03
61.13
52.93
65.97
31.41
55.81
51.23
Model Settings. For bidirectional contrasted learning training, we conduct full fine-tuning for the 0.5B and 1.8B models, and use LoRA (Hu et al., 2021) for parameter ad- justments on the 4B and 7B models. We fix the learning rate 0.00003 but vary the batch size and gradient accu- mulation steps according to the model parameters, ensuring their product remains 32, and apply the same hyperparam- eter settings within the same scale models. For LoRA, set r 16, 32, LoRA dropout to 0.05, and enable reLoRA (Lialin et al., 2023).
the training process, we freeze parameter updates except for those of the gating layer. We trained Qwenl hh-hat with sizes 0.5B, 1.8B, and 4B on a GeForce RTX 4090. For Qwen 1. 55-7B-Chat and Llama3-8B, we train them on NVIDIA L20. For the evaluation of UBMoE-LLM, we use the scripts provided by Language Model Evaluation Harness for evaluation (Gao et al., 2023).
5.2. Analysis
Bidirectional embedding expert The embedding tasks experiment results shown in Table In all sizes of models, although the optimal results of fine-tuning different layers are different, merely fine-tuning the FFN attention layers has achieved robust performance in the text similar-
For the training of UBMoE-LLM, we only sample data with token counts less then 512. The batch size set to 1, gradient accumulation
32, and the learning rate is 0.0003, with training limited 1000 steps. During
8

---

A Uni-Bi-Directional Mixture-of-Expee -Expert Method For Bidirectional Finc-tuning
Table 5. Contrastive learning subsequent dependence ablation.
Trained Module
0-256
256-512
512-1024
1024-2048
Model
DP-M
DP-B
DP-A
DP-M
DP-B
DP-A
DP-M
DP-B
DP-A
DP-M
DP-B
DP-A
FFN
3.86
61.74
34.40
2,58
57.87
39.55
2.22
54.73
43.05
1.92
49.56
48.52
Qwen1.5-0.5B Chat
ATT
4.33
59.28
36.39
2.82
56.44
40.74
2.39
53.49
44.12
2.00
48.53
49.48
FFN
3.82
62.81
33.37
2.65
60.79
36.56
2.34
58.85
38.81
2.04
52.72
45.24
Owen1.5-11.B0 Chat
ATT
4.09
62.49
33.41
2.52
60.68
36.79
2.10
58.41
39.48
1.69
52.98
45.33
FFN
3.64
65.60
30.76
2.71
64.31
32.98
2.43
63.14
33.43
2.19
59.62
38.18
Qwen (weel..5-4B-CC -.-----hh
ATT
3.62
62.52
33.86
2.63
63.09
35.28
2.38
61.17
36.44
2.17
57.91
39.92
FFN
4.05
69.64
26.32
2.87
68.16
28.97
2.57
67.17
30.25
2,41
63.44
34.15
Qwen (weel..5-7B-- 15-7BBhhat
ATT
3.98
65.39
30.63
2.63
64.81
32.56
2.29
63.67
34.04
2.06
59.67
38.29
training on word embeddings, did not undergo instruction alignment, leading to a decline in their generative abilities. Notably, our approach manages to enhance overall gener- ative performance at the 1.8B scale, which suggests the benefits of incorporating bidirectional experts- experts-enhancing the model's 's resistance to hallucinations and improving its language comprehension capabilities and the richness of semantic modeling. More experiment results can found in Appendix B.
Table Experiment results UBMoE-LLM and base model with different model scales on the generation tasks. All results are evaluated by us.
Model Scale
Model
MMLU
Winogrande
Truthfulqa
Average
Qwent.5-Chat
32.5
53,0
43,0
42.8
0,5B
UBMoB-LLLM
28.1
54.0
43.9
42.0
Qwen1.5-Chat
44.3
59.7
40.5
48.2
1.8B
UBMoE-LLM
45.2
58.5
40,9
48.2
Qwen1
54.2
44.5
54.9
4B
UBMoE-LLM
51,9
64.7
47.6
54.7
6. Conclusion
Qwenl.5-Chat
60.2
67.4
53.7
60.4
7B
UBMoE-LLM
59.6
66.0
55.4
60.2
To explore whether bidirectional training of unidirectional models would enhance subsequent dependence, we reinter- preted attention weights as a form of dependency, dividing it into preceding dependency, subsequent dependence, and self/main dependency. We have provided the formulas for calculating these dependencies and tested our hypothesis on four models ranging from 0.5B to 7B - bidirectional training does increase subsequent dependence. Addition- ally, we performed an ablation study on the bidirectional contrastive learning training part, revealing that training only the FFN layers results in robust embedding effects and lower increase in subsequent dependence. We used the fine-tuned FFN layers as bidirectional embedding experts, paired with the unidirectional generative experts of the main model. By adding gating mechanism, we constructed a hybrid expert language model that effectively utilizes the illusion resistance brought by bidirectionality and the ro- bust generative capabilities unidirectionality. This model showed consistent improvements on the realness benchmark dataset TruthfulQA.
ity computation tasks across all scales. We evaluated the dependence distribution of models trained via bidirectional contrastive learning for these two components and token evaluations as show in Table 5. The results indicate that solely fine-tuning the FFN consistently yielded the lowest subsequent dependence across various model scales. It is shown that only fine-tuning the FFN has the lowest impact on the dependence of the model. This is primarily because, in the initial layers where the attention layer is not updated, the attention distribution during the bidirectional training phase is exactly the same as the unidirectional generation. In this context, the FFN layer mainly learns how to process hidden states that contain backward token information un- der the same attention distribution. Updating the ATT layer, however, would disrupt this consistency in distribution. This implicit separation of the two modeling capabilities reduces the performance degradation of the FFN in unidirectional modeling during training.
Acknowledgments
UBMoE-LLM
The generative performance of UBMoE- LLM is illustrated in Table 6, where our method consistently achieves improvements on the TruthfulQA dataset while maintaining robust generative capabilities. At the same time, due to only learning token allocation capabilities dur- ing the training phase, degradation in some generation tasks inevitable. This is because the bidirectional experts, after
This work was supported by the National Natural Science Foundation of China (No. 62306216), the Technology In- novation Program of Hubei Province (No. 2024BAB043) and the National Social Science Fund of China (No. 24&ZD186).
9

---

A Birrei--irectionll Mixture-off-xpelr Method For Bidirectional Finc-tuning
Impact Statement
Dong, L.,, Yang, N., Wang, W., Wei, F., Liu, X., Wang. Y.,
Gao, J., Zhou, M., and Hon, H.-W. Unified language model pre-training for natural language understanding and generation, 2019.
This paper presents work whose goal to advance the field of Machine Learning. There are many potential societal consequences of our work, none which feel must be specifically highlighted here.
Du, Z., Qian, Y., Liu, X., Ding, M., Qiu, J., Yang. Z., and Tang. J. Glm: General language model pretraining with autoregressive blank infilling. 2022.
References
Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K.,, Deng, X., Fan,
Gao, L., Tow, J., Abbasi, B., Biderman, S., Black, S., DiPofi,
Y., Ge, W., Han, Y., Huang, F., Hui, B., Ji, L., Li, M., Lin, J., Lin, R., Liu, D.,. Liu, G., Lu, C.. Lu, K. Ma, J., Men, R., Ren, X., Ren, X., Tan, C., Tan, S., Tu, J., Wang, P., Wang. S., Wang, W., Wu, S., Xu, B., Xu, J., Yang,
A., Foster, C.,, Golding, L.,, Hsu, J., Le Noac' h, A., Li,
H., McDonell, K., Muennighoff, N., Ociepa, C., Phang,
J., Reynolds, L.,, Schoelkopf, H., Skowron, A., Sutawika,
L., Tang, E., Thite, A., Wang, B., Wang, K., and Zou, A. A framework for few-shot language model evaluation,
A., Yang. H., Yang, J., Yang, S., Yao, Y., Yu, B., Yuan, H., Yuan, Z., Zhang, J., Zhang, X., Zhang, Y., Zhang, Z., Zhou, C., Zhou, J., Zhou, X., and Zhu, T. Qwen technical report, 2023.
12 2023. URL https: zenodo. org/ records/ 10256836.
Gao, T., Yao, X., and Chen, D. SimCSE: Simple con- trastive learning of sentence embeddings. In Moens, M.-F, Huang, X., Specia, L., and Yih, S. W.tt (eds.), Proceedings of the 2021 Conference Empirical Meth- ods in Natural Language Processing, pp. 1894-6910, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 1211663//12222..mmlpp aclanthology: wwwpogg......r.eeee. 2ttgg//22111 www.ppmaama.aaannnnmmmmmmmmm-mam 552.
Bao, H., Dong, L.,, Wei, F., Wang, W., Yang, N., Liu, X., Wang, Y., Piao, S., Gao, J., Zhou, M., and Hon, H.-W. Unilmv2: Pseudo-masked language models for unified language model pre-training, 2020.
BehnamGhader, P. Adlakha, V., Mosbach, M., Bahdanau, D., Chapados, N., and Reddy, S. Llm2vec: Large lan- guage models are secretly powerful text encoders, 2024.
URL https://
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., -erbert- A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners, 2020.
Gao, T., Yao, X., and Chen, D. Simese: Simple contrastive learning of sentence embeddings, 2022.
He, Z., Sun, T., Tang. Q.. Wang, K., Huang, X., and Qiu, X. DiffusionBERR: Improving generative masked language models with diffusion models. In Rogers, A., Boyd- Graber, J., and Okazaki, N. (eds.), Proceedings of the 61st Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: LLnng Papers), pp. 4521- 4534, Toronto, Canada, July 2023. Association for Com- putational Linguistics. doi: 10.188653/1/22233.cd 248. URL htwps: http:: httt::ss/acclaathhllogoo wtt.oogg...rg...3//////// acl-long. 248.
Campos, J. A., Otegi, A., Soroa, A., Deriu, J., Cieliebak, M., and Agirre, E. via conversational qa. arXiv preprint arXiv:2005.01328, 2020.
domain-specific faqs
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song. D., and Steinhardt, Mcasuring massive multitask language understanding, 2021.
Chrysostomou, G. and Aletras, N. Improving the faithful- ness of attention based explanations with task-specific information for text classification, 2021.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models, 2021.
Contributors, P. Paddlenlp: An easy-to-use and high performance nlp library. https: //ggtttuuuu migtt.u..com com/ Bod6leePaddle/Paddleee.
2021.
Ivison, H., Wang, Y., Pyatkin, V., Lambert, N., Peters, M.,
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for lan- guage understanding. 2019.
Dasigi, P., Jang, J., Wadden, D., Smith, N. A., Beltagy, I., and Hajishirzi, H. Camels in changing climate: Enhancing Im adaptation with tulu 2, 2023.
Ding, N., Levinboim, T., Wu, J.. Goodman, S., and Soricut, R. Causallm is not optimal for in-context learning, 2024.
Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C.,
Chaplot, D. S., de las Casas, D., Bressand, F., Lengyel,
10

---

A Uni-Bi-Directional Mixture-of-Expee Method For Bidirectional Fine-tuning
G., Lample, G., Saulnier, L.,, Lavaud, L.. R., Lachaux, M.- A., Stock, P., Seao, T. L.,, Lavril, T., Wang, T., Lacroix, T., and Sayed, W. E. Mistral 7b, 2023.
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with unified text-to-text transformer, 2023.
Lan, Z.., Chen, M., Goodman, S., Gimpel, K., Sharma, P., and Soricut, R. Albert: A lite bert for self-supervised learning of language representations, 2020.
Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. Winogrande: An adversarial winograd schema challenge at scale, 2019.
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mo- hamed, A., Levy, O.. Stoyanov, V., and Zettlemoyer, L. Bart: Denoising sequence-to-sequeee pre-training for natural language generation, translation, and comprehen- sion, 2019.
Springer, J. M., Kotha, S., Fried, D., Neubig, G., and Raghu- nathan, A. Repetition improves language model embed- dings, 2024.
Touvron, H.,, Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., and Lam- ple, G. Llama: Open and efficient foundation language models, 2023a.
Li, H., Zhang, Y., Koto, F., Yang, Y., Zhao, H., Gong, Y., Duan, N., and Baldwin, T. Cmmlu: Measuring massive multitask language understanding in chinese, 2023a.
Li, X. and Li, J. Angle-optimized text embeddings. arXiv preprint arXiv:2309. 12871, 2023.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Ferrer, C. C.,, Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn, A., Hosseini, S., Hou, R., Inan, H., Kardas, M., Kerkez, V., Khabsa, M., Kloumann, I., Korenev, A., Koura, P. S.. Lachaux, M.-A., Lavril, T., Lee, J., Liskovich, D.,, Lu, Y., Mao, Y., Martinet, X., Mihaylov, T., Mishra, P., Molybog, I., Nie, Y., Poulton, A., Reizenstein, J., Rungta, R., Saladi, K., Schelten, A., Silva, R., Smith, E. M., Subramanian, R., Tan, X. E., Tang. B., Taylor, R., Williams, A., Kuan, J., Xu, P., Yan, Z., Zarov, I., Zhang, Y., Fan, A., Kambadur, M., Narang, S., Rodriguez, A., Stojnic, R., Edunov, S., and Scialom, T. Llama 2: Open foundation and fine-tuned chat models, 2023b.
Li, X. and Li, J. Angle-optimized text embeddings, 2024a.
Li, X. and Li, J. Bellm: Backward dependency enhanced large language model for sentence embeddings, 2024b.
Li, Z., Zhang, S., Zhao, H., Yang, Y., and Yang, D. Batgpt: A bidirectional autoregessive talker from generative pre- trained transformer. arXiv preprint arXiv:2307.00360, 2023b.
Lialin, V., Shivagunde, N., Muckatira, S., and Rumshisky, A. Relora: High-rank training through low-rank updates, 2023.
Lin, S., Hilton, J., and Evans, O. Truthfulqa: Mcasuring how models mimic human falsehoods, 2022.
Vashishth, S., Upadhyay, S., Tomar, S., and Faruqui, M. Attention interpretability across nlp tasks, 2019.
Mishra, S., Khashabi, D., Baral, C., and Hajishirzi, H. Nat- ural instructions: Benchmarking generalization tasks from natural language instructions. arXiv preprint arXiv:2104.08873, 2021.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need, 2023.
Muennighoff, N., Tazi, N., Magne, L., and Reimers, N. Mteb: Massive text embedding benchmark. arXiv preprint 17Xiv:2210.07316. 2022. doi: 10.48550/ ARXIV.2210.07316. URL abs/2210. 07316.
Wang. T., Roberts, A., Hesslow, D., Scao, T. L., Chung, H. W., Beltagy, I., Launay, J., and Raffel, C. What lan- guage model architecture and pretraining objective work best for zero-shot generalization?, 2022.
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., Davison, J., Shleifer, S., von Platen, P., Ma, C., Jer- nite, Y., Plu, J., Xu, C., Scao, T. L.., Gugger, S., Drame, M., Lhoest, Q., and Rush, A. M. Transformers: State- of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natu- ral Language Processing: System Demonstrations, pp-
Muennighoff, N., Su, H., Wang, L., Yang, N., Wei, F., Yu, T., Singh, A., and Kiela, D. Generative representational instruction tuning, 2024.
Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., and Finn, C. Direct preference optimization: Your language model is secretly reward model, 2023.

---

A Uni-Bi-Directionn Mixture-of-Expee ofpxrr Method For Bidirectional Finc-tuning
38-45, Online, October 2020, Association for Compu- tational Linguistics. URL https: / www. aalawwlewe web.
org/ hrggaanhhology/200 /2020 emnlp demos. 6.
Zeng, A., Liu, X., Du, Z., Wang, Z., Lai, H., Ding, M., Yang, Z., Xu, Y., Zheng, W., Xia, X., Tam, W. L., Ma, Z., Xue, Y., Zhai, J., Chen, W., Zhang, P., Dong, Y., and Tang. Glm-1300 An open bilingual pre-trained model, 2023.
12

---

lii-ei-Directional Mixture-of-Expee Expxert Method For Bidirectional Finc-tuning
A. Effect of bidirectional attention model
Table 7, we follow et (Li al., 2023b) 's work to train a bidirectional attention model BatGPT and test on CMMLU (Li al., 2023a) dataset. We can find that BatGPT with bi-directional attention has excellent performance, fully demonstrating the benefits of bi-directional attention.
Table 7. Performance comparison of BATGPT and other Chinese-oriented large language models on the CMMLU benchmark. Results are presented average accuracy within each categories based on five-shot experiment setting.
Model
STEM
Humanities
Social Science
Other
China-specific
Average
Random
25.00
25.00
25.00
25.00
25.00
25.00
27.23
30.41
28.84
32.56
28.68
29.57
Maanese-LLLaM--1333
27.12
33.18
34.87
35.10
32.97
32.63
Chinese-GI.M-II
25.49
27.05
27.42
29.21
28.05
27.26
Bhinese-LLaMA-78
25.79
27.45
26.35
26.06
25.45
26.36
ChatGLMM6B
32.35
39.22
39.65
38.62
37.70
37.48
BATGPT
33.49
35.38
36.31
42.14
37.00
36.72
B. Further Study
Performance on more base models. We analyzed the relationship between generation performance and attention-basee forward or subsequent dependence on other models. The experimental results are shown Table As the level of subsequent dependence increased, the generation performance of the model declined. In contrast, our method reduced the subsequent dependence introduced by incorporating bidirectional experts while improving generation performance. This indicates correlation between the distribution of attention weights and the model's generation performance.
Model
DPB
DPA
MMLU
Qwen2.5-0.5B Instruct
55.68
39.75
46.9
UBMoE-LLM
54.68
38.91
43.85
FNN-Bi
54.40
41.40
35.54
Dween.55-1.B-IInstme
58.27
36.57
60.21
UBMoE-LLM
58.08
36.80
59.41
FNN-Bi
57.00
38.13
54.35
Dwven2.5-7B-Instruus
60.94
35.07
73.86
UBMoE-LLM
60.37
35.61
70.86
FNN-Bi
46.17
49.05
21.28
Table Dependence and Performance Qwen 2.5.
Besides, we add MNTP-LLM as baseline, which is trained using the Wikipedia dataset through the MNTP method. As shown in Table 9, the experimental results demonstrate that UBMoE-LLM still exhibits strong performance. we also test the generation ability on models beyond 7B parameters. As shown Table 10, the experimental results of the 14B model remain consistent with the previous conclusions.
Model
MMLU
Winogrande
Truthfulqa
Avg
Qwwen.5-00.5BImmee
46.90
55.60
41.86
48.12
UBMoE-LLM
43.85
56.94
43.02
47.94
MNTP-LLM
41.33
54.40
41.65
45.79
Qwen2.5-7B-Insttune
73.86
73.60
64.72
70.72
UBMoE-LLM
70.86
75.42
62.65
69.64
MNTP-LLLM
70.06
74.21
62.54
68.93
Table 9. Performance generation tasks.
13

---

A Uni-Bi-Directional Mixture-of-Expeee Method For Bidirectional Finc-tuning
Model
MMLU
Winogrande
Truthfulqa
Avg
Qwen2.5-14B Instruct
79.54
78.74
67.51
75.26
UBMoE-LLM
78.44
77.83
68.01
74.76
Table 10. Performance on 14B model.
Comparison of computational overhead. UBMoE activates only one expert for each token, adding computational overhead during inference solely for token routing. Compared the computational cost of the causal language model itself, this additional overhead is minimal. As shown in Table 11, our approach does not result in significant additional computational overhead.
Model
GFLOPs
Frren..5-0.5B-Insttuu
505.82
UBMoE-LLM (0.5B)
505.86
Qwen2.5-1.5B-img
1,580.62
UBMoE-LLM (1.5B)
1,580.70
Qwen2.5-7B-Inntrruet
7,239.97
UBMoE-LLM (7B)
7,240.18
Table 11. Comparison of computational overhead.
Performance on reasoning tasks. To evaluate the reasoning ability of UBMOE-LLM, we evaluate the performance of UBMoE-LLM a physies problem benchmark and mathematics problem benchmark. As shown in Table 12, the experimental results indicate that UBMoE-LA -LLM still demonstrates performance comparable to that of causal language model.
Model
Math
PIQA
Avg
Qwen2.5-0.5B-Inniiie
32.5
70.62
51.56
UBMoE-LLM
32.67
69.85
51.26
1inm2.5-1.5BBInsttrre
41.07
76.22
58.64
UBMoE-LLM
42.68
76.36
59.52
Twee2.5-7B-Innstiee UBMoE-LLM
54.51
79.49
67.00
53.91
79.04
66.48
Table 12. Performance reasoning tasks.
Ablation study of gate control layer. To test the effectiveness gate control layer, we carry out ablation experiments. As shown in Table 13, we conduct experiments on the STS task. The gate control layer effectively enhances the ability to embed tasks.
Model
Method
Avg
Sween.5-00.B--Instt
FNN
81.63
Qwen2.5-0.5B-Inste Instruct
FNN w/o gate
80.14
Owen2.5-1.5B-IIntrrue
FNN
80.15
Owen2.5-1.5B-Inntruect
FNN w/o gate
78.49
Qwen2.5-7B-Insttu
FNN
81.79
Qwen2.5-7B-Instru
FNN w/o gate
80.39
Table Ablation study of gate control layer.
14


Reviews:
[{"review_id": "irLpG5uyiQ", "reviewer": "Reviewer_bS72", "summary": "Due to the unidirectional attention mechanism, current LLMs underperform in embedding tasks. Some studies have modified the unidirectional attention to bidirectional attention in LLMs and fine-tuned them using contrastive learning, resulting in models better suited for embedding tasks. However, this modification compromises the model's generative capabilities, rendering it ineffective for generation tasks.\n\nThe authors propose a two-stage training approach to address this issue. In the **first stage**, a feedforward neural network (FNN_b) suitable for embedding tasks is trained. Specifically, the causal attention is replaced with bidirectional attention, while freezing all model parameters except the FNN, and training embedding tasks using contrastive learning. In the **second stage**, the trained FNN from the first stage is combined with the original FNN, and a gate router is added to form a Mixture of Experts (MoE) model. Then, all parameters except the router are frozen, and auto-regressive training is conducted on generation tasks. This results in a new model that excels in both embedding and generation tasks.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> W1: The new model developed ... computational demands for generation tasks. \n\nUBMoE activates only one expert for each token, adding computational overhead during inference solely for token routing. Compared to the computational cost of the causal language model itself, this additional overhead is minimal. As shown in the table below, our approach does not result in significant additional computational overhead.\n\n| Model                 | GFLOPs   |\n| --------------------- | -------- |\n| Qwen2.5-0.5B-Instruct | 505.82   |\n| UBMoE-LLM (0.5B)      | 505.86   |\n| Qwen2.5-1.5B-Instruct | 1,580.62 |\n| UBMoE-LLM (1.5B)      | 1,580.70 |\n| Qwen2.5-7B-Instruct   | 7,239.97 |\n| UBMoE-LLM (7B)        | 7,240.18 |\n\n> W2:  For tasks with clear distinctions such ... requirement to train a router).\n\nA model with both generation and embedding capabilities can reduce deployment costs by eliminating the need to deploy multiple models.  Meanwhile, the method proposed in this paper can also enhance the model’s resistance to hallucination to some extent by introducing bidirectional experts, which improve the model’s performance against linguistic priors. Some work has reflected the advantages of combining unidirectional  and bidirectional  models \\[1\\]\\[2\\]. Our results at Truthfulqa benchmark in Table 4 also show that the bidirectional module effectively reduces the LLM hallucination.\n\n[1] LLM2Vec: Large Language ModelsAreSecretly Powerful Text Encoders\n[2] BatGPT: A Bidirectional Autoregessive Talker from Generative Pre-trained Transformer\n\n> W3:  In Section 3.1, under \"Preliminaries,\" ...  based on this point is insufficient.\n\nAs shown in Table 1, there is a consistent trend between subsequent dependence and the general capability of the model. This indicates a correlation between subsequent dependence and the model’s general performance, suggesting a potential causal relationship.\n\nBesides, we analyzed the relationship between generation performance and attention-based forward or subsequent dependence. As shown in the table below. As the level of subsequent dependence increased, the generation performance of the model declined. In contrast, our method reduces the subsequent dependence introduced by incorporating bidirectional experts while improving generation performance. This indicates a correlation between the distribution of attention weights and the model's generation performance.\n\n|         Model         |  DPB  |  DPA  | MMLU  |\n| :-------------------: | :---: | :---: | :---: |\n| Qwen2.5-0.5B-Instruct | 55.68 | 39.75 | 46.9  |\n|       UBMoE-LLM       | 54.68 | 38.91 | 43.85 |\n|        FNN-Bi         | 54.40 | 41.40 | 35.54 |\n| Qwen2.5-1.5B-Instruct | 58.27 | 36.57 | 60.21 |\n|       UBMoE-LLM       | 58.08 | 36.80 | 59.41 |\n|        FNN-Bi         | 57.00 | 38.13 | 54.35 |\n|  Qwen2.5-7B-Instruct  | 60.94 | 35.07 | 73.86 |\n|       UBMoE-LLM       | 60.37 | 35.61 | 70.86 |\n|        FNN-Bi         | 46.17 | 49.05 | 21.28 |\n\n> Q1: What is the experimental setup for ... employ bidirectional attention or causal attention?\n\nWe provide detailed experimental settings in Section 5.1. For the evaluation of embedding capability in Table 2, we maintain the same settings as in the training phase and enable bidirectional attention.\n\n> Q2: In the results of Table 3, which layer of ... used during the evaluation?\n\nFollowing the computation method described in Section 3.2, we use the average across all layers. During the calculation, we adopt causal attention to remain consistent with the generation task.\n\n> W4: The experiments seem to lack any comparison with previous state-of-the-art (SOTA) methods.\n\nThanks for your advice. We added MNTP-LLM as a baseline, which is trained using the Wikipedia dataset through the MNTP method. As shown in the table below, the experimental results demonstrate that UBMoE-LLM still exhibits strong performance.\n\n|         Model         | MMLU  | Winogrande | Truthfulqa |  Avg  |\n| :-------------------: | :---: | :--------: | :--------: | :---: |\n| Qwen2.5-0.5B-Instruct | 46.90 |   55.60    |   41.86    | 48.12 |\n|       UBMoE-LLM       | 43.85 |   56.94    |   43.02    | 47.94 |\n|       MNTP-LLM        | 41.33 |   54.40    |   41.65    | 45.79 |\n\n|        Model        | MMLU  | Winogrande | Truthfulqa |  Avg  |\n| :-----------------: | :---: | :--------: | :--------: | :---: |\n| Qwen2.5-7B-Instruct | 73.86 |    73.6    |   64.72    | 70.72 |\n|      UBMoE-LLM      | 70.86 |   75.42    |   62.65    | 69.64 |\n|      MNTP-LLM       | 70.06 |   74.21    |   62.54    | 68.93 |\n\n> Summary\n\nThank you for your helpful advice. We will improve our work based on your suggestions. It is worth noting that the concept of attention dependence proposed by us is highly correlated with the generative ability of the model in experimental verification, and has important significance for the study of unidirectional and bidirectional architecture of the model. We hope our response provides a clearer perspective on our work.", "claims_and_evidence": "1. The article provides a detailed experimental ablation on the effectiveness of fine-tuning layers of LLM (FFN, embedding, attention) for embedding tasks.\n\n2. The article lacks a discussion on the proposition of using MoE (Mixture of Experts) models for both embedding tasks and generation tasks, such as efficiency or performance, which are not addressed (given the inherent router in both tasks, it is deemed unnecessary in my opinion).", "methods_and_evaluation_criteria": "The new model developed using the author's approach has increased computational demands for generation tasks. Moreover, for tasks with clear distinctions such as embedding tasks and generation tasks, it is not particularly necessary to employ a single model to achieve both (unless it is training-free). This is because these two tasks inherently have a natural router (the task category is known in advance), and a more efficient and cost-effective solution can be directly implemented through an if-else branch (eliminating the need for a second training phase, as there is no requirement to train a router).", "theoretical_claims": "In Section 3.1, under \"Preliminaries,\" the concepts of \"Subsequent Dependence\" and \"decline of productive ability\" are only correlated, but causality cannot be inferred. Therefore, the logic of introducing the MoE architecture based on this point is insufficient.", "experimental_designs_or_analyses": "1. What is the experimental setup for downstream task testing in the article? For example, is the model used for the evaluation of embedding tasks taken after the second stage of training or before the second stage of training? Does the embedding task evaluation employ bidirectional attention or causal attention?\n2. In the results of Table 3, which layer of the model was used to test **subsequent dependence**, and was unidirectional attention or bidirectional attention used during the evaluation?\n3. The experiments seem to lack any comparison with previous state-of-the-art (SOTA) methods.", "relation_to_broader_scientific_literature": "None", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "#### Strengths:\n\n- The author's proposal of \"attention dependence to explain this phenomenon\" in Section 3.1 is quite novel, offering a unique perspective for observing the characteristics of attention.\n\n#### Weaknesses:\n\n- In my opinion, the author's motivation is not sufficiently justified. For details, please refer to **Methods and Evaluation Criteria**.", "other_comments_or_suggestions": "None", "questions_for_authors": "None", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QO9anMwHQy", "reviewer": "Reviewer_u1oZ", "summary": "The paper investigates the impact of bidirectional fine-tuning on unidirectional language models. The authors argue that bidirectional attention mechanisms, while enhancing embedding tasks, degrade generative performance. To address this, they integrate unidirectional FNN layers with bidirectional ones trained via unsupervised contrastive learning. Extensive experiments demonstrate the model's ability to improve embedding performance without compromising generative capabilities.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Weakness 1: Limited discussion of potential trade-offs in training complexity.\n\nThanks for the helpful comments. Our training consists of two main parts: bidirectional expert training and UBMoE-LLM model training. For bidirectional expert training, compared to previous bidirectional approaches, we only fine-tune the FNN layers, which significantly reduces training costs. As for UBMoE-LLM training, we train for only 1,000 steps, using just 32 samples per step. The training of UBMoE-LLM (7B) can be completed in just 1 hour on a single H100 GPU.\n\n> Weakness 2: Additional analysis on the interpretability of attention weights could enrich the findings.\n\nThanks for the helpful comments. We analyzed the relationship between generation performance and attention-based forward or subsequent dependence. The experimental results are shown in the table below. As the level of subsequent dependence increased, the generation performance of the model declined. In contrast, our method reduced the subsequent dependence introduced by incorporating bidirectional experts while improving generation performance. This indicates a correlation between the distribution of attention weights and the model's generation performance.\n\n|         Model         |  DPB  |  DPA  | MMLU  |\n| :-------------------: | :---: | :---: | :---: |\n| Qwen2.5-0.5B-Instruct | 55.68 | 39.75 | 46.9  |\n|       UBMoE-LLM       | 54.68 | 38.91 | 43.85 |\n|        FNN-Bi         | 54.40 | 41.40 | 35.54 |\n| Qwen2.5-1.5B-Instruct | 58.27 | 36.57 | 60.21 |\n|       UBMoE-LLM       | 58.08 | 36.80 | 59.41 |\n|        FNN-Bi         | 57.00 | 38.13 | 54.35 |\n|  Qwen2.5-7B-Instruct  | 60.94 | 35.07 | 73.86 |\n|       UBMoE-LLM       | 60.37 | 35.61 | 70.86 |\n|        FNN-Bi         | 46.17 | 49.05 | 21.28 |\n\n> Weakness 3: Clarify the distinction between preceding and subsequent dependencies.\n\nWe provide a detailed explanation of the computation methods for preceding and subsequent dependencies in Section 3.2. As shown in formula 1, preceding dependence represents the average score of attention before token i in the attention layer. Subsequent dependence represents the average score of attention after token i.\n\n> Weakness 4: Include a broader comparison with other hybrid models.\n\nThanks for your advice and we added MNTP-LLM as a baseline, which is trained using the Wikipedia dataset through the MNTP method. \n\n|         Model         | MMLU  | Winogrande | Truthfulqa |  Avg  |\n| :-------------------: | :---: | :--------: | :--------: | :---: |\n| Qwen2.5-0.5B-Instruct | 46.90 |   55.60    |   41.86    | 48.12 |\n|       UBMoE-LLM       | 43.85 |   56.94    |   43.02    | 47.94 |\n|       MNTP-LLM        | 41.33 |   54.40    |   41.65    | 45.79 |\n\n|        Model        | MMLU  | Winogrande | Truthfulqa |  Avg  |\n| :-----------------: | :---: | :--------: | :--------: | :---: |\n| Qwen2.5-7B-Instruct | 73.86 |    73.6    |   64.72    | 70.72 |\n|      UBMoE-LLM      | 70.86 |   75.42    |   62.65    | 69.64 |\n|      MNTP-LLM       | 70.06 |   74.21    |   62.54    | 68.93 |\n\n> Question 1: How does UBMoE-LLM perform on tasks beyond text generation and embedding, such as reasoning tasks?\n\nAs shown in the table below, we evaluated the performance of UBMoE-LLM on a physics problem benchmark and a mathematics problem benchmark. The experimental results indicate that UBMoE-LLM still demonstrates performance comparable to that of a causal language model.\n\n| Model                 | Math  | PIQA  | Avg   |\n| --------------------- | ----- | ----- | ----- |\n| Qwen2.5-0.5B-Instruct | 32.5  | 70.62 | 51.56 |\n| UBMoE-LLM             | 32.67 | 69.85 | 51.26 |\n| Qwen2.5-1.5B-Instruct | 41.07 | 76.22 | 58.64 |\n| UBMoE-LLM             | 42.68 | 76.36 | 59.52 |\n| Qwen2.5-7B-Instruct   | 54.51 | 79.49 | 67    |\n| UBMoE-LLM             | 53.91 | 79.04 | 66.48 |\n\n\n> Question 2: Have you explored the impact of scaling UBMoE-LLM beyond 7B parameters?\n\nAs shown below, we present the experimental results on Qwen2.5-14B-instruct, where UBMoE-LLM maintains performance consistent with that of a causal language model in terms of generation quality.\n\n| Model                | MMLU  | Winogrande | Truthfulqa | Avg   |\n| -------------------- | ----- | ---------- | ---------- | ----- |\n| Qwen2.5-14B-Instruct | 79.54 | 78.74      | 67.51      | 75.26 |\n| UBMoE-LLM            | 78.44 | 77.83      | 68.01      | 74.76 |\n\n> Summary\n\nThank you for your advice. Your suggestions have brought great help to the improvement of our work. Our results show that the method efficiently combines both generative and embedding tasks efficiently, which supports our hypothesis of attention dependence. We hope these additional analyses address your concerns and improve the manuscript's rigor. We sincerely appreciate your constructive feedback and have carefully addressed all the points raised. We hope the revised version demonstrates significant improvements and better aligns with your expectations.", "claims_and_evidence": "The primary claim is that bidirectional fine-tuning increases \"subsequent dependence,\" negatively impacting generative performance. Evidence includes ablation studies comparing various fine-tuning strategies and datasets, showing UBMoE-LLM effectively balances embedding and generation tasks. The study also claims the FNN layer is least affected by bidirectional training, supported by experiments revealing minimal impact on generative performance when only the FNN layer is fine-tuned.", "methods_and_evaluation_criteria": "The methodology includes training with bidirectional contrastive learning using LoRA for larger models and testing on diverse datasets like Natural Instructions, DOQA, and Dureader. The evaluation metrics include F1 scores, accuracy, and rouge-l, providing a comprehensive performance assessment.", "theoretical_claims": "The paper introduces a novel attention dependence measure to quantify preceding and subsequent dependencies. The derivation is consistent with established attention weight calculations, though a deeper examination of convergence guarantees would strengthen the claims.", "experimental_designs_or_analyses": "The experimental results involve models of varying sizes (0.5B to 7B) and controlling factors like learning rate and batch size. However, further experiments with more diverse datasets would enhance the generalizability of the findings.", "relation_to_broader_scientific_literature": "The contribution builds on previous works exploring bidirectional modeling in causal language models.", "essential_references_not_discussed": "n/a", "other_strengths_and_weaknesses": "Strengths:\n\n- Comprehensive evaluation across diverse datasets.\n\n- The introduction of the attention dependence measure is novel and insightful.\n\nWeaknesses:\n\n- Limited discussion of potential trade-offs in training complexity.\n\n- Additional analysis on the interpretability of attention weights could enrich the findings.", "other_comments_or_suggestions": "- Clarify the distinction between preceding and subsequent dependencies.\n\n- Include a broader comparison with other hybrid models.", "questions_for_authors": "- How does UBMoE-LLM perform on tasks beyond text generation and embedding, such as reasoning tasks?\n\n- Have you explored the impact of scaling UBMoE-LLM beyond 7B parameters?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "g1rkWEJEdd", "reviewer": "Reviewer_y1xr", "summary": "There exists a common belief that causal language models (i.e., unidirectional models) perform better in generation tasks while bidirectional models perform better in embedding tasks. However, bidirectional finetuning of unidirectional models usually leads to significantly inferior generation performance, which makes it difficult to obtain a model that excels in both tasks. This paper analyzes the performance degradation from a new perspective, i.e., they analyze the attention scores and observe that bidirectional finetuning enhances subsequence dependence. Furthermore, they find that training only the FNN layers results in a lower increase in subsequent dependence. Based on that, they propose UBMoE-LLM, a new bidirectional finetuning paradigm that combines the original FNN layer of the unidirectional model with the bidirectional FNN layer trained by unsupervised contrastive learning through MoE and exhibits impressive performance in generation and embedding tasks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">  Weakness 1: The authors claim that the inferior performance ... this conclusion more convincing.\n\nAs we replied to Reviewer bS72, Table 1 shows that there is a consistent trend between subsequent dependence and the general capability of the model. This indicates a correlation between subsequent dependence and the model’s general performance, suggesting a potential causal relationship.\n\nBesides, we analyzed the relationship between generation performance and attention-based forward or subsequent dependence. The experimental results are shown in the table below. As the level of subsequent dependence increased, the generation performance of the model declined. In contrast, our method reduced the subsequent dependence introduced by incorporating bidirectional experts while improving generation performance. This indicates a correlation between the distribution of attention weights and the model's generation performance.\n\n|         Model         |  DPB  |  DPA  | MMLU  |\n| :-------------------: | :---: | :---: | :---: |\n| Qwen2.5-0.5B-Instruct | 55.68 | 39.75 | 46.9  |\n|       UBMoE-LLM       | 54.68 | 38.91 | 43.85 |\n|        FNN-Bi         | 54.40 | 41.40 | 35.54 |\n| Qwen2.5-1.5B-Instruct | 58.27 | 36.57 | 60.21 |\n|       UBMoE-LLM       | 58.08 | 36.80 | 59.41 |\n|        FNN-Bi         | 57.00 | 38.13 | 54.35 |\n|  Qwen2.5-7B-Instruct  | 60.94 | 35.07 | 73.86 |\n|       UBMoE-LLM       | 60.37 | 35.61 | 70.86 |\n|        FNN-Bi         | 46.17 | 49.05 | 21.28 |\n\n> Weakness 2: The results in Tables 2, 3, 4 demonstrate ... current results are not enough to support the claims proposed by the authors.\n\nThank you for your advice. We added MNTP-LLM as a baseline, which is trained using the Wikipedia dataset through the MNTP method. As shown in the table below, the experimental results demonstrate that UBMoE-LLM still exhibits strong performance.\n\n|         Model         | MMLU  | Winogrande | Truthfulqa |  Avg  |\n| :-------------------: | :---: | :--------: | :--------: | :---: |\n| Qwen2.5-0.5B-Instruct | 46.90 |   55.60    |   41.86    | 48.12 |\n|       UBMoE-LLM       | 43.85 |   56.94    |   43.02    | 47.94 |\n|       MNTP-LLM        | 41.33 |   54.40    |   41.65    | 45.79 |\n\n|        Model        | MMLU  | Winogrande | Truthfulqa |  Avg  |\n| :-----------------: | :---: | :--------: | :--------: | :---: |\n| Qwen2.5-7B-Instruct | 73.86 |    73.6    |   64.72    | 70.72 |\n|      UBMoE-LLM      | 70.86 |   75.42    |   62.65    | 69.64 |\n|      MNTP-LLM       | 70.06 |   74.21    |   62.54    | 68.93 |\n\n> Weakness 3: However, I believe additional ablation studies would be helpful to analyze the role of the gate control layer.\n\nThank you for your helpful advice. As shown in the table below, we conducted experiments on the STS task. The gate control layer effectively enhances the ability to embed tasks.\n\n| Model                 | Method       | Avg   |\n| --------------------- | ------------ | ----- |\n| Qwen2.5-0.5B-Instruct | FNN          | 81.63 |\n| Qwen2.5-0.5B-Instruct | FNN w/o gate | 80.14 |\n| Qwen2.5-1.5B-Instruct | FNN          | 80.15 |\n| Qwen2.5-1.5B-Instruct | FNN w/o gate | 78.49 |\n| Qwen2.5-7B-Instruct   | FNN          | 81.79 |\n| Qwen2.5-7B-Instruct   | FNN w/o gate | 80.39 |\n\n> Question 1: This paper focuses on bidirectional finetuning ... embeddings tasks compared to bidirectional pretraining tasks.\n\nWe appreciate the reviewer’s suggestion. However, it is well known that there is currently a lack of proper similar scale bidirectional LLMs to serve as a baseline. We are trying to train to the bidirectional model for testing.\n\n> Question 2: I note that UBMoE-LLM improves the generation performance on TruthfulQA. Is it possible to provide an additional explanation for the improvements?\n\nWe're glad you're paying attention to that. We believe this is mainly due to the introduction of bidirectional experts, which enhances the model’s resistance to hallucination.\n\n> Summary\n\nThank you for your valuable advice. We have addressed these suggestions through additional experiments and analyses. We hope these revisions strengthen the validity of our findings.", "claims_and_evidence": "The authors claim that the inferior performance of bidirectional finetuning is due to the increased subsequence dependence. The motivation is easy to follow. However, although the results in Table 1 are interesting, I am afraid they are not enough to verify the relationship between subsequence dependence and generation performance. I think more verification experiments will make this conclusion more convincing.", "methods_and_evaluation_criteria": "The solutions cooperate well with the empirical findings (tuning FFN layers results in a lower increase in subsequent dependence). However, I believe additional ablation studies would be helpful to analyze the role of the gate control layer.", "theoretical_claims": "I do not find theoretical analysis in this paper.", "experimental_designs_or_analyses": "The results in Tables 2, 3, 4 demonstrate the performance of bidirectional training in both generation and embeddings tasks. However, to show the effectiveness of UBMoE-LLM, I believe additional comparisons with other bidirectional training methods and bidirectional models are necessary. The current results are not enough to support the claims proposed by the authors.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "Most of the essential related works are discussed in this paper.", "other_strengths_and_weaknesses": "1. Obtaining a model that performs well in both generation and embeddings tasks is a significant topic and this paper provides novel insights.\n2. The paper is organized well and easy to follow.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. This paper focuses on bidirectional finetuning to enhance the performance of causal language models. I wonder whether it is possible to achieve comparable performance on embeddings tasks compared to bidirectional pretraining tasks.\n2. I note that UBMoE-LLM improves the generation performance on TruthfulQA. Is it possible to provide an additional explanation for the improvements?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OzbD7SvXmq", "reviewer": "Reviewer_39Yf", "summary": "This paper explores the impact of bidirectional fine-tuning on the performance of unidirectional language models, particularly focusing on the decline in generative ability caused by bidirectional fine-tuning. The authors attribute this decline to subsequent dependence and perform an in-depth analysis to support this explanation. A key finding is that fine-tuning the Feed-Forward Network (FFN) layer has the least impact on generative performance. To address the trade-off between generative and embedding tasks, the paper introduces UBMoE-LLM, a novel model that leverages the Mixture-of-Experts (MoE) method. UBMoE-LLM integrates both the original FFN layer and the bidirectionally fine-tuned FFN layer to preserve generative capabilities while improving performance on embedding tasks. Experimental results demonstrate that UBMoE-LLM achieves good performance, showcasing its potential to balance efficiency and effectiveness in practical applications. This work has significant implications for advancing the adaptability of pre-trained models across diverse downstream tasks.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Weakness 1&2: Test on More Models: The authors have only tested the Qwen 1.5 series of models. The experiments on more models are yet to be conducted. Test on Recent Models: It is beneficial to conduct experiments on recent LLMs.\n\nThank you for your advice. We evaluate the effectiveness of our proposed method on the recently released Qwen2.5 series, and UBMoE still achieved performance comparable to that of causal language models.\n\n| Model                 | MMLU  | Winogrande | Truthfulqa | Avg   |\n| --------------------- | ----- | ---------- | ---------- | ----- |\n| Qwen2.5-0.5B-Instruct | 46.90 | 55.60      | 41.86      | 48.12 |\n| UBMoE-LLM             | 43.85 | 56.94      | 43.02      | 47.94 |\n| Qwen2.5-1.5B-Instruct | 60.21 | 64.48      | 46.58      | 57.09 |\n| UBMoE-LLM             | 59.41 | 63.95      | 49.26      | 57.54 |\n| Qwen2.5-7B-Instruct   | 73.86 | 73.60      | 64.72      | 70.72 |\n| UBMoE-LLM             | 70.86 | 75.42      | 62.65      | 69.64 |\n| Qwen2.5-14B-Instruct  | 79.54 | 78.74      | 67.51      | 75.26 |\n| UBMoE-LLM             | 78.44 | 77.83      | 68.01      | 74.76 |\n\n\n> Weakness 3: Experimental Details: Several important experimental details are missing or not properly explained.\n\nWe provide the experimental setup for Figure 2 in Appendix B. For the model, we use Qwen1.5-0.5B.\n\nWe set the value of λ (lambda) to 0.001.\n\nWe provide detailed data sources in Section 5.1. The data used to train the gating layer is sampled from the Tulu-v2-SFT mixture, with only 32,000 samples used for training. These details will be added to Section 5.1 in the revised manuscript.\n\n> Question 1: It seems that FFN+ATT, as well as FFN+EMB perform better on the model size of 0.5B in Figure 2. I would like to understand why this is the case.\n\nFor LLMs with fewer parameters, fine-tuning both the FFN and attention layers can yield modest performance gains due to the limited model capacity, as shown in our results in Table 2. However, as the model size increases, this performance gap narrows significantly.\n\n> Typos\n\nThank you for your thorough review. We will address this and carefully check our paper in a next revision.\n\n> Summary\n\nThank you for your careful reading and suggestions. We will improve our work according to your suggestion. We hope these additional analyses address your concerns and improve the manuscript's rigor.", "claims_and_evidence": "Yes, the claims made in the submission are largely supported by evidence provided in the paper, though the strength of the evidence may vary depending on the specific claim.", "methods_and_evaluation_criteria": "The proposed UBMoE-LLM are well-suited to the bidirectional finetune degeneration problem. However, the generality of the findings could be enhanced by broadening the evaluation to include more baselines, datasets, and task types.", "theoretical_claims": "Yes, the paper provides a theoretical explanation for subsequent dependence, which is likely grounded in how bidirectional fine-tuning alters the model’s internal representations or dependencies that are crucial for unidirectional generation.", "experimental_designs_or_analyses": "The paper evaluates UBMoE-LLM on both generative and embedding tasks, comparing it to baseline models and demonstrating improvements. The overall experimental setup is reasonable, but more types of foundation models can be further evaluated.", "relation_to_broader_scientific_literature": "The key contributions of the paper—identifying subsequent dependence as a cause of generative performance degradation, proposing UBMoE-LLM to balance task performance, and demonstrating the importance of FFN layer fine-tuning—are well-grounded in the broader scientific literature. The paper builds on prior work in bidirectional vs. unidirectional models, MoE methods, and layerwise fine-tuning while advancing the field by addressing a specific trade-off between generative and embedding tasks. These contributions represent a meaningful extension of existing research, with the potential to influence future work in fine-tuning paradigms and multi-task NLP models.", "essential_references_not_discussed": "Based on the current analysis, the references cited in the paper appear to cover the core prior work in the field and provide sufficient background to support the key contributions of the study.", "other_strengths_and_weaknesses": "1. This paper demonstrates an interesting exploration in attention. The authors propose the concepts of preceding and subsequent dependence to explain the relationships between tokens in the attention layer. The experiments on attention show the dependence of attention when bidirectionally fine- tuning different layers.\n\n2. The authors have validated the method at four different scales, and the wide experimental scope provides insights into the relationship between performance and parameter count, offering guidance for scaling up to larger models.\n\n3. The authors combine the bidirectionally fine-tuned FFN layer through the MoE method to achieve the model's embedding ability while retaining its generative ability. The proposed method is innovative and well-grounded. This series of models holds potential application value. \n\n4. Although there are some minor issues in wording, this paper provides a well-justified and impactful solution to the bidirectional fine-tuning of LLMs that deserves to be published.", "other_comments_or_suggestions": "There are also some minor improvements.\n1. Test on More Models: The authors have only tested the Qwen 1.5 series of models. The experiments on more models are yet to be conducted.\n\n2. Test on Recent Models: It is beneficial to conduct experiments on recent LLMs.\n\n3. Experimental Details: Several important experimental details are missing or not properly explained.\n\n    a) The experimental setup for Figure 2 is not provided in the main text. The authors need to clarify the specific LLMs used in the experiments, as this information is not available in Appendix C.\n\n    b) The value of λ (lambda) used in the loss function is not specified. The authors need to explicitly provide this information.\n\n    c) In Line 258, the authors mention using a \"small amount of data\" to train the gating layer, but they do not define what this dataset consists of. Further clarification is needed.\n\n4. Typos: Some typos remain to be corrected.\n\n    a) The word \"FNN\" should be corrected to \"FFN\".\n\n    b) The word \"Casual\" should be corrected to \"Causal\".", "questions_for_authors": "It seems that FFN+ATT, as well as FFN+EMB perform better on the model size of 0.5B in Figure 2. I would like to understand why this is the case.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposes a two-stage training framework to unify the strengths of bidirectional and unidirectional attention for large language models, aiming to improve both embedding and generation capabilities. The authors first train a bidirectional expert for embedding tasks and later integrate it with the original model using a Mixture-of-Experts (MoE) structure, guided by a gating mechanism. One key concern from reviewers was the potential computational overhead introduced by the MoE design, especially during generation. The authors addressed this with detailed GFLOPs comparisons, showing the overhead is negligible. In the response, the authors also responded by highlighting deployment benefits and improvements on hallucination benchmarks, providing expanded empirical evidence to support their attention dependence hypothesis, and added a baseline comparison with MNTP-LLM to address the lack of SOTA comparisons. 

The AC has calibrated the reviewers’ comments, particularly for those who did not actively engage during the discussion phase. Overall, the reviewers’ concerns have been adequately addressed, and the authors’ responses are thoughtful and well-supported.
Decision: Accept (poster)