ICML 2025

What Limits Bidirectional Model’s Generative Capabilities?
A Uni-Bi-Directional Mixture-of-Expert Method For Bidirectional Fine-tuning
Zuchao Li 1 Yonghua Hei 2 Qiwei Li 3 Lefei Zhang 3 Ping Wang 4 Hai Zhao 5 Baoyuan Qi 6 Guoming Liu 6
Abstract
Large Language Models (LLMs) excel in genera-
tion tasks, yet their causal attention mechanisms
limit performance in embedding tasks. While
bidirectional modeling may enhance embeddings,
naively fine-tuning unidirectional models bidi-
rectionally severely degrades generative perfor-
mance. To investigate this trade-off, we analyze
attention weights as dependence indicators and
find that bidirectional fine-tuning increases subse-
quent dependence, impairing unidirectional gen-
eration. Through systematic Transformer module
evaluations, we discover the FFN layer is least
affected by such dependence. Leveraging this dis-
covery, we propose UBMoE-LLM, a novel Uni-
Bi-directional Mixture-of-Experts LLM, which
integrates the original unidirectional FFN with a
bidirectionally fine-tuned FFN via unsupervised
contrastive learning. This MoE-based approach
enhances embedding performance while preserv-
ing robust generation.
Extensive experiments
across diverse datasets and model scales vali-
date our attention dependence metric and demon-
strate UBMoE-LLM’s superior generative qual-
ity and reduced hallucination. Code is available
at: https://github.com/heiyonghua/
ubmoe_llm.
1. Introduction
Creating a model capable of both generation and text em-
bedding has been a long-term goal in the field of natural
1School of Artificial intelligence, Wuhan University 2This work
was conducted during Yonghua Hei’s internship at the School of
Artificial Intelligence, Wuhan University. 3School of Computer
Science, Wuhan University 4School of Information Management,
Wuhan University 5Department of Computer Science and Engineer-
ing, Shanghai Jiao Tong University 6Xiaomi, Beijing, China. Cor-
respondence to: Lefei Zhang <zhanglefei@whu.edu.cn>, Ping
Wan...