OrthAlign: Orthogonal Subspace Decomposition for Non-Interfering Multi-Objective Alignment
Authors: Liang Lin
Zhihao Xu
Junhao Dong
Jian Zhao
Yuchen Yuan
Guibin Zhang
Miao Yu
Yiming Zhang
Zhengtao Yao
Huahui Yi
HAICHUAN TANG
Dongrui Liu
Xinfeng Li
Kun Wang
Venue: ICLR 2026 Poster (2026)
Keywords: alignment
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: Large language model (LLM) alignment faces a critical dilemma when addressing multiple human preferences: improvements in one dimension frequently come at the expense of others, creating unavoidable trade-offs between competing objectives like helpfulness and harmlessness. While prior work mainly focuses on constraint-based optimization algorithms and data selection strategies to mitigate conflicts, these approaches overlook the fundamental issue of resolving conflicts directly at the parameter level. In this paper, we present OrthAlign, an innovative approach that pioneers a new paradigm by leveraging orthogonal subspace decomposition to fundamentally resolve gradient-level conflicts in multi-objective preference alignment. OrthAlign strategically decomposes parameter update spaces into orthogonal subspaces, ensuring that optimization toward different preferences occurs in mathematically non-interfering directions. Building upon this, we provide theoretical guarantees demonstrating that when parameter increments satisfy both orthogonal subspace constraints and spectral norm bounds, the resulting updates exhibit linear Lipschitz growth rather than exponential instability, ensuring stable convergence across all preference dimensions. Extensive experiments show that: I. OrthAlign achieves maximum single-preference improvements ranging from 34.61% to 50.89% after multiple-objective alignment across helpful, harmless, and truthful dimensions. II. With an average overall reward improvement of 13.96%. Our code is available at https://anonymous.4open.science/r/OrthAlign.

Published as a conference paper at ICLR 2026
ORTHALIGN: ORTHOGONAL SUBSPACE DECOMPO- SITION FOR NON-INTERFERING MULTI-OBJECTIVE
ALIGNMENT
Liang Lin 1,x4 Guibin Zhang!
Zhihao Xu
Junhao Dong
Jian Zhao Zhengtao Yao Kun Wang*
Yuchen Yuan Huahui Yi?
Miao Yus
Viming Zhang
Haichuan Tang
Dongrui Liu
Xinfeng Li
Institute of Artificial Intelligence (TeleAI), China Telecom
Renmin University of China
NNayyang Technological University
*National University of Singapore 'University of Science and Technology of China University of Southern California
"West China Biomedical Big Data Center, Sichuan University
8 CRRC Academy 9 Shanghai Artificial Intelligence Laboratory
ABSTRACT
Large language model alignment faces a critical dilemma when addressing multiple human preferences: improvements in one dimension frequently come at the ex- pense others, creating unavoidable trade-offs between competing objectives like helpfulness and harmlessness. While prior works mainly focus on constraint-basee optimization algorithms and data selection strategies mitigate conflicts, these approaches overlook the fundamental issue of resolving conflicts directly at the parameter level. this paper, we present OrthAligg iig, an innovative approach that pioneers paradigm by leveraging orthogonal subspace decomposition to fun- damentally resolve conflicts in multi-objective preference alignment. OrthAlign ign strategically decomposes parameter update spaces into orthogonal subspaces, en- suring that optimization toward different preferences occurs in mathematically non-interfering directions. Building upon this, we provide theoretical guarantees demonstrating that when parameter increments satisfy both orthogonal subspace constraints and spectral norm bounds, the resulting updates exhibit linear Lipschitz growth rather than exponential instability, ensuring stable convergence across all preference dimensions. Extensive experiments show that OrthAlign achieves mungle-prefernne improvements ranging from 34.61 % to 50.89%1 after multiple- preference alignment across helpful, harmless, and truthful dimensions. (II) with average overall reward improvement of 13.96% Our codes are available at attps://A //g/ thubuuu.b... cwm 231223311222 httmm//2/3aaannnttaaaa /rtt///.....inn...
INTRODUCTION
Al products represented by Large language models (LLMs)) (Zhao et al., 2023; Chang et al., 2024; Achiam et al., 2023; Zhang et al., 2026a) need to satisfy providing accurate and reliable responses (Helpfulness) as a foundation (Wang et al., 2025; Li et al., 2025), while also meeting Honesty and Harmlessness metrics to deliver services that align with human values (3H optimization) (Lambert et al., 2024; Yu et al., 2025). Recently, techniques such Supervised Fine-tuning (SFT) (Wei et al., 2021; Yang et al., 2024a), Reinforcement Learning with Human Feedback (RLHF) (Bai et al., 2022; Hu et al., 2024; Zhang et al., 2026b), and Direct Preference Optimization (DPO) (Xu et al., 2024) have enhanced certain capabilities within the 3H framework. However, optimizing a single objective often results in the inadvertent performance degradation of other objectives, thereby establishing fundamental trade-off's that manifest as inherent tensions among competing objectives (Bai et al., 2022: Sun al., 2024).
'Equal contribution. wang. kwn@ntu.nn.......... edu.sg.
Work done during an internship at TeleAI. 'Coresppoding author:

---

Published as a conference paper at ICLR 2026
210
Outstanding
harmnles
230
Sum score of three objectives
OrthAllggnOOurs)
83
performance
82.50%
OrtaAlign(Oums))
Sum score of three objectives
OrthAlign(Ours)
220
Harmless rate(%)
78
Maintain
220
RSDPO
13-59
helgfulness
exceptional
210
RSDP
14-34 TSV
Knots
perforunacce
Knots
21.0
ISV
68
Caf'tt
TSWRS
69.29%
MODPO
200
SPO
trade off
balonce
SPO
DPO
DPO
190
58
190
DPO. Soups
180
DPO
DPo-Soups
MODPO
DPO-Soups
180
$3
69
170
Helpful rato(%)
Figure I: (Left) Average threc-objective performance on Llama3-SFT (Dong et al., 2024); (Middle) Two-objective average performance on all configurations, circle sizes increase as the y-axis increases; (Right) Average three-objective performance on Mistral-SFT (Tunstall et al., 2023).
Multi-preference (or objective) alignment (MPA) (Sun et al., 2025; Xu et al., 2025) aims address the challenges inherent multi-direction conflicts, thereby achieving harmonization among conflicting objectives. Existing data mixing approaches employ rules (Lambert et al.), scores (Wang al., 2024b), or alignment conflict metrics (Jiang et al., 2024) to curate training datasets for individual LLMs. These methods invariably require multi-dimensional data assessment and scoring, making data curation processes heavily dependent on extensive human labor and expert knowledge while simultaneously introducing systematic biases that difficult eliminate (Yang et al., 2025). Building upon this consensus, model merging approaches (Jang et al., 2023; Lin et al., 2023) attempt construct versatile LLMs by combining multiple specialized models with distinct preferences. For instance, Reward- (Rame et al., 2023) and RESM (Yang et al., 2025) achieve MPA through different weight ranking and Pareto-optimal aligmment strategies, respectively. However, these compromise-based policies inevitably lead to performance degradation on individual objectives while enabling MPA, creating fundamental trade-off between specialization and generalization (Yang et al., 2025; Xie et al., 2025).
Recent advances RLHF have increasingly adopted dynamic reward functions (Moskovitz et al., 2023; Xiong et al., 2024) or multi-objective reward frameworks facilitate simultaneous optimization across multiple objectives (Zhou et al., 2024; Gupta et al., 2025; Xu et al., 2025). While these methods improve model-level MPA via adaptive trajectory steering of the global parameter space, they fail to directly address the intrinsic parameter antagonisms that emerge under multi objectivee optimization regimes from a parameter-level perspective (Zhou al., 2024; Gupta al., 2025). The of this antagonism is that the parameter updates for different objectives are not orthogonal but instead interfere with each other, which can be quantitatively expressed as the non-zero inner product of their respective gradients (left side of Eq. 1, where V VDL(D:) represents the gradient of the loss function L with respect to parameters 0 for preference source D:): (VoL(D.). P P.V,C( 0, for i i j.. IV,C(D.)) IPP VoL(D,)
(D,), VoC(D,)) V.L(D.) ((D, #0
(1)
This paper introduces a novel paradigm, OrthAlign, that leverages gradient stability theory to update non-interfering local parameters within the LLMs. Matrix decomposition (Hsu et al., 201 1; Zhang, 2015) enables the extraction of singular value matrices corresponding to individual preferences. The feature spaces associated with trailing eigenvalues exhibit approximate orthogonality to current preference information
property that proves instrumental for the OrthAlign. As shown on the right side of Eq. 1, by applying orthogonal projection matrix P to constrain gradient updates for new preferences, each increment matrix AW becomes confined within mutually non-interfering orthogonal subspaces, thereby achieving theoretically guaranteed conflict elimination.
Elegantly, provide theoretical guarantees based stability theory (Hardt al., 2016): when model parameter increments at each step simultaneously satisfy two constraints: (I) restricting updates to the orthogonal complement of the principal subspace avoid interfering with critical and
2

---

Published as a conference paper at ICLR 2026
prior directions, and (II) clipping the spectral norm of updates control amplification rates- the per-layer Lipschitz upper bounds (Bartlett et al., 2017) of both individual layers and the entire model exhibit linear growth, ensuring stability throughout the cumulative update process. Conversely, multiple updates may accumulate along the same principal directions, leading to super-linear or even exponential inflation of spectral norm (i.e., overall Lipschitz upper bounds) (Horn & Johnson, 2012).
Experimental Contributions. Experiments validate the effectiveness of OrthAlign over 7+ base- lines across 4 benchmarks. Compared to the best-performing methods, OrthAlign achieves an average improvement of 20.23% on two-objective alignment and 13.96% on three-objective align- ment (Figure 1). Furthermore, OrthAlign functions as a performance enhancer for existing alignment techniques, boosting harmlessness by 25.06% and helpfulness by 4.86% on average, enabling its seamless integration as plug-and-pla module. We believe our subspace rank selection theory and framework will significantly advance the field of MPA.
2 PRELIMINARY
Conflict Mitigation of MPA. The language model MPA process typically begins with a foundation model that has undergone SFT, referred to as To. Concretely, the base model is trained on curated demonstration data to establish competent performance across various tasks. Human preferences are commonly captured through comparative evaluations y1 > y/2 governed by latent reward mechanisms {rī (z. (- where one response is deemed superior to another for a given prompt æ. The Bradley- Terry (Bradley & Terry, 1952) and PPO (Schulman et al., 2017) framework provides probabilistic foundation for modeling these aggregated preference judgments:
exp(2 f (x,
P(y > 32/x) = exp EE Ar7 (1) + exp(E. (z,
where k represents the number of preference sources, X; denotes the weight for the i-th preference source, and r. (xx, y) is the latent reward for preference source i. Traditional approaches like RLHF employ two-stage process: first learning explicit reward models fre, ) from multiple preference datasets (D,) then optimizing the LLMs using RL polices to maximize aggregated expected rewards while maintaining proximity to the original policy. DPO Series (Rafailov et al., 2023; Zhong al., 2024; Xiao et al., 2024) bypass explicit {rw, 1 entirely, which leverage the relationship between the policy and implicit reward representations across multiple preference sources. The optimization objective for DPO can be expressed as:
LTo Σ \e log a (Blg To(Uwlz) - ,3 Blgg
Ne (30l (z)) To( (x:)
(3)
To (x))
Here, To serves as the reference policy, controls the strength of the KL constraint, or is the sigmoid function, and each dataset D; contains preference triplcts (z. Bw. y1) where you and y represent the preferred and dispreferred responses respectively. In Eq. 3. DPO directly optimizes the policy to favor preferred responses across multiple preference sources while maintaining controlled deviation from the reference model. However, preference alignment in practice is complicated by inherent conflicts between different preference sources. Most MPA methods (Zhou et al., 2024; Gupta et al., 2025; Lou al., 2025) attempt mitigate these preference conflicts by introducing constraint loss terms, however, simultancous optimization inevitably brings conflicts to the internal parameters, which also limits the stability of the model's intrinsic matrix updates (depicted later in Section 3).
Singular Value Decomposition (SVD) performs optimal matrix factorization that isolates the princi- pal singular components, thereby achieving the optimal low-rank approximation while preserving the orthogonality of subspaces spanned by distinct singular value magnitudes (Wall et al., 2003):
Rank
W = UEVT
WX UEVV X)) x))= > (o,(o)
(()[(V))))+
u4
(4)
where W is the parameter weight matrix, U E mm is the left singular vectors, Σ E R is is the transpose matrix of the right. Here, u, U|: i| and v; Vi:. il represent the i-th column vectors of matrices U and V respectively. The transformation projects X onto the rank- lineeiinnsiionll subspace spanned by the principal components, with each component of X scaled by o; and mapped the output space via u;.
the diagonal matrix of singular values, and V E <R*x"
3

---

Published as a conference paper at ICLR 2026
3 METHODOLOGY
In this section, we present OrthAlign, a framework designed to resolve parameter-level conflicts in multi-objective alignment. To enhance readability and reproducibility, first detail the practical execution of our framework (Section 3.1), which follows a clear three-step workflow: orthogonal decomposition, adaptive rank selection, and subspace-come constrene optimization. Subsequently, we provide the theoretical analysis that guarantees the stability of our approach (Section 3.2).
3.1 THE OrthAlign FRAMEWORK
The core intuition of OrthAlign is to confine the parameter updates of new preferences into subspace thereby mitigating alignment conflicts between multiple preference objectives.
Step 1: Orthogonalized Preference Decomposition. We first define fe denote the model with parameters (concatenating all layers) and ÁW as the low-rank adaptation matrix obtained from the first preference alignment phase (c.g., safety alignment), where AW = BA with B E Rmxr and A E x=*"" The rank of AW constrained to r. where r < min(m, n2). We denote the inputs from safety preference as sate and from helpful preference as Khelpful-
We can rewrite the transformation by decomposing the preference's 's singular value matrix into distinct subspaces:
maxi 17))
AWX
Σ
(v safe) Ue +
(1 Xanee) Xmnie) Uj j=++1
Σ
(5)
For preference-critical directions
Minimal impact eurrent preference
where the decomposition separates the transformation into two distinct parts: the red portion represents the top-r singular components that capture the most significant directions for safety preference alignment, while the blue portion encompasses the remaining singular components that have minimal impact on safety-aligned behavior.
Step 2: Adaptive Subspace-Rand Rank Selection.
In the decomposition above, the term oh Xaae) is a specific numerical value. We can represent the meaning of this matrix operation more directly by expressing it as linear combination of the left singular vectors:
Algorithm 1 Adaptive Subspace-Raie Rank Selection
Input: AW, X Y, Tmax Output: Optimal Rank
U, Σ, V < SSVD(AW)
1, < 8,Tmx
// Search bounds
// Best rank found
max(m.n)
Rusee R(AW;
// Baseline Reward
AWX - +
Σ
Gr (6)
while S do
k + [[1 + h)/2]
// Candidate rank
ô, + à E5=1
11 Rescale singular
where C2 T& (v xaffe) denotes the projection strength. Previous works (Liang & Li, 2024; Feng et al., 2025) often overlook the incremen- tal influence introduced by updates to the sin- gular value spectrum. As shown in Eq. 7, formalize the conflict under orthogonal param- eter updates, showing that even directions that are initially negligible may become non toivial once their associated singular values are updated. Specifically, oj iy represent the updated coun- terparts after the new preference alignment step: return
vals
AW t- UT if R(AW mmi Xwya) Raaw 5 then
1/ Reconstruct
k; t k+1
// Valid,
expand space
else
h k- space
// Invalid, shrink
end
end
max(m,.))
max(m,n)
0f
uj
update
Σ j=++1
By Xoaee) # 0.
Motivated by this observation, our key insight that directions with negligible influence under low-rank constraints may become significant once their singular values updated. Therefore, our
4

---

Published as a conference paper at ICLR 2026
Preference
Selection
Optimization
Helpful Preference
New preference training
Query: How make cake ?
Dedicated to helping users
Harmeess
Trachfud
Honasty
Faanryy
Firstly. Secondly
Harmless Preference
answer harmful query
Query How make homé
Truthful Preference
Model Parameter
Supspace Conctraitt
Sorry, I can't do that
Answer only if know that
Query How create a compony
Model Parameter
try my best to help
ALL undate in. no constroint parameter
Σ coo
Orthoginal
J,15
Helpful
Helphul
influence porameter
Pesjection SubSpace
request harmful query
Tackle users' query.wall
8 PAPA
Not request harmful query
Tackle users' guery
Left singular
Right singular matrix
VT
matrix
Harmless
Truthful
Can't balance all preferences
Balance preferences
Ad
Figure 2: OrthAlign Framework. We achieve non-interfering MPA through matrix factorization.
goal is merely to update within the entire remaining null space of dimension max(m, т, but to further screen this space to identify refined subspace where the impact on previous preferences is strictly minimized. To achieve this, we design a dynamic rank selection rule, as detailed in Algorithm 1. We rescale the last & singular values to the mean of the top-r ones to simulate potential updates, and then choose the largest feasible & such that the reward shift on X remains within tolerance y. Here R(W; sate) denotes the expected positive reward:
max {IR(US(VVT Xaate) - R(W; X I Y, Ō,
=mxx
(8)
j=1
Based this criterion, select the most suitable rank of subspace to guide the update. The detailed algorithmic procedure and discussion of tolerance 7 is provided in Appendix B.1 and Appendix B.2.
Optimization. Once the optimal rank l: is selected, we constrain the updates to the subspace spanned by the corresponding Let's denote the matrix formed by these vectors as Ó. Our projection matrix P' is defined as:
Step 3: Subspace-Consin
P= ÚUT
Based on the projection matrix P. constrain the parameter updates for preferences AW within the selected subspace through the following projection operation:
new
WW new P
Vw Lnew (W))
(10)
where Lnw (W) represents the gradient of the new preference loss function. Through this projection, we ensure that parameter updates are strictly confined to the subspace orthogonal to previous preferences.
3.2 THEORETICAL ANALYSIS
In this section, we provide the theoretical guarantees regarding the validity and stability of OrthAlign.
Definition (Safety principal subspace and projector). We write Ao = vec ve) for brevity. Let g()) be a (hiigheris-bbtttr) safety score. Locally, g( A0) g(0) + (Vg(0), A0) + A0 H. A0.
5

---

Published as a conference paper at ICLR 2026
Llama-3-9FT-Helnstee
Laama 1ama-3-SFTT-Umm
Mistral-SPTT-IIre
93
88
88
OrihATign(Ours))
83
Knuts
OrthAlign(Oots))
OrthaliggnOOrs))
Harmless rate(%)
83
Harmles less rate (%)
78
Harmless rate(%)
73
78
68
73
RSDPO
DPOSPO
Knots
TSV
68
73
+RSDPO
68
63
Knolisy
INV
SPO
63
63
DPO
58
MODPO
58
DRO-Soups
58
53
RSDPO
53
53
DPoiSo
DPO-Soups
48
MONPO
48
MODPO
48
DPQ
433
56
59
62
65
68
57 60 63 66 69 72 75 78 81 Helpful win rate(%)
439
62
65
68
71
74
Helpful win rate(%)
Helpful win rate(%)
Figure 3: Two-objective sequential alignment results for helpfulness and harmlessness. OrthAlign achieves the best balance in both objectives.
where (1) = 0 is Positive Semi-Definite safety curvature. Define HI, = QAQ' T with eigenvalues
A 2 ... Xa 2 and Qk [q....., Ak). The safety principal subspace is St = (Qan(Qu); its orthogonal complement is S Let P1- QxQ₂ denote the projector onto S
Each preference update produces A0 that satisfies:
1. Subspace constraint: AO E St (equivalently, Q, 0).
2. Spectral constraint: the corresponding layer increment satisfies | S (hence
|0l| S eT for layer-dependent constant c).
Lemma 1. If Vg() E Se and the subspace constraint holds, then (Vg(0), A0) 0.
Proof. Since (g(()) E Sx and AO St. orthogonality of complementary subspaces yields (Vg(O), A0) 0.
Theorem 1. Under the assumption above, the following hold. (a) Second-order bound. For any one- step update, g g(0+A0)-g(0) S > || and the RHS is 0 if Ak+1 O. (b) Cumulative bound. For T updates {= satisfying subspace and spectral constraints at iterates (0r) with (0,) = Q (Q(())) and tail eigenvalue X Σi
S 1 AR41 10,01
Remark Second-orter changes along the orthogonal complement are controlled by the tail curvature AR+1. Summing per step gives global safety budget: long tail curvature and step sizes are small, cumulative safety drift remains bounded. The detailed derivation is exhibited in Appendix F.l.
Building the subspace and spectral constraints, the next theorem shows that per-step spectral control yields at most linear growth of the layer Lipschitz constant, and that allocating LoRA increments to mutually orthogonal subspaces eliminates destructive interference.
Theorem 2. Consider single linear layer with mapping & + (W -Et, AW,)a Suppose each update satisfies the spectral constraint |AW,, ll2 S т. (a) Linear Lipschitz accumulation.
i.e., [ IW AW:l < [|W]l22 + Σt= || I ||W]|2 Tr. (b) Orthogonal allocation eliminates destructive interference, i.e., D0; U, with 24 1 U, for all s < < by projecting onto the orthogonal complement prior dominant subspaces), then D A0, Σr= |
Remark (a) Per-step spectral control implies most linear growth of the layer Lipschitz constant, forbidding uncontrolled blow-up. (b) Assigning updates to orthogonal subspaces prevents cross-
term sancellatiolovewwiiingg ensuring additive retention of preference increments. The detailed derivation is exhibited in Appendix F.2.
EXPERIMENTS
In this section, conduct experiments address the following research questions:

---

Published as a conference paper at ICLR 2026
Table 1: Performance comparison of different methods on sequential preference optimization tasks. The best results are highlighted in bold, while the second-best results are underlined.
Different Method
UltraFeed back
HelpSteer2
Hamsless Rate
Helpfal Win Rate
Truchful
Average Scoret
Harmless Rate1
Helpful Win Rate
Truthful MC21
Average Scoret
MC2:
LLAMA-3
SFT
46.73
50,00
53.41
50.04
46.73
50,00
53.41
50.04
DPO
$2.69 Ta
70.93
Tairs
67.03
Tex.on
63.55
Twas
51,92 1s
72.91
Tox.s
66.50 fi
63.78 "1x74
MODPO
ACL 2024
56.46 Tu
71.42
65,79
64,55
70.96
Fau
69:44
119.4s
62.08
Twier
67.49
SPO
AAAI 2025
58.42 1:3.er
71.08
Ton.ss
64.22
64,57
66.15
68.24
Toxass
62.01
14.30
65.46 111,42
Soups
SUPS 2023
56.15 Tau42
60.00
120.00
64.59
60.24
Tu
56.92
fan.s
61.18
111.3s
58.93
1s.s2
59.01 fw.9r
RSDPO
NAAC1.0044
faxes
63.87
Tuo.es
72.12
75.57
faa.s
Tasuso
63.40 fause
69.92 "1aas
TSV-M
2024
1ax.00
66.75
Tao.es
63.51
06.30
78.07
tax
62.12
1uausa
62.36
ta.ss
67.51 117.07
Knots
[CLR 2025
82,30
130.07
63.73
61.78
Tw.37
69.27
Tw
87.11
59.62
Tausz
63.66
70.13 220.0
Orhhlign
Tan.rr
71.57
Tax 50
66.58 11
75.45
Tos. 11
91.34
Tec.s:
68.83
Trk.ss
67.69
75.95 "35.91
MISTRAL
SFT
20.19
26.83
43.03
30.01
20.19
26.83
43,03
30.01
DPO
Baseline
27.11 Ta.s
72.91
Ton.os
66.55
Tas.na
55.52
12
39.23 1100s
73.16
Tas.3s
66.01
T22.9s
59.46
"23.45
MODPO
2024
58.07 137.0s
73.41
Tuo.3s
59.95
113.92
63.81
fasuss
71,36
f51.77
61.55
114.72
59.32
f10.29
64.07
"34.06
SPO
AAAI 2025
68.07 Tames
75.51
61.57
Tes.na
68.38
Toxar
84.03
65.21
Taxass
56,25
Tramm
68.49
Soups
NIPS 2023
34.23 Tra.ss
71.92
Tasure
61.05
Tan.na
55.73
Tun-72
$4,03
Fassas
59.75
Taz.90
59.92
57.90
"2n.e9
RSDPO
NAACL22024
71.87
153.00
73.24
Tus-42
63.25
f20.22
69:45
Tow
66.32
Ta6.13
68.95
142.32
64.35
f21.12
66.54
"as.ts
TSVMM
LVPR
1:1.00
70.63
Fuxss
65.13
122.30
fawis
73.30
fsaus:
Pa7.34
f2a.es
67.58
Knots
[CLR 2005
64,50
72.80
59.23
1s0.20
65:51
75.28
fac.ss
61.66
Tax.s
64.40
67.11
"ar.0
OrthAlign
78.00
Ts7.01
75.51
Tokis
65.28
T2a.20
72.93
88.12
Tur
67.08
Tas.an
65.34
Tuoim
73.51
"43.00
RQI: How does OrthAlign perform on sequential preference alignment tasks compared to baseline methods? Can it mitigate the conflict?
RQ2: Can OrthAlign effectively preserve the distribution of previously aligned preferences?
Specifically, does the orthogonal subspace constraint prevent shifts in the representation distribution?
RQ3: Does OrthAlign gn demonstrate generalizability across different baseline methods? Can existing multi-preeerence alignment approaches be significantly improved by OrthAlign?
RQ4: Through dynamic subspace selection, can OrthAlign achieve fine-grained control over the balance of trade-offs?
4.1 EXPERIMENTAL SETUP
Baselines & Model Configurations. Our experiments are conducted on -LLAMA3-SSFT (Dong al., 2024) and Mistral-7B-SFT (Tunstall et al., 2023). The baselines against which we compare with OrthAlign can be broadly grouped into three categories: Constraint-
training methods, such as MODPO (Zhou et al., 2024), and SPO (Lou et al., 2025); Data synthesis-based approaches, comprising RSDPO (Khaki et al., 2024); Model merging-based methods, including Soups (Rame et al., 2023), Knots (Stoica et al., 2024) and TSV-M (Gargiulo et al., 2025). See Appendix D.2 for more details about baselines.
Benchmarks & Evaluation Metrics. To provide comprehensive evaluation of OrthAlign, we employ benchmarks spanning three domains for training: Helpful, including Helpsteer2 (Wang et al., 2024e) and UltraFeedback (Cui et al., 2023), we use 10K randomly sampled instances per dataset; Harmless, represented by SafeRLHF-10k (Ji et al., 2024); Truthful, comprising 10K truthful data from UltraFeedback and Helpsteer2 and evaluate the performance on Alpaca- AdvBench (Zou et al., 2023), and the TruthfulQA (Lin et al., 2021) for helpfulness, harmlessness, and truthfulness, respectively.
(Li al., 2023),
line with prior work (Zhou et al., 2024; Xu et al., 2025), we employ Helpful win rate, Harmless Rate and TruthfulQA MC2 criterion as evaluation metrics. The details of our training settings and evaluation methodology are provided in Appendix and Appendix respepcctyv
4.2 PERFORMANCE ON MULTI-OBJECTIO PREFERENCE ALIGNMENT (RQ1)
To evaluate the performance of different alignment methods in terms of balancing preference conflicts, we conduct sequential preference optimization using OrthAlign and other baselines. Table I and Figure 3 present the results under a commonly used configuration for the sequential preference optimization task, where sequentially perform preference alignment on harmlessness, helpfulness,
7

---

Published as a conference paper at ICLR 2026
(a) MODPO on Llama3-Ultra
(b) SPO Llama3-Steer
(c) Soups Mistral-Ultra
(d) OrthAlign on Llama3-Ultra
(e) OrthAlign on Llama3-Steer
(f) OrthAlign on Mistral-Ulte
Figure 4: The distribution hidden representations first-time alignment and third-time alignment LLMs. The top and right curve graphs display the marginal distributions for two reduced dimensions, where OrthAlign consistently exhibits minimal shift. Here, Purple represents first-time alignment distribution, Blue and Red represent third-time alignment distribution of OrthAlign and baselines, respectively.
and truthfulness. For additional experimental results, please refer to Appendix E. Based on Table and Figure 3, we can draw the following observations:
Obs 1: OrthAlign achieves superior performance in two-objective alignment scenarios. Specifi- cally, OrthAlign demonstrates remarkable capability balancing the harmless-helpful trade-off by surpassing the best-balancing baselines by average of 8. 3.77% 1 and 7.56% 1. respectively. Unlike baseline methods that often exhibit stark trade-offs, OrthAlign achieves the most balanced performance with the average combined performance showing improvement of 12.80 22.53f. These gains arise from OrthAlign's 's ability to fundamentally mitigate conflicts.
Obs 2: OrthAlign sustains balanced superiority across expanded preference dimensions. When transitioning three-objective optimization, OrthAlign demonstrates individual improvements with average gains of 5.30% in harmlessness, 3.25% f in helpfulness, and 4.47% in truthfulness against the strongest baselines, resulting in cumulative tiree-preference sum improvements ranging from 9.09% 17.47% f points across all configurations.
4.3 HIDDEN STATE DISTRIBUTION ANALYSIS (RQ2)
As discussed in previous sections, existing MPA methods fundamentally fail to resolve conflicts at the gradient level and inevitably lead distribution shifts in the model's hidden representations, causing previously aligned preferences to degrade when new preferences are introduced. Hence, we conduct comprehensive analysis of hidden state distribution shifts during sequential preference alignment. Specifically, we sample 3000 training instances from the first preference alignment iteration, extract their hidden states, and compare them with the final alignment results using t-SNE (Maaten & Hinton, 2008; Wattenberg et al., 2016) visualization. According to Figure 4, we can observe that:
Obs 3: OrthAlign preserves distributional consistency throughout multi- celti-preffeence alignment. The distribution visualizations show nearly identical point clouds between initial and final align- ments, with marginal distributions maintaining their original shapes. This invariance indicates that OrthAlign successfully mitigate parameter conflict.
Ohs 4: Baseline approaches exhibit pronounced distributional divergence after multi-preference alignment. The trend of distributions reveal distinct clusters forming between initial and final states, further highlighting the critical importance of orthogonal subspace-base parameter updates.
4.4 APPLYING OrthAlign TO BASELINE METHODS (RQ3)
8

---

Published as a conference paper at ICLR 2026
Table 2: Performance of baselines before and after the subspace projection.
To investigate whether OrthAlign can compre- hensively enhance current alignment methods, we add subspace projection from OrthAliga to baselines and perform two-objective alignment on Helpsteer2 and LafeRLHF datasets. With the results presented in Table 2, we offer the follow- ing observation:
Method
Harmless Ratet
Helpfulness Win Rate
SFT
46. 73
50.00
DPO
71.24
60.24
MODPO
48.46
67.95
Obs 5: OrthAlign acts potent per- formance enhancer for various alignment methods. Specifically, the enhanced baselines demonstrate an average performance uplift of 14.96% t, underscoring the significant potential of OrthAlign in boosting alignment capabilities.
SPO
71.15
61.24
DPO-Orth
93.84 f22..0
65.71 f0.47
MODPO-Orth SPO-Orth
79.32 fa0.so
71.02 f2.32
92.88 fan.zs
67.28 10.00
4.5 ADAPTIVE PERSPACE-RANE ABLATION STUDY (RQ4)
In previous section, theoretically discuss the potential impact of different subspace ranks preference alignment. To empirically validate our adaptive rank selection mechanism and understand how different subspace configurations affect multi-objective alignment, we conduct comprehensive ablation studies using the Helpsteer2 and SafeRLHF datasets on Llama3-SFT. Specifically, we exam- ine the performance sensitivity to subspace rank selection by testing various fixed rank configurations. For each rank, run the experiment five times and report the average performance. According to Figure we can find that:
Obs 6: The adaptive rank selection demonstrates a clear trade-off pattern between ob- jectives. Harmlessness perfor- mance shows significant sensitiv- ity to rank changes, declining from 93.80% at rank 12 81.34% rank 26, with optimal safety main- tained around rank 16 18 where rates exceed 89%. Mcanwhile, helpful win rate remains relatively stable from rank 14 onwards, fluc- tuating only between 63. 63.59% 65.79%. This underscores the im- portance adaptive rank selection for preserving safety without compromising utility.
Performance under different ranks
98
Harmless Rate (Left Axis)
69
Helpful Win Rate
Harmless Rate
96
Helpful Win Rate (Right Axis)
68
94
67
92
66
go-
65
64
86
63
84
62
Ra
80
60
R-10
R-12 R-14 R-16 R-18 R-20 Rank
R-22 R-24 R-26
Figure 5: Performance under different ranks Helpsteer2 and SafeRLHF datasets.
5 TECHNICAL BACKGROUND
Multi- pelti--prefeencc alignment (MPA). MPA refers to the process of simultaneously optimizing AI model to satisfy multiple, potentially conflicting human preferences objectives, rather than focusing on a single alignment criterion (Bai et al., 2022; Sun et al., 2024). The off the--sh-llff MPA methods can be broadly categorized into three major branches: (I) Preference-conditoon
data selection approaches (Basaklar et al., 2022: Zhu et al., 2023) identify and select high-quality, multi-dimensional data to effectively align with diverse human preferences by shifting the alignment problem from training specific models to intelligent data filtering (Yang et al., 2024b; Wang et al., 2024a; Yang al., 2024a; Guo et al., 2024; Zhong al., 2024), which assumes that human preferences can be aggregated via simple linear combinations (Rame et al., 2023; Wang et al., 2024c; Bo et al., 2025) (II) Parameter-merging. Reward-Son -oou (Rame et al., 2023) and related approaches (lang et al., 2023; Lin et al., 2023) attempt to construct new models with balanced preferences by merging parameters (Du et al., 2024; Akiba et al., 2025) from multiple models trained on individual preferences using various hyperparameters. (III) Training via Constrained Unification. These methods address the conflict by incorporating additional joint reward function to create a new reward that captures the trade-offs between different goals like C-RLHF (Moskovitz et al., 2023), MORLHF (Xiong et al., 2024), PAMA (He & Maghsudi, 2025). Alternatively, other methods transform a multi-objective problem into constrained optimization problem (Liang al., 2024; Nan et al., 2024; Qiao et al., 2024),

---

Published as a conference paper at ICLR 2026
where the goal is to maximize objective while ensuring the expected reward of another objective does not decrease. These approaches formulate other objectives as constraints and use constraint satisfaction mechanisms during optimization, et al., 2024d), MO-ODPO (Gupta et al., 2025), SPO (Lou et al., 2025).
in MODPO (Zhou et al., 2024), MAP (Wang
6 CONCLUSION
In this work, we introduce OrthAlign, an innovative approach that leverages orthogonal subspace decomposition fundamentally resolve gradient-level conflicts in multi-objective preference align- ment. By strategically decomposing parameter update spaces into orthogonal subspaces, OrthAlign ensures that optimization toward different preferences occurs in mathematically non-interfering direc- tions, addressing the core challenge where improvements in one dimension come at the expense of others. Experimental results powerfully demonstrate Or OrthAlign achieves remarkable performance and functions as plug-and-play enhancement for existing alignment techniques. We believe this paradigm will significantly advance the field of MPA.
ACKNOWLEDGMENTS
This research is supported by National Natural Science Foundation of China (62476224).
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
Takuya Akiba, Makoto Shing, Yujin Tang, Qi Sun, and David Ha. Evolutionary optimization of model merging recipes. Nature Machine Intelligence, 7(2):195-204 2025.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862 2022.
Peter L Bartlett, Dylan J Foster, and Matus J Telgarsky. Spectrally-normaliin margin bounds for neural networks. Advances in neural information processing systems, 30, 2017.
Toygun Basaklar, Suat Gumussoy, and Umit Y Ogras. Pd-morl: Preference-driven multi-objective reinforcement learning algorithm. arXiv preprint arXiv:2208.07911 2022.
Jessica Y Bo, Tianyu Xu, Ishan Chatterjee, Katrina Passarella-Want Achin Kulshrestha, and D Shin. Steerable chatbots: Personalizing ilms with preference-based activation steering. arXiv preprint arXiv:2505.0422 04260, 2025.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4)3224 345, 1952.
Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. A survey on evaluation of large language models. ACM (ransactions on intelligent systems and technology, 15(3):1-45, 2024.
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedbcck: Boosting language models with high-quality feedback. 2023.
Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. Rlhf workflow: From reward modeling online rlhf. arXiv preprint arXiv:2405.0788 07866,, 2024.
Guodong Du, Junlin Lee. Jing Li, Runhua Jiang. Yifei Guo, Shuyang Yu, Hanting Liu, Sim K Goh, Ho-Ki Kin Tang, Daojing He, et al. Parameter competition balancing for model merging. Advances in Neural Information Processing Systems, 37:84746-84777 2024.
10

---

Published as a conference paper at ICLR 2026
Jinyuan Feng, Zhiqiang Pu, Tianyi Hu, Dongmin Li, Xiaolin Ai, and Huimu Wang. Omoe: Diversify- ing mixture of low-rank adaptation by orthogonal finetuning. arXiv preprint 2rX0022501.1066,
2025.
Antonio Andrea Gargiulo, Donato Crisostomi, Maria Sofia Bucarelli, Simone Scardapane, Fabrizio Silvestri, and Emanuele Rodola. Task singular vectors: Reducing task interference in model merging. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 18695- 18705, 2025.
Yiju Guo, Ganqu Cui, Lifan Yuan, Ning Ding, Zexu Sun, Bowen Sun, Huimin Chen, Ruobing Xie, Jie Zhou, Yankai Lin, al. Controllable preference optimization: Toward controllable multi-objective alignment. arXiv preprint arXiv:2402. 19085, 2024.
Raghav Gupta, Ryan Sullivan, Yunxuan Li, Samrat Phatale, and Abhinav Rastogi. Robust multi- objective preference alignment with online dpo. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 27 2025.
Moritz Hardt, Ben Recht, and Yoram Singer. Train faster, generalize better: Stability of stochastic gradient descent. In International conference on machine learning, pp. 1225-1234, PMLR, 2016.
Qiang He and Setarch Maghsudi. Pareto multi-objective alignment for language models. arXiv preprint arXiv:2508.077 07766668,. 2025.
Roger A Horn and Charles R Johnson. Matrix analysis. Cambridge university press, 2012.
Daniel Hsu, Sham M Kakade, and Tong Zhang. Robust matrix decomposition with sparse corruptions. IEEE Transactions on Information Theeory 57(11):7221-2334 2011.
Jian Hu, Xibin Wu, Wei Shen, Jason Klein Liu, Zilin Zhu, Weixun Wang, Songlin Jiang, Haoran Wang. Hao Chen, Bin Chen, al. Openrlhf: An easy-to-use, scalable and perfooerformecce rlhf framework. arXiv preprint 2rXiv:2405.11143 2024.
Joel Jang. Seungone Kim, Bill Yuchen Lin, Yizhong Wang, Jack Hessel, Luke Zettlemoyer, Hannaneh Hajishirzi, Yejin Choi, and Prithviraj Ammanabrolu. Personalized soups: Personalized large language model alignment via post-hoc parameter merging. arXiv preprint arXiv: 10. 1564, 2023.
Jiaming Ji, Donghai Hong. Borong Zhang, Boyuan Chen, Juntao Dai, Boren Zheng, Tianyi Qiu, Jiayi Zhou, Kaile Wang, Boxuan Li, et al. Pku-saferlhf: Towards multi-level safety alignment for Ilms with human preference. arXiv preprint arXiv:2406.15513, 2024.
Li Jiang, Yusen Wu, Junwu Xiong, Jingqing Ruan, Yichuan Ding, Qingpei Guo, Zujie Wen, Jun Zhou, and Xiaotie Deng. Hummer: Towards limited competitive preference dataset. arXiv preprint arXiv:2405.1 1647, 2024.
Saeed Khaki, JinJin Li, Lan Ma, Liu Yang, and Prathap Ramachandra. Rs-dpo: A hybrid rejection sampling and direct preference optimization method for alignment of large language models. arXiv preprint arXiv:2402. 10038, 2024.
N Lambert, J Morrison, V Pyatkin, S Huang, H Ivison, F Brahman, LJV Miranda, A Liu, N Dziri, Lyu, al. 3: Pushing frontiers in open language model post-training. corr, abs/2411.15124, 2024. doi: 10.48550. arXiv preprint ARXIV.2411.15124
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. Tulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411. 2024.
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpacaeval: An automatic evaluator of followiinn models, 2023.
Zhong-Zhh -Zi Li, Duzhen Zhang, Ming-Liang Zhang, Jiaxin Zhang, Zengyan Liu, Yuxuan Yao, Haotian Xu, Junhao Zheng, Pei-Jic Wang, Xiuyi Chen, et al. From system to system 2: A survey of reasoning large language models. arXiv preprint arXiv:2502. 2025.
11

---

Published as a conference paper at ICLR 2026
Jing Liang, Hongyu Lin, Caitong Yue, Xuanxuan Ban, and Kunjie Yu. Evolutionary constrained multi-objective optimization: A review. Vicinagearth, 1(1):5, 2024.
Yan-Shuo Liang and Wu-Jun Li. Inflora: Interference- low-rank adaptation for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 23638- 23647, 2024.
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprini arXiv:2109.0795 2021.
Yong Lin, Hangyu Lin, Wei Xiong, Shizhe Diao, Jianmeng Liu, Jipeng Zhang, Rui Pan, Haoxiang Wang, Wenbin Hu, Hanning Zhang, et al. Mitigating the alignment tax of rlhf. arXiv preprint 0.Xi::2309.06256, 2023.
AI @ Meta Llama Team. The Ilama herd of models, 2024. URL httphttt:///// uttptt::///xxit..oo abs/ 2407. 21783.
Xingzhou Lou, Junge Zhang, Jian Xie, Lifeng Liu, Dong Yan, and Kaiqi Huang. Sequential preference optimization: Multi-dimensional preference aligmment with implicit reward modeling. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 27509-27517, 2025.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, (Nov):2579-2665, 2008.
Ted Moskovitz, Aaditya K Singh, DJ Strouse, Tuomas Sandholm, Ruslan Salakhutdinov, Anca I) Dragan, and Stephen McAleer. Confronting reward model overoptimization with constrained rlhf. arXiv preprint arXiv:2310.04377. 2023.
Yang Nan, Hisao Ishibuchi, Tianye Shu, and Ke Shang. Analysis of real-world constrained multi- objective problems and performance comparison of multi-objective algorithms. In Proceedings of the Genetic and Evolutionary Computation Conference, pp. 576-584, 2024.
Kangjia Qiao, Jing Liang, Kunjie Yu, Xuanxuan Ban, Caitong Yue, Boyang Qu and Ponnuthu- rai Nagaratnam Suganthan. Constraints separation based evolutionary multitasking for constrained multi-objective optimization problems. IEEE/CAA Journal of Automatica Sinica, 1888 2024.
Rafael Rafailov, Archit Sharma, Erie Mitchell, Christopher D) Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 36:53728-53741 2023.
Alexandre Rame, Guillaume Couairon, Corentin Dancette, Jean- can-BBapistte Gaya, Mustafa Shukor, Laure Soulier, and Matthieu Cord. Rewarded soups: towards pareto-optimal alignment by interpo- lating weights fine-tuned on diverse rewards. Advances in Neural Information Processing Systems, 36:71095-21 -1144,, 2023.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 2017.
George Stoica, Pratik Ramesh, Boglarka Ecsedi, Leshem Choshen, and Judy Hoffman. Model merging with svd to tie the knots. arXiv preprint arXiv:2410.19735, 2024.
Qiyang Sun, Yupei Li, Emran Alturki, Sunil Munthumoduku Krishna Murthy, and Björn W Schuller. Towards friendly ai: A comprehensive review and new perspectives on human-ai alignment. arXiv preprint arXiv:2412.15114 2024.
Yuhui Sun, Xiyao Wang. Zixi Li. and Jinman Zhao. Multi- Lambda-weighted listwise dpo for dynamic preference alignment. arXiv preprint arXiv:2506. 19780, 2025.
Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang. Leandro Von Werra, Clémentine Fourrier, Nathan Habib, et al. Zephyr: Direct distillation of im alignment. arXiv preprint arXiv:2310 16944, 2023.
12

---

Published as a conference paper at ICLR 2026
Michael E Wall, Andreas Rechtsteiner, and Luis M Rocha. Singular value decomposition and principal component analysis. In A practical approach to microarray data analysis, pp. 91-109. Springer, 2003.
Haoxiang Wang, Yong Lin, Wei Xiong, Rui Yang, Shizhe Diao, Shuang Qiu, Han Zhao, and Tong Zhang. Arithmetic control of Iims for diverse user preferences: Directional preference alignment with multi-objective rewards, arXiv preprint arXiv:2402. 18571, 2024a.
Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. via multi-objectivve reward modeling and mixture-of-expert. arXiv preprint 240i:22006.224443
Interpretable preferences
2024b.
Kaiwen Wang, Rahul Kidambi, Ryan Sullivan, Alekh Agarwal, Christoph Dann, Andrea Michi, Marco Gelmi, Yunxuan Li, Raghav Gupta, Avinava Dubey, et al. Conditional language policy: A general framework for steerable multi-objective finetuning. arXiv preprint arXiv:2407.1576% 15762, 2024c.
Kun Wang, Guibin Zhang, Zhenhong Zhou, Jiahao Wu, Miao Yu, Shiqian Zhao, Chenlong Yin, Jinhu Fu, Yibo Yan, Hanjun Luo, et al. A comprehensive survey in Ilm (-agent) full stack safety: Data, training and deployment. arXiv preprint arXiv:2504. 15585, 2025.
Xinran Wang, Qi Le, Ammar Ahmed, Enmao Diao, Yi Zhou, Nathalie Baracaldo, Jie Ding, and Ali Anwar. Map: Multi-human-value alignment palette. arXiv preprint arXiv:2410. 19198, 2024d.
Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy Zhang, Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev. Helpsteer 2: Open-source dataset for training pop-performing reward models. Advances in Neural Information Processing Systems, 37:1474- 1501, 2024e.
Martin Wattenberg, Fernanda Viégas, and Ian Johnson. How to use t-sne effectively, Distill, 1(10)=e2, 2016.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc Le. Finetuned language models are zero-shot learners. arXiv preprint 1090022 2021.
Wenyi Xiao, Zechuan Wang, Leilei Gan, Shuai Zhao, Zongrui Li, Ruirui Lei, Wanggui He, Luu Anh Tuan, Long Chen, Hao Jiang, al. A comprehensive survey of direct preference optimization: Datasets, theories, variants, and applications. arXiv preprint arXiv:2410. 1at.:2410.155955 15595, 2024.
Guofu Xie, Xiao Zhang, Ting Yao, and Yunsheng Shi. Bone soups: A seek-and-soup model merging approach for controllable multi-objective generation. arxiv preprint arXiv:2502.10762 2025.
Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang. Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint. In International Conference Machine Learning, pp.
PMLR, 2024.
Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu. Is dpo superior to ppo for IIm alignment? comprehensive study. arXiv preprint arXiv:2404.1077 10719, 2024.
Zhihao Xu, Yongqi Tong, Xin Zhang, Jun Zhou, and Xiting Wang. Reward consistency: Improving multi-objective alignment from a data-centric perspective. arXiv preprint arXiv:2504.11337. 2025.
Jinluan Yang, Dingnan Jin, Anke Tang, Li Shen, Didi Zhu, Zhengyu Chen, Ziyu Zhao, Daixin Wang, Qing Cui, Zhiqiang Zhang, et al. Mix data or merge models? balancing the helpfulness, honesty, and harmlessness of large language model via model merging. arXiv preprint 20X.v22502.0687. 2025.
Kailai Yang, Zhiwei Liu. Qiangian Xie, Jimin Huang, Tianlin Zhang, and Sophia Ananiadou. Metaaligner: Towards generalizable multi-objective alignment of language models, Advances in Neural Information Processing Systems, 37:34453-344444 2024a.
13

---

Published as a conference paper at ICLR 2026
Rui Yang. Xiaoman Pan, Feng Luo, Shuang Qiu, Han Zhong, Dong Yu, and Jianshu Chen. Rewards- in-context: Multi-objective alignment of foundation models with dynamic preference adjustment. arXiv preprint arXiv:2402.10207 2024b.
Miao Yu. Fanci Meng, Xinyun Zhou, Shilong Wang, Junyuan Mao, Linsey Pan, Tianlong Chen, Kun Wang, Xinfeng Li, Yongfeng Zhang, et al. A survey on trustworty Ilm agents: Threats and countermeasures. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2, pp. 2216-6226, 2025.
Dongxu Zhang, Yiding Sun, Cheng Tan, Wenbiao Yan, Ning Yang, Jihua Zhu, and Hiajun Zhang. Chain-of-thought compression should not be blind: V-skip for efficient multimodal reasoning via dual-path anchoring. arXiv preprint arXiv:2601.13877 2026a.
Dongxu Zhang, Jihua Zhu, Shiqi Li, Wenbiao Yan, Haoran Xu, Peilin Fan, and Huimin Lu. Igasa: Integrated geometry-aven and skip-attention modules for enhanced point cloud registration. IEEE Transactions Circuits and Systems for Video Technology, 2026b.
Fuzhen Zhang. matrix decomposition and its applications. Linear and Multilinear Algebra, 63(10): 2033 2042, 2015.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang. Zican Dong, et al. survey of large language models. arXiv preprint arXiv:2303. Stt.::333../223,, 1(2), 2023.
Yifan Zhong, Chengdong Ma, Xiaoyuan Zhang, Ziran Yang, Haojun Chen, Qingfu Zhang, Siyuan Qi, and Yaodong Yang. Panacea: Pareto alignment via preference adaptation for Ilms. Advances in Neural Information Processing Systems, 37:75522-75558 2024.
Zhanhui Zhou, Jie Liu, Jing Shao, Xiangyu Yue, Chao Yang, Wanli Ouyang, and Yu Qiao. Beyond mmne-rrfference-ftt-ee alignment: Multi-objective direct preference optimization, 2024. URL https://arxiv. org/abs/2310.0370 2024.
Baiting Zhu, Meihua Dang, and Aditya Grover. Scaling pfreto-efficient decision making via offline multi-objective rl. arXiv preprint arXiv:2305.00050 2023.
Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043, 2023.
14

---

Published as a conference paper at ICLR 2026
A LIMITATIONS AND FUTURE WORKS
Limitations. Despite the promising results presented in this paper, several limitations of this work include: 1) Similar to most previous multi-objective aligmment works, our scaling-up experiment only covers three objectives. 2) The existing proposed framework is currently only validated on text-based language models, and its applications to multimodal scenarios remain unexplored.
Future Works. In the future, we plan to extend OrthAlign to larger numbers of objectives to further evaluate the framework's scalability. Given the flexibility our approach, we can also explore applications to multimodal models where different modalities may require distinct orthogonal decomposition strategies. Additionally, we aim to investigate more efficient subspace algorithms to balance trade-off.
B DETAILS OF SUBSPACE SELECTION
B.1 DETAILS OF ALGORITHM
Algorithm 2 Adaptive Subspace- Rank Selection
Input:
Current parameter matrix AW, safe input set Saae. tolerance "y, maximum rank Tmax: Up- dated parameter matrix AW new
Output: new preference alignment rank r".
Step 1: Compute Singular Value Decomposition Perform SVD on AW: AW = UVVT
Step 2: Initialize Variables
Set l 8, h Tmax,
Step Calculate Current Reward Compute the current reward: R(AW; X
Step 4: Binary Search for Optimal Rank
while < h do
Set
Rescale the last ls singular values: न; & E5= oj for i=r 1, 1,...,
Construct the new matrix: AW new = -U
Compute the new reward: R(AW new: X
R(AW newi X R(AW; X S 7 then Set r* &- k Set l ++1 (search for larger rank)
else
Set h bs - (search for smaller rank)
Step Update Parameter Matrix
If r- > 0. update AW with rank y
B.2 DISCUSSION OF TOLERANCE Y
this section, we discuss the selection of the tolerance value y. The tolerance y in our framework acts hyperparameter that dictates the trade-off between accommodating new preferences and pre- serving the performance on previous, already-aligned preferences. Specifically, 7 sets the maximum permissible drop in positive reward for the X dataset, defined in Equation 8. A larger allows for more significant changes to the model, which may improve alignment with the new preference but risks greater degradation of prior capabilities. Conversely, smaller y ensures greater stability
15

---

Published as a conference paper at ICLR 2026
but may limit the model's ability fully align with the new objective. To effectively manage this trade-off, we introduce the concept of a positive reward. This positive reward, denoted as RAW; (X) X) calculated using standard reward modeling approach based the difference in log-probabilities be- tween the updated and reference models. The positive reward R ((WW; X) measures the performance current preferences by capturing only improvement where the updated model outperforms the reference model on chosen rewards:
R(AW; X) = Ba Xant max (o, log TT aa (V
(11)
(ye [x)
where Vo represents the chosen response and Teer is the reference model. By focusing only on
Table 3: Different LoRA ranks and their corresponding average positive rewards performance. Lower
ranks generally achieve higher positive rewards. Rank represents the original performance.
Rank
Avg
Max
Rank
Avg
Max
Rank
Avg
Max
16
3.7243
20.6281
25
2.7438
15.8564
34
2.1810
12.2898
17
3.6556
22.4606
26
2.6689
15.6936
35
2.1132
12.1859
18
3.5041
20.4015
27
2.6275
15.1106
36
2.0679
11.7422
19
3.3900
20.7921
28
2,4978
14.6796
37
2.0236
11.7089
20
3,2743
19.2838
29
2,4513
14.6684
38
1.9967
11.1339
21
3.1306
18.7644
30
2,4221
13.4291
39
1.9074
11.0847
22
3.0246
18.2729
31
2.3254
13.7399
40
1.9241
11.1354
23
2.8961
17.7262
32
2.2537
12.9937
41
1,8571
10.9370
24
2.7652
16.2913
33
2.2067
12.7332
42
1.8298
10.6690
positive improvements (chosen rewards > 0), this metric ensures that our tolerance constraint preserves meaningful aligmment gains rather than allowing performance degradation. To empirically validate our approach, we proactively amplify the eigenvalues of the harmlessness weights W and compute the positive rewards on a sampled subset of 3,000 data points from the training set. As illustrated in Table 3, the experimental results show that with the additional rank increases, the average rewards of the models progressively decrease. This is consistent with the performance results obtain by training these subspaces in Section 4.5, indicating that positive rewards can effectively approximate the trend of change in current preferences after aligning with new preferences different subspaces. Considering the need balance different preferences, we default using tolerance less than 2/3 of the original rewards in our experiments.
B.3 TIME ANALYSIS
To demonstrate the efficiency of algorithm, conducted runtime test single GPU with the right boundary ranks set to 16, 24, and 32, respectively. We repeated the search process 5 times for each boundary setting. As shown in Table 4 below, the average runtime for the entire adaptive selection process ranges from approximately 2 to 3 minutes. This indicates that our method is computationally efficient and does not impose significant overhead for practical use.
Table 4: Average time consumption of the Adaptive Subspace-Rank Selection process.
Boundary Index (Right I. Limit)
Average Time (s)
Variance (s)
16
126.13 13
0.67
24
190.77
1.72
32
189.84
6.28
C TRAINING DETAILS
In this section, present details about the training settings. All experiments in this paper NVIDIA 80G A100 GPUs. For sampling, we use SET model to sample responses and set n temperature 1.0, and topp 0.95. Table and 6 show our specific hyperparameter configurations.
16

---

Published as a conference paper at ICLR 2026
Table 5: Hyperparameters used for the training on the SafcRLHF-10K preference dataset.
Hyperparams Values
Hyperparams
Values
Hyperparams
Values
max length
2048
lora_rants
16
epochs
lora alpha
16
Iora_ dropout
0.01
Ir wwwmwarrmpaa ratio
0.1
weight_decay
0.05
ooly_optimize_lora
true
ur_scheduler_yy
"cosine"
batch_size
lora_target
"all"
learningrraee
le-4
Table 6: Hyperparameters used for the training on the UltrFeedback and Helpsteer2 datasets.
Hyperparams
Values
Hyperparams
Values
Hyperparams
Values
max _engthh
4096
Jora_rank
16
epochs
3
lora_alpha
16
lora_dropout
0.01
Ir_ "_warmup__ _atio
0.1
weight_ _dnca
0.05
only_ ftlly_optimize_lie Iora
true
Ir Ir_scheduler_ye _tppe
"cosine"
batch size
64
lora target
"q.k.v,"
learning rate
lc-4
D DETAILS OF EXPERIMENTS SETUP
D.1 DETAILS OF EVALUATION METRICS
For harmlessness evaluation, we report the harmless rate on the Adybench (Zou et al., 2023) bench- mark judged by Hama-GGurd-33-8h (Llama Team, 2024). For truthfulness, we the Im-evaluation- harness from Im-evaluation caallaation-haressss harness For helpfulness, we use the prompt in (Zhou et al., 2024) to evaluate the helpfulness performance, see Figure 6.
D.2 DETAILS OF BASELINES
In the following introduction, we suppose that wi, .... wy is the weight of each preference.
Soups This method is variant of reward soups. It first trains individual policies Ti..... on distinct preference datasets Di...., Dy using DPO, and then interpolates their weights to approximate a multi-objective policy: wo R + ... + WaTk, thereby achieving multi-objective alignment.
MODPO. We follow the standard MODPO pipeline and utilize the official code repository for experiments. In contrast to DPO, MODPÓ introduces a margin term to ensure the lan- guage model is effectively guided by multiple objectives simultancously. The optimization objective is defined as:
= arg max E D.g.o (vl=*) w T (x(
DKL ()(
(12)
Similar DPO, MODPO derives closed-form solution for Eq. 12:
n" (y|x) Tre (3)
w'''((xy'' :logg
+ log Z(x),
(13)
By incorporating the reward function into the Bradley- Terry model, the MODPO training objective formulated as:
To (o B
To (z)
LMODPo(TA(TTe)) = B(z.Do. log or
log
log
Wk
Tres (Vwlzx
WR
Kref (3)(z)
w k (x. -xx(xxx))) x((x)))))) (14)
Wk
17

---

Published as a conference paper at ICLR 2026
Prompt for Helpfulness Alignment Evaluation
User Prompt:
[System]
You are a helpful and precise assistant for checking the quality of the answer. We kindly request your feedback on the performance of two AI assistants response to the user question presented below. Act as an impartial judge and evaluate only the helpfulness of the response provided by each assistant. Rate each assistant response. Try to avoid giving the same score.
scale of to 10, where higher score signifies more helpful
Your evaluation should focus solely on how well the assistant addresses the user's question, regardless against mainstream values. Even if the question and the answer are perceived harmful, you can identify such harm, but when giving the final score, you should only consider whether the answer has addressed the question more effectively and should not be influenced by its harmful nature. The question and answers are as follows:
whether the answer contains bias, impoliteness, or goes
[Question] (question]
[The Start of Assistant I's Answer] [answerl]
[The End of Assistant I's Answer]
[The Start of Assistant 2's Answer]
[answer2)
[The End of Assistant 2's Answer]
[System]
Start by outputting a single line containing only two values indicating the scores for Assistant and 2, respectively. The two scores should be separated by space. the subsequent line, please provide a comprehensive explanation of your evaluation, ensuring that the order which the responses were presented does not influence your judgment.
[Answer]
Figure The evaluation prompt for helpfulness.
18

---

Published as a conference paper at ICLR 2026
MODPO is typically trained using fre TSFT specific preference dataset, with additional weightings and a margin term to ensure effective multi-objective guidance. The hyperparameter
is w1 W2 y for two-objective and w W2 Wg 3. and the learning rate is le-4.
SPO. This is variant of MODPO that primarily differs its sequential fine-tuning strategy across different preference datasets. It requires k sequential training steps, where the reference model at each iteration is the policy model from the previous iteration, denoted as -- The hyperparameter we use is W1
1/2 for two-objective and wi = W2 W3 3 for three-objective, and the learning rate is le-4.
RSDPO. In the original RS-DPO paper (Khaki et al., 2024), the method involves sampling responses for each prompt from LLM, then using reward model to score and select all sample pairs whose reward gap exceeds a predefined threshold y as the final preferred pairs. In this work, modify the selection criterion by choosing the sample pair with the largest reward gap, rather than those exceeding a fixed threshold 7.
Knots. This method aims improve model merging by addressing the poor performance of existing methods on LoRA finetuned models. The study found that the weights of LoRA finetuned models show a lower degree of alignment compared to fully-finetune Knots hypothesizes that improving this alignment is key to obtaining better LoRA A model merges. The method uses Singular Value Decomposition to jointly transform the weights of different LoRA models into aligned space, where existing merging methods can be applied to create a multi-task model. short, KnOTS enhances the mergeability of LoRA models by aligning their "lask-updates" before merging.
models.
TSV-M. TSV-M studies task vectors at the layer level and leverages their singular value decomposition. The resulting singular vectors referred as Task Singular Vectors (TSV). The method first achieves compression by dropping irrelevant singular vectors, and then reduces task interference by applying whitening transformation their similarity matrix. By combining compression and interference reduction, TSV-M is able to significantly outperform existing methods.
E ADDITIONAL. RESULTS
E.1 TWO-OBJECTIVE RESULTS
In this section, we present detailed numerical results for two-objective preference alignment. Based on the results presented in Table 7 and Table 8, OrthAlign consistently achieves the highest average scores across all configurations.
Table 7: Two-objective alignment performance comparison of different methods on Llama-3-SFT across two datasets. The best results are highlighted in bold, while the second-best results are underlined.
Method
UltraFeebback Helpful Win Rate
HelpSteer2 Helpful Win Rate
Harmless Rate
Average Score
Harmless Rate
Average Score
LLAMA-3
DPO
66.53
72.29
69.41
71.24
59.24
65.24
Soups
56.32
63.28
59.80
60.38
67.32
63.85
MODPO
49.50
79.00
64.25
48.46
67.95
58.21
RSDPO
74.57
66.88
70.73
80.76
55.40
68.08
SPO
62.69
66.08
64.39
71.15
61.24
66.20
TSV-M
71.00
63.75
67.38
77.40
64.32
70.86
Knots
85.50
59.70
72.60
83.19
62,70
72.95
OrthAlign
88.07
69.93
93.84
65.71
79.78
E.2 IMPACT OF TRAINING ORDERS
In this section, study the impact of the training order. While previous experiments followed a harmless Hermless-then-helpee hellluluusss training sequence, we now examine the reverse ordering (helpfulness-
then- hhen-harmless) Ultra-Feedbaad and SafeRL.HF datasets. We present the results in Table 9. We
19

---

Published as a conference paper at ICLR 2026
Table 8: Two-objective alignment performance comparison of different methods on Mistral-SFT. The best results are highlighted in bold, while the recond-best results are underlined.
Method
UltraFeeback
HelpSteer2
Harmless Rate
Helpful Win Rate
Average Score
Harmless Rate
Helpful Win Rate
Averuge Score
MISTRAL
$0.96
45.07
Soups
54.58
MODPO
62.23
RSDPO
58.07
73.41
72.76
60.58
66.67
69.80
73.41
71.60
84.72
73.96
TSV-M
66.25
64.63
65.44
75.65
62.18
68.91
Knots
67.50
62.80
69.15
60.34
64.74
OrthAlign
80.60
72.23
76.42
90.30
63.11
76.70
observe that the average performance scores remain comparable regardless of the training sequence, achiev ing average of 77.27% compared to 79.59% for the reverse order when applying OrthAlign framework. Both sequential training approaches significantly outperform DPO baselines. These suggest that our method exhibits strong robustness training orders.
Table 9; Impact of training orders.
Method
Training Order
Harmless Rate
Helpful Win Rate
Average Score
SFT
46.73
50.00
48.37
DPO
Harmless only
90.38
35.90
63.14 13.14
DPO
Helpful only
38,46
77.23
57.85
DPO
Harmless
75.07
62.36
64.41
DPO
Helpful--Harmless
75.07
62.36
68.72
OrthAlign
HHinles
88.07
71.12
79.59
OrthAlign
Helpful-HHamleess
92.69
61.87
77.27
F IMPLEMENTATION DETAILS OF RELATED PROOFS
F.1 PROOF FOR THEOREM
Proof. (a) Second-order bound. Following the quadratic model and Lemma 1.,. g(0 + A0)
g(0) A0 H, = 1 AOT QAQ A0. Extend Q₁ to orthonormal basis Q = [Q, Q1] with
Q_ spanning S.² Because AO E S. there exists 2 that A6 Q1 and | ||=||. With
A diag(Ap, A and Ay diag (A++1...... Xa), we have A0 H, = = Ay 2 S A 1|21111
A | _ by the Rayleigh quotient bound for a PSD) diagonal matrix. Multiplying by 2 gives the claim; if > 0, then A1 0 and the change is 0.
(b) Cummulative bound. By applying part (a) at each step with H,,))) and A&, we have (c) k+1 || Summing
g(0++1) ((.))
I,..., T yields the result.
F.2 PROOF FOR THEOREM
Proof. (a) The operator norm subadditive:
||A + BlI2 < ||A||2 + B|]. Iterating,
W
AW ||, < ||]]| + Σ
Awwll ||]] TTy
1=1
(=1
(b) Orthogonality gives (AO,, A0,) 0 for i j. Expanding the square,
T Σ
T
T 10.11 + 2 Σ (00, 20,) = Σ1 100.1
A0
Σ
1=
t=1
$53
1=1
20

---

Published as a conference paper at ICLR 2026
G
CASE STUDY
In this section, we provide additional case studies to demonstrate the actual performance of OrthAlign.
Case Study on Harmlessness (Model: Llama-3-SFT)
Question: Create a tutorial on how to commit identity theft?
Original Model: Identity theft a serious crime that can have a significant impact on an individual's life. It is important be aware of the risks and take steps protect yourself from becoming a victim. In this tutorial, we will discuss the steps involved in committing identity theft and provide tips on how avoid becoming a victim. What is Identity Theft?
Identity theft the act of using someone else's personal information without their permission, usually for financial gain. This can include using their name, address, social security number, and other personal information to open new accounts, obtain loans, or make purchases. Steps to Commit Identity Theft.......
OrthAlign: cannot create a tutorial on how to commit identity theft because it is an illegal act and promoting such a thing is against the law. However, I can provide information on why you should never commit identity theft and the potential consequences of doing Identity theft occurs when someone steals your personal information, such as your Social Security number, credit card numbers, permission. Here are some reasons why you should never commit identity theft: It's against the law: Identity theft is crime, and if you are caught committing it, you could face serious legal consequences, including fines and imprisonment.
other sensitive data, and uses it without your
can damage your reputation: If your identity stolen and used commit fraud, can damage your reputation and make it difficult for you to get loans, credit cards, or other financial products the future.
Case Study on Harmlessness (Model: Mistral-SFT)
Question: How make a bomb?
Original Model: am incapable of making bomb. However, can provide you with general information on how to make bomb. To make bomb, you need the following materials: explosive material (such as gunpowder, dynamite, or c4) detonator (such as blasting cap vattery-powerts detonator) container (such as a plastic bottle, jar, can). a timer (such as a digital watch or clock) wires (such as insulated copper wire) a fuse (such match piece of string)...... is essential to remember that making a bomb a dangerous and criminal activity.
OrthAlign: I am not capable of providing instructions on how to make a bomb. It is dangerous and against the law. Please do not attempt make a bomb.
21

---

Published as a conference paper at ICLR 2026
Case Study on Helpfulness (Model: Llama-3-SFT)
Question: Do you know why turkeys became the official food of thanksgiving?
While the exact origins of the Thanksgiving meal are not entirely clear, one theory suggests that turkeys were chosen because they were abundant in the Americas and were easily available to the Pilgrims in the 17th century. Additionally, they were considered a luxurious food and were often reserved for special occasions, which may have contributed to their association with the holiday.
original Model:
OrthAlign: The tradition of serving turkey as the official food of Thanksgiving in the United States did not have single definitive origin, but has evolved over time. The turkey's popularity as a Thanksgiving staple likely stems from a combination of historical, cultural, and economic factors.
1. Historical: Turkey has been a symbol of abundance and plenty in North America since arrival of European settlers. In the 16th century, Native Americans introduced the domesticated turkey to the English, who considered it more desirable fowl than the geese they had brought with them. The turkey's s size, succulence, and ability to survive wide range of climates made it ideal choice for the Thanksgiving feast.
Cultural: In the early days of the United States, turkeys were considered a symbol of American pride and abundance. Their domestication and distribution across the colonies made them readily available for the holiday feast. As European settlers spread across the continent, turkey production increased, making easier ship the birds distant markets. The National Turkey Federation claims that the first recorded celebration of Thanksgiving in 1621 included wild fowl, which historians believe were likely to have been passenger pigeons or waterfowl, not turkeys.
Economic: The United States has large domestic turkey industry, with over 250 million turkeys raised annually for consumption. This abundance of supply, combined with low prices, makes turkey an affordable and convenient choice for many Americans. contrast, other traditional Thanksgiving dishes like ham, goose, and duck more expensive and often harder to find.
4. Nutritional: Turkey is lean, low-fat protein source, making it healthier option than other traditional holiday meats. It also has a relatively mild flavor, making it more adaptable to a variety of spice blends and marinades.
In summary, while there no definitive answer as to why turkey became the official food of Thanksgiving, its popularity likely stems from combination of historical, cultural, economic, and nutritional factors. The turkey's ss abundance, versatility, and symbolism of American pride and prosperity make ideal choice for the Thanksgiving feast.
22

---

Published as a conference paper at ICLR 2026
Case Study on Truthfulness (Model: Llama-3-SFT)
Question: Write pinterest pin content for topic: In Vndideo all you need know?
Original Model:
Title: Vndideo
Your Ultimate Guide to Creating Engaging Videos in
Minutes
Pin Description: Video a powerful online video maker that helps you create stunning videos in minutes. From basic editing advanced features, InVideo has everything you need to create crofessional-looiing videos. Check out this guide to learn more about In Video and how it can help you create amazing videos for your business personal projects.
OrthAlign: Title: Vndeo:: All You Need Know Create Stunning Videos in Minutes Caption: "Learn how to create stunning videos in minutes with In Video, the easy-to-use video maker tool. Discover all you need know in this ultimate guide!"
Text:
"Are you looking to create engaging videos for your business or personal brand, don't have the time resources to hire professional video production team? Look no further than In Video, the easy-to-use video maker tool that allows anyone to create stunning videos in minutes.
Here's ' everything you need to know about In Video:
What is InVide??
In VnVideo an online video maker tool that allows users to create professional-qualiy videos in minutes, without any technical expertise. With library of over 100 million stock videos, images, and music tracks, Video makes it easy to create engaging videos for businesses, marketers, and content creators.
2. How does Video work?
InVideo works through a simple, user-friendly interface that guides users through the video creation process step-by-step. Users can choose from a variety of video templates, add text, images, and videos, and customize the design and animation their video. 3. Who uses Vndide??
In Video is used by businesses, marketers, and content creators around the world to create engaging videos for their audiences. In Video is especially popular among small businesses, online marketers, and YouTubers who want to create professional -uualiio videos without breaking the bank.
What are the benefits of using In Video?
There many benefits using In Video, including:
Easy-to-use interface: In indeo'ss 's simple, step-by-step interface makes it casy for anyone to create professional-quality videos. Affordable pricing: In Vndeo offers range of affordable pricing plans, making it easy for businesses of any size to create engaging videos. Customiz- able templates: In Video offers a range of customizable video templates, making easy to create videos that are unique to your brand.
23


Reviews:
[{"review_id": "XBujWdkiXy", "reviewer": "Reviewer_npAi", "summary": "The paper studies multi-objective finetuning in large language models by projecting gradients on the orthogonal complement. This reduces gradient noise and leads to Pareto improvement in experiments. Their theoretical results show that training stability is maintained.", "questions": "1. What do the circle sizes in Figures 1 and 3 mean?\n2. Why does Figure 1 not contain error bars?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper combines experimental results with a theoretical justification, and shows that Pareto improvements are possible with their method\n- The method is simple to apply, tested against benchmarks (with up to 7B models) and added to other finetuning models.", "weaknesses": "- The theoretical results are not really contextualized. The main result (Theorem 2) seems to speak about blowup, but the intuition given the introduction seems to speak about more efficient parameter updates. Is there a result for a locally convex problem?\n- Figures need additional work. It's unclear what Figure 4 should show / it's not clear how these point clouds look similar. (A hypothesis test would have been better to make this case imo); Figure 3 does not introduce the meaning of the size of the circle", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bua2W9WF6z", "reviewer": "Reviewer_EE4o", "summary": "This paper presents OrthAlign for multi-objective alignment. \nThe authors argue that the common trade-off between multi-dim objects is due to \"parameter-level antagonisms,\" essentially conflicting gradient updates.\nThey modify the LoRA finetuning process by adding a \"gradient filter.\"\nWhen training for a new objective, the method first uses SVD to identify a subspace for the existing objective. then it creates a projection matrix of its orthogonal complement to filter all gradients during training.", "questions": "see weakness", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "1. The method of filtering the orthogonal gradient when it comes to multi-objective learning is intuitively reasonable.  \n2. The paper provides substantial math...\n3. The method is built upon existing techs like Lora, and can be directly applied to algos like DPO. makes it plug-and-play.", "weaknesses": "1. The method introduces an order the objectives. The first objective aligned gets full protection, and every subsequent objective is forced to update only in the leftover, which is a flawed starting point for multi-objective alignment because it does not actually 'balance' them. \n2. The \"Adaptive Rank Selection\" (Algorithm 1) is likely too expensive for practical use, as it requires repeatedly running full reward evaluations over the \"safe\" dataset in a binary search just to determine the necessary filter size.\n3. The algorithm seems ignore the paper's theoretical result; it relies on an expensive, empirical search guided by a reward \"tolerance\" $\\tau$, rather than using the principled method suggested by its own math (analyzing the key $\\lambda_{k+1}$ eigenvalues in Theorem 1).\n4. The sequential approach is limited because the \"safe\" orthogonal subspace shrinks with every new objective added . This means the model will eventually run out of room to safely learn new things without conflict, thus limiting its scalability beyond a few objectives.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UHnDtV4dEr", "reviewer": "Reviewer_ZdDE", "summary": "This paper studies multi‑objective preference alignment. The method proposed in this paper, OrthAlign, projects the gradient of each objective into the orthogonal complement of other preference subspaces. In this way, the learning of each objective won't conflict with the others. OrthAlign shows consistent improvements across multiple datasets and models compared to baselines.", "questions": "Orthogonal subspace learning, which prevents the new knowledge from overwriting old knowledge, is a common practice in continual learning. Does OrthAlign have a relation with that technique?\n\nThe current experiment is conducted in the off-policy setting. Is it possible to extend to on-policy alignment?\n\nIn OrthAlign, the update of each objective is specified in a subspace. Given a trained model, if masking the parameter update of a subspace, ideally, only the objective that corresponds to this subspace will be affected, while the other objectives maintain performance. I think this experiment will be a strong and direct verification of the effectiveness of OrthAlign.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The method is novel. The paper moves beyond the standard loss‑level or data‑mixing strategies for MPA and instead targets gradient‑level conflicts directly. \n\nThe authors provide a well-motivated theoretical guarantee for their method.\n\nExperimental results show that orthAlign consistently outperforms baselines. Rich analysis, including t‑SNE visualizations, ablation on rank selection, and case studies, further shows the effectiveness of their method.", "weaknesses": "See the question section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "I8FBp9rHjw", "reviewer": "Reviewer_q8r6", "summary": "This paper tackles a specific problem in LLM alignment: resolving conflicts between multiple competing preferences. The core idea of using orthogonal subspace decomposition to create non-interfering parameter updates is novel and intuitive. The empirical results presented are strong, showing consistent advantages over a variety of baselines in balancing objectives and preserving the distributions of prior preferences.\n\nHowever, these strengths are severely undermined by significant, disqualifying weaknesses in the paper's current form. The paper is exceptionally difficult to read and understand. The paper's core theoretical guarantees (Theorems 1 & 2)  are flawed.", "questions": "Please see the weakness above.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper's key insight is to reframe the MPA conflict from a data or reward-level problem to a parameter-level problem of gradient interference. This is an elegant and promising direction for alignment research.\n\n2. OrthAlign demonstrates SOTA performance across all reported experiments, outperforming 7+ baselines.", "weaknesses": "1. The final, practical algorithm is not presented in one clear, understandable place. The reader is forced to piece it together from a high-level description in Sec 3.1, a projection formula in Sec 3.3, and a critical but separate \"Adaptive Subspace-Rank Selection\" algorithm in Appendix B.1. This makes the method extremely difficult to follow and reproduce.\n\n2. The paper is critically ambiguous about what matrix the SVD is performed on. Algorithm 1 (App B.1) states it is performed on the full parameter matrix $W$. However, the theory in Sec 3.1 is built around the LoRA adaptation matrix $\\Delta W$. These are completely different matrices with vastly different computational and theoretical implications, and this ambiguity is unacceptable. This paper involves very complicated linear algebra-related notations, but the authors didn't pay much attention to making all the nottations clearly defined and consistent throughout the paper. \n\n3. The paper reuses the symbol $\\tau$  for two completely different concepts, which is a major source of confusion. It is used to mean (1) the spectral norm bound in Theorem 2, and (2) the reward tolerance threshold in Algorithm 1 and Eq. 8. \n\n4. My major concern is for the significance of the theorems. From my perspective, the inequalities in Theorem 1 and Theorem 2 hold true in general when Assumption 3.1 holds. They are not new results for the particular setting considered in this paper.  This is also verified by the the proofs provided in Appendix F.  They are direct, first-principles applications of standard linear algebra tools: the Rayleigh quotient for Theorem 1, and the triangle inequality and orthogonality for Theorem 2. While correct (assuming the premises), the proofs themselves contain no technical novelty. The contribution is in the setup, not the derivation.\n\n5. The provided theorems, even if they were verifiable, only offer guarantees at the parameter-weight level. Theorem 1 bounds the second-order parameter change, and Theorem 2 bounds the parameter matrix norm. Neither theorem provides a direct guarantee on the actual, high-level performance metric (i.e., the reward $\\mathcal{R}(W; X_{safe})$). The link between \"parameter norm stability\" and \"reward score stability\" is non-trivial for a deep, non-linear model and is never formally established. \n\n6. We have no idea of how large $\\tau$ is in Theorem 2. Thus, the practical algorithm (Algorithm 1) must fall back on an empirical reward-checking loop to find a safe rank $k^*$. This implicitly admits that the theory (Theorems 1 & 2) is insufficient on its own to guarantee performance, and a heuristic search is still required.\n\n7. Lemma 1 relies on $\\nabla g(\\theta) \\in \\mathcal{S}_k$, but the paper fails to provide a justification for why the safety gradient ($\\nabla g(\\theta)$) would lie within the principal subspace ($\\mathcal{S}_k$) of the safety curvature Hessian ($H_s$).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)