Heterogeneous Label Shift: Theory and Algorithm
Authors: Chao Xu
Xijia Tang
Chenping Hou
Venue: ICML 2025 poster (2025)
Keywords: Open environment; Heterogeneous domain adaptation; Label shift; Generalization theory.
Primary area: theory->domain_adaptation_and_transfer_learning
TL;DR: Research on Heterogeneous Cross-modal Knowledge Transfer in Open Scenarios.

Abstract: In open-environment applications, data are often collected from heterogeneous modalities with distinct encodings, resulting in feature space heterogeneity. This heterogeneity inherently induces label shift, making cross-modal knowledge transfer particularly challenging when the source and target data exhibit simultaneous heterogeneous feature spaces and shifted label distributions. Existing studies address only partial aspects of this issue, leaving the broader problem unresolved. To bridge this gap, we introduce a new concept of Heterogeneous Label Shift (HLS), targeting this critical but underexplored challenge. We first analyze the impact of heterogeneous feature spaces and label distribution shifts on model generalization and introduce a novel error decomposition theorem. Based on these insights, we propose a bound minimization HLS framework that decouples and tackles feature heterogeneity and label shift accordingly. Extensive experiments on various benchmarks for cross-modal classification validate the effectiveness and practical relevance of the proposed approach.

Heterogeneous Label Shift: Theory and Algorithm
Chao Xu
Xijia Tang'
Chenping Hou
Abstract
Cross Cross-Modal Knowledge Transfer
Source Domaln
In open-envvroommnt applications, data are often collected from heterogeneous modalities with dis- tinct encodings, resulting in feature space hetero- geneity. This heterogeneity inherently induces label shift, making cross-modal knowledge trans- fer particularly challenging when the source and target data exhibit simultaneous heterogeneous feature spaces and shifted label distributions. Ex- isting studies address only partial aspects of this issue, leaving the broader problem unresolved. To bridge this gap, we introduce a new concept of Heterogencous Label Shift (HLS), targeting this critical but underexplored challenge. We first ana- lyze the impact of heterogeneous feature spaces and label distribution shifts model generaliza- tion and introduce a novel eπror decomposition theorem. Based on these insights, we propose a bound minimization HI. framework that decou- ples and tackles feature heterogeneity and label shift accordingly. Extensive experiments on var- ious benchmarks for cross-modal classification validate the effectiveness and practical relevance of the proposed approach.
Image
Encoder
Input Data
Heterogeneous Feature space
Shifted Label Distribations
Target Domain
An eag
stand ing
Encoder
branch
Figure I. Illustration for the Heterogeneous Label Shift (HLS), in which the data is with simultaneous heterogeneous feature spaces and shifted label distributions.
referred as vocabularies in different languages, and the features across domains neither the same feature space nor of the same dimensionality. Due to the differences be- tween languages, label distribution shift is inevitable. We formulate these real-world cross-modal knowledge transfer problems as the Heterogeneous Label Shift (HLS), in which the data is with simultaneous hetcrogeneous feature spaces and shifted label distributions.
Reviewing traditional domain adaptation (DA) label shift (LS) approaches, most existing works assume homoge- neous cross-domain feature space (Pan & Yang, 2010; Ben- David et al., 2010; Wang et al., 2023). The heterogeneous feature spaces block the path existing methods to directly deal with the HLS problem. To relax this assumption, het- erogeneous domain adaptation (HDA) (Dai al., 2008; Yang al., 2009; Day & Khoshgoftaar, 2017; Zhou et al., 2019), which tackle domain adaptation problems with dif- ferent cross-domain feature spaces, has been an increased focus. In such a context, plenty of practical HDA (Zhou et al., 2014; Sukhija et al., 2016; Wang & Mahadevan, 201 1; Li et al., 2019a; Shi et al., 2010: Fang et al., 2023) have been proposed. For example, Sukhija et al. proposed using shared label distributions as pivots learn a sparse feature transformation in supervised HDA framework (Sukhija et al., 2016). Xiao and Guo develop semi-supervised ker- nel matching framework that simultaneously maps the target domain instance into the source domain instances and learns prediction function on the labeled source instances (Xiao & Guo, 2015).
1. Introduction
In many real-world tasks, data are collected from open and dynamic environments, and the source and target data may exhibit simultancous heterogeneous feature spaces and shifted label distributions. For instance, as in the cross- modal classification task from images to text shown in Fig- ure 1, the source and target domains may share the same set classes but differ modality. At the same time, the the heterogeneity of modalities also causes a certain degree of label distribution shift. A similar situation also occurs in lross-language (e.g., English/Spanish) text classification task (Prettenhofer & Stein, 2010), the feature spaces are
'College of Science, National University of Defense Technol- ogy, Changsha, 410073, China.. Correspondence to: Chenping Hou <hepnudt @arruutt@@ottall hotmail.com>..
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Heterogeneous Label Shift: Theory and Algorithm
Despite the prominent performance achieved by these meth- ods in applications, applying them directly to the HLS sce- nario will suffer from the following obstacles. (I) They ignore the adaptation to the shifted label distributions in the target domain, which is crucial for practical machine learn- ing systems (des Combes et al., 2020; Zhao et al., 202 aa; Wu et al., 2021; Bai et al., 2022: Zhou et al., 2023). (2) They inherently depend on the availability of labeled data in the target domain, which is either searce even absent in many tasks. (3) The mismatched label distribution in- evitably disrupts the existing theoretical guarantee of HDA, so it is desirable to establish a theoretical guarantee that is simultaneous applicable to heterogeneous feature spaces and shifted label distributions. This paper aims to fill the gap in terms of both theoretical understanding and practi- cal methods for the heterogeneous label shift setting and thereby move step closer towards having more complete understanding on the cross-modal knowledge transfer for heterogeneous label shift data.
2. Related Works
Label shift. Label shift (Long et al., 2018b; 2019; Cai et al., 2022a;b; Garg et al., 2020; Li et al., 2019b; Zhang et al., 2021; Zhao et al., 2021b) assumes the source and target domains have different class distributions but the same feature distribution within each class. Lipton et al. (Lipton et al., 2018b) explained that label shift is simpler than covariate shift due to the simplicity of the label space and exploited arbitrary black-box predictors to estimate the importance weights using the confusion matrix. On this basis, Azizzadenesheli (Azizzadenesheli et al., 2019b) proposed regularized weight estimator, which obtains good statistical guarantees without a requirement the problem- dependent minimum sample complexity, and introduced a novel regularization method to compensate for the high estimation error of the importance weights in low target sample settings (Azizzadeneshelli 2022). Alexandari et al. (Alexandari et al., 2020) combined maximum likelihood with bias-corrected calibration and introduced a principled strategy for estimating source-domain priors that improves robustness to poor calibration. Based on the above label shift methods, Garg et al. (Garg et al., 2020) proposed a unified view of label shift estimation based on confusion matrices and maximum likelihood. These approaches are not suitable for our investigations since they are limited to homogeneous domain adaptation problems and cannot handle the heterogeneous feature spaces directly.
To solve the aforementioned HLS problem, we first ana- lyze the role that heterogeneous feature spaces and shifted label distributions play in the generalization ability of the model and present a novel error decomposition theorem. Motivated by this, we suggest a bound minimization HLS framework that decouples and tackles feature heterogene- ity and label shift accordingly. For illustration, we devise a novel Heterogeneous Label Shift Adversarial Network (HLSAN) algorithm within the framework.
Heterogeneous domain adaptation. Heterogeneous do- main adaptation (HDA) (Li et al., 2014; Zhou et al., 2014;
Extensive experiments on various benchmarks for cross- modal classification validate the effectiveness and practical relevance of the proposed approach. In summary, the contri- butions of our paper are listed as follows.
Sukhija et al., 2016; Shen & Guo, 2018; Li & Zhang, 2019;
Xiao & Guo, 2015; Tsai et al., 2016; Hsieh et al., 2016;
Yao et al., 2019; Fang et al., 2023) aims to enable effective cross-modal knowledge transfer between domains with in- herently distinct feature spaces. This framework addresses the challenges posed by feature-space heterogeneity, ensur- ing meaningful alignment and adaptation across domains. Early, Duan et aL. et al., 2014) propose a heterogeneous feature augmentation method that transforms the data in both domains into a common subspace and then augments the projected data with original features. Later, Sukhija et al. proposed using shared label distributions as pivots to learn sparse feature transformation in a supervised HDA frame- work (Sukhija et al., 2016). Xiao and Guo (Xiao & Guo, 2015) develop a semi-supervised kernel matching frame- work that simultancously maps the target domain instance into the source domain instances and learns a prediction function on the labeled source instances. Shen et al. (Shen & Guo, 2018) apply the linear transformation to map source domain features onto the target domain and propose sparse feature transformation method to tackle the unsupervised HDA problem. More recently, motivated by compatibility condition in semi-supervised probably approximately cor- rect (PAC) theory, Fang et al. (Fang et al., 2023) propose
We introduce and investigate a novel learning problem, namely, Heterogencous Label Shift (HLS), which is rarely studied and arisen from many real application areas. To our knowledge, this may be the first attempt concerning knowledge transfer in this simultaneous heterogeneous feature spaces and shifted label distri- butions scenario with a theoretical guarantee.
We present a novel error decomposition theorem that directly suggests a bound minimization HLS frame- work. Motivated by the theoretical analysis, we de- vise Heterogeneous Label Shift Adversarial Network (HLSAN) algorithm as an illustration within the frame- work.
Comprehensive experimental studies demonstrate the effectiveness of our proposal on multiple benchmarks with varying degrees of shifts for different types of cross-modal classification tasks.
2

---

Heterogeneous Label Shift: Theory and Algorithm
generalization error estimation for target risk in semi- supervised HDA and then develop two SsHeDA algorithms based on the theoretical result. These prominent approaches can not derectly tackle HLS problem either since (1) HLS considers the unsupervised scenario, the requirement of la- beled target instances remains to be a limitation for these (semi-)supert suppevised methods. (2) These methods only fo- cus on one aspect of our setting and they ignore that label distribution shifts.
Notation
Definition
X. Ne
latent, source, target feature space label space random variables on X,, A' random variables on y source, target joint distributions source, target marginal distributions discrete label distrbutions source, target feature transformation
y f1....,h
Xs, Xt
Ye, Yi
Ps. Pt
Px., Px,
(Y) P.(Y)
T.
Fe.Fi
feature transformation space
3. Problem Setting and Concepts
S
The labeled source data The unlabeled target data The unlabeled parallel instances
T
This section outlines the problem setting and key concepts, with Table providing a summary of primary notations.
Table 1. Main Notations and Corresponding Definitions
3.1. Problem Setting
Considering a general k-class classification problem, let y = {I..... 3 the label space, Denote the feature spaces of the source and target domains by X, C Rd. and X gd respectively. The source and target domains correspond two distinct joint distributions, P(X,, Y,) and P(,,, Y.). where X, E Xs, X₁ E X,, and Ys, Yi y are random variables. For simplicity, we use P. and P. for short to denote P(X,, Y,) and P(X,, Yt), respectively. Marginal feature distributions are indicated by subscripts, for example, Px, represents the marginal distribution of X The discrete label distribution is represented by specifying random variables, such as P. PY) represents the marginal distribution of Y.. Then, the HLS problem is defined follows.
space X' C Rd. we denote
F, (T, X2, & F (T Xx
source and target transformation spaces, respectively. The feature transformations T, and T, transform the heteroge- neous feature spaces into a common latent feature space and induce similar distributions on Px, and Px,
Importance Weights. In the label shift setting, importance weights play crucial role.
Definition 3.2 (Importance Weights). For discrete label distributions P.(Y) and P:(Y) of the source and target domains, the importance weight R is defined as
Problem (Heterogeneous Label Shift, HLS)). Given sets of samples, namely labeled source set, unlabeled target set, and few additional unlabeled parallel instances
w (1) = P.(Y v)
P.(Y = =y) Vy E (1, (k)...,)) ...
Hypothesis Space and Risks. Considering a general k- class classification task with a hypothesis space H consist- ing of scoring functions
S = {x 1 Px,., i.i.
T= (x pu
Px, i.i.d
h X 1x lxk x x (x), ................................ (x))
O {xx x 12
Px, Px, i.idd
where (x) = 1,... .k) indicates the confidence in the prediction of label j. Given é IR x R + R20 as the symmetric loss function, the expected risks of h E w.r.t. and the weighted expected risks
Here Px, X Px, the marginal distribution of the parallel instances and Mp & nt: Up Bt. The aim of HLS is to enable effective knowledge transfer between the hetero- geneous source and target domains, utilizing S, T, and O to develop a model that performs well in classifying data within the target domain.
loss l under (X,,, Y)) of h H w.r.t. loss l under (XT,(X.), Y,) are given by
R2 o Te) = E & (h (T, (x;)), o(8)))
R,(w, h o T.) E (y. ((. (x.)),
3.2. Concepts
where & maps a label to the corresponding one-hot vector. Correspondingly, the weighted empirical source risk are defined as
Feature Transformations. Inspired by (Ben- et al., 2010), a common strategy involves learning representations that invariant to domain shifts. For the HLS problem, begin by introducing the following definition.
(, ho T,) = Σ ((w))((h(T,(())) ((.))
Definition 3.1 (Feature Transformations). Given a latent
n2
3

---

Heterogeneous Label Shift: Theory and Algorithm
4. Theoretical Analysis of HLS
Theorem 4.3 presents decomposition of the error gap be- tween source and target domains, yielding generalization bound for the target risk R& (h, Ts, Te). This bound consists of three key components: 1) (h which measures the divergence between the conditional distributions P()[Y); 2) |/ -ww-w//2. representing the weight estimation error as the discrepancy between estimated and true importance weights; and 3) Finite sample errors. Unlike previous works [(Ben- David et al., 2010), Theorem 2; (Zhao et al., 2019), Theorem 4. which decompose the error gap based on the distance between marginal feature distributions (Px.. Px.) ) and opti- mal labeling functions (f.x , f(²), Theorem 4.3 introduces a novel decomposition that circumvents reliance on unknown optimal labeling functions, providing a distinctive and prac- tical perspective.
To dissect the HLS problem, we make a theoretical analysis of HLS and present a novel error decomposition theorem that directly suggests a bound minimization HLS framework. Due to space limitations, the detailed proofs are listed in Appendix A.
4.1. Error Decomposition Theorem of HLS
Definition 4.1 (Heterogeneous Feature Alignment, HFA). Considering two feature transformations T, and T, trans- form the heterogeneous feature spaces into a common latent feature space, i.e., X = (X,) U TX())) The representa- tion X E X satisfies HFA if
Px(X = y) = Pi(XY = y). Vy e..
Notably, the conditional error gap Acr(h) (h) can be minimized by aligning the transformed conditional feature distributions across domains. The importance weights w can be esti- mated with reference to the investigation of (Lipton et al., 201 2018a), which the authors proved to be effective for large enough sample sizes, and the distance || - w/l2 can be bounded effectively. Synthesizing conditional error gap and weight estimation error, the result suggests that, minimize the eπror gap. it suffices to align the transformed conditional distributions P(X|Y = y) while simultaneously minimiz- ing the weight estimation error. Before applying a bound- minimization algorithm inspired by Theorem 4.3, two key challenges must be addressed, i.e., Heterogeneous Feature |Po( = -'lY- = - -)-P = = f) Alignment and Importance Weight Estimation. v2y'E)²
HFA extends the concept of Generalized Label Shift (GLS) introduced in (des Combes et al., 2020), addressing the more complex challenge of aligning both feature spaces and marginal feature distributions. Specifically, HFA employs dual feature transformations to induce similar latent-space distributions for Pe and P., thereby achieving alignment of heterogeneous feature spaces.
Definition 4.2 (Conditional Error Gap). For a joint distribu- tion P and a classifier h, where Y = (X), the conditional error gap of h is defined as follows.
ACE (h) :
max
4.2. Heterogencous Feature Alignment
Under the HFA assumption, it follows that ÁCE O. We proceed to establish an upper bound on the error gap be- tween the source and target domains, which also facilitates deriving generalization bound for the target risk.
In HLS, the absence of target domain labels prevents direct alignment of conditional label distributions. By leverag- ing relative class weights between domains can provide a necessary condition for HFA, we can bypass the explicit alignment of conditional feature distributions.
Theorem 4.3 (Error Decomposition Theorem). Consider- ing a general k-class classification lask. Let H be the family of hypothesis set, and denote the hypothesis returned by the model trained with the available data S, 'T, and O as h. Suppose the loss function E is M -bounded and L-Lipschitz w.r.. Euclidean norm. Then, for any 5 > O, with probability at least ö over the source domain sample of size ns and target domain sample of size ne, the following inequalities holds for any h E H.
Lemma 4.4 (Necessary condition for HFA). Considering two feature transformations Tg and Tr, which transform the heterogeneous source and target feature spaces into a common latent feature space, i.e., X- = TX.) Tx(X,). If X X satisfies HFA, then
P((X)= = Σ w(U)Ps(X, v) =: PM"
VEY
[h T,) R, h No,,hoT))1 Ace(h)+ + || -wl12
In contrast to prior approaches that align PX( with P,(X) maximum mean discrepancy (MMD) (Long et al., 2015) or
+ +LR,, (H T.) + M/ log(1/5)
aim align P. ( with P (Long et al., 2018a),
2ng
(1)
Lemma 4.4 proposes aligning P((X) with a reweighted marginal distribution, (X).
Here, Rr (H o T.) denotes the Rademacher complexity of the hypothesis class H associated with T.. w and w denote the true importance weights and the estimated importance weights, respectively.
Theorem 4.5. Given the feature transformations T, and To the common latent feature space X = (X,) U Tx(X.). Let p = mingeyp.(Y = y) and War =
4

---

Heterogeneous Label Shift: Theory and Algorithm
mingey w(y). For hypothesis h, the following inequality holds.
probability at least 1 ô, the following inequality holds
|| wwll S ((nss n4, 110122 0),
(4)
max (AA (P.(X - =)) NEY
where
WM.,(hoT.) R. T.) (dd
P((X))
P
72k: log
12k
9k
6k
(2)
1| wl12 S v
+ c21
log
where drv and dıs denote the the total variation and distance, respectively.
8
n2y
8
(5)
and
[100112 + +1) C1 = Omin ( (C)
Theorem 4.5 confirms that using d.s to match Px(X) and (X) an appropriate objective for approximating the discrepancy of feature distribution conditioned the label. It demonstrates that if marginal feature distributions are aligned and the source error is zero, successful domain adaptation (i.e., zero target error) implies that HFA holds.
C2
(6)
Tmin
|| -wull characterizes the distance between the Irue weight and the initialized weight.
0
Theorem 4.8 establishes the effectiveness of estimating the importance weight w via RQP, demonstrating that the weight estimation error is tightly bounded. This bound is directly affected by the sample sizes n, and ni, as well as the number of classes k. Specifically, larger sample sizes ni Ti result in tighter bound, whereas an inerease in the number of classes & leads to looser bound. Moreover, the initial weight wo indirectly impacts the bound; closer alignment of wo to the true weight w reduces ||0||, further tightening the bound.
4.3. Importance Weight Estimation
Accompanied by feature transformations, can be esti- mated within a latent invariant space. Drawing inspiration from the moment-matching technique proposed by Lipton et al. (Lipton et al., 2018a) for estimating w under label shift, we develop a method to estimate w under HL. by solving regularized quadratic program (RQP). The approach begins with the following definition.
Definition 4.6. Let y = h(X) be the predictions. Denote C E | as the confusion matrix of the classifier the source domain and l E R as the distribution of predictions on the target domain, Vy. yf y
5. Deep Neural Network Implementation
This section shows how to design theoretically guaranteed algorithm, i.e., bringing HLS theory into the reality. As discussed in above Theorems, the algorithm is carried out from the following three aspects. (1) estimate w via the available data, (2) align the target feature distributions with the reweighted source feature distribution and, (3) minimize the weighted souce risk. Overall, we should consider the optimization problem as follows
Cy. = -P.Y y.
ply) : Pi( P y).
Lemma 4.7. If the Feature Transformations T, and Tr achieved HFA, and if the confusion matrix C is invertible, then w = C--
Lemma 4.7 indicates that estimating the importance weight w under HLS does not require target domain labels. How- ever, matrix inversion can be numerically unstable, particu- larly with finite sample estimates Č and j for C and μ. To address this, propose regularized quadratic program (RQP) to estimate w
min TE F.EF.
(s ho T.) @AC
According to Lemma 4.4, we can bypasses an explicit align- ment of the conditional feature distributions by aligning (x( with a reweighted marginal distribution PM (X). Then (7) can be transformed into
w = arg min f - Cw 1|2 + A||w woll
min Ra(w, ho +adx toxx (X)). E
s.t. W 2 0. w P. (Y) =
Here, Wo is an initial weight incorporating prior information, and À is parameter balancing the estimated loss with the regularization term. In contrast to the estimation method in (des Combes et al., 2020), the regularization term mitigates extreme label shift, making the estimated weight w more robust to estimation biases of Č and j..
To measure the distribution discrepancy introduce the integral probability metric ((Mller, 1997), IPM).
Definition 5.1 (H-IPM). Denote H set of real-valued functions. For two distributions D and D', the H-IPM between distributions D and D' is
Theorem 4.8. Denote w as the estimated importance weight obtained by solving the RQP in (3). Then, with
(D, dx dx (D, = sup [Ex=D[h(X]] - EX-D [h(X]] MEH
5

---

Heterogeneous Label Shift: Theory and Algorithm
Classifier
Encoder
Labeled Source Data
Loss Le +aG
Latent Space
Classification
Loss be +ack
Predictions
W
Parallel Instance
Helerogeneous Feature space
Adversarial
ibis stands in the lake
Loss Cn
gradient reversal
Unlabeled Target Data
eagle standing branch
Loss Cy
Loss E
Feature Space
Alignment
Discriminator
Output
Input
Encoder
Figure Illustration for proposed HLSAN architecture. HLSAN consists of two main components Feature Space Alignment and Feature Distribution Alignment, which form positive feedback loop, reinforcing each other to achieve an satisfactory result.
By approximating any function class H. using parametrized models, e.g., neural networks, we propose general HLS framework that is compatible with H -iinnteraal probability metrics.
5.2. Loss in HLSAN
Recalling the HLS framework formulated in Eq.(8),
min bEX.T,EF..T.C. (,(w, ho h.,)) ad (P((),, PM (X)).
5.1. Network-Based HLS Algorithm
According the investigation of (des Combes et al., 2020), di (X),),, PM (X)) ) is instantiated with the Jensen- Shannon Divergence. Specifically, the loss include domain discriminator loss L and classifier loss CV.
Inspired by domain-ad adversarial neural networks [(Ganin et al., 2016), DANN], we instantiate F, and H as fully- connected neural networks and propose a network-bor based algorithm Heterogeneous Label Shift Adversarial Network (HLSAN), As illustrated in Figure the HLSAN structure consists of two main components. The first part, Feature Space Alignment, projects the heterogeneous space into latent common space through two feature transformation networks, T.e, and Tr,,,, parameterized by 0, and O,, re- spectively. In the second part, the distributions in the latent common space are aligned while training a classifier suited to the target domain. This second part includes two mod- ules: a label classifier h. parameterized by é, and domain discriminator d'p, parameterized by ip. The design of the HISANN network is guided by two main objectives: (1) optimizing T., Tr, and h to minimize the label classifier loss, i.e., Weighted Source Risk Minimization, (2) optimiz- ing far St. and d maximize the domain discriminator loss, achieving Feature Distribution Alignment.
Domain Discriminator Loss. We align P.(X) and (X) using discriminator. For batches (x', B and {x{) of size B, the weighted domain discriminator loss is
LB (1xi, v. xxx
To, T2,
B w(v() log (dx(T((x))) log (I-dx(T.(x))))
B
Morcover, also verify that the standard ADA framework applied to CEA indeed minimizes d (P (X) || P((X)) in Theorem 5.2.
Theorem 5.2. Let D(x, y) and D'(x) be two density dis- tributions, and w(y) be a positive function such that J (Dyy)w(y)dy = 1. Let D= = D(x, v)w(y) de- note the w-rreeighted marginal distribution of under D. Define the following function with respect d(z)
In nutshell, Feature Space Alignment and Feature Distri- bution Alignment collaborate to achieve Heterogeneous Fea- ture Aligument. Dynamically updated importance weights mitigate the challenges of label shift, effectively linking Heterogeneous Feature Alignment with Weighted Source Risk Minimization.
T(d) = E(..,))D,,e D (-w(y) (owo(y)og((e log(d(x)) - log(1 d(x))))
The analysis shows that when
D=
d(z) d' (=
D')x)'
I(d) attains the minimum value
(x)) = -log(4) d,s(DD (x))
6

---

Heterogeneous Label Shift: Theory and Algorithm
Instantiating D(a,2) y) and (x) to (X) and Px(X) proves that the L leads minimizing d dss(P: (X) II
Take all the losses of Feature Distribution Alignment into account, we obtain the overall knowledge transfer (KT) loss as
P((X)).
Classifier Loss. As for Weighted Souce Risk Minimization, we adopt the commonly used cross-entropy loss. For batches {x), S of size S, LC is formulated as
(0, 0. C6 + aLr BL
(14)
where the hyper-parameee a and 3 are used to tune the trade-off between these three quantities during the learn- ing process. In summary, optimizing HLSAN involves two key components, i.e. Feature Space Alignment and Fea- ture Distribution Alignment, governed by the respective loss functions Lo and LKT. HLSAN alternately optimizes Lo and LKT either until a convergence criterion satisfied or for a predefined number of iterations. The more optimiza- tion and implementation details are provided in Appendix B.1 and B.2.
(1x v... {x]] Te, 0)
B
(10)
Σ w(v)) log (he (T.(x)))
= -
B
Overall Loss. Combining LA and Lg we obtain the overall loss
(11) where the hyper-parameee À is used to tune the trade-off between these two quantities during the learning process.
C= LC aLDA
6. Experiment
In this section, we evaluate the performance of the HLS approach with other closely related methods. Subsequently, we present an in-depth analysis of the proposed CMAN method from different aspects, including ablation study, parameter sensitivity analysis and convergence behavior. Before going into the details, let us introduce datasets and baselines first.
Everything seemed ready, however, a potential risk in Fea- ture Space Aligument is that poor initialization can lead to failed knowledge transfer, as demonstrated (Zhao et al., 2019; Ye al., 2021). We further illustrate this scenario with a simple example in the Appendíx B.3. To mitigate this risk, we aim to find a appropriate representation for both source and target domain data within the common space. Fortunately, parallel instances provide a natural bridge for aligning source and target modalities, as they share a common label (albeit unknown). By aligning the marginal feature distributions of parallel instances across the two modalities, we can effectively prevent extreme cases of negative transfer. Specifically, we add loss Lo for Fea- ture Space Aligument based on the parallel instances. For of size S, Lo is formulated as batches (xé_s) xo,1}²
6.1. Configuration
Dataset. We design cross-modal knowledge transfer tasks using two real-world datasets: the Multilingual Reuters Col- lection (Li et al., 2014) and Wikipedia (Fang et al., 2023). Detailed descriptions of these tasks are provided in Ap- pendix B.4. To simulate shifted label distributions in the benchmark datasets, we employ the Dirichler shift approach proposed in (Guo et al., 2020). In this setup, the target dataset is assumed to follow uniform label distribution, achieved through sampling, while the source domain label distribution is altered. Specifically, the source label distribu- tion is sampled from a Dirichlet distribution parameterized by the concentration factor y. Experiments are conducted under three label distribution shift settings, 7 2, 7 5, and y 10. Additional implementation details are provided in Appendix B.2.
Lo(O., = Σ I|T., (xo.)) TT,,, (x
(12)
The parallel instances create a cross-modal channel, en- abling a solution to the HLS problem. Through this channel, Feature Space Aligument and Feature Distribution Align- ment form a positive feedback loop, reinforcing each other to achieve an satisfactory result.
Within this cross-modal channel, leveraging the implicit label information in parallel instances allows us to fur- ther enhance Feature Distribution Alignment. Specifically, d (X), (X)) is instantiated with the projected max- imum mean discrepancy (MMD) (Gretton et al., 2012) on the parallel instances. For batches {xt,e> x 1 of size S, Cp is formulated as
Compared Methods. We evaluate the performance of the HLSAN algorithm by comparing it against two groups of baseline methods with distinct objectives: modified label shift (LS) approaches and heterogeneous domain adapta- tion (HDA) methods. (1) To illustrate the challenges posed by heterogeneous spaces and show the effectiveness of the Heterogeneous Feature Alignment (HFA) module, we com- pare HLSAN with traditional LS approaches, such as BBSL (Lipton et al., 201 8a) and RLLS (Azizzadenesheli et al., 2019a). These methods are tailored to adapting heteroge- neous source and target domains by employing dimension-
1 5 (T..,, (xx,e)) ho (2 (x
Cp(0., Oz, 0) = - 1=1
(13)

---

Heterogeneous Label Shift: Theory and Algorithm
7
2
5
7. 10
Method
SP-EN
SP-FR
SP-GE
SP-IT
SP-EN
SP-FR
SP-GE
SP-IT
SP-EN
SP-FR
SP-GE
SP-IT
BBSI.
23.7+1.4 18.7+0.9
7.7+0.8
15.6+1.0
13.8+1.6 18.7+1.3
9.2+1.11 20.3+1122
15.2+1.3 18.9+1.7 15.8+1. 7.0
RLLS
22,4+1.3 18.2+..0
6.4+0.8
15.6+1.6
13.6+1.1 19.
9.8+1.0
21.041.11
14.3+1.2 18.7+1.4
15.41111
6.91111
TNT
26.711..8 25.2.:3.5
18.9-2.5
22.413.7
25.21:444 24.0....0
20.2.1.00
19.412.4
25.0.3.1 20.413.7
21.912.6
24.712..
JEMA
48.613.1 45.7113.
37.2...88
32.542.9
46.514455
42.512.0
41.143.7
39.914.0
45.114.4
40.243.1
40.915.1
39.3129
HILAAN
53.513.1 54.7.12.0 3.. 46.4.1..3
57.4
53.7 1.6 53.5.1..0 50.8.12.9
59.9 57.3.4.4 56.3.1188
55.1
Table Accuracy (%) with standard error Multilingual Reuters Collection datasct from Texz-+Text. The best accuracy among all algorithms are highlighted in boldface.
6.3. Insight Analyses
7 2
7
5
7 10
Method
6.3.1. ABLATION STUDY
Wiki T-I
We perform ablation experiments four tasks, SP + EN, SP - FR, SP - GE and SP - IT, under Dirichlet shift (y = 10) to evaluate the contributions of HLSAN compo- nents. HLSAN is divided into three variants. (1) w/o P, ex- cluding the parallel instances loss Lp (13); (2) w/o W, ex- cluding the importance weight w from (14); and (3) w/o D, excluding the domain discriminator loss D (9). Results, shown in Figure 3, reveal two key findings. 1) Removing any component degrades performance, demonstrating the importance of each term. 2) Incorporating with importance weight enhances performance, indicating the necessity of aligning label distribution shifts.
BBSL
16.11::2.99
4..94
34.56:13.35
RLLS
18.2641388
11.07=4.15
9.02:14.88
TNT
60.26:44.24
54.07=6.38
56.25+55.54
JEMA
72.02+55.01
68.71 - 3.05
82.28
HLSAN
81.73+3.20
77.61=2.52
85.83+10655
Table Accuracy (%) with standard error on Wikipedia dataset from Image->Tere Text. The hest accuracy among all algorithms are highlighted boldface.
ality reduction techniques to enforce an equal number of features across the two domains. (2) To evaluate the effec- tiveness of HLS eliminating the dependency on labeled target domain data and addressing label shift, we conduct comparisons with HDA methods, including TNT (Chen et al., 2016) and JMEA (Fang et al., 2023). For a fair com- parison, we provide only a minimal amount of labeled target domain data (one sample per class) to the SsHDA methods, closely approximating the unsupervised setting.
Actumey (Stardad Devitta))
16 16
SP-RN
SP-FR
SP-OR
SP-IT
Figure 3. The accuracy (%) of ablation study of HLSAN on four cross-modal knowledge transfer tasks with 7 10.
6.2. Experiments Result
Based on the aforementioned datasets, we encounter two types of cross-modal knowledge transfer tasks, i.e., Tex- and Texe- Table and 3 report the com- parison results and we can obtain the following conclusions. HLSAN consistently outperforms the comparison base- line methods. 2) The modified LS performs the worst, high- lighting that heterogeneous feature spaces introduce new challenges for traditional LS methods. 3) The performance of SsHDA methods heavily depends on label availability in the target domain and significantly declines under inade- quate supervision.
6.3.2, PARAMETER ANALYSIS
The HISANN algorithm includes two key hyperparameters, a and whose impact on performance is analyzed through parameter sensitivity experiments on two tasks: SP - EN and SP - IT, under the Dirichlet shift scenario with y 5. The values of a and /3 are systematically varied within the range [0.01, 0.05, 0.1, 0.5, 5,5). The results, presented in Figure 4, demonstrate that the performance of HL.SAN sensitive to these parameters, underscoring the importance of their proper tuning. Notably, HI.SAN achieves prominent
8

---

Heterogeneous Label Shift: Theory and Algorithm
label shift data. Building on these insights, developed bound minimization framework effectively decouple and address feature heterogeneity and label shift. Our findings highlight the importance of tackling complex, real-world distribution shifts and lay a strong foundation for future research this area.
0.0
50.24
16.30
68
68
51,
51.6055
92.11
53,42
0.01
41.53
41.63
4:93
47:01
461
50
06
0.03
52.5
02
68.
08.50
68.9
52.5
03.10
56.22
0.3
41.60
02.37
64.84
40.04
07:05
45.43
0.5
50.29
$1.50
50
77
05.34
63.05
0.5
42.91
45.38
64.64
68.1
48
45.80
08.37
09.09
07.00
20
43.85
06.32
68:63
05.01
07.87
45.23
42.95
0.01
0.05
0.1
0.5
0.01
0.05
0.1
Impact Statement
(a) SP EN
(b) SP IT
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Figure 4. The sensitivity analysis of parameters and / for HLSAN on two cross- modal knowledge transfer tasks with y 5.
Acknowledgments
Toean Low
This work was partially supported by the NSF for Distin- guished Young Scholars under Grant No. 62425607, the Key NSF of China under Grant No. 62136005. Chenping Hou is the corresponding author.
References
(a) SP EN
(b) Wiki T-+1
Alexandari, A., Kundaje, A., and Shrikumar, A. Maximum likelihood with bias-corrected calibration is hard-to-bea beat at label shift adaptation. In Proceedings of the 37th Inter- national Conference on Machine Learning. ICML, vol- ume 119 of Proceedings of Machine Learning Research, pp. 232. PMLR, 2020.
Figure 5. The convergence behavior of HLSAN on two cross- modal knowledge transfer tasks with y
performance when a and A are within the range (0.5, 1} for both two tasks, indicating that the parameter interval [0.5, 1] more suitable for HLSAN.
Azizzadenesheli, K. Importance weight estimation and gen- eralization in domain adaptation under label shift. IEEE Transactions Pattern Analysis and Machine Intelli- gence, 44(10))6578-6555 2022.
6.3.3. CONVERGENCE BEHAVIOR
The HLSAN network utilizes alternating optimization for its two components: Feature Space Alignment and Feature Distribution Alignment. Figure 5 illustrates the evolution of the losses Lo and LKT for these components over training epochs, alongside the accuracy curve to depict the knowl- edge transfer process. As training progresses, both losses steadily decrease and stabilize, while accuracy improves until reaching plateau. These observations suggest that Feature Space Alignment and Feature Distribution Align- ment interact synergistically. forming positive feedback loop that enhances overall performance.
Azizzadenesheli, K., Liu, A., Yang, F., and Anandkumar, A. Regularized learning for domain adaptation under label shifts. In 7th International Conference Learning Representations, ICLR. OpenReview.net, 2019a.
Azizzadenesheli, K., Liu, A., Yang. F., and Anandkumar, A. Regularized learning for domain adaptation under label shifts. In 7th International Conference on Learning Represemattonss ICLR. OpenReview.net, 2019b.
Bai, Y., Zhang, Y., Zhao, P., Sugiyama, M., and Zhou, Z. Adapting online label shift with provable guarantees. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems, NeurlPS, 2022.
7. Conclusion
This paper addresses the critical yet underexplored chal- lenge of Heterogeneous Label Shift (HLS), characterized by simultaneous feature space heterogeneity and label distribu- tion shifts in cross-modal knowledge transfer. We bridge the gap in both theoretical understanding and practical methods for the HLS setting, advancing our comprehension of cross- modal knowledge transfer the presence of heterogeneous
Ben- Den-David, S., Blitzer, J., Crammer, K., Kulesza, A., Pereira, F., and Vaughan, J. W. A theory of learning from different domains. Mach. Learn., 79(1-2):151-175 2010.

---

Heterogeneous Label Shift: Theory and Algorithm
Briet, J. and Harremoes, P. Properties of classical and quantum jensen-shannon divergence. Physical Review A, 2009.
Guo, J., Gong, M., Liu, T., Zhang, K., and Tao, D. LTF: A label transformation framework for correcting label shift. In Proceedings of the 37th International Conference on Machine Learning, ICML, volume 119 of Proceedings of Machine Learning Research, pp. 3843-3853. PMLR, 2020.
Cai, Z., Jing, X., and Shao, L. Visual-depth matching network: Deep RGB-D domain adaptation with unequal categories. IEEE Transactions on Cybernetics, 52(6):
4623 4623-4635, 2022a.
Hsieh, Y., Tao, S., Tsai, Y. H., Yeh, Y., and Wang, Y. F. Recognizing heterogeneous cross-domain data via gener- alized joint distribution adaptation. In IEEE International Conference on Multimedia and Expo, ICME, pp. 1-6.
Cai, Z., Zhang, T., Jing, X., and Shao, L. Unequal adaptive visual recognition by learning from multi-mode data. Information Sciences, 600:1-21, 2022b.
IEEE Computer Society, 2016.
Chen, W., Hsu, T. H., Tsai, Y. H., Wang, Y. F., and Chen, M. Transfer neural trees for heterogeneous domain adapta- tion. In Computer Vision ECCV 2016 14th European Conference, volume 9909 of Lecture Notes in Computer Science, pp. 399 414. Springer, 2016.
Jennings, N. R. and Wooldridge, M. J. Foundations of
Machine Learning. Foundations of machine learning, 2012.
Kolesnikov, A., Beyer, L., Zhai, X., Puigcerver, J., Yung, J., Gelly, S., and Houlsby, N. Big transfer (bit): General visual representation learning. In Vedaldi, A., Bischof, H., Brox, T., and Frahm, J. (eds.), Computer Vision ECCV 2020 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part V, volume 12350 of Lec- ture Notes in Computer Science, pp. 491-507. Springer, 2020.
Dai, W., Chen, Y., Xue, G., Yang, Q., and Yu, Y. Translated learning: Transfer learning across different feature spaces. In Advances in Neural Information Processing Systems 21, NeurlPS, pp. 353-360, Curran Associates, Inc., 2008
Day, and Khoshgoftaar, T. M. A survey on heterogeneous transfer learning. J. Big Data, 4:29, 2017.
des Combes, R. T., Zhao, H., Wang, Y., and Gordon, G. J. Domain adaptation with conditional distribution match- ing and generalized label shift. In Advances in Neural In- formation Processing Systems 33: Annual Conference on Neural Information Processing Systems, NeurIPS, 2020.
Li, C., Chang, W., Cheng, Y., Yang, Y., and Póczos, B. MMD GAN: towards deeper understanding of moment matching network. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural In- formation Processing Systems, NewrIPS, pp. 2203 203-22113, 2017.
Fang, Z., Lu, J., Liu, F., and Zhang, G. Supi-supe het- erogeneous domain adaptation: Theory and algorithms. IEEE Trans. Puttern Anal. Mach. Intell., 1081) 1105, 2023.
Li, H., Pan, S. J., Wan, R., and Kot, A. C. Heterogeneous transfer learning via deep matrix completion with adver- sarial kernel embedding. In The Thirty-Third Conference on Artificial Intelligence, AAAI, pp. 8602-8609. AAAI Press, 2019a
Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., Marchand, M., and Lempitsky, S. adversaral training of neural networks. J. Mach.
Li, L. and Zhang, Z. Semi-supervised domain adaptation by covariance matching. IEEE Trans. Pattern Anal. Mach. Intell., 41(11):724--273, 2019.
Learn. Res., 17:59:1-59:35 2016.
Garg, S., Wu, Y., Balakrishnan, S., and Lipton, Z. C. A unified view of label shift estimation. In Advances in Neu- ral Information Processing Systems 33: Annual Confer- ence on Neural Information Processing Systems, NewrlPS, 2020.
Li, W., Duan, L., Xu, D., and Tsang, I. W. Learning with augmented features for supervised and semi- aumi-suerviseed heterogeneous domain adaptation. IEEE Trans. Pattern Anal. Mach. Intell., 3666):1134-1141 2014.
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial nets. Advances in neural informa- tion processing systems, 27, 2014.
Li, Y., Murias, M., Major, S., Dawson, G., and Carlson, E. On target shift in adversarial domain adaptation. In The 22nd International Conference on Artificial Intelligence and Statisties, AISTATS, volume 89 of Proceedings of Machine Learning Research, pp. 616-625. PMLR, 2019b.
Gretton, A., Borgwardt, K. M., Rasch, M. J., Schölkopf, B., and Smola, J. A kernel tao-sample test. Mach. Learn. Res., 13:723-773, 2012.
Lipton, Z. C., Wang, Y., and Smola, A. J. Detecting and correcting for label shift with black box predictors. In
10

---

Heterogeneous Label Shift: Theory and Algorithm
Proceedings of the 35th International Conference on Ma- chine Learning, ICMI,, volume 80 of Proceedings of Ma- chine Learning Research, pp. 1136. PMLR, 2018 8a.
Shi, X., Liu, Q. Fan, W., Yu, S., and Zhu, R. Transfer learning on heterogenous feature spaces via spectral trans- formation. In The 10th IEEE International Conference on Data Mining, ICDM, pp. 1049-1054. IEEE Computer Society, 2010
Lipton, Z. C., Wang. Y., and Smola, A. J. Detecting and correcting for label shift with black box predictors. In Proceedings of the 35th International Conference on Ma- chine Learning, ICMI, volume 80 of Proceedings of Ma- chine Learning Research, pp. 3128-3136. PMLR, 2018b.
Sukhija, S., Krishnan, N. C., and Singh, G. Supervised heterogeneous domain adaptation via random forests. In Proceedings of the Twenty-Fift Fifth International Joint Con- ference on Artificial Intelligence, IJCAI, pp. 2039-2045. LJCAT/AAI Press, 2016.
Long, M., Cao, Y., Wang, J., and Jordan, M. I. Learning transferable features with deep adaptation networks. In Proceedings of the 32nd International Conference on Ma- chine Learning, ICML, volume 37 of JMLR Workshop and Conference Proceedings, pp. 97-105. JMLR.org, 2015.
Tsai, Y. H., Ych, Y., and Wang, Y. F. Learning dross-domain landmarks for heterogeneous domain adaptation. In 2016 IEEE Conference on Computer Vision and Pattern Recog- nition, CVPR, pp. 1-5900 IEEE Computer Society, 2016.
Long, M., Cao,, Z., Wang, J., and Jordan, M. I. Conditional adversarial domain adaptation. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems, NeurlPS, pp. 1647-1657, 2018a.
Wang, C. and Mahadevan, S. Heterogeneous domain adap- tation using manifold alignment. In Proceedings of the 22nd International Joint Conference on Artificial Intelli- gence, IJCAI, pp. 1541-1546. ICCAVAAAI, 2011.
Long, M., Cao, Z., Wang, J., and Jordan, M. I. Conditional adversarial domain adaptation. In Advances in Neural Information Processing Systems 31: Annual Conference Neural Information Processing Systems, NeurlPS, pp- 1647-1657, 2018b.
Wang, J., Lan, C., Liu, C., Ouyang. Y., Qin, T., Lu, W., Chen, Y., Zeng, W., and Yu, S. Generalizing unseen domains: A survey on domain generalization. IEEE Trans. Knowl. Data Eng., (5(8))8052-807) 2023.
Wu, R., Guo, C., Su, Y., and Weinberger, K. Q. Online adap- tation to label distribution shift. In Advances in Neural Information Processing Systems 34: Annual Conference Neural Information Processing Systems, NeurIPS, pp. 11340- 1 1351, 2021.
Long, M., Cao, Y., Cao, Z., Wang, J., and Jordan, M. I. Transferable representation learning with deep adaptation networks. IEEE Transactions on Pattern Analysis and
Machine Intelligence, 41(12):3071-3083 2019.
Miler, A. Integral probability metrics and their generating classes of functions. Advances in Applied Probability, 29 (2):429-443) 1997.
Xiao, M. and Guo, Y. Feature space independent semi- supervised domain adaptation via kernel matching. IEEE Trans. Puttern Anal. Mach. Intell., 37(1):54 66, 2015.
Pan, S. J. and Yang, Q. A survey on transfer learning. IEEE Trans. Knowl. Data Eng., 22110):1455-139, 2010.
Yang, Q.. Chen, Y., Xue, G., Dai, W., and Yu, Y. Hetero- geneous transfer learning for image clustering via the socialweb. In Proceedings of the 47th Annual Meeting of the Association for Computational Linguisties, ACL, PP. 1-9. The Association for Computer I Linguistics, 2009.
Pires, B. Á. and Szepesvári, C. Statistical linear estimation with penalized estimators: an application to reinforce- ment learning. In Proceedings of the 29th International Conference on Muchine Learning, ICML. ieml.ee / Omni- press, 2012.
Yao, Y., Zhang, Y., Li, X., and Ye, Y. Heterogeneous domain adaptation via soft transfer network. In Proceedings of the 27th ACM International Conference on Multimedia, MM, pp. 1578-1586. ACM, 2019.
Prettenhofer, P. and Stein, B. Cross-language text clas- sification using structural correspondence learning. In Proceedings of the 48th Annual Meeting of the Associa- tion for Computational Linguistics, ACL, pp. 1118-1127. The Association for Computer Linguistics, 2010.
Ye, H., Xie, C., Cai, Ti, R., Li, Z., and Wang, L. Towards a theoretical framework of out-of-distribution general- ization. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems, NeurIPS, pp. 225311, 2021.
Shen, C. and Guo, Y. Unsupervised heterogeneous domain adaptation with sparse feature transformation. In Proceed- ings of The 10th Asian Conference on Machine Learning, ACML, volume 95 of Proceedings of Machine Learning Research, pp. 375-390. PMLR, 2018
Zaheer, M., Guruganesh, G., Dubey, K. A., Ainslie, J., Al- berti, C., Ontañón, S., Pham, P., Ravula, A., Wang, Q., Yang, L.,, and Ahmed, A. Big bird: Transformers for

---

Heterogeneous Label Shift: Theory and Algorithm
longer sequences. In Advances in Neural Information Processing Systems 33: Annual Conference Neural Information Processing Systems, NeurlPS, 2020.
Zhang. J., Menon, A. K., Veit, A., Bhojanapalli, S., Kumar, S., and Sra, S. Coping with label shift via distributionally robust optimisation. In 9th International Conference Learning Representations, ICLR. OpenReview.net, 2021.
Zhao, E., Liu, A., Anandkumar, A., and Yue, Y. Active learning under label shift. In The 24th International Conference on Artificial Intelligence and Statistics, AIS- TATS. volume 130 of Proceedings of Machine Learning Research, pp. 3412-3420. PMLR, 2021a.
Zhao, E.. Liu, A., Anandkumar, A., and Yue, Y. Active learning under label shift. In The 24th International Conference Artificial Intelligence and Statistics, AIS- TATS, volume 130 of Proceedings of Machine Learning Research, pp. 3412-3420. PMLR, 2021b.
Zhao, H., des Combes, R. T., Zhang, K., and Gordon, G. On learning invariant representations for domain adapta- tion. In Proceedings of the 36th International Conference on Machine Learning, ICML, volume 97 of Proceedings of Machine Learning Research, pp. 7523-7532 PMLR, 2019.
Zhou, J. T., Tsang, I. W., Pan, S. J., and Tan, M. Het- erogeneous domain adaptation for multiple classes. In Proceedings of the Seventeenth International Conference on Artificial Intelligence and Statistics, AISTATS, vol- ume of JMLR Workshop and Conference Proceedings, pp. 1095-1103, JMLR.org, 2014.
Zhou, J. T., Tsang, I. W., Pan, S. J., and Tan, M. Multi-class heterogeneous domain adaptation. J. Mach. Learn. Res., 20:57:1-57:31, 2019.
Zhou, Z. Guo, L., Jia, L., Zhang, D., and Li, Y. ODS: test-time adaptation in the presence of open-world data shift. In International Conference on Machine Learning. ICML, volume 202 of Proceedings of Machine Learning Research, pp. 42574 42588. PMLR, 2023.
12

---

Heterogeneous Label Shift: Theory and Algorithm
A. Omitted Proofs
A.1. Proof of Error Decomposition Theorem 4.3
Proof. The proof of Theorem 4.3 will start with addition and subtraction based on [R T,) - (t,, T.)l. By addition and subtraction we have
(2 T₂) - Ro (w, h T,) = Rh o TT) - Ro(w, n.w,her h T.) (ow T.) - Ro ho T.)
(0)
(6)
(15)
Rg (w. h o TT) - h
ho T,).
Here (a) is the heterogeneous space alignment error, (b) is the weight estimation error and (c) is the finite sample error.
Bounding term (a) Remember that Re o T,) is expected risks of E w.r.t. loss & under (r,(,,),, and R2 (, o Tς) is the weighted expected risks of h E H. w.r.t. loss under PT,(,,),, w E R is the importance weights of the target and source label distributions. k | is the cardinality of the finite domain of y Let γ l(X) be the
prediction of h, then we have
RR o TŞ) - R. he T.)I= E
- (ww.)))((((((()x)))).
Pr = j)P:({ = - j) Σ j)pr, j)px() = i|Y, = j)
191
Σ
Pr = =))Pa( = i|Y, = - w(Y -j)Pr = j)P,(( = =i =
$1
Σ
Pr, 1) P.((f- = -- -) =|lY, = =))
Σ Pr, =)
|P,(-= y = v) - -PP(P =v1VV =v)
max
V EV2
i
(h).
(16)
Here, equality ! holds due to the law of total probability.
Bounding term (b) Recalling that (w, h o T.). (w,, h o Ts) are the weighted expected risks of h E 7 w.r.t. loss É under Pr, (x.)X. w, W E R& are the true importance weights and the estimated importance weights. == = is the cardinality of the finite domain y., Let us define é g* with & (yy'; h(x')) Notice that by definition ||]]|, S 1 and ||e||.0 from which it follows by Hoelder's inequality that [| I 1. Therefore, for all h we have via the Cauchy Schwarz. inequality that
R.(w,hoT. - R,(W, ho T.)l = F
v/ - = =l(w(V() ( ((h(T. wi)l
vill
(17)
(w(Y = - w(Y = 5)1e S -wl
Bounding term (c) Inspired by the proof of Theorem 3.1 in (Jennings & Wooldridge, 2012), we bound (c) starting from the standard Rademacher complexity bound for H. Let C H be the family of loss function associated to H. Suppose the loss function L is M-bounded and T--Lipschitz w.r.t. Fuclidean norm. Now defined a random variable t as follows,
(S)) sup (w, h TT) R. he T.).
he7
We further define a ghost data set S' and the corresponding ghost loss Rh, w, Te, Tr), where and S' are two samples differing by exactly one instance, and say S., (x), v() in and S'm = (x), v in St. Then, since the difference of
13

---

Heterogeneous Label Shift: Theory and Algorithm
suprema does not exceed the supremum of the difference, have
Lo h o (.(Sm) - (S(
M
((S) +(S) sup (w, h T,) R (w. ho T,) he7
sup he
(18)
74g
n2
Similarly, can obtain - ((8))= < M/nn, thus ( ((8)) & M/n.. Then, by McDiarmids inequality, for any 6 > 0, with probability at least 6/2 over sample S of size nas the following inequality holds:
log
( - Es [+) + M
V
2n.
Now will proceed bound Es [+] the random variable & has the following properties
Es [+()) = Ess
sup Lhe
Rg ho T.) (w, ho TT)
Es
sup
ER h TT) Ro ho TT))
(19)
LheH
We can rewrite as
Es [(()]
Es
Es Lhe
sup
(w, ho TT))) - RR, ho T.) (x Px.v.]
(20)
and swapping the sup with the expectation
(w, ho T.) (w, ho T.) (x), ~ Px.xi]]
Es [S)] <
Es
Es
sup lhH
(21)
According to the law of iterated conditional expectation, we remove the condition have expectation on both of the samples;
Rt's (w, h Ta) - Ro. ho ho To) T.)
Es [0() < Es,s'
sup hhW
Es.s'
sup heH
1
w ((b -
n4 ==1
S.s.
sup hEH
(( - )
n4
S
sup heH
1
(ML)))((((T(<<)))))))
+ Ens
sup
_ Σ oi (wwxE)((c(TT,(""))) ]
1
T4g 1=1
hew
Y1,s i=1
2E.
sup heh
(wT)))(((((T(())))))))
f4
= 29,, ( T.) 2LR (H T.).
(22)
where the equality ① holds by using the usual symmetrizing technique through Rademacher variables. After propagation sup, the latter inequality ② is obtained. By propagating the expectation and again symmetry in the Rademacher variable obtain the inequality ④ Last, the inequality ④ holds due to the loss function & is L-Lipschitz. At this point, bound term (c) as follows:
log(1/6)
R. (w, h T.) R. ho T.) ho < 2LRR. (HT + M
(23)
sup
heH
2n1
where R,, (H o T.) is Rademacher complexity of hypothesis function class H associated T..
14

---

Heterogeneous Label Shift: Theory and Algorithm
A.2. Heterogeneous Space Alignment
A.2.1. PROOF OF LEMMA 4.4
Proof. According to the definition of HFA, we know that P,(X y) P. =y) Applying the conditional probability formula, we will maintain the equality. By the definition of w, obtain
P.(X) = Σ P.(Y = PP(XIY = =y)
NEY
Σ
w())P.(Y -P,(X =y)
yEY
Σ
wXy) Y= = (X).
yEY
A.2.2. PROOF OF THEOREM 4.5
Proof. The proof of Theorem 4.5 essentially follow Theorem 3.4 from R.T. des Combes et al. (des Combes et al., 2020),
except for Theorem 3.4 needs to adapted to the HLS scenario.
Denote y h(X) as the prediction for some h X + y. Now consider any measurable subset E Xx we would like upper bound the following quantity:
IP.(X E EIY== = y) - P.(X E EY = 1)l = Pi(Y = v) E = ()w()) E =
(24)
1 P.(X - () E, v)
$
Next, we upper bound (P,(X E E.Y = s)w(()) - PA(E = -)]l Considering the following decomposition:
E, v)
w)w(r) P 3) - P((X E E, v)
(25)
Pi(X P E) - E,Ý= = P E =) - E E, E)
And then we have
E,Y = - Pi(x E,Y = v)l S [(() (0) + [())
(26)
We bound the above three terms in turn. First, consider E E F.Y= g) - P. P,
= /
Pi(EEE = v) - PIX =v)
Σ -UN Σ Pi(x - v/)
WEY
N'EY
$
Σ PIXEEEY=-Y--- E
(27)
$ Σ ( -V) -
+
VAv
Σ
(Pi(V = + v =1=1)) v1)
$ P(( #Y) = R. To).
15

---

Heterogeneous Label Shift: Theory and Algorithm
where the last inequality is due the fact that the definition of error rate corresponds the sum of all the off-diagonal elements in the confusion matrix while the sum here only corresponds to the of all the elements in slices. Similarly, we can bound the third term follows:
( D) P.(XEE, - =n)ww))
--'ww())) E = v))(() v, Σ - = -
V'EY
V'EV
Σ PAxEEE =,, y,y= V.Y. -/Y- -(X -
(28)
Mty
WM Σ (P = -,y = /) -
S wMMP((+(M)
S WM ( T
Now we bound the last term. Recall the definition of total variation, we have
Pi(XEE = y) - (X = =n
(29)
sup
PP((X -XX
E'EN' mensurable
(Tv((:(X),, PM (X)).
Combining the above three parts yields
[Pa(X FJY = - Pi(X E EVY =VV= 1 ( T,) R. (T) + drv (xV,
Now realizing that the choice of y and the measurable subset E. is arbitrary, this leads to
_. max (X B|Y v) P((XEE b)1 < p (warR, T.) R, T,) + (xx,
yEy F
From Briet and Harremoes (Briet & Harremoes, 2009), we have drv(P)(X), P. (X)) S (Xx)s P(())).
To sum up, we obtain
drv (P.( =)) WarR((h (T) + R, T,) + Sd)s(Pp(X) Pc(X)) yEy
max
P
which completes the proof.
A.2.3. PROOF OF THEOREM 5.2
Before proving Theorem 5.2, we first introduce the Jensen-Shannon Divergence.
Definition A.1 (Jensen-Shannon Divergence). For two distributions D and D', the Jensen-Shannon (JS) divergence
dDs || D') is defined as:
1 dss(D) || D') dxr.(D Dar) dkr. (D' DM)
1 2
2
where d || the Kullback-Leibler (KI.) divergence and DM == 1(D + D').
16

---

Heterogeneous Label Shift: Theory and Algorithm
Proof. Recalling the function [d), we that
/ (w(y)) log(d())) log(1 d(x')))) p(z, y)g(z')drdd dy
T(d)
/
/ w(V)p(x,))d log(d(x)) + g(>) log(1 d(x))dx
(30)
p" (o) ((x) log(1 d(z))dr.
From the last line, we follow the exact method from Goodfellow et al. (Goodfellow et al., 2014) to see that point-wise in the minimum is attained for d' (ax) ((())) and that [(d") log(4) 4) sd (())).
p=(x)
Instantiating pd (x) and ((xr to P (X) and (X) proves that the LA leads to minimizing d.s(PP. (
P.())).
A.3. Importance Weight Estimation
A.3.1. PROOF OF LEMMA 4.7
Proof. According to the definition of HLS, and with the joint hypothesis y = h(X) over both source and target domains, straightforward to see that the induced conditional distributions over predicted labels match between the source and target domains, i.e.:
P. = h(X))Y = v) = P.(Y h(X)jy = v). Vy E V.
(31)
Then we can compute ply), Vy E y
u(y) p.( y)
Σ Pi(Ý y 3/) Pi(Y = v)
V'EY
Σ v'EY
P&( = y = -)) P,( = v/)
(32)
P.(Y = y. = P.(Y = v/) P. 3²)
Σ
V'EY
Σ
Cya w(y')
D'E
Rewrite the above equation in matrix form yields μ Cw, and C is invertible, thus w C-1 4..
A.3.2. PROOF OF THEOREM 4.8
Proof. The proof of Theorem 4.8 starts from Lemma 3.5 and theorem 3.7 proposed by (Pires & Szepesvári, 2012). For
a squared penalized loss L² (0) + 12 where (()) = ||b A0 2. is the regularization parameter. Let
arg mino ( + x1[0|12) with À = AA. A Á All2 and Ap | bll2, the following inequality holds with probability at least 1 - 6:
(()) < inf ( (C(0') 3
[2 3.b.
(33)
0/
Consider feasible 0 = 0' which satisfies ||b A0 || 0 and then we can get upper bound on the right hand side of Eq.(33),
inf + 0' [2]] < C()) + 34 0112 34x 0012.
(34)
Combining (33) and (34), the following conclusion is established.
(( = |b AQl12 = ( 0)112 S 34x||011 ||01|2 +3Ab
(35)
17

---

Heterogeneous Label Shift: Theory and Algorithm
Recalling the estimation of w, i.e., w arg min | Cw 112 + A/[w wolle- Denote Ô Wo- A C and
= μ CWn- We have,
b (1 (4) (Cwo Cwo)lla & 111e 14/12 + ||C Cll2 ||wll22 S SAn + Ac
(36)
A II(C C Ac-
Denote Umin (C) as the minimum singular value C and ||A 0)[12 - ||C( w)ll2 2 mmn [C]](C]]ll wl12. Thus,
3
(Ac(1 + 1) + An). | w v mmiu (C)
|IA 0)112 S
(AA 1| +bb)
min (C)
(37)
m (C)
Applying the Lemma 4.2 proposed by (Azizzadeneshell, 2022)) with probability at least 1 - 6, the following inequality holds:
2k 2/
(4h)
[k
k
2k 8
Ac 2
log
A S
V
log
(38)
nss
12
As result, we have with probability at least 1 - 8 that
w/12 S (110112 1)
72k:
12k
9k:
6k:
Omin ( ()) V
n
log
5
Tmin (C)
V
nh
log
(39)
Here, |00|| ||w woll2 characterizes the distance between the weight and the initialized weight.
B. Experimentation Details
B.1. The Optimization of HISAN
As mentioned in the main text, HLSAN alternately optimizes Lo and LKT either until convergence criterion is satisfied or for predefined number of iterations. Regarding to LxT, the three losses contained in LKT can be optimized simultaneously simultaneously via stochastic gradient descent (Li al., 2017) a unified neural network architecture introduced by HLSAN. Specifically, the optimization of LKT is equivalent to finding a saddle point (9., te 3, ()) such that
(Ó., Oz Φ)
arg min LKT (0., Oz, x),
0.,0
arg min CKT 0,, 3.
And the saddle point can identified as a stationary point through the following gradient updates.
OLp aLYA a
0 t 0
00,
00,
00,
JL
8Lp
B aLDA
0, - 0 P
+
00,
00,
00,
dLp
① - ④ P
+ a
06
06
OLW
n1
Oip
In addition, the HLSAN algorithm includes two stages, i.e., warming- up and training. the warming-up up stage, a suitable initial weight wo is provided for HLSAN, and in the starting up stage. the distribution shifts alignment both features and labels are performed based on the initial weights. To summarize, HI.SAN algorithm is summarized in Algorithm
18

---

Heterogeneous Label Shift: Theory and Algorithm
Algorithm 1 HLLSNN Algorithm
I: Input: Data S, T and O; Parameters a, 3, learning rate μ,, warming up epochs Ej and traning epochs Eg and batches
per epoch B:
2: Output: neural network ,T Te, heg dhe Jand the predicted target labels Y, ho (X;).
3: Initial S.. ft. h and d.p:
4: Shuffle datasets S, T and O;
Warming-up
5: for i to E do
6: Sample batches {xg, w {xi} and (x x
7: Minimize Lo w.r.t. T, and Tr;
8: Maximize CDA w.r.t 0g and fr. minimize LDA w.r.t &p and minimize Le alp w.r.t. o. 0, and 0s
9: Calculate the initial weight Wo according Lemma 4.7.
Starting-up
10:
while stopping criterion is not met do for i to EΙ + E2 do
11:
12:
Sample batches {x", v {x:] and {x x 18,
13:
Minimize Lo w.r.t. fs and ft;
14:
Maximize LA w.r.t. 0, and 6., minimize LA W.r.t. ip and minimize Cg + aLp w.r.t. o, 0, and 0s;
15:
Update the weight by solving RQP(3).
16:
End for
17:
End while
B.2. Implementation Details
Nenwork Architecture. In CMAN, the feature transformation network Ts,0, and Te, are instanced with four-layer fully-connected neural networks. The label classifier h and domain discriminator de are instanced with three-layer fully-connected neural networks. All network parameters are optimized using SGD with momentum. And the learning rate set to 0.02, The activation function is RELU.
Parameters Setting. There are two parameters and 3 in HLSAN, the value ranges of a and B are to [0.01, 0.050.5 0.1, 0.5, 1, 51. As for the comparison methods, we utilize the suggested default parameter settings provided by their original authors. In addition, the number of parallel instances is set to 100, i.e., np = 100. The warming-up epochs E₁ = 20 and the starting-up up epochs Es = 50. For the Dirichlet shift, we draw P. P.(Y) from a dirichlet distribution with concentration parameter as 10.
Remark B.I. Note that in the Importance Weight Estimation, an appropriate initial weight should be provided before solving the RQP in (3). Specifically, we first warm HLSAN by training without weighting for the first 20 epochs, and then we build estimators and P of C and fe by averaging the predictions of the classifier on the source (per true class) and target (overall) to calculate Wo. Subsequently,
importance weight is estimated on the fly during training.
B.3. Examples of knowledge transfer failures
In the main text we mentioned that the initialization of common space the extreme case could lead to the failure of knowledge transfer. Take a simple example, as shown in Figure 6(a), the transformed source and target domains have perfectly aligned marginal feature distributions in the latent common space, but this an impossible knowledge transfer task, which has been fully demonstrated in (Zhao al., 2019; Ye al., 2021). addiotion, as highlighted in the main text, parallel instances serve natural bridge for aligning source and target modalities due to their shared (albeit unknown) labels. Aligning the marginal feature distributions of parallel instances across modalities effectively mitigates the risk of extreme negative transfer. To validate the effectiveness of this technique, we conducted comparative experiments between HLSAN and its variant w/o o O, which excludes the Feature Space Aligument parallel instances loss Lo (12). The results, presented in Figure 6(b), demonstrate that omitting Lo induces significant negative shifts, ultimately leading to the failure of the knowledge transfer task.
19

---

Heterogeneous Label Shift: Theory and Algorithm
Failure Case
Negntive Trunder Compuring Revults
T'rrget Domann
A/OO SILSAN
Feature
SPENN
SP-FR
SP-Git lisk
SP-IT
Win-T4
Feature 1
(a)
(b)
Figure Examples knowledge transfer failures. The left a failure case, where the marginal feature distributions are perfectly aligned. but knowledge transfer fails. The right is the comparative experiment results of HLSAN and w/o O.
B.4. Description of the cross-modal knowledge transfer tasks
Three real-world datasets are used in the experiments, i.c., Multilingual Reuters Collection, NUS+ImageNet, and Wikipedia, which arise from different cross-modal knowledge transfer tasks.
Wikipedia is sourced from Wikipedia feature articles and contains 2,866 image-text pairs across 10 semantic categories, Following the the settings in (Fang et al., 2023), image features (I) are extracted using the Big Transfer-M (BiT-M) model with ResNet-101 (Kolesnikov et al., 2020), while text features (T) are obtained using the Big Bird model (Zaheer et al., 2020), which is well-suited for long sequences like Wikipedia texts. the Text->lmage task, text features (T) serve the source domain, and image features (I) the target domain.
Multilingual Reuters Collection contains over 11,000 news articles spanning six categories across five languages, i... English, French, German, Italian, and Spanish. Each text data is represented by a bag-of- weighted by TF-IF. Since the original data is not tractable due its high dimensionality, following (Li al., 2014), we perform dimensionality reduction of features using PCA with 60% energy preserved. After reduction, the feature dimensions for English, French, German, Italian, and Spanish 1,131, 1,230, 1,471, 1,041, and 807, respectively. Inspired by the setup from (Hsich et al., 2016; Fang al., 2023), designate Spanish the source domain, while the other four languages serve target domains, resulting in four Tex-->ext tasks.
20


Reviews:
[{"review_id": "nBHC8xiBpi", "reviewer": "Reviewer_SZhU", "summary": "This paper introduces Heterogeneous Label Shift (HLS), a problem where cross-modal knowledge transfer must address simultaneous heterogeneous feature spaces and shifted label distributions. The work presents a theoretical error decomposition, proposes a bound minimization framework (HLSAN), and validates it empirically on cross-modal classification tasks. Key contributions include formalizing HLS, deriving theoretical guarantees, and demonstrating superior performance over baseline methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q1. How does HLSAN scale with the number of classes ($k$)? Does the RQP estimator degrade for $k > 100$? \n\nA1. Thank you for your thoughtful question. Based on our theoretical analysis (Theorem 3.8) as shown in the following formula, the weight estimation error is sublinear with respect to $k$ so HLSAN can efficiently scale with $k$. Of course, when k is large, such as $k \\leq100$, it will inevitably reduce the performance of the RQP estimator due to the increased classification difficulty. In fact, weight estimation with large number of classes has been a recognized difficult problem in label shift. But by increasing the sample size we can mitigate that the impact of the increase in categories at this time.\n$$||\\hat{\\mathbf{w}} - \\mathbf{w}||_2\\le c_1\\sqrt {\\frac{{72k}}{{{n_s}}}\\log \\left( {\\frac{{12k}}{\\delta }} \\right)}+ c_2\\sqrt {\\frac{{9k}}{{{n_t}}}\\log \\left( {\\frac{{6k}}{\\delta }} \\right)}.$$ \n\nQ2. Can the framework handle non-parallel modalities (e.g., audio→text) without instance pairs $\\mathcal O$? \n\nA2. Thank you for your insightful question. Parallel instances provide a cross-modal channel, which enables HLSAN to perform effective knowledge transfer. Different from previous settings with some assumptions to connect source and target domains, instance pairs are vital for our approach since there is no labeled data in the target domain. Therefore, it is difficult for HLSAN to handle non-parallel modalities without instance pairs $\\mathcal O$. In fact, we investigate this setting since the parallel instances are now common and easily accessible, such as Image-text data, news broadcasts. We recognize the challenges due to the lack of parallel instances. In the future, we'll explore strategies such as self-supervised alignment, semantic similarity, and contrastive learning to enhance performance in this case. \n\nQ3. What is the computational overhead of adversarial alignment vs. standard DANN?\n\nA3. We appreciate the reviewer’s concern regarding the computational cost. Compared to standard DANN, adversarial alignment introduces additional computations primarily due to the alternating optimization introduces additional updates for $L_O$ and $L_{KT}$. Specifically, each training step requires solving two subproblems sequentially, which adds approximately 60% increase in training time per epoch. Fortunately, our method can level off within 30 epochs in most cases, and the additional computational cost is also affordable in real applications.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Error Decomposition Theorem (Theorem 3.3)\nAssumes invertible confusion matrix (Lemma 3.7), which may fail in high-dimensional settings.", "experimental_designs_or_analyses": "Text→Text: Spanish→English/French/German/Italian with TF-IDF features (Reuters).\nImage→Text: Wikipedia articles with BiT-M (image) and BigBird (text) features.", "relation_to_broader_scientific_literature": "Label Shift: Extends Lipton et al. (2018b)’s confusion matrix approach to heterogeneous settings via RQP.\nHeterogeneous DA: Builds on Fang et al. (2023)’s semi-supervised HDA but addresses label shift via adversarial alignment.\nAdversarial DA: Connects to DANN (Ganin et al., 2016) but aligns reweighted source/target distributions.", "essential_references_not_discussed": "Cross-Modal DA: CLIP (Radford et al., 2021) or ViLBERT (Lu et al., 2019) for vision-language alignment.\nUnsupervised Weight Estimation: Kernel Mean Matching (Huang et al., 2007) or IWCV (Sugiyama et al., 2007).", "other_strengths_and_weaknesses": "Strengths:\n1.Novel problem formulation with practical relevance to cross-modal applications.\n2.Theoretically grounded framework integrating feature alignment and label shift adaptation.\nWeaknesses:\n1.Reliance on parallel instances (O) for initialization may limit real-world applicability.\n2.Computational cost of alternating optimization (L_O and L_KT) is not analyzed.", "other_comments_or_suggestions": "Explore self-supervised alignment (e.g., contrastive learning) to reduce dependence on parallel data.\nCompare to transformer-based models (e.g., CLIP) for cross-modal transfer.\nAnalyze scalability to large-scale datasets (e.g., ImageNet-21K).", "questions_for_authors": "1.How does HLSAN scale with the number of classes (k)? Does the RQP estimator degrade for k > 100k>100?\n2.Can the framework handle non-parallel modalities (e.g., audio→text) without instance pairs O?\n3.What is the computational overhead of adversarial alignment vs. standard DANN?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AJG6jQ8N1y", "reviewer": "Reviewer_Aznb", "summary": "The paper introduces a new Heterogeneous Label Shift (HLS) method to tackle heterogeneous label shift. After analyzing the impact of feature spaces and label shift, the authors propose a new error bound. They show, with experiments on 2 real-life datasets, the efficiency of their method for multi-modal data.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q1. Typos and suggestions.\n\nA1. Thank you for your valuable feedback. We have corrected the typo in the manuscript and checked it carefully throughout. In addition, as suggested by the reviewers, we added a brief summary of the comparison methods, and the related work section has been moved earlier to improve the logical flow. Finally, we have revised Figure 2 with meaningful colors to distinguish learnable and fixed components, added explicit labels (e.g., $T_s, T_t, h$), and clarified the role of the encoder.\n\nQ2. I don't get why a heterogeneous domain should imply a label shift. You can distribute the label in two different modalities. Even in the experiment, the label shifts are simulated. \n\nA2. Thank you for your comment. In reality, since the collected data come from different modalities, it is more likely to cause label shift. For example, when images and text data are collected from different websites, it is almost impossible to maintain the same category ratio of picture and text data, which unavoidably cause the label shift problem. Of course, as the reviewer said, we can distribute label in two different modalities to avoid label shift. Nevertheless, in practice, this approach is not feasible since there is no label data in target domain. In fact, label shift is a well-documented phenomenon in domain adaptation and transfer learning [1,2], and the HLS problem is also raised from reality. Regarding the experiments, in order to evaluate our approach from multiple angles and multiple label shift cases, we follow standard setup in label shift research [3,4] by simulating label shifts to systematically evaluate our method. This allows us to isolate the impact of label shift and better analyze its effects.\n\n[1] Zachary C. Lipton, Yu-Xiang Wang, Alexander J. Smola: Detecting and Correcting for Label Shift with Black Box Predictors. ICML 2018: 3128-3136.\n\n[2] \tRuihan Wu, Chuan Guo, Yi Su, Kilian Q. Weinberger: Online Adaptation to Label Distribution Shift. NeurIPS 2021: 11340-11351.\n\n[3] Kamyar Azizzadenesheli, Anqi Liu, Fanny Yang, Animashree Anandkumar: Regularized Learning for Domain Adaptation under Label Shifts. ICLR (Poster) 2019.\n\n[4] Ruidong Fan, Xiao Ouyang, Tingjin Luo, Dewen Hu, Chenping Hou: Incomplete Multi-View Learning Under Label Shift. IEEE Trans. Image Process. 32: 3702-3716 (2023).\n\nQ3. The ablation study is done only over two adaptations SP$\\rightarrow$EN and SP$\\rightarrow$IT. Can we have the ablation study for both experiments' overall adaptations? You choose to take the worst case $\\gamma=10$; what are the results for smaller $\\gamma$?\n\nA3. Thank you for your suggestions. Due to space constraints, we conducted the ablation study on two representative adaptations (SP$\\rightarrow$EN and SP$\\rightarrow$IT) and selected $\\gamma=10$ as Illustration. In fact, we have done all the experiments. As shown in the following tables. The results can also reveal the same two key findings. 1) Removing any component degrades performance, demonstrating the importance of each term. 2) Incorporating with importance weight enhances performance, indicating the necessity of aligning label distribution shifts. \n\n# The accuracy (%) of ablation study of HLSAN with $\\gamma = 10$.\n|Task| w/o/D | w/o/P | w/o/W |HLSAN|\n|-----------------|-------------------|--------------------|--------------------|--------------------|\n| SP$\\rightarrow$EN |50.1|49.6|54.7|**55.9**|\n| SP$\\rightarrow$FR |53.7|51.7|55.4|**57.3**|\n| SP$\\rightarrow$GE|53.4|49.4|54.0|**56.3**|\n| SP$\\rightarrow$IT|44.4|42.5|48.3|**55.1**|\n| Wiki T$\\rightarrow$I|75.9|74.6|77.2|**85.8**|\n\n# The accuracy (%) of ablation study of HLSAN with $\\gamma = 5$.\n|Task| w/o/D | w/o/P | w/o/W |HLSAN|\n|-----------------|-------------------|--------------------|--------------------|--------------------|\n| SP$\\rightarrow$EN |49.1|47.3|53.1|**57.4**|\n| SP$\\rightarrow$FR |48.8|48.6|50.2|**53.7**|\n| SP$\\rightarrow$GE|47.7|46.5|51.6|**53.5**|\n| SP$\\rightarrow$IT|46.9|45.5|48.2|**50.8**|\n| Wiki T$\\rightarrow$I|73.3|72.6|75.2|**77.6**|\n\n# The accuracy (%) of ablation study of HLSAN with $\\gamma = 2$.\n|Task| w/o/D | w/o/P | w/o/W |HLSAN|\n|-----------------|-------------------|--------------------|--------------------|--------------------|\n| SP$\\rightarrow$EN |48.2|47.3|52.1|**53.5**|\n| SP$\\rightarrow$FR |50.1|48.6|52.4|**54.7**|\n| SP$\\rightarrow$GE|46.8|45.4|47.9|**50.2**|\n| SP$\\rightarrow$IT|41.3|40.5|44.2|**46.4**|\n| Wiki T$\\rightarrow$I|72.9|70.6|76.2|**81.7**|", "claims_and_evidence": "The authors claim that in heterogeneous scenarios, \"label shift is inevitable.\" In my opinion, this is not so obvious, and I would like to have some references to show the necessity of this application. Especially when, in the experiments, the label shift is made artificially.", "methods_and_evaluation_criteria": "The dataset makes sense for the heterogeneous Domain Adaptation, but maybe not for label shift without making them shift.", "theoretical_claims": "I checked briefly the proofs that seemed good, but I did not have time to go in depth.", "experimental_designs_or_analyses": "My only concern is the ablation study that is done only over 2 tasks, while other experiments are done over 13 tasks. cf questions.", "relation_to_broader_scientific_literature": "NA", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "Strengths:\n- The paper deals with an interesting problem in domain adaptation that has not been treated much.\n\nWeaknesses:\n- the paper is very dense and could be hard to follow.\n- The notation seems to change in the paper. cf comments", "other_comments_or_suggestions": "- typo Table 1: discrete sourece -> discrete source\n\n- Figure 2 is complex to understand. It could be nice to have meaningful colors to explain what is learnable and fixed. \nI don't understand why a network like encoders represents the latent space. I don't see the shared network encoder in the paper. Maybe put the name of each to understand better the illustration (i.e., Ts, Tt, h ...).\n\n- You use T to name the encoder, and in eq. 12 and 13, you are using f. Is it the same or a different encoder?\n\n- It can be nice to have a small summary of the compared methods.\n\n- I suggest putting the related work part at the beginning. At this point of the paper, we want to continue reading your method.", "questions_for_authors": "- I don't get why a heterogeneous domain should imply a label shift. You can distribute the label in two different modalities. Even in the experiment, the label shifts are simulated. \n\n- The ablation study is done only over two adaptations SP -> EN and SP -> IT. Can we have the ablation study for both experiments' overall adaptations? You choose to take the worst case $\\gamma = 10$; what are the results for smaller $\\gamma$?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PD4vBCIIaM", "reviewer": "Reviewer_cVaJ", "summary": "The paper introduces the concept of Heterogeneous Label Shift to address cross-modal knowledge transfer challenges, where both feature heterogeneity and shifted label distributions affect model performance. It presents an error decomposition theorem and a bound minimization framework that tackle these issues. Extensive experiments validate the effectiveness of this approach.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q1. What is the primary role of the unlabeled parallel instances, and would the proposed method become inapplicable if these data were lacking?\n\nA1. Thank you for your insightful question. As the second response to the Reviewer SzhU, parallel instances establish a cross-modal channel that facilitates effective knowledge transfer, linking the source and target domains while mitigating extreme negative transfer, as discussed in Appendix B.3. Thus, our method has a reliance on parallel instances. In reality, parallel instances are common and easy to obtain compared to labeled target data. We then investigate this problem setting and propose an theoretical inspired models. Besides, we also recognize the challenges due to the lack of parallel instances. In future work, we will continue to explore strategies such as self-supervised alignment, semantic similarity, and contrastive learning to explore the task of cross-modal knowledge transfer without parallel instance.\n\nQ2. If the unlabeled parallel instances are crucial, should the experiments analyze the impact of the number of these instances on the final performance?\n\nA2. Thank you for the insightful suggestion. To validate this, we have conducted preliminary experiments to analyze the impact of the number of parallel instances on final performance and the results are shown in the following tables. More thorough investigations will be added to the final version. Based on these observations, we can see that performance gradually improves as more parallel instances are available.\n# The influence of Parallel Instances for HLSAN.\n|Task| 50 | 100 | 150 |200|\n|-----------------|-------------------|--------------------|--------------------|--------------------|\n| SP$\\rightarrow$EN |49.2|53.5|54.3|56.2|\n| SP$\\rightarrow$FR |52.1|54.7|56.1|56.8|\n| SP$\\rightarrow$GE|48.1|50.2|52.1|55.8|\n| SP$\\rightarrow$IT|44.3|46.4|50.1|53.2|\n| Wiki T$\\rightarrow$I|79.7|81.7|84.6|87.3|\n\nQ3. While there is existing work on both heterogeneous domain adaptation and label shift, what unique challenges does the Heterogeneous Label Shift problem introduce?\n\nA3. Thanks for your question. We think coupled Heterogeneous Feature and Label Shift is the unique challenges of HLS problem. Unlike traditional label shift, where only the label distribution changes, HLS involves simultaneous shifts in both feature space and label distribution, making standard importance-weighting methods insufficient. Unlike traditional heterogeneous domain adaptation, HLS considers the more general case where heterogeneous features are accompanied by label shift, which makes the existing feature-alignment methods fail to achieve joint distribution alignment. More seriously, our HLS problem can not be decoupled into two sequent or parallel problems since Heterogeneous Feature Alignment and Importance Weight Estimation are intertwined. It is not ideal to simply use existing heterogeneous domain adaptation and label shift techniques to solve the two problems separately. To solve this problem, HLSAN designs two interactive modules of Heterogeneous Feature Alignment and Importance Weight Estimation respectively. By interacting them synergistically, we form a positive feedback loop that enhances overall performance.", "claims_and_evidence": "The claims made in the submission are supported by evidence.", "methods_and_evaluation_criteria": "Yes, the proposed methods and the constructed benchmark make sense for the problem at hand.", "theoretical_claims": "I have generally checked the proofs, but some details have not been thoroughly verified.", "experimental_designs_or_analyses": "I have generally checked the experimental design and analysis, and they appear sound.", "relation_to_broader_scientific_literature": "The paper contributes to the study of heterogeneous domain shift by focusing on heterogeneous label shift.", "essential_references_not_discussed": "No, essential prior works are appropriately cited and discussed.", "other_strengths_and_weaknesses": "No additional strengths or weaknesses are observed beyond those already mentioned.", "other_comments_or_suggestions": "I have no other suggestions.", "questions_for_authors": "1. What is the primary role of the unlabeled parallel instances, and would the proposed method become inapplicable if these data were lacking?\n2. If the unlabeled parallel instances are crucial, should the experiments analyze the impact of the number of these instances on the final performance?\n3. While there is existing work on both heterogeneous domain adaptation and label shift, what unique challenges does the Heterogeneous Label Shift problem introduce?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LEdu42DG0B", "reviewer": "Reviewer_kepV", "summary": "This paper introduces Heterogeneous Label Shift (HLS), a novel challenge in cross-modal knowledge transfer where both feature spaces and label distributions differ. It presents a new error decomposition theorem and a bound minimization framework to separately tackle feature heterogeneity and label shift. The authors propose HLSAN, an adversarial deep learning algorithm that aligns feature spaces, estimates label shifts, and improves target domain classification. Experiments show HLSAN outperforms existing methods, validating its theoretical and practical effectiveness.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q1. Overall, it feels like there is a hole that needs to be filled between the entangled importance weight and HFA condition, that is, what happens when HFA is not perfectly achieved (which is almost surely the case), should we change the $\\mathbf w$ estimation strategy and estimation characterization?\n\nA1. We appreciate your insightful observation. In practice, achieving perfect Heterogeneous Feature Alignment (HFA) is challenging, which may introduce biases in importance weight estimation. To mitigate this, we adopt a joint optimization approach that simultaneously optimizes Importance Weight Estimation and Heterogeneous Feature Alignment, allowing the two modules to interact synergistically and form a positive feedback loop that enhances overall performance. Specifically, better HFA reduces distribution mismatch, leading to more accurate importance weight estimation, while improved weight estimation provides a more reliable training signal, further refining feature alignment. Of course, when there is a significant deviation in HFA, the estimated importance weight $\\mathbf{w}$ may become biased due to residual feature mismatches between domains, which will further affect the adaptation performance. In this case, we need to adjust the $\\mathbf w$ estimation strategy. For example, prior information can be incorporated to design regularization terms that correct $\\mathbf w$.\n\nQ2. Regarding Theorem 3.5, the result seems rather counter intuitive, because the total variation distance between the two measures can be arbitrarily large when the probability of one class diminishes in distribution shift ($\\rho \\rightarrow 0$). This kind of implies the bound could be potentially vacuous. Can the authors clarify on this issue?\n\nA2. Thanks for your insightful comments. Theorem 3.5 gives an upper bound on the discrepancy of the feature distribution conditional on the label. We think it is not counter-intuitive since $\\rho \\rightarrow 0$ only implies a loose upper bound and does not imply that the arbitrarily large total variation. In other words, an infinite upper bound is not equivalent to an infinite total variation. In fact, as shown in Eq. (1), the total variation is a value within the range [0,1]. In addition, Theorem 3.5 reflects the natural limitation that, in extreme cases of label shift, adaptation becomes fundamentally more difficult. Most theoretical results in domain adaptation and importance weighting [1, 2] similarly exhibit such behavior as $\\rho \\rightarrow 0$. In this paper, we mainly consider the mild case consistent with the mainstream Label Shift [2, 3], and first give a theoretical analysis of the brand-new HLS problem. Based on the theory, we propose a bound minimization HLS framework and achieve good performance. In fact, the extreme imbalance problem has always been an important puzzle, and how to construct more tight upper bounds in this case is a very interesting research problem.\n$$d_{TV}(P,Q)= \\mathop {\\sup }\\limits_{E~\\text{is measurable}}\\left|P(X\\in E)-{Q} (X\\in E)\\right|.~~~~(1)$$ \n\n[1] Remi Tachet des Combes, Han Zhao, Yu-Xiang Wang, Geoffrey J. Gordon: Domain Adaptation with Conditional Distribution Matching and Generalized Label Shift. NeurIPS 2020.\n\n[2] Kamyar Azizzadenesheli, Anqi Liu, Fanny Yang, Animashree Anandkumar: Regularized Learning for Domain Adaptation under Label Shifts. ICLR (Poster) 2019.\n\n[3] Ruidong Fan, Xiao Ouyang, Tingjin Luo, Dewen Hu, Chenping Hou: Incomplete Multi-View Learning Under Label Shift. IEEE Trans. Image Process. 32: 3702-3716 (2023).\n\nQ3. The same concern goes with Theorem 3.8, as it might be the case that for a \"bad\" predictor, the importance weight could also be arbitrarily large (${\\sigma _{\\min }(\\mathbf{C})}\\rightarrow 0$). Is there any guarantee or general explanation on how can this be avoided?\n\nA3. Thanks for your insightful comments. Theorem 3.8 gives an upper bound for Importance Weight Estimation. ${\\sigma _{\\min }(\\mathbf{C})}\\rightarrow 0$ leads to a loose upper bound for weight estimation error. Similarly, an infinite upper bound is not equivalent to an arbitrarily large weight estimation error. It typically occurs when the model is poorly trained or when the source and target domains have extreme shifts. In our paper, to avoid this problem, as mentioned in Appendix B.2, we first warm up HLSAN by training without weighting for the first 20 epochs to obtain an acceptable predictor, and then we build estimators for $\\mathbf{C}$. In this case, $\\mathbf{C}$ is a diagonally dominant real symmetric probability matrix, which guarantees that $\\mathbf{C}$ is invertible and its singular values are all greater than 0. Through such way, the extreme cases suggested by the reviewer are largely avoided. Since this issue is a common challenge for label shift, and we have employed certain strategies to mitigate it as much as possible, how to fully resolve it from a theoretical perspective remains an open problem.", "claims_and_evidence": "Most of the claims and evidence makes sense by themselves. However, my major concern is on the interrelationship between the two estimation. The design of the loss function is rooted in estimating a good importance weight, and then use the importance weight to align the latent feature space. However, unlike a simple competing paradigm like min max optimization, in there, the importance weighting estimation scheme only makes sense when HFA is achieved. And we can only get closer to HFA (almost impossible to achieve perfect HFA as shown in equation (2)) with a good importance weight estimation. This is especially concerning because the estimation bound can be arbitrarily large, which makes it looks difficult from an optimization perspective. \n\nOverall, it feels like there is a hole that needs to be filled between the entangled importance weight and HFA condition, that is, what happens when HFA is not perfectly achieved (which is almost surely the case), should we change the w estimation strategy and estimation characterization?", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Yes.", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "The paper provides an interesting piece of theoretical analysis for domain/distribution shift.", "essential_references_not_discussed": "Not to my knowledge.", "other_strengths_and_weaknesses": "See above.", "other_comments_or_suggestions": "Minor Comments:\n\nThere are confusing typos throughout the paper. For example, page3 line 116 second column, it should be under HFA assumption correct?", "questions_for_authors": "Questions:\n\n1. Regarding Theorem 3.5, the result seems rather counter intuitive, because the total variation distance between the two measures can be arbitrarily large when the probability of one class diminishes in distribution shift ($\\rho \\rightarrow 0$). This kind of implies the bound could be potentially vacuous. Can the authors clarify on this issue?\n2. The same concern goes with Theorem 3.8, as it might be the case that for a \"bad\" predictor, the importance weight could also be arbitrarily large ($\\sigma_{min}(\\mathbf{C}) \\righarrow 0$). Is there any guarantee or general explanation on how can this be avoided?\n\nIf these two answers and the previous concern is addressed, I would be happy to raise my score, as I do find the theoretical framework interesting.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces Heterogeneous Label Shift (HLS), a new learning problem addressing knowledge transfer when both feature spaces and label distributions differ between source and target domains. It provides a theoretical error decomposition and proposes the HLSAN algorithm, demonstrating its effectiveness in cross-modal classification tasks with varying label shifts.  

The main strengths of this paper are:
1. HLS is a new and exciting framework that generalizes label shift in a way that encompasses more applications.
2. The theoretical results not only show that their algorithm achieves theoretically good performance, but also provides some fundamental facts about the problem at hand.

Addressing concerns from reviewers:
1. Two assumptions were brought up by the reviewers.  The existence of parallel instances and the invertibility of the confusions matrix.
2. What happens if HFA does not precisely hold, and how does this interact with the weighting scheme?
3. Providing more ablation studies

Regarding (1), I think that without these assumptions we lose identifiability and the problem becomes impossible (from what I can tell).  (2) is outside of the scope of this work, and typically, model misspecification is handled in followup works.  (3) was addressed by the authors.
Decision: Accept (poster)