Title: Conditional Distribution Compression via the Kernel Conditional Mean Embedding
Venue: NeurIPS 2025
Accepted: yes
Keywords: compression

herding

distribution compression

conditional distribution

joint distribution

inducing points

kernel conditional mean embedding

kernel mean embedding

rkhs

reproducing kernel hilbert space
Abstract: Existing distribution compression methods, like Kernel Herding (KH), were originally developed for unlabelled data. However, no existing approach directly compresses the conditional distribution of *labelled* data. To address this gap, we first introduce the *Average Maximum Conditional Mean Discrepancy* (AMCMD), a metric for comparing conditional distributions, and derive a closed form estimator. Next, we make a key observation: in the context of distribution compression, the cost of constructing a compressed set targeting the AMCMD can be reduced from $\mathcal{O}(n^3)$ to $\mathcal{O}(n)$. Leveraging this, we extend KH to propose *Average Conditional Kernel Herding* (ACKH), a linear-time greedy algorithm for constructing compressed sets that target the AMCMD. To better understand the advantages of *directly* compressing the conditional distribution rather than doing so via the joint distribution, we introduce *Joint Kernel Herding* (JKH), an adaptation of KH designed to compress the joint distribution of labelled data. While herding methods provide a simple and interpretable selection process, they rely on a greedy heuristic. To explore alternative optimisation strategies, we also propose *Joint Kernel Inducing Points* (JKIP) and *Average Conditional Kernel Inducing Points* (ACKIP), which *jointly* optimise the compressed set while maintaining linear complexity. Experiments show that directly preserving conditional distributions with ACKIP outperforms both joint distribution compression and the greedy selection used in ACKH. Moreover, we see that JKIP consistently outperforms JKH.

Conditional Distribution Compression via the Kernel Conditional Mean Embedding

Abstract

Existing distribution compression methods, like Kernel Herding (KH), were originally developed for unlabelled data. However, no existing approach directly compresses the conditional distribution of labelled data. To address this gap, we first introduce the Average Maximum Conditional Mean Discrepancy (AMCMD), a metric for comparing conditional distributions, and derive a closed form estimator. Next, we make a key observation: in the context of distribution compression, the cost of constructing a compressed set targeting the AMCMD can be reduced from O ( n 3 ) to O ( n ) . Leveraging this, we extend KH to propose Average Conditional Kernel Herding (ACKH), a linear-time greedy algorithm for constructing compressed sets that target the AMCMD. To better understand the advantages of directly compressing the conditional distribution rather than doing so via the joint distribution, we introduce Joint Kernel Herding (JKH), an adaptation of KH designed to compress the joint distribution of labelled data. While herding methods provide a simple and interpretable selection process, they rely on a greedy heuristic. To explore alternative optimisation strategies, we also propose Joint Kernel Inducing Points (JKIP) and Average Conditional Kernel Inducing Points (ACKIP), which jointly optimise the compressed set while maintaining linear complexity. Experiments show that directly preserving conditional distributions with ACKIP outperforms both joint distribution compression and the greedy selection used in ACKH. Moreover, we see that JKIP consistently outperforms JKH.

1 Introduction

Given a large unlabelled dataset D := { x i } n i =1 ⊂ X sampled i.i.d. from the distribution P X , a major challenge is constructing a compressed set, C := { ˜ x j } j m =1 with m ≪ n , that preserves the essential statistical properties of the original data. This compressed set can then replace the full dataset in downstream tasks, significantly reducing computational costs while maintaining statistical fidelity. Existing distribution compression algorithms leverage the theory of Reproducing Kernel

Hilbert Spaces (RKHS) [1] to embed distributions into function space. Specifically, these methods minimise the Maximum Mean Discrepancy (MMD) [2] between the true kernel mean embedding of the distribution P X , denoted µ X , and the kernel mean embedding estimated with the compressed set, denoted ˜ µ X . This ensures that the empirical distribution of the compressed set, ˜ P X , remains close to the true distribution P X in terms of MMD.

Several well-known distribution compression algorithms include Kernel Herding [3], Kernel Quadrature [4-9], Support Points [10], Gradient Flow [11], and Kernel Thinning [12-14]. Kernel Herding was the first method proposed for distribution compression and remains one of the most intuitive approaches. It is a greedy algorithm that iteratively constructs a compressed set via gradient descent, optimising super-samples that are not part of the original dataset. Kernel Herding was originally designed to compress distributions over un- labelled data. However, we demonstrate that it can be adapted to compress the joint distribution P X,Y of a labelled dataset D := { ( x i , y i ) } n i =1 ⊂ X × Y using the theory of tensor product RKHSs. This is achieved by optimising a compressed set C := { (˜ x j , ˜ y j ) } j m =1 targeting the Joint Maximum Mean Discrepancy (JMMD) [15]. In a similar fashion, we also adapt the gradient flow approach of [16], which optimises all points in the compressed set jointly.

No existing distribution compression method targets the family of conditional distributions P Y | X of a labelled dataset. To address the gap, we require the kernel conditional mean embedding (KCME), denoted µ Y | X . The KCME provides a way to embed the family of conditional distributions P Y | X into an RKHS, and is a widely used technique for non-parametric modelling of complex conditional distributions. Given n labelled samples, it can be consistently estimated with a computational cost of O ( n 3 ) [17, 18]. Despite this high cost, the KCME has been successfully applied in various fields, including conditional distribution testing [18, 19], conditional independence testing [18, 20], conditional density estimation [21-23], likelihood-free inference [24], Bayesian optimisation [25], probabilistic inference [17, 22, 26-28], calibration of neural networks [29], reinforcement learning [30-32], and as a consistent multi-class classifier [33]. Intuitively, we posit that directly compressing the conditional distribution should be preferable to indirect joint compression, much as direct conditional density estimation outperforms approaches based on separate joint and marginal estimates [34]. Proofs of all our theoretical results are in Section B.

Our key contributions are :

In Section 4.1, we define the Average Maximum Conditional Mean Discrepancy (AMCMD), show that it satisfies the properties of a proper metric on the space of conditional distributions, and derive a closed form estimate.

In Section 4.2, we make a crucial observation: the cost of estimating the AMCMD, excluding terms irrelevant for distribution compression, can be reduced from O ( n 3 ) to O ( n ) via application of the tower property.

This observation enables the development of Average Conditional Kernel Herding (ACKH), a linear-time algorithm which constructs a compressed set such that ˜ P Y | X = x ≈ P Y | X = x a.e. x wrt P X . Furthermore, in Section 4.3, we propose Average Conditional Kernel Inducing Points (ACKIP) as a non-greedy , linear-time alternative that jointly optimises the compressed set to the same end.

For comparison purposes, in Section 3, we propose Joint Kernel Herding (JKH) and Joint Kernel Inducing Points (JKIP), extending existing compression algorithms to target the joint distribution.

In Section 5, across various datasets and evaluation metrics, we show that directly targeting the conditional distribution via ACKIP is preferable to compressing the joint distribution via JKH or JKIP. We also demonstrate the limitations of the greedy heuristic used by JKH and ACKH, with JKIP and ACKIP outperforming their counterparts.

2 Preliminaries

In this section we briefly introduce the relevant theory of RKHSs, for a more thorough treatment see the various detailed surveys which exist in the literature [35-37].

Throughout this work, we consider (Ω , F , P ) as the underlying probability space. Let ( X , X ) and ( Y , Y ) be separable measure spaces, and let X : Ω → X and Y : Ω → Y be random variables with distributions P X and P Y , respectively. We denote the joint distribution of X and Y by P X,Y and the conditional distribution, in the measure-theoretic sense of [18], by P Y | X . Given a labelled dataset D := { ( x i , y i ) } n i =1 , we refer to the empirical distribution of D as ˆ P X,Y . For a compressed set C := { (˜ x i , ˜ y i ) } i m =1 , m ≪ n , we instead denote the empirical distribution as ˜ P X,Y .

Reproducing Kernel Hilbert Spaces : Each positive definite kernel function k : X × X → R induces a vector space of functions from X to R , known as a Reproducing Kernel Hilbert Space (RKHS) [1], denoted by H k . An RKHS H k is defined by two key properties: (i) For all x ∈ X , the function k ( x , · ) : X → R belongs to H k ; (ii) The kernel function k satisfies the reproducing property , meaning that for all f ∈ H k and x ∈ X , we have f ( x ) = ⟨ f ( · ) , k ( x , · ) ⟩ H k , where ⟨· , ·⟩ H k denotes the inner product in H k . A key property of kernels is the concept of universality [38]. Intuitively, if a kernel k ∈ C 0 ( X ) is universal, then for every function f ∈ C 0 ( X ) , there exists a function g ∈ H k that approximates it arbitrarily well. This property is satisfied for many common kernel functions such as the Gaussian Laplacian, and Matérn, for example [38].

Tensor Products of Reproducing Kernel Hilbert Spaces : Let l : Y × Y → R be the reproducing kernel inducing the RKHS H l , and denote H k ⊗ H l to be the tensor product of the RKHSs H k and H l , consisting of functions g : X × Y → R . Then, for h, h ′ ∈ H k and f, f ′ ∈ H l , the inner product in H k ⊗H l is given by ⟨ f ⊗ g, f ′ ⊗ g ′ ⟩ H k ⊗H l := ⟨ g, g ′ ⟩ H k ⟨ f, f ′ ⟩ H l . Under the integrability condition E P X [ k ( X,X )] < ∞ , E P Y [ l ( Y, Y )] < ∞ , one can define the joint kernel mean embedding µ X,Y := E P X,Y [ k ( X, · ) l ( Y, · )] ∈ H k ⊗ H l such that E P X,Y [ g ( X,Y )] = ⟨ µ X,Y , g ⟩ H k ⊗H l for all g ∈ H k ⊗ H l [18]. The joint kernel mean embedding can be estimated straightforwardly as ˆ µ X,Y := ∑ n i =1 k ( x i , · ) l ( y i , · ) with i.i.d. samples from the joint distribution. The tensor product structure is advantageous as it permits the natural construction of a tensor product kernel from kernels defined on X and Y . This insight is particularly important when X and Y have distinct characteristics that make a direct definition of a p.d. kernel difficult, for example, if X = R d and Y = N p .

Given additional random variables X ′ : Ω →X , Y ′ : Ω →Y , and the embedding µ X ′ ,Y ′ of P X ′ ,Y ′ , one can define the Joint Maximum Mean Discrepancy (JMMD) [15] as

For a particular class of characteristic tensor product kernels [39], the mapping P X,Y ↦→ µ X,Y is injective [40]. Hence, by the virtue of Theorem 5 [2], it is the case that JMMD ( P X,Y , P X,Y ) = ∥ µ X,Y -µ X ′ ,Y ′ ∥ H k ⊗H l = 0 if and only if P X,Y = P X ′ ,Y ′ .

Conditional Kernel Mean Embedding : Under the integrability condition E P Y [ √ l ( Y, Y )] < ∞ , the kernel conditional mean embedding (KCME) of P Y | X is defined as µ Y | X := E P Y | X [ l ( Y, · ) | X ] where µ Y | X : Ω →H l is an X -measurable random variable outputting functions in H l . Similar to the unconditional case, for any f ∈ H l , it can be shown that E P Y | X [ f ( Y ) | X ] a.s. = ⟨ f, µ Y | X ⟩ H l [18].

The KCME can be written as the composition of a deterministic function F Y | X : X → H l and the random variable X : Ω → X , i.e. µ Y | X = F Y | X ◦ X (Theorem 4.1, [18]). For consistency in notation, throughout the remainder of this work, whenever we refer to the KCME µ Y | X , we mean F Y | X . The KCME can be estimated directly [18, 30] using i.i.d. samples from the joint distribution D as

where the superscript D refers to the data used to estimate µ Y | X , we define W := ( K + λI ) -1 , [ K ] ij = k ( x i , x j ) , i, j = 1 , . . . , n , and λ > 0 is a regularisation parameter.

Maximum Conditional Mean Discrepancy : Given two conditional distributions P Y | X and P Y ′ | X ′ , with KCMEs µ Y | X and µ Y ′ | X ′ , the Maximum Conditional Mean Discrepancy (MCMD) was defined

by [18] as a function of the conditioning variable x ∈ X , which returns a metric on P Y | X = x and P Y ′ | X ′ = x , that is

In the following sense sense, the MCMD is a natural metric on the space of conditional distributions:

Theorem 2.1. (Theorem 5.2. [18]) Suppose l : Y × Y → R is characteristic, that P X and P X ′ are absolutely continuous with respect to each other, and that P ( · | X ) and P ( · | X ′ ) admit regular versions. Then MCMD [ P Y | X , P Y ′ | X ′ ] ( · ) = 0 almost everywhere x wrt P X (or P X ′ ) if and only if, for almost all x ∈ X wrt P X (or P X ′ ), we have P Y | X = x ( B ) = P Y ′ | X ′ = x ( B ) for all B ∈ Y .

Related Work : Existing distribution compression methods focus on unlabelled data, where the goal is to approximate P X using a smaller representative set that minimises the Maximum Mean Discrepancy (MMD). Perhaps the most intuitive of these is Kernel Herding [3, 41], which greedily constructs a compressed set by iteratively optimising points that minimise the MMD. While simple and interpretable, its greedy nature can lead to suboptimal solutions, motivating alternatives such as Gradient Flow [11], which jointly optimises all points via gradient descent, and Kernel Thinning [13, 14], which restricts the compressed set to a subset of the original data to support theoretical guarantees. Despite these advances, existing approaches target unlabelled distributions. In contrast, this work introduces algorithms for joint and conditional distribution compression. For additional discussion of related work see Section A.1.

3 Joint Distribution Compression

Given a labelled dataset D , one may be interested in compressing the joint distribution P X,Y , rather than the marginals P X , P Y .

3.1 Joint Kernel Herding

By inducing an RKHS H k ⊗ H l with the tensor product kernel k ( · , · ) l ( · , · ) , one can modify the Kernel Herding [3] algorithm to instead target the joint distribution. First, we assume that ∥ k ( x , · ) l ( y , · ) ∥ H k ⊗H l = R for all x ∈ X and y ∈ Y , where R is a constant. This condition holds for commonly used stationary kernels such as the Gaussian, Laplace, and Matérn kernels. Then, assuming we are at the ( m +1) th iteration, having already constructed a compressed set of size m , C = { (˜ x j , ˜ y j ) } j m =1 , the next pair is chosen as the solution to the optimisation problem

We refer to this algorithm as Joint Kernel Herding (JKH). The optimisation problem in (2) can be interpreted as reducing at each iteration the JMMD ( P X,Y , ˜ P X,Y ) ; see Section B.2.

3.2 Joint Kernel Inducing Points

The greedy optimisation approach of JKH is a convenient heuristic which focuses computational effort on optimising one new pair at a time while previously selected pairs are fixed. However, this strategy may give poor solutions, as it never revisits or adjusts earlier selections. Instead, we can first select an initial compressed set of m pairs through uniformly at random subsampling of D , followed by refining all pairs jointly . Going forward, we refer to this algorithm as Joint Kernel Inducing Points (JKIP), noting that it may be viewed as a discretised Wasserstein gradient flow [11].

We target the JMMD ( P X,Y , ˜ P X,Y ) by solving the optimisation problem

via gradient descent. In the general case, one cannot compute the expectation above, hence we instead target its empirical alternative, namely JMMD ( ˆ P X,Y , ˜ P X,Y ) . Defining X := [ x 1 , x 2 , . . . , x n ] ⊤ ,

Y := [ y 1 , y 2 , . . . , y n ] ⊤ , ˜ X := [˜ x 1 , ˜ x 2 , . . . , ˜ x m ] ⊤ , and ˜ Y := [˜ y 1 , ˜ y 2 , . . . , ˜ y m ] ⊤ , this reduces to targeting

where [ K ˜ X ˜ X ] ij := k (˜ x i , ˜ x j ) , [ K ˜ XX ] ij := k (˜ x i , x j ) , [ L ˜ Y ˜ Y ] ij := l (˜ y i , ˜ y j ) , and [ L ˜ Y Y ] ij := l (˜ y i , y j ) . One might initially assume that JKIP is more computationally expensive than JKH due to its higher cost per gradient step, but this is not the case. In fact, to construct a compressed set of size m , both JKH and JKIP have time complexity of O ( mn + m 2 ) ; see Section D.2.1 and D.2.2.

4 From Joint to Conditional Distribution Compression

4.1 The Average Maximum Conditional Mean Discrepancy

In Section 2, we recalled the MCMD, which is a function of x ∈ X that outputs a metric on the conditional distributions P Y | X = x , P Y ′ | X ′ = x with fixed conditioning values. However, for the purposes of distribution compression, we require a discrepancy measure that applies to entire families of conditional distributions P Y | X , P Y ′ | X ′ . Prior work has introduced the discrepancy

which was independently proposed as the Average Maximum Mean Discrepancy (AMMD) in [42] and the Kernel Conditional Discrepancy (KCD) in [19]. Throughout this work, we will refer to (5) as the KCD. We introduce a more general alternative, and show it is a proper metric: given an additional random variable X ∗ : Ω →X with distribution P X ∗ , we define the Average Maximum Conditional Mean Discrepancy (AMCMD) as

̸

In the above definition, the expectation is taken with respect to a distinct probability measure P X ∗ = P X , P x ′ , which broadens its applicability compared to the KCD. See Section C.2 for an illustrative example.

The following theorem establishes that the AMCMD is indeed a proper metric.

Theorem 4.1. Suppose that l : Y × Y → R is a characteristic kernel, that P X , P X ′ , and P X ∗ are absolutely continuous with respect to each other, and that P ( · | X ) and P ( · | X ′ ) admit regular versions. Then, AMCMD [ P X ∗ , P Y | X , P Y ′ | X ′ ] = 0 if and only if, for almost all x ∈ X wrt P X ∗ , P Y | X = x ( B ) = P Y ′ | X ′ = x ( B ) for all B ∈ Y .

Moreover, assuming the Radon-Nikodym derivatives d P X ∗ d P X , d P X ∗ d P X ′ , d P X ∗ d P X ′′ are bounded, then the triangle inequality is satisfied, i.e. AMCMD [ P X ∗ , P Y | X , P Y ′′ | X ′′ ] ≤ AMCMD [ P X ∗ , P Y | X , P Y ′ | X ′ ] + AMCMD [ P X ∗ , P Y ′ | X ′ , P Y ′′ | X ′′ ] .

Remark 4.2. The boundedness condition on the Radon-Nikodym derivative may be intuitively understood as a condition on the relative heaviness of the tails of the distribution P X ∗ compared to P X . For example, if P X ∗ = N ( µ, σ 2 ∗ ) and P X = N ( µ, σ 2 ) , then d P X ∗ d P X is bounded whenever σ 2 > σ 2 ∗ .

Given sets of i.i.d. samples { x ∗ i } q i =1 ∼ P X ∗ , N := { ( x i , y i ) } n i =1 ∼ P X,Y , and M := { ( x ′ i , y ′ i ) } i m =1 ∼ P X ′ ,Y ′ , we define a plug-in estimate of the AMCMD 2 as

and derive a closed form expression as follows:

Lemma 4.3.

where, for example, we have defined [ K X ′ X ∗ ] ij := k ( x ′ , x ∗ ) , [ L Y Y ′ ] ij := l ( y i , y ′ j ) , and W X ′ X ′ := ( K X ′ X ′ + λ m I ) -1 .

This estimate is O ( n 3 + m 3 + q ( n 2 + m 2 )) to compute. As a corollary of Theorem 4.5 in [19], we establish its consistency in the special case that X ∗ = X ′ = X , which corresponds to the regime under which our conditional distribution compression algorithms will operate.

Corollary 4.4. Assume that k ( · , · ) and l ( · , · ) are bounded, k ( · , · ) is universal, and let the regularisation parameters λ n and λ m decay at slower rates than O ( n -1 / 2 ) and O ( m -1 / 2 ) respectively. Then, ̂ AMCMD [ P X , P Y | X , P Y ′ | X ] p → AMCMD [ P X , P Y | X , P Y ′ | X ] as n, m, q →∞ .

Remark 4.5. The conditions in Corollary 4.4 ensure that ˆ µ Y | X and ˆ µ Y ′ | X converge to µ Y | X and µ Y ′ | X respectively in L 2 ( X , P X ; H l ) norm, that is, the norm of the space of square P X -integrable H l -valued functions [19].

Remark 4.6. The conditions on k and l are satisfied by many common choices. For example, with continuous conditional distributions, both can be Gaussian kernels; for discrete conditional distributions, l can be replaced with an indicator kernel [33].

4.2 Average Conditional Kernel Herding

We now introduce Average Conditional Kernel Herding (ACKH), a greedy algorithm that constructs a compressed set targeting the AMCMD 2 [ P X , P Y | X , ˜ P Y | X ] . The ACKH objective is derived by expanding the squared norm in (6), and ignoring the term which is invariant wrt the compressed set.

Assuming we are at the ( m + 1) th iteration, having already constructed the compressed set C = { (˜ x j , ˜ y j ) } j m =1 , we expand the compressed set C = C ∪ ( x , y ) by solving

As we do not have access to µ Y | X and P X , we must estimate this objective; this is equivalent to targeting the AMCMD 2 [ ˆ P X , ˆ P Y | X , ˜ P Y | X ] . Naïvely, one might assume that we must estimate µ Y | X using D , incurring a high computational cost of O ( n 3 ) . The following result allows us to avoid this:

Lemma 4.7. Let h : X → H l be a vector-valued function, then

Remark 4.8. If we let h ( x ) = ˜ µ C Y | X = x , then Lemma 4.7 eliminates the need to explicitly estimate µ Y | X in (8), and hence, the cost of estimating the objective is reduced from O ( n 3 ) to O ( n ) .

Even after applying Lemma 4.7 we still have an expectation in (8) which we may not be able to compute in general. Estimating this expectation using D , we obtain the closed form objective

where we define [ ¯ K ( x )] ij := k ( x i , ˜ x j ) , i = 1 , . . . , n , j = 1 , . . . , m , [ ˜ K ( x )] ij := k (˜ x i , ˜ x j ) , i, j = 1 , . . . , m , and ˜ W ( x ) := ( ˜ K ( x ) + λI ) -1 . Analogous definitions hold for the response kernel l ( · , · ) , yielding matrices ¯ L ( y ) ∈ R n × m and ˜ L ( y ) ∈ R m × m . To construct a compressed set of size m , ACKH can be shown to have an overall time complexity of O ( m 4 + m 3 n ) ; see Section D.2.3.

4.3 Average Conditional Kernel Inducing Points

Unlike JKH, where updates depend only on the newest pair in the compressed set, the presence of an inverse in the objective (9) prevents a similar simplification. 1 This leads to a quartic dependence of ACKH on the compressed set size m , which is a fairly significant limitation. By instead optimising each pair in the compressed set simultaneously, we can reduce this to just cubic dependence on m .

We refer to this algorithm as Average Conditional Kernel Inducing Points (ACKIP), solving the optimisation problem

over each pair in C = ( ˜ X , ˜ Y ) via gradient descent. Once again, even after applying Lemma 4.7 to (10) we still retain an expectation which in general is difficult to compute. Estimating this ecoectatuion using D , we obtain the closed form objective

where W ˜ X ˜ X := ( K ˜ X ˜ X + λI ) -1 . To construct a compressed set of size m , ACKIP has an overall complexity of O ( m 3 + m 2 n ) , i.e. a factor of m faster than ACKH; see Section D.2.4.

5 Experiments

Building on the experimental setup of Kernel Herding [3], we demonstrate how the methods proposed in this paper can be applied to compress the conditional distribution. We report the root mean square error RMSE ( C ) := √ 1 n ∑ n i =1 ( E [ h ( Y ) | X = x i ] -⟨ ˆ µ C Y | X = x i , h ⟩ H l ) 2 , across a range of test functions h : Y → R . This aligns with the standard applications of the KCME [17-33], where one estimates the conditional expectation of a function of interest h . When the exact value of the conditional expectation is unavailable, we approximate E [ h ( Y ) | X = x i ] via its full-data estimate, ⟨ ˆ µ D Y | X = x i , h ⟩ H l . Note that when h is the identity function, the estimate reduces to the familiar regression setting i.e. E [ Y | X ] , and when h is an indicator function, it corresponds to estimating class-conditional probabilities e.g. P ( Y = 0 | X ) . For full details of the experiments, including results on additional test functions omitted from the main text due to space constraints, see Section C.

5.1 Matching the True Conditional Distribution

In general, the expectations in (2), (3), (8), and (10) must be estimated. However, when the kernels k and l are Gaussian, and we let P X = N ( µ, σ 2 ) and P Y | X = N ( a 0 + a 1 X,σ 2 ϵ ) for µ, a 0 , a 1 ∈ R and σ 2 , σ 2 ϵ > 0 , the integrals can be evaluated analytically. See Section C.3 for details. We construct compressed sets of size m = 500 , and compute the AMCMD 2 [ P X , P Y | X , ˜ P Y | X ] achieved by each method. Figures 2 and3 highlight the advantages of directly targeting the conditional distribution, with ACKH and ACKIP achieving lower AMCMD compared to JKH and JKIP. Additionally, in the case of ACKIP versus ACKH, it demonstrates the superiority of joint optimisation over herding, where the inability to revisit previous selections limits ACKH's performance in comparison to ACKIP. Moreover, we can see that the reduced AMCMD achieved by ACKH and ACKIP translates to improved performance in estimating conditional expectations across a variety of test functions.

h(y)=sin(y)

h（y)=1(y<0}

X

>

101

'X

10-1,

102

RMSE

E

10-1.

10-1

AMCMD2[

10-3

JKH

JKIP

ACKH

102

10-4

ACKIP

10-2

100

101

102

100

101

102

100

101

102

100

101

102

Compressed Set Size

Compressed Set Size

Compressed SetSize

CompressedSetSize

Random

10-1

h(y)=sin(y)

h(y)=e-y²

h（y)=1(y<0}

JKH

ACKH

10-1

JKIP

ACKIP

RMSE

ACKIP

10-2

ACKIP

10

-2

ACKIP

ACKIP

主

10-2

5.2 Matching the Empirical Conditional Distribution

In this section, we present experiments targeting the empirical conditional distribution of synthetic and real-data. Across all datasets, we generate or subsample down to n = 10 , 000 pairs, split off 10% for validation, 10% for testing, and construct compressed sets up to size m = 250 .

5.2.1 Continuous Conditional Distributions

Real : We use the Superconductivity dataset from UCI [44]. Superconductivity is composed of d = 81 features relating to the chemical composition of superconductors with the target being its critical temperature [45]. In Figure 4 and 5 we see that ACKIP achieves the lowest RMSE across each of the test functions, with ACKH in second for all but one. We also note that JKIP achieves favourable performance versus JKH.

×10-1

100

h(y)=sin(y)

h(y)=e-y²

h（y)=1(y<0}

6

4

RMSE

3

Random

10-1

JKH

2

ACKH

JKIP

ACKIP

10-1

100

101

102

100

101

102

100

101

102

100

101

102

Compressed Set Size

Compressed Set Size

Compressed Set Size

Compressed Set Size

×10-1

×10-1

Random

h(y)=sin(y)

h(y) =e-y²

h(y)=1(y<0}

JKH

ACKH

JKIP

3

ACKIP

RMSE

10-1

3

ACKIP

ACKIP

ACKIP

ACKIP

Synthetic : We design a challenging dataset with a highly non-linear feature-response relationship and pronounced heteroscedastic noise, referring to it as Heteroscedastic going forward. Let

P X = N (0 , 2 2 ) and P Y | X = x = N ( f ( x ) , σ 2 ( x )) , with f ( x ) := ∑ 4 i =1 a i exp ( -1 b i ( x -c i ) 2 ) , and σ 2 ( x ) := σ 2 1 + ∣ ∣ σ 2 2 sin( x ) ∣ ∣ . Figure 1 compares a compressed set constructed by ACKIP, with the random sample which initialised the optimisation procedure. It demonstrates how random sampling fails to adequately represent key areas of the data cloud, and how ACKIP can construct a better representation. In Figures 6 and 7 we see that ACKIP attains the lowest RMSE across three of the four test functions. For the remaining function, all methods exhibit relatively similar performance. We also note that JKIP outperforms or achieves similar performance versus JKH across the test functions.

×10-1

100

Full Data

h(y)=sin(y)

h(y)=e-y²

h（y)=1(y<0}

Random

JKH

ACKH

4

JKIP

ACKIP

RMSE

3

10-1

100

101

102

100

101

102

100

101

102

10-1

100

101

102

Compressed Set Size

Compressed Set Size

Compressed Set Size

Compressed SetSize

×10-1

x10-1

h(y=y

Full Data

h(y) = sin(y)

h(y)=e-y²

h(y)=l{y<0}

3

Random

2.0

JKH

ACKH

JKIP

ACKIP

1.8

RMSE

1.6

ACKIP

ACKIP

ACKIP

ACKIP

1

1.4

10-1

1.2

10-

5.2.2 Discrete Conditional Distributions

Conditional distributions may also be discrete, as in classification settings where responses take one of C distinct values y ∈ { 0 , 1 , . . . , C } . In such cases, applying an indicator kernel on the response space enables the KCME to serve as a consistent multi-class classifier, in contrast to methods like SVCs and GPCs [33]. The use of the indicator kernel renders standard gradient-based optimisation inapplicable on the response space, necessitating an alternative optimisation strategy (see Section C.1.3 for details). Figure 8 illustrates the strong performance of ACKIP on a challenging, synthetic, imbalanced four-class classification dataset, which we refer to as Imbalanced . We see that the KCME, trained with the compressed set constructed by ACKIP, estimates the class-conditional probabilities with accuracy that very closely matches that of the full dataset at just 3% of the size. In contrast, the figure also exposes the limitations of the herding approach: ACKH performs worse than random on three of the four classes, and JKIP outperforms JKH on three out of the four classes. For full details on Imbalanced , and additional experimental results, including on MNIST, see see Section C.1.3.

×10-1

h(y)=1{y=0}

h(y)=1(y=1}

h(y)=1{y=2}

×10-1

h(y)=1{y=3}

2.6

Full Data

ACKH

2.2

2.4

+

Random

JKIP

2.0

JKH

ACKIP

2.2

1.8

RMS

S

E

2.0

1.6

1.8

1.6

10-1

10-1

1.4

ACKIP

1.4

ACKIP

ACKIP

ACKIP

6 Discussion

Applicability : ACKIP generates a compressed set that enables efficient estimation (from O ( n 3 ) to O ( m 3 + m 2 n ) ) and evaluation (from O ( n 2 ) to O ( m 2 ) ) of the KCME while maintaining a close approximation to the true KCME in terms of the AMCMD. The KCME is widely used across various applications [17-33], despite its original O ( n 3 ) computational cost. By reducing this to O ( n ) , whilst impacting empirical performance minimally, our approach may significantly expand the range of scenarios where the KCME can be practically applied. In particular, wherever one may have used a random subsample of size m to estimate the KCME with cost O ( m 3 ) , ACKIP can be inserted, where running even just a few iterations of the algorithm increases the efficacy of the random sample at just O ( nm 2 ) additional cost.

Limitations : Our algorithms currently lack a formal convergence guarantee. Although empirical results consistently show convergence across a wide range of experimental settings, including both real-world and synthetic conditional distributions, continuous and discrete, a rigorous theoretical proof remains open. In practical terms, our approach depends on computing kernel gradients, which may be unsuitable for data domains where gradients are ill-defined or difficult to interpret, such as graphs or text [46, 47]. In such cases, gradient-free alternatives inspired by Kernel Thinning [12-14] may be preferable to versions of JKH and ACKH that greedily select optimal sample pairs directly from the existing dataset (see Algorithms 5 and 6).

7 Conclusions

We showed that existing distribution compression methods can be extended to target the joint distribution, introducing JKH and JKIP. In particular, JKIP removes the heuristic limitations of greedy optimisation by jointly optimising the compressed set, while preserving the same computational cost. We then presented the AMCMD, an extension of the MCMD that defines a proper metric on families of conditional distributions. We derive a closed form estimate of the AMCMD and demonstrate that it can be consistently estimated in the regime of our compression algorithms. Then, leveraging the AMCMD, we proposed ACKH and ACKIP, two linear-time conditional distribution compression algorithms that are the first of their kind. Experimentation across a range of scenarios indicates that it is preferable to compress the conditional distribution directly using ACKIP or ACKH, rather than through the joint distribution via JKH or JKIP. Moreover, we see that the greedy optimisation approach of ACKH limits its empirical performance, and increases its computational cost, versus ACKIP. Finally, we also note that JKIP consistently outperforms JKH across our experimental settings. This work opens up numerous promising avenues for future research; for a detailed discussion of potential directions, see Section A.2.

References

N Aronszajn. Theory of reproducing kernels. Trans. Amer. Math. Soc. , 68:337-404, 1950. doi: 10.2307/1990404.

Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. Journal of Machine Learning Research , 13(25):723-773, 2012. URL http://jmlr.org/papers/v13/gretton12a.html .

Yutian Chen, Max Welling, and Alex Smola. Super-samples from kernel herding. In Proceedings of the Twenty-Sixth Conference on Uncertainty in Artificial Intelligence , UAI'10, page 109-116, Arlington, Virginia, USA, 2010. AUAI Press. ISBN 9780974903965.

Ferenc Huszár and David Duvenaud. Optimally-weighted herding is bayesian quadrature. In Proceedings of the Twenty-Eighth Conference on Uncertainty in Artificial Intelligence , UAI'12, page 377-386, Arlington, Virginia, USA, 2012. AUAI Press. ISBN 9780974903989.

Francis Bach. On the equivalence between kernel quadrature rules and random feature expansions. J. Mach. Learn. Res. , 18(1):714-751, January 2017. ISSN 1532-4435.

Ayoub Belhadji, Rémi Bardenet, and Pierre Chainais. Kernel quadrature with dpps. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019. URL https://proceedings.neurips.cc/paper_files/paper/2019/file/ 7012ef0335aa2adbab58bd6d0702ba41-Paper.pdf .

Satoshi Hayakawa, Harald Oberhauser, and Terry Lyons. Positively weighted kernel quadrature via subsampling, 2022. URL https://arxiv.org/abs/2107.09597 .

Satoshi Hayakawa, Harald Oberhauser, and Terry Lyons. Sampling-based nyström approximation and kernel quadrature, 2023. URL https://arxiv.org/abs/2301.09517 .

Ethan N. Epperly and Elvira Moreno. Kernel quadrature with randomly pivoted cholesky. ArXiv , abs/2306.03955, 2023. URL https://api.semanticscholar.org/CorpusID: 259095887 .

Simon Mak and V. Roshan Joseph. Support points, 2018. URL https://arxiv.org/abs/ 1609.01811 .

Michael Arbel, Anna Korba, Adil Salim, and Arthur Gretton. Maximum mean discrepancy gradient flow, 2019. URL https://arxiv.org/abs/1906.04370 .

Raaz Dwivedi and Lester Mackey. Generalized kernel thinning, 2021. URL https://arxiv. org/abs/2110.01593 .

Raaz Dwivedi and Lester Mackey. Kernel thinning. In Mikhail Belkin and Samory Kpotufe, editors, Proceedings of Thirty Fourth Conference on Learning Theory , volume 134 of Proceedings of Machine Learning Research , pages 1753-1753. PMLR, 8 2021. URL https://proceedings.mlr.press/v134/dwivedi21a.html .

Abhishek Shetty, Raaz Dwivedi, and Lester Mackey. Distribution compression in near-linear time, 2022. URL https://arxiv.org/abs/2111.07941 .

Mingsheng Long, Han Zhu, Jianmin Wang, and Michael I. Jordan. Deep transfer learning with joint adaptation networks. In Proceedings of the 34th International Conference on Machine Learning - Volume 70 , ICML'17, page 2208-2217. JMLR.org, 2017.

Zonghao Chen, Toni Karvonen, Heishiro Kanagawa, François-Xavier Briol, and Chris. J. Oates. Stationary mmd points for cubature, 2025. URL https://arxiv.org/abs/2505.20754 .

Le Song, Jonathan Huang, Alex Smola, and Kenji Fukumizu. Hilbert space embeddings of conditional distributions with applications to dynamical systems. In Proceedings of the 26th Annual International Conference on Machine Learning , ICML '09, page 961-968, New York, NY, USA, 2009. Association for Computing Machinery. ISBN 9781605585161. doi: 10.1145/1553374.1553497. URL https://doi.org/10.1145/1553374.1553497 .

Junhyung Park and Krikamol Muandet. A measure-theoretic approach to kernel conditional mean embeddings. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 21247-21259. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper_files/paper/ 2020/file/f340f1b1f65b6df5b5e3f94d95b11daf-Paper.pdf .

J. Park, U. Shalit, B. Schölkopf, and K. Muandet. Conditional distributional treatment effect with kernel conditional mean embeddings and u-statistic regression. In Proceedings of 38th International Conference on Machine Learning (ICML) , volume 139 of Proceedings of Machine Learning Research , pages 8401-8412. PMLR, July 2021. URL https://proceedings.mlr. press/v139/park21c.html .

Kun Zhang, Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Kernel-based conditional independence test and application in causal discovery. In Proceedings of the Twenty-Seventh Conference on Uncertainty in Artificial Intelligence , UAI'11, page 804-813, Arlington, Virginia, USA, 2011. AUAI Press. ISBN 9780974903972.

Ingmar Schuster, Mattes Mollenhauer, Stefan Klus, and Krikamol Muandet. Kernel conditional density operators. In Silvia Chiappa and Roberto Calandra, editors, Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics , volume 108 of Proceedings of Machine Learning Research , pages 993-1004. PMLR, 26-28 Aug 2020. URL https://proceedings.mlr.press/v108/schuster20a.html .

Jake D Spiteri. Nonparametric Estimation with Kernel Mean Embeddings . PhD thesis, Bristol Doctoral College, School of Mathematics, 2024.

Eiki Shimizu, Kenji Fukumizu, and Dino Sejdinovic. Neural-kernel conditional mean embeddings, 3 2024. URL https://arxiv.org/abs/2403.10859 .

Kelvin Hsu and Fabio Tozeto Ramos. Bayesian learning of conditional kernel mean embeddings for automatic likelihood-free inference. In International Conference on Artificial Intelligence and Statistics , 2019. URL https://api.semanticscholar.org/CorpusID:67855374 .

Sayak Ray Chowdhury, Rafael Oliveira, and Fabio Tozeto Ramos. Active learning of conditional mean embeddings via bayesian optimisation. In Conference on Uncertainty in Artificial Intelligence , 2020. URL https://api.semanticscholar.org/CorpusID:220623530 .

Le Song, Byron Boots, Sajid M. Siddiqi, Geoffrey Gordon, and Alex Smola. Hilbert space embeddings of hidden markov models. In Proceedings of the 27th International Conference on International Conference on Machine Learning , ICML'10, page 991-998, Madison, WI, USA, 2010. Omnipress. ISBN 9781605589077.

Le Song, Arthur Gretton, and Carlos Guestrin. Nonparametric tree graphical models. In Yee Whye Teh and Mike Titterington, editors, Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics , volume 9 of Proceedings of Machine Learning Research , pages 765-772, Chia Laguna Resort, Sardinia, Italy, 5 2010. PMLR. URL https://proceedings.mlr.press/v9/song10a.html .

Le Song, Arthur Gretton, Danny Bickson, Yucheng Low, and Carlos Guestrin. Kernel belief propagation. In Geoffrey Gordon, David Dunson, and Miroslav Dudík, editors, Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics , volume 15 of Proceedings of Machine Learning Research , pages 707-715, Fort Lauderdale, FL, USA, 11-13 Apr 2011. PMLR. URL https://proceedings.mlr.press/v15/song11a.html .

Spencer Young, Cole Edgren, Riley Sinema, Andrew Hall, Nathan Dong, and Porter Jenkins. Beyond calibration: Assessing the probabilistic fit of neural regressors via conditional congruence, 2024. URL https://arxiv.org/abs/2405.12412 .

Steffen Grünewälder, Guy Lever, Arthur Gretton, Luca Baldassarre, Sam Patterson, and Massimiliano Pontil. Conditional mean embeddings as regressors. ArXiv , abs/1205.4656, 2012. URL https://api.semanticscholar.org/CorpusID:14671775 .

Yu Nishiyama, Abdeslam Boularias, Arthur Gretton, and Kenji Fukumizu. Hilbert space embeddings of pomdps. Uncertainty in Artificial Intelligence - Proceedings of the 28th Conference, UAI 2012 , 10 2012.

Byron Boots, Arthur Gretton, and Geoffrey J. Gordon. Hilbert space embeddings of predictive state representations. In Proceedings of the Twenty-Ninth Conference on Uncertainty in Artificial Intelligence , UAI'13, page 92-101, Arlington, Virginia, USA, 2013. AUAI Press.

Kelvin Hsu, Richard Nock, and Fabio Ramos. Hyperparameter learning for conditional kernel mean embeddings with rademacher complexity bounds, 2018. URL https://arxiv.org/ abs/1809.00175 .

Rafael Izbicki and Ann B. Lee. Nonparametric conditional density estimation in a highdimensional regression setting. Journal of Computational and Graphical Statistics , 25(4): 1297-1316, October 2016. ISSN 1537-2715. doi: 10.1080/10618600.2015.1094393. URL http://dx.doi.org/10.1080/10618600.2015.1094393 .

Alain Berlinet and Christine Thomas-Agnan. Reproducing Kernel Hilbert Space in Probability and Statistics . Springer US, 01 2004. ISBN 978-1-4613-4792-7. doi: 10.1007/ 978-1-4419-9096-9.

Charles A. Micchelli and Massimiliano Pontil. On Learning Vector-Valued Functions. Neural Computation , 17(1):177-204, 01 2005. ISSN 0899-7667. doi: 10.1162/0899766052530802. URL https://doi.org/10.1162/0899766052530802 .

Krikamol Muandet, Kenji Fukumizu, Bharath Sriperumbudur, and Bernhard Schölkopf. Kernel mean embedding of distributions: A review and beyond. Foundations and Trends® in Machine Learning , 10(1-2):1-141, 2017. ISSN 1935-8245. doi: 10.1561/2200000060. URL http: //dx.doi.org/10.1561/2200000060 .

Claudio Carmeli, Ernesto De Vito, Alessandro Toigo, and V. Umanità. Vector valued reproducing kernel hilbert spaces and universality. Analysis and Applications , 08, 11 2011. doi: 10.1142/S0219530510001503.

Bharath K. Sriperumbudur, Kenji Fukumizu, and Gert R.G. Lanckriet. Universality, characteristic kernels and rkhs embedding of measures. Journal of Machine Learning Research , 12(70): 2389-2410, 2011. URL http://jmlr.org/papers/v12/sriperumbudur11a.html .

Zoltán Szabó and Bharath K. Sriperumbudur. Characteristic and universal tensor product kernels. Journal of Machine Learning Research , 18(233):1-29, 2018. URL http://jmlr. org/papers/v18/17-492.html .

Francis Bach, Simon Lacoste-Julien, and Guillaume Obozinski. On the equivalence between herding and conditional gradient algorithms. In Proceedings of the 29th International Coference on International Conference on Machine Learning , ICML'12, page 1355-1362, Madison, WI, USA, 2012. Omnipress. ISBN 9781450312851.

Ziyi Huang, Henry Lam, and Haofeng Zhang. Evaluating aleatoric uncertainty via conditional generative models, 2022. URL https://arxiv.org/abs/2206.04287 .

D.K. Faddeev and V.N. Faddeeva. Computational methods of linear algebra . A series of undergraduate books in mathematics. W.H. Freeman, 1963. URL https://books.google. co.uk/books?id=KsIV0QEACAAJ .

Kolby Nottingham Markelle Kelly, Rachel Longjohn. The uci machine learning repository. URL https://archive.ics.uci.edu .

Kam Hamidieh. A data-driven statistical model for predicting the critical temperature of a superconductor, 2018. URL https://arxiv.org/abs/1803.10260 .

Thomas Gärtner, Peter A. Flach, and Stefan Wrobel. On graph kernels: Hardness results and efficient alternatives. In Annual Conference Computational Learning Theory , 2003. URL https://api.semanticscholar.org/CorpusID:10856944 .

Huma Lodhi, John Shawe-Taylor, Nello Cristianini, and Christopher Watkins. Text Classification using String Kernels. In T Leen, T Dietterich, and V Tresp, editors, Advances in Neural Information Processing Systems , volume 13. MIT Press, 2000. URL https://proceedings.neurips.cc/paper_files/paper/2000/file/ 68c694de94e6c110f42e587e8e48d852-Paper.pdf .

Dino Sejdinovic, Bharath Sriperumbudur, Arthur Gretton, and Kenji Fukumizu. Equivalence of distance-based and rkhs-based statistics in hypothesis testing. The Annals of Statistics , 41(5), October 2013. ISSN 0090-5364. doi: 10.1214/13-aos1140. URL http://dx.doi.org/10. 1214/13-AOS1140 .

Noveen Sachdeva and Julian McAuley. Data distillation: A survey, 2023. URL https: //arxiv.org/abs/2301.04272 .

Yong Ren, Jialian Li, Yucen Luo, and Jun Zhu. Conditional generative moment-matching networks. In Proceedings of the 30th International Conference on Neural Information Processing Systems , NIPS'16, page 2936-2944, Red Hook, NY, USA, 2016. Curran Associates Inc. ISBN 9781510838819.

Pengfei Ge, Chuan-Xian Ren, Dao-Qing Dai, and Hong Yan. Domain adaptation and image classification via deep conditional adaptation network, 2022. URL https://arxiv.org/abs/ 2006.07776 .

Tushar and Souvik Chakraborty. Deep physics corrector: A physics enhanced deep learning architecture for solving stochastic differential equations. Journal of Computational Physics , 479:112004, April 2023. ISSN 0021-9991. doi: 10.1016/j.jcp.2023.112004. URL http: //dx.doi.org/10.1016/j.jcp.2023.112004 .

Boya Hou, Sina Sanjari, Nathan Dahlin, Alec Koppel, and Subhonmesh Bose. Nonparametric sparse online learning of the koopman operator, 2025. URL https://arxiv.org/abs/2405. 07432 .

Guy Lever, John Shawe-Taylor, Ronnie Stafford, and Csaba Szepesvari. Compressed conditional mean embeddings for model-based reinforcement learning. In AAAI Conference on Artificial Intelligence , 2016. URL https://api.semanticscholar.org/CorpusID:6059767 .

Tamim El Ahmad, Luc Brogat-Motte, Pierre Laforgue, and Florence d'Alch'e Buc. Sketch in, sketch out: Accelerating both learning and inference for structured prediction with kernels. ArXiv , abs/2302.10128, 2023. URL https://api.semanticscholar.org/CorpusID: 257038736 .

Michael W. Mahoney. Randomized algorithms for matrices and data, 2011. URL https: //arxiv.org/abs/1104.5557 .

Boya Hou, Sina Sanjari, Nathan Dahlin, and Subhonmesh Bose. Compressed decentralized learning of conditional mean embedding operators in reproducing kernel hilbert spaces. In Proceedings of the Thirty-Seventh AAAI Conference on Artificial Intelligence and Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence and Thirteenth Symposium on Educational Advances in Artificial Intelligence , AAAI'23/IAAI'23/EAAI'23. AAAI Press, 2023. ISBN 978-1-57735-880-0. doi: 10.1609/aaai.v37i7.25956. URL https://doi.org/ 10.1609/aaai.v37i7.25956 .

Albert Gong, Kyuseong Choi, and Raaz Dwivedi. Supervised kernel thinning, 2025. URL https://arxiv.org/abs/2410.13749 .

Masashi Sugiyama, Matthias Krauledat, and Klaus-Robert Müller. Covariate shift adaptation by importance weighted cross validation. Journal of Machine Learning Research , 8(35):985-1005, 2007. URL http://jmlr.org/papers/v8/sugiyama07a.html .

Jose G. Moreno-Torres, Troy Raeder, Rocío Alaiz-Rodríguez, Nitesh V. Chawla, and Francisco Herrera. A unifying view on dataset shift in classification. Pattern Recognition , 45(1):521-530, 2012. ISSN 0031-3203. doi: https://doi.org/10.1016/j.patcog.2011.06.019. URL https: //www.sciencedirect.com/science/article/pii/S0031320311002901 .

Fuzhen Zhuang, Zhiyuan Qi, Keyu Duan, Dongbo Xi, Yongchun Zhu, Hengshu Zhu, Hui Xiong, and Qing He. A comprehensive survey on transfer learning, 2020. URL https: //arxiv.org/abs/1911.02685 .

Abolfazl Farahani, Sahar Voghoei, Khaled Rasheed, and Hamid R. Arabnia. A brief review of domain adaptation, 2020. URL https://arxiv.org/abs/2010.03978 .

Jiayuan Huang, Arthur Gretton, Karsten Borgwardt, Bernhard Schölkopf, and Alex Smola. Correcting sample selection bias by unlabeled data. In B. Schölkopf, J. Platt, and T. Hoffman, editors, Advances in Neural Information Processing Systems , volume 19. MIT Press, 2006. URL https://proceedings.neurips.cc/paper_files/paper/2006/ file/a2186aa7c086b46ad4e8bf81e2a3a19b-Paper.pdf .

Liwen Ouyang and Aaron Key. Maximum mean discrepancy for generalization in the presence of distribution and missingness shift, 2022. URL https://arxiv.org/abs/2111.10344 .

Hongliang Yan, Yukang Ding, Peihua Li, Qilong Wang, Yong Xu, and Wangmeng Zuo. Mind the class weight bias: Weighted maximum mean discrepancy for unsupervised domain adaptation, 2017. URL https://arxiv.org/abs/1705.00609 .

Wei Wang, Haojie Li, Zhengming Ding, and Zhihui Wang. Rethink maximum mean discrepancy for domain adaptation, 2020. URL https://arxiv.org/abs/2007.00689 .

Kun Zhang, Bernhard Schölkopf, Krikamol Muandet, and Zhikun Wang. Domain adaptation under target and conditional shift. In Proceedings of the 30th International Conference on International Conference on Machine Learning - Volume 28 , ICML'13, page III-819-III-827. JMLR.org, 2013.

Xuezhi Wang and Jeff Schneider. Flexible transfer learning under support and model shift. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems , volume 27. Curran Associates, Inc., 2014. URL https://proceedings.neurips.cc/paper_files/paper/2014/file/ 21085aa904b9fe66bf35f67c34d176d0-Paper.pdf .

Remi Tachet des Combes, Han Zhao, Yu-Xiang Wang, and Geoff Gordon. Domain adaptation with conditional distribution matching and generalized label shift. In Proceedings of the 34th International Conference on Neural Information Processing Systems , NIPS '20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829546.

Junhyunng Park and Krikamol Muandet. Regularised least-squares regression with infinitedimensional output space, 2022. URL https://arxiv.org/abs/2010.10973 .

Zhu Li, Dimitri Meunier, Mattes Mollenhauer, and Arthur Gretton. Optimal rates for regularized conditional mean embedding learning, 2023. URL https://arxiv.org/abs/2208.01711 .

Zhu Li, Dimitri Meunier, Mattes Mollenhauer, and Arthur Gretton. Towards optimal sobolev norm rates for the vector-valued regularized least-squares algorithm, 2024. URL https: //arxiv.org/abs/2312.07186 .

Timothy Nguyen, Zhourong Chen, and Jaehoon Lee. Dataset meta-learning from kernel ridgeregression. CoRR , abs/2011.00050, 2020. URL https://arxiv.org/abs/2011.00050 .

Trevor Campbell and Boyan Beronov. Sparse variational inference: Bayesian coresets from scratch, 2019. URL https://arxiv.org/abs/1906.03329 .

Dionysis Manousakas, Zuheng Xu, Cecilia Mascolo, and Trevor Campbell. Bayesian pseudocoresets. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 14950-14960. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper_files/paper/2020/file/ ab452534c5ce28c4fbb0e102d4a4fb2e-Paper.pdf .

Ingo Steinwart. On the influence of the kernel on the consistency of support vector machines. J. Mach. Learn. Res. , 2:67-93, March 2002. ISSN 1532-4435. doi: 10.1162/153244302760185252. URL https://doi.org/10.1162/153244302760185252 .

James E. Gentle. Matrix Algebra: Theory, Computations, and Applications in Statistics . Springer Publishing Company, Incorporated, 1st edition, 2007. ISBN 0387708723.

K. B. Petersen and M. S. Pedersen. The matrix cookbook, October 2008. URL http://www2. imm.dtu.dk/pubdb/p.php?3274 . Version 20081110.

James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http://github.com/jax-ml/jax .

Damien Garreau, Wittawat Jitkrittum, and Motonobu Kanagawa. Large sample analysis of the median heuristic, 2018. URL https://arxiv.org/abs/1707.07269 .

Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017. URL https://arxiv.org/abs/1412.6980 .

DeepMind, Igor Babuschkin, Kate Baumli, Alison Bell, Surya Bhupatiraju, Jake Bruce, Peter Buchlovsky, David Budden, Trevor Cai, Aidan Clark, Ivo Danihelka, Antoine Dedieu, Claudio Fantacci, Jonathan Godwin, Chris Jones, Ross Hemsley, Tom Hennigan, Matteo Hessel, Shaobo Hou, Steven Kapturowski, Thomas Keck, Iurii Kemaev, Michael King, Markus Kunesch, Lena Martens, Hamza Merzic, Vladimir Mikulik, Tamara Norman, George Papamakarios, John Quan, Roman Ring, Francisco Ruiz, Alvaro Sanchez, Laurent Sartran, Rosalia Schneider, Eren Sezener, Stephen Spencer, Srivatsan Srinivasan, Miloš Stanojevi´ c, Wojciech Stokowiec, Luyu Wang, Guangyao Zhou, and Fabio Viola. The DeepMind JAX Ecosystem, 2020. URL http://github.com/google-deepmind .

Yann LeCun and Corinna Cortes. Mnist handwritten digit database, 2010. URL http://yann. lecun.com/exdb/mnist/ .

Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278-2324, 1998.

F.W. Byron and R.W. Fuller. Mathematics of Classical and Quantum Physics . Number v. 1-2 in Dover books on physics and chemistry. Dover Publications, 1992. ISBN 9780486671642. URL https://books.google.co.uk/books?id=pndBQOYzlP8C .

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Our claims are backed by both theoretical and experimental evidence. Our main contributions are outlined in Section 1, with links to the relevant sections of the paper. See Section B for proofs of our theoretical claims and Section C for further experimental evidence.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: See Section 6 for a discussion of limitations. Whenever assumptions have been made, effort has been put forth to describe their strength e.g. in Section B.1. Computational complexity of our algorithms has been discussed in Section D.2.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: General assumptions are detailed in Section B.1, with additional assumptions listed in the theorems/lemmas themselves. All proofs are in Section B.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We use publicly accessible datasets, and all details required to reproduce the experiments are included in Section C. Code to reproduce the results can be found at https: //github.com/conditionaldummy/dummy_repo , with scripts to run experiments and a notebook to make figures included in the supplemental material.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We use publicly accessible datasets, and we have open-sourced our code at https://github.com/conditionaldummy/dummy_repo .

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: In Section C we reproduce all necessary details of our experiments. All details are also further included in the experiment scripts provided.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Where appropriate, we have included error bars in the form of 25 th and 75 th percentiles.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We have included the hardware and software specifications used to conduct our experiments in Section C.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We followed the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [No]

Justification: Our work is not tied to any particular applications, and is not related to any private or personal data, and there is no explicit negative social impact.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [No]

Justification: The work is not tied to any particular applications, and so we do not foresee any high risk for misuse of this work

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The creators of any of the assets used in this paper, such as code and data, have been appropriately recognised, with licenses and terms of use clearly mentioned and respected.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [NA]

Justification: LLMs were not used in the development of this work.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

Appendix

Table of Contents

A Further Discussion

In this section we include further discussions and conclusions that could not fit in the main body, including related work, applications, limitations and future work.

A.1 Related Work

A.1.1 Standard Distribution Compression

As noted in the introduction, numerous distribution compression methods exist which target the distribution of unlabelled data. Most of these are kernel-based methods that target the MMD, with one notable exception (Support Points), which can actually be shown to be equivalent to a kernel-based method for a specific choice of kernel. To the best of our knowledge, no existing method performs joint or conditional distribution compression as defined in this work. We now briefly summarise the existing approaches.

Kernel Herding [3, 41] constructs a compressed set by greedily minimising the MMD, optimising one point at a time. Let X := [ x 1 , x 2 , . . . , x n ] ⊤ ⊂ R d be an i.i.d. sample from the target distribution P X , and let Z := [ z 1 , z 2 , . . . , z m ] ⊤ ⊂ R d denote the current compressed set. At each iteration, the next point is chosen by solving

after which the compressed set is updated to Z := [ z 1 , z 2 , . . . , z m , z m +1 ] ⊤ , and the process is repeated. This greedy strategy has the advantage of focusing computational effort on optimising one new point at a time while leaving previously selected points fixed. However, it may yield suboptimal solutions, as earlier selections are never revisited or refined. This optimisation approach is used to target the joint distribution in Joint Kernel Herding, and the conditional distribution in Average Conditional Kernel Herding.

Gradient Flow [11] methods address the problem of distribution compression by solving a discretised Wasserstein gradient flow of the MMD. In practice, this corresponds to performing gradient descent on all points in the compressed set simultaneously, i.e., solving

Under certain technical convexity assumptions on the objective [11], this method can be shown to converge to the global optimum; in practice, however, one should only expect to obtain a locally optimal solution. We take this optimisation approach to target the joint distribution in Joint Kernel Inducing Points, and the conditional distribution in Average Conditional Kernel Inducing Points.

Kernel Thinning [12-14] constructs a compressed set that is a proper subset of the original dataset, i.e., all points in the compressed set are drawn directly from the input data. This restriction stems from the method's original motivation of thinning the output of Markov Chain Monte Carlo (MCMC) methods, where subsampling is commonly referred to as standard thinning. Kernel thinning proceeds via a two-stage procedure targeting the MMD. First, the input dataset is probabilistically split into 2 m MMD-balanced candidate sets, each of size ⌊ n/ 2 m ⌋ , discarding the remaining n -2 m ⌊ n/ 2 m ⌋ points if n is not evenly divisible. Second, the best candidate set from this partitioning is selected, and then greedily refined by iteratively replacing points with others from the original dataset whenever doing so improves the MMD. This construction enables the derivation of convergence rates that are state of the art in the literature, however practically it has the significant disadvantage of throwing away potentially significant amounts of information in the n -2 m ⌊ n/ 2 m ⌋ points.

Support Points [10] is a method for constructing compressed sets that takes a similar optimisationbased approach to Gradient Flow methods, but targets the Energy Distance (ED) rather than the MMD. The ED between distributions P X and Q X is defined as

where a , a ′ ∼ P X and b , b ′ ∼ Q X . Much like the MMD, the energy distance is zero if and only if P X = Q X [10, Theorem 1]. Although Support Points is not initially expressed in kernel form, the energy distance is in fact equivalent to the MMD for a particular choice of negative-definite kernel [48].

A.1.2 Dataset Distillation

Dataset distillation produces a compressed set which attempts to replicate the performance of the original data on a downstream task, most typically image classification [49]. However, these algorithms are model-dependent and preserve task-specific performance. In contrast, distribution compression is model-agnostic: a distributional discrepancy is targeted, independent of any downstream model. The aim is not to preserve performance on a particular model, but rather to preserve the distribution itself under compression. Therefore, the approaches introduced in this work are task-agnostic : the compressed set could be reused across diverse downstream applications, as discussed in the introduction.

A.1.3 Maximum Mean Discrepancies for Conditional Distributions

The Maximum Mean Discrepancy (MMD) was introduced as a metric on the space of distributions P X and has become widely used in machine learning [2]. More recently, there has been growing interest in developing MMD-like discrepancy measures for conditional distributions [18, 19, 42, 50]. One such discrepancy:

was introduced as the Kernel Conditional Discrepancy (KCD) by [19] to measure conditional distributional treatment effects. Independently, [42] proposed the same object under the name

Average Maximum Mean Discrepancy (AMMD) in the context of generative modelling. They are limited to cases the outer expectation must be taken with respect to the distribution of the shared conditioning variable P X . In contrast, we introduce the Average Maximum Conditional Mean Discrepancy (AMCMD):

which allows the outer expectation to be taken with respect to a distinct distribution over the conditioning variable. This generalisation extends the applicability of AMCMD beyond KCD and AMMD while still recovering their results when setting X = X ′ = X ∗ . Moreover, we show that the AMCMD satisfies the identity of indiscernibles, and the triangle inequality under the conditions of Theorem 4.1, therefore satisfying the properties of a proper metric over the space of conditional distributions. Furthermore, unlike in [42], we show that the AMCMD can be consistently estimated using i.i.d. samples from the joint distribution, rather than requiring access to i.i.d. features x i with conditionally independent responses y i,j at each x i -a highly restrictive assumption. We derive a closed-form estimator for the AMCMD that is similar in form to the one proposed for KCD by [19], and show consistency of the estimate in the case that X ∗ = X ′ = X (Corollary 4.4), which is the case under which our compression algorithms lie.

The Conditional Maximum Mean Discrepancy (CMMD) [50], defined as

measures the Hilbert-Schmidt norm of the operator difference µ Y | X -µ Y ′ | X . Originally developed for use in moment-matching networks, the CMMD has since been applied to domains such as domain adaptation [51] and stochastic differential equations (SDEs) [52]. However, as noted in [42, 18], the strong assumptions required to ensure that µ Y | X and µ Y ′ | X exist as Hilbert-Schmidt operators imply that CMMD may not even be well-defined at the population level in many settings, unlike the AMCMD.

A.1.4 Accelerating the Kernel Conditional Mean Embedding

The most closely related work is that of [53], which leverages the equivalence between the operatortheoretic estimate of the KCME and the solution to a vector-valued kernel ridge regression problem. They develop an operator-valued stochastic gradient descent algorithm to learn the KCME operator from streaming data. While both our approach and theirs utilise gradient-based methods, our work is fundamentally different. Instead of learning the operator , we learn the compressed set itself via gradient descent. Moreover, by identifying an MMD-based objective function, we establish connections with the distribution compression literature. This shift in perspective leads to a significantly different formulation and set of theoretical insights.

Beyond this, other approaches exist that are less similar. Some methods aim to speed up evaluation of the trained KCME at arbitrary input, rather than the training process itself: [30] and [54] use LASSO regression to construct sparse KCME estimates for efficient repeated queries. Working in Bayesian Optimisation, [25] introduce a greedy algorithm that sequentially optimises the conditional expectation of a fixed function f ∈ H k using the KCME. Meanwhile, [55] apply sketching techniques [56] to approximate the KCME, however they do not deliver a compressed set. Finally, [57] propose a decentralised approach where a network of agents collaboratively approximates the KCME by optimising sparse covariance operators and exchanging them across the network.

In contrast to these methods, by framing the problem through an MMD-based objective function and directly optimising the compressed set, we introduce a new perspective that enables both theoretical advancements and practical improvements in scalable conditional distribution compression.

A.1.5 Supervised Kernel Thinning

In [58], the authors apply the method of Kernel Thinning [12-14] in order to accelerate the training of two non-parametric regression models: Nadaraya-Watson (NW) kernel regression, and kernel ridge regression (KRR). That is, given a labelled dataset { ( x i , y i ) } n i =1 , x ∈ R d , y ∈ R , they construct a compressed set with Kernel Thinning, using a specialised input kernel, and then derive better-than-iid-subsampling bounds on the MSE achieved by the model trained with the compressed set.

More specifically, they use

for the construction of the compressed set targeting the Nadaraya-Watson model, and

for the construction of the compressed set targeting the kernel ridge regression model. For NW regression the feature kernel k can be infinite dimensional, and their results still hold, however for KRR, their better-than-iid bounds hold only for finite dimensional feature kernels k .

Very importantly, they make no claims about compression of the joint or conditional distribution, and indeed it is straightforward to see that k NW and k KRR are not characteristic, as the linear kernel l ( y, y ′ ) := ⟨ y, y ′ ⟩ R applied in both k NW and k KRR is not characteristic, and can recover changes only in the first moment between distributions.

A.2 Future Work

Distribution Shift : The closely related fields of Covariate Shift [59], Distribution Shift [60], Transfer Learning [61], and Domain Adaptation [62] have been the focus of significant research in recent years. Notably, the MMD has become widely used across these areas, as evidenced by works such as [15, 63-66], among others. In this context, the AMCMD metric introduced in this work has natural applications, e.g. in covariate shift scenarios, as the choice of the distribution used for the outer expectation, denoted by P X ∗ , can differ from the distributions from which the observed features arise, P X and P X ′ . Furthermore, the AMCMD is especially relevant when one encounters Conditional Shift [67-69], where the conditional distribution of the data changes across domains.

Two-Sample Testing : The MMD was originally introduced as a metric for two-sample testing-that is, for determining whether two datasets are drawn from the same underlying distribution [2]. In that work, the authors propose an MMD-based hypothesis test and analyse its statistical properties. It would be natural to undertake a similar investigation for the AMCMD, and additionally to study how conditional distribution compression affects the resulting test.

̸

̸

Estimator Consistency : The consistency of the AMCMD estimator is established-via Corollary 4.4, which follows from Theorem 4.5 of [19]-in the special case where X = X ′ = X ∗ . This setting aligns with our application of the AMCMD, as it corresponds to the regime in which our compression algorithms operate. However, it would be interesting to extend the consistency result to the most general case, where X = X ′ = X ∗ . Moreover, a promising direction for future work is to characterise the conditions under which convergence rates can be guaranteed. This includes both the well-specified case, where µ Y | X ∈ H Γ [18, 19, 70], and the more general misspecified setting, where µ Y | X is not assumed to lie in H Γ [71, 72].

̸

̸

Differential Privacy : In the work of [73], an optimisation procedure similar to that used in JKIP/ACKIP is used to compress a dataset for training a Kernel Ridge Regression model. Notably, they demonstrate that test performance remains strong even with significant corruption of the compressed set, suggesting potential applications in privacy preservation. In the context of Bayesian Coresets [74], [75] introduced the concept of pseudocoresets , where they apply stochastic gradient descent to optimise a compressed set targeting a Bayesian posterior, and further establish differential privacy guarantees by corrupting gradients. It would be interesting to investigate the impact of corruption on the performance of compressed sets in our setting and derive corresponding differential privacy guarantees.

Global Conditional Distribution Compression : The compressed sets generated by ACKH and ACKIP focus on compressing the conditional distribution in regions where P X has high density, due to the use of the AMCMD objective function. An interesting direction would be to develop methods that provide a more uniform weighting across the entire conditioning space, ensuring balanced compression regardless of feature density variations. This may be particularly valuable for cases where data observed at the tails of the feature distribution are especially important e.g. in health related scenarios.

Alternative Optimisation Strategies : Further exploration of alternative optimisation strategies targeting the AMCMD could also be valuable. Potential approaches include algorithms inspired by Kernel Thinning, second-order methods such as Newton or Quasi-Newton techniques, and metaheuristic strategies like simulated annealing for identifying global optima.

Real-world Applications : Finally, it would be interesting to see how the compressed sets generated by ACKIP perform when used to estimate a KCME applied in the important real-world downstream tasks [17-32], beyond multi-class classification [33] which we have explored in this work.

B Proofs

In this section, we provide technical proofs for the results in the main paper, and rigorously describe and discuss the assumptions that we adopt throughout the paper.

B.1 Technical Assumptions

The assumptions that we work under are laid out in this section, alongside commentary on their restrictiveness.

In order to guarantee the existence of the kernel conditional mean embedding µ Y | X := E P Y | X [ l ( Y, · )] , [18] require that l : Y × Y → R is measurable and E P Y [ √ l ( Y, Y )] < ∞ . However, to ensure that µ Y | X is an element of the space of equivalence classes of measurable functions L 2 ( X , P X ; H l ) , we actually require that E P Y [ l ( Y, Y )] < ∞ [18]. This fact is needed for the proof of Theorem 4.1, hence we make this slightly stronger integrability assumption.

For there to exist a deterministic function F Y | X : X → H l such that µ Y | X = F Y | X ◦ X , we require that H l is separable, which is not a restrictive assumption, and is guaranteed if the kernel l : X × X → R is continuous [18], or if H l is finite dimensional, e.g. when l is the indicator kernel defined on a subset of the natural numbers. Recall that in the main body of this work, and for the remainder of this section, when we refer to µ Y | X , we mean F Y | X .

In various theorems throughout this section we will make assumptions that kernels are bounded , characteristic or universal . Assuming a kernel k defined on X is

bounded ensures that there exists some constant B > 0 such that sup x ∈X k ( x , x ) ≤ B . This assumption is trivially satisfied for many commonly used kernels such as the Gaussian, Laplacian and indicator kernels.

characteristic ensures that the corresponding kernel mean embedding µ X is injective, and hence the corresponding MMD ( P X , P X ′ ) is a proper metric. On R d , the Gaussian, Laplacian, B-spline, inverse multi-quadratics, and the Matérn class of kernels can be shown to be characteristic [40], and on N C := { 0 , 1 , . . . , C } the indicator kernel is characteristic [33]. Note that k ⊗ l : ( X × Y ) × ( X × Y ) → R is characteristic in the case that k and l are continuous, bounded and translation-invariant kernels (e.g. the Gaussian, Matérn and Laplace kernels) (Theorem 4 [40]).

universal ensures it is continuous and that the RKHS H k is dense on the space of continuous functions C ( X ) . That is, for every function f ∈ C ( X ) , and every ϵ > 0 , there exists a function g ∈ H k such that ∥ f -g ∥ ∞ ≤ ϵ [76]. This assumption is satisfied for many common kernel functions, e.g. the Gaussian of the Laplacian [18].

B.2 Equivalence of Optimising the JMMD and Joint Kernel Herding

In this subsection, in order to guarantee the existence of the joint kernel mean embedding µ X,Y ∈ H k ⊗H l , we must assume that k : X × X → R is measurable with E P X [ k ( X,X )] < ∞ , and that l : Y × Y → R is measurable with E P Y [ l ( Y, Y )] < ∞ . We also reiterate the assumption from the main body that ∥ k ( x , · ) , l ( y , · ) ∥ = R for all x ∈ X and y ∈ Y , where R is a constant. This of course implies that

Now, assuming we are at the ( m +1) th iteration of the Joint Kernel Herding algorithm, having already constructed a compressed set of size m , C m := { (˜ x j , ˜ y j ) } j m =1 , the next pair is chosen as the solution

to the optimisation problem

We now show that these updates greedily optimise the JMMD ( P X,Y , ˜ P X,Y ) between the joint kernel mean embedding estimated with the compressed set, and the true joint kernel mean embedding. Adding ( x , y ) to the compressed set C m , we compute the JMMD ( P X,Y , ˜ P X,Y ) as

where (a) follows from expanding the squared norm; (b) follows from linearity of inner products and the definition of the joint kernel mean embedding; and (c) follows from setting C 1 := ⟨ µ X,Y , µ X,Y ⟩ H k ⊗H l , C 2 := -2 m +1 ∑ m j =1 E ( x ′ , y ′ ) ∼ P X,Y [ k (˜ x j , x ′ ) l (˜ y j , y ′ )] , C 3 := ∑ m i,j =1 k (˜ x i , ˜ x j ) l (˜ y i , ˜ y j ) , and C 4 = 1 ( m +1) 2 k ( x , x ) l ( y , y ) , where we have assumed C 4 is constant. Now, as we are optimising with respect to ( x , y ) , we can ignore those invariant terms, and solve the optimisation problem

which is the change in JMMD ( P X,Y , ˜ P X,Y ) from adding the point ( x , y ) to the compressed set. Note that this is exactly equivalent to solving the optimisation problem

which is precisely the update used in Joint Kernel Herding.

Remark B.1. In general, one can not usually evaluate the joint expectation in (2), and hence this is estimated using the samples from D , corresponding to optimising the JMMD ( ˆ P X,Y , ˜ P X,Y ) .

B.3 Derivation of JKH Objective and Gradients

We denote L D m : R d × R p → R to be the estimate of the objective in (2), using the entire dataset D , then

where ˜ K m ( x ) := [ k ( x , ˜ x 1 ) , . . . , k ( x , ˜ x m )] ⊤ , and K n ( x ) := [ k ( x , x 1 ) , . . . , k ( x , x n )] ⊤ , ˜ L m ( y ) := [ l ( y , ˜ y 1 ) , . . . , l ( y , ˜ y m )] , and L n ( y ) := [ l ( y , y 1 ) , . . . , l ( y , y n )] ⊤ .

We solve the optimisation problem in (2) using gradient descent, hence we need to derive gradients of (12) with respect to both x and y . It is straightforward to see that

and

where we have defined

Hence, it is easy to see that the gradient can be computed with O ( m + n ) time and storage complexity.

B.4 Derivation of JKIP Objective and Gradients

Before we state and prove our lemma, we first recall some properties of tensor calculus.

B.4.1 Tensor Calculus

Let F : R m × d → R m × m be a matrix-valued function taking matrix-valued inputs with

where k : R d × R d → R will be some kernel, and X = [ x 1 , . . . , x m ] ⊤ ∈ R m × d , x i ∈ R d for i = 1 , . . . , m . Then, we have

such that ∇ X K ( X , X ) ∈ R m × m × m × d , i.e. a fourth-order tensor. With some abuse of notation, we reproduce the usual derivative identities below, for a more in-depth review see, for example, [77, 78]:

Trace Rule : Given some matrix A ∈ R m × m we have

which establishes the linearity of the gradient operator with respect to the trace. Note that the trace on the RHS is understood here to be a partial trace over the last two dimensions of the fourth-order tensor, i.e.

Inverse Rule : We also have

assuming K ( X ) -1 ∈ R m × m exists, where

Product Rule : Given a second function L : R m × d → R m × m , defined similarly to K with kernel l : R d × R d → R , then we have

where we have

B.4.2 Statement and Proof of Lemma

Letting L D : R m × d × R m × p → R be the estimate of the objective in (3) using the entire dataset D , then we have

where [ K ˜ X ˜ X ] ij := k (˜ x i , ˜ x j ) , [ K ˜ XX ] iq := k (˜ x i , x q ) , [ L ˜ Y ˜ Y ] ij := l (˜ y i , ˜ y j ) , and [ L ˜ Y Y ] iq := l (˜ y i , y q ) , for i, j = 1 , . . . , m and q = 1 , . . . , n .

Note that optimising this objective with respect to the compressed set C = ( ˜ X , ˜ Y ) is equivalent to optimising the JMMD between ˆ P X,Y and ˜ P X,Y , that is, the empirical joint distributions of the full dataset D and the compressed set C respectively:

Lemma B.2. We compute the gradients of the objective function L D : R m × d × R m × p → R as

where

with O ( mn + m 2 ) time and storage complexity, i.e. linear with respect to the size of the full dataset D .

Proof : We have

Applying rules (15) and (18), it is straightforward to see that

and

Now, in order to show that these gradients can be computed with O ( mn + m 2 ) time and storage complexity, the critical observation is that the majority of the elements of these fourth-order tensors will be equal to zero, i.e.

̸

̸

and

Then, by using the identity in (16), we have that,

for q = 1 . . . , m and r = 1 , . . . , d , and where (a) follows trivially from the symmetry of the kernel functions l ( · , · ) and k ( · , · ) . Hence, here, the only terms we have to compute and store are

and L ˜ Y , ˜ Y ∈ R m × m , which can be accomplished with cost O ( m 2 ) in both storage and time, ignoring any dependence on the dimension of the feature space d . A very similar derivation holds for the term

Now, tackling the cross term, we can use (16) to get that

with q = 1 . . . , m and r = 1 , . . . , d . Hence the only terms we must compute and store are

̸

̸

̸

̸

and L Y , ˜ Y ∈ R n × m , which can be accomplished with cost O ( nm ) in both storage and time, ignoring any dependence on the dimension of the feature space d . Again, a very similar derivation holds for the term

Hence, the final computation and storage cost of computing the gradients is O ( nm + m 2 ) , i.e. linear in the size of the target dataset D . ■

Remark B.3. Above we have derived analytical gradients of the objective function, and shown they can be computed in linear time. In practice one computes the gradients using JAX's [79] autodifferentiation capabilities. The authors observed minimal slowdown from using auto-differentiation.

B.5 Proof of Theorem 4.1

Theorem B.4. Suppose that l : Y × Y → R is a characteristic kernel, that P X , P X ′ , and P X ∗ are absolutely continuous with respect to each other, and that P ( · | X ) and P ( · | X ′ ) admit regular versions. Then, AMCMD [ P X ∗ , P Y | X , P Y ′ | X ′ ] = 0 if and only if, for almost all x ∈ X wrt P X ∗ , P Y | X = x ( B ) = P Y ′ | X ′ = x ( B ) for all B ∈ Y .

Moreover, assuming the Radon-Nikodym derivatives d P X ∗ d P X , d P X ∗ d P X ′ , d P X ∗ d P X ′′ are bounded, then the triangle inequality is satisfied, i.e. AMCMD [ P X ∗ , P Y | X , P Y ′′ | X ′′ ] ≤ AMCMD [ P X ∗ , P Y | X , P Y ′ | X ′ ] + AMCMD [ P X ∗ , P Y ′ | X ′ , P Y ′′ | X ′′ ] .

Proof :

It is clear that

is non-negative and symmetric in P Y | X and P Y ′ | X ′ .

We first prove the equivalence result:

( = ⇒ ) Assume that AMCMD [ P X ∗ , P Y | X , P Y ′ | X ′ ] := √ E x ∼ P X ∗ [ ∥ µ Y | X = x -µ Y ′ | X ′ = x ∥ 2 H l ] = 0 . This implies that µ Y | X = x = µ Y ′ | X ′ = x almost everywhere x wrt P X ∗ . Now, by the fact that P X ∗ and P X (or P X ′ ) are absolutely continuous with respect to each other, we also have that µ Y | X = x = µ Y ′ | X ′ = x almost everywhere x wrt P X (or P X ′ ). Hence, we must have MCMD ( P Y | X = · , P Y ′ | X ′ = · ) := ∥ ∥ µ Y | X = · -µ Y ′ | X ′ = · ∥ ∥ H l = 0 almost everywhere x ∈ X wrt P X (or P X ′ ). Thus, by Theorem 2.1, we have that for almost all x ∈ X wrt P X (or P X ′ ) , P Y | X = x ( B ) = P Y ′ | X ′ = x ( B ) for all B ∈ Y . However, again by absolute continuity of measures, this is equivalent to stating that for almost all x ∈ X wrt P X ∗ , P Y | X = x ( B ) = P Y ′ | X ′ = x ( B ) for all B ∈ Y .

( ⇐ = ) Assume that for almost all x ∈ X wrt P X ∗ , P Y | X = x ( B ) = P Y ′ | X ′ = x ( B ) for all B ∈ Y . Then, by the fact that P X ∗ and P X (or P X ′ ) are absolutely continuous with respect to each other, and Theorem 2.1, we have that MCMD ( P Y | X = · , P Y ′ | X ′ = · ) := ∥ ∥ µ Y | X = · -µ Y ′ | X ′ = · ∥ ∥ H l = 0 almost everywhere x wrt P X (or P X ′ ). This implies that µ Y | X = x = µ Y ′ | X ′ = x almost everywhere x ∈ X wrt P X ′ (or P X ′ ), and by absolutely continuity, P X ∗ also. Hence, we must have AMCMD [ P X ∗ , P Y | X , P Y ′ | X ′ ] := √ E x ∼ P X ∗ [ ∥ µ Y | X = x -µ Y ′ | X ′ = x ∥ 2 H l ] = 0 .

Finally, we show the triangle inequality, that is, given additional random variables X ′′ : Ω → X , Y ′′ : Ω →Y , with conditional distribution P Y ′′ | X ′′ and KCME µ Y ′′ | X ′′ , we show (suppressing the first argument P X ∗ )

Firstly, denote L 2 ( X , P X ∗ ; H l ) to be the Banach space of (equivalence classes of) measurable functions f : X → H l such that ∥ f ∥ 2 H l is P X ∗ -integrable with norm defined by

Now, it is shown in [18], that µ Y | X belongs to L 2 ( X , P X ; H l ) , where we stress that the measure is P X , not P X ∗ . They arrive at this conclusion by the measurability of µ Y | X , and by noting that

where (a) follows by the definition of the KCME; (b) follows by the Generalised Conditional Jensen's Inequality (Theorem A.2 [18]); and (c) by the tower property, the reproducing property, and by our integrability assumption.

Therefore, by further assuming that the Radon-Nikodym derivative d P X ∗ d P X is bounded, i.e. there exists some constant M > 0 such that d P X ∗ d P X ( x ) ≤ M for all x ∈ X , we have

where (a) follows directly from the boundedness condition. Thus, it is now clear that µ Y | X ∈ L 2 ( X , P X ∗ ; H l ) . Hence, assuming that d P X ∗ d P X ′ and d P X ∗ d P X ′′ are also bounded, the functions f, g : X → H l defined by

belong to L 2 ( X , P X ∗ ; H l ) . The triangle inequality then follows by a straightforward application of the Minkowski inequality, i.e.

for the special case where p = 2 . ■

Remark B.5. Assuming the Radon-Nikodym derivatives d P X ∗ d P X and d P X ∗ d P X ′ are bounded, by the arguments above it is clear that

that is, the AMCMD can be understood as the norm of the difference of µ Y | X and µ Y ′ | X ′ in L 2 ( X , P ∗ X ; H l ) space.

B.6 Proof of Lemma 4.3

Lemma B.6.

where we have defined W X ′ X ′ := ( K X ′ X ′ + λ m I ) -1 with [ K X ′ X ′ ] ij := k ( x ′ i , x ′ j ) , W XX := ( K X ′ X ′ + λ m I ) -1 with [ K XX ] ij := k ( x i , x j ) , [ K X ′ X ∗ ] ij := k ( x ′ , x ∗ ) , K X ∗ X := K ⊤ X ′ X ∗ , [ L Y Y ] ij := l ( y i , y j ) , [ L Y Y ′ ] ij := l ( y i , y ′ j ) , and [ L Y ′ Y ′ ] ij := l ( y ′ i , y ′ j ) .

Proof : We have defined

Using equation (1), we have

then we can expand the MCMD 2 as

Now,

where (a) follows from the linearity of inner products; (b) follows from the reproducing property on H l ; and (c) follows from symmetry of the kernel k ( · , · ) . Therefore,

where the second line follows from Tr ( AB ) = ∑ n i,j =1 a ij b ji . Here we have defined [ K X ∗ X ] ij := k ( x ∗ i , x j ) , W XX := ( K XX + λI ) -1 , L Y Y ′ := [ l ( y i , y ′ j ] ij , W X ′ X ′ := ( K X ′ X ′ + λI ) -1 , and [ K X ′ X ∗ ] ij := k ( x ′ i , x ∗ j ) .

Noting that the derivation of the first and third term of (23) follow very similarly, we can easily see that

■

B.7 Proof of Corollary 4.4

To state Corollary 4.4 in full context, we first introduce some additional definitions and notation. Let L ( H l ) denote the space of bounded linear operators from H l to itself. An operator-valued kernel Γ : X × X → L ( H l ) induces a vector-valued reproducing kernel Hilbert space (vvRKHS), denoted H Γ , which consists of functions f : X → H l (see [38] for further details). The vvRKHS H Γ is C 0 if H Γ ⊆ C 0 ( X , H l ) , the space of continuous functions from X to H l that vanish at infinity. Moreover, the kernel Γ is called C 0 -universal if it is C 0 and H Γ is dense in L 2 ( X , P X ; H l ) for any probability measure P X on X . Denoting the identity operator on H l by I H l , it is shown in [38] that the kernel Γ( x, x ′ ) = k ( x, x ′ ) I H l is C 0 -universal whenever k is a universal scalar kernel, such as the Gaussian or Laplacian kernel. Hence, in the statement of Corollary B.7, the assumption that k ( · , · ) is universal implies that Γ is C 0 -universal, with H Γ as described above.

Note that the specific choice of kernel Γ( x, x ′ ) = k ( x, x ′ ) I H l leads to the form of the KCME estimator given in equation (1) [19], which we adopt in this work. More generally, the corollary could be stated under the assumption that H Γ is induced by an arbitrary C 0 -universal operator-valued kernel Γ , thereby removing the requirement that k be universal-following the more general formulation in Theorem 4.5 of [19]. However, as stated, this would yield a potentially different form of the KCME estimator than the one used in (1).

Corollary B.7. Assume that k ( · , · ) and l ( · , · ) are bounded, k ( · , · ) is universal, and let the regularisation parameters λ n and λ m decay at slower rates than O ( n -1 / 2 ) and O ( m -1 / 2 ) respectively. Then, ̂ AMCMD [ P X , P Y | X , P Y ′ | X ] p → AMCMD [ P X , P Y | X , P Y ′ | X ] as n, m, q →∞ .

Proof : Given sets of i.i.d. samples { x i } q i =1 ∼ P X , M := { ( x , y ′ i ) } i m =1 ∼ P X,Y ′ , and N := { ( x i , y i ) } n i =1 ∼ P X,Y , the estimate of the AMCMD [ P X , P Y | X , P Y ′ | X ] is defined as

with population counterpart

The assumptions in Corollary B.7 satisfy the assumptions of Theorem 4.5 [19], which states that

Now, it is enough to note that convergence in probability is conserved under continuous mappings, i.e. A n p → A as n →∞ implies √ A n p → √ A as n →∞ by the continuity of the square root function on [0 , ∞ ) for non-negative random variables A,A n . Hence, our result follows. ■

B.8 Proof of Lemma 4.7

Lemma B.8. Let h : X → H l be a vector-valued function, then

Proof : We apply the definition of the KCME, then the tower rule to see that

■

B.9 Derivation of ACKH Objective and Gradients

In ACKH, assuming we are at the m th iteration, having already constructed a compressed set of size m -1 , C m -1 := { (˜ x j , ˜ y j ) } m -1 j =1 , and let C m = C m -1 ∪ ( x , y ) , then we solve the optimisation problem

via gradient descent. Letting G D m : R d × R p → R be the estimate of the objective function in (24) computed using the entire dataset D = { ( x i , y i ) } n i =1 , i.e.

then we have the following lemma:

where we let

and ˜ W m ( x ) := ( ˜ K m ( x ) + λI m ) -1 ∈ R m × m . Moreover, G D m ( x , y ) can be computed with time complexity of O ( m 2 n + m 3 ) and storage complexity of O ( mn + m 2 ) .

Proof : In order to reduce the notational burden, we write [ ˜ W m ( x )] ij = ˜ W ij , and ( x , y ) = (˜ x m , ˜ y m ) , then, using the estimate of the KCME from (1), we have

where (a) follows from inserting the estimate of the KCME and the definition of the norm; (b) follows from the reproducing property and the symmetry of the kernels; and (c) follows from the fact that Tr ( AB ) = ∑ n i,j =1 a ij b ji for symmetric A,B ∈ R n × n .

The storage complexity of O ( mn + m 2 ) comes from the fact we have to store

The computational complexity of O ( m 2 n + m 3 ) arises from solving the linear system,

which dominates the O ( m 2 n ) cost of the singular remaining matrix multiplication, and the O ( m 2 + mn ) cost of taking Hadamard products required to compute the traces. ■

We now derive the gradients of G D m , and show that they are cheap to compute and store. Note that the derivative identities used in this section are similar to those in Section B.4.1, except for third-order tensors, as we deal with derivatives of matrix-valued functions with respect to vectors.

Lemma B.10. We compute the gradients of the objective function G D m : R d × R p → R as

Where, in order to reduce notational burden, we write ( x , y ) = (˜ x m , ˜ y m ) , then we have

with O (( m 2 n + m 3 ) time and O ( mn + m 2 ) storage complexity, i.e. linear with respect to the size n of the full dataset D .

Proof : Firstly, by applying the trace and product rule, we can immediately see that

Now, we have

where (a) follows from the linearity of the gradient operator; (b) follows from a combination of the trace and product rules; and (c) follows from the symmetry of the feature kernel k ( · , · ) . Then, by applying the inverse rule, we have

Now, to establish the cost of computing this estimate, the first thing to notice is that there is a significant amount of symmetry and shared computation between the terms. In particular, avoiding the gradients

for now, we need to compute

then we have

where (a) follows from the cyclic property of the trace and the symmetry of the kernels. Now, the cost of computing A , B and C is O ( nm 2 + m 3 ) , and given these matrices, the cost of computing D := CA ⊤ ∈ R m × m and E := AB ∈ R m × m is O ( nm 2 ) . So, we are now in a position where we have to compute

We can reduce the cost of computing these terms by noticing that the majority of the elements of our third-order gradient tensors will be equal to zero, that is

Hence, we have

for r = 1 , . . . , d . Hence this term, given C , can be computed with cost O ( n ) . We also have

where the last equality follows by the symmetry of the kernels. Hence this term, given D , can be computed with cost O ( m ) . Similar derivations hold for Tr ( ∇ x ¯ K ( x ) ⊤ B ) and Tr ( ∇ x ˜ K ( x ) E ) .

Therefore, we have an overall storage cost of O ( mn + m 2 ) , and time cost of O ( m 3 + m 2 n ) , i.e. linear with respect to the size n of the full dataset D . ■

Remark B.11. Above we have derived analytical gradients of the objective function, and shown they can be computed in linear time. In practice one computes the gradients using JAX's [79] autodifferentiation capabilities. The authors observed minimal slowdown from using auto-differentiation.

B.10 Derivation of ACKIP Objective and Gradients

Noting that C = ( ˜ X , ˜ Y ) , in ACKIP we solve the optimisation problem

via gradient descent. Letting J D : R m × d × R m × p → R be the estimate of the objective function in (27) computed using the entire dataset D = { ( x i , y i ) } n i =1 , i.e.

we have the following lemma:

,

where [ K ˜ X ˜ X ] ij := k (˜ x i , ˜ x j ) , [ K ˜ XX ] iq := k (˜ x i , x q ) , [ L ˜ Y ˜ Y ] ij := l ( ˜ y i , ˜ y j ) , [ L ˜ Y Y ] iq := l ( ˜ y i , y q ) , and [ W ˜ X ˜ X ] ij := ( K ˜ X ˜ X + λI ) -1 , for i, j = 1 , . . . , m and q = 1 , . . . , n .

Moreover, J D ( ˜ X , ˜ Y ) can be computed with time complexity of O ( m 2 n + mn + m 3 ) and storage complexity of O ( mn + m 2 ) .

Proof : Using the estimate of the KCME from (1), and writing [ W ˜ X ˜ X ] ij = ˜ W ij , we have

where (a) follows from inserting the estimate of the KCME and the definition of the norm; (b) follows from the reproducing property and the symmetry of the kernels; and (c) follows from the fact that Tr ( AB ) = ∑ n i,j =1 a ij b ji for symmetric A,B ∈ R n × n .

The storage complexity of O ( mn + m 2 ) comes from the fact we have to store

The overall computational complexity of O ( m 2 n + m 3 ) arises from the cost of solving the linear system,

which dominates the O ( m 2 n ) cost of the singular remaining matrix multiplication, and the O ( m 2 + mn ) cost of taking Hadamard products required to compute the traces. ■

We now derive the gradients of J D , and show that they are cheap to compute and store:

Lemma B.13. We compute the gradients of the objective function J D → R m × d × R m × d → R as

where

and W ˜ X ˜ X := ( K ˜ X ˜ X + λI ) -1 , with O ( m 2 n + m 3 ) time complexity and O ( mn + m 2 ) storage complexity, i.e. linear with respect to the size of the full dataset D .

Proof : We use the matrix-by-matrix derivative identities from Section B.4.1. Firstly, by applying rules (15) and (18) we can immediately see that

Now, we have

where (a) follows from the linearity of the gradient operator; (b) follows from a combination of rules (15) and (18); and (c) follows from the symmetry of the feature kernel k ( · , · ) . Then, by applying rule (17), we have

Now, in order to establish the cost of computing this estimate, the first thing to notice is that there is a significant amount of symmetry and shared computation between the terms. In particular, avoiding the gradients

for now, we need to compute

then we have

where (a) follows from the cyclic property of the trace and the symmetry of the kernels. Now, the cost of computing A , B and C is O ( nm 2 + m 3 ) , and given these matrices, the cost of computing D := CB ∈ R m × m and E := A ⊤ B ∈ R m × m is O ( nm 2 ) . So, we are now in a position where we have to compute

but, following the exact same logic as in section B.4.2, we can compute these traces as sums of products where the majority of the elements of ∇ ˜ X K ˜ X ˜ X and ∇ ˜ X K X ˜ X are zeros, and hence much wasted computation and storage can be avoided. Therefore, similar to equations (21) and (22), we can compute the quantities Tr ( ∇ ˜ X K X ˜ X C ) and Tr ( ∇ ˜ X K ˜ XX B ⊤ ) with O ( mn ) time and storage cost, and Tr ( ∇ ˜ X K ˜ X ˜ X E ) and Tr ( ∇ ˜ X K ˜ X ˜ X D ) with O ( m 2 ) time and storage cost.

Therefore, we have an overall storage cost of O ( mn + m 2 ) , and time cost of O ( m 3 + m 2 n ) , i.e. linear with respect to the size n of the full dataset D . ■

Remark B.14. Above we have derived analytical gradients of the objective function, and shown they can be computed in linear time. In practice one computes the gradients using JAX's [79] autodifferentiation capabilities. The authors observed minimal slowdown from using auto-differentiation.

B.11 Accelerating Objective Computation for Discrete Conditional Distributions

As outlined in C.1.3, for discrete conditional distributions e.g. those encountered in classification data, the response kernel l ( · , · ) is chosen to be the indicator kernel. In this case, gradient descent on the responses is no longer possible. For herding-type algorithms it is straightforward to alternate between taking a step on the feature x , and then, given this new x , exhaustively search over the possible values of the paired response y .

For KIP-style algorithms, given a step on each of the features in the compressed set ˜ X , it would be very expensive to exhaustively search over each possible combination of responses in ˜ Y to find the jointly optimal combination. To reduce this cost, we take a greedy approach and iteratively, response-by-response, exhaustively search over the possible values carrying forward the optimal value of each response to the next iteration. For pseudocode for these procedures see Section D.1.

In this section, by treating each KIP-style objective just as a function of each response ˜ y in ˜ Y , we show that one can accelerate computation of the JKIP and ACKIP objective functions, reducing the cost of the corresponding exhaustive search procedure significantly.

B.11.1 Joint Kernel Inducing Points

Defining ˜ X := [˜ x 1 , ˜ x 2 , . . . , ˜ x m ] ⊤ and ˜ Y := [˜ y 1 , ˜ y 2 , . . . , ˜ y m ] ⊤ , in JKIP, we optimise the following objective

Now, if one treats this objective solely as a function of a single ˜ y t ∈ ˜ Y , fixing ˜ X and the remaining points in ˜ Y , then it easy to see that optimising (30) is equivalent to optimising

where ˜ K m ( x ) := [ k ( x , ˜ x 1 ) , . . . , k ( x , ˜ x m )] ⊤ , and K n ( x ) := [ k ( x , x 1 ) , . . . , k ( x , x n )] ⊤ , ˜ L m ( y ) := [ l ( y , ˜ y 1 ) , . . . , l ( y , ˜ y m )] ⊤ , and L n ( y ) := [ l ( y , y 1 ) , . . . , l ( y , y n )] ⊤ . This reduces the cost of evaluating the objective function by a factor of m , both in storage and time. Note that if using a non-stationary kernel l ( · , · ) , one must avoid double counting the diagonal term in Equation 30 by subtracting 1 m k (˜ x t , ˜ x t ) l (˜ y t , ˜ y t ) .

B.11.2 Average Conditional Kernel Inducing Points

In Section B.10, we saw that the objective of ACKIP can be written as

Now, if one treats this objective solely as a function of a single ˜ y t ∈ ˜ Y , fixing ˜ X and the remaining points in ˜ Y , then it easy to see that optimising the above is equivalent to optimising

where (a) follows from the symmetry of the kernel functions and a simple reordering of the terms, and (b) follows from defining w t to be the t th row of W ˜ X ˜ X . Note that if using a non-stationary kernel l ( · , · ) one must again subtract a correction term 1 n ( w ⊤ t K ˜ XX K X ˜ X w t ) · l (˜ y t , ˜ y t ) to avoid double counting the diagonal.

Now, the important observation to make is that one only needs to compute the terms involving ˜ X once, as we iterate through each ˜ y t ∈ ˜ Y with ˜ X fixed. Hence, if we ignore the one-time cost of computing W ˜ X ˜ X K ˜ XX , then this objective is O ( m + n ) . A very similar derivation holds for ACKH.

C Experiment Details

All the experiments were performed on a single NVIDIA GTX 4070 Ti with 12GB of memory, CUDA 12.2 with driver 535.183.01 and JAX version 0.4.35.

As is ubiquitous in kernel methods, unless stated otherwise, we standardise the features and responses such that each dimension has zero mean and unit standard deviation.

For continuous conditional distributions, the kernel functions k : X × X → R and l : Y × Y → R are chosen to be the Gaussian kernel, defined as

where the lengthscales α k , α l > 0 are set via the median heuristic. That is, given a dataset { z i } p i =1 , the median heuristic is defined to be

with lengthscale α := √ H p / 2 such that r ( z , z ′ ) := exp ( -1 H p ∥ z -z ′ ∥ 2 ) . This heuristic is a very widely used default choice that has shown strong empirical performance [80]. For discrete conditional distributions, we replace the response kernel with the indicator kernel, defined as

.

See Section C.1.3 for additional details on the impact of this change in kernel.

The regularisation parameter λ > 0 is selected using a two-stage cross-validation procedure on a validation set consisting of 10% of the data. In the first stage, a coarse grid of candidate λ values are used to identify a preliminary range for the regularisation parameter. In the second stage, a finer grid is constructed within this range, and searched over. To avoid the O ( n 3 ) cost of training the KCME, we randomly sample 10 subsets of the training data of size 1000 such that the optimal λ value is averaged over these random training sets to determine the final regularisation parameter.

For all experiments, we use the Adam optimiser [81] as a sensible default choice. However, the implementations of ACKIP and ACKH allow for an arbitrary choice of optimiser via the Optax package [82]. We set a default learning rate of 0.01 across all experiments.

To provide reasonable seeds for minimisation across each iteration of JKH and ACKH, we follow the approach of [3] and draw 10 random auxiliary pairs from the training data, choosing the seed to be the auxiliary sample which achieves the smallest value of the relevant loss. In comparison, to initialise JKIP and ACKIP, we draw 10 auxiliary sets of size m , and choose the initial seed set to be the auxiliary set which achieves the smallest value of the relevant loss.

In order to compare the performance of the above approaches, we note that the primary goal of the kernel conditional mean embedding is to approximate the conditional expectation E [ h ( Y ) | X = x ] for arbitrary functions h ∈ H l and conditioning variables x ∈ X . Hence, to assess this approximation, we report the root mean square error (RMSE), where the mean is taken with respect to the distribution of the conditioning variable, P X .

For continuous conditional distributions, we report the RMSE for the first, second and third moments, as well as the functions h ( y ) = sin( y ) , h ( y ) = cos( y ) , h ( y ) = exp( -y 2 ) , h ( y ) = | y | , and h ( y ) = ✶ y> 0 . These choices of test functions extend those chosen to evaluate Kernel Herding [3].

C.1 Additional Figures and Experiments

In this section we include additional figures for the experiments in the main body, and further experiments on discrete conditional distributions.

C.1.1 Matching the True Conditional Distribution

In this section we include some additional figures for the true conditional distribution compression task outlined in Section 5.

Figure 9 displays an example of a compressed set of size m = 500 constructed by each method. We note that JKH and JKIP have clearly constructed a representation of the joint distribution; with the JKIP construction seemingly more structured. It is interesting to note the extreme disparity between the compressed sets constructed by ACKH and ACKIP, versus the relative similarity of JKH and JKIP. We know that ACKIP achieves superior performance, hence it may be the case that the greedy heuristic is particularly poorly suited to targeting the AMCMD.

8

8

6

2o:

9

10-1

[x1

10-2

2

AMCMD

10-3

RandomSampling

ACKH

JKH

ACKIP

10-4

JKIP

100

101

102

Compressed Set Size

h（y)=y²

h(y)=y²

h(y)=sin(y)

10-1

RMSE

10-1

10-1

Random

JKH

ACKH

10-1

10-2

JKIP

ACKIP

h（y)=cos(y)

h(y)=e-y²

h（y)=1y1

h（y)=1(y<0}

RMSE

10-1

10-1

10-1

10-1

10-2

10-2

10

100

101

102

100

101

100

101

102

100

101

102

102

Random

h(y)=y²

h(y)=y3

10-

h(y)=sin(y)

JKH

ACKH

10-1

JKIP

ACKIP

RMSE

10-1

101

10-2

h(y)=cos(y)

h(y)=e-y²

h(y) = lyl

10-1

h（y)=1(y<0}

101

RMSE

10-2

102

102

C.1.2 Matching the Empirical Conditional Distribution - Continuous / Regression

Real : In this section we include some additional figures for the Superconductivity data outlined in Section 5. Figure 13 shows the AMCMD 2 [ ˆ P X , ˆ P Y | X , ˜ P Y | X ] achieved by each distribution compression method as the compressed set size increases. ACKIP reaches the lowest AMCMD, followed by ACKH, JKIP, then JKH. Figures 14 and 15 are enlarged versions of 4 and 5 respectively, showing results on a larger number of test functions. We see that ACKIP achieves the lowest RMSE across each of the test functions, with ACKH in second for all but one. We also note that JKIP tends to achieve favourable performance versus JKH.

AMCMD2[px, PY1x,

10-1

ACKH

10-2

JKH

ACKIP

JKIP

100

101

102

Compressed Set Size

×10-1

100

h(y)=y²

h(y)=y3

h(y)=sin(y)

2 × 100

100

6 ×10-1

RMSE

4 × 10-1

5 × 101

100

F

3 ×10-1

Random

JKH

2

JKIP

ACKH

8 × 10-1

6 × 10-1

2 ×10-1

ACKIP

h(y)=cos(y)

h(y)=e-y²

100

h（y)=1y1

h（y)=1(y<0}

6 × 10-1

4 ×10-1

6 ×10-1

3 × 101

4 × 101

RMSE

4 × 10-1

3 × 10-1

2 ×10-1

3 ×10-1

10-1

2 × 10-1

2 × 10-1

10-1

101

100

101

102

100

101

102

100

101

102

100

101

102

Random

h(y)=y²

h(y)=y3

h(y) = sin(y)

JKH

JKIP

ACKH

6 × 10-1

3 × 10-1

4 × 10

ACKIP

100

RMSE

3 × 10-1

4 × 101

2 ×10-1

3 × 101

6 × 101

2 ×10-1

h(y) = cos(y)

h(y) =e-y²

年

h(y) =lyl

2.75 × 101

h(y)= 1(y<0}

2 × 10-1

2.5 × 101

3 × 10-1

2.25 × 101

2 ×10-1

RMSE

10-1

1.75 × 101

2 × 101

1.5 × 101

101

6 × 10-2

1.25 × 101

10-1

Synthetic : In this section we include some additional figures for the Heteroscedastic data outlined in Section 5. Figures 16 and 17 are enlarged versions of 6 and 7 respectively, showing results on a larger number of test functions. They show that ACKIP consistently outperforms the other methods across a range of test functions, achieving the lowest RMSE as the size of the compressed set increases. In particular, Figure 7 demonstrates that with m = 250 pairs in the compressed set, ACKIP attains the lowest median RMSE on seven out of eight test functions. For the remaining function, all methods exhibit similar median performance. This highlights the advantage of directly compressing the conditional distribution with ACKIP, rather than targeting the joint distribution with JKH or JKIP. Finally, we note that JKIP consistently outperforms JKH across all test functions.

100

Full Data

h(y)=y²

h（y)=y3

h(y)=sin(y)

Random

JKH

4 × 10°

6 ×10-1

ACKH

JKIP

4 × 10-1

ACKIP

RMSE

4 × 101

100

3 × 100

8 × 101

3 ×10-1

2 ×10-1

× 10-1

2 × 100

6 ×10-1

h(y)=cos(y)

× 10*

h(y)=e-y²

100

h（y)=1yl

h（y）=1(y<0}

×10-1

4 ×10-1

4 × 10-1

3 × 10-1

6 ×10-1

4 × 10-1

RMSE

3 ×10-1

4 × 10-1

3 × 10-1

2 ×10-1

3 ×10-1

2 ×10-1

× 101

2 × 10-1

10-1

100

101

102

100

102

102

100

101

102

10-1

100

10

102

h(y[=y

Full Data

h(y)=y²

h(y)=y3

h(y) = sin(y)

3 × 10-1

Random

JKH

100

2 × 10-1

ACKH

JKIP

ACKIP

9 × 10-1

4 × 100

0 00 0

0 0 080 6

1.8 × 10-1

008

RMSE

2 × 10-1

8 × 10-1

3 × 100

1.6 × 101

7 ×10-1

6 × 101

1.4 × 101

2 × 100

5 ×10-1

1.2 × 101

2.2 × 101

h(y) = cos(y)

1.6 × 101

h(y)=e-y²

3 × 10-1

：

h(y)= lyl

h(y)=1(y<0}

2 × 101

2.2 × 10-1

1.4 × 10-1

2 × 10-1

1.8 × 101

：

1.8 × 101

：

1.6 × 10-1

1.2 × 101

1.6 × 101

1

2 × 10-1

1.4 × 101

1.4 × 101

101

1.2 × 101

1.2 × 101

101

Figure 18 displays an example of a compressed set of size m = 250 constructed by each method. We note that JKH and JKIP have clearly constructed a representation of the joint distribution, whereas ACKH and ACKIP have constructed something that is more difficult to straightforwardly interpret.

A

(

AMCMD2[px, P1x,

10-1

10-2

ACKH

JKH

ACKIP

JKIP

100

101

102

Compressed Set Size

Ablation Study : Using the same setup for the Heteroscedastic data, we repeat the experiment with the Gaussian kernels replaced by inverse multi-quadratic kernels. We report the results in Figures 20 and 21 where we can see that ACKIP still achieves the best results across a variety of test functions.

InverseMulti QuadraticKernel

h(y)=y²

h(y)=y3

h(y)=sin(y)

100

6 × 101

4 ×10°

4 × 10-1

RMSE

100

3 × 10°

3 ×10-1

10-

Full Data

Random

JKH

5 × 101

2 × 100

2 × 10-1

ACKH

JKIP

ACKIP

h(y)=cos(y)

4 × 10-1

h(y)=e-y²

h（y)=1yl

h(y)=1(y<0}

4 ×10-1

× 10-1

4×10-1

6 × 101

3 ×10-1

×10-1

3 ×10-1

RMSE

4 ×10-1

2 × 10-1

3 × 10-1

3 × 10-1

2 ×10-1

2 ×10-1

× 10°

101.

10-1

100

101

102

100

101

102

100

101

102

100

101

102

Compressed Set Size

Compressed Set Size

Compressed SetSize

Compressed SetSize

Inverse MultiQuadraticKernel

Random

Full Data

h(y)=y²

4 × 100

·

h(y)=y3

4 × 10-1

·

h(y)=sin(y)

JKH

ACKH

100

JKIP

ACKIP

3 × 100

3 ×10-1

RMSE

101

6 × 10-1

2 × 100

2 × 101

4 × 101

2.6 × 101

2.4 × 101

2.2 × 10-1

h(y)= cos(y)

2 × 10-1

h(y) =e-y²

4 × 10-1

1

：

h(y)=|yl

4 × 10-1

h(y)=1(y<0}

2 × 101

3 × 10-1

3 × 10-1

1.8 x 101

RMSE

1.6 × 10

2 × 101

2 × 101

1.4 × 10*

101

1.2 × 101

101

10-1

10-1

Wall-Clock Time : In Section D.2 we derive the computational and storage complexity for each of the methods introduced in this work. In Figure 22 and Table 1 we report the wall-clock time for the Heteroscedastic data experiment. Despite JKIP and JKH having the same time complexity, we see that JKIP is significantly faster. As noted in Section D.2.5, the algorithms in this paper were implemented using JAX. JAX enables Just-In-Time (JIT) compilation, which significantly increases execution speed. However, to achieve this speed, JAX relies on an immutable array structure, meaning the arrays must not change shape during program execution. As a result, JKH and ACKH cannot fully leverage the speed benefits of JIT compilation, as the size of the compressed set, and hence the corresponding arrays, increases at every iteration. Conversely, the arrays considered in JKIP and

ACKIP stay the same shape throughout. This presents a notable practical advantage of JKIP and ACKIP over JKH and ACKH.

ACKH

JKIP

ACKIP

102

JKH

Wall Clock Time (s)

100

10-1

50

100

150

200

250

Compressed Set Size

C.1.3 Matching the Empirical Conditional Distribution - Discrete / Classification

For classification tasks with C possible classes, we replace the Gaussian kernel on the responses with the indicator kernel l : N C × N C → [0 , 1] defined by

where N C := [0 , 1 , . . . , C ] . In this case, standard gradient descent on the responses is no longer possible. Moreover, solving the optimisation problems of JKH, JKIP, ACKH and ACKIP now constitute a mixed-integer programming problem, which is known to be NP-complete. Various heuristic approaches exist for problems of this type, such as relaxation-based methods (e.g., continuous relaxations followed by rounding), greedy algorithms, and metaheuristic strategies like genetic algorithms or simulated annealing. We develop a simple two-step optimisation procedure based on exhaustive search, leaving investigating the above techniques in the context of our algorithms for future work.

For JKH and ACKH, we alternate between performing a gradient step on the feature and selecting the optimal response class via exhaustive search. For JKIP and ACKIP, after each gradient step on ˜ X , we iterate over the responses ˜ y in ˜ Y , updating each in turn with the optimal class by exhaustive search, carrying forward these selections. It is important to note that one can reduce the cost of evaluating the JKIP and ACKIP objective functions significantly when ˜ X is fixed; see Section B.11 for a derivation of the relevant objectives and Algorithms 7 and 8 for the pseudocode.

For classification tasks with C classes, we report the overall classification accuracy and F1 scores, as well as the RMSE for the indicator functions h i ( y ) = 1 { y = i } , where i = 1 , . . . , C . As shown in [33], the KCME naturally functions as a multiclass classification model, since

which expresses the class probabilities given X . Moreover, the empirical decision probabilities are guaranteed to converge to the true population probabilities (Theorem 1, [33]), unlike, for example in multi-class SVCs and GPCs. However, in the finite case, the predicted probabilities are not guaranteed to lie in the range [0 , 1] , nor form a normalised distribution. In order to produce a valid distribution, we clip-normalise the estimates (Equation 6, [33]).

Synthetic : We generate an unbalanced 4-class dataset using the multinomial logistic regression model, where conditional class probabilities are given by P ( Y = 0 | X = x ) = 1 1+ ∑ 3 exp( β j · x )

j

=1

model with 100 components. We assign classes using β := [ 10 8 1 45 40 45 40 10 ] ⊤ and, to ensure class overlap, introduce additive noise ϵ ∼ N (0 , 100) to the exponential. This setup results in class proportions of approximately 32% (class 0), 12% (class 1), 19% (class 2), and 37% (class 3).

Figures 23 and 24 show that ACKIP achieves clearly superior performance versus ACKH, JKH and JKIP, both in predicting the class probabilities, as well as in overall accuracy and F1 score, achieving parity with the full data at only 3% of the size.

h（y)=1(y=0}

h（y)=1(y=1}

0.9

Classification Accuracy

3 ×10-1

0.8

6 ×10-1

0.7

RMSE

4×10-1

2 ×10-1

0.6

0.5

3 ×10-1

Full Data

0.4

Random

JKH

0.3

2 ×10-1

ACKH

JKIP

ACKIP

10-1

0.2

h（y）=1(y=2}

h（y)=1(y=3}

0.9

4 × 10-1

0.8

6 ×10-1

3 ×10-1

0.7

0.6

RMSE

4 × 10-1

0.5

2×10-1

3 ×10-1

0.4

0.3

2 × 10-1

0.2

0.1

100

101

102

100

101

102

100

101

102

2.6 × 10-1.

h(y)=1(y=0}

Full Data

2 × 10

h(y)=1{y=1}

Random

0.89

十

2.4 × 10-1

ACKH

Classification Accuracy

ACKIP

JKIP

9060

0.88

22 ×10-1

0.87

RMSE

2 ×10-1

0.86

1.8 × 10-1

10-1

0.85

1.6 × 10-1

0.84

1.4 × 10-1

h(y)=1(y=2}

2.2 × 101

h(y)=1(y=3}

0.90

2 × 101

0.89

2 ×10-1

0.88

1.8 × 101

Score

0.87

RMSE

0.86

1.6 × 101

10.85

0.84

10-1

1.4 × 10

0.83

0.82

In Figure 25 we see that ACKIP achieves by far the best AMCMD, noting that the final value achieved is effectively zero. Due to floating point errors in the computation of AMCMD 2 [ ˆ P X , ˆ P Y | X , ˜ P Y | X ] , it can become slightly negative, resulting in the line leaving the log-log plot. We also see how the herding optimisation approach hinders ACKH as it initially matches ACKIP, but ends up performing worse than random. Finally, we note that JKIP outperforms JKH, which only matches random.

101

10°

10-1

10-2

10-3

10-4

ACKH

JKH

ACKIP

10-5

JKIP

100

101

102

Compressed Set Size

For completeness, in Figures 26 and 27 we also include an example of the compressed set constructed by each method, as well as the corresponding decision boundary.

Class 0

Class 1

Class 2

Class 3

00

00

8

8

8°

00

B

8

?

8

。

8

8

8

8

C

8

Full Data

JKH

ACKH

8

6

O

8

8

X

6

O

8

8

8

0

JKIP

ACKIP

Random

9

C

o

00

8

8

8

g

O

O

8

8

O

8p

go

6

O

O

O

O

+

8

O

8

O

60

60

O

10°

AMCMD2[px, P1x, Py1x]

10-1

ACKH

JKH

ACKIP

JKIP

100

101

102

Compressed Set Size

h（y)=1(y=0}

100

h（y）=1(y=2}

h（y)=1(y=3}

2 ×10-1

2 × 10-1

RMSE

10-1

10-1

10-1

10-1

6×10-2

6 ×10-2

4×10-2

h（y)=1(y=4}

2 ×10-1

h（x)=1(y=5}

h（y）=1(y=6}

h（y）=1(y=7}

2 ×10-1

2 ×10-1

RMSE

10-1

10-1

10-1

10-1

6 ×10-2

6 ×10-2

6 ×10-2

4 ×10-2

4×10-2

4 ×10-2

h（y)=1(y=8}

h（y)=1（y=9}

3 × 10-1

0.8

0.8

2 ×10-1

RMSE

0.6

0.6

10-1

0.4

0.4

Random

Full Data

10-1

HX

0.2

ACKH

6×10-2

0.2

JKIP

0.0

ACKIP

100

101

102

100

101

102

100

101

102

100

101

102

h(y) =1(y=0}

h(y)={y=1}

h(y)=1(y=2}

h(y)=1(y=3}

10-1

1

宁

宁

10-1

RMSE

6×10-2

10-1

10-1

主

6 × 10~2

4 × 10-2

6x10-2

3 × 10~2

4 × 102

6 × 10-2

4 × 10-2

h(y)=1(y=4}

h(y)=1(y=5}

h(y) =1(y=6}

h(y)=1(y=7}

RMSE

10-1

10-1

宁

10-1

10-1

宁

6 × 10-2

6 × 10-2

6 × 102

6x10-2

工

4 × 10-2

4 ×10-2

4 × 10-2

3 × 102

：

4×10-2

h(y)=1(y=8}

h(y)=1(y=9}

0.950

0.95

0.925

H

0.900

0.90

RMSE

10-1

+

10-1

壬

0.850

0.85

·干

0.825

0.80

Full Data

6 ×10-2

0.800

Random

JKH

6×10-2

0.775

0.75

JKIP

ACKH

0.750

ACKIP

C.2 Flexibility of AMCMD versus KCD/AMMD

In this section we demonstrate how the increased flexibility of the AMCMD allows for application to tasks that AMMD/KCD are not suitable for.

̸

Let P X := N ( µ, σ 2 ) , P X ′ := N ( -µ, σ 2 ) , and P X ∗ := N (0 , σ 2 ∗ ) with µ, σ 2 , σ 2 ∗ chosen such that the Radon-Nikodym derivatives d P X ∗ d P X , d P X ∗ d P X ′ are bounded. These three distributions are also clearly absolutely continuous with respect to each other, hence the conditions on the distributions in Theorem 4.1 are satisfied. Importantly, we have P X = P X ′ = P X ∗ , and thus the AMMD/KCD is not defined for this setup. Now, let f a : R → R be a function with

̸

̸

for a ∈ R , and let P Y | X = x := N ( x , σ 2 ϵ ) , and P Y ′ | X ′ = x := N ( f ( x ) , σ 2 ϵ ) . Then, we have that P Y | X = x = P Y ′ | X ′ = x for all x ∈ [ -a, a ] and P Y | X = x = P Y ′ | X ′ = x for all x / ∈ [ -a, a ] . The AMCMDallows us to detect this change in behaviour over regions by changing the location of the weighting distribution P X ∗ ; see Figure 31. In fact, using Lemma 4.3, we estimate the AMCMD in Figure 31 to be approximately equal to 1e-2.

4.0

(x, y) ~ P(x, Y)

(x',y') ~ p(x', Y")

Response Space

3.0

Y=X

2.5

Y' =f(x")

2.0

P

1.5

1.0

0.5

0.0

2.0

1.5

1.0

0.5

Feature Space

0.0

0.5

1.0

1.5

2.0

4.0

3.5

3.0

2.5

RND

1.0

0.5

00

2.0

1.5

1.0

0.5

Feature Space

00

0.5

1.0

1.5

2.0

2.0

1.5

1.0

0.5

Feature Space

0.0

0.5

1.0

1.5

2.0

In contrast, the relative inflexibility of the KCD/AMMD would mean one would not be able to detect over which regions of the conditioning space the conditional distributions are equal; see Figure 32 for an illustration of this. Using Lemma 4.3, we estimate the AMCMD to be approximately 0.5.

(x,y)~P(X,Y)

0.8

Response Space

(x',y)~P(X',Y)

0.7

Y=X

0.6

Y'=f(X)

0.5

PDF

0.4

0.3

0.2

0.1

0.0

2.0

-1.5

1.0

0.5

FeatureSpace

0.0

0.5

1.0

1.5

2.0

C.3 Targeting a Family of Conditional Distributions Exactly

In order to construct a compressed representation that exactly targets a family of conditional distributions P Y | X , we require access to analytical expressions for the expectations

for arbitrary x ′ , x ′′ ∈ X and y ′ ∈ Y . Furthermore, in order to compute the exact AMCMD between the true family of conditional distributions and the family of conditional distributions generated by the compressed set, we must also be able to evaluate

In general we cannot to exactly evaluate the expectations in (33) and (34), however it is possible by restricting our attention to specific choices of the kernel functions k : X×X → R and l : Y×Y → R ,

and a specific data generation process. In particular, we set

i.e. Gaussian kernels, with α k , α l ∈ R > 0 . Moreover, we let P X = N ( µ, σ 2 ) , and given coefficients a 0 , a 1 ∈ R we let y = a 0 + a 1 x + ϵ where ϵ ∼ N (0 , σ 2 ϵ ) , i.e. P Y | X = x = N ( a 0 + a 1 x , σ 2 ϵ ) .

C.3.1 Deriving the Marginal Expectation

In this section, we derive the marginal expectation in (33), under the conditions on the kernel and data-generating process previously laid out:

Now,

where A := ( 1 σ 2 + 2 α 2 k ) , B := ( µ σ 2 + ( x ′ + x ′′ ) α 2 k ) . Completing the square, we have

and therefore we can write that,

C.3.2 Deriving the Joint Expectation

In this section, we derive the joint expectation in (33), under the conditions on the kernels and data-generating process previously laid out. We first derive the joint distribution.

We have that

where using

we notice that ( X,Y ) ∼ N (( µ a 0 + a 1 µ ) , ( σ 2 a 1 σ 2 a 1 σ 2 a 2 1 σ 2 + σ 2 ϵ )) .

Now, we want to derive the expectation E ( x , y ) ∼ P X,Y [ k ( x , x ′ ) l ( y , y ′ )] for arbitrary x ′ ∈ X and y ′ ∈ Y :

where k ( · , · ) and l ( · , · ) are both Gaussian kernels. We need this integral to end up in the form

for ω := ( x , y ) ⊤ as by completing the square, it can be shown [85] that

Let us first interrogate the product k ( x , x ′ ) l ( y , y ′ ) :

where A 1 := ( 1 α 2 k 0 0 1 α 2 l ) , b 1 := ( x ′ /α 2 k y ′ /α 2 l ) ⊤ and c 1 := -x ′ 2 2 α 2 k -y ′ 2 2 α 2 l . Now, we need to write f X,Y ( x , y ) in the same form:

Now,

hence,

Therefore, we have that

where A := A 1 + A 2 , b := b 1 + b 2 and c = c 1 + c 2 .

C.3.3 Computing the AMCMD Exactly

In order to compute the AMCMD exactly, we require an analytical expression for (34). First note that we have

where the second and third equalities follow straightforwardly from the definition of the KCME. Now, the first step is to derive an analytical expression for

Writing, f ( x ) = a 0 + a 1 x , we have

Now,

where A := ( 1 α 2 l + 1 σ 2 ϵ ) and B := ( y α 2 l + f ( x ) σ 2 ϵ ) . Completing the square, we get

and therefore we can write that,

We can further simplify by removing the unwieldy constants A and B , writing that

The next step is therefore to compute,

Therefore,

where we see that the integrand with respect to the expectation over P X is constant. Note that the above computations hold for arbitrary f : X → R , however we require that the error has constant variance σ 2 ϵ . If the error is not constant and it evolves as a function of x , then the final expectation with respect to P X may become very difficult to compute exactly.

D Algorithm Details

In this section we include additional details about the algorithms developed in this work including pseudocode and complexity analysis.

D.1 Pseudocode

In this section we include pseudocode for the algorithms introduced in this work, including gradientfree variants of the Kernel Herding type algorithms suitable for X ̸ = R d and Y ̸ = R p . In all gradient-based algorithms, the pseudocode assumes standard gradient descent. In practice, however, any gradient descent variant may be used. In our implementation, we employed the Optax [82] package, which provides access to a wide range of gradient-based optimisation methods, including ADAM [81], which we used in our experiments.

Algorithm 1 Joint Kernel Herding

Algorithm 2 Average Conditional Kernel Herding

Algorithm 3 Joint Kernel Inducing Points

Input : Dataset D = { ( x i , y i ) } n i =1 ⊂ X ×Y , Coreset size M ∈ N , Feature kernel k : X ×X → R , Response kernel l : Y ×Y → R , Candidate batch size C ∈ N , Maximum iteration number T , Step size α

Uniformly at random, select C sets of candidate sets { ( ˜ X i , ˜ Y i ) } C i =1 from D , | ˜ X i | = | ˜ Y i | = M , i = 1 , . . . C

Estimate

S

i

←L

D

( ˜

X

end for

end if end for

Algorithm 4 Average Conditional Kernel Inducing Points

Input : Dataset D = { ( x i , y i ) } n i =1 ⊂ X ×Y , Coreset size M ∈ N , Feature kernel k : X ×X → R , Response kernel l : Y × Y → R , Regularisation parameter λ ∈ R > 0 m Candidate batch size C ∈ N , Maximum iteration number T , Step size α

Uniformly at random, select C sets of candidate sets { ( ˜ X i , ˜ Y i ) } C i =1 from D , | ˜ X i | = | ˜ Y i | = M , i = 1 , . . . C

Estimate S i ←J D ( ˜ X i , ˜ Y i ) using equation (11)

end for

end if end for

˜

i

,

Y

i

)

using equation (4)

Algorithm 5 Gradient-Free Joint Kernel Herding

Algorithm 6 Gradient-Free Average Conditional Kernel Herding

Algorithm 7 Joint Kernel Inducing Points with Exhaustive Search

Input : Dataset D = { ( x i , y i ) } n i =1 ⊂ X ×Y , Coreset size M ∈ N , Feature kernel k : X ×X → R , Indicator response kernel l : Y×Y → R , Candidate batch size C ∈ N , Maximum iteration number T , Step size α , Set of possible classes A := { 0 , 1 , . . . , a }

Uniformly at random, select C sets of candidate sets { ( ˜ X i , ˜ Y i ) } C i =1 from D , | ˜ X i | = | ˜ Y i | = M , i = 1 , . . . C

Estimate S i ←L D ( ˜ X i , ˜ Y i ) using equation (4)

end for

∗

Compute

∇

˜

L

D

( ˜

Using equation (31), compute F D (˜ y i ) for each possible value of ˜ y i ∈ { 0 , 1 , . . . , a }

Update

˜

Y

j

with the optimal choice end for

if converged then break

end if end for

Algorithm 8 Average Conditional Kernel Inducing Points with Exhaustive Search

Input : Dataset D = { ( x i , y i ) } n i =1 ⊂ X ×Y , Coreset size M ∈ N , Feature kernel k : X ×X → R , Indicator response kernel l : Y × Y → R , Regularisation parameter λ ∈ R > 0 Candidate batch size C ∈ N , Maximum iteration number T , Step size α , Set of possible classes A := { 0 , 1 , . . . , a }

Uniformly at random, select C sets of candidate sets { ( ˜ X i , ˜ Y i ) } C i =1 from D , | ˜ X i | = | ˜ Y i | = M , i = 1 , . . . C

Estimate S i ←J D ( ˜ X i , ˜ Y i ) using equation (11)

end for

Using equation (32), compute R D (˜ y ) for each possible value of ˜ y ∈ { 0 , 1 , . . . , a }

i i Update ˜ Y j with the optimal choice

end for

if converged then break

end if end for

X

j

,

˜

Y

j

)

using equation (19)

D.2 Complexity Analysis

In this section we derive the overall storage and time complexity of constructing a compressed set of size m , where we use n ≫ m datapoints to estimate the objective functions of JKH, JKIP, ACKH and ACKIP respectively. The results are summarised in Table 2.

D.2.1 Joint Kernel Herding

From Section B.3, we know that each gradient computation of JKH has O ( m + n ) storage and time complexity.

Storage cost : The overall storage cost is just the cost of storing the gradients of the last iteration, i.e. O ( m + n ) , that is, linear in the size of the target dataset n .

Time cost : Assuming we take T gradient steps per optimisation of each pair in the compressed set, then the cost of the m th iteration of JKH is O (( m + n ) T ) . Therefore, the final cost is

i.e. linear in the size of the target dataset n .

D.2.2 Joint Kernel Inducing Points

From Section B.4, we know that each gradient computation of JKIP has O ( m 2 + mn ) storage and time complexity.

Storage cost : The overall storage cost is O ( m 2 + mn ) i.e. linear in the size of the target dataset n .

Time cost : Assuming we take J gradient steps, then the final cost of JKIP is simply O (( m 2 + mn ) J ) i.e. linear in the size of the target dataset n .

D.2.3 Average Conditional Kernel Herding

From Section B.9, we know that each gradient computation of ACKH has O ( m 2 + mn ) storage and O ( m 3 + m 2 n ) time complexity.

Storage cost : The overall storage cost is O ( m 2 + mn ) i.e. linear in the size of the target dataset n .

Time cost : Assuming we take T gradient steps per optimisation of each pair in the compressed set, then the cost of the m th iteration of ACKH is O (( m 3 + m 2 n ) T ) . Therefore, the final cost is

that is, linear in the size of the target dataset n , but suffering from quartic cost in m .

D.2.4 Average Conditional Kernel Inducing Points

From Section B.10, we know that each gradient computation of ACKIP has O ( m 2 + mn ) storage and O ( m 3 + m 2 N ) time complexity.

Storage cost : The overall storage cost is O ( m 2 + mn ) i.e. linear in the size of the target dataset n .

Time cost : Assuming we take J gradient steps, then the final cost of ACKIP is simply O (( m 3 + m 2 n ) J ) i.e. linear in the size of the target dataset n , but suffering from only cubic cost in m versus quartic for ACKH.

D.2.5 Discussion

Experimentation suggests that the number of gradient steps required by JKIP and ACKIP to achieve convergence is of the same order as JKH and ACKH, i.e., J ≈ T . Consequently, JKIP and JKH have the same time complexity, but JKIP incurs a slightly higher storage cost due to an additional factor of m , which arises from the joint optimisation of pairs in the compressed set.

For ACKIP and ACKH, their storage costs are identical, as the nature of the ACKH objective prevents it from being expressed solely in terms of the newest pair in the compressed set (unlike in JKH).This same property causes ACKH to have an additional factor of m in its time complexity compared to ACKIP. This difference becomes significant in complex problems which call for large m , or more generally when n is very large.

Throughout, we have omitted the contribution of the feature dimension d and response dimension p from the stated time complexities, as it is implicitly assumed that in the distribution compression context, n ≫ d, p . For commonly used kernels, evaluation is typically linear in d or p , so at most, one should expect an additional multiplicative factor of d + p .

The algorithms in this paper were implemented using the free, open-source Python library JAX [79]. JAX enables Just-In-Time (JIT) compilation, which significantly increases execution speed. However, to achieve this speed, JAX relies on an immutable array structure, meaning the arrays must not change shape during program execution. As a result, JKH and ACKH cannot fully leverage the speed benefits of JIT compilation in their current form, as the size of arrays increases at every iteration. Conversely, the arrays considered in JKIP and ACKIP stay the same shape throughout. This presents a notable practical advantage of JKIP and ACKIP over JKH and ACKH in the JAX implementation.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper extends the distribution‐compression problem to the conditional‐distribution setting and proposes a family of kernel metrics and algorithms for selecting the best compression set in an RKHS. Specifically, the authors introduce AMCMD (Average Maximum Conditional Mean Discrepancy), prove it is a valid metric, and show it can be estimated in O(n) time—rather than O(n^3)—by leveraging the tower property.\nBuilding on AMCMD, they present ACKH, a compression algorithm for conditional distributions, and demonstrate experimentally that ACKH outperforms the joint‐distribution approach (JKH). They also propose an inducing‐point variant, ACKIP, which achieves superior performance on both synthetic and real‐world datasets. Overall, the paper is well motivated and clearly written.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The method is theoretically sound and intuitive.\n2. The benefit of modeling $P(Y|X)$ directly—rather than via the joint distribution—is clearly demonstrated through experiments.\n3. Empirical results are consistent, and the improvements are substantial.\n4. Computational complexity is carefully addressed, and the algorithms appear practical for real‐world use.\n\nWeaknesses:\n1. The discussion focuses narrowly on kernel herding; positioning this work relative to other distribution‐compression methods would strengthen the paper.\n2. The circumstances under which one should choose ACKH over JKH are shown empirically but lack a more intuitive, conceptual explanation.\n\nQUESTIONS:\n1. What types of real-world data motivate this approach? In particular, under what practical conditions does ACKH outperform JKH? Clarifying this would better justify focusing on conditional distributions.\n2. Could you clarify in the main text how ACKIP's complexity improves over ACKH --- specifically, why the inducing-point variant is $m$ times faster? \n3. In Eq. (6), you introduced random-variable notations: X*, Y|X, Y'|X'. I understand this is for generality, but could you explain what these distributions represent? For example, is X* the empirical marginal of P(X,Y)?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of compressing labeled data while preserving the properties of the original distribution. The authors extend concepts in Kernel Herding, which is used for compressing unlabeled data. They also consider joint distribution where they first extend Kernel Herding to Joint Kernel Herding (JKH) and also develop a new algorithm called Joint Kernel Inducing Points (JKIP). They introduce a new metric called Average Maximum Conditional Mean Discrepancy (AMCMD) and derive a closed-form formula for it. They extend the Kernel Herding algorithm to Average Conditional Kernel Herding (ACKH). They also develop a new algorithm called Average Conditional Inducing Points (ACKIP) which, instead of sequentially constructing the compression set, works with sets of points and iteratively updates the set via gradient descent. The point-based algorithm move away from greedy optimization that sequentially constructs a set and they turn out to be more efficient. The proposed algorithms are evaluated through comprehensive set of experiments.\n\nSTRENGTHS AND WEAKNESSES:\nStrength: \n- paper studies a fundamental problem which has been considered before and so many of the properties have been explored before however they gather all these findings from various resources and present a comprehensive study of previous work. \n- Introduction of new way of measuring the discrepancy between original and compressed distribution. Showing that it is metric and providing a closed-formula for it. \n- Proposing new algorithms with better runtime and measuring their performance in practice. \n- Strong empirical evidence of superiority of the proposed algorithms. \n\nWeakness: \n-  There are no discussion in terms of convergence of the point-based methods? Similarly for initialization, they mention uniform sampling but no other discussion. \n- Samples that are necessarily a part of the original training set  but appear in compression set are not thoroughly discussed, e.g., what is conditional property for these samples? how are they handled generally?\n\n\nMinor: \n- Typo Eq(2) l(y,.) instead of k(y,.) in the sum. \n- l248 add space\n\nQUESTIONS:\n- Why are they 20 points (instead of 25) in the Figure 1 for ACKIP, do we have points that are repeated? \n- Why do the point-based algorithm converge? what is their convergence time? Also what is the effect of initialization step in these methods? \n- What is the runtime of the algorithms? Can you add plots or simply report the runtime? \n- Are they non-Kernel methods are used for compression? how do they perform? How do they compare to the proposed algorithms?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 1, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses conditional distribution compression for labeled data, a problem distinct from existing joint distribution compression approaches. The authors introduce the Average Maximum Conditional Mean Discrepancy (AMCMD) as a novel metric for comparing families of conditional distributions, proving it satisfies standard metric properties (triangle inequality, identity of indiscernibles). Leveraging this metric, they propose two algorithms: (1) Average Conditional Kernel Herding (ACKH), a greedy linear-time algorithm, and (2) Average Conditional Kernel Inducing Points (ACKIP), a joint optimization alternative. For comparison, they also develop joint distribution compression methods: Joint Kernel Herding (JKH) and Joint Kernel Inducing Points (JKIP). A key theoretical contribution is reducing AMCMD estimation complexity from O(n³) to O(n) via the tower property (Lemma 4.7). Experiments across synthetic and real datasets demonstrate that (1) conditional compression outperforms joint compression, and (2) joint optimization consistently outperforms greedy approaches, with ACKIP achieving near full-data performance using only 3% of the original dataset.\n\nSTRENGTHS AND WEAKNESSES:\n### Strength\n\n- The paper has considered both the joint and conditional kernel herding and uses experiments to prove the effectiveness of doing conditional distribution compression, rather than the joint one.\n- This paper is the first direct approach to conditional distribution compression, addressing a genuine gap in the literature where existing methods compress joint distributions and hope conditional structure is preserved.\n- The paper presents a clear logical progression, going from joint to conditional and from greedy to the joint optimization.\n- The paper also addresses a computational bottleneck of estimating AMCMD from O(n^3) to O(n).\n- Demonstrates substantial compression ratios (97% data reduction) while maintaining statistical performance, with potential to significantly expand KCME applicability.\n\n### Weakness\n\n- The experiments are a bit weak in terms of the downstream tasks or practical guidance. For example, the Kernel Herding paper[3]’s experiments setting that this paper claims to follow also includes the Bayesian Posterior.\n- Experiments also lack the comparison to other baseline methods.\n- The paper claims AMCMD is \"more general\" than existing metrics (KCD/AMMD) but provides no compelling use cases requiring this generality\n- The writing of the paper requires substantial work. There is dense notation in Section 4 with insufficient motivation for theoretical developments\n- Memory constraints limit mnIST to 10K samples, reducing confidence in large-scale applicability\n\nQUESTIONS:\n- Since JKH and JKIP both have the same complexity of O(mn + m^2), do you observe any empirical speed difference between the two? Is there any scenario that the greedy one is preferred?\n- Is there a mistake in equation (2) - should be $k(x, \\tilde{x_j})l(y, \\tilde{y}_j)$ instead of $k(x, \\tilde{x_j})k(y, \\tilde{y}_j)$\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nExisting methods like Kernel Herding (KH) were originally designed to compress unlabelled data distributions, but no prior approach directly compresses the conditional distribution of labelled data. To fill this gap, the paper first introduce Average Maximum Conditional Mean Discrepancy (AMCMD), a metric for comparing conditional distributions, along with a closed-form estimator. Then show that constructing a compressed set targeting AMCMD can be made much more efficient—reducing the cost from O(n³) to O(n). Based on this insight, we extend KH and propose Average Conditional Kernel Herding (ACKH), a linear-time greedy algorithm for compressing conditional distributions.\n\nTo compare with joint distribution compression, the paper also introduce Joint Kernel Herding (JKH), a straightforward extension of KH for joint data. While herding methods are simple and interpretable, they rely on greedy heuristics. \n\nTo explore better alternatives, the paper  propose Joint Kernel Inducing Points (JKIP) and Average Conditional Kernel Inducing Points (ACKIP)—which jointly optimize the compressed set while still maintaining linear complexity.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper extends existing kernel compression techniques to the conditional distribution setting and proposes a linear-time optimization approach based on gradient descent. I did not identify any major issues regarding the correctness of the method.\n2. Experimental results under various data distribution settings demonstrate that the proposed method consistently achieves lower RMSE.\n\nWeaknesses: \n1. Writing and Structure:\nThe writing of the paper needs improvement. While several method variants are introduced, including some mentioned in the abstract, it is difficult to follow the key logical connections between them. The discussion of related work is insufficient in the main text; although some references are included in the appendix, this weakens the overall motivation of the paper. For example, it is unclear why existing methods perform poorly and why compressing the joint distribution is less effective than compressing the conditional distribution. I recommend adding a dedicated section to clearly outline the main contributions and structure of the paper. In addition, many important details require flipping back and forth between the main text and the appendix, which disrupts the flow of reading.\n\n2. Theoretical Contribution:\nThe theoretical novelty is relatively weak. The proposed criteria are derived based on well-studied conditional kernel mean embedding estimators, and the derivation mainly relies on direct computation. The paper lacks convergence analysis or sample error bounds, which are essential for establishing the robustness of the method.\n\n3. Experimental Evaluation:\nThe experimental section lacks clarity in dataset presentation. For example, in line 255, one dataset is briefly mentioned without any description of its features or relevance. Additionally, the meaning of the x-axis in the figures is not explained, making the visualizations less informative.\n\n4. Experimental Limitations:\nMost experiments rely on the Gaussian kernel and are conducted on relatively low-dimensional variables, which may limit the generality of the conclusions. Furthermore, the evaluation metric is solely based on RMSE. While RMSE is appropriate for regression tasks, the goal of distribution compression is often to support downstream tasks more broadly. Therefore, including results on a wider range of downstream applications would make the evaluation more convincing and practical.\n\nminor: A few minor points:\n1. The notation for the separable measure space in lines 85 and 86 is inconsistent.\n2. Why is it m+1​ in Equation (2) instead of m?\n3. In line 219, two matrix dimensions appear to be inconsistent.\n4. line 248: Figures 2 and3 ->Figures 2 and 3\n\nQUESTIONS:\nsee weakness.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a novel distribution compression method specifically designed to compress conditional distributions, addressing a gap in existing techniques (e.g., Kernel Herding) that primarily focus on marginal or joint distributions. The method belongs to the kernel family, minimizing the maximum mean discrepancy (MMD) between kernel embeddings to yield a distilled dataset that best represents the original data. To measure differences between conditional distributions efficiently, they introduce the AMCMD metric. Two optimization approaches for MMD minimization are developed and compared: (a) A greedy algorithm (ACKH) with linear-time complexity (when using a specially designed estimator), (b) A non-greedy, joint-optimization method (ACKIP) with $O(m^3 + m^2 n)$ complexity, which is a factor of $m$ faster than ACKH, where $m$ is the cardinality of  a compressed dataset.\n\nA conventional compression algorithm for marginal distributions is also extended to the joint distribution case to provide a fair comparison to the methods of the same family. An experimental evaluation is conducted to benchmark all the four approaches considered. The results suggest that the proposed conditional approach is superior.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\nThis work addresses an important challenge in machine learning: reducing dataset size without significant quality degradation, which could enable faster model training and inference. The study is both comprehensive and mathematically rigorous, with clear presentation. Experiments on synthetic data and the superconductivity dataset demonstrate that the proposed conditional approach consistently outperforms joint distribution methods and random subsampling. Additionally, the Appendix provides extensive supporting details.\n\n**Weaknesses:**\n\n1. The background review focuses exclusively on kernel methods, omitting discussion of non-kernel approaches. The experimental comparison similarly fails to include non-kernel baselines, limiting the perspective on the method's relative advantages.\n2. While distribution compression has critical applications in generative modeling, computer vision, speech, and NLP (domains characterized by high-dimensional data) the evaluation is conducted primarily in low-dimensional settings. The authors use only one real-world dataset and notably avoid addressing their method's scalability with increasing dimensionality.\n3. The analysis concentrates on AMCMD and RMSE for test function expectation approximations, while downstream task performance (e.g., regression/classification accuracy using compressed data for training ML models) would provide more practical insight. This omission leaves the method's real-world utility unclear.\n4. The argument for the consistency of the proposed AMCMD assumes that the distributions of $X^*$ and $X$ are equal (Corollary 4.4). However, the original definition emphasizes the use of a distribution different from $X$, rendering the argument non-rigorous.\n\nQUESTIONS:\n1. What are the contemporary neural approaches (or methods of other, non-kernel families) for distribution compression? How does your method performs in comparison to them?\n2. Are there any kernel preferences specific to the task in question (kernel embeddings construction, distribution compression)? Why have not you tried other kernels?\n3. How does your method scale with dimensionality (in theory and in practice)?\n\nLIMITATIONS:\nThe authors discuss limitations of their method throughout the text. However, other, more practical limitations are overlooked; see **Weaknesses**.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}