Title: On Certified Generalization in Structured Prediction
Venue: NeurIPS 2023
Accepted: yes
Keywords: Structured Prediction

PAC-Bayes

Concentration Inequalities

Statistical Learning Theory

Knothe-Rosenblatt Rearrangement
Abstract: In structured prediction, target objects have rich internal structure which does not factorize into independent components and violates common i.i.d. assumptions. This challenge becomes apparent through the exponentially large output space in applications such as image segmentation or scene graph generation.
We present a novel PAC-Bayesian risk bound for structured prediction wherein the rate of generalization scales not only with the number of structured examples but also with their size.
The underlying assumption, conforming to ongoing research on generative models, is that data are generated by the Knothe-Rosenblatt rearrangement of a factorizing reference measure. This allows to explicitly distill the structure between random output variables into a Wasserstein dependency matrix. 
Our work makes a preliminary step towards leveraging powerful generative models to establish generalization bounds for discriminative downstream tasks in the challenging setting of structured prediction.

On Certified Generalization in Structured Prediction

Abstract

In structured prediction, target objects have rich internal structure which does not factorize into independent components and violates common i.i.d. assumptions. This challenge becomes apparent through the exponentially large output space in applications such as image segmentation or scene graph generation. We present a novel PAC-Bayesian risk bound for structured prediction wherein the rate of generalization scales not only with the number of structured examples but also with their size. The underlying assumption, conforming to ongoing research on generative models, is that data are generated by the Knothe-Rosenblatt rearrangement of a factorizing reference measure. This allows to explicitly distill the structure between random output variables into a Wasserstein dependency matrix. Our work makes a preliminary step towards leveraging powerful generative models to establish generalization bounds for discriminative downstream tasks in the challenging setting of structured prediction.

1 Introduction

1.1 Overview

Structured prediction is the task of predicting an output which itself contains internal structure. As an example, consider the problem of image segmentation. The output to be predicted is an assignment of semantic classes to each image pixel. However, the segmentation problem is not merely a pixelwise classification, because each pixel is not independently assigned a semantic class. If two pixels are adjacent in the image plane, it is more likely that they belong to the same class than to different ones. A way to remedy this problem is to enumerate all possible segmentations of an image and treat the problem as classification. However, the output space of this classification is now exponentially large. This exponential (in the number of pixels) size of the output space indicates a much more difficult problem than the unstructured case of classification. In fact, it also presents an immediate challenge for any statistical learning theory which requires assumptions on the number of output classes [44].

From a practical perspective, structured prediction problems present a challenge of label aquisition. For instance, dense manual segmentation of an image is significantly more labour intensive than manual classification. In turn, one would expect that pixelwise segmentation of an image contains much richer information to be exploited in supervised learning. However, this is not represented in typical statistical learning theory which assumes all data to be independently drawn from the true underlying distribution. In the extreme case of only a single structured example being available for training, these statistical learning theories can not make any meaningful statement on generalization.

Addressing this point in particular, [38] presents an analysis of the dependency structure in the output and proves a risk certificate which decays in both the number of structured examples m and their size d . Refering back to the example of image segmentation, this amounts to a high-probability bound on the fraction of misslabeled pixels which decays with the number of labeled pixels observed during training as opposed to merely the number of segmented images.

At the core of statistical learning theory lies the concentration of measure phenomenon which posits that a stable function of a large number of weakly dependent random variables will take values close to its mean [37, 9]. This is relevant because model risk, the expected loss on unseen data, is the mean of empirical risk under the draw of the sample. Learning theories can thus be built on concentration of measure results. In the present work, we focus on PAC-Bayesian learning theory [51, 43, 42]. For an overview of PAC-Bayesian theory we refer to [13, 27, 1].

In addition to a concentration result such as a moment-generating function (MGF) bound, PACBayesian arguments employ a change of measure, typically via Donsker and Varadhan's variational formula. In particular, stochastic predictors, i.e. distributions over a hypothesis class of models are considered. A core objective is to construct generalization bounds which hold uniformly over all stochastic predictors called PAC-Bayes posteriors . Model complexity is then measured as relative entropy to a reference stochastic predictor called PAC-Bayes prior . As this terminology alludes to, the PAC-Bayes posterior is informed by more data than the PAC-Bayes prior. Note however, that PAC-Bayesian theory generalizes Bayesian theory because prior and posterior are not typically connected via likelihood.

PAC-Bayesian theory has gained considerable attention in recent years due to the demonstration of non-vacuous risk certificates in deep learning [23]. Since then, a line of research has succeeded in tightening the risk bounds of deep classifiers [48, 47, 16]. In addition, multiple authors have studied ways to weaken underlying assumptions of bounded loss [29, 28] and i.i.d. data [3].

The majority of recent works in this field focuses on classification or regression. Structured prediction has received comparatively little recent attention, an exception being the work [11] which provides a PAC-Bayesian perspective on the implicit loss embedding framework for structured prediction [15].

1.2 Related Work

Here, we continue a line of research started by [39, 40, 38] which aims to construct PAC-Bayesian risk bounds for structured prediction that account for generalization from a single (but large) structured datum. Instrumental to their analysis is the stability of inference and quantified dependence in the data distribution. The latter is expressed in terms of ϑ -mixing coefficients, the total variation distance between data distributions conditioned on fixed values of a subset of variables. For structured prediction with Hamming loss, a coupling of such conditional measures can be constructed [24] such that ϑ -mixing coefficients yield an upper-bound that allows to invoke concentration of measure arguments [36]. The result is an MGF bound which the authors employ in a subsequent PAC-Bayesian construction - achieving generalization from a single datum.

The underlying assumption of these previous works is that data are generated by a Markov random field (MRF). This model assumption is somewhat limiting because Markov properties, certain conditional independences, likely do not hold for many real-world data distributions. In addition, MRFs are difficult to work with computationally. Exact inference in MRFs is NP-hard [57] and thus learning, which often contains inference as a subproblem, presents significant computational roadblocks. Even once an MRF has been found which represents data reasonably well, numerical evaluation of the PAC-Bayesian risk certificate proposed in [38] will again be computationally difficult.

Nevertheless, we share the sentiment of previous authors that some assumption on the data-generating process is required in structured prediction. This is because conditional data distributions, the distribution of data conditioned on a fixed set of values for a subset of variables, are central to establishing concentration of measure via the martingale method [35]. Consider again the example of image segmentation. Once we have fixed a sufficiently large number of pixels to arbitrary values (and class labels), even a large dataset will not contain an abundance of data which match these values and thus provide statistical power to learn the conditional distribution. This problem is well-known in conditional density estimation [56].

1.3 Contribution, Organization

In this work, we propose to instead assume a triangular and monotone transport, a Knothe-Rosenblatt (KR) rearrangement [32, 49, 12, 8, 41] of a reference measure as data model. This choice is attractive for multiple reasons. First, any data distribution which does not contain atoms can be represented

uniquely in this way [6] which should suffice to represent many distributions of practical interest. With regard to conditional distributions, the KR-rearrangement has the convenient property that conditioning on a fixed value for a subset of variables can again be represented by KR-rearrangement. We will use this property in our construction of coupling measures between conditional distributions.

Specifically,

We present a novel PAC-Bayesian risk bound for structured prediction wherein the rate of generalization scales not only with the number of structured examples but also with their size.

Based on data generated by KR-rearrangement of a tractable reference measure, we distill relevant structure of the data distribution into a Wasserstein dependency matrix. Our analysis hinges on state-of-the-art results in concentration theory [35] which serve to bound momentgenerating functions by properties of the Wasserstein dependency matrix. We subsequently invoke a PAC-Bayesian argument to derive the desired risk certificate.

We also propose to leverage a construction of bad input data as a computational tool to find entries of the Wasserstein dependency matrix.

We stress the fact that many established approaches to generative modelling can be seen as instances of measure transport. For instance, it includes normalizing flows [54, 53, 33, 46, 50], diffusion models [52, 30], generative adversarial networks and variational autoencoders [10, 26]. While most measure transport models which currently enjoy empirical success are not KR-rearrangements, we hope that the methods presented here can lay the foundation of leveraging powerful generative models to build risk certificates for discriminative downstream tasks.

Organization The central concepts of concentration theory and measure transport are described in the preliminary Sections 2 and 3. The main results are presented in Section 4. In Section 5 we present a first discussion of computational aspects related to the presented framework. The paper closes on a discussion of limitations in Section 6 and a conclusion in Section 7.

Basic Notation For any d ∈ N , denote [ d ] = { 1 , . . . , d } . If z ∈ Z d is a vector, we refer to the subvector of entries with index in a set I ⊆ [ d ] as z I . In particular, index sets of interest will be half-open and closed intervals ( i, d ] ⊆ [ d ] and [ i, d ] ⊆ [ d ] . Analogously, we will index the output of vector-valued functions f I and marginal measures µ I . For a set B ⊆ Z d , we denote its complement in Z d by B c = Z d \ B and for a measure µ on Z d , we denote the conditional measure given B c as µ |B c . If Z is a random variable with distribution µ on Z d and I, J ⊆ [ d ] are disjoint index sets with I ∪ J = [ d ] , we denote the conditional law of Z I given Z J = z J as µ ( dz I | z J ) .

2 Concentration of Measure and Generalization

Let X denote an input space and Y denote an output space. Let µ be a distribution on Z d = ( X ×Y ) d . There are two restrictions inherent to this setup. First, an input is always paired with an output and thus the number of inputs needs to match the number of outputs. Second, all structured data will be drawn from µ and thus the size of each structured datum will be the same. Otherwise, X and Y can in principle be arbitrary sets which admit metrics. For concreteness, think of X = [0 , 1] ⊆ R as being a set of gray values and Y = R containing signed distances from a semantic boundary [45] in an image with d pixels. In this case, Z d contains all binary segmentations of grayvalue images.

The goal of learning is to find parameters θ which define a predictor ϕ θ : X d →Y d such that the risk

with respect to some loss function L : Y d ×Y d → R is minimized. However, the true risk (1) is typically intractable because µ is unknown. A related tractable quantity is the empirical risk

of a sample D m = ( X ( k ) , Y ( k ) ) k ∈ [ m ] drawn from µ m . We further assume that the loss of structured outputs is the mean of bounded pointwise loss ℓ : Y × Y → [0 , 1]

In PAC-Bayesian constructions, we consider stochastic predictors ζ , i.e. measures on a hypothesis space H of predictors ϕ θ : X d →Y d as identified with measures on the underlying parameter space from which θ is selected. We then take the above notions of risk and empirical risk in expectation over parameter draws

The expected value of empirical risk with respect to the sample is the true risk. For this reason, a central tool for the study of generalization is the concentration of measure phenomenon. Informally, it states that a stable function of many weakly dependent random variables concentrates on its mean. We will invoke a line of reasoning put forward in [35] and propose a novel approach to structured prediction based on the measure-transport framework outlined in Section 1. To this end, we first define the following formal notions of stability and dependence .

Let ρ be a metric such that Z has finite diameter

and let ρ d ( z, z ′ ) = ∑ i ∈ [ d ] ρ ( z i , z ′ i ) denote the corresponding product metric on Z d .

Definition 1 ( Local oscillation ) . Let f : Z d → R be Lipschitz with respect to ρ d . Then the quantities

are called the local oscillations of f .

The vector of local oscillations gives a granular account of stability. In order to discuss interdependence of data in a probability space ( Z d , µ, Σ) , define the Markov kernels

as well as K ( d +1) ( z, dw ) = δ z ( dw ) and their action on functions

where in the edge case i = 1 , the condition on z [ i -1] = z {} is removed. Here K ( i ) ( z, dw ) is a Borel measure for every z and (8) computes the expected value of f at z , conditioned on the fixed realization of the subvector z [ i -1] . It turns out that the effect of the kernel (7) on local oscillations serves to quantify dependence of data with joint distribution µ .

Definition 2 ( Wasserstein matrix ) . For i ∈ [ d +1] , let K ( i ) denote the Markov kernel (8) . A matrix V ( i ) ∈ R d × d ≥ 0 is called a Wasserstein matrix [25] for K ( i ) , if

for any function f : Z d → R which is Lipschitz with respect to ρ d .

The two concepts defined above will be used in Section 4 to construct a moment-generating function bound via the martingale method.

3 Triangular Measure Transport

Suppose a structured output is composed of d > 0 unstructured data in a space Z . Then the target measure µ of interest is a measure on Z d which does not factorize into simpler distributions. A popular method of representing complex joint distributions of interdependent random variables is to define a map T : Z d →Z d which transports a tractable factorizing reference measure ν d to the target measure µ , i.e. T ♯ ν d = µ . This abstract framework encompasses many generative models such as normalizing flows [54, 53, 33, 46, 50], diffusion models [52, 30], generative adversarial networks and variational autoencoders [10, 26]. Here, we focus on transport maps T which are monotone and triangular in the sense that T ( z ) i only depends on the inputs z [ i ] and each T ( z [ i -1] , · ) i is an increasing function. Such a map is called a Knothe-Rosenblatt (KR) rearrangement [32, 49, 12, 8, 41]. If both ν d and µ have no atoms then the KR rearrangement exists and is unique [6]. In particular, normal distribution ν d and any absolutely continuous (with respect to the Lebesgue measure) distribution µ meet these criteria. The KR rearrangement has the useful property that certain conditional distributions have a simple representation.

Lemma 3 ( Lemma 1 of [41] ) . Let T : Z d →Z d be the KR-rearrangement which satisfies T ♯ ν d = µ . For arbitrary i ∈ [ d ] , let z [ i ] ∈ Z i be fixed. Then

where z [ i ] is the unique element of Z i such that T [ i ] ( z [ i ] ) = z [ i ] .

Numerical realization of KR rearrangements has recently received attention [5] and more broadly, a variety of triangular transport architectures exists [20, 21]. However, we do not focus on numerical considerations in the present theoretical work.

One may wonder if data generated by KR-rearrangement implicitly restricts the choice of possible input and output spaces since the monotonicity requirement on T can only be satisfied if the underlying set Z is ordered. However, many feature spaces of practical interest are still permissible for what follows. In particular, both X and Y may be Euclidean spaces or hypercubes. Inkeeping with the introductory image segmentation example, suppose X = [0 , 1] 3 contains RGB color values and Y = [ -1 , 1] contains signed distance from a semantic boundary in the image plane. Then we can interpret Z d = ([0 , 1] 3 × [ -1 , 1]) d as a product of compact intervals in R 4 d which is clearly permissible as underlying space for KR-rearrangement. All presented results hold irrespective of whether Z contains such internal dimensions as long as a natural ordering of the set Z is induced.

4 PAC-Bayesian Risk Certificate

In this section we present a novel PAC-Bayesian risk bound for structured prediction which combines three main ingredients.

A concentration of measure theorem for dependent data (Theorem 4) which builds on the notion of a Wasserstein dependency matrix;

a simple construction of coupling measures between conditional distributions (Lemma 5) which serves to represent the Wasserstein dependency matrix;

a PAC-Bayesian argument (Theorem 7) employing Donsker-Varadhan's variational formula in concert with concentration of measure results.

The first theorem summarizes key results from [35] on the concentration of measure phenomenon for dependent random variables. We have slightly generalized by augmenting the underlying Doob martingale construction with the inclusion of a set B of bad inputs . For inputs in this set, data stability requirements do not necessarily hold. We call the complement B c = Z d \ B the set of good inputs . The concept of good and bad inputs as well as related proof techniques were originally proposed by [38]. Here, we incorporate them into the more general concentration of measure formalism of [35]. A full proof of the following theorem is deferred to Appendix A.

Theorem 4 ( Moment-generating function (MGF) bound for good inputs ) . Let B ⊆ Z d be a measurable set of bad inputs. Suppose for each i ∈ [ d +1] , V ( i ) is a Wasserstein matrix for the

Markov kernel K ( i ) defined in (7) on the set of good inputs, that is

for all Lipschitz (with respect to ρ d ) functions ˜ f : B c → R . Define the Wasserstein dependency matrix

Then for all Lipschitz functions f : Z d → R , the following MGF bound holds

An upper bound on the moment generating function will be used in the PAC-Bayesian argument concluding this section. The function f in question will be the loss of a structured datum z . Regarding (13), our goal is to bound the norm ∥ Γ δ ( f ) ∥ 2 2 through properties of the data distribution. We will use the fact that data is represented by measure transport to establish such a bound after the following preparatory lemma.

Lemma 5 ( Coupling from transport ) . Let ν d be a reference measure on Z d and F, G : Z d →Z d be measurable maps. Define the map ( F, G ) by

Then ( F, G ) ♯ ν d is a coupling of F ♯ ν d and G ♯ ν d .

Proof. Let A ⊆ Z d be measurable, then

which shows that F ♯ ν d is the first marginal of ( F, G ) ♯ ν d . An analogous argument for the second marginal shows the assertion.

By assuming µ to be represented via KR-rearrangement of a factorizing reference measure, Lemma 3 gives an explicit representation of KR-rearrangement for conditional distributions. From there, we invoke Lemma 5 to construct a coupling between conditional distributions and subsequently follow a line of reasoning put forward in [35] to explicitly construct Wasserstein matrices for the kernels (7) which yield a bound on (13) by Theorem 4. This leads to the following proposition. A full proof is deferred to Appendix A.

Proposition 6 ( Wasserstein dependency matrix from KR-rearrangement ) . Let ( Z d , Σ , µ ) be a probability space with µ = T ♯ ν d for the KR-rearrangement T : Z d →Z d and a reference measure ν d on Z d . Let each Z be equipped with a metric ρ and have finite diameter ∥ ρ ∥ < ∞ . Let f : Z d → R be a Lipschitz function with respect to the product metric ρ d . Let B ⊆ Z d denote a set of bad inputs and define the corresponding set A = T -1 ( B ) ⊆ Z d . Let ̂ T be the unique KR-rearrangement that satisfies ̂ T ♯ ν d = ν d |A c and denote ˜ T = T ◦ ̂ T . Suppose there exist constants L ij such that for all v, z ∈ B c with v [ d ] \{ i } = z [ d ] \{ i } it holds

where ̂ v [ i ] and ̂ z [ i ] are uniquely defined through ˜ T [ i ] ( ̂ v [ i ] ) = v [ i ] and ˜ T [ i ] ( ̂ z [ i ] ) = z [ i ] . Then Γ = ∥ ρ ∥ d D is a Wasserstein dependency matrix for µ |B c with

We remark that Γ indeed distills the dependency structure of µ . To illustrate this, let µ { i } be independent from µ { j } for some i ∈ [ d ] , j ∈ ( i, d ] . Then conditioning on a different value of µ { i } does not change the distribution µ { j } . Thus,

The following theorem states the main result of the present work.

Theorem 7 ( PAC-Bayesian risk certificate for structured prediction ) . Fix δ ∈ (0 , exp( -e -1 )) , let µ be a data distribution on Z d with T ♯ ν d = µ the Knothe-Rosenblatt rearrangement for a reference measure ν d on Z d and fix a measurable set B ⊆ Z d of bad inputs with µ ( B ) ≤ ξ . Fix a PAC-Bayes prior π on a hypothesis class H of functions ϕ : X d → Y d and a loss function ℓ which assumes values in [0 , 1] . Define the oscillation vector ˜ δ by

where L ( h, · ) ∣ ∣ B c denotes the restriction of L ( h, · ) to Z d \ B . Suppose all oscillations ˜ δ i are finite, suppose the condition (16) is satisfied and denote by D the matrix with entries (17) . Then, with probability at least 1 -δ over realizations of a training set D m = ( Z ( k ) ) m k =1 drawn from ( µ |B c ) m it holds for all PAC-Bayes posteriors ζ on H that

Note that the generalization gap on the right hand side of (20) decays with d . This accounts for generalization from a single example. In fact, if only m = 1 structured example is available, but d ≫ 1 , Theorem 7 still certifies risk. This effect can however be negated by the norm ∥ D ˜ δ ∥ . If structured data contain strong global dependence, then ∥ D ˜ δ ∥ will not be bounded independently of d and thus, in the worst case of ∥ D ˜ δ ∥ ∈ O ( d ) the assertion is no stronger than PAC-Bayesian bounds for unstructured data. The same point was observed in [38].

The measure of the bad set under the data distribution µ is assumed to be bounded by ξ . This is to account for a small number of data which contain strong dependence. In order to prevent these bad data from dominating D , thereby negating the decay of the bound in d as described above, it is preferable to exclude them from the sample, reduce the sample size m and pay the penalty ξ in (20).

To prove Theorem 7, we broadly follow the argument put forward in [38]. This augments typical PAC-Bayesian constructions in the literature by the inclusion of a set of bad inputs. We first reconcile the data being conditioned on B c with risk certification for unconditioned data, leading to the addition of ξ on the right hand side of (20). The model complexity term KL[ ζ : π ] is due to Donsker and Varadhand's variational formula. Subsequently, the moment generating function bound of Theorem 4 is instantiated through the Wasserstein dependency matrix constructed in Proposition 6. Markov's inequality then gives a pointwise risk bound for fixed value of a free parameter. In order to optimize this parameter, the bound is made uniform on a discrete set of values through a union bound. A full proof is presented in Appendix A.

In Theorem 7, we combine the PAC-Bayesian construction of [38] with the more general concentration of measure theory of [35]. Crucially, concentration of measure results used in [38] are predicated on the assumption of data generated by a Markov random field [57, 34]. Our work is more flexible in two major ways.

Our assumption on the data-generating distribution is likely more representative of real-world data as measure transport models have repeatedly been shown to yield convincing data generators.

Markov random fields are difficult to handle computationally because inference in general Markov random fields is NP-hard [57] such that one is forced to learn based on approximate inference procedures [31, 22, 55].

Our work also allows for more general metrics ρ as opposed to the singular choice of Hamming norm required in [38]. Additionally, the key results of [38] are constructed to ensure all data drawn from the unconditioned distribution µ are in the good set. This reduces the probability of correctness 1 -δ by mξ . Instead, we assume data drawn from µ |B c , effectively reducing the number of available samples by a factor of 1 -ξ , but keeping the probability of correctness high. This allows the set of bad inputs to be used more effectively as a computational tool in Section 5.

Comparing the dependency of (20) on d with the respective result in [38], it first appears as though our bound decays with a faster rate ( d instead of √ d ). However, this will not typically be the case in practice because ∥ D ˜ δ ∥ 2 grows with rate √ d in most situations. To see this, consider the case of local

dependency in the sense that

for local neighborhoods N i ⊆ [ d ] which contain a fixed number of c elements and let ˜ δ i = α for all i ∈ [ d ] and some constant value α > 0 . Then

Clearly, if dependence is localized and the oscillations ˜ δ do not decay in d , then ∥ D ˜ δ ∥ 2 contains a factor that grows with rate √ d , leading to the same asymptotic rate of (20) already observed in [38].

Note that [38] additionally allows for a set of bad hypotheses B F ⊆ H which do not conform to stability assumptions. In our construction, this means restricting the bound (19) to oscillations on the set of good hypotheses. We omit this extension for clarity of exposition, but do not expect it to necessitate major changes to the presented proofs. The same applies to the derandomization strategy proposed by [38] which is based on hypothesis stability.

Further, [38] considers a large number of applicable orderings for random variables by introducing a filtration of their index set. This notion is not easily compatible with our assumption of KRrearrangement, because triangularity of transport depends on the order of variables.

5 Bounding the Bad Set

With regard to numerical risk certificates, a key technical aspect of Section 4 concerns the quantities L ij in (16). Here, we propose a way to use the set of bad inputs as a computational tool to this end. Suppose we assign arbitrary fixed values to L ij and subsequently define B ⊆ Z d as the set of inputs on which the condition (16) fails. Then we have fulfilled the prerequisites of Proposition 6 by construction and are left with bounding µ ( B ) . Note that

and the indicator function 1 assumes values in the bounded set { 0 , 1 } . Therefore, Hoeffding's inequality gives the following.

Proposition 8 ( Upper bound on the bad set ) . Let µ be a data distribution and let ˜ D n ∼ µ n be a sample of size n . Fix an error probability ϵ ∈ (0 , 1) . Then

with probability at least 1 -ϵ over the sample.

Checking the condition Z ∈ B requires evaluating (16) which comes down to finding a Lipschitz constant for a one-dimensional function. If this is computationally feasible for the given data model, then the concentration argument (24) can be used to bound µ ( B ) with high probability. Because (24) decays only in the number of structured examples O ( √ n ) , it can not be used to show generalization from a single structured example. However, PAC-Bayesian risk certificates are typically dominated by the KL complexity term in (20) which decays with the size of structured examples as well. Thus, Proposition 8 should still be useful in practice.

Note that Proposition 8 makes a pointwise statement about a fixed value of L ij which has limited utility for learning L ij from data. To remedy this problem, we can first define a discrete set L = ( L ( k ) ) k ∈ [ l ] of candidate matrices and select error probabilities ϵ k for each of the events

such that ∑ k ∈ [ l ] ϵ k = ϵ . Then, by a union bound with probability at least 1 -ϵ over the sample none of the events (25) occurs. We have thus constructed a uniform bound over the set of candidate matrices which allows us to select the one which minimizes the generalization gap in (20).

In order to make this strategy most effictive, domain knowledge on the application at hand should be applied when constructing candidate matrices and assigning error probabilities. For instance, the limited empirical findings of [14] on ImageNet [17] indicate that the majority of natural images contain mostly local signal. In image segmentation, this is conducive to concentration, because it can lead to many small values in an optimal Wasserstein dependency matrix. In particular, if dependency decays with distance in the image domain, one should select configurations L ( k ) in which L ( k ) ij is small if i is distant from j in the image domain and allowed to assume larger values if i is close to j in the image domain.

We give an intuitive interpretation of the relationship between Proposition 8 and Theorem 7 as follows. Suppose the majority of samples from a structured data distribution contain mostly local signal. The locality of signal in samples indicates weak global dependence of random variables which in turn manifests in small entries of a Wasserstein dependency matrix. However, a small number of bad data may contain only weak local signal. For instance, an image in which every pixel has the same value does not give more information to a learner if it is doubled in size. Even worse, a small (but not null) set of bad data will dominate the Wasserstein dependency matrix and prevent generalization that scales with d . Proposition 8 thus estimates an upper bound on the likelihood of bad data under µ which are then excluded from the concentration argument underlying the bound (20). This relationship is further illustrated in a numerical toy example in Appendix C.

6 Limitations

We assume that structured data are drawn from a distribution µ which arises from a reference measure through KR-rearrangement. This is motivated by the fact that a large class of data distributions can be represented in this way and that some assumption on the data-generating distribution is required in order to make statements on conditional distributions required to quantify dependence. However, it is an open question how closely a measure transport distribution learned from data approximates the unknown distribution from which the data are drawn. A first step towards this goal is made in the recent work [4], but a non-asymptotic theory of generalization for generative models is left for future work. Accordingly, we do not present any empirical results on real-world data.

We describe how dependency in the data distribution which is relevant for generalization of discriminative models can be written as properties of the KR-rearrangement through a Wasserstein dependency matrix. Section 5 further outlines how the construction of bad data can be used as a computational tool to this end. We do not cover how to compute the involved Lipschitz constants L ij in (16). In practice, this will require nontrivial numerical machinery because a deterministic bound on the expected value under the reference distribution appears needed. One possible approach is to employ quasi-Monte-Carlo methods [19, 18] which come with deterministic error bounds and have recently been applied to PAC-Bayesian certification of ordinary (non-structured) classification risk [7]. This entails a fine-grained stability analysis of the involved integrands and is likely to be computationally expensive because deterministic error bounds tend to be pessimistic compared to their stochastic counterparts.

7 Conclusion

We have presented a PAC-Bayesian risk certificate for structured prediction. Compared to earlier work, we make the assumption of data generated by KR-rearrangement of a reference measure. This approach is able to represent all atom-free data distributions and yields an explicit quantification of dependence via Wasserstein dependency matrices. It also indicates a way of leveraging powerful generative models to compute risk certificates for downstream discriminative tasks.

Acknowledgments and Disclosure of Funding

This work is funded by the Deutsche Forschungsgemeinschaft (DFG), grant SCHN 457/17-1, within the priority programme SPP 2298: 'Theoretical Foundations of Deep Learning'. This work is funded by the Deutsche Forschungsgemeinschaft (DFG) under Germany's Excellence Strategy EXC-2181/1 - 390900948 (the Heidelberg STRUCTURES Excellence Cluster).

References

Pierre Alquier. User-friendly introduction to pac-bayes bounds. arXiv preprint arXiv:2110.11216 , 2021.

Pierre Alquier. User-friendly introduction to pac-bayes bounds, 2023.

Pierre Alquier and Benjamin Guedj. Simpler pac-bayesian bounds for hostile data. Machine Learning , 107(5):887-902, 2018.

Ricardo Baptista, Bamdad Hosseini, Nikola B Kovachki, Youssef M Marzouk, and Amir Sagiv. An approximation theory framework for measure-transport sampling algorithms. arXiv preprint arXiv:2302.13965 , 2023.

Ricardo Baptista, Youssef Marzouk, and Olivier Zahm. On the representation and learning of monotone triangular transport maps, July 2022. arXiv:2009.10303 [cs, math, stat].

V I Bogachev, A V Kolesnikov, and K V Medvedev. Triangular transformations of measures. Sbornik: Mathematics , 196(3):309, apr 2005.

B. Boll, A. Zeilmann, S. Petra, and C. Schnörr. Self-Certifying Classification by Linearized Deep Assignment. PAMM: Proc. Appl. Math. Mech. , 23(1):e202200169, 2023.

N. Bonnotte. From Knothe's Rearrangement to Brenier's Optimal Transport Map. SIAM J. Math. Anal. , 45(1):64-87, 2013.

Stéphane Boucheron, Gábor Lugosi, and Pascal Massart. Concentration Inequalities: A Nonasymptotic Theory of Independence . Oxford University Press, February 2013.

O. Bousquet, S. Gelly, I. Tolstikhin, C.-J Simon-Gabriel, and B. Schölkopf. From optimal transport to generative modeling: the VEGAN cookbook. preprint arXiv:1705.07642 , 2017.

Théophile Cantelobre, Benjamin Guedj, María Pérez-Ortiz, and John Shawe-Taylor. A pacbayesian perspective on structured prediction with implicit loss embeddings. arXiv preprint arXiv:2012.03780 , 2020.

G. Carlier, A. Galichon, and F. Santambrogio. From Knothe's Transport to Brenier's Map and a Continuation Method for Optimal Transport. SIAM Journal on Mathematical Analysis , 41(6):2554-2576, 2010.

O. Catoni. PAC-Bayesian Supervised Classification: The Thermodynamics of Statistical Learning . Institute of Mathematical Statistics, 2007.

Carlo Ciliberto, Francis Bach, and Alessandro Rudi. Localized Structured Prediction. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019.

Carlo Ciliberto, Lorenzo Rosasco, and Alessandro Rudi. A general framework for consistent structured prediction with implicit loss embeddings. The Journal of Machine Learning Research , 21(1):3852-3918, 2020.

Eugenio Clerico, George Deligiannidis, and Arnaud Doucet. Conditionally gaussian pac-bayes. In International Conference on Artificial Intelligence and Statistics , pages 2311-2329. PMLR, 2022.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A largescale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248-255. IEEE, 2009.

Josef Dick, Frances Y Kuo, and Ian H Sloan. High-dimensional integration: the quasi-monte carlo way. Acta Numerica , 22:133-288, 2013.

Josef Dick and Friedrich Pillichshammer. Digital nets and sequences: discrepancy theory and quasi-Monte Carlo integration . Cambridge University Press, 2010.

Laurent Dinh, David Krueger, and Yoshua Bengio. NICE: Non-linear Independent Components Estimation, April 2015. arXiv:1410.8516 [cs].

Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp, 2017.

J. Domke. Learning Graphical Model Parameters with Approximate Marginal Inference. IEEE Trans. Pattern Analysis and Machine Intelligence , 35(10):2454-2467, 2013.

Gintare Karolina Dziugaite and Daniel M Roy. Data-dependent pac-bayes priors via differential privacy. In Advances in Neural Information Processing Systems , volume 31 of NIPS . Curran Associates, Inc., 2018.

Doris Fiebig. Mixing properties of a class of bernoulli-processes. Transactions of the American Mathematical Society , 338(1):479-493, 1993.

H. Föllmer. Tail structure of markov chains on infinite product spaces. Zeitschrift für Wahrscheinlichkeitstheorie und Verwandte Gebiete , 50(3):273-285, Jan 1979.

A. Genevay, G. Peyré, and M. Cuturi. GAN and VAE from an Optimal Transport Point of View. preprint arXiv:1706.01807 , 2017.

B. Guedj. A Primer on PAC-Bayesian Learning. CoRR abs/1901.05353 , 2019.

Benjamin Guedj and Louis Pujol. Still No Free Lunches: The Price to Pay for Tighter PACBayes Bounds. Entropy , 23(11):1529, November 2021.

Maxime Haddouche, Benjamin Guedj, Omar Rivasplata, and John Shawe-Taylor. Pac-bayes unleashed: Generalisation bounds with unbounded losses. Entropy , 23(10):1330, 2021.

J. Ho, C. Saharia, W. Chan, D. J. Fleet, M. Norouzi, and T. Salimans. Cascaded Diffusion Models for High Fidelity Image Generation. J. Machine Learning Research , 23:1-33, 2022.

M. I. Jordan, T. S. Ghahramani, Z. abd Jaakkola, and L. K. Saul. An Introduction to Variational Methods for Graphical Models. Machine Learning , 37:183-233, 1999.

Herbert Knothe. Contributions to the theory of convex bodies. Michigan Mathematical Journal , 4(1):39-52, 1957.

I. Kobyzev, S.J. D. Prince, and M. A. Brubaker. Normalizing Flows: An Introduction and Review of Current Methods. IEEE Trans. Pattern Anal. Mach. Intell. , 43(11):3964-3979, 2021.

D. Koller and N. Friedman. Probabilistic Graphical Models: Principles and Techniques . MIT Press, 2009.

Aryeh Kontorovich and Maxim Raginsky. Concentration of Measure Without Independence: A Unified Approach Via the Martingale Method. In Eric Carlen, Mokshay Madiman, and Elisabeth M. Werner, editors, Convexity and Concentration , volume 161, pages 183-210. Springer New York, New York, NY, 2017. Series Title: The IMA Volumes in Mathematics and its Applications.

Leonid (Aryeh) Kontorovich and Kavita Ramanan. Concentration inequalities for dependent random variables via the martingale method. The Annals of Probability , 36(6):2126-2158, 2008.

M. Ledoux. The Concerntration of Measure Phenomenon . Amer. Math. Soc., 2001.

Ben London, Bert Huang, and Lise Getoor. Stability and generalization in structured prediction. Journal of Machine Learning Research , 17(221):1-52, 2016.

Ben London, Bert Huang, Ben Taskar, and Lise Getoor. Collective stability in structured prediction: Generalization from one example. In International Conference on Machine Learning , pages 828-836. PMLR, 2013.

Ben London, Bert Huang, Ben Taskar, and Lise Getoor. Pac-bayesian collective stability. In Artificial Intelligence and Statistics , pages 585-594. PMLR, 2014.

Youssef Marzouk, Tarek Moselhy, Matthew Parno, and Alessio Spantini. Sampling via Measure Transport: An Introduction. In Roger Ghanem, David Higdon, and Houman Owhadi, editors, Handbook of Uncertainty Quantification , pages 1-41. Springer International Publishing, Cham, 2016.

David A McAllester. Pac-bayesian model averaging. In Proceedings of the twelfth annual conference on Computational learning theory , pages 164-170, 1999.

David A. McAllester. Some pac-bayesian theorems. Machine Learning , 37(3):355-363, Dec 1999.

Waleed Mustafa, Yunwen Lei, Antoine Ledent, and Marius Kloft. Fine-grained analysis of structured output prediction. International Joint Conferences on Artificial Intelligence, 2021.

Stanley Osher and Ronald Fedkiw. Level set methods and dynamic implicit surfaces . Springer, 2003.

G. Papamakarios, E. Nalisnick, D.J. Rezende, S. Mohamed, and B. Lakshminarayanan. Normalizing Flows for Probabilistic Modeling and Inference. J. Machine Learning Research , 22(57):1-64, 2021.

Maria Perez-Ortiz, Omar Rivasplata, Benjamin Guedj, Matthew Gleeson, Jingyu Zhang, John Shawe-Taylor, Miroslaw Bober, and Josef Kittler. Learning pac-bayes priors for probabilistic neural networks. arXiv preprint arXiv:2109.10304 , 2021.

Marıa Pérez-Ortiz, Omar Rivasplata, John Shawe-Taylor, and Csaba Szepesvári. Tighter Risk Certificates for Neural Networks. Journal of Machine Learning Research , 22(227):1-40, 2021.

Murray Rosenblatt. Remarks on a multivariate transformation. The annals of mathematical statistics , 23(3):470-472, 1952.

L. Ruthotto and E. Haber. An Introduction to Deep Generative Modeling. GAMMMitt. , 44(2):24 pages, 2021.

John Shawe-Taylor and Robert C Williamson. A pac analysis of a bayesian estimator. In Proceedings of the tenth annual conference on Computational learning theory , pages 2-9, 1997.

Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole. Score-Based Generative Modeling Through Stochastic Differential Equations. In ICLR , 2021.

Esteban G Tabak and Cristina V Turner. A family of nonparametric density estimation algorithms. Communications on Pure and Applied Mathematics , 66(2):145-164, 2013.

Esteban G Tabak and Eric Vanden-Eijnden. Density estimation by dual ascent of the loglikelihood. Communications in Mathematical Sciences , 8(1):217-233, 2010.

Vera Trajkovska, Paul Swoboda, Freddie Åström, and Stefania Petra. Graphical model parameter learning by inverse linear programming. In François Lauze, Yiqiu Dong, and Anders Bjorholm Dahl, editors, Scale Space and Variational Methods in Computer Vision , pages 323-334, Cham, 2017. Springer International Publishing.

Brian L. Trippe and Richard E. Turner. Conditional Density Estimation with Bayesian Normalising Flows, February 2018. arXiv:1802.04908 [stat].

M.J. Wainwright and M.I. Jordan. Graphical Models, Exponential Families, and Variational Inference. Found. Trends Mach. Learn. , 1(1-2):1-305, 2008.

A Full Proofs of Presented Results

In this appendix, we present full proofs of all results which are not already complete in the main text.

Proof of Theorem 4 For any i ∈ [ d ] and z ∈ Z d define

with the edge case

Due to E Z ∼ µ [ f ( Z ) |B c , Z = z ] = f ( z ) for z ∈ B c we have

Since the conditions Z [ i ] = z [ i ] generate a nested sequence of σ -algebras, the quantities K ( i +1) f ( z ) = E µ [ f ( Z ) |B c , Z [ i ] = z [ i ] ] are a Doob martingale and (26) is a martingale difference sequence. In order to bound the moment generating function of f , we will bound every M ( i ) from above and below and apply the Azuma-Hoeffding theorem 10. We have

by the tower property of conditional expectations. Because µ ( dw [ i,d ] | z [ i -1] , B c ) is a probability measure, it holds

and we find

Now bound A ( i ) ≤ M ( i ) ≤ B ( i ) almost surely with

where B c i ( z [ i -1] ) contains all z i ∈ Z such that there exist z ( i,d ] ∈ Z d -i with ( z [ i -1] , z i , z ( i,d ] ) ∈ B c . Because every realization of a random variable conditioned on B c is in the set of good inputs, the difference ∥ B ( i ) -A ( i ) ∥ ∞ can be written as

By seeing this expression in terms of oscillation of the kernel action K ( i +1) f , we find

where ˜ f : B c → R is the restriction of f to B c . The assertion then follows from the Azuma-Hoeffding theorem [35, Theorem 4.1] which we recite as Theorem 10 to make this paper self-contained.

Proof of Proposition 6 For arbitrary z, z ′ ∈ Z d it holds

and thus, by summing over all indices we get

Let v, z ∈ Z d with v [ d ] \{ i } = z [ d ] \{ i } be given for some i ∈ [ d ] . Recall the action (8) of Markov kernels K ( i +1) is an expected value with respect to conditional distributions µ ( i,d ] ( dw ( i,d ] | v [ i ] ) .

Because ν d has no atoms, ν d |A c also has no atoms. Therefore, there is a unique KR-rearrangement ̂ T with ̂ T ♯ ν d = ν d |A c . Then ˜ T = T ◦ ̂ T is a KR-rearrangement with

by Lemma 9 and we have ˜ T ( ̂ v ) = v . Lemma 3 implies

An analogous expression holds for the distribution conditioned on z . We have therefore found two transport functions pushing the reference measure to the respective conditional distributions. By Lemma 5, a coupling of the conditional distributions is then given by

Using a change of measure we find

which shows

for good inputs. We have thus found a Wasserstein matrix V ( i +1) for K ( i +1) with entries

in row i which shows the assertion.

Proof of Theorem 7 For any hypothesis h ∈ H , we have

where in (46c) we have used that pointwise loss is in [0 , 1] . Note that the underlying distribution of the risk R ( h ) is µ , while D m are drawn from µ |B c . The above inequality reconciles this such that a concentration argument for the conditional distribution becomes applicable. For any PAC-Bayes posterior distribution ζ and any β > 0 , this implies

by Donsker and Varadhan's variational formula [2, Lemma 2.2]. Focusing on the first term, we find exp ( β ( E Z ∼ µ |B c [ L ( h, Z )] -R m ( h, D m )) ) = exp ( β m ∑ k ∈ [ m ] ( E Z ∼ µ |B c [ L ( h, Z )] -L ( h, Z ( k ) ) ) ) (48a)

Each structured datum Z ( k ) is drawn independently from µ |B c . By Proposition 6 there exists a Wasserstein dependency matrix Γ = ∥ ρ ∥ d D for µ |B c where D has entries (17). Then

Denote the shorthand

By Markov's inequality it holds

and combining this with (49) we have

with probability at least 1 -δ over the sample. Using (47) we thus have

Ideally, we would minimize the right hand side with respect to β . However, this would mean to have β depend on ζ and we thus would not have a uniform bound for all posterior distributions.

Instead, [38] approaches the problem by defining a sequence of constant ( δ j , β j ) j ∈ N 0 and bounding the probability that the bound does not hold for any sequence element. Since in the opposite (highprobability) case, the bound holds for all sequence elements, an optimal one can subsequently be chosen dependent on the posterior.

For all j ∈ N 0 , define

which are independent of ζ . Now consider the event E j that

By the above argument leading up to (52), the probability for E j under a random sample of the conditioned data distribution µ |B c is at most δ j . Therefore, the probability that any E j occurs is bounded by

Thus, for all posteriors ζ with probability at least 1 -δ none of the events (55) occurs. We may therefore select an index j dependent on ζ to obtain a sharper risk certificate which still holds with probability at least 1 -δ over the sample conditioned on the good set. For a fixed posterior ζ , the optimizer of (53b) would be

Equating this to (55) and rounding down to the nearest integer gives

Denote this number before rounding by r , i.e. j ∗ = ⌊ r ⌋ . For any real number r it holds r -1 ≤ ⌊ r ⌋ ≤ r . Therefore

which gives the following bounds on u j ∗

Likewise, we bound

The assumption δ ≤ exp( -e -1 ) yields -log log 1 δ ≤ 1 and because x +1 ≤ exp( x ) for all x ∈ R , we find

We can now use the bounds (62c) and (60) in (53b) to bound the expected generalization error

Note that β ∗ would attain the optimal value

which only differs from the above uniform bound by a factor of two. Finally, recall Γ = ∥ ρ ∥ d D where D has entries (17).

B Additional Lemmata

Lemma 9. Let T : Ω → Ω be a measurable function on a measurable space (Ω , Σ) and let ν, µ be measures on Ω with T ♯ ν = µ . Let B ∈ Σ be a fixed set with µ ( B ) > 0 and A = T -1 ( B ) its preimage under T . Then

Proof. Let S ∈ Σ be arbitrary and let ˜ µ = T ♯ ( ν | A ) . Then

as well as

Note that z ∈ T -1 ( S ) ∩ T -1 ( B ) ⇔ T ( z ) ∈ S ∧ T ( z ) ∈ B ⇔ T ( z ) ∈ S ∩ B ⇔ z ∈ T -1 ( S ∩ B ) (68) thus T -1 ( S ) ∩ T -1 ( B ) = T -1 ( S ∩ B ) and consequently ˜ µ ( S ) = ( µ | B )( S ) . Since S was arbitrary, this shows the assertion.

The following theorem exists in various forms in the literature. To make this paper self-contained, we recite the version in [35] which is used to bound moment-generating functions in Proposition 4. Note that we only use the MGF bound (69) in our analysis. However, the concentration inequality (70) also holds analogously under the assumptions of Proposition 4 which may be of independent interest.

Theorem 10 ( Azuma-Hoeffding [35, Theorem 4.1] ) . Let ( M ( i ) ) i ∈ [ m ] be a martingale difference sequence with respect to a filtration (Σ i ) i ∈ [ m ] of sigma algebras. Suppose that for each i ∈ [ m ] there exist Σ i -1 -measurable random variables A ( i ) , B ( i ) such that A ( i ) ≤ M ( i ) ≤ B ( i ) almost surely. Then for all λ ∈ R it holds that

and consequently, for any t ≥ 0

C Numerical Toy Example

0

.

4

0

.

6

0

.

8

1

.

0

largest

L

12

0

.

25

0

.

50

0

.

75

1

.

00

bad set prob.

ξ

0

.

0

0

.

5

1

.

0

□□

To illustrate the concept of transport stability discussed in Proposition 6, as well as the bad set construction of Section 5, we devised a toy example of two-dimensional transport. As a reference distribution, we choose the uniform distribution on the unit cube [0 , 1] 2 , i.e. Z = [0 , 1] . The metric ρ d , d = 2 is chosen as the ℓ 1 -distance ρ 2 ( z, z ′ ) = ∥ z -z ′ ∥ 1 = | z 1 -z ′ 1 | + | z 2 -z ′ 2 | . The KR transport map has component functions T ( z 1 , z 2 ) = ( T 1 ( z 1 ) , T 2 ( z 1 , z 2 ) ) . We make a polynomial ansatz, defining

where B i m denotes the i -th Bernstein polynomial of degree m , β i ≥ 0 are parameters and (positive) coefficient functions ˜ β i ( z 1 ) are again a parameterized combination of Bernstein polynomials (degree 8). Because Bernstein polynomials assume non-negative values on [0 , 1] , the components in (71) define a valid KR-rearrangement. The resulting measure transport is illustrated in Figure 1. In order to evaluate the bound of Theorem 7, we need to compute the Lipschitz constant L 12 according to equation (16)

for all good inputs z 1 , z ′ 1 . Figure 2 (left) shows the values of L 12 satisfying (72) for each input ( z 1 , z ′ 1 ) ∈ [0 , 1] 2 . The sought Lipschitz constant is the largest of these values. In order to avoid regions with large values, Theorem 7 allows for the exclusion of bad inputs . The probability ξ of this bad set under the data distribution is incurred as a penalty in equation (20). We thus construct bad sets to exclude large values and consider their probability under the data distribution Figure 2 (middle). Finally, the exclusion of such a bad set is shown on the right of Figure 2.

Peer reviews:
comments: SUMMARY:
This paper proposes a PAC-Bayesian risk bound for the task of structured prediction. Under the assumption that the data is generated by  Knothe-Rosenblatt rearrangement, this method distills random output variables into a Wasserstein dependency matrix, which paves the way for improved generalization bounds of generative models.

STRENGTHS:
1. The core method is based on the triangular measure transport and KR rearrangement of a tractable reference measure, which is novel and allows for flexible distillation. 

2. The set of bad inputs is bounded so that the risk is more certificated.

3. The presented approach has the potential to compute risk certificates for many downstream discriminative tasks.

WEAKNESSES:
1. My main concern is that the samples are drawn from a distribution arising from a reference measure through KR rearrangement. Can authors do some toy experiments of some popular generative models to support this assumption?

QUESTIONS:
Please see the weaknesses and limitations

LIMITATIONS:
I had a rough check on the derivation and proofs, and they seem correct. My only concern is whether the assumption is valid in practice.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-19

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper establishes a new PAC-Bayesian risk bound for the structured prediction problem. Technically, it assumes the data are generated by the Knothe-Rosenblatt rearrangement of a factorizing reference measure, and then obtains generalization bounds with the Wasserstein dependency matrix.\n\nSTRENGTHS:\n1. Different from the existing PAC-Bayesian bound with ϑ-mixing dependency matrix, this paper uses the Wasserstein dependency matrix to measure the interdependence of the data. Built upon the Wasserstein dependency matrix, novel concentration inequalities and general PAC-Bayesian risk bounds are established.\n\n2. The Knothe-Rosenblatt rearrangement adopted in this paper is related to the measured transport and generative models, which may bring some insights into this field.\n\nWEAKNESSES:\n1. The discussion on the existing work [1*] should be clarified. The authors argue that [1*] requires that data are generated by a Markov random field. However, their general PAC-Bayesian risk bounds (Section 5 in [1*]) do not need this assumption. The ϑ-mixing dependency matrix adopted by them exists for any distribution. [1*] only choose Markov random fields as special examples to concretely clarify the implications of their theoretical framework. Thus, PAC-Bayesian risk bounds in this paper may not be more general than that in [1*].\n\n2. There does not exist any theoretical example, simulation experiment, or real-world experiment in this paper. To show the superiority of the proposed framework, the authors should at least consider adding some concrete theoretical examples (e.g. Markov random field) in this paper.\n\n3. The paper is not written very clearly. More descriptions and discussions can be added for the technical notions and definitions. For example, when I read this paper for the first time, some notions (e.g., Equation 7) confuse me, and I can not get intuitions of some definitions (e.g., Wasserstein dependency matrix).\n\nI will consider improving the score after you discuss my concerns.\n\n[1*] Ben London, Bert Huang, and Lise Getoor. Stability and generalization in structured prediction. JMLR 2016.\n\nQUESTIONS:\nPlease see the weaknesses comments.\n\nLIMITATIONS:\nNone.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work derives a novel PAC-Baeysian risk bound for structured prediction based on generative models, a triangular and monotone transport and Wasserstein dependency matrices.\n\nSTRENGTHS:\nThis is technical a paper with rigorous theoretical analysis. The flow is easy to follow. The authors have made very detailed comparisons to [1], on which this work is largely based. Many examples such as image data are included to help readers understand some of the intuitions behind the derived theoretical results.\n\n\n[1] London, Ben, Bert Huang, and Lise Getoor. \"Stability and generalization in structured prediction.\" The Journal of Machine Learning Research 17, no. 1 (2016): 7808-7859.\n\nWEAKNESSES:\nThere are no empirical results to showcase the tightness or usefulness of the risk bounds compared to bounds that ignore the size of the structure object. The assumptions are probably limited. See my comments on these points below.\n\nQUESTIONS:\nI have a few minor concerns as follows:\n\n1. In line 134, it is assumed that the structured loss is an additive, bounded and pointwise loss. This looks like the conclusion in this paper can only be generalized to affine losses. This assumption does not hold for, e.g., log loss, which goes to infinity, or F1 score, which is non-decomposable into components. So what family of losses can the risk bound generalize to?\n\n2. The successful derivation of the risk bound in Theorem 7 is greatly thanks to, I believe, the introduction of the bad set. I appreciate the detailed explanations on the intuitions for this set. But in practice, how do you define bad and how can you identify those data points? As the authors suggest by themselves, one could use Equation 16 to decide on this but the Lipschitz constants $L_{ij}$ may not be easy to compute.\n\n3. Does the learned measure transport distribution converge to the true underlying distribution asymptotically? If not, how close are they?\n\nLIMITATIONS:\nYes, the authors have adequately discussed the limitations of their findings.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper develops a PAC-Bayesian bound on the risk of structured predictors which decreases with the number and size of examples. The work builds on concentration of measure results (e.g., [33]) and continues the line of work in [36] by removing the assumption that data are generated by a Markov Random Field (MRF). Instead, the present work assumes a triangular and monotone transport, a Knothe-Rosenblatt (KR) rearrangement of a reference measure, as the data model.\n\nSTRENGTHS:\n**S1.** This theoretical work advances a series of works on bounding the risk of structured predictors via more recent concentration of measure results.\n\n**S2.** The submission does a great job at providing an overview of prior results and presenting new developments.\n\nWEAKNESSES:\n**W1.** The theoretical results in this submission are presented with little connection to implications to practice. For example, L341-344 state that how closely a measure transport distribution learned from data approximates the actual (unknown) distribution of the data is an open question and thus, no empirical results are provided. I wonder if more could be said in this regard.\n\nAs another example, in [36] bounds are applied to specific models. On one hand, the specifics of the model and training loss are taken into account. On the other hand, the PAC-Bayes bound is derandomized in order to apply it to a learned predictor. Are similar applications not possible for the results in the present submission?\n\n**W2.** One of the claims is that the present work “makes a preliminary step towards leveraging powerful generative models to establish generalization bounds for discriminative downstream tasks.” In my view, providing further insight into next steps in this direction would help other researchers build upon this work.\n\nQUESTIONS:\n**Q1.** How are the theoretical results in this submission (perhaps given further development) to inform practice?\n\n**Q2.** In general, what would be possible progressions of developments building on this work?\n\n**Q3.** How does the restriction to atom-free data distributions limit relevance and applicability of the results in this submission, e.g., to discrete output spaces?\n\nLIMITATIONS:\nPotential negative societal impact was not discussed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies PAC-Bayesian risk bound for structured prediction using concentration of measure. Specifically, authors characterize stability and dependency with Wasserstein dependency matrix under the data assumption that data are given by Knothe-Rosenblatt (KR) rearrangement of a factorizing reference measure. PAC-Bayesian risk bound for structured prediction has been studied in the previous studies [1], [2] with different stability notions and the main contribution of this paper is to apply measure-theoretic stability notions from [3] to the problem and provide a new PAC Bayesian bound that scales with the number of examples and their dimensions. This result agrees with the PAC risk bound provided in [2] under specific conditions, giving a more generalized interpretation based on Wasserstein dependency matrix.\n\n[1] London, Ben, et al. \"Collective stability in structured prediction: Generalization from one example.\" International Conference on Machine Learning. PMLR, 2013.\n\n[2] London, Ben, Bert Huang, and Lise Getoor. \"Stability and generalization in structured prediction.\" The Journal of Machine Learning Research 17.1 (2016): 7808-7859.\n\n[3] Kontorovich, Aryeh, and Maxim Raginsky. \"Concentration of measure without independence: a unified approach via the martingale method.\" Convexity and Concentration. Springer New York, 2017.\n\nSTRENGTHS:\n1. New stability notion is considered for PAC bayes risk bound, providing additional analysis and interpretations up on previous works.\n2. Provided examples and intuitions are helpful for readers to follow the manuscript.\n3. KR rearrangement assumption on data is not that unrealistic, considering that generative models using KR rearrangement assumption have been studies recently, e.g. [4]. This entails the possibility of practical implications.\n\n\n[4] Irons, Nicholas J., et al. \"Triangular flows for generative modeling: Statistical consistency, smoothness classes, and fast rates.\" International Conference on Artificial Intelligence and Statistics. PMLR, 2022.\n\nWEAKNESSES:\n1. Definitions, notations, some lemma & theorem look pretty similar to ones in [3] --- authors may want to highlight the contribution of this paper by differentiating from [3] or giving more explanation.\n2. More background on PAC Bayes risk bounds could be helpful for readers. Especially, it would be nicer if how stability and dependency affects PAC Bayes risk bounds can be discussed.\n\nQUESTIONS:\n1) If measure space is discrete, can it be reduced to previously known PAC Bayes bounds? I am curious how measure-theoretic characterization can be connected with the previous results.\n2) What's the intuition about \"a bad set\" --- is it the set possibly locally unstable?\n3) Is there any possible simulation setup we can see the provided PAC Bayes bound works?\n4) L151: What is $\\delta_x$? --- confusion with $\\delta_i$. $\\delta$ is also overloaded in the probability $1-\\delta$, authors may want to change the notation to reduce confusion.\n\nLIMITATIONS:\nLimitations are properly discussed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-25"}