Achieving Domain-Independent Certified Robustness via Knowledge Continuity

Authors: alan  sun
chiyu  ma
kenneth  ge
soroush  vosoughi
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 22:39:28

Abstract: We present *knowledge continuity*, a novel definition inspired by Lipschitz continuity which aims to certify the robustness of neural networks across input domains (such as continuous and discrete domains in vision and language, respectively). Most existing approaches that seek to certify robustness, especially Lipschitz continuity, lie within the continuous domain with norm and distribution-dependent guarantees. In contrast, our proposed definition yields certification guarantees that depend only on the loss function and the intermediate learned metric spaces of the neural network. These bounds are independent of domain modality, norms, and distribution. We further demonstrate that the expressiveness of a model class is not at odds with its knowledge continuity. This implies that achieving robustness by maximizing knowledge continuity should not theoretically hinder inferential performance. Finally, to complement our theoretical results, we present several applications of knowledge continuity such as regularization, a certification algorithm, and show that knowledge continuity can be used to localize vulnerable components of a neural network.

Main paper: 
 introduction: 
 Deep neural networks (DNNs) have demonstrated remarkable generalization capabilities. Their robustness, however, has been considerably more difficult to achieve. Robustness refers to the preservation of model performance under natural or adversarial alterations of the input [18] . DNNs' lack of robustness, highlighted by seminal works such as [24, 66] and recently [7, 5] , poses significant challenges to their adoption in critical applications, underscoring concerns for AI safety and trustworthiness [20, 30, 9, 7] .Though issues of robustness emerged from computer vision applications, they have since spanned multiple domains [1, 35, 72, 75, 7] . This research trajectory has not only prompted significant advancements in robustness improvements through architectural, training, and dataset augmentations, but also unveiled the sophistication of adversarial attacks-the process through which counterexamples to robustness are generated [1, 35, 72, 75, 7] . Along the progress made in these parallel directions, a great deal of work has gone into certified robustness which seeks to provide theoretical robustness guarantees. Certification is desirable as it generally transcends any particular task, dataset, or model. As a result, Lipschitz continuity has emerged, promising certified robustness by essentially bounding the derivative of a neural network's output with respect to its input. In this way, Lipschitz continuity directly captures the volatility of a model's performance, getting at the heart of robustness. Such an approach has proven its merit in computer vision, facilitating robustness under norm and distributional assumptions [29, 59, 78, 76] . Its inherent ease and interpretability has lead to widespread adoption as a means to measure and regulate robustness among practitioners as well [71, 12, 21, 68, 54] . Despite these successes in computer vision, there are fundamental obstacles when one tries to apply Lipschitz continuity into discrete or non-metrizable domains such as natural language. Firstly, characterizing distance in this input-output space is highly nontrivial, as language does not have a naturally-endowed distance metric. Additionally, suppose we impose some distance metric on the input-output space [49, 16] . For such a metric to meaningfully characterize adversarial perturbations, it cannot be universally task-invariant. Consider the two sentences (a) "I am happy," (b) "I am sad." The ground-truth label of (a) is invariant to the perturbation (a) → (b), if the task is sentence-structure identification, but it would not be preserved for a task like sentiment classification. Lastly, key architectures such as the Transformer [70] are provably not Lipschitz continuous [36] . Most of these challenges are not unique to language, and they represent a strong divide of our understanding of robustness in discrete/non-metrizable and continuous domains [22, 46] .To address these issues, we propose a new conceptual framework which we call knowledge continuity. At its core, we adopt the following axiom:Robustness is the stability of a model's performance with respect to its perceived knowledge of input-output relations.Concretely, our framework is grounded on the premise that robustness is better achieved by focusing on the variability of a model's loss with respect to its hidden representations, rather than forcing arbitrary metrics on its inputs and outputs. Our approach results in certification guarantees independent of domain modality, norms, and distribution. We demonstrate that the expressiveness of a model class is not at odds with its knowledge continuity. In other words, achieving robustness by improving knowledge continuity should not theoretically hinder inferential performance. We show that in continuous settings (i.e. computer vision) knowledge continuity generalizes Lipschitz continuity and inherits its tight robustness bounds. Finally, we present an array of practical applications using knowledge continuity both as an indicator to predict and characterize robustness as well as an additional term in the loss function to train robust classifiers. In sum, our contributions are threefold:• Introduction of knowledge continuity, a new concept that frames robustness as variability of a model's loss with respect to its hidden representations. • We theoretically show that knowledge continuity results in certified robustness guarantees that generalize across modalities (continuous, discrete, and non-metrizable). Moreover, this robustness does not come at the expense of inferential performance. • We present several practical applications of knowledge continuity such as using it train more robust models, in both language processing and vision, identify problematic hidden layers, and using its theoretical guarantees to formulate a novel certification algorithm.Although our results apply to all discrete/non-metrizable and continuous spaces, throughout the paper we invoke examples from natural language as it culminates the aforementioned challenges. Further, the ubiquity of large language models make their robustness a timely focus.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes]Justification:Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related works: 
 There have been extensive studies on developing robust neural networks with theoretical guarantees. With respect to our contributions, they can be organized into the following categories.Certified robustness with Lipschitz continuity. The exploration of Lipschitz continuity as a cornerstone for improving model robustness has yielded significant insights, particularly in the domain of computer vision. This principle, which ensures bounded derivatives of the model's output with respect to its input, facilitates a smoother model behavior and inherently encourages robustness against adversarial perturbations. This methodology, initially suggested by [24] , has since been rigorously analyzed and expanded upon. Most theoretical results in this area focus on certifying robustness with respect to the 𝓁 2 -norm [11, 86, 25, 2, 38, 29, 4] . A recent push, fueled by new architectural developments, has also expanded these results into 𝓁 ∞ -norm perturbations [89, 88, 90] . Further, Lipschitz continuity-inspired algorithms also serve practitioners as a computationally effective way to train more robust models [68, 78, 69, 13] . This stands in contrast to (virtual) adversarial training methods which brute-force the set of adversarial examples, then iteratively retrain on them [50, 63, 80] .Though Lipschitz continuity has seen much success in continuous domains, it does not apply to nonmetrizable domains such as language. Further, architectural limitations of prevalent models such as Bounding this volatility in expectation then directly leads to our notion of knowledge continuity. With these tools, we demonstrate a host of theoretical properties of knowledge continuity including its certification of robustness, guarantees of expressiveness, and connections to Lipschitz continuity in continuous settings. We summarize our theoretical contributions as follows:• We define the perceived knowledge of a model as well as volatility and knowledge continuity within a model's representation space (see Def. 1, 2, 3, 4, respectively). • We prove that knowledge continuity implies probabilistic certified robustness under perturbations in the representation space and constraining knowledge continuity should not hinder the expressiveness of the class of neural networks (see Thm. 4.1 and Prop. 4.3, 4.4, respectively). • We prove that in some cases knowledge continuity is equivalent (in expectation) to Lipschitz continuity. This shows that our axiomization of robustness aligns with existing results when perturbation with respect to the input is well-defined (see Prop. 4.6, 4.8).Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: We present a detailed discussion of the limitations in Section K.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
theory assumptions and proofs: 
 Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [Yes]Justification: For all of the theoretical results in the paper, we include all of its assumptions. We include full proofs of each theoretical result in Appendices A, B, C, G. To the best of our knowledge, the proofs are correct.Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.• All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
defining perceived knowledge: 
 Knowledge is generally understood as a relational concept: it arises from the connections we make between ideas, experiences, and stimuli [26] . Herein, we capture the perceived knowledge of a model by focusing on the relations it assigns to input-input pairs. Specifically, these relations are exposed by decomposing a function 𝑓 ∶  →  into projections to intermediate metric spaces. Formally, Definition 1 (Metric Decomposition). We say that 𝑓 admits a metric decomposition if there exists metric spaces ( ∶  →  1 , ℎ 𝑘 ∶  𝑘 →  𝑘+1 for 𝑘 ∈ [𝑛 -1], and ℎ 𝑛 ∶  𝑛 → . 3. 𝑓 = ℎ 𝑛 •ℎ 𝑛-1 • … •ℎ 1 •ℎ 0 . Remark 1.If  is a metric space with metric 𝑑  and   is its Borel 𝜎-algebra, then for any measurable mapping 𝑓 ∶  →  there exists the trivial metric decomposition 𝑓 = 𝑓 •Id  . (4.1) Therefore, in computer vision applications where (, 𝑑  ) = (ℝ 𝑛 , 𝓁 𝑝 ) for some 𝑛 ∈ ℤ + , we can apply this trivial decomposition to yield bounds which mirror the certification guarantees of Lipschitz continuity. This is discussed in detail in Section 4.5.To the best of our knowledge, all deep learning architectures admit metric decompositions, since their activations are generally real-valued. So, for all subsequent functions from  to , unless otherwise specified, we assume they are measurable and possess a metric decomposition. Further, we denote 𝑓 𝑘 = ℎ 𝑘 •ℎ 𝑘-1 • … •ℎ 1 •ℎ 0 and adopt the convention of calling ℎ 𝑘 the 𝑘 th hidden layer. In Appendix A, we present several metric decompositions for a variety of architectures.For any metric-decomposible function, an immediate consequence of our definition is that its metric decomposition may not be unique. However, in the context of neural networks, this is a desirable property. Seminal works from an array of deep learning subfields such as semi-supervised learning [57] , manifold learning [51] , and interpretability [10, 47] place great emphasis on the quality of learned representation spaces by examining the induced-topology of their metrics. This often does not affect the typical performance of the estimator, but has strong robustness implications [33] . Our results, which are dependent on particular metric decompositions, capture this trend. In Section 4.4, we discuss in detail the effects of various metric decompositions on our theoretical results.We first introduce what it means for a model's performance to be volatile at a data point relative to its metric decomposition. Then, we contrast knowledge continuity with Lipschitz continuity, pointing out key differences that will allow us to prove more general bounds. Definition 2 (𝑘-Volatility). Let 𝑓 ∶  →  and  be any loss function. The 𝑘-volatility of a point (𝑥, 𝑦) ∈  ×  which we denote as 𝜎 𝑘 𝑓 (𝑥, 𝑦) is given by Notice that the classification of points is independent of input/output clustering behavior since ,  may not be endowed with a metric.EQUATIONwhere 𝑑 𝑘 is the distance metric associated with 𝑓 's 𝑘 th hidden layer.By performing some algebra on the definition, we see that it decomposes nicely into two distinct terms: sparsity of the representation and variation in loss.EQUATIONOur notion of volatility essentially measures the change in performance with respect to perturbations to a model's perceived knowledge. In particular, Eq. 4.3 reveals that there are two interactions in play which we illustrate in Fig. 1 . Informally, we say that (𝑥, 𝑦) is highly volatile if there is a large discrepancy in performance between it and points that are perceived to be conceptually similar. Therefore, highly volatile points capture inaccurate input-input knowledge relations. Additionally, (𝑥, 𝑦) experiences low volatility if the space around it is sparse with respect to  , . In other words, any set of perturbations applied in  𝑘 would push (𝑥, 𝑦) far away, with high probability. This makes (𝑥, 𝑦) an isolated concept with little knowledge relationships associated with it.Similar to Lipschitz continuity, the boundedness of the 𝑘-volatility of 𝑓 across the data distribution is crucial and we denote this class of functions as knowledge continuous. Though the functional forms of Lipschitz continuity and knowledge continuity are similar, there are important differences that allow us to prove more general results. Firstly, unlike Lipschitz continuity which is an analytical property of the model 𝑓 , knowledge continuity is a statistical one. In this way, non-typical data points, even if they are volatile, are ignored, whereas Lipschitz continuity treats all points equally. This is necessary in many discrete applications, as projecting a countable input space onto a non-countable metric space inevitably results in a lack of correspondence thereof. Moreover, ground-truth relations from  →  may not be well-defined on all of : consider sentiment classification of an alpha-numeric UUID string or dog-cat classification of Gaussian noise. Secondly, the knowledge continuity of an estimator is measured with respect to the loss function rather than its output. This property allows us to achieve the expressiveness guarantees in Section 4.4, since it places no restrictions on the function class of estimators. Lastly, knowledge continuity measures the distance between inputs with the endowed metric in its hidden layers. This flexibility allows us to define knowledge continuity even when the input domain is not a metric space.Our first main result demonstrates that 𝜖-knowledge continuity implies probabilistic certified robustness in the hidden representation space. In Theorem 4.1, given some reference set 𝐴 ⊂  × , we bound the probability that a 𝛿-sized perturbation in the representation space away from 𝐴 will result in an expected 𝜂 change in loss. In other words, knowledge continuity is able to characterize the robustness of any subset of data points with positive measure.Theorem 4.1. Let 𝐴 ⊂  ×  such that ℙ  , [𝐴] > 0 and 𝛿, 𝜂 > 0. Let 𝐴 ′ = { (𝑥 ′ , 𝑦 ′ ) ∈  ×  ∶ 𝔼 (𝑥,𝑦)∼ , (𝑥,𝑦)∈𝐴 Δ (𝑥,𝑦) 𝑓 (𝑥 ′ , 𝑦 ′ ) > 𝜂 } . If 𝑓 ∶  →  is 𝜖-knowledge continuouswith respect to the hidden layer indexed by 𝑘 and ( 𝑘 , 𝑑 𝑘 ) is bounded by 𝐵 > 0, thenℙ (𝑥,𝑦)∼ , [𝐴 ′ | 𝑑 𝑘 (𝑓 𝑘 (𝑥), 𝑓 𝑘 (𝐴)) < 𝛿] ≤ 𝜖𝛿 𝜂 ( 1 -exp [ -Ω ( 𝛿 𝐵 - √ log 1 ℙ[𝐴]) 2 ]) .(4. ℙ (𝑥,𝑦)∼ , [𝐴 ′ | 𝑑 𝑘 (𝑓 𝑘 (𝑥), 𝑓 𝑘 (𝐴)) < 𝛿] ≤ 𝜖𝛿 𝜂(1 -ℙ[𝐴]) . (4.6) If ℙ[𝐴] = 0, then ℙ (𝑥,𝑦)∼ , [𝐴 ′ | 𝑑 𝑘 (𝑓 𝑘 (𝑥), 𝑓 𝑘 (𝐴)) < 𝛿] ≤ 𝜖𝛿 𝜂 . (4.7)Proof. These results follow from directly taking the limit as 𝐵 → ∞ and applying some of the bounds acquired in the proof of Thm. 4.1. This yields Eq. 4.6. Next, jointly taking the limit as ℙ[𝐴] → 0 and 𝐵 → ∞ results in Eq. 4.7.In both Thm. 4.1 and Cor. 4.7, we yield probabilistic guarantees like [12] , rather than deterministic ones. Though deterministic bounds are desirable, the stochasticity of our framework is necessary for its generalization across different domains. For most continuous, metrizable applications (like computer vision), models learn a hidden representation space where most minute changes in this space correspond to tangible inputs. The same cannot be said for many discrete or non-metrizable applications. In natural language processing, the correspondence between the learned representation space and the input is sparse, resulting in lots of "dead space": portions of the hidden representation space that do not correspond to any input [3, 19] . And so, by incorporating the data distribution into our bounds, we implicitly adjust for this: assigning zero-measure to the aforementioned "dead space."Our second main result demonstrates that 𝜖-knowledge continuity can be achieved without theoretically compromising the accuracy of the model. Proof. Choose 𝑓 𝑛 ∈  such that ∫ (𝑓 𝑛 (𝑥), 𝑦) 𝑑𝜇 𝑓 < 1 2 𝜖 𝑛 . Consider the 1-layer metric decomposition of 𝑓 , ℎ 1 ∶  →  1 where  1 =  equipped with the trivial metric (𝑑 1 (𝑥, 𝑦) = 1 if 𝑥 ≠ 𝑦 and 0 otherwise). Then, 𝑓 𝑛 = 𝑓 𝑛 •ℎ 1 . So, it follows that𝔼 𝜎 1 𝑓 𝑛 (𝑥, 𝑦) = ∫ Δ (𝑥,𝑦) 𝑓 𝑛 (𝑥 ′ , 𝑦 ′ ) 𝑑 1 (ℎ 1 (𝑥), ℎ 1 (𝑥 ′ )) 𝑑𝜇 𝑓 , ≤ ∫ Δ (𝑥,𝑦) 𝑓 𝑛 (𝑥 ′ , 𝑦 ′ ) 𝑑𝜇 𝑓 , ≤ 𝜖 𝑛 .and by the construction of 𝑓 𝑛 , the proof is completed.In other words, if our estimator was given "infinite representational capacity," robustness can be trivially achieved by isolating every point as its own concept (as discussed in Section 4.2). More generally, if we instead considered a generalized discrete metric (fix 𝑐 ∈ [0, ∞], 𝑑(𝑥, 𝑦) = 𝑐 if and only if 𝑥 = 𝑦 and 𝑑(𝑥, 𝑦) = 0, otherwise), then as 𝑐 → ∞, 𝑘-volatility converges pointwise to 0 almost everywhere assuming that the loss is finite almost everywhere. In practice, we find these degenerate decompositions to be unreasonable as they also trivialize robustness. For example, if 𝑐 = ∞, then robustness is not well-defined as any perturbation would lead to a point that is perceived to be infinitely far away. In this sense, our framework accounts for different notions of robustness, strong and weak. The next result builds on Prop. 4.3 and demonstrates how a stronger notion of robustness will affect expressiveness. These added constraints make it so that trivial metric decompositions are no longer possible unless the metric in  is also trivial. We state this formally below, note the highlighted differences between this and Prop. We now demonstrate that our axiomization of robustness presented in Section 1 aligns with the notion of robustness2 commonly prescribed in vision [18] . This unifies the certified robustness bounds with respect to the representation space derived in Thm. 4 𝔼 (𝑥,𝑦),(𝑥 ′ ,𝑦 ′ )∼ , 𝑑  (𝑓 (𝑥), 𝑓 (𝑥 ′ )) 𝑑  (𝑥, 𝑥 ′ ) ≤ 𝜖𝜂 𝑛 ∏ 𝑗=1 𝐾 𝑗 . (4.9)The proof is presented in Appendix D and follows easily through some algebriac manipulation. It is easy to see that if 𝑓 is knowledge continuous with respect to some identity (or contractive) metric decomposition, then we can loose the repeated product. Analogous to Remark 1, the concepts of Lipschitz continuity and knowledge continuity become similar when we can assign metrics to the input-output spaces. Next, combining this proposition with an auxiliary result from [89] , we directly yield a certification on the input space. Corollary 4.7. Suppose that assumptions of Prop. 4.6 are true. And also assume that(, 𝑑  ) = (ℝ 𝑛 , 𝓁 𝑝 ), (, 𝑑  ) = (ℝ 𝑚 , 𝓁 𝑝 ), for 1 ≤ 𝑝 ≤ ∞. Define a classifier from 𝑓 ∶ ℝ 𝑛 → ℝ 𝑚 , 𝑔, where 𝑔(𝑥) ∶= arg max 𝑘∈[𝑚] 𝑓 𝑘 (𝑥) for any 𝑥 ∈ ℝ 𝑛 . Then, with probability 1 -𝜖𝜂 𝑡 ∏ 𝑛 𝑗=1 𝐾 𝑗 , 𝑔(𝑥) = 𝑔(𝑥 + 𝛿) for all ‖𝛿‖ 𝑝 < (2 1∕𝑝 ∕2𝑡)margin(𝑓 (𝑥)) and 𝑡 > 0. 𝑓 𝑘 (𝑥) is the 𝑘 th coordinate of 𝑓 (𝑥)We present the proof in Appendix D. Our second result identifies conditions under which Lipschitz continuity, implies knowledge continuity. Proposition 4.8. Let (, 𝑑  ), (, 𝑑  ) be a metric spaces. Let 𝑓 ∶  →  be 𝜖-Lipschitz continuous and (𝑓 (𝑥), 𝑦) be 𝜂-Lipschitz continuous with respect to both coordinates. If the first 𝑛 metric decompositions of 𝑓 are 𝐾 𝑖 -Lipschitz continuous, then 𝑓 is knowledge continuous with respect to the 𝑛 th hidden layer. That is, [35] on a host of models [58, 57, 16, 44] and the IMDB [48] dataset regressed with the average of knowledge continuity coefficients across all hidden layers (𝑅 2 = 0.35). (b) 𝑘-Volatility as 𝑘 is varied across a model's relative depth. (c) Correlation between 𝑘-volatility and adversarial vulnerability (averaged across all models shown in (b)) with respect to TextFooler [35] as 𝑘 varies.EQUATIONWe detail the proof of this proposition in Appendix D. We note that in continuous applications such as computer vision, the assumptions of both propositions are generally met (i.e. our input-output spaces are metric spaces, all hidden layers are Lipschitz, and loss functions are locally Lipschitz). Furthermore, common architectures such as fully connected networks, CNNs, RNNs, and even vision transformers are Lipschitz continuous [71, 55] .Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: We present all of the hyperparameters in the experiments that require training in Appendix G. Additionally, our compute resources are detailed in Appendix J.Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully. (c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closedsource models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
practical applications: 
 In addition to the theoretical guarantees given by knowledge continuity in Section 4, we also demonstrate that knowledge continuity can be easily applied in practice. First, we find that knowledge continuity, similar to Lipschitz continuity, can be used to gauge adversarial robustness. Along these lines, our measure of volatility (see Def. 2) can be used to isolate particularly vulnerable hidden representations. These applications then directly motivate regulation of knowledge continuity as a means to enforce robustness.Unless otherwise specified, we run all of our experiments on the IMDB dataset [48] (a sentiment classification task) using a host of language models from different model families (encoder, decoder, encoder-decoder). We also present additional experiments on vision tasks. These experiments can be found in the Appendix G.Knowledge continuity can predict adversarial robustness. For a given model, 𝑓 , with 𝑛 hidden representations, choose some 𝑘 ∈ [𝑛]. Then, consider the hidden representation index by 𝑘. For this fixed 𝑘, we determine its 𝑘-volatility by directly estimating Def. 2 through a naive Monte-Carlo algorithm (see Appendix G for more details). Repeating this for all 𝑘 ∈ [𝑛], we yield a collection of 𝑘-volatilities which we denote as {𝜖 1 , … , 𝜖 𝑛 }, one for each hidden layer. When we regress a simple average of these coefficients, 𝑛 -1 ∑ 𝑛 𝑘=1 𝜖 𝑘 , with the empirical adversarial robustness (estimated using TextFooler [35] ), a strong correlation is observed. This is shown in Fig. 2(a) . In particular, knowledge continuity alone is able to explain 35% of the variance in adversarial attack success rate. When we combine 𝑘-volatility with other model properties like size, model family, even more variance can be explained (𝑅 2 = 0.48). Thus, knowledge continuity may be used as a computationally efficient method to estimate adversarial vulnerability with respect to the input space as compared to iteratively applying real adversarial attacks. Moreover, when the adversary is unknown a priori, knowledge continuity can also be used in this way as a diagnostic tool. A detailed discussion of these experiments are presented in Appendix E.We plot the relationship between the 𝑘-volatility, 𝜖 𝑘 , and the relative depth of the model (i.e. 𝑘∕𝑛). We find that language models belonging to different model families (encoder, decoder, encoder-decoder) admit different 𝑘volatility trajectories. This is shown in Fig. 2(b ). In this way, knowledge continuity may provide a more Table 1 : Comparison of our knowledge continuity algorithm to existing works across various model families and adversarial attack methods. TF, BA, ANLI denote adversarial attacks [35] , [40] , and [52] , respectively. Regulating knowledge continuity to improve robustness is superior across almost all tasks and attacks. nuanced picture of a model's inductive biases and robustness beyond a scalar value like "accuracy under adversarial attack." We present a detailed analysis of this in Appendix F. Further, these dynamics may act as a diagnostic tool and offer a starting point for designing model-specific robustness interventions or adversarial defenses. For example, when insights from Fig. 2 (b) are combined with a knowledge continuity regularization algorithm, this yields superior empirical robustness compared to existing methods. This is shown in the next subsection and in Appendix G. In addition, knowledge continuity can also quantitatively characterize an adversarial attack against a host of models which is useful for online or adaptive defenses [84, 64, 14] . This is shown in in Fig. 2 (c), where TextFooler [35] largely exploits the knowledge continuities in middle/final layers of the model to decrease performance.Regulating knowledge continuity. Motivated by the theoretical results in Section 4, we augment the loss function during training to mitigate knowledge continuity. Specifically, on each training iteration (batch), we start by choosing a hidden layer at random according to a Beta distribution determined a priori: 𝑋 ∼ Beta(𝛼, 𝛽) and let 𝑘 = ⌊𝑛𝑋⌋. Here, 𝛼, 𝛽 are chosen according to Fig. 2(b, c ). We assign larger sampling probability to layers where both 𝑘-volatility is high and where knowledge continuity is highly correlated with adversarial robustness. In this way, our regularization objective is both model and attack specific (if the attack method is unknown, then we only apply the former). Then, we devise a Monte-Carlo algorithm to estimate this layer's 𝑘-volatility, 𝜖 𝑘 , (see Appendix G) on this minibatch. And so, the augmented loss function becomes  ′ (𝑓 (𝑥), 𝑦) = (𝑓 (𝑥), 𝑦) + 𝜆𝜖 𝑘 with 𝜆 ≥ 0 as a hyperparameter, controlling the regularization strength. In contrast to existing adversarial training methods that perform inner-optimization steps [50, 43, 85] , our method requires only additional zeroth-order computations. As a result, it outperforms existing works in training speed (up to 2× for TextFooler [35] and 3× for ALUM [43] ), while improving robustness. We present a discussion of the results, ablation studies, and training details in Appendix G.Certifying robustness with knowledge continuity. We present an algorithm based on Thm. 4.1 to certify robustness during test-time. Similar to [12] , we estimate the probability of there existing an adversarial example within some fixed radius (in the representation space, according to a pre-defiend distance metric) through bootstrapping a one-side confidence interval. Applying these methods to our regularization results, we show that regularizing knowledge continuity increases the certified robustness. The certification algorithm, its proof of correctness, and certifications of our regularized models are presented in Appendix H.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes]Justification: We attach all of the code used to generate the figures and the experimental results in the supplementary materials.Guidelines:• The answer NA means that paper does not include experiments requiring code.• Please see the NeurIPS code and data submission guidelines (https://nips.cc/public/ guides/CodeSubmissionPolicy) for more details. • While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).• The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://nips.cc/public/ guides/CodeSubmissionPolicy) for more details. • The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc. • The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why. • At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable). • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
conclusion: 
 In this paper, we propose a novel definition, knowledge continuity, which addresses some of the key limitations of Lipschitz robustness. We demonstrate that our definition certifies robustness across domain modality, distribution, and norms. We also show that knowledge continuity, in contrast to Lipschitz continuity, does not affect the universal approximation property of neural networks. We also establish conditions under which knowledge continuity and Lipschitz continuity are equivalent. Lastly, we present several practical applications that directly benefit the practitioner. The broader impacts, reproducibility, and limitations of our work can be found in Appendix I, J, K, respectively. . It remains to define a metric on these hidden spaces. There are many ways of doing this. For example,• For any 1 ≤ 𝑝 ≤ ∞, endow each intermediate space with the 𝓁 𝑝 -norm.• Define 𝑑(𝑥, 𝑦) = 1 -cos(𝜃 𝑥,𝑦 ) where 𝜃 𝑥,𝑦 is the angle between 𝑥, 𝑦. Then, if we choose 𝜎 𝑖 to restrict the image of ℎ 𝑖 to be the unit sphere, we may endow each intermediate space with this cosine distance.Note here that there are two steps here: we first identify what the intermediate spaces are, then assign metrics to them. The process of identfying these intermediate spaces may be independent of the metrics we end of assigning them.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: For the experiments that require training, we discuss in detail the hyperparameters in Appendix G. Moreover, we also attach the code used to generate all results and figures in the supplementary materials of the submission.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: The experiments in our paper serve as a type of sanity check and demonstrate possible explorations rather than a benchmark against existing state-of-the-art methods. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: We provide details on the compute resources we use in Appendix J.Guidelines:• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [NA]Justification: The paper does not use existing assets.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: We do not introduce any new assets in the paper.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: We do not perform any crowdsourcing experiments nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: We do not perform any experiments that involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: For simplicity, we only consider the case of a single 2d-convolution layer, a convolutional network with higher dimensions or more layers can be derived inductively.Let 𝑓 ∶ ℝ 𝑐×ℎ×𝑤 → ℝ 𝑐 ′ ×ℎ ′ ×𝑤 ′ . Suppose that this layer is parameterized by kernels 𝑊 𝑖 ∈ ℝ 𝑘×𝑘 for 𝑖 ∈ [𝑐 ′ ] and some 𝑘 ∈ ℕ as well as a bias 𝑏 ∈ ℝ 𝑐 ′ . Then, it follows that𝑓 (𝑥) 𝑗 = ( 𝟏 ℎ ′ ×𝑤 ′ 𝑏 𝑗 + 𝑐 ∑ 𝑖=1 𝑊 𝑗 * 𝑥[𝑖, ∶, ∶]) ,for 𝑗 ∈ [𝑐 ′ ] where 𝑓 (𝑥) 𝑗 ∈ ℝ ℎ ′ ×𝑤 ′ for ℎ ′ , 𝑤 ′ being the resulting dimension after convolution with a 𝑘 × 𝑘 kernel. Here, 𝟏 ℎ ′ ×𝑤 ′ ∈ ℝ ℎ ′ ×𝑤 ′ is a one matrix. To induce a distance metric on this output space, we can simply define a matrix norm on each of the output channels and sum them. Let {‖⋅‖ 𝑖 } 𝑐 ′ 𝑖=1 be a collection of matrix norms. Then, we define𝑑(𝑓 (𝑥), 𝑓 (𝑥 ′ )) = 𝑐 ′ ∑ 𝑖=1 ‖ ‖ 𝑓 (𝑥) 𝑖 -𝑓 (𝑥 ′ ) 𝑖 ‖ ‖𝑖 .It is easy to verify that this is a metric. Thus, the availability of a metric decomposition is not affected by parameter sharing.Instead of incorporating every individual channel into our metric, we may also consider applying a pooling operation before passing the result through a single matrix norm, ‖⋅‖. For example,𝑑(𝑓 (𝑥), 𝑓 (𝑥 ′ )) = 1 𝑐 ′ ‖ ‖ ‖ ‖ ‖ ‖ 𝑐 ′ ∑ 𝑖=1 𝑓 (𝑥) 𝑖 - 𝑐 ′ ∑ 𝑖=1 𝑓 (𝑥 ′ ) 𝑖 ‖ ‖ ‖ ‖ ‖ ‖ .This, however, is no longer a metric, as definiteness is not preserved. That is, there exists 𝑓 (𝑥) ≠ 𝑓 (𝑥 ′ ) where 𝑑(𝑓 (𝑥), 𝑓 (𝑥 ′ )) = 0. This issue can be easily resolved by having 𝑑(⋅, ⋅) operate on a quotient space with respect to the equivalence relation 𝑓 (𝑥) ∼ 𝑓 (𝑥 ′ ) if and only if∑ 𝑐 ′ 𝑖=1 𝑓 (𝑥) 𝑖 = ∑ 𝑐 ′ 𝑖=1 𝑓 (𝑥 ′ ) 𝑖 .This technique is further explored in the next subsection.Residual Connections. We present two distinct metric decompositions of a residual network. Consider two fully-connected layers with one residual connection. This is visualized below.Let us assume that𝐴 ∶ ℝ 𝑑 1 → ℝ 𝑑 2 and 𝐵 ∶ ℝ 𝑑 2 → ℝ 𝑑 1 .Here, the input 𝑥 feeds back into the output layer 𝐵 creating a residual block (the set of layers between the input and the residual connection).Trivially, we can aggregate the entire residual block as one metric decomposition. That is, let ℎ(𝑥) = 𝐵(𝐴(𝑥)) + 𝑥 be our metric decomposition. Then, define a metric on the image of ℎ, ℝ 𝑑 1 , analogous to the hidden layers of a fully-connected neural network. This is the approach we use throughout our practical applications section (Section 5), and it is the standard way to counter layers in computer vision [27] and natural langauge processing [16] .To operate at a finer lever of granularity, we can also represent each layer within the residual block as a part of a metric decomposition. Let us redefine the residual block such that at every layer, we keep track of the input. The computational graph for this is shown below.Define 𝐴 ′ ∶ 𝑥 ↦ (𝐴(𝑥), 𝑥), 𝐵 ′ ∶ (𝐴(𝑥), 𝑥) ↦ (𝐵(𝐴(𝑥)), 𝑥) and 𝑥 ′ ∶ (𝐵(𝐴(𝑥)), 𝑥) ↦ (𝐵(𝐴(𝑥))+𝑥, 𝑥).Then, it follows that 𝑥 → 𝐴 ′ → 𝐵 ′ → 𝑥 ′ forms a metric decomposition. Here, the metric in each layer is with respect to the quotient space where (𝑎, 𝑎 ′ ) ∼ (𝑏, 𝑏 ′ ) if and only if 𝑎 = 𝑏. Therefore, we also recover the same vector space structure.Transformers. By chaining our metric decompositions for the residual blocks with our metric decompositions for the fully-connected networks we can easily create a metric decomposition for any transformer. Throughout the paper, we use two distinct methods to generate representations of its hidden layers:• After each attention block which consists of multiheaded attention and multilayered perceptrons, we retrieve the last token.• We average all of the tokens together.In both of these methods, we are significantly reducing the dimension of the hidden layer. Thus, to formalize these metrics, we need to quotient out points that break the definiteness of our metric, as we have done before with the residual block.We have shown that our notion of a metric decomposition can well-describe many deep learning architectures, but what about models that are not neural networks (like a decision tree)? Herein, we demonstrate that we can induce metric decompositions even when the model itself does not have explicit hidden layers.Let us now consider an arbitrary function 𝑓 ∶  → . We can induce a metric decomposition on 𝑓 through an auxiliary function 𝑓 ′ ∶  → , for a metric-decomposable 𝑓 ′ . If 𝑓 ′ = Id  , then, 𝑓 = 𝑓 •𝑓 ′ and the metric decomposition of 𝑓 would be exactly the metric decomposition of 𝑓 ′ . This is visualized below. ( 1 , 𝑑 1 ) … ( 𝑛 , 𝑑 𝑛 )   𝑓 ′ 0 𝑓 ′ 1 𝑓 ′ 𝑛-1 𝑓 ′ 𝑛 𝑓 Metric Decomposition of 𝑓 ′Essentially, we have created an autoencoder for . This is common in many applications where a neural network or some other method is used as a feature extractor. In this way, we can simply define our metric with respect to these extracted features. However, this requires that either the autoencoder to be exact or that our function 𝑓 is invariant under representations that collide. Thus, this would allow models such as decision trees to also be metric decomposed.Theorem (See Thm. 4.1).Let 𝐴 ⊂  ×  such that ℙ  , [𝐴] > 0 and 𝛿, 𝜂 > 0. Let 𝐴 ′ = {(𝑥 ′ , 𝑦 ′ ) ∈  ×  ∶ 𝔼 (𝑥,𝑦)∼ ,(𝑥,𝑦)∈𝐴 Δ (𝑥,𝑦) 𝑓 (𝑥 ′ , 𝑦 ′ ) > 𝜂}. If 𝑓 ∶  →  is 𝜖-knowledge continuous with respect to the hidden layer indexed by 𝑘 and ( 𝑘 , 𝑑 𝑘 ) is bounded by 𝐵 > 0, thenℙ (𝑥,𝑦)∼ , [𝐴 ′ | 𝑑 𝑘 (𝑓 𝑘 (𝑥), 𝑓 𝑘 (𝐴)) < 𝛿] ≤ 𝜖𝛿 𝜂 ( 1 -exp [ -Ω ( 𝛿 𝐵 - √ log 1 ℙ[𝐴]) 2 ]) ,(B.1)where 𝑓 𝑘 (𝐴) = {𝑓 𝑘 (𝑎) ∶ 𝑎 ∈ 𝐴}.Proof.ℙ (𝑥,𝑦)∼ , [ 𝐴 ′ | 𝑑 𝑘 (𝑓 𝑘 (𝑥), 𝑓 𝑘 (𝐴)) < 𝛿 ] = ℙ (𝑥,𝑦)∼ , [ 𝐴 ′ ∩ 𝑑 𝑘 (𝑓 𝑘 (𝑥), 𝑓 𝑘 (𝐴)) < 𝛿 ] ℙ (𝑥,𝑦)∼ , [𝑑 𝑘 (𝑓 𝑘 (𝑥), 𝑓 𝑘 (𝐴)) < 𝛿] . (B.2)We bound the numerator and denominator of Eq. B.2 separately. The denominator is given by Cor. B.3. We upper-bound the numerator using Markov's inequality. Firstly, we find the expectation of  (𝑥,𝑦) 𝑓 (𝑥 ′ , 𝑦 ′ ) over 𝐴 ′ ∩ 𝑑 𝑘 (𝑓 𝑘 (𝑥), 𝑓 𝑘 (𝐴)) < 𝛿:𝔼 (𝑥,𝑦)∼ , 𝜎 𝑘 𝑓 (𝑥, 𝑦) = 𝔼 (𝑥,𝑦)∼ , ⎛ ⎜ ⎜ ⎝ 𝔼 (𝑥 ′ ,𝑦 ′ )∼ , ⎡ ⎢ ⎢ ⎣ Δ (𝑥,𝑦) 𝑓 (𝑥 ′ , 𝑦 ′ ) 𝑑 𝑘 (𝑓 𝑘 (𝑥), 𝑓 𝑘 (𝑥 ′ )) ⎤ ⎥ ⎥ ⎦ ⎞ ⎟ ⎟ ⎠ , (B.3) = 𝔼 (𝑥,𝑦),(𝑥 ′ ,𝑦 ′ )∼( , × , ) ⎡ ⎢ ⎢ ⎣ Δ (𝑥,𝑦) 𝑓 (𝑥 ′ , 𝑦 ′ ) 𝑑 𝑘 (𝑓 𝑘 (𝑥), 𝑓 𝑘 (𝑥 ′ )) ⎤ ⎥ ⎥ ⎦ . (B.4)The previous inequality follows from Fubini's theorem, then𝔼 (𝑥,𝑦)∼ , 𝜎 𝑘 𝑓 (𝑥, 𝑦) ≥ 𝔼 (𝑥,𝑦),(𝑥 ′ ,𝑦 ′ )∼( , × , ) (𝑥 ′ ,𝑦 ′ )∈𝐴 𝑑 𝑘 (𝑓 𝑘 (𝑥),𝑓 𝑘 (𝐴))<𝛿 ⎡ ⎢ ⎢ ⎣ Δ (𝑥,𝑦) 𝑓 (𝑥 ′ , 𝑦 ′ ) 𝑑 𝑘 (𝑓 𝑘 (𝑥), 𝑓 𝑘 (𝑥 ′ )) ⎤ ⎥ ⎥ ⎦ , (B.5) ≥ 1 𝛿 𝔼 (𝑥,𝑦),(𝑥 ′ ,𝑦 ′ )∼( , × , ) (𝑥 ′ ,𝑦 ′ )∈𝐴 𝑑 𝑘 (𝑓 𝑘 (𝑥),𝑓 𝑘 (𝐴))<𝛿 [ Δ (𝑥,𝑦) 𝑓 (𝑥 ′ , 𝑦 ′ ) ] , (B.6) 𝛿 𝔼 (𝑥,𝑦)∼ , 𝜎 𝑘 𝑓 (𝑥, 𝑦) ≥ 𝔼 (𝑥,𝑦),(𝑥 ′ ,𝑦 ′ )∼( , × , ) (𝑥 ′ ,𝑦 ′ )∈𝐴 𝑑 𝑘 (𝑓 𝑘 (𝑥),𝑓 𝑘 (𝐴))<𝛿 [ Δ (𝑥,𝑦) 𝑓 (𝑥 ′ , 𝑦 ′ ) ] . (B.7)And by 𝜖-knowledge continuity,𝛿𝜖 ≥ 𝔼 (𝑥,𝑦),(𝑥 ′ ,𝑦 ′ )∼( , × , ) (𝑥 ′ ,𝑦 ′ )∈𝐴 𝑑 𝑘 (𝑓 𝑘 (𝑥),𝑓 𝑘 (𝐴))<𝛿 [ Δ (𝑥,𝑦) 𝑓 (𝑥 ′ , 𝑦 ′ ) ] . (B.8)This gives us an upper-bound of expectation of Δ (𝑥,𝑦) 𝑓 (𝑥 ′ , 𝑦 ′ ) over the set of all points that are within 𝛿-radius from 𝑓 𝑘 (𝐴). Since Δ (𝑥,𝑦) 𝑓 (𝑥 ′ , 𝑦 ′ ) ≥ 0 everywhere, by Markov's inequality,ℙ (𝑥,𝑦)∼ , [𝐴 ′ ∩ 𝑑 𝑘 (𝑓 𝑘 (𝑥), 𝑓 𝑘 (𝐴)) < 𝛿] ≤ 𝛿 𝔼 𝜎 𝑘 𝑓 (𝑥, 𝑦) 𝜂 , (B.9) ≤ 𝛿𝜖 𝜂 . (B.10)The last inequality follows from 𝔼 (𝑥,𝑦)∼ , 𝜎 𝑘 𝑓 (𝑥, 𝑦) < 𝜖, by the definition of 𝜖-knowledge continuity. Now, by applying the complement of Lem. B.2, we lower-bound the denominator and yield the followingEQUATION)The proof is concluded by applying Ω(⋅) notation to the denominator. ℙ[|𝑓 (𝑋 1 , … , 𝑋 𝑛 ) -𝔼[𝑓 (𝑋 1 , … , 𝑋 𝑛 )| ≥ 𝜖] ≤ 2 exp ( - 2𝜖 2 ∑ 𝑛 𝑖=1 𝑐 2 𝑖 ) . (B.13) Lemma B.2. Suppose that (, 𝑑) is a bounded metric space such that sup 𝑥,𝑥 ′ ∈ 𝑑(𝑥, 𝑥 ′ ) < 𝐵 for some 𝐵 > 0. Let 𝐴 ⊂ 𝑋 such that ℙ[𝐴] > 0 and 𝜖 > 0. Then, ℙ[𝑑(𝑥, 𝐴) ≥ 𝜖] ≤ exp ⎛ ⎜ ⎜ ⎝ - 2 𝐵 2 ( 𝜖 -𝐵 √ 1 2 log 2 ℙ[𝐴] ) 2 ⎞ ⎟ ⎟ ⎠ .Proof. For brevity, denote 𝑓 𝐴 (𝑥) = 𝑑(𝐴, 𝑥) = inf 𝑎∈𝐴 𝑑(𝑥, 𝑎). Since (, 𝑑) is a bounded metric space, by Lem. B.1,ℙ[|𝑓 𝐴 (𝑥) -𝔼𝑓 𝐴 (𝑥)| ≥ 𝜖] = 2 exp ( - 2𝜖 2 𝐵 2 ) , (B.14) ℙ[𝑓 𝐴 (𝑥) -𝔼𝑓 𝐴 (𝑥) ≥ 𝜖] + ℙ[𝑓 𝐴 (𝑥) -𝔼𝑓 𝐴 (𝑥) ≤ -𝜖] ≤ 2 exp ( - 2𝜖 2 𝐵 2 ) , (B.15) ℙ[𝑓 𝐴 (𝑥) -𝔼𝑓 𝐴 (𝑥) ≤ -𝜖] ≤ 2 exp ( - 2𝜖 2 𝐵 2 ) , (B.16) Let 𝜖 = 𝔼𝑓 𝐴 (𝑥). Then, ℙ[𝑓 𝐴 (𝑥) ≤ 0] ≤ 2 exp ( - 2(𝔼𝑓 𝐴 (𝑥)) 2 𝐵 2) , (B.17)ℙ[𝐴] ≤ 2 exp ( - 2(𝔼𝑓 𝐴 (𝑥)) 2 𝐵 2 ) , (B.18) 𝔼𝑓 𝐴 (𝑥) ≤ √ 𝐵 2 2 log ( 2 ℙ[𝐴] ) . (B.19)The second inequality follows fromℙ[𝑓 𝐴 (𝑥) ≤ 0] = ℙ[𝑓 𝐴 (𝑥) = 0] ≥ ℙ[𝐴]. By Eq. B.15, ℙ[𝑓 𝐴 (𝑥) -𝔼𝑓 𝐴 (𝑥) ≥ 𝜖] + ℙ[𝑓 𝐴 (𝑥) -𝔼𝑓 𝐴 (𝑥) ≤ -𝜖] ≤ 2 exp ( - 2𝜖 2 𝐵 2 ) , ℙ[𝑓 𝐴 (𝑥) -𝔼𝑓 𝐴 (𝑥) ≥ 𝜖] ≤ 2 exp ( - 2𝜖 2 𝐵 2 ) , ℙ[𝑓 𝐴 (𝑥) ≥ 𝜖 + 𝔼𝑓 𝐴 (𝑥)] ≤ 2 exp ( - 2𝜖 2 𝐵 2 ) , ℙ ⎡ ⎢ ⎢ ⎣ 𝑓 𝐴 (𝑥) ≥ 𝜖 + √ 𝐵 2 2 log ( 2 ℙ[𝐴] ) ⎤ ⎥ ⎥ ⎦ ≤ 2 exp ( - 2𝜖 2 𝐵 2) , (by Eq. B.19,)for any 𝛿 > 0, let 𝜖 = 𝛿 -√𝐵 2 2 log 2 ℙ[𝐴]. And so,ℙ[𝑓 𝐴 (𝑥) ≥ 𝛿] ≤ 2 exp ⎛ ⎜ ⎜ ⎜ ⎝ - 2 𝐵 2 ⎛ ⎜ ⎜ ⎝ 𝛿 -𝐵 √ 1 2 log ( 2 ℙ[𝐴] ) ⎞ ⎟ ⎟ ⎠ 2 ⎞ ⎟ ⎟ ⎟ ⎠, which is the desired expression.■ Corollary B.3. ℙ[𝑓 𝐴 (𝑥) < 𝛿] ≥ 1 -2 exp ⎛ ⎜ ⎜ ⎝ -2 𝐵 2 ( 𝛿 -𝐵 √ 1 2 log ( 2 ℙ[𝐴] ) ) 2 ⎞ ⎟ ⎟ ⎠. (𝑥) , 𝑦) 𝑑𝜇 𝑓 < 𝜖 𝑛 . It remains to show that each of these functions are in fact knowledge continuous. Since  is a metric space, we consider the trivial metric decomposition of our sequence of functions (see Remark 1). Specifically, we denote ℎ 1 = Id  and proceed to bound 𝔼 𝜎 1 𝑓 (𝑥, 𝑦). Eq. D.11 follows from inductively applying the definition of Lipschitz continuity on the metric decompositions of 𝑓 . Specifically, 𝑑 𝑖+1 (𝑓 𝑖+1 (𝑥), 𝑓 𝑖+1 (𝑥 ′ )) ≤ 𝐾 𝑖 𝑑 𝑖 (𝑓 𝑖 (𝑥), 𝑓 𝑖 (𝑥)). Then, by the Lipschitz continuity of  in both coordinates we yield Eq. D.12. Since the Lebesgue integral preserves order, Eq. D.13 directly implies the statement of the proposition and this concludes the proof.𝔼 𝜎 1 𝑓 𝑛 (𝑥, 𝑦) = ∬ Δ (𝑥,𝑦) 𝑓 𝑛 (𝑥 ′ , 𝑦 ′ ) 𝑑  (𝑥, 𝑥 ′ ) (𝑑𝜇 𝑓 × 𝑑𝜇 𝑓 ), (C.1) ≤ ∬ |(𝑓 𝑛 (𝑥), 𝑦) -(𝑓 𝑛 (𝑥 ′ ), 𝑦) + (𝑓 𝑛 (𝑥 ′ ), 𝑦) -(𝑓 𝑛 (𝑥 ′ ), 𝑦 ′ )| 𝑑  (𝑥, 𝑥 ′ ) (𝑑𝜇 𝑓 × 𝑑𝜇 𝑓 ), (C.2) ≤ ∬ |(𝑓 𝑛 (𝑥), 𝑦) -(𝑓 𝑛 (𝑥 ′ ), 𝑦)| 𝑑  (𝑥, 𝑥 ′ ) 𝑑(𝜇 𝑓 × 𝜇 𝑓 ) (C.3) + ∬ |(𝑓 𝑛 (𝑥 ′ ), 𝑦) -(𝑓 𝑛 (𝑥 ′ ), 𝑦 ′ )| 𝑑(𝑥, 𝑥 ′ ) (𝑑𝜇 𝑓 × 𝑑𝜇 𝑓 ), (C.4) ≤ ∬ 𝐾𝑑  (𝑓 (𝑥), 𝑓 (𝑥 ′ )) 𝑑  (𝑥, 𝑥 ′ ) 𝑑(𝜇 𝑓 × 𝜇 𝑓 ) + ∬ 𝐾𝑑  (𝑦, 𝑦 ′ ) 𝑑  (𝑥,In this section, we detail the experimental methods and results that use knowledge continuity to predict adversarial vulnerability, briefly discussed in Section 5. We focus on langauge models of various sizes and their ability to perform sentiment classification on the IMDB dataset [48] . Before computing any statistics of the model, we finetune it against the IMDB dataset and reserve a test set on which we compute a vulnerability score and estimate the model's adversarial vulnerability.Vulnerability Score. As described in the main text, given a model with 𝑛 hidden layers, we compute all of its 𝑘-volatility scores. This is done with a naive Monto-Carlo algorithm which we present in Appendix G. This results in a list of 𝑘-volatility scores {𝜖 1 , … , 𝜖 𝑛 }, one for each hidden layer. Then, we perform a simple average 𝑛 -1 ∑ 𝑛 𝑘=1 𝜖 𝑘 . Let us denote this quantity as the vulnarability score. Estimating Adversarial Robustness. It remains to estimate the adversarial vulnerability of a given model. We do this empirically by applying an out-of-the-box adversarial attack (specifically, TextFooler [35] ) on the given model with respect to the reserved test set. We then measure the number of successful adversarial attacks defined as♯Successful Adversarial Attacks = | adversarial ∩  correct ) | correct |, where  correct is the set of examples in the test set that are correctly classified by the model (after finetuning) without any intervention. And,  adversarial are the set of examples that are incorrectly classified after an adversarial attack is applied. In other words, we only consider points where a perturbation will worsen performance. In expectation, this estimate of adversarial robustness should be a 1 ∕2 factor of the notion of vulnerability we present in Thm. 4.1, where we also consider a point to be vulnerable if perturbation increases its performance.We then perform a linear regression using vulnerability score and a host of other model properties to predict the number of successful adversarial attacks. Concretely, we seek to learn the relationship:♯Successful Adversarial Attacks = 𝑚 𝑇 ( 𝑛 -1 𝑛 ∑ 𝑘=1 𝜖 𝑘 ⊕ … ⏟ ⏟ ⏟ additional architectural variables ) + 𝑏,where 𝑚 ∈ ℝ 𝑑 and 𝑏 ∈ ℝ are the learnable regression parameters. We also incorporate 𝑑 -1 size and architectural variables into our regression as we found that significantly increases its predictiveness. And so, the input variables to our regression and their types are: We also run permutation tests for each coefficient and the change in 𝑅 2 is shown in the column Δ𝑅 2 (higher the better).For the regression itself, we perform a Ridge regression with 𝛼 = 1. We test three experimental conditions where we regress the model's adversarial robustness against: (1) only vulnerability score, ( 2) vulnerability score and model characteristics, (3) only model characteristics. We experiment with seven models: RoBERTa (Base/Large) [44] , BERT-Uncased (Base/Large) [16] , GPT2, and T5 (Small/Base) [58] . Our regression results are shown in Table . 2.After yielding an initial line-of-best fit (see Fig. 2 (a)), we run permutation tests to determine the contribution of each feature to the explained variance. Specifically, for each feature, keeping all else constant, we permute its values. If this feature is a significant contributor to the explained variance, intuitively, we should see a large decrease in 𝑅 2 after this intervention. If 𝑠 is the 𝑅 2 without any intervention and 𝑠 𝜎 𝑖 (𝑑) is the 𝑅 2 after permuting the data by 𝜎 𝑖 (⋅) ∶ [𝑛] → [𝑛] (for a dataset of 𝑛 data points). Then, we defineΔ𝑅 2 = 𝑠 - 1 𝑁 𝑁 ∑ 𝑘=1 𝑠 𝜎 𝑘 (𝑑) ,where 𝑁 controls the number of permutations that we apply. For all experiments we choose 𝑁 = 100.For formal theory on permutation tests, see [8] .We find that when our vulnerability score is added to the regression, it contributes significantly to the explained variance. Moreover, in (2), we see that vulnerability score has the highest feature importance among all regression variables.In this section, we localize adversarially vulnerable hidden representations in two ways. Firstly, we use 𝑘-volatility to gauge which layers are vulnerable across a selection of models. Then, we focus on model-specific characterizations of robustness with respect to 𝑘-volatility. We present experiments on the same selection of models in Appendix E, the same dataset (IMDB [48] ), and the same adversarial attack (TextFooler [35] ) to empirically measure adversarial vulnerability.As mentioned in the previous section (Section E), for a given model with 𝑛 hidden layers, we can measure its 𝑘-volatility for 𝑘 ∈ [𝑛] through a Monte-Carlo algorithm. For each model, we then plot its 𝑘-volatility against its relative depth which is defined as ⌊𝑘∕𝑛⌋. These curves are shown in Fig. 2(b ). We see that models which have different architectures independent of size have very different 𝑘-volatility curves.We have already shown in the previous section that there is a positive correlation between 𝑘-volatility and adversarial vulnerability. However, this correlation is derived from the simple average of all 𝑘volatility scores. Are the 𝑘-volatility scores in some layers more predictive of adversarial vulnerability than others? If the 𝑘-volatility in some layers is more correlated with 𝑘-volatility in others, then it should suffice to minimize 𝑘-volatility in these former layers. This would also speed up regularization and training.We repeat the experiments in the previous settings. But, instead of collating 𝑘-volatility through a simple average, we run one regression for each relative depth across all models (which we discretize into 9 bins). This result is shown in Fig. 2(c ). Surprisingly, we find that the magnitude of 𝑘-volatility is not necessarily predictive of adversarial vulnerability. For example, in Fig. 2(b) , almost all of the models exhibit low average 𝑘-volatility in the latter layers. However, the 𝑘-volatility of latter layers predict adversarial vulnerability the best.We start by exploring the 𝑘-volatility across each of our test models. We notice that 𝑘-volatility cannot be predicted by surface-level features such as size or model type alone. This is shown clearly in Fig. 3 . Yet, as discussed in Appendix E, it is still able to predict actual adversarial vulnerability with moderate power. Thus, we conjecture that 𝑘-volatility captures a complex aspect of the model's vulnerability which cannot be solely attributed to its size or type. Figure 3 : Average 𝑘-volatility plotted against the log of number of model parameters (left). We see that although there is a strong negative correlation, the exactly relationship is nontrivial. Moreover, this negative correlation is also consistently observed across model families (right).In this section, we provide a comprehensive overview of regulating knowledge continuity to achieve robustness. We first show a simple algorithm that estimates 𝑘-volatility. Then, we demonstrate how this can incorporated into any loss function as a regularization term. We then prove guarantees that revolve around the unbiasedness of our estimation algorithm. Lastly, we present detailed discussion of the results shown in Table 1 including training details and ablation studies over the hyperparameters.We first present a method for estimating 𝑘-volatility. This is shown in Alg. 1(ESTKVOL). In theory, one should choose 𝑀 = 𝑁, as this will lead to a most accurate estimate. This is similar to contrastive learning methods where it is desirable to make the minibatch sizes as large as possible [56] . However, if 𝑁 ≫ 1, this can become quickly intractable. In practice, during regularization we keep 𝑁 to be the same as if we were doing normal finetuning (i.e. 32/64) and set 𝑀 = 𝑁. This works well, and, anecdotally, we find that in contrast to contrastive learning increasing 𝑁 or 𝑀 past this threshold yields marginal returns. Further work could examine this relationship in more detail.As discussed in the main text, the choice of metric (or representation space) which we enforce knowledge continuity against is crucial as it determines the type of robustness we will achieve. Therefore, in Alg. 1(KCREG), we incorporate this detail by sampling a hidden layer of interest using a Beta distribution specified by hyperparameters 𝛼, 𝛽. Then, on that minibatch, regularize 𝑘-volatility Algorithm 1 A Monte-Carlo algorithm for estimating 𝑘-volatility of some metric decomposable function 𝑓 with 𝑛 hidden layers (left). Augmenting any loss function to regularize 𝑘-volatility (right), given some Beta distribution parameterized by 𝛼, 𝛽 and regularization strength 𝜆 ≥ 0.procedure ESTKVOL({(𝑥 𝑖 , 𝑦 𝑖 )} 𝑁 𝑖=1 , 𝑀, 𝑓 , 𝑘) Sample {𝑛 1 , … , 𝑛 𝑀 } ⊂ [𝑁] uniformly 𝜎 𝑘 𝑓 ← 0 Losses ← {(𝑓 (𝑥 𝑛 𝑖 ), 𝑦 𝑛 𝑖 )} 𝑀 𝑖=1 for (𝑖, 𝑗) ∈ [𝑀] × [𝑀] do Dist ← 𝑑 𝑘 (𝑓 𝑘 (𝑥 𝑛 𝑖 ), 𝑓 𝑘 (𝑥 𝑛 𝑗 )) 𝜎 𝑘 𝑓 ← 𝜎 𝑘 𝑓 + |Losses 𝑖 -Losses 𝑗 |∕DIST return 𝜎 𝑘 𝑓 procedure KCREG(𝛼, 𝛽, 𝑀, 𝜆) 𝑋 ∼ Beta(𝛼, 𝛽) 𝑘 ← max(⌊𝑋𝑛⌋, 1) 𝜎 𝑘 𝑓 ← ESTKVOL({(𝑥 𝑖 , 𝑦 𝑖 )} 𝑁 𝑖=1 , 𝑓 , 𝑀, 𝑘) return 1 𝑁 ∑ 𝑁 𝑖=1 (𝑓 (𝑥 𝑖 ), 𝑦 𝑖 ) + 1 𝑀 2 𝜆𝜎 𝑘 𝑓with respect to that sampled layer. Note that we choose the Beta distribution for simplicity, however, it can be replaced by any distribution like a mixture of Gaussians.In contrast to existing adversarial training methods such as [32] and [63] which only use the embeddings, our algorithm gives the practitioner more control over which hidden layer (or distance metric) to enforce smoothness. In this way, if the practitioner has some knowledge a priori of the attacker's strategy, they may choose to optimize against the most suitable metric decomposition. We present a brief discussion of the various tradeoffs when choosing 𝛼, 𝛽 in the following section as well as a detailed empirical analysis in the following subsections.𝜆 is the weight we put on the regularizer in relation to the loss function . We provide a detailed ablation study of the effects of 𝜆 in the following subsections. We surprisingly find that even for 𝜆 ≪ 1 we can achieve significant edge in terms of robustness over existing methods. This is in contrast to virtual adversarial training methods such as [43] which requires applying a 𝜆-value magnitudes larger. Moreover, for larger 𝜆, we find that the accuracy of the model is not compromised. This provides some empirical support for Theorem 4.3.In this subsection, we show that our Monte-Carlo algorithm presented in Alg. 1(ESTKVOL) is an unbiased estimator. The proof is simple and follows from some bookkeeping. Proposition G.1 (Alg. 1(ESTKVOL) is an Unbiased Estimator). Assuming that each data point in the batch, {(𝑥 𝑖 , 𝑦 𝑖 )} 𝑁 𝑖=1 ∼  , , is sampled i.i.d., then Alg. 1(ESTKVOL) is an unbiased estimator for 𝔼 𝜎 𝑘 𝑓 (𝑥, 𝑦).Proof. Let θ be the random variable representing the output of Alg. 1. It suffices to show that𝔼[ θ] = 𝔼 𝜎 𝑘 𝑓 (𝑥, 𝑦), where the expectation on the left-hand side is taken over the set of all batches. By the definition of Alg. 1(ESTKVOL),𝔼[ θ] = 𝔼 ⎛ ⎜ ⎜ ⎜ ⎝ 𝑀 ∑ 𝑖=1 𝑀 ∑ 𝑗=1 1 𝑀 2 Δ (𝑥 𝑛 𝑗 ,𝑦 𝑛 𝑗 ) 𝑓 (𝑥 𝑛 𝑖 , 𝑦 𝑛 𝑖 ) 𝑑 𝑘 (𝑓 𝑘 (𝑥 𝑛 𝑖 ), 𝑓 𝑘 (𝑥 𝑛 𝑗 ) ) ⎞ ⎟ ⎟ ⎟ ⎠ , (G.1) = 𝑀 ∑ 𝑖=1 𝑀 ∑ 𝑗=1 1 𝑀 2 𝔼 ⎛ ⎜ ⎜ ⎜ ⎝ Δ (𝑥 𝑛 𝑗 ,𝑦 𝑛 𝑗 ) 𝑓 (𝑥 𝑛 𝑖 , 𝑦 𝑛 𝑖 ) 𝑑 𝑘 (𝑓 𝑘 (𝑥 𝑛 𝑖 ), 𝑓 𝑘 (𝑥 𝑛 𝑗 ) ) ⎞ ⎟ ⎟ ⎟ ⎠ , (G.2) = 𝔼 𝜎 𝑘 𝑓 (𝑥, 𝑦). (G.3)The second equality follows from the linearity of expectation.We emphasize that our estimator is very naive. Improving its efficiency could form the basis of possible future work. For example, Rao-Blackwellizing [6] Alg. 1(ESTKVOL) to yield an estimator with smaller variance, applying rejection sampling to deal with the potential sparsity of the representation space discussed in Section 4.4, or adapting the regularization weight based on some bootstrapped confidence interval (if the estimate has higher variance then decrease weight on regularization and vice versa). However, we see that even with this naive algorithm we achieve improvements in robustness as well as training speed.In addition to regulating language models, we also demonstrate that KCREG is effective for vision tasks. This provides empirical support for the equivalences we proved in Section 4.5. The exact same method of 𝑘-volatility estimation and loss augmentation is applied. We finetune three models ResNet50 [28] , MobileNetV2 [61] , and ViT16 [17] on the MNIST dataset both with and without our regularization algorithm. We then apply two different adversarial attacks: FGSM [24] and SI-NI-FGSM [41] . We find that in both cases, regularization 𝑘-volatility improves/stabilizes robustness across attack strengths (see Fig. 4 ).Herein, we present ablation studies for the crucial hyperparameters in our regularization algorithm (across the natural language tasks that we explored in the main text), Alg. 1(KCREG): 𝜆 which is the weight we assign the knowledge continuity regulation loss and (𝛼, 𝛽) which determines the sampling behavior of the index of the hidden representation space.Ablation Study of 𝜆 (Fig. 5 (right)). The weight given to the regularizer (𝜆) is ablated over, with the results shown in Fig. 5 . For any positive 𝜆, there is an immediate large improvement in adversarial robustness. Next, as 𝜆 is systematically increased by factors of 10, we do not see a significant change in the accuracy (not under attack). This corroborates Theorem. 4.3, as it demonstrates that regulating knowledge discontinuities (no matter how strongly) is not at odds with minimizing the empirical risk of our model. On the other hand, we also do not see a significant increase in adversarial robustness as 𝜆 increases. This may imply that we have reached the threshold of adversarial robustness under TextFooler [35] . Specifically, the adversarial attacks generated by TextFooler may not be valid in that they have flipped the ground-truth label. Therefore, we believe that a good 𝜆 for this particular application should lie somewhere between 0 and 1 × 10 -4 . 3) and its effect on test accuracy (right). We see that moderate regularization significantly improves robustness across all attack strengths. This improvement does not come at the expense of test accuracy.The attack-strength is measured using the minimum angular similarity between the perturbed and original text. Both ablations are done with respect to GPT2 on the IMDB [48] dataset with respect to the TextFooler attack [35] .Ablation Study of Adversarial Attack Strength (Fig. 5 (left)). For every value of 𝜆, we also vary the strength of the adversarial attack. The adversarial attack strength is measured through the angular similarity of the embeddings between the original text and the perturbed text. Intuitively, if this constraint is loosened the adversary is allowed to find text that is semantically very different and vice versa. We see that moderate 𝑘-volatility regulation achieves the best adversarial robustness across all attack strengths.Ablation Study of (𝛼, 𝛽) In this subsection, we briefly discuss how the 𝛼, 𝛽 hyperparameters which determine the shape of the Beta distribution in Alg. 1(KCREG) affect the final performance and robustness of our model on the IMDB dataset. Recall that the shape of the Beta distribution determines the index of the hidden layers we are using the compute the knowledge continuity. Thus, they are crucial in determining the behavior of our regularizer.We finetune {BERT, T5, GPT2} models on the IMDB dataset with the hyperparameters described in the next subsection. The results are displayed in Table 3 . Across all models we observe a decrease in robustness for 𝛼 = 1, 𝛽 = 2. These values correspond to a right-skewed distribution which places high sampling probability on the earlier (closer to the input) hidden layers. Intuitively, perturbations in the early layers should correspond to proportional textual perturbations in the input text. Pure textual perturbations with respect to some metric like the Levenshtein distance should be only loosely if not completely (un)correlated with the actual labels of these inputs. Therefore, enforcing knowledge continuity with respect to this metric should not see increase robustness. Moreover, we also observe a larger decrease in accuracy (not under attack) with the same parameters. This suggests that maintaining this sort of knowledge continuity in the earlier layers is harder to converge on and there may be a "push-and-pull" behavior between optimizing knowledge continuity and accuracy (not under attack). Surprisingly, we observe no significant difference between the other 𝛼, 𝛽 values shown in the table.We did not formally benchmark other configurations of 𝛼, 𝛽 such as increasing their magnitude to impose a sharper distribution. 94.9 87.8 GPT2+Reg (1, 2) 93.1 84.9T5 BASE 93.7 53.9 T5 BASE +Reg (2, 1) 95.0 88.9 T5 BASE +Reg (2, 2) 94.9 89.3 T5 BASE +Reg (1, 2) 94.6 88.1 In this section, we describe in detail the training objectives, procedures, algorithms, and hyperparmeters that we used in the main text and further experiments done in the appendix.Training. For all models undergoing adversarial training, we first finetune the model against the training set. Then, attack it using the TextFooler [35] algorithm with examples from the training set. After the attacks are concluded, we then incorporate the text of successful adversarial attacks back into the training set and proceed to finetune again. This procedure iteratively continues. For the sake of computational efficiency, for all models we applied this procedure once. The parameters we are using during the adversarial attack is the same hyperparameters we actually use at test-time. Specifically, we impose a query budget of 300 queries.Plain Finetuning on IMDB. The IMDB dataset consist of 50,000 examples with 25,000 for training and 25,000 for testing. We split the test set 40%-60% to create a validation and test set of 10,000 and 15,000 examples, respectively. Examples were sampled uniformly at random during the splitting process. Since adversarial attacks were costly, we uniformly subsampled 5,000 examples from this 15,000 to benchmark robustness in the experiments related to the regularizer. However, for the experiments estimating the knowledge vulnerability score, we performed adversarial attacks on all 15,000 datapoints in the test set. We found no significant difference between robustness estimation on this 5,000 subsample versus and the entire 15,000 dataset.We train all models using the hyperparameter and optimizer configurations shown in Table 4 .• The answer NA means that the paper does not include experiments. • The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean.• It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.Codebase for our experiments can be found at https://github.com/alansun17904/kc 38th Conference on Neural Information Processing Systems (NeurIPS2024).Small perturbations on the input result in small changes in performance which implies small changes in output when the loss function is Lipschitz continuous. 


Decision: The paper received all positive ratings before the rebuttal. The reviewers think the idea is novel and the empirical results are promising. After the rebuttal, most of the minor concerns have been addressed. The AC made the acceptance recommendation.

Reviews: review: paper_summary: paper_summary: This paper proposes "knowledge continuity", a metric inspired by the Lipschitz continuity that aims to certify the robustness of neural networks. Compared to existing methods, the proposed metric yields certification guarantees that only depend on loss functions and the intermediate learned metric spaces of neural networks, which are independent of domain modality, norms, and data distribution. The authors further show that the knowledge continuity is not conflicted with the expressiveness of a model class. Several practical applications are provided.
strength_weakness: strength_weakness: - A lot of details of the empirical results are deferred to the appendix which makes me hard to understand what are the experimental settings. The authors should consider reorganize the paper - right now it seems that the paper is actually compressed from a longer version (say journal) but not dedicated to the conference. For example, the regression test seems to be very interesting but there is no any actual result shown in the main text. 
- Some numbers are wrongly bolded in the table (or the meaning of bolded results is not clearly conveyed) since they are not the highest number.
questions: questions: I don't have questions.
limitations: limitations: Yes, the authors have discussed the limitation in the main text.
paper_summary: paper_summary: This paper proposes a new concept inspired by Lipschitz continuity, aimed at certifying the robustness of neural networks across different input domains, such as continuous and discrete domains in vision and language. This certification guarantees based on the loss function and intermediate learned metric spaces, independent of domain modality, norms, and distribution. In this paper, the authors define knowledge continuity as the stability of a model's performance to its perceived knowledge of input-output relations. This approach focuses on the variability of a model's loss to its hidden representations rather than imposing arbitrary metrics on inputs and outputs.
strength_weakness: strength_weakness: - The authors claim that their method generalizes across modalities (continuous, discrete, and non-metrizable). However, they have not conducted experiments in other domains. The experiments in Table 1 are limited to the discrete text domain.
- The paper does not provide specific implementation details. Section 5 directly presents the experimental results in Table 1, but the metrics in Table 1 are not clearly defined.
- From the paper "On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspective," it is observed that model robustness improves with increased model capacity. It is necessary to demonstrate that the proposed method works on larger models like LLaMA3-70B/Qwen-1.5-70B to prove that the effectiveness of the method is not merely due to limited model capacity.
- The proposed method does not consistently achieve better performance on ANLI. It remains competitive with previous methods rather than showing clear superiority.
- The datasets used for comparison, IMDB and ANLI, focus more on word-level attacks, which are not particularly challenging for current large language models (LLMs). There is a lack of analysis on LLMs.
- It is unclear how much benefit this method provides to bidirectional attention models like BERT and RoBERTa, which already have some error correction capabilities.
- When dealing with out-of-distribution (OOD) data, the distributions themselves are different. It is uncertain whether this method will still be effective under such conditions.
questions: questions: - See "Weaknesses"
- The equation in line 252 is not numbered and contains a typo.
limitations: limitations: N/A
paper_summary: paper_summary: This paper proposes a generalization of adversarial robustness and certified robustness for continuous, discrete, or non-metrizable input domains. The authors introduce _knowledge continuity_, a novel definition inspired by Lipschitz continuity which aims to certify the robustness of neural networks across input domains (such as continuous and discrete domains in vision and language, respectively). While existing approaches that aim to certify robustness (e.g., Lipschitz continuity) lie in the continuous domain with norm and distribution-dependent guarantees, the new proposed framework proposes certification guarantees that depend only on the loss function and the intermediate learned metric spaces of the neural network. 

The current definition of robustness considers the stability of a model's performance with respect to its perceived knowledge of input-output relationships. However, the authors argue that robustness is better achieved by focusing on the variability of a model's loss with respect to its hidden representations, rather than imposing arbitrary metrics on its inputs and outputs.

To summarize the authors' contributions:
- They introduce _knowledge continuity_, a new concept that frames robustness as variability of a model's loss with respect to its hidden representations.
-  They show theoretically that knowledge leads to certified robustness guarantees that generalize across modalities (continuous, discrete, and non-metrizable), and show that this new robustness definition does not come at the expense of inferential performance.  
- They present several practical applications of knowledge continuity such as using it  to train more robust models and to identify problematic hidden layers.
strength_weakness: strength_weakness: - While the method seems to be useful for robustness in the context of discrete input domains such as language, the new framework also seems to inherit drawbacks of current certified methods, namely Lipschitz continuity and randomized smoothing:
  - Obtaining robustness from Lipschitz continuity requires having knowledge of the architecture of the network, and constraining the architecture to fit the definition; an example of this is that transformers are not _globally_ Lipschitz continuous, and thus certified robustness for transformer architectures via Lipschitz continuity is difficult. While knowledge continuity could in theoretically be used on any architecture where the hidden layers are metric spaces, I wonder if specific architectures might lead to vacius bounds. Similarly, quantized networks would not fall under the definition of having hidden layers that are metric spaces . 
  - As randomized smoothing, the robustness derived from knowledge continuity is not deterministic. This could lead to a high computation of robustness accuracy and certified accuracy.  
- The implementation of knowledge continuity is not very clear, I understand that the authors have proposed a regularization scheme via Algorithm 1 for "Estimating knowledge continuity". Algorithm 1 estimates the the expected value that defines knowledge continuity, is this approximation accurate and how does the accuracy of the approximation affect the training speed?
- While the authors have shown that there is theoretically no tradeoff between the robustness of knowledge continuity and natural accuracy, this does not mean that there is no tradeoff in practice. It would be interesting to see the tradeoff between between the strength of the attacks and the strength of the regularization, which seems to be missing from the paper.
- The authors proposed a certified robustness approach, but no certification algorithm was provided. How to compute the certified accuracy with knowledge continuity. Given the non-deterministic approach of their framework, some statistical testing, as in randomized smoothing, may be necessary.
questions: questions: See Weaknesses.
limitations: limitations: See Weaknesses.
paper_summary: paper_summary: In this work, the authors propose a novel definition of "continuity": knowledge continuity, which measures the continuity of a model with respect to its hidden representations instead of the inputs. The knowledge continuity can be used to certify adversarial robustness across domain modality. Then, they provide theorems of robustness certification based on the new definition of continuity and demonstrate that in some cases the knowledge continuity implies Lipschitz continuity. Further, they prove that knowledge continuity doesn't affect the universal approximation property of neural networks, which means constraining a network to be knowledge-continuous may not hinder its expressiveness. Lastly, the authors provide some applications of knowledge continuity in improving robustness.
strength_weakness: strength_weakness: Some typos:
* In Line 193, delete "with respect to a function $f$"? 
* In Line 218, $P_X[A]$ -> $P_{X\times Y}[A]$.
questions: questions: In this work, the authors decompose the network $f^k$ as $h^k\circ h^{k-1}\circ \dots \circ h^1\circ h^0$. However, some architectures such as ResNet and transformers, have residual structures. So, how can we decompose these networks as above?
limitations: limitations: As the authors said, this work provides some probability results that don't protect against OOD attacks. More deterministic results are needed in future studies.

Comments: title: Thanks for your reply
comment: The authors' reply has answered my questions. And I keep the score unchanged.
Thank you for your detailed responses. I appreciate your clarifications, and most of my concerns have been addressed. I've raised the rating score accordingly.
Thank you for the rebuttal, I think adding the algorithm to compute the certified robustness strengthens the paper. I am in favor of accepting this paper at the conference and have raised my score accordingly.