SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Rattana Pukdee*
Carnegie Mellon University rpukdee@cs.cmu.edu
Dylan Sam*
Carnegie Mellon University dylansam@andrew.cmu.edu
J. Zico Kolter
Carnegie Mellon University Bosch Center for AI zkolter@cs.cmu.edu
Carnegie Mellon University  
ninamf@cs.cmu.edu
Pradeep Ravikumar
Carnegie Mellon University  
pkr@cs.cmu.edu
As larger deep learning models are hard to interpret, there has been a recent focus on generating explanations of these black-box models. In contrast, we may have apriori explanations of how models should behave. In this paper, we formalize this notion as learning from explanation constraints and provide a learning theoretic framework to analyze how such explanations can improve the learning of our models. One may naturally ask, "When would these explanations be helpful?" Our first key contribution addresses this question via a class of models that satisfies these explanation constraints in expectation over new data. We provide a characterization of the benefits of these models (in terms of the reduction of their Rademacher complexities) for a canonical class of explanations given by gradient information in the settings of both linear models and two layer neural networks. In addition, we provide an algorithmic solution for our framework, via a variational approximation that achieves better performance and satisfies these constraints more frequently, when compared to simpler augmented Lagrangian methods to incorporate these explanations. We demonstrate the benefits of our approach over a large array of synthetic and real-world experiments.
There has been a considerable recent focus on generating explanations of complex black-box models so that humans may better understand their decisions. These can take the form of feature importance [31, 35], counterfactuals [31, 35], influential training samples [18, 43], etc. But what if humans were able to provide explanations for how these models should behave? We are interested in the question of how to learn models given such apriori explanations. A recent line of work incorporates explanations as a regularizer, penalizing models that do not exhibit apriori given explanations [33, 32, 15, 36]. For example, Rieger et al. [32] penalize the feature importance of spurious patches on a skin-cancer classification task. These methods lead to models that inherently satisfy "desirable" properties and, thus, are more trustworthy. In addition, some of these empirical results suggest that constraining models via explanations also leads to higher accuracy and robustness to changing test environments. However, there is no theoretical analysis to explain this phenomenon.
We note that such explanations can arise from domain experts and domain knowledge, but also other large "teacher" models that might have been developed for related tasks. An attractive facet of the latter is that we can automatically generate model-based explanations given unlabeled data points. For instance, we can use segmentation models to select the background pixels of images solely on
Figure 1: A restricted hypothesis class  $\mathcal{H}_{\phi, \alpha}$  (left). Our algorithmic solution to solve a proposed variational objective in Section 5 (right).
unlabeled data, which we can use in our model training. We thus view incorporating explanation constraints from such teacher models as a form of knowledge distillation into our student models [13].
In this paper, we provide an analytical framework for learning from explanations to reason when and how explanations can improve the model performance. We first provide a mathematical framework for model constraints given explanations. Casting explanations as functionals  $g$  that take in a model  $h$  and input  $x$  (as is standard in explainable AI), we can represent domain knowledge of how models should behave as constraints on the values of such explanations. We can leverage these to then solve a constrained ERM problem where we additionally constrain the model to satisfy these explanation constraints. Since the explanations and constraints are provided on randomly sampled inputs, these constraints are random. Nevertheless, via standard statistical learning theoretic arguments [38], any model that satisfies the set of explanation constraints on the finite sample can be shown to satisfy the constraints in expectation up to some slack with high probability. In our work, we term a model that satisfies explanations constraints in expectation, an CE model (see Definition 1). Then, we can capture the benefit of learning with explanation constraints by analyzing the generalization capabilities of this class of CE models (Theorem 3.2). This analysis builds off of a learning theoretic framework for semi-supervised learning of Balcan and Blum [1, 2]. We remark that if the explanation constraints are arbitrary, it is not possible to reason if a model satisfies the constraints in expectation based on a finite sample. We provide a detailed discussion on when this argument is possible in Appendix B,D. In addition, we note that our work also has a connection with classical approaches in stochastic programming [16, 4] and is worth investigating this relationship further.
Another key contribution of our work is concretely analyzing this framework for a canonical class of explanation constraints given by gradient information for linear models (Theorem 4.1) and two layer neural networks (Theorem 4.2). We focus on gradient constraints as we can represent many different notions of explanations, such as feature importance and ignoring spurious features. These corollaries clearly illustrate that restricting the hypothesis class via explanation constraints can lead to fewer required labeled data. Our results also provide a quantitative measure of the benefits of the explanation constraints in terms of the number of labeled data. We also discuss when learning these explanation constraints makes sense or is possible (i.e., with a finite generalization bound). We note that our framework allows for the explanations to be noisy, and not fully satisfied by even the Bayes optimal classifier. Why then would incorporating explanation constraints help? As our analysis shows, this is by reducing the estimation error (variance) by constraining the hypothesis class, at the expense of approximation error (bias). We defer the question of how to explicitly denoise noisy explanations to future work.
Now that we have provided a learning theoretic framework for these explanation constraints, we next consider the algorithmic question: how do we solve for these explanation-constrained models? In general, these constraints are not necessarily well-behaved and are difficult to optimize. One can use augmented Lagrangian approaches [33, 7], or simply regularized versions of our constrained problems [32] (which however do not in general solve the constrained problems for non-convex parameterizations but is more computationally tractable). We draw from seminal work in posterior regularization [9], which has also been studied in the capacity of model distillation [14], to provide a variational objective. Our objective is composed of two terms; supervised empirical risk and the discrepancy between the current model and the class of CE models. The optimal solution of our objective is also the optimal solution of the constrained problem which is consistent with our theoretical analysis. Our objective naturally incorporates unlabeled data and provides a simple way to
control the trade-off between explanation constraints and the supervised loss (Section 5). We propose a tractable algorithm that iteratively trains a model on the supervised data, and then approximately projects this learnt model onto the class of CE models. Finally, we provide an extensive array of experiments that capture the benefits of learning from explanation constraints. These experiments also demonstrate that the variational approach improves over simpler augmented Lagrangian approaches and can lead to models that indeed satisfy explanations more frequently.
Explainable AI. Recent advances in deep learning have led to models that achieve high performance but which are also highly complex [20, 11]. Understanding these complex models is crucial for safe and reliable deployments of these systems in the real-world. One approach to improve our understanding of a model is through explanations. This can take many forms such as feature importance [31, 35, 23, 37], high level concepts [17, 44], counterfactual examples [39, 12, 25], robustness of gradients [41], or influential training samples [18, 43].
In contrast to generating post-hoc explanations of a given model, we aim to learn models given apriori explanations. There has been some recent work along such lines. Koh et al. [19], Zarlenga et al. [45] incorporates explanations within the model architecture by requiring a conceptual bottleneck layer. Ross et al. [33], Rieger et al. [32], Ismail et al. [15], Stacey et al. [36] use explanations to modify the learning procedure for any class of models: they incorporate explanations as a regularizer, penalizing models that do not exhibit apriori given explanations; Ross et al. [33] penalize input gradients, while Rieger et al. [32] penalize a Contextual Decomposition score [26]. Some of these suggest that constraining models via explanations leads to higher accuracies and more robustness to spurious correlation, but do not provide analytical guarantees. On the theoretical front, Li et al. [22] show that models that are easier to explain locally also generalize well. However, Bilodeau et al. [3] show that common feature attribution methods without additional assumptions on the learning algorithm or data distribution do no better than random guessing at inferring counterfactual model behavior.
Learning Theory. Our contribution is to provide an analytical framework for learning from explanations that quantify the benefits of explanation constraints. Our analysis is closely related to the framework of learning with side information. Balcan and Blum [2] shows how unlabeled data can help in semi-supervised learning through a notion of compatibility between the data and the target model. This work studies classical notions of side information (e.g., margin, smoothness, and constraining). Subsequent papers have adapted this learning theoretic framework to study the benefits of representation learning [10] and transformation invariance [34]. On the contrary, our paper focuses on the more recent notion of explanations. Rather than focus on the benefits of unlabeled data, we characterize the quality of different explanations. We highlight that constraints here are stochastic, as they depend on data points which differs from deterministic constraints that have been considered in existing literature, such as constraints on the norm of weights (i.e., L2 regularization).
Self-Training. Our work can also be connected to the self-training literature [5, 42, 40, 8], where we could view our variational objective as comprising a regularized (potentially simpler) teacher model that encodes these explanation constraints into a student model. Our variational objective (where we use simpler teacher models) is also related to distillation, which has also been studied in terms of gradients [6].
3 Learning from Explanation Constraints
Let  $\mathcal{X}$  be the instance space and  $\mathcal{Y}$  be the label space. We focus on binary classification where  $\mathcal{Y} = \{-1,1\}$ , but which can be naturally generalized. Let  $\mathcal{D}$  be the joint data distribution over  $(\mathcal{X},\mathcal{Y})$  and  $\mathcal{D}_{\mathcal{X}}$  the marginal distribution over  $\mathcal{X}$ . For any classifier  $h:\mathcal{X}\to \mathcal{Y}$ , we are interested in its classification error  $\mathrm{err}(h)\coloneqq \operatorname*{Pr}_{(x,y)\sim D}(h(x)\neq y)$ , though one could also use other losses to define classification error. Our goal is to learn a classifier with small error from a family of functions  $\mathcal{H}$ . In this work, we use the words model and classifier interchangeably. Now, we formalize local explanations as functionals that take in a model and a test input, and output a vector:
Definition 1 (Explanations). Given an instance space  $\mathcal{X}$ , model hypothesis class  $\mathcal{H}$ , and an explanation functional  $g: \mathcal{H} \times \mathcal{X} \to \mathbb{R}^r$ , we say  $g(h, x)$  is an explanation of  $h$  on point  $x$  induced by  $g$ .
For simplicity, we consider the setting when  $g$  takes a single data point and model as input, but this can be naturally extended to multiple data points and models. We can combine these explanations with prior knowledge on how explanations should look like at sample points in term of constraints.
Definition 2 (Explanation Constraint Set). For any instance space  $\mathcal{X}$ , hypothesis class  $\mathcal{H}$ , an explanation functional  $g: \mathcal{H} \times \mathcal{X} \to \mathbb{R}^r$ , and a family of constraint sets  $\{C(x) \subseteq \mathbb{R}^r \mid x \in \mathcal{X}\}$ , we say that  $h \in \mathcal{H}$  satisfies the explanation constraints with respect to  $C$  iff:
$$
g (h, x) \in C (x), \forall x \in \mathcal {X}.
$$
In our definition,  $C(x)$  represents values that we believe our explanations should take at a point  $x$ . For example, "an input gradient of a feature 1 must be larger than feature 2" can be represented by  $g(h,x) = \nabla_{x}h(x)$  and  $C(x) = \{(x_1,\ldots ,x_d)\in \mathbb{R}^d\mid x_1 > x_2\}$ . In practice, human annotators will be able to provide the constraint set  $C(x^{\prime})$  for a random sample  $k$  data points  $S_{E} = \{x_{1}^{\prime},\dots ,x_{k}^{\prime}\}$  drawn i.i.d. from  $\mathcal{D}_{\mathcal{X}}$ . We then say that any  $h\in \mathcal{H}$ $S_{E}$ -satisfies the explanation constraints with respect to  $C$  iff  $g(h,x)\in C(x),\forall x\in S_E$ . We note that the constraints depend on random samples  $x_{i}^{\prime}$  and therefore are random. To tackle this challenge, we can draw from the standard learning theoretic arguments to reason about probably approximately satisfying the constraints in expectation. Before doing so, we first consider the notion of explanation surrogate losses, which will allow us to generalize the setup above to a form that is amenable to practical estimators.
Definition 3. (Explanation surrogate loss) An explanation surrogate loss  $\phi : \mathcal{H} \times \mathcal{X} \to \mathbb{R}$  quantifies how well a model  $h$  satisfies the explanation constraint  $g(h, x) \in C(x)$ . For any  $h \in \mathcal{H}, x \in \mathcal{X}$ :
1.  $\phi (h,x)\geq 0$
2. If  $g(h,x) \in C(x)$  then  $\phi (h,x) = 0$ .
For example, we could define  $\phi(h,x) = 1\{g(h,x) \in C(x)\}$ . Given such a surrogate loss, we can substitute the explanation constraint that  $g(h,x) \in C(x)$  with the surrogate  $\phi(h,x) \leq 0$ . We now have the machinery to formalize how to reason about the random explanation constraints given a random set of inputs. First, denote the expected explanation loss as  $\phi(h,\mathcal{D}) \coloneqq \mathbb{E}_{x \sim \mathcal{D}}[\phi(h,x)]$ . We are interested in models that satisfy the explanation constraints up to some slack  $\tau$  (i.e. approximately) in expectation. We define a learnability condition of this explanation surrogate loss as EPAC (Explanation Probably Approximately Correct) learnability.
Definition 4 (EPAC learnability). For any  $\delta \in (0,1),\tau >0$  the sample complexity of  $(\delta ,\tau)$  - EPAC learning of  $\mathcal{H}$  with respect to a surrogate loss  $\phi$  denoted  $m(\tau ,\delta ;\mathcal{H},\phi)$  is defined as the smallest  $m\in \mathbb{N}$  for which there exists a learning rule  $\mathcal{A}$  such that every data distribution  $\mathcal{D}_{\mathcal{X}}$  over  $\mathcal{X}$ , with probability at least  $1 - \delta$  over  $S\sim \mathcal{D}^m$ ,
$$
\phi (\mathcal {A} (S), \mathcal {D}) \leq \inf  _ {h \in \mathcal {H}} \phi (h, \mathcal {D}) + \tau .
$$
If no such  $m$  exists, define  $m(\tau, \delta; \mathcal{H}, \phi) = \infty$ . We say that  $\mathcal{H}$  is EPAC learnable in the agnostic setting with respect to a surrogate loss  $\phi$  if  $\forall \delta \in (0,1), \tau > 0$ ,  $m(\tau, \delta; \mathcal{H}, \phi)$  is finite.
Furthermore, for a constant  $\tau$ , we denote any model  $h \in \mathcal{H}$  with  $\tau$ -Approximately Correct Explanation where  $\phi(h, \mathcal{D}) \leq \tau$ , with a  $\tau$ -CE models. We define the class of  $\tau$ -CE models as
$$
\mathcal {H} _ {\phi , \mathcal {D}, \tau} = \{h \in \mathcal {H}: \phi (h, \mathcal {D}) \leq \tau \}. \tag {1}
$$
We simply use  $\mathcal{H}_{\phi, \tau}$  to denote this class of CE models. From natural statistical learning theoretic arguments, a model that satisfies the random constraints in  $S_{E}$  might also be a CE model.
Proposition 3.1. Suppose a model  $h$ $S_{E}$ -satisfies the explanation constraints then
$$
\phi (h, \mathcal {D} _ {\mathcal {X}}) \leq 2 R _ {k} (\mathcal {G}) + \sqrt {\frac {\ln (4 / \delta)}{2 k}},
$$
with probability at least  $1 - \delta$ , when  $k = |S_E|$  and  $\mathcal{G} = \{\phi(h, \cdot) \mid h \in \mathcal{H}\}$ .
We use  $R_{k}(\cdot)$  to denote Rademacher complexity; please see Appendix A where we review this and related concepts. Note that even when  $h$  satisfies the constraints exactly on  $S$ , we can only guarantee a bound on the expected surrogate loss  $\phi(h, \mathcal{D}_{\mathcal{X}})$ . We can achieve a bound similar to that in Proposition 3.1 via a single and simpler constraint on the empirical expectation  $\phi(h, S_E) =$
$\frac{1}{|S_E|}\sum_{x\in S_E}\phi (h,x)$ . We can then extend the above proposition to show that if  $\phi (h,S_E)\leq \tau$  then  $\phi (h,\mathcal{D}_{\mathcal{X}})\leq \tau +2R_k(\mathcal{G}) + \sqrt{\frac{\ln(4 / \delta)}{2k}}$ , with probability at least  $1 - \delta$ . Another advantage of such a constraint is that the explanation constraints could be noisy, or it may be difficult to satisfy them exactly, so  $\tau$  also serves as a slack. The class  $\mathcal{G}$  contains all surrogate losses of any  $h\in \mathcal{H}$ . Depending on the explanation constraints,  $\mathcal{G}$  can be extremely large. We remark that the surrogate loss  $\phi$  allows us to reason about satisfying an explanation constraint on a new data point and in expectation. However, for many constraints,  $\phi$  does not have a closed-form or is unknown on an unseen data point. The question of which types of explanation constraints are generalizable may be of independent interest, and we further discuss this in Appendix B and provide further examples of learnable constraints in Appendix D.
EPAC-ERM Objective. Let us next discuss combining the two sources of information: the explanation constraints that we set up in the previous section, together with the usual set of labeled training samples  $S = \{(x_{1},y_{1}),\ldots ,(x_{n},y_{n})\}$  drawn i.i.d. from  $\mathcal{D}$  that informs the empirical risk. Combining these, we get what we call EPAC-ERM objective:
$$
\min  _ {h \in \mathcal {H}} \frac {1}{n} \sum_ {i = 1} ^ {n} \ell \left(h, x _ {i}, y _ {i}\right) \text {s . t .} \frac {1}{k} \sum_ {i = 1} ^ {k} \phi \left(h, x _ {i} ^ {\prime}\right) \leq \tau . \tag {2}
$$
We provide a learnability condition for a model that achieve both low average error and surrogate loss in Appendix F.
3.1 Generalization Bound
We assume that we are in a doubly agnostic setting. Firstly, we are agnostic in the usual sense that there need be no classifier in the hypothesis class  $\mathcal{H}$  that perfectly labels  $(x,y)$ ; instead, we hope to achieve the best error rate in the hypothesis class,  $h^* = \arg \min_{h\in \mathcal{H}}\mathrm{err}_{\mathcal{D}}(h)$ . Secondly, we are also agnostic with respect to the explanations, so that the optimal classifier  $h^*$  may not satisfy the explanation constraints exactly, so that it incurs nonzero surrogate explanation loss  $\phi (h^{*},D) > 0$
Theorem 3.2 (Generalization Bound for Agnostic Setting). Consider a hypothesis class  $\mathcal{H}$ , distribution  $\mathcal{D}$ , and explanation loss  $\phi$ . Let  $S = \{(x_1, y_1), \ldots, (x_n, y_n)\}$  be drawn i.i.d. from  $\mathcal{D}$  and  $S_E = \{x_1', \ldots, x_k'\}$  drawn i.i.d. from  $\mathcal{D}_{\mathcal{X}}$ . With probability at least  $1 - \delta$ , for  $h \in \mathcal{H}$  that minimizes empirical risk  $\mathrm{err}_S(h)$  and has  $\phi(h, S_E) \leq \tau$ , we have
$$
\operatorname {e r r} _ {D} (h) \leq \operatorname {e r r} _ {D} \left(h _ {\tau - \varepsilon_ {k}} ^ {*}\right) + 2 R _ {n} \left(\mathcal {H} _ {\phi , \tau + \varepsilon_ {k}}\right) + 2 \sqrt {\frac {\ln (4 / \delta)}{2 n}},
$$
$$
\varepsilon_ {k} = 2 R _ {k} (\mathcal {G}) + \sqrt {\frac {\ln (4 / \delta)}{2 k}},
$$
when  $\mathcal{G} = \{\phi (h,x)\mid h\in \mathcal{H},x\in \mathcal{X}\}$  and  $h_\tau^* = \arg \min_{h\in \mathcal{H}_{\phi ,\tau}}\mathrm{err}_D(h)$ .
Proof. The proof largely follows the arguments in Balcan and Blum [2], but we use Rademacher complexity-based deviation bounds instead of VC-entropy. We defer the full proof to Appendix E.  $\square$
Our bound suggests that these constraints help with our learning by shrinking the hypothesis class  $\mathcal{H}$  to  $\mathcal{H}_{\phi, \tau + \varepsilon_k}$ , reducing the required sample complexity. However, there is also a trade-off between reduction and accuracy. In our bound, we compare against the best classifier  $h_{\tau - \varepsilon_k}^* \in \mathcal{H}_{\phi, \tau - \varepsilon_k}$  instead of  $h^*$ . Since we may have  $\phi(h^*, \mathcal{D}) > 0$ , if  $\tau$  is too small, we may reduce  $\mathcal{H}$  to a hypothesis class that does not contain any good classifiers. Recall that the generalization bound for standard supervised learning — in the absence of explanation constraints — is given by
$$
\operatorname {e r r} _ {D} (h) \leq \operatorname {e r r} _ {D} \left(h ^ {*}\right) + 2 R _ {n} (\mathcal {H}) + 2 \sqrt {\frac {\ln (2 / \delta)}{2 n}}.
$$
We can see the difference between this upper bound and the upper bound in Theorem 3.2 here as a possible notion of the goodness of an explanation constraint. We further discuss this in Appendix C.
Figure 2: Visualization of the piecewise constant function of  $\nabla_{x}h(x) - \nabla_{x}h^{\prime}(x)$  when  $h$  is a two layer NNs with 1 node. Background colors represent regions with non-zero value.
4 Gradient Explanations for Particular Hypothesis Classes
In this section, we further quantify the usefulness of explanation constraints on different concrete examples and characterize the Rademacher complexity of the restricted hypothesis classes. In particular, we consider an explanation constraint of a constraint on the input gradient. For example, we may want our model's gradient to be close to that of some  $h' \in \mathcal{H}$ . This translates to  $g(h, x) = \nabla_x h(x)$  and  $C(x) = \{x \in \mathbb{R}^d \mid \| x - \nabla_x h'(x) \| \leq \tau\}$  for some  $\tau > 0$ .
4.1 Gradient Explanations for Linear Models
We now consider the case of a uniform distribution on a sphere, and we use the symmetry of this distribution to derive an upper bound on the Rademacher complexity (full proof to Appendix H).
Theorem 4.1 (Rademacher complexity of linear models with a gradient constraint, uniform distribution on a sphere). Let  $\mathcal{D}_{\mathcal{X}}$  be a uniform distribution on a unit sphere in  $\mathbb{R}^d$ , let  $\mathcal{H} = \{h : x \mapsto \langle w_h, x \rangle \mid w_h \in \mathbb{R}^d, \|w_h\|_2 \leq B\}$  be a class of linear models with weights bounded by a constant  $B$ . Let  $\phi(h, x) = \theta(w_h, w_{h'})$  be a surrogate loss where  $\theta(u, v)$  is an angle between  $u, v$ . We have
$$
R _ {n} \left(\mathcal {H} _ {\phi , \tau}\right) \leq \frac {B}{\sqrt {n}} \left(\sin (\tau) \cdot p + \frac {1 - p}{2}\right),
$$
where  $p = \operatorname{erf}\left(\frac{\sqrt{d}\sin(\tau)}{\sqrt{2}}\right)$  and  $\operatorname{erf}(x) = \frac{2}{\sqrt{\pi}}\int_{0}^{x}e^{-t^2}dt$  is the standard error function.