Title: A Theory of PAC Learnability under Transformation Invariances
Authors: Han Shao
Omar Montasser
Avrim Blum
Venue: NeurIPS 2022
Keywords: transformation invariance
data augmentation
PAC learning
sample complexity

Abstract: Transformation invariances are present in many real-world problems. For example, image classification is usually invariant to rotation and color transformation: a rotated car in a different color is still identified as a car. Data augmentation, which adds the transformed data into the training set and trains a model on the augmented data, is one commonly used technique to build these invariances into the learning process. However, it is unclear how data augmentation performs theoretically and what the optimal algorithm is in presence of transformation invariances. In this paper, we study PAC learnability under transformation invariances in three settings according to different levels of realizability: (i) A hypothesis fits the augmented data; (ii) A hypothesis fits only the original data and the transformed data lying in the support of the data distribution; (iii) Agnostic case. One interesting observation is that distinguishing between the original data and the transformed data is necessary to achieve optimal accuracy in setting (ii) and (iii), which implies that any algorithm not differentiating between the original and transformed data (including data augmentation) is not optimal. Furthermore, this type of algorithms can even ``harm'' the accuracy. In setting (i), although it is unnecessary to distinguish between the two data sets, data augmentation still does not perform optimally. Due to such a difference, we propose two combinatorial measures characterizing the optimal sample complexity in setting (i) and (ii)(iii) and provide the optimal algorithms.

TL;DR: We study the PAC sample complexity of learning under transformation invariances and the performance of data augmentation.

Review discussion:
1) As far as I know, data augmentation techniques are rather simple in the case of image data, but not so simple in the case of textual data. Say, to implement the technique that the authors mentioned at the beginning of the paper (replacement of noun phrases), some external resources will be required (for example, an already trained constituent parser) which we may not assume to be readily available for a wide variety of languages.
Therefore, it seems to me that the proposed theory has a limited scope.
2) The proposed theory does not provide a simple characterization of realizable and agnostic learnability, as does the VC dimension in classical PAC learning. In the proposed theory, different modes of realizability entail different characterizations of learnability.
3) The proposed theory does not provide such a simple principle as the ERM in classical PAC learning. The optimal learning algorithm is again conditioned by the realizability mode.
Moreover, in some cases the optimal algorithm is transductive (based on a 1-inclusion-graph predictor), i.e. to calculate the label on any new test point, the entire training set will be used (as it is, for example, in the k nearest neighbors algorithm), which calls into question the computational efficiency of learning.
Perhaps I missed it in the papaer, but I have this question: if we perform data augmentation under standard PAC learning, but define the sample size as the number of examples in the original (non-augmented) dataset, how does this compare with your estimate of the sample complexity $\tilde{O}\left(\frac{\text{VC}_{\text{ao}}(\mathcal{H},\mathcal{G})}{\epsilon}\right)$ in the invariantly realizable case?
It seems to me that the proposed theory has a limited scope. See item 1 in Strengths and Weaknesses above.
No
Thanks for your review!
To your question on application in textual data: To clarify, this work considers the setting where the transformations and how to apply these transformations are known. The difficulty of applying transformations in textual data is an interesting but totally different question.
To your question on a simple characterization of realizable/agnostic learnability: To clarify,  we have three settings in this paper, which are the invariantly realizable setting, the relaxed realizable setting and the agnostic setting. The relaxed realizable setting and agnostic setting are analogous to the realizable and agnostic setting in classical PAC learning. And we actually have a single simple characterization $VC_{ao}$ for both settings (Theorem 7 and 8). For the invariantly realizable setting, it has a different characterization $VC_o$ because it has a further constraint on the target function, i.e., the target function has to be invariant (see line 121-122 for a formal definition).
Hence, we have a different characterization in this setting since we have more prior information about the target function.
To your question on a simple principle as ERM in PAC learning: To clarify, we indeed have a simple principle called ERM-INV (which is a variant of ERM) in the relaxed realizable and agnostic settings. This algorithm is mentioned in line 363, and the details are provided in Appendix H (line 773-780) due to space limitations. In the invariantly realizable setting, as mentioned above, there is more prior information about the target function and because we exploit this, our algorithm for this setting is different. Whether there exists a unifying optimal learning rule in all three settings is an interesting open question.
We didn’t fully understand your last question "if we perform data augmentation under standard PAC learning, but define the sample size as the number of examples in the original (non-augmented) dataset, how does this compare with your estimate of the sample complexity in the invariantly realizable case?". To clarify, throughout the paper and in all of our settings, we define the sample complexity as the number of examples needed from the original distribution (without any augmentations).  If "performing DA under standard PAC learning" means applying DA without transformation invariance assumption, then DA will fail in the worst case since the transformed data might not be labeled the same as the original data (e.g., applying rotations in MNIST). We are happy to clarify more if you could elaborate on this or have more questions.
Oh right, PAC learning from augmented data may fail without transformation invariance assumption.
I am satisfied with your answers to my questions. I will update my score.
Thank you for the detailed response.
### **Strength**
- The sample complexity bounds concerning DA depend on the VC dimension of/across the orbits, depending on the exact setting. These bounds provide a new theoretical lens to examine data augmentation.
For example, it is argued from the VC dimension bounds that
- In the invariantly realizable setting, DA helps compared to ERM.
However, it is not necessarily optimal.
- In the relaxed realizable setting, DA may actually hurt.
- This paper also proposes an ERM-based algorithm that achieves the optimal sample complexity. The design of this algorithm is quite interesting in my opinion: It works by first checking whether the test instance can be generated by a transformation of the input. If not, it outputs the ERM prediction.
- This paper first provides a rigorous mathematical definition of PAC learning under group transformations. I find this formulation nice to read. The contributions and their proofs are well laid out.
- I find the writing of this paper to be easy to follow. I have some minor comments mentioned below.
- The discussions section provides a candid narrative of the limitations of this paper.
### **Weakness**
- The discussion concerning the 1-inclusion-graph, starting from L273, could be improved to better explain this concept.
- In the relaxed realizable setting, the sample complexity of DA depends on a certain edge orientation notion $\mu(\mathcal{H}, \mathcal{G})$. It might help provide some more explanations to give the intuition behind the notion.
- Some of the proofs in the appendix could use more work to give greater clarity.
- L762: The sentence following "Every labeling" is broken.
- In the problem setup, it seems that the transformations are assumed to be known a prior. What if they are not given?
For example, works like Autoaugment often require finding the "right" transformation to apply for a particular task/dataset.
The authors provided a discussion of the limitations of this paper in the main text. The potential negative societal impact is not discussed (possibly but due to the technical nature of the paper); this should be addressed in a revised version.
No
Thanks for your review, especially for checking the details in the appendix!
You are totally correct that the transformations are assumed to be known in this work. Learning under unknown transformations will be an interesting future direction. One intuitive way is applying an augmentation policy searching method like Autoaugment to learn the transformations first and then using the methods in this work based on the learned transformations.
But then the results should depend on the quality of the transformations we have learned.
### Strengths:
1. This paper addresses two important problems: PAC learnability under transformation invariance and theoretical understanding of data augmentation.
2. The theoretical analyses are sound and interesting — the two “invariant versions” of VC-dimension are novel and the lower and upper sample complexity bounds in each realizatiliby case are comprehensive.
3. The conclusions, suggesting that the DA can be sub-optimal in practice and using the 1-inclusion-graph-based algorithm could help achieve the optimal sample complexity, are inspiring.
### Weaknesses:
1. My major concern is to what extent the theoretical results in this paper hold on real data and how much practical insight we could gain from this paper.
Specifically,
1. The two proposed combinatorial complexity measures seem hard to evaluate in practice. Evaluating the VC-dimension of a neural network class while limiting data points to lie in disjoint orbits seems non-trivial to me. I wonder how many previous results on VC-dimension calculation for neural networks could be adapted here. If we cannot evaluate the proposed combinatorial complexity measures in practice, how can we gain insight from the sample complexity results based on it?
2. Many of the main theoretical results, including the arbitrarily large gap between different versions of VC-dimension and all the lower bounds of the sample complexity (which are used to show the sub-optimality and harm of data augmentation), are based on specific data distribution constructions. I understand that these are common proof practices for VC-dimension but am also wondering to what extent they hold on real data?
2. Some descriptions could be made clearer.
For example, throughout the paper I do not see how the probability measure on the transformation group is defined for the data-augmented ERM.
1. I suggest that the authors give an intuitive explanation for the term “distinguish between the original and transformed data” in the main text.
2. Readers could also benefit from an intuitive description about the 1-inclusion-graph-based learning algorithm in the main text.
Yes, the authors address the limitation of assuming invariant data distributions.
No
Thank you for your review!
For evaluating the proposed combinatorial complexity measures, we agree that evaluating these measures will give us a better understanding of the quantitative gaps introduced in this work, and evaluating them efficiently is non-trivial in practice. How to evaluate them in neural networks would be an interesting future direction. Without evaluating these measures, we can still gain some qualitative insights, e.g., DA is *suboptimal* (implied by Theorem 2); DA can *harm* in some cases (implied by Theorem 5). There are also some practical insights for algorithm design.
For example, our optimal algorithm in the invariantly realizable setting (in Section 4.2) takes the invariance over the test point into account, which provides some theoretical justification for test-time adaptation such as [Wang et al., 2021].  Our Theorem 5 shows that distinguishing between the original and the transformed data (which means that algorithms are only given the augmented data $GS_{trn}$ without knowing the original training set $S_{trn}$) is important in the relaxed realizable setting and agnostic setting, which provides some high-level hints on designing practical algorithms.
To your question about to what extent the theoretical results hold on real data:
* The arbitrarily large gaps between the different versions of VC-dimension: To clarify, the gaps between different dimensions do not depend on data distributions as these dimensions are distribution independent.
But the gaps depend on instance space X, transformation set G and hypothesis class H and they can be large in real cases. Consider a simple case, where the natural data is k images and the transformation set is rotation by 0, 360/n, 2*360/n, ..., (n-1)*360/n degrees for some integer n. For an expressive hypothesis class (e.g., neural networks) that can shatter all rotated versions of these images, we have VC dimension = nk and vco = vcao = k. For a hypothesis class composed of all hypotheses labeling all upright images and their upside-down variations differently, we have VC dimension = (n-1)k, vcao = k and vco = 0.
* The lower bounds of sample complexity: The constructions in the lower bounds of the sample complexity depend on data distribution.
But the dependence on the dimensions (ignoring the dependence on $\epsilon$) holds for a wide class of data distributions. Let's consider a specific example of classifying land birds vs water birds. The natural data is k images of land birds with land background and water birds with water background. The transformation contains keeping the current background and changing the background from land (water) to water (land).
* For the lower bound used to show the sub-optimality of DA (Theorem 2): Let the marginal data distribution be any distribution with support being half of the k original images (which can be extended to any constant fraction of k images). Let the hypothesis class be composed of all hypotheses classifying exactly half of the k images with the original background incorrectly (and the other k/2 images with the original background and all k images with changed background correctly). We can show that DA needs at least vcao = k/2 samples using the same proof techniques of our Theorem 2.
But the optimal algorithm only needs zero samples to achieve zero error given that we are in the invariantly realizable setting, which shows the sub-optimality of DA.
* For the lower bound used to show the harm of DA (Theorem 5): If we consider a simple hypothesis class, which predicts only based on backgrounds.
That is to say, $H=$ {$h_1,h_2$} with $h_1$ predicting all images with water background as water birds and $h_2$ predicting all images with water background as land birds. Let the data distribution be any distribution with distribution support being all the original images.
Then given any training data, $h_1$ and $h_2$ have the same empirical loss on the augmented training data.
Thus, for any unobserved image, DA will make a mistake with constant probability.
Hence DA requires at least $\Omega(k)$ samples. This can be proved by the proof techniques of Theorem 5 with Theorem 6.
But it is direct to check that we only need one sample if applying standard ERM, showing the harm of DA.
We will add the real data examples in the revised version. Thanks for your question!
Probability measure on the transformation group: To clarify, we didn't define any probability measure on the transformation group. As introduced in line 160-165, if there exists an $h\in H$ s.t. $h(x) = y$ for all $(x, y) \in G S_{trn}$, data-augmented ERM will output such an $h$; If there does not exist such an $h$, data-augmented ERM outputs an $h\in H$ with the minimal loss $L(GS_{trn}, h)$. Here the loss can be any function, which can be defined based on any probability measure on the transformation group. Thanks for pointing this out and we will make it clearer.
We will clarify the 1-inclusion-graph algorithms further.
My concerns are addressed. I have increased my score by one. Looking forward to seeing the revised version.

Meta-review:
role: AC
data: metareview: All reviewers are positive. 

Clear accept.
role: PC

Decision: Accept