Sparse Filtering is a popular feature learning algorithm for image classification pipelines. In this paper, we connect the performance of Sparse Filtering with spectral properties of the corresponding feature matrices. This connection provides new insights into Sparse Filtering; in particular, it suggests early stopping of Sparse Filtering. We therefore introduce the Optimal Roundness Criterion (ORC), a novel stopping criterion for Sparse Filtering. We show that this stopping criterion is related with pre-processing procedures such as Statistical Whitening and demonstrate that it can make image classification with Sparse Filtering considerably faster and more accurate. Introduction Standard ways to improve image classification are to collect more samples or to change the representation and the processing of the data. In practice, the number of samples is typically limited, so that the second approach becomes relevant. An important tool for this second approach are feature learning algorithms, which aim at easing the classification task by transforming the data. Recently proposed deep learning methods intend to jointly learn learn a feature transformation and the classification (Krizhevsky, Sutskever, and Hinton 2012). In this work, however, we focus on unsupervised feature learning, especially on Sparse Filtering, because of their simplicity and scalability. Feature learning algorithms for image classification pipelines typically consists of three steps: pre-processing, (un)supervised dictionary learning, and encoding. An abundance of procedures is available for each of these steps, but for accurate image classification, we need procedures that are effective and interact beneficially with each other (Agarwal and Triggs 2006; Coates and Ng 2011; Coates, Ng, and Lee 2011; Jia, Huang, and Darrell 2012; Le 2013; LeCun, Huang, and Bottou 2004). Therefore, a profound understanding of these procedures is crucial to ensure accurate results and efficient computations. In this paper, we study the performance of Sparse Filtering (Ngiam et al. 2011) for image classification. Our main contributions are: Copyright © 2015, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. • we show that Sparse Filtering can strongly benefit from early stopping; • we show that the performance of Sparse Filtering is correlated with spectral properties of feature matrices on tests sets; • we introduce the Optimal Roundness Criterion (ORC), a stopping criterion for Sparse Filtering based on the above correlation, and demonstrate that the ORC can considerably improve image classification. Feature Learning for Image Classification Feature learning algorithms often consist of two steps: In a first step, a dictionary is learned, and in a second step, the samples are encoded based on this dictionary. A typical dictionary learning step for image classification is sketched in Figure 1: First, random patches (samples) are extracted from the training images. These patches are then pre-processed using, for example, Statistical Whitening or Contrast Normalization. Finally, an unsupervised learning algorithm is applied to learn a dictionary from the pre-processed patches. Once a dictionary is learnt, several further steps need to be applied to finally train an image classifier, see, for example, (Coates and Ng 2011; Coates, Ng, and Lee 2011; Jia, Huang, and Darrell 2012; Le 2013). Our pipeline is similar to the one in (Coates and Ng 2011): We extract square patches comprising 9 × 9 pixels, pre-process them with Contrast Normalization1 and/or Statistical Whitening, and finally pass them to Random Patches or Sparse Filtering. (Note that our outcomes differ slightly from those in (Coates and Ng 2011) because we use square patches comprising 9×9 pixels instead of 6×6 pixels.) Subsequently, we apply soft-thresholding for encoding, 4×4 spatial max pooling for extracting features from the training data images, and finally L2 SVM classification (cf. (Coates and Ng 2011)). Numerous examples show that feature learning can considerably improve classification. Therefore, insight in the underlying principles of feature learning algorithms such as Statistical Whitening and Sparse Filtering is of great interest. Contrast normalization consists of subtracting the mean and dividing by the standard deviation of the pixel values. ar X iv :1 40 9. 46 89 v2 [ cs .C V ] 2 4 M ay 2 01 5 Training images Extraction of random patches Pre-processing Unsupervised learning Dictionary Figure 1: A typical dictionary learning step. Statistical Whitening and Contrast Normalization are examples for preprocessing procedures; Random Patches and Sparse Filtering are examples for unsupervised learning procedures. In mathematical terms, a feature learning algorithm provides a transformation F : Rl×p → Rn×p X 7→ F(X) (1) of an original feature matrix X ∈ Rl×p to a new feature matrix F(X) ∈ Rn×p. We adopt the convention that the rows of the matrices correspond to the features, the columns to the samples; this convention implies in particular that l ∈ N is the number of original features, p ∈ N the number of samples, and n ∈ N the number of new features. The Optimal Roundness Criterion Roundness of Feature Matrices Feature learning can be seen as trade-off between reducing the correlations of the feature representation and preservation of relevant information. This trade-off can be readily understood looking at Statistical Whitening. For this, recall that pre-processing with Statistical Whitening transforms a set of image patches into a new set of patches by changing the local correlation structure. More precisely, Statistical Whitening transforms patches XPatch ∈ R ′×p (n′ < n), that is, subsets of the entire feature matrix, into new patches FPatch(XPatch) such that FPatch(XPatch)FPatch(XPatch) = n′ In′ . Statistical Whitening therefore acts locally: while the correlation structures of the single patches are directly and radically changed, the structure of the entire matrix is affected only indirectly. However, these indirect effects on the entire matrix are important for the following. To capture these effects, we therefore introduce the roundness of a feature matrix F := F(X) given an original feature matrix X . On a high level, we say that the new feature matrix F is round if the spectrum of the associated Gram matrixFF ∈ Rn×n is narrow. To specify this notion, we denote the ordered eigenvalues of FF by σ1(F ) ≥ · · · ≥ σn(F ) ≥ 0 and their mean by σ(F ) := 1 n ∑n i=1 σi(F ) and define roundness as follows: Definition 1. For any matrix F 6= 0, we define its roundness as
Output: Compute Less to Get More: Using ORC to Improve Sparse Filtering