In this paper, we address learning problems for high dimensional data. Previously, oblivious random projection based approaches that project high dimensional features onto a random subspace have been used in practice for tackling highdimensionality challenge in machine learning. Recently, various non-oblivious randomized reduction methods have been developed and deployed for solving many numerical problems such as matrix product approximation, low-rank matrix approximation, etc. However, they are less explored for the machine learning tasks, e.g., classification. More seriously, the theoretical analysis of excess risk bounds for risk minimization, an important measure of generalization performance, has not been established for non-oblivious randomized reduction methods. It therefore remains an open problem what is the benefit of using them over previous oblivious random projection based approaches. To tackle these challenges, we propose an algorithmic framework for employing non-oblivious randomized reduction method for general empirical risk minimizing in machine learning tasks, where the original high-dimensional features are projected onto a random subspace that is derived from the data with a small matrix approximation error. We then derive the first excess risk bound for the proposed non-oblivious randomized reduction approach without requiring strong assumptions on the training data. The established excess risk bound exhibits that the proposed approach provides much better generalization performance and it also sheds more insights about different randomized reduction approaches. Finally, we conduct extensive experiments on both synthetic and real-world benchmark datasets, whose dimension scales to O(10), to demonstrate the efficacy of our proposed approach. Introduction Recently, the scale and dimensionality of data associated with machine learning and data mining applications have seen unprecedented growth, spurring the BIG DATA research and development. Learning from largescale ultrahigh-dimensional data remains a computationally challenging problem. The big size of data not only increases the memory footprint but also increases the computational costs pertaining to optimization. A popular approach for addressing the high-dimensionality challenge is Copyright c © 2017, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. to perform dimensionality reduction. Nowadays, randomized reduction methods are emerging to be attractive for dimensionality reduction. Compared with traditional dimensionality reduction methods (e.g., PCA and LDA), randomized reduction methods (i) can lead to simpler algorithms that are easier to analyze (Mahoney 2011); (ii) can often be organized to exploit modern computational architectures better than classical dimensional reduction methods (Halko, Martinsson, and Tropp 2011); (iii) can be more efficient without loss in efficacy (Paul et al. 2013). Generally, randomized reduction methods can be cast into two types: the first type of methods reduces a set of high-dimensional vectors into a low dimensional space independent of each other. These methods usually sample a random matrix independent of the data and then use it to reduce the dimensionality of the data. The second type of methods projects a set of vectors (in the form of a matrix) onto a subspace such that the original matrix can be well reconstructed from the projected matrix and the subspace. Therefore, the subspace to which the data is projected depends on the original data. These methods have been deployed for solving many numerical problems related to matrices, e.g., matrix product approximation, low-rank matrix approximation, approximate singular value decomposition (Boutsidis and Gittens 2013a; Halko, Martinsson, and Tropp 2011). To differentiate these two types of randomized reduction methods, we refer to the first type as oblivious randomized reduction, and refer to the second type as non-oblivious randomized reduction. We note that in literature oblivious and non-oblivious are used interchangeably with data-independent and data-dependent. Here, we use the terminology commonly appearing in matrix analysis and numerical linear algebra due to that the general excess risk bound depends on the matrix approximation error. However, we have not seen any comprehensive study on the statistical property (in particular the excess risk bound) of these randomized reduction methods applied to risk minimization in machine learning. The excess risk bound measures the generalization performance of a learned model compared to the optimal model from a class that has the best generalization performance. The excess risk bounds facilitate a better understanding of different learning algorithms and have the potential to guide us to design better algorithms (Kukliansky and Shamir 2015). It is worth noting that several studies have been devoted to understanding the theoretical properties of oblivious randomized reduction methods applied to classification and regression problems. For example, (Blum 2005; Shi et al. 2012; Paul et al. 2013) analyzed the preservation of the margin of SVM based classification methods with randomized dimension reduction. (Zhang et al. 2014; Yang et al. 2015; Pilanci and Wainwright 2015) studied the problem from the perspective of optimization. Nonetheless, these results are limited in the sense that (i) they focus on only oblivious randomized reduction where the data is projected onto a random subspace independent of the data; (ii) they depend heavily on strong assumptions of the training data or the problem, e.g., low-rank of the data matrix, linear separability of training examples, or the sparsity of optimal solution, and (iii) some of these results do not directly carry over to the excess risk bounds. To tackle the above challenges, we propose an algorithmic framework for employing non-oblivious randomized reduction (NOR) method to project the original high-dimensional features onto a random subspace that is derived from the original data. We study and establish the excess risk bound of the presented randomized algorithms for risk minimization. Different from previous results for oblivious randomized reduction methods, our theoretical analysis does not require assumptions of the training data or the problem, such as low-rank of the data matrix, linear separability of training examples, and the sparsity of optimal solution. When the data matrix is of low-rank or has a fast spectral decay, the excess risk bound of NOR is much better than that of oblivious randomized reduction based methods. Empirical studies on synthetic and real data sets corroborate the theoretical results and demonstrate the effectiveness of the proposed methods.
Output: Efficient Non-oblivious Randomized Reduction for Risk Minimization with Improved Excess Risk Guarantee