Sparse Methods for Machine Learning: Theory and Algorithms
Channel: VideoLecturesChannel
Date: 2012-12-11

VideoLectures.Net
View the talk in context: http://videolectures.net/nips09_bach_smm/
View the complete 23rd Annual Conference on Neural Information Processing Systems (NIPS), Vancouver 2009: http://videolectures.net/nips09_vancouver/
Speaker: Francis R. Bach, INRIA - SIERRA project-team 

License: Creative Commons CC BY-NC-ND 3.0
More information at http://videolectures.net/site/about/
More talks at http://videolectures.net/

Regularization by the L1-norm has attracted a lot of interest in recent years in statistics, machine learning and signal processing. In the context of least-square linear regression, the problem is usually referred to as the Lasso or basis pursuit. Much of the early effort has been dedicated to algorithms to solve the optimization problem efficiently, either through first-order methods, or through homotopy methods that leads to the entire regularization path (i.e., the set of solutions for all values of the regularization parameters) at the cost of a single matrix inversion. A well-known property of the regularization by the L1-norm is the sparsity of the solutions, i.e., it leads to loading vectors with many zeros, and thus performs model selection on top of regularization. Recent works have looked precisely at the model consistency of the Lasso, i.e., if we know that the data were generated from a sparse loading vector, does the Lasso actually recover the sparsity pattern when the number of observations grows? Moreover, how many irrelevant variables could we consider while still being able to infer correctly the relevant ones? The objective of the tutorial is to give a unified overview of the recent contributions of sparse convex methods to machine learning, both in terms of theory and algorithms. The course will be divided in three parts: in the first part, the focus will be on the regular L1-norm and variable selection, introducing key algorithms and key theoretical results. Then, several more structured machine learning problems will be discussed, on vectors (second part) and matrices (third part), such as multi-task learning, sparse principal component analysis, multiple kernel learning and sparse coding.

0:56  Supervised learning and regularization
2:38  Regularizations (1)
2:58  Regularizations (2)
3:43  ℓ2 vs. ℓ1 - Gaussian hare vs. Laplacian tortoise
5:14  Lasso - Two main recent theoretical results (1)
5:53  Lasso - Two main recent theoretical results (2)
6:26  Going beyond the Lasso (1)
7:13  Going beyond the Lasso (2)
7:43  Going beyond the Lasso (3)
7:58  Going beyond the Lasso (4)
8:38  Sparse methods for machine learning, Outline
9:03  Why ℓ1-norm constraints leads to sparsity?
10:26  ℓ1-norm regularization (linear setting)
11:26  A review of nonsmooth convex analysis and optimization
11:53  Optimality conditions for smooth optimization Zero gradient (1)
12:58  Optimality conditions for smooth optimization Zero gradient (2)
13:32  Directional derivatives - convex functions on Rp
15:43  Optimality conditions for convex functions
16:08  Directional derivatives for ℓ1-norm regularization
16:48  Optimality conditions for ℓ1-norm regularization