Scalable streaming learning of dyadic relationships (שער נוסף בעברית: למידת יחסים דיאדיים ממידע זורם.)

Modern machine learning problems are often about learning a mapping between two high-dimensional domains. For example, learning a mapping from the domain of images to the domain of words, or learning which documents are similar to others. We refer to these mappings as dyadic relationships. In this thesis we focus on learning dyadic relationships encoded in matrix form. For example, a matrix mapping between image representations and semantic representations, or a matrix acting as a bilinear form encoding similarity between pairs of documents. Specifically, we address the challenge of learning dyadic relationships for large-scale highdimensional datasets. We employ two sets of tools in developing the methods presented here: First, we use the rich geometric and algebraic structure of matrices. This includes the Riemannian manifold structure of the set of low-rank matrices, decompositions of orthogonal matrices, and properties of positive definite matrices. In addition, we concentrate on streaming access algorithms. These are algorithms which have access to a small subset of the data samples or features at any one time. Streaming algorithms are key for handling huge datasets which do not fit in memory, and for obtaining fast predictions even before the entire data is available or processed. This thesis is based on four papers. First, we deal with the problem of learning a similarity measure between sparse high-dimensional instances, applied to image retrieval. We focus on building a fast algorithm which scales to large datasets, and investigate the role of symmetry and positive definiteness in its performance. The second paper presents two streaming algorithms for learning low-rank and low-rank positive definite matrices, applied to learning similarity measures and multi-label models. The algorithms are based on Riemannian stochastic gradient descent, and prove to be both fast and robust despite the highly non-convex nature of the low-rank constraint. The third paper deals with the challenge of learning interpretable relations how can an algorithm explain its output in terms which make sense to humans? The work is motivated by the need to understand a large dataset of high-resolution gene expression images in the mouse brain. The images exhibit elaborate multi-scale patterns, and the genes themselves exhibit complex interactions, making the data difficult to comprehend. Our method uses the semantic information gained from the rich existing knowledge on genes and their functions, and gives researchers means for exploring and understanding the dataset. Finally, the fourth paper deals with the general problem of efficiently learning orthogonal matrices, motivated by recent applications to tensor decompositions for method-of-moments estimation of latent variable models. We present a novel Riemannian coordinate descent algorithm, based on simple sparse orthogonal matrices called Givens rotations. We show our method produces better results for tensor decomposition compared with the stateof-the-art, as well as being faster and outperforming a state-of-the-art algorithm for sparse PCA. We conclude by discussing several future research directions stemming from the work presented here, as well as building on recent advancements by other researchers. Letter of contribution to chapters with joint authors Contributions to Chapter 2.1 • Jointly with Gal Chechik, designed and carried out the Caltech256 experiment (section 5.3). • Jointly with Gal Chechik, designed and carried out the parallel training experiment (section 5.4). • Jointly with Gal Chechik, designed and carried out the symmetry and positivity experiment (section 6). • Jointly with Gal Chechik, analyzed the above experiments and wrote corresponding sections 5.3, 5.4 and 6. • Performed and wrote the theoretical loss bound analysis (section 2.2). • Jointly with Gal Chechik and Samy Bengio, wrote the related work section. • Jointly with Gal Chechik, Samy Bengio and Varun Sharma, wrote the discussion (section 7). Contributions to Chapter 2.3 • Jointly with Noa Liscovitch, conceived and designed the project. • Designed the machine learning model presented and used in the paper. • Designed the similarity measures and interpretable similarity concept presented and used in the paper. • Jointly with Noa Liscovitch and Gal Chechik, designed and carried out the experiments and their analysis. • Wrote the paper jointly with Noa Liscovitch and Gal Chechik, was the lead writer of sections 2.2 and 2.3 and prepared figures 2, 4 and S2. • Jointly with Noa Liscovitch and Gal Chechik, wrote Introduction and Summary.