Gradient-based Training of Slow Feature Analysis by Differentiable Approximate Whitening

1 INTRODUCTION . Finding meaningful representations in data is a core challenge in modern machine learning as the performance in many goal-directed frameworks , such as reinforcement learning or supervised learning , is strongly influenced by the quality of the former . Usually , features are either domain specific or acquired through learning . Most currently successful approaches for either deep supervised learning ( Goodfellow et al . ( 2016 ) ) or reinforcement learning ( Sutton & Barto ( 1998 ) ) rely on a training signal , i.e. , a classification label or reward , to provide sufficient indication which features of the input data should be extracted to increase performance . However , in most real-world scenarios labels have to be provided by expert knowledge and reward signals are sparse . In unsupervised representation learning , one tries to find and apply a principle by which to extract meaning from data without assuming the availability of any goal-driven metrics . Examples for such principles are based on reconstruction error in principal component analysis or autoencoder networks ( Bourlard & Kamp ( 1988 ) ) , statistical dependence of extracted features in independent component analysis ( Comon ( 1994 ) ) , indistinguishability of synthetically generated data from samples of the input distribution in generative adversarial nets ( Goodfellow et al . ( 2014 ) ) . Further examples include : fitting the probability distribution of input data with variational autoencoders ( Kingma & Welling ( 2013 ) ) , neighborhood preservation used in locally linear embedding ( LLE , Roweis & Saul ( 2000 ) ) , Laplacian eigenmaps ( LEM , Belkin & Niyogi ( 2003 ) ) , as well as temporal coherence as used in slow feature analysis , ( SFA , Wiskott & Sejnowski ( 2002 ) ) or regularized slowness optimization ( Bengio & Bergstra ( 2009 ) ) . Temporal coherence is the focus of this work and has been shown to provide a useful proxy for extracting underlying causes from time-series data such as position , head direction , identity of spatial view similar to those observed in rodent brains ( Franzius et al . ( 2007 ) ) from ego-visual data . The principle has also been successfully applied to determining object configuration ( identity , position , angle ) from a tabletop view of moving objects ( Franzius et al . ( 2011 ) ) in the form of SFA . A graph-based generalization of SFA has been used to achieve ( at that time ) state-of-the-art age estimation results on the MORPH dataset in Escalante-B . ( 2017 ) . We propose a variant of SFA that approximately enforces the same constraints while being differentiable and thus allows training by gradient-descent-derived methods . This makes it possible to leverage the representational power of complex models , such as deep neural networks , and useful ideas from that domain ( dropout , batch normalization , activation regularization ) to extract slow and informative features from data . To demonstrate the applicability of this approach , we provide three distinct experimental evaluations . 2 RELATED WORK . 2.1 SLOWNESS-BASED METHODS . While the original proposal of SFA by Wiskott & Sejnowski ( 2002 ) uses non-linear basis functions as a method to introduce non-linearity to the otherwise linear model , this classical approach has two limitations : ( a ) The basis functions are fixed and thus have to be chosen beforehand by expert knowledge or trial-and-error , and ( b ) as the resulting model is shallow , its expressivity tends to scale unfavorably in the dimension of the expansion ( Raghu et al . ( 2016 ) ) compared to hierarchical extensions discussed later . In the case of polynomial expansion , expanding to degree d on e-dimensional input data results in ( d+e d ) -dimensional expanded data , e.g. , quadratically expanding grayscale images of 180×90 pixels results in an output dimension > 131 ·106 . Extracting a single feature with a linear model would thus require 5.7-times more parameters than the modern and powerful Xception network ( Chollet ( 2016 ) ) . An alternative approach to increase expressivity using expansions is to apply low-degree nonlinearities repeatedly in a receptive-field fashion , interlacing them with projection steps . This has been done in Hierarchical SFA ( HSFA , Franzius et al . ( 2007 ) and Escalante-B . & Wiskott ( 2016 ) ) and deep architectures in general . But while the latter are typically trained in an end-to-end fashion by variants of stochastic gradient-descent , HSFA is trained in a layer-wise procedure , solving the linear SFA problem consecutively for each layer as closed-form solution and thereby assuming that globally slow features can be composed of decreasingly local slow features . Escalante-B . & Wiskott ( 2016 ) shows that this assumption can be partially relaxed by adding information by-passes to the model . Compared to HSFA , our method allows directly modifying parameters in different layers to optimize a global slowness objective . Our algorithm is in line with recent work harnessing the temporal coherence prior by Bengio et al . ( 2013 ) in deep , self-supervised feature learning . This can be done by having a slowness term in the loss function . To avoid the trivial , constant solution , another term is usually added . For example , a reconstruction loss in an auto-encoder ( Goroshin et al . ( 2015a ) ) or one-step latent code prediction ( Goroshin et al . ( 2015b ) ) . Deep temporal coherence has also been considered via the lens of similarity metric learning , for example by optimizing a contrastive loss ( Jayaraman & Grauman ( 2016 ) , Mobahi et al . ( 2009 ) ) or a triplet loss ( Jansen et al . ( 2017 ) , Wang & Gupta ( 2015 ) ) . These metric learning approaches manage to avoid degenerate solutions by pushing points away from each other ( in feature space ) that are not temporal neighbors . Our work differs from these approaches as we seek to directly approximate the optimization problem as originally stated by Wiskott & Sejnowski ( 2002 ) in a deep learning setting , automatically handling the constant solution and ensuring that different features code for different information . 2.2 GRAPH-BASED METHODS . SFA has been generalized to graph-based SFA ( GSFA , Escalante-B . & Wiskott ( 2013 ) ) and generalized SFA ( Sprekeler ( 2011 ) ) . The former adapts only the objective function , while the latter additionally generalizes the constraints to D-orthogonality ( D being the degree matrix of an underlying graph ) . Following generalized SFA , standard SFA can be shown to be a special case of Laplacian Eigenmaps . Spectral Inference Networks ( SpIN , Pfau et al . ( 2018 ) ) utilize this connection to successfully derive a gradient-based SFA training as a special case . They are based on correcting a biased gradient when directly optimizing the Rayleigh-Quotient with respect to the model ’ s parameters . The constraints are implicitly enforced through the loss function as opposed to directly whitening the output . Similar to our work , SpINs allow for employing any architecture to find these embeddings . However , the whitening proposed in this paper is applicable to any loss function as it is part of the model architecture and not inherently a part of the optimized objective . The generalization of PowerSFA proposed in section 6.3 is close to GSFA , assuming regularity ( or rather : ignoring non-regularity ) in the graph for the orthogonality constraint . SpectralNets ( SN , Shaham et al . ( 2018 ) ) are another closely related approach in which a differentiable approximator is trained to learn spectral embeddings used in subsequent k-means clustering . Opposed to PowerSFA and SpINs , they split a single optimization step into two parts : an ortho-normalization step based on explicitly calculating the Cholesky decomposition of the batch covariance matrix to set and freeze the weights of a linear output layer , followed by a stochastic optimization step . While this can be considered end-to-end depending on the paradigm , Shaham et al . ( 2018 ) do not indicate if and how this can be implemented as fully differentiable architecture . 2.3 OTHER RELATED APPROACHES . In section 6.3 we consider a generalization of PowerSFA ’ s loss similar to GSFA and apply it to the NORB dataset ( LeCun et al . ( 2004 ) ) . We thereby loosely follow the experimental procedure in Hadsell et al . ( 2006 ) . The authors use a siamese neural network architecture ( Bromley et al . ( 1993 ) ) for optimizing pair-wise distances of embedded points to reflect similarity and dissimilarity structure of the data . In particular , they do not enforce orthogonality of the embeddings but rely on the optimization procedure to maximize informativeness . 3 SLOW FEATURE ANALYSIS . Slow Feature Analysis ( SFA ) is based on the hypothesis that interesting high-dimensional streams of data that vary quickly in time , are typically caused by a low number of underlying factors that vary comparably slow . Therefore , slowness can be used as a proxy criterion by which to extract meaningful representations of these low-dimensional underlying causes , even in the absence of labels . There is strong evidence in favor of this hypothesis , as it has been shown that features extracted by SFA tend to encode highly relevant information about the data-generating environments , e.g. , slow features encode and disentangle object identity , rotation , and position in visual tasks ( Franzius et al . ( 2011 ) ) as well as agent position and orientation from visual first-person recordings of random movement similar to place cells in rodents , or head-direction cells in primates ( Franzius et al . ( 2007 ) ) . The notion of extracting slow features from a time-series dataset has been formalized as a sequential optimization problem . Given a time-series { xt } t=0 ... τ with xt ∈ Rd , sequentially find continuous functions gi : Rd → R with : min gi 〈 ( gi ( xt+1 ) − gi ( xt ) ) 2 〉 t ( 1a ) s.t . 〈 gi ( xt ) 〉 t = 0 , ( 1b ) 〈 gi ( xt ) 2 〉 t = 1 , ( 1c ) 〈 gi ( xt ) gj ( xt ) 〉 t = 0 , ∀j < i ( 1d ) where 〈 · 〉 t is the time-average . The constraints ensure that each of the extracted features is informative ( decorrelated to all others , equation ( 1d ) ) and non-trivial ( unit variance , equation ( 1c ) ) . Originally , solutions to SFA directly were only proposed for the space of affine functions gi ∈ G , for which a closed-form solution exists , and in a kernelized version that requires strong regularization . 4 ( APPROXIMATE ) WHITENING . The standard implementation of linear SFA ( Zito et al . ( 2008 ) ) is based on computing the closedform solution to a generalized eigenvalue problem ( Berkes & Wiskott ( 2005 ) ) . Another approach is to first whiten the time-series data , followed by a projection onto the minor components of the difference time-series { ẋt = xt+1 − xt } t=0 ... τ−1 . Whitened data has three important properties : ( a ) it is mean-free ( constraint ( 1b ) ) , ( b ) has unit variance if projected onto an arbitrary unit vector ( constraint ( 1c ) ) , and ( c ) projections onto orthonormal vectors are decorrelated ( constraint ( 1d ) ) . For a dataset X̃ ∈ Rd×N with N and d being size and dimension of the dataset , respectively , the corresponding whitened dataset is defined as X = WX̃ with W = D− 1 2UT and C = UDUT ∈ Rd×d being the whitening matrix and the canonical eigendecomposition of the covariance matrix , respectively . U ’ s columns contain the eigenvectors of C and D contains the corresponding eigenvalues on its diagonal . As C is typically assumed to be positive definite , D− 1 2 is well-defined . One widely used method to extract eigenvector/-value pairs is power iteration . Starting from a random vector u0 ∈R Rd , repeatedly applying : ui+1 = Cui ‖Cui‖ converges to the eigenvector u corresponding to the largest ( absolute ) eigenvalue λ . The eigenvalue can then be extracted as λ = ‖Cu‖ and the spectral component corresponding to this eigenvector can be removed as C← C− λuuT . Repeating the procedure converges to the eigenvector corresponding to the next largest eigenvalue and so on . We use a previously fixed number of iterations for this method as experiments have shown that approximate whitening is enough to take meaningful optimization steps . In practice , a relatively small number of iterations results in acceptable whitening for most non-degenerate cases . Data : covariance matrix C , number of iterations Niter , data dimension d Result : whitening matrix W W← { 0 } d×d for i = 0 ; i < d ; i++ do Sample r ∼ U [ −1 , 1 ] d for j = 0 ; j < Niter ; j++ do r← Cr‖Cr‖ end λ← ‖Cr‖ C← C− λrrT wi· = 1√ λ rT end return W Algorithm 1 : Constructing W by power iteration Note that in algorithm 1 each operation is differentiable with respect to C .

Summary: The authors state a clear summary of their contribution to Slow Feature Analysis (SFA) in Section 5: "The key idea for gradient-based SFA is that a whitening layer can be applied subsequently to any differentiable architecture (such as deep neural networks) to enforce outputs that approximately obey the SFA constraints while still being a differentiable architecture.” As a result, the proposed method can replace the previous SFA pipeline of [fixed, non-linear feature extraction + learned linear feature extraction] with simply end-to-end [learned, non-linear feature extraction]. The resulting method is thus more flexible and expressive and less hand-crafted than traditional SFA approaches. The paper shows experiments that validate that the method can learn meaningful representations in practice.