Learnable Group Transform For Time-Series

1 INTRODUCTION . The selection of the time-frequency representation for analyzing , classifying , and predicting time-series has long been studied ( Coifman & Wickerhauser , 1992 ; Mallat & Zhang , 1993 ; Gribonval & Bacry , 2003 ) . To this day , the front-end processing of time-series remains a keystone toward the improvement of a wealth of applications such as health-care ( Saritha et al. , 2008 ) ) , environmental sound ( Balestriero et al. , 2018 ; Lelandais & Glotin , 2008 ) , and seismic data analysis ( Seydoux et al. , 2016 ) . The common denominator of the recorded signals in these fields is their undulatory behavior . While these signals share this common behavior , two significant factors imply the need of learning the representation : 1 ) time-series are intrinsically different because of their physical nature , 2 ) the machine learning task can be different even within the same type of data . Therefore , the representation should be induced by both the signal and the task at hand . An all too common approach to performing inference on time-series consists of building a Deep Neural Network ( DNN ) that operates on a spectral decomposition of the time-series such as Wavelet Transform ( WT ) or Mel Frequency Spectral Coefficients ( MFSC ) . The selection of the judicious transform is either performed by an expert in the signal at hand , or by considering the aforecited selection methods and their derivatives . However , an inherent drawback is that the selection of the time-frequency transform is often achieved with criteria that do not align with the task . For instance , a selection based on the sparsity of the representation while the task is the classification of the signals . Besides , these selection methods and transformations require substantial cross-validations of a large number of hyperparameters such as mother filter family , number of octaves , number of wavelets per octave , size of the window ( Cosentino et al. , 2017 ) . To alleviate these drawbacks , Ravanelli & Bengio ( 2018 ) ; Balestriero et al . ( 2018 ) ; Cakir et al . ( 2016 ) ; Zeghidour et al . ( 2018 ) investigated the learnability of a mother filter . This learnable mother filter is transformed by deterministic affine maps . These transformations constitute the filter-bank . The representation of the signal is obtained by convolving the filter-bank atoms with the signals . Recently , Khan & Yener ( 2018 ) investigated the learnability of the affine transformations , that is , the sampling of the dilation parameter of the affine group inducing the wavelet filter-bank . Optimized jointly with the DNN , their method allows an adaptive transformation of the mother filter . Another approach consists of building equivariant-invariant representations . In Mallat ( 2012 ) ; Bruna ( 2013 ) they propose a translation-invariant representation , the Scattering Transform , which is stable under the action of small diffeomorphisms . In Oyallon et al . ( 2018 ) ; Cohen & Welling ( 2016 ) , they focus on equivariant-invariant representations for images , which reduces the sample complexity and endow DNN ’ s layers with interpretability . In this work , we focus on GT , which is achieved by taking the inner product between the filter-bank , which is built by taking the action of a transformation map on a mother filter , and the signal . Well-known GTs are the Short-Time Fourier Transform ( STFT ) and the Continuous Wavelet Transform ( CWT ) . We propose to extend these GTs and improve their flexibility by introducing the Learnable Group Transform ( LGT ) by 1 ) generalizing the affine transformations of a mother filter leading to wavelet filter-bank by introducing non-linear transformations ( Section 3.1 ) , 2 ) proposing a parameterization of such non-linear map such that it can be learned efficiently and jointly with any DNN , ( Sections 3.2 , 3.3 ) , 3 ) Replacing the affine transformations of CWT by non-linear maps allows for greater flexibility in the learnable spectral decomposition which displays different equivariance properties ( Section 3.4 ) . This flexibility improves the linearization capability of the representation as it eases the learning of a spectral decomposition that is able to discard intricate patterns in the time-series that are nuisances . This specific transformation of a filter induces a filter with a non-linear instantaneous phase , which in turn , allows to span filters a la chirplets , which are of interest in a variety of domains such as biology and medicine , mechanics and vibrations , and sonar systems ( Flandrin , 2001 ) . Also , this generalization implies that for fixed network topology , replacing the learnable affine group with the continuous group leads to a larger class of spannable functions , which improves the approximation property of the DNN at hand ( Winkler & Le , 2017 ; Balestriero & Baraniuk , 2018 ) . In order to show the generality of our approach , we apply our algorithm on two diverse time-series classification problems ( Section 4 ) . 2 BACKGROUND AND NOTATIONS . We first highlight the properties of particular GTs by expressing their time-frequency tiling . 2.1 TIME-FREQUENCY TILING . The spread of a filter and its Fourier transform are inversely proportional as per the Heisenberg uncertainty principle ( Mallat , 1999 ) . Following this principle , we can observe that in the case of STFT ( respectively WT with a Gabor wavelet ) , at a given time τ , the signal is transformed by a window of constant bandwidth ( respectively proportional bandwidth ) modulated by complex exponential resulting in a uniform tiling ( respectively proportional ) on the frequency axis , Figure ( 1 ) . In the case of a chirp-like filter , as proposed in Baraniuk & Jones ( 1996 ) , each tile is a sheared rectangular , more generally , an affinely trans- formed rectangular . In this case , as well , the lower bound area of the sheared rectangular is constrained by the uncertainty principle . As such , the understanding of the benefits of various time-frequency decompositions can be achieved by analyzing how they tile the time-frequency plane . For instance , in the case of WT , the precision in frequency degrades as the frequency increases while its precision in time increases ( Mallat , 1999 ) . In the case of STFT , the uniform tiling implies that the precision is constant along the frequency axis . In our proposed framework , the LGT allows for an adaptive tiling , as illustrated in Figure ( 1 ) such that the trade-off between time and frequency precision depends on the task and data . 3 LEARNABLE GROUP TRANSFORM . To extend the filter-bank derivation as proposed in a wavelet decomposition we introduce a learnable group transform . We now define a subset of invertible maps on R enabling the transformation of a mother filter . Then , we provide a parameterization of such functions and show how one can efficiently learn these parameters . Finally , we derive the equivariance properties of the induced group transform . The overall building block the LGT and its application on a signal is depicted in Figure ( 2 ) . 3.1 STRICTLY INCREASING AND CONTINUOUS TRANSFORMATIONS . In order to generalize the classical affine transformations used in WT , we propose the utilization of strictly increasing and continuous functions defined as Cinc ( R ) = { g ∈ C ( R ) |g is strictly increasing } , ( 1 ) where C ( R ) defines the space of continuous functions defined on R. This set of function is composed of invertible maps which is crucial in order to derive invariance properties as well as avoid artifacts in the transformed filters . We define the linear operator ρinc ( g ) by [ ρinc ( g ) ψ ] ( t ) = ψ ( g ( t ) ) , ∀ψ ∈ L2 ( R ) , ∀g ∈ Cinc ( R ) , ( 2 ) where ψ denotes a mother filter . We can see that the increasing and continuous group representation operator ρinc induces a mapping which depends on the function g ∈ Cinc ( R ) . If for instance g = e , i.e. , the identity map , then we have ρinc ( e ) ψ = ψ , it is in fact the identity operator in the space of the mother filter . Given a mother filter ψ ∈ L2 ( R ) , ρinc ( g ) ψ , ∀g ∈ Cinc ( R ) induces a nonlinear transformation of the mother filter which can be visualized in Figure ( 3 ) . Note that in signal processing , such a transformation is called warping ( Goldenstein & Gomes , 1999 ; Kerkyacharian et al. , 2004 ) . Among the possible transformations induced on a mother filter by the mapping g ∈ Cinc ( R ) , some of them correspond to well-known filters ( Table 1 ) . 3.2 SAMPLING THE TRANSFORMATION MAPS . In this work , we are specifically interested in the learnability of such an increasing and continuous map . As such , we provide a way to sample such a space via its parameterization . We propose to use piece-wise affine functions constrained such that they belong to the class of strictly increasing and continuous functions . This constrained piece-wise affine map is defined as g ( a , b ) ( t ) = n∑ l=1 ( alt+ bl ) 1Il ( t ) , ∀t ∈ R , ( 3 ) s.t . : al > 0 , ∀l ∈ { 1 , . . . , n } , ( 4 ) bl+1 = ( al − al+1 ) tl+1 + bl , ∀l ∈ { 1 , . . . , n− 1 } , ( 5 ) where a = ( a1 , . . . , an ) , b = ( b1 , . . . , bn ) , 1Il is the indicator function of the intervals Il = [ tl , tl+1 ) , ∀l ∈ { 2 , . . . , n− 1 } and I1 = ( −∞ , t1 ) , In = [ tn , +∞ ) , and al and bl denote respectively the slope and offset of each piece of the function and n is the number of pieces . As such , for each ( a , b ) satisfying the constraints ( 4 ) and ( 5 ) the function g ( a , b ) is a sample from the set Cinc ( R ) . Notice that this mapping can be performed using a 1-layer ReLU Neural Network ( Arora et al. , 2016 ) . This implementation implies a knot-free piece-wise affine mapping , providing more flexibility regarding the transformation map . The knot-free mapping is defined such that the uniform support , i.e. , the intervals Il ( 3 ) , is replaced with varying support for different l ∈ { 1 , . . . , n } . As such , this flexibility induces better approximation property ( Jupp , 1978 ) . 3.3 LEARNING THE PIECE-WISE AFFINE TRANSFORMATION MAPS . The parameters ( ak , bk ) , ∀k ∈ { 1 , . . . , K } are differentiable with respect to the filter and thus any deep learning pipeline using those filters can be used and optimized jointly with the other DNN parameters by stochastic gradient descend methods . Given a set of signals { si ∈ L2 ( R ) } Ni=1 and given a task specific loss function L , we aim at solving the following optimization problem min ( a1 , b1 ) ∈Ω1 , ... , ( aK , bK ) ∈ΩK N∑ i=1 L ( F ( W [ si , ψ ] ( g , . ) ) , ( 6 ) where N denotes the number of signals , K the number of filters , F represents a DNN , Ωk = { ak ∈ Rn+ , bk ∈ Rn|b ( k , l+1 ) = ( a ( k , l ) −a ( k , l+1 ) ) t ( l+1 , k ) + b ( k , l ) } ∀k ∈ { 1 , . . . , K } , and W [ si , ψ ] ( g , . ) = [ W [ si , ψ ] ( g ( a1 , b1 ) , . ) , . . . , W [ si , ψ ] ( g ( aK , bK ) , . ) ] T , and W [ si , ψ ] ( g ( ak , bk ) , . ) = ( si ? ρinc ( g ( ak , bk ) ) ψ̄ ) ( . ) , g ( ak , bk ) ∈ Cinc ( R ) , ∀k ∈ { 1 , . . . , K } , ( 7 ) where ψ̄ ( t ) = ψ ( −t ) and ( . ) corresponds to the time axis . We propose different settings that will impact the type of filter-bank our method can reach . First , We propose a normalization of the frequency of the transform filter ( denoted in the result tables by nLGT ) . This normalization helps to reduce the aliasing induced by the filters . We propose to use f̂ , the normalized frequency f with respect to the maximum slope of the piece-wise affine mapping . For instance , in the case of a Morlet wavelet , the normalization is as follows [ ρinc ( g ( a , b ) ) ψ ] ( t ) = π − 14 exp ( 2πjf̂g ( a , b ) ( t ) ) exp ( −1 2 ( g ( a , b ) ( t ) /σ ) 2 ) , where f̂ = f/maxl∈ { 1 , ... , n } al , j is the imaginary unit , and σ is the width parameter defining the localization of the wavelet in time and frequency . This normalization will be performed for each sample of the group , and thus for each generated filter k ∈ { 1 , . . . , K } of the filter-bank . Second , we constrain the domain of the piece-wise affine map , as derived in ( 3 ) ( denoted in the result tables by cLGT ) . In the following experiments , we propose a dyadic constraint of the domain as in the WT . The support of the filter is close to the support of a wavelet filter-bank . However , the envelope of the filter and the instantaneous frequency still vary as in the Chirplet Transform ( Baraniuk & Jones , 1996 ) .

Summary: This paper defines a set of learnable basis functions and a joint learning algorithm to estimate them. It is based on the premise that the common learning approach in time-series is to first represent them in some spectral domain; thus, the main problem is to define and estimate the basis functions. However, this premise is not accurate and many learning algorithms operate just in the actual time domain (even in speech).