UNIVERSAL MODAL EMBEDDING OF DYNAMICS IN VIDEOS AND ITS APPLICATIONS

1 INTRODUCTION . Extracting underlying dynamics of objects in video frames is one of the challenging problems in video processing Erichson et al . ( 2016 ) . Meanwhile , dynamic mode decomposition ( DMD ) Rowley et al . ( 2009 ) ; Schmid ( 2010 ) has recently attracted attention as a way of obtaining modal representations of nonlinear dynamics from ( general multivariate time-series ) data without explicit prior knowledge about the dynamics . It is closely related to the spectral analysis of the Koopman operator and has been successfully applied for the extraction of spatiotemporal patterns which is crucial in many applications of engineering fields , such as fluid dynamics , epidemiology , neuroscience , controlled systems , analysis of power systems , oceanography , molecular kinetics and many others . We consider two major applications which are based on the underlying dynamics of videos , one is the background/foreground extraction and the other is video classification . Since DMD can extract low rank spatio-temporal features of complex dynamical systems and gives information of growth rates and frequencies of the dynamics , which allows us to interpret spatial structural and temporal information in the data . Therefore , it is possible to perform analysis in a dimensionality reduction data by considering dynamics embedding with the modes obtained by DMD . However , standard or exact DMD method is unable extract the complex non-linear dynamics in time series data , since in the standard approach vertically aligned spatio-temporal image sequences are processed all at once with any further processing , so it becomes difficult to extract the required dynamic information from the data . In this paper , we propose a universal modal embedding of dynamics in videos by CAE-DMD method ; a variant of extended dynamic mode decomposition ( EDMD ) to obtain a compact modal representations of non-linear dynamics from a general multivariate time-series data . In CAE-DMD , at first latent vectors of complex video sequences are obtained by training a CAE network and then DMD is applied on these latent vectors to obtain modal representation of input image sequences , those contain the spatial information of respective image sequences . This method exploits the spatial and temporal information in a low-rank modal with latent vector representations that gives more meaningful representation of input image sequences . We investigate the empirical performance of CAE-DMD on publicly available datasets and results show that our method achieves competitive performance and can cope with complex dynamics in videos or any time-series data . Algorithm 1 Dynamic Mode Decomposition Schmid ( 2010 ) Require : V1 and V2 input sequences . Ensure : Dynamic modes Φ and eigenvalues ∆ . 1 : Ur , Sr , Qr ← compact SVD of V1 . 2 : Ã← U∗rV2QrS−1r . 3 : W̃ , ∆← eigenvectors and eigenvalues of Ã ; 4 : Φ← V2QrS−1r W̃r 5 : return : Φ , ∆ ; The remainder of this paper is organized as follows : First , we give a brief review on Koopman spectral analysis and Extended DMD in Section 2 . In Section 3 , we describe our proposed model , and video processing with CAE-DMD is presented in Section 4 . Experiments and results are discussed in in Section 5 along with performance evaluations on the basis of benchmark datasets . Finally , Section 6 summarizes and concludes the study . 2 BACKGROUND : EXTENDED DMD . Consider a ( possibly nonlinear ) dynamical system : vt+1 = f ( vt ) , v ∈M , where f : M → M , M is the state space , and t is the time index . In this system , the Koopman operator K for ∀v ∈M can be defined as follows : Kg ( v ) = g ( f ( v ) ) , where g : M → C ( ∈ F ) denotes an observable that is an element in some function space F Koopman ( 1931 ) . By definition , K is a linear operator in F . Here , we assume that there exists a subspace of F invariant to K , which can be denoted by G ⊂ F . Additionally , we assume that G is finite-dimensional and that a set of observables { g1 , . . . , gn } that spans G is available . If we let g = [ g1 , . . . , gn ] > : M→ Cn , the one-step evolution of g for ∀v ∈M can be expressed as follows : Kg ( v ) = g ( f ( v ) ) , where the finite dimensionalK is the restriction ofK to G. Then , if all eigenvalues ofK are distinct , any value of g can be expressed as follows : g ( v ) = ∑n i=1ϕ ( v ) ξi , with some coefficients ξi , where ϕ : M → Cn is an eigenfunction of K with the corresponding eigenvalue λ ∈ C , i.e. , Kϕ ( v ) = λϕ ( v ) . Thus , we obtain g ( vt ) = ∑n i=1λ t ici , ci = ϕi ( v0 ) ξi . As a result , g is decomposed into modes { ci } , and the modulus and argument of λi express the decay rate and frequency of ci , respectively . Next , we summarize EDMD as a method to approximates the Koopman operator Williams et al . ( 2015 ) . It requires a pair of time series data as V1 = [ v1 , v2 , v3 , . . . , vT−1 ] , V2 = [ v2 , v3 , v4 , . . . , vT ] , where vt ∈ M and vt+1 = f ( vt ) . And then , suppose we are given a set of independent basis functions B = { b1 , b2 , . . . , bn } , where bi ∈ K. A vector valued function b : M → C1×n , where b ( v ) = [ b1 ( v ) , b2 ( v ) , . . . , bn ( v ) ] . Next , we generate K ∈ Rn×n , a finite dimensional approximation of K and a function bd ∈ Fb ⊂ F , written as bd = n∑ i=1 wibi = bw , where w contain the weights of n elements of B . Since Fb is not an invariant subspace of K , Therefore Kbd = b ( Kw ) + r. The above equation includes a residual term r ∈ F . Therefore , to findK , we minimize the following optimization J = 1 2 T−1∑ i=1 | ( b ( vi+1 ) − b ( vi ) K ) a| , which is a least square problem whose unique solution can be found as , K = G†A , where † denotes the pseudoinverse , while G and A can be obtained by solving the following equations G = 1 T − 1 T−1∑ i=1 b ( vi ) ∗ b ( vi ) , A = 1 T − 1 T−1∑ i=1 b ( vi ) ∗ b ( vi+1 ) , whereK , G , A ∈ Cn×n . Thus , K is a finite dimensional approximation of K and EDMD approximation of an eigenfunction of K can be written as ϕi = bξi . In the above equation ξi is the i-th eigenvector ofK . Note that here we briefly summarized EDMD , a more detailed explanation can be found in Williams et al . ( 2015 ) . 3 PROPOSED MODEL : CAE-DMD . In the propose model of CAE-DMD , we first train the covolutional auto-encoder network on input image sequences to obtain the compact representation of these sequences in the form of latent vectors/encoded sequences . These encoded sequences of input videos capture the complex underlying dynamics after training the CAE network . In the next step , we apply DMD on these compact representations to obtain modal representations , those contain the spatial information of image sequences . Most of the spatial information of video sequences can be found in few DMD modes . Therefore , a few set of modes can capture dynamics of even longer video sequences , that is one of the main advantages of dimensionality reduction temporally using DMD . Also , the spatial dimension of input sequences can be reduced by choosing the length of latent vectors . Therefore , we exploit the dimensionality reduction spatially and temporally for modal embedding of dynamics in time series sequences . Also , extraction of non-linear information in the data is achieved by representing the input image sequences in the form of encoded vectors using deep convolutional autoencoder . The architecture of CAE-DMD for foreground and background extraction is illustrated in Figure 1 . The details of the main blocks of CAE-DMD model are described as follows : Encoder : In the first step , input video sequences { v1 , v2 , . . . , vT−1 } are trained on convolutional autoencoder network . For this training we used sixteen layers deep CAE network . The filter size of each convolutional layer was fixed to 5× 5 with increasing number of total filters at each layer . The Encoder takes flattened video sequences as input and generates the feature map in the next layer as zi = f ( pi ×V1 + bi ) , ( 1 ) where V1 contains the input images , pi and bi are the corresponding filter and bias , and f is the activation function . In this model , we used Rectified Linear Unit ( ReLU ) as activation function after each convolutional layer Zeiler et al . ( 2013 ) , which is further followed by a max pooling layer . This helps reducing the computational cost in the upper layers . Then , the latent spaces are obtained by a fully connected layer at the output of encoder as H = Z×W + C. ( 2 ) In equation 2 , H denotes the hidden layer which contains the compressed sequences of input images , Z represents the feature maps , whereas W and C are the feature weights and bias of the fully connected layers , respectively . Note that in this model we choose the length of a latent vector to half the size of an input image , which gives promising results to extract dynamics from image sequences and for classification of videos . Decoder : The proposed model of decoder follows a symmetric pattern of layers as in the encoder . To reconstruct the input images at the decoder side , the latent sequences obtained from the encoder are reshaped to a reconstructed version of feature maps Z′ via fullyconnected layer as Z′ = H×W′ + C′ , ( 3 ) where W′ and C′ are the weights and bias of fullyconnected layer at the decoder side . Then , up-sampling is applied after each convolutional layer and the final approximated image Ṽ1 is reconstructed at the output of the last layer of Decoder by reshape function . Note that too much compression in CAE will result in loss of information . Typically a simple loss function ; Mean Square Error ( MSE ) is calculated for convergence of the CAE model between the network output Ṽ1 i.e. , output of decoder and the corresponding vectorized original video sequences V1 . However , in the proposed method DMD loss term is introduced and a modified objective function is implemented for end-to-end learning . MSE = 1 bs T/bs∑ i=1 ‖vbsi− ˜vbsi‖2 + λd‖H2 −UÃU ′H1‖2 , ( 4 ) where T is the total number of video sequences and bs is the batch size for training . The incorporated DMD loss term in the objective function is used to minimize the DMD reconstruction error that is controlled by the parameter λd . H1 and H2 are the encoded sequences and shifted encoded sequences in time step , respectively . U , U ′ are orthogonal matrices and Ã is a low-dimensional linear operator obtained by applying DMD on latent vectors ( see Alg . 1 ) . DMD over Encoded Sequences : In the next step , original image sequences { v1 , v2 , . . . , vT−1 } and { v2 , v3 , . . . , vT } are fed to the trained CAE network to obtain the latent vectors of input image sequences . The dimension of these latent vectors is spatially compressed as compared to input sequences , DMD is applied on these vectors to obtain dynamic modes , those contain the complex dynamic with low and high frequency information of the data with their corresponding eigenvalues . These filtered latent vectors are further fed to the decoder for final approximation of video sequences .

Summary: The paper considers the problem of extracting the underlying dynamics of objects in video frames. The paper focuses on two major applications: background/foreground extraction and video classification. The paper proposes a method that first obtains latent vectors from a video sequence by training a neural network and then applies dynamic mode decomposition (by Schmidt, 2010).