UNIVERSAL MODAL EMBEDDING OF DYNAMICS IN VIDEOS AND ITS APPLICATIONS
Authors: Israr Ul Haq
Yoshinobu Kawahara
Venue:  (2020)
Keywords: Non-linear dynamics
Convolutional Autoencoder
Foreground modeling
Video classification
Dynamic mode decomposition

Abstract: Extracting underlying dynamics of objects in image sequences is one of the challenging problems in computer vision. On the other hand, dynamic mode decomposition (DMD) has recently attracted attention as a way of obtaining modal representations of nonlinear dynamics from (general multivariate time-series) data without explicit prior knowledge about the dynamics. In this paper, we propose a convolutional autoencoder based DMD (CAE-DMD) that is an extended DMD (EDMD) approach, to extract underlying dynamics in videos. To this end, we develop a modified CAE model by incorporating DMD on the encoder, which gives a more meaningful compressed representation of input image sequences. On the reconstruction side, a decoder is used to minimize the reconstruction error after applying the DMD, which in result gives an accurate reconstruction of inputs. We empirically investigated the performance of CAE-DMD in two applications: background/foreground extraction and video classification, on publicly available datasets.

Under review as a conference paper at ICLR 2020
UNIVERSAL MODAL EMBEDDING OF DYNAMICS IN VIDEOS AND ITS APPLICATIONS
Anonymous authors Paper under double-blind review
ABSTRACT
Extracting underlying dynamics of objects in image sequences is one of the chal- lenging problems in computer vision. On the other hand, dynamic mode decom- position (DMD) has recently attracted attention resentations of nonlinear dynamics from (general multivariate time-series) data without explicit prior knowledge about the dynamics. In this paper, we propose a convolutional autoencoder based DMD (CAE-DMD) that is an extended DMD (EDMD) approach, to extract underlying dynamics in videos. To this end, we de- velop modified CAE model by incorporating DMD on the encoder, which gives a more meaningful compressed representation of input image sequences. On the reconstruction side, decoder is used to minimize the reconstruction error after applying the DMD, which in result gives an accurate reconstruction of inputs. We empirically investigated the performance of CAE-DMD in two applications: background/foreegoumn extraction and video classification, on publicly available datasets.
way of obtaining modal rep-
INTRODUCTION
Extracting underlying dynamics of objects in video frames is one of the challenging problems in video processing Erichson al. (2016). Meanwhile, dynamic mode decomposition (DMD) Rowley et al. 2009); [Schmid] (2010) has recently attracted attention as a way of obtaining modal repre- sentations nonlinear dynamics from (general multivariate time-series) data without explicit prior knowledge about the dynamics. It is closely related to the spectral analysis of the Koopman operator and has been successfully applied for the extraction of spatiotemporal patterns which is crucial in many applications of engineering fields, such as fluid dynamics, epidemiology, neuroscience, con- trolled systems, analysis of power systems, oceanography, molecular kinetics and many others. We consider two major applications which are based the underlying dynamics of videos, one is the background/foregroune extraction and the other is video classification. Since DMD can extract low rank apatio-tempolt and frequencies of the dynamics, which allows us to interpret spatial structural and temporal infor- mation in the data. Therefore, it is possible perform analysis in dimensionality reduction data by considering dynamics embedding with the modes obtained by DMD. However, standard or exact DMD method is unable extract the complex non-linear dynamics in time series data, since in the standard approach vertically aligned spatio-temporal image sequences are processed all at once with any further processing, so becomes difficult extract the required dynamic information from the data.
features of complex dynamical systems and gives information of growth rates
In this paper, we propose a universal modal embedding of dynamics in videos by CAE-DMD method; a variant of extended dynamic mode decomposition (EDMD) to obtain a compact modal representations of non-linear dynamics from a general multivariate time- series data. CAE-DMD, at first latent vectors of complex video sequences are obtained by training a CAE network and then DMD is applied on these latent vectors obtain modal representation of input image sequences, those contain the spatial information of respective image sequences. This method exploits the spa- tial and temporal information in a low-rank modal with latent vector representations that gives more meaningful representation of input image sequences. We investigate the empirical performance of CAE-DMD on publicly available datasets and results show that our method achieves competitive performance and can cope with complex dynamics in videos any time-series data.

---

Under review as a conference paper at ICLR 2020
Algorithm Dynamic Mode Decomposition Schmii (2010)
Require: V1 and V2 input sequences.
Ensure: Dynamic modes & and eigenvalues A.
Ur S,, Qr + compact SVD of Vi.
2: A UUV
3: W, A <- eigenvectors and eigenvalues of A;
4: + V.Q.S,1W,
5: return: Φ, A;
The remainder of this paper is organized follows: First, we give brief review on Koopman spec- tral analysis and Extended DMD in Section 2 In Section [ we describe our proposed model, and video processing with CAE-DMD presented in Section Experiments and results are discussed in in Section 5 along with performance evaluations on the basis of benchmark datasets. Finally,
Section 6 summarizes and concludes the study.
BACKGROUND: EXTENDED DMD
Consider (possibly nonlinear) dynamical system:
ut+1 f(v))
v E M,
f:M where f: MM- + M, M is the state space, and is the time index. In this system, the Koopman
operator K. for Vu M can be defined as follows:
Kg(v)= = ((f(v)),
where 9: M + C (E F) denotes an observable that is an element in some function space F Koopman (193I). By definition. K. linear operator in F. Here, we assume that there exists subspace of F invariant to K, which can be denoted by g C F. Additionally, we assume that G finite-dimensional and that a set of observables (91,-..,9t that spans available. we let g 991 ]n] M C", the one-step evolution of g for Yu E M can be expressed as follows:
kg(v) (ff()),
where the finite dimensional K is the restriction of KC to G. Then, all eigenvalues K are distinct, any value of g can be expressed follows:
= = Ei
with some coefficients E, where sp: M + C" is an eigenfunction of with the corresponding eigenvalue À E C, i.e., Ky((v) Xy((v). Thus, we obtain
g(v)) Σ Ei=Me, c, = (w))
As a result, g is decomposed into modes (ci), and the modulus and argument of X; express the decay rate and frequency of c, respectively.
Next, we summarize EDMD as a method to approximates the Koopman operator Williams et al. 2015). It requires pair of time series data
V1 = U1: 22 V3...... 1) V22 = (U2, 13, F4 ....... PT),
where V, E M and Ut+1 = f(v)). And then, suppose we are given a set of independent basis functions {b), .2...... . b. where b, K. A vector valued function b -M - Clx", where
b(u)= = [b (v), bz(v)...., b, (v)). Next, we generate K E Raxx, finite dimensional approxima- tion of K and function by E F. C F, written
11
bd Σ w.b, = bw. i=1
where contain the weights of elements of B. Since Fo is not an invariant subspace of K, Therefore
Kha b (ww) +
2

---

Under review as a conference paper at ICLR 2020
The above equation includes a residual term E F. Therefore, to find K, we minimize the following optimization
T
Σ
(b(0,+1) Ku))))aa
which is a least square problem whose unique solution can be found as,
K = G* A,
where i denotes the pseudoinverse, while G and can be obtained by solving the following equa- tions
G T b(v;) b(v,),
A b(u:) b(0++1);
where K, G, E Cuxn, Thus, K a finite dimensional approximation K. and EDMD approxi- mation of an eigenfunction of K. can be written as
In the above equation & is the i-th eigenvector of K. Note that here we briefly summarized EDMD, more detailed explanation can be found in Williams al. 2015).
PROPOSED MODEL: CAE-DMD
In the propose model of CAE-DMD, we first train the covolutional auto-encoder network on input image sequences to obtain the compact representation of these sequences in the form of latent vec- tors/encoded sequences. These encoded sequences of input videos capture the complex underlying dynamics after training the CAE network. In the next step, apply DMD on these compact repre- sentations to obtain modal representations, those contain the spatial information of image sequences. Most of the spatial information of video sequences a few set of modes can capture dynamics of even longer video sequences, that one of the main advantages of dimensionality reduction temporally using DMD. Also, the spatial dimension of in- put sequences can be reduced by choosing the length of latent vectors. Therefore, we exploit the dimensionality reduction spatially and temporally for modal embedding of dynamics in time series sequences. Also, extraction of non-linear information in the data is achieved by representing the input image sequences in the form of encoded vectors using deep convolutional autoencoder. The architecture of CAE-D DMD for foreground and background extraction is illustrated in Figure The details of the main blocks of CAE-DMD model are described as follows:
be found in few DMD modes. Therefore,
Encoder: In the first step, imput video sequences {U1, U -1 are trained on convolutional
autoencoder network. For this training we used sixteen layers deep CAE network. The filter size of each convolutional layer was fixed to 5 x 5 with increasing number of total filters at each layer. The Encoder takes flattened video sequences as input and generates the feature map in the next layer as
z, = f(p V1 b.).
(1)
where V, contains the input images, p, and b, the corresponding filter and bias, and is the activation function. In this model, we used Rectified Linear Unit (ReLU) as activation function after each convolutional layer Zeiler al. (2013), which is further followed by max pooling layer. This helps reducing the computational cost in the upper layers. Then, the latent spaces are obtained by fully connected layer the output of encoder as
Z CC
In equation 2
H denotes the hidden layer which contains the compressed sequences of input images, represents the feature maps, whereas W and C are the feature weights and bias of the fully connected layers, respectively. Note that in this model choose the length of a latent vector half
3

---

Under review as a conference paper at ICLR 2020
secuences
Impur image sequenees:
foregrounds
DMD
Decoder
V(ln)
Estl mated backgrounds
Foreyround modes
DMD modes
Amplitadio
Time Evolutions
Buckground modes
Figure Convolutional autoencoder block diagram for foreground/bacckrround extraction.
the size of an input image, which gives promising results to extract dynamics from image sequences and for classification videos.
Decoder: The proposed model of decoder follows symmetric pattern of layers as in the encoder. To reconstruct the input images at the decoder side, the latent sequences obtained from the encoder reshaped to reconstructed version of feature maps Z' via fullyconnected layer
= C
where W and C' the weights and bias of fullyconnected layer the decoder side. Then, up-sampling is applied after each convolutional layer and the final approximated image V, is recon- structed at the output of the last layer of Decoder by reshape function. Note that too much com- pression in CAE will result in loss of information. Typically simple loss function; Mean Square Error (MSE) is caleulated for convergence of the CAE model between the network output V1 i.c., output of decoder and the corresponding vectorized original video sequences V1. However, in the proposed method DMD loss term introduced and a modified objective function implemented for end-to-end learning.
1 T/be
MSE
be Σ D 22 Aal AÄ
(4)
where i'ss is the total number of video sequences and is the batch size for training. The incorporated DMD loss term in the objective function is used to minimize the DMD reconstruction error that controlled by the parameter Ad- H1 and H. are the encoded sequences and shifted encoded sequences in time step, respectively. linear operator obtained by applying DMD on latent vectors (see Alg.
U' are orthogonal matrices and Á is a low-dimensional
DMD over Encoded Sequences: the next step, original image sequences (v1, U2...., 0T-1 and [2y, .,,..., Ur are fed to the trained CAE network to obtain the latent vectors of input image sequences. The dimension of these latent vectors is spatially compressed as compared to input sequences, DMD is applied on these vectors to obtain dynamic modes, those contain the complex dynamic with low and high frequency information of the data with their corresponding eigenvalues. These filtered latent vectors are further fed to the decoder for final approximation of video sequences.
4 VIDEO PROCESSING WITH CAE-DMD
For a given video frames V E I X xT each frame {v), V2...., Uy is flattened as column vector and ordered in time. This arrangement of video frames gives spatiotemporal grid, that is arranged
4

---

Under review as a conference paper at ICLR 2020
in two overlapping sets of data called left and right image sequences defined as:
Vi v1, 22 ................................ UT V2
12 03 03..... VT
(5)
To obtain the DMD of a given video, it assumed that each frame d++1 at time interval (t+1) is connected to the previous frame vt by linear mapping A as V/2 = AV, the estimate of and its eigen decomposition is obtained by Alg. D
A set of dynamic modes + :- { 3 and the corresponding eigenvalues A = {AI, }
obtained are used to reconstruct these image sequences V1 and V22 which are the approximation of original video sequences. Here, is the number of adopted eigenvectors. These modes represent the slowly varying rapidly moving objects at time points & E 10, 1, 7 - 1} in the video frames with associated continuous-time frequencies and can be expressed as follows:
log(A,) At
W/
(6)
Further, the approximated video frames for low- and high-freyuency modes at any time point can be reconstructed as
v(t) ~ Σ exp((w,))) = $eex((2tt)a,
(7)
3=1
where Φ is a column vector of the j-th dynamic mode that contains the spatial structure information and or the initial amplitude of the corresponding DMD mode. The vector of the initial amplitudes ar can be obtained by taking the initial video frame at time = 0, which reduces equation 7 to Vi &a. Note that the matrix of cigenvectors is not square; thus, the initial amplitudes can be observed using the following pseudoinverse process:
Q Divi.
Table I: F-measures by CAE-DMD and the existing algorithms for foreground extraction Wallflower datasct.
Dataset
DECOLOR 3TD
DMM
LSD
TVRPCASIICA
LICFSO
MODSM
GFL
RESA
GRASTAMSCL
CAL
Wallioower
0.59
0.75
0.78
0.75
0.6
0.8
0.74
0.73
0.54
0.33
0.92
037
4.1 FOREGROUND INFORMATION EXTRACTION
Foreground segmentation and background estimation has been originated many computer vision applications such moving object detection Zhou et al. (2013), video surveillance Bouwmans & [2hzah] motion detection Gao et al. (2014) and many others. number of methods have been proposed in literature achieve this task. Ortego et al. (2016) proposed multipath reconstruction for background estimation. Liu et al. (2015) estimated the background and foreground based on low-rank and structured sparse matrix, respectively. The performance of these methods degrades the background is visible for short period or in case of dynamic background. To overcome these limitations Javed et al. (2017) proposed a method to incorporate the spatial and temporal sparse subspace clustering into the Robust Principal Component Analysis (RPCA) Candès et al. (201)) However in our proposed CAE-DMD method, we split the foreground and background based on the DMD modes and their corresponding eigenvalues (frequencies). Thus one can extract not only the background or foreground information but also other dynamics the data, based on the frequency information obtained after applying DMD latent vectors. Figure shows the case of splitting foreground and background in the video.
The key principle to separate the video frames into foregrounds and the background is by thresh- olding of low frequency modes based on the corresponding eigenvalues. Generally, the portion that represents the background in videos is constant among the frames and satisfies | as O, where p E (1.2.....7) r}. Typically, single mode represents the background, which located near the origin in complex space, whereas Wyy Vj # p are the eigenvalues that represent the foreground
5

---

Under review as a conference paper at ICLR 2020
structures bounding away from the origin in complex plane. Therefore, the reconstructed video frames can be separated into the background and foreground structures as follows:
V DMD @ycxplupt)ay
Σ
Φ @)exp(wyt)a
(9)
Background
sfp
Foreground
where / {0, ..,,, T 1} is the time indices up -T- frames. Note that the initial amplitude Op [/111 of the stationary background is constant for all the future time points, whereas
0j (vI). Vj + are the initial amplitudes of varying foreground structures.
After filtering the DMD modes based on the eigenvalues, encoded sequences are reconstructed and fed the decoder to approximate the background and foreground structures in the video sequences. set five frames of Wallflower dataset (Bootstrap video sequences) of moving peo- ple. Figure 3 second row shows the obtained eigenvalues each of which corresponds to either fore- ground or background modes.
Figure
4.2 CLASSIFICATION OF VIDEOS
To extract the dynamics in videos several methods have been proposed for different applications such as dynamic scene classification Derpanis et al. (2012), activity and action recognition Donahue JaasiiiiationnDeepaniss
et al. (2015) in videos. Video classification using Convolutional Neural Networks (CNNs) have been considered as an effective class models, where the networks not only learn the appearance information but also able to learn temporal evolution Karpathy et al. (2014); Du et al. (2015). In these methods the temporal encoding is learned at a single level that is not sufficient to interpret and understand the complex dynamics in video sequences. more well suited method is proposed by| Cherian et al. 2017). they proposed a rank pooling method that takes input, features from the intermediate layer of (CNN) trained on tiny subspaces. They use this low-rank approximation of features while maintaining temporal order for compact representation of video sequences and then use this for classification. A recently introduced discriminative hierarchical rank pooling method Fernando et al. (2 cope with the limitations in previous methods and proposes a novel method to encode video sequences at multiple levels to capture the non-linear and complex dynamics. Some of the famous traditional methods for one activity classification per video are: bag of features (BoF), kernel dynamic system (KDS) Laptev et al. (2008), binary dynamic system (BDS) Niebles et al.) (2010), DTM Blei & Lafferty (2006), Attributes based LI & Vasconcelos (2012) and TOT those explained by LI & Vasconcelos (2012).
In our proposed method, we classify the complex dynamics in videos through dynamic modes, ob- tained by applying the DMD on latent spaces input video sequences. the next step of video classification, orthogonal vectors of these DMD modes are calculated by applying singular value decomposition (SVD) and then a projection kernel Hamm Lee (2008) is computed on these or- thogonal vectors. Finally, a supervised classifier is trained over this kernel with respective labels of videos. The method to classify videos along with the projection kernel method is summarized in Alg.
2
EXPERIMENTAL RESULTS
5.1 EXTENDED (CAE-DMD) AND STANDARD DMD
To demonstrate the effectiivnnes of the proposed method, another experiment is performed on video of SBMne(" dataset, where people are strolling in a terrace with no original background pro- vided, the extraction of dynamics becomes more challenging. To visualize and verify our method we extracted foreground information from a set of 150 consecutive image sequences and then ap- plied both methods i.e., standard and CAE-DMD. Results of standard and CAE-DMD are shown in Figure 2 (a) and Figure 2 (b), respectively. It can be visualized in Figure P second row, that standard DMD method is unable reconstruct original image sequences because of the presence of non-linear complex dynamics in video frames. On the other side, CAE-DMD can reconstruct
hhtt:///eechaakygoouddddennnneee

---

Under review as a conference paper at ICLR 2020
Algorithm 2 CAE-DMD to classify videos.
Require: D and L. DMD modes and videos labels, respectively.
Ensure: predicted label.
1: U compact SVD of D.
2: for
do
total number of videos train.
if 2 2 then
4:
for j = to i do
5:
K(i. = real (trace (U, - (U, U;))
6:
K (i. K(i,)
K: kernel
end for
end if
9: end for
10: Train KNN classifier over K and L.
KNN: k-nearest neighbors
11: L < predict with test data.
original image sequences more accurately, which shows that by applying DMD on latent vectors can effectively minimize the reconstruction error; that shows the method is capable of handling the non-linearity in data. Further, the dynamic and static information in the video can easily be sepa- rated with CAE-DMD shown in Figure 2 thirddrow. Figure fourth-rou shows the threshold image sequences after foreground extraction.
q
(a)
(b)
Figure 2: Dynamics extraction in Standard and Extended (CAE) DMD; (a) Standard DMD, (b) CAE-DMD. First row: original Image sequences of walking people; Second row: Image reconstruction by standard and CAE-DMD; Third row: dynamics extraction; Last row: Threshold image sequences.
We also empirically investigated the performance of CAE-DMD based on comparisons with several existing algorithms in the scenarios of foreground extraction in Subsection 5.2 and video classifica- tion Subsection (5.3
We utilize convolutional autoencoder network to represent each video frame by a latent vector obtained by fully connected layer, the length of each latent vector is set to half of the input image size, with filter size of 5x5 and stride of one, unless otherwise specified. The network is implemented in python with Tensorflow package and parameters such as batch size of 16, learning rate 1e -4, number of epochs 500 1000, and optimization is done with adam optimizer. To avoid overfitting dropout of 0.5 was used. Test, validation and train data is separated as 10%, 10% and 80%, respectively for foreground extraction tasks. Note that the optimal values of these parameters were manually tuned and set to those where the hest performance was achieved. Total time to reconstruct a batch of images was 1-2.5s, after training of video frames.
Experimental details:
5.2 FOREGROUND EXTRACTION
We applied the proposed method to extract foreground structures from the background using the wallflower dataset This dataset contains six video sequences (Moved Object, Camouflage, Fore- ground Aperture, Light Switch, Time of Day and Waving Trees) and single hand segmented ground-truth image for each sequence. The results are compared with the state-of-the-art meth- ods, such as DECOLOR Zhou al. (0013), 3TD Oreifej al. (2013). DP-GMMHaines & Xiang

---

Under review as a conference paper at ICLR 2020
(2014) LSD Liu al. (2015). TVRPCA Cao et al. 2016), SRPCA Javed al. (2018) LR- FSO Xue et al. (2013), MODSM, GFL Xin et al. 2015), RFSA Guo et al. (2014), GRASTA He et al. (2012) and MSCL-FLL Javed ground extraction. The first column shows the original frames of video sequences. The estimated backgrounds by CAE-DMID is shown in the second column followed by difference images in the third column. The estimated foreground images by our method and the hand segmented ground-truth images of corresponding videos shown fourth and fifth columns, respectively. The difference image was calculated by considering the absolute difference between the original frames and their respective backgrounds. To further enhance the accuracy of the extracted foreground structures, morphological operations were applied to fill the empty holes and to connect the unconnected bina- ry pixels followed by thresholding. The performance of CAE-DMD calculated by F-measure values are shown in Table T that shows the our method is competitive with other state-of-the-ar proposed methods.
Faavee et al. (017)). Figure shows some examples of results by CME-DMD for fore-
20
10
Budggeed zade
Emaginary
Figure Wallflower dataset; first column frames o- riginal videos; Estimated backgrounds are shown in the second column; Difference and threshold images are shown in column three and fourth respectively. Fifth Figure 3: Splitting foreground and the background in column shows the enhanced images after applying mor- phological operations.
Eninsaalongrreol
Roal
Wallflower dataset (Bootstrap video sequences).
Table 2: Classification accuracy on Weizmann Activities.
Dataset
BoF
ATTRIBUTES
DTM
TOT
KDS
BDS
CAE-DMD
Weizmann Activities
57.8%
72.6%
84.6%
88.2%
90.2%
94.8%
91.1%
5.3
VIDEO CLASSIFICATION
Our second experiment the classification of videos based on the dynamics they contain. We used Weizmann datase to classify different actions videos. We consider this dataset suitable for experiments, since it contains one particular action per video with 10 classes of different actions egg. (Bend, Jack, Jump, Pjump, Run, Side, Skip, Walk, Wavel and Wave2) performed by 9 different subjects. Next we classify these activities with a 9-fold lave-oneeout-creee where for each trial the activities of one subject were used test set and those of the remaining 8 as training set. The classification accuracy of our method is presented Table 2 that shows classification through modal representations using DMD latent vectors be used classify videos containing different actions.
oott-cross-allidation validation (LOOCV),
CONCLUSIONS
We proposed universal modal embedding of dynamics by CAE-DMD that is an extended DMD method to extract complex underlying dynamics in general multivariate time series data. The ex- perimental results show that our modal embedding method; a dimensionality reduction spatially and temporally can effectively extract the underlying dynamics in videos. To verify our method we con- sider two applications: foreground extraction and videos classification. Comparable performance achieved these applications exhibit that this method can be applied any multivariate-time series data to extract complex and non-linear dynamics.
thtp://wwwwwisddeb weizmann.ac.ii wwt.mam...cil/vssso///ssseeeeee Disiom/SpaceTTimActiion
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Moshe Blank, Lena Gorelick, Eli Shechtman, Michal Irani, and Ronen Basri. Actions as space-time shapes. In null, pp. 1395-1402. IEEE, 2005.
David M Blei and John D Lafferty. Dynamic topic models. In Proceedings of the 23rd international conference on Machine learning, pp. 113-120. ACM, 2006.
Thierry Bouwmans and El Hadi Zahzah. Robust pea via principal component pursuit: A review for a comparative evaluation in video surveillance. Computer Vision and Image Understanding, 122: 22-34, 2014.
Emmanuel J Candès, Xiaodong Li, Yi Ma, and John Wright. Robust principal component analysis? Journal of the ACM (JACM), 58(3):11, 2011.
Xiaochun Cao, Liang Yang, and Xiaojie Guo. Total variation regularized rpea for irregularly moving object detection under dynamic background. IEEE transactions on cybernetics, 4614):101 2016.
Anoop Cherian, Basura Fernando, Mehrtash Harandi, and Stephen Gould. Generalized rank pooling for activity recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 322222223333 22311 2017.
Konstantinos G Derpanis, Matthieu Lecce, Kostas Daniilidis, and Richard Wildes. Dynamic scene understanding: The role of orientation features in space and time in scene classification. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1306-131 1313. IEEE, 2012.
Jeffrey Donahue, Lisa Anne Hendricks, Sergio Guadarrama, Marcus Rohrbach, Subhashini Venu- gopalan, Kate Saenko, and Trevor Darrell. Long- leng-term recurrent convolutional networks for visual recognition and description. In Proceedings of the IEEE conference on computer vision and pat- tern recognition, pp.
2015.
Yong Du, Wei Wang, and Liang Wang. Hierarchical recurrent neural network for skeleton based action recognition. In Proceedings of the IEEE conference on computer vision and pattern recog- nition, pp. 1110-1118, 2015.
Benjamin Erichson, Steven L Brunton, and Nathan Kutz. Compressed dynamic mode decom- position for background modeling. Journal of Real-Time Image Processing, pp. 1-14, 2016.
Basura Fernando, Peter Anderson, Marcus Hutter, and Stephen Gould. Discriminative hierarchical rank pooling for activity recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1924-1933 2016.
Zhi Gao, Loong-Fah Cheong, and Yu-Xiang Wang. Block-sparse rpea for salient motion detection. IEEE transactions on paltern analysis and machine intelligence, 36(10):1975-1987, 2014.
Xiaojie Guo, Xinggang Wang, Liang Yang, Xiaochun Cao, and Yi Ma. Robust foreground detection using smoothness and arbitrariness constraints. In European Conference on Computer Vision, pp- 535-550. Springer, 2014.
Tom SF Haines and Tao Xiang. Background subtraction with dirichletprocess mixture models. IEEE transactions on pattern analysis and machine intelligence, 36(4):670-683, 2014.
Jihun Hamm and Daniel D Lee. Grassmann discriminant analysis: a unifying view on subspace- based learning. In Proceedings of the 25th international conference on Machine learning, pp. 376-383, ACM, 2008
Jun He, Laura Balzano, and Arthur Szlam. Incremental gradient on the grassmannian for online fore- ground and background separation in subsampled video. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1 1568-1575. IEFF, 2012.
Sajid Javed, Arif Mahmood, Thierry Bouwmans, and Soon Ki Jung. beokgrouundforeerroone mod- cling based on spatiotemporal sparse subspace clustering. JEEE Transactions on Image Process- ing,
2017.

---

Under review as a conference paper at ICLR 2020
Sajid Javed, Arif Mahmood, Thierry Bouwmans, and Soon Ki Jung. Spatiotemporal low-rank mod- eling for complex scene background initialization. IEEE Transactions on Circuits and Systems for Video Technology, 28(6):1315-133 1329, 2018.
Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, and Li Fei- Fei. Large-scale video classification with convolutional neural networks, In Proceedings of the IEEE conference on Computer Vision and Puttern Recognition, pp. 1725-1732, 2014.
B. Koopman. Hamiltonian systems and transformation in Hilbert space. Proceedings of the National Academy of Sciences of the United States of America, 17(5):155331, 1931.
Ivan Laptev, Marcin Marszalek, Cordelia Schmid, and Benjamin Rozenfeld. Learning realistic human actions from movies. In Computer Vision and Pattern Recognition, 2008. CVPR 2008.
IEEE Conference on, pp. 1-8. IEEE, 2008.
Qianxiao Li, Felix Dietrich, Erik M Bollt, and loannis G Kevrekidis. Extended dynamic mode decomposition with dictionary learning: data-driven adaptive spectral decomposition of the koopman operator. Chaos: An Interdisciplinary Journal of Nonlinear Science, 27(10):103111, 2017.
Wei-Xin LI and Nuno Vasconcelos. Recognizing activities by attribute dynamics. volume 2, pp- 1106-1114 14, 12 2012.
Weixin Li and Nuno Vasconcelos. Recognizing activities by attribute dynamics. In Advances in Neural Information Processing Systems, pp. 1106-1114, 2012.
Xin Liu, Guoying Zhao, Jiawen Yao, and Chun Qi. Background subtraction based on low-rank and structured sparse decomposition. IEEE Transactions on Image Processing, 24()):200222544 2015.
Juan Carlos Niebles, Chih-War Chen, and Li Fei-Fei. Modeling temporal structure of decomposable motion segments for activity classification. In European conference on computer vision, pp. 392- 405. Springer, 2010.
Omar Oreifej, Xin Li, and Mubarak Shah. Simultaneous video stabilization and moving object detection turbulence. IEEE transactions on pattern analysis and machine intelligence, 35(2): 450-462, 2013.
Diego Ortego, Juan C SanMiggul,, and José M Martínez. Rejection based multipath reconstruction for background estimation in sbmnet 2016 dataset. In 2016 23rd International Conference on Pattern Recognition (ICPR), pp. 114-119, IEEE, 2016.
Clarence W Rowley, Igor Mezić, Shervin Bagheri, Philipp Schlatter, and Dan S Henningson. Spec- tral analysis of nonlinear flows. Journal of fluid mechanics, 641:1 11:1125 2009.
Peter Schmid. Dynamic mode decomposition of numerical and experimental data. Journal fluid mechanics, 656:5-28, 2010.
Matthew O Williams, Ioannis G Kevrekidis, and Clarence W Rowley. A data- approximation of the koopman operator: Extending dynamic mode decomposition. Journal of Nonlinear Science, 25(6):1307-1344 2015.
Bo Xin, Yuan Tian, Yizhou Wang, and Wen Gao. Background subtraction via generalized fused lasso foreground modeling. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 4676-4684, 2015.
Gengjian Xue, Song, and Jun Sun. Foreground estimation based on linear regression model with fused sparsity on outliers. IEEE Transactions Circuits and Systems for Video Technology, 23 18):1346-1357, 2013.
Matthew D Zeiler, Ranzato, Rajat Monga, Min Mao, Kun Yang, Quoc Viet Le, Patrick Nguyen, Alan Senior, Vincent Vanhoucke, Jeffrey Dean, et al. On rectified linear units for speech process- ing. In Acoustics, Speech and Signal Processing (ICASSP). 2013 IEEE International Conference on, pp. 3517-3521. IEFE, 2013.
10

---

Under review as a conference paper at ICLR 2020
Xiaowei Zhou, Can Yang, and Weichuan Yu. Moving object detection by detecting contiguous outliers in the low-rank representation. IEEE transactions on pattern analysis and machine intel- ligence, 35(3):597-610, 2013.


Reviews:
[{"review_id": "rJxRE-91cS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I found the topic of this paper interesting and I believe I understand what the authors are trying to achieve but I'm afraid this was after several readings and I do think the paper could be presented differently that would make it more accessible. My suggestion would be to explain how the model will be applied first (identify the required properties) to motivate the need for the learned basis and then present the DMD as a method for providing a basis that meets the properties required. I acknowledge that different communities have different styles of presentation so apologies if this is just me.\n\nFirst I would just like to check that I have understood correctly so please could the authors point out if I have missed something or misunderstood in the following?\n\nOur goal is to establish a basis invariant to the video dynamics that can then be used, for example, to partition the video into parts with differing dynamics - e.g. foreground/background. To do this we need to identify such a basis from a specific video - we will use the collection of pairs of neighboring frames.\n\nThe Koopman operator acts on a differential system to identify a function space invariant to the dynamics. If we instantiate this with a finite number of dimensions we can essentially establish the invariance as an eigenvalue problem. From this and our pairs of successive videos we can establish a vector basis for the space and then project the video into this basis. The spectral properties of the coefficients of the projection will determine whether something is static (omega = 0) or transitory in the scene and these can be used to identify foreground and background.\n\nNext there is the issue that this method operates in a linear domain with something like Gaussian noise which is not a good fit for image space videos so the authors propose to identify the dynamics in a linear latent space determined by an autoencoder to handle the non-linear mapping to image space.\n\nI hope I have understood the main points?\n\nIf this is the case, I think that much more needs to be said about the second part, which is the essential novelty of the paper, with a discussion of the merits of different approaches and full details - at the moment there is just one small paragraph at the end of 4.2 which contains the majority of the contribution.\n\nMy main concern about the paper is that I find it very difficult to appreciate the efficacy of the method given the current presentation of the results. There are no error bars to ascertain significance for any of the results and the summarization of multiple experiments to a single percentage gives very little insight into where this method works and where it doesn't. There are a number of ways that a dynamic prior could be added to a latent space and it is unclear why we would expect this approach to be preferred given the evidence presented in the paper.\n\nOther Notes:\n\nI found that the notation is not always consistent and sometimes could be simplified - it is unclear whether some operators are convolutions or multiplications (vector or scalar). To me the asterisk does not represent straight forward multiplication but it might be being used for this?\n\nCould Table 1 be placed in the experiments section rather than in the middle of page 5?\n\nDo the authors mean half the number of pixels or half the edge size (e.g. a quarter of the area) in terms of the latent space?\n\nPlease can all equations be numbered so that they can be referred to - there are no equation numbers in all of section 2.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Syl2HTO2tB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper considers the problem of extracting the underlying dynamics of objects in video frames. The paper focuses on two major applications: background/foreground extraction and video classification. The paper proposes a method that first obtains latent vectors from a video sequence by training a neural network and then applies dynamic mode decomposition (by Schmidt, 2010). \n\nThe paper is not well written and even after reading it the second time I, unfortunately, have difficulties understanding the exact contributions and experiments.\nHere are concrete examples that lead to this critique:\n- Section 2: Letters are not defined (e.g., \\mathcal F is not defined when first used), and sentences are not finished and/or do not make sense.\n- Sec. 5.1: not clear whether the experiment is only done for one sequence or for many. If done on only one sequence, this is not sufficient to demonstrate that the method works well, if done on more than one then the results are not reported.\n- Conclusion: States that ``this method can be applied to any multivariate-time series data to extract complex and non-linear dynamics''. That statement sounds overlay general given the experimental evaluation.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "SyeTbYISqH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents an application of convolutional autoencoder networks and a nonlinear dynamic systems analysis method known as extended dynamic mode analysis (EDMD) to a data-driven analysis of multivariate time series.  \nThe DMD method appears to be well-known in the physics community but is outside my area of expertise and unfortunately I have limited time to make a quick study of it.  However, from what I gather, it involves empirical approximation of a nonlinear dynamical system as a high-dimensional linear dynamical system, which in turn enables analysis in terms of eigendecomposition of the resulting linear operator, revealing basic modes of the dynamics.   In the proposed method, DMD is used in the latent representations of a convolutional autoencoder for image sequences.   The DMD objective is incorporated into the autoencoder training loss to minimize its reconstruction error.    The DMD is also used, by conditioning on the eigenvalues, to split the reconstruction into high frequency (quickly varying) foreground modes and low-frequency (slowly varying) background modes.   Although end-to-end training is mentioned, it is not made clear how the derivatives of the DMD decomposition are implemented, especially considering that the DMD involves an SVD, which can have unstable/ singular derivatives when two or more singular values are close to the same / exactly the same.   The resulting methods are applied to a foreground extraction and a classification tasks, and compared with numerous baselines.  It is not clear to me what the state of the art is on these tasks, but the proposed methods compare favorably to reported baselines, and the images of results look convincing.   However the experimental results seem a little thin and I would expect a more thorough study.  Overall the method looks very interesting.  \n\nSome complaints:\n -  the tables are a bit sloppy and should be formatted to fit in the document with normal sized fonts, \n - the images are too small to see well.\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper focuses on extracting the underlying dynamics of objects in video frames, for background/foreground extraction and video classification. Generally speaking, the presentation of the paper should be improved. Novelty should be clarified, contrasting the proposed approach with existing literature. All reviewers also agree the experimental section is also too weak in its current form.

Decision: Reject