# Online Structured Sparsity-Based Moving-Object Detection From Satellite Videos

Junpeng Zhang, Xiuping Jia, Jiankun Hu, Jocelyn Chanussot

► To cite this version:

Junpeng Zhang, Xiuping Jia, Jiankun Hu, Jocelyn Chanussot. Online Structured Sparsity-Based Moving-Object Detection From Satellite Videos. IEEE Transactions on Geoscience and Remote Sensing, 2020, 58 (9), pp.6420-6433. 10.1109/TGRS.2020.2976855. hal-03142158

HAL Id: hal-03142158

https://hal.science/hal-03142158v1

Submitted on 26 Sep 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
arXiv:1911.12989v3 [cs.CV] 10 Dec 2019

# Online Structured Sparsity-based Moving Object Detection from Satellite Videos

Junpeng Zhang, Student Member, IEEE, Xiuping Jia, Senior Member, IEEE, Jiankun Hu, Senior Member, IEEE and Jocelyn Chanussot, Fellow, IEEE

**Abstract**—Inspired by the recent developments in computer vision, low-rank and structured sparse matrix decomposition can be potentially be used for extract moving objects in satellite videos. This set of approaches seeks for rank minimization on the background that typically requires batch-based optimization over a sequence of frames, which causes delays in processing and limits their applications. To remedy this delay, we propose an Online Low-rank and Structured Sparse Decomposition (O-LSD). O-LSD reformulates the batch-based low-rank matrix decomposition with the structured sparse penalty to its equivalent frame-wise separable counterpart, which then defines a stochastic optimization problem for online subspace basis estimation. In order to promote online processing, O-LSD conducts the foreground and background separation and the subspace basis update alternately for every frame in a video. We also show the convergence of O-LSD theoretically. Experimental results on two satellite videos demonstrate the performance of O-LSD in term of accuracy and time consumption is comparable with the batch-based approaches with significantly reduced delay in processing.

**Index Terms**—Satellite Video Processing, Moving Object Detection, Online Robust Principle Component Analysis, Structured Sparsity-Inducing Norm, Background Subtraction

## I. INTRODUCTION

OBJECT detection on high resolution aerial images has been actively investigated in recent years [1], [2]. Inspired by the state-of-the-art Deep Learning methods, such as Regional Convolution Neural Network (R-CNN) [3], Fast R-CNN [4], Faster R-CNN [5], You Only Look Once (YOLO) [6] and Single Shot MultiBox Detector (SSD) [7], object detection performance on these images has been improved significantly [8], [9], [10], [11]. These approaches are mainly exploring the spectral (or color) and spatial (texture or context) information on objects of interest, they detect objects of interest image by image, as none temporal information is available on those images. Recently, with satellite videos captured by Jilin-1 [12] and Skybox [13], dense temporal information becomes available, which benefits moving objects detection from space. Target tracking becomes possible and can then be conducted for various applications [14], [15], [16], [17].

Detecting moving objects in a video is achieved by separating the temporal varying foreground, which is associated with the moving objects, and the background that lays in a low dimensional subspace from a video [18], [19], [20]. Given the moving objects account for a limited number of pixels in the foreground, the foreground is assumed sparse. Robust Principle Component Analysis (RPCA), as one fundamental method in foreground extraction, defines a low-rank matrix decomposition problem with a sparse penalty [21], which is

solved by Principle Component Pursuit (RPCA-PCP) [22], [21], [23] and Fast Low Rank Approximation (GoDec) [24]. Based on the duality between sparsity and Laplace distribution, Probabilistic Robust Matrix Factorization (PRMF) provides a probabilistic interpretation to RPCA by combining Laplace error and Gaussian prior [25].

As a moving object is commonly a set of neighboring pixels, spatial prior on the foreground is considered in low-rank matrix decomposition to improve moving object detection performance. Total Variation (TV) regularization is introduced to enforce smoothness on the foreground in the matrix decomposition [26]. DETecting Contiguous Outliers in the LOw-rank Representation (DECOLOR) constrains the edges of moving objects to be contiguous, then first-order Markov Random Field (MRF) is integrated into low-rank matrix decomposition [27]. Another possible spatial prior on the foreground is the sparsity over groups of spatial neighboring pixels on the foreground other than pixel-wise sparse, which is measured by Structured Sparsity-Inducing Norm [28]. Low-rank and Structured Sparse Decomposition (LSD) obeys this prior and penalizes the low-rank matrix decomposition by the structured sparsity-inducing norm of the foreground [29]. As moving object detection in satellite video is more sensitive to random noises, integrating spatial prior should improve the quality of the estimated foreground, thus the moving object detection performance. By integrating structured sparsity, LSD presents boosted Moving Object Detection (MOD) performance in satellite videos [30]. DECOLOR, however, has a limited improvement in MOD performance for satellite videos, as the introduced MRF constraint tends to merge neighboring targets when the distance between them is too small.

Regardless of the detection performance of the algorithms above, a pitfall of them is that their solutions are based on optimization in a batch manner. These approaches use Singular Value Decomposition (SVD) for low-rank background estimation, which couples all the samples in each iteration of the optimization. The detection results are not available until the optimization terminates, which results in delays in processing. Another shortcoming of batch-based approaches is the difficulty in handling a video with an incremental length. Both issues of the batch-based algorithms limit their application in various online systems.

In order to reduce the delay and to make MOD adaptable to videos of incremental length, online method is expected to sequentially estimate foreground and background for each new incoming frame. By low-rank matrix decomposition, the estimated Principal Component basis vectors represent a
<table><caption>TABLE I
COMPARISON ON ONLINE LOW-RANK MATRIX DECOMPOSITION ALGORITHMS FOR MOVING OBJECT DETECTION</caption><thead><tr><th>Method</th><th>Objective Function</th><th>Constraints</th><th>Spatial Prior</th><th>Optimization Scheme</th><th>Proven Convergence</th></tr></thead><tbody><tr><td>OPRMF [25]</td><td>min λ ||<b>D</b> − <b>L</b><b>R</b>||<sub>1</sub><br>+ λ<sub>1</sub> 2 ||<b>L</b>||<sub>F</sub><sup>2</sup> + λ<sub>2</sub> 2 ||<b>R</b>||<sub>F</sub><sup>2</sup></td><td><b>L</b><sub>ij</sub>|λ<sub>1</sub> ~ N(<b>L</b><sub>ij</sub>|0, λ<sub>1</sub><sup>−1</sup>),<br><b>R</b><sub>ij</sub>|λ<sub>2</sub> ~ N(<b>R</b><sub>ij</sub>|0, λ<sub>2</sub><sup>−1</sup>)</td><td>-</td><td>Online Expectation<br>Maximization</td><td>No</td></tr><tr><td>GRASTA [31]</td><td>min ||<b>s</b>||<sub>1</sub></td><td><b>d</b> = <b>L</b><b>r</b> + <b>s</b>,<br><b>L</b><sup>T</sup><b>L</b> = <b>I</b></td><td>-</td><td>Incremental Gradient<br>Descent Method on<br>Grassmannian Manifold</td><td>No</td></tr><tr><td>OR-PCA [32]</td><td>min 1/2 ||<b>D</b> − <b>L</b><b>R</b> − <b>S</b>||<sub>F</sub><sup>2</sup><br>+ λ<sub>1</sub> 2 ||<b>L</b>||<sub>F</sub><sup>2</sup> + λ<sub>1</sub> 2 ||<b>R</b>||<sub>F</sub><sup>2</sup><br>+ λ<sub>2</sub> ||<b>S</b>||<sub>1</sub></td><td>-</td><td>-</td><td>Stochastic<br>Optimization</td><td>Yes</td></tr><tr><td>COROLA [33]</td><td>min 1/2 ||<b>P</b><sub>S⊥</sub>(<b>D</b> − <b>L</b><b>R</b>)||<sub>F</sub><sup>2</sup><br>+ α/2 ||<b>P</b><sub>S⊥</sub>(<b>L</b>)||<sub>F</sub><sup>2</sup><br>+ α/2 ||<b>P</b><sub>S⊥</sub>(<b>R</b>)||<sub>F</sub><sup>2</sup><br>+ β ||<b>S</b>||<sub>1</sub> + γ ||<b>Avec</b>(<b>S</b>)||</td><td><b>S</b><sub>ij</sub> ∈ {0, 1}</td><td>Edge<br>Contiguousness</td><td>Stochastic<br>Optimization</td><td>Yes</td></tr><tr><td>GOSUS [34]</td><td>min ∑<sub>i=1</sub><sup>l</sup> μ<sub>i</sub> ||<b>G</b><b>s</b>||<sub>2</sub><br>+ λ/2 ||<b>L</b><b>r</b> + <b>s</b> − <b>d</b>||<sub>2</sub><sup>2</sup></td><td><b>L</b><sup>T</sup><b>L</b> = <b>I</b></td><td>Structured Sparsity<br>of Foreground</td><td>Incremental Gradient<br>Descent Method on<br>Grassmannian Manifold</td><td>No</td></tr><tr><td>Proposed O-LSD</td><td>min 1/2 ||<b>D</b> − <b>L</b><b>R</b> − <b>S</b>||<sub>F</sub><sup>2</sup><br>+ λ<sub>1</sub> 2 ||<b>L</b>||<sub>F</sub><sup>2</sup> + λ<sub>1</sub> 2 ||<b>R</b>||<sub>F</sub><sup>2</sup><br>+ λ<sub>2</sub> ∑<sub>s∈<b>S</b></sub> ||<b>s</b>||<sub>ℓ<sub>1</sub>/ℓ<sub>∞</sub></sub></td><td>-</td><td>Structured Sparsity<br>of Foreground</td><td>Stochastic<br>Optimization</td><td>Yes</td></tr></tbody></table>

subspace, where the background lays. This subspace can be identified by a point on the Grassmannian manifold, and the incremental gradient descent method on Grassmannian Manifold is employed for online subspace tracking or updating [36], [37], [38]. For moving object detection, Grassmannian Robust Adaptive Subspace Tracking Algorithm (GRASTA) [31] and Grassmannian Online Subspace Updates with Structured-sparsity (GOSUS) [34] are developed for online low-rank matrix decomposition with the pixel-wise sparse penalty and the structured sparse penalty, respectively. These set of approaches, however, provide no theoretical guarantee on their convergence, and their performance is heavily sensitive to the selection of the learning rate.

Another possibility for online low-rank matrix decomposition is the increasingly common matrix factorization approximation of nuclear norm, where rank-minimization is replaced by the sum of square penalties of its factorization [39], [32], [40], [33], [41]. With this reformulation, iterative optimization scheme is then developed based on stochastic optimization for solving low-rank decomposition problem online. Online Robust Principle Component Analysis (OR-PCA) solves the online low-rank matrix decomposition problem with pixel-wise sparsity penalty, which, more importantly, proves that iterative optimization algorithm converges to the global optimum of the original RPCA approach [32]. Utilizing first-order Markov Random Field, spatial prior on contiguous edges is also integrated with this reformulation for online low-rank matrix decomposition [33].

It has been observed that, by introducing the structured sparse penalty, LSD boosts the moving object detection performance in satellite videos [30]. While GOSUS combines online low-dimensional subspace tracking with structured sparsity, no theoretical guarantee on its convergence is provided. To the best of our knowledge, there exists a gap between online algorithm with theoretically guaranteed convergence and the

one with the structured sparsity penalty, as demonstrated in Table I. In order to fill this gap, we present an online low-rank matrix decomposition approach with structured sparse penalty, named as **Online Low-rank and Structured Sparse Decomposition (O-LSD)**, which not only combines the structured sparsity penalty but also provides theoretically guaranteed convergence. We follow the matrix factorization approximation of nuclear norm for online learning in [32], [40], [33], [41], and decompose the background matrix to a set of background frames that are reconstructed by the estimated subspace basis and their associated coefficients. To promote online processing, the proposed O-LSD algorithm is composed of two building blocks. For each frame, its corresponding foreground and background frames are reconstructed by the current subspace basis, then the subspace basis is updated for this new input. This procedure defines a stochastic optimization problem, and we show that O-LSD algorithm converges almost surely. Existing convergence analysis in [42], [32], [41], [43] is built on necessary and sufficient conditions on the unique solution in sparse encoding. In O-LSD, no such conditions exist for structured sparsity encoding to the best of our knowledge, and we show the convergence of O-LSD based on the boundedness of the sub-gradients in structured sparsity encoding, then a set of related properties of O-LSD are demonstrated. Experimental evaluations and analysis were performed on a satellite video dataset with two videos, where we compared our algorithm with five state-of-the-art algorithms.

In summary, the main contributions of this work are four-fold:

1) We propose an **Online Low-rank and Structured Sparse Decomposition (O-LSD)** for moving object detection in satellite videos by reformulating batch-based LSD using the matrix factorization approximation of nuclear norm.

2) To solve the new reformulated optimization problem,

<footnote>*There exist a variety of low-rank decomposition algorithm in the literature, however, we select the most related works here. Interested readers may refer to [18], [19], [20], [35] for more comprehensive reviews in low-rank matrix decomposition and their applications in video processing.</footnote>
two iterating steps are designed and developed. For each frame, the corresponding foreground and background frames are first reconstructed by the current subspace basis, then the subspace basis is updated by the given frame.

3) We show that O-LSD converges almost surely. In contrary to most current online algorithms, we show that O-LSD can converge without meeting the conditions for the unique solution of structured sparsity encoding. Due to the lack of these conditions, the solution of O-LSD can converge to neither a stationary point nor the global optimum of its batch-based counterpart LSD. This finding and its corresponding proof are useful beyond the scope of this paper.

4) Due to the better convergence characteristics of O-LSD, it can further reduce its processing delay with negligible effects on the detection performance, by down-sampling in the temporal domain.

The remainder of this paper is organized as follows. The proposed O-LSD is presented in Section II, where its convergence analysis is provided in Section II-E. The experimental parameter settings and performance comparison against state-of-the-art approaches are presented in Section III. Finally, conclusions and suggestions for future research are given in Section IV.

## II. PROPOSED METHOD

### A. Matrix Factorization to LSD

Low-rank and Structured Sparse Decomposition (LSD) seeks for a low-rank matrix decomposition of an observation matrix, which at the same time imposes the structured sparse penalty on the foreground. Given a fixed-length sequence from n video frames and each frame contains p pixels, LSD decomposes its corresponding matrix $\mathbf{D} \in \mathbb{R}^{p \times n}$ to a low-rank background matrix $\mathbf{B} \in \mathbb{R}^{p \times n}$ plus a structured sparse foreground matrix $\mathbf{S} \in \mathbb{R}^{p \times n}$, and defines a batch-based optimization problems as

$$ (\mathbf{B}^*, \mathbf{S}^*) = \arg \min_{\mathbf{B},\mathbf{S}} \|\mathbf{B}\|_* + \lambda \Omega(\mathbf{S}), \quad (1) \\ \text{s.t. } \mathbf{D} = \mathbf{B} + \mathbf{S} $$

where $\Omega(\mathbf{S})$ refers to the structured sparsity-inducing norm of $\mathbf{S}$ and $\lambda$ is a scalar that assigns the weight of structured sparsity. The structured sparsity-inducing norm [28], [44], [45] indicates the sparsity over groups of neighboring pixels as

$$ \Omega(\mathbf{S}) = \sum_{\mathbf{s} \in \mathbf{S}} \|\mathbf{s}\|_{\ell_1/\ell_\infty} = \sum_{\mathbf{s} \in \mathbf{S}} \sum_{\mathbf{g} \in \mathcal{G}} \eta_g \|\mathbf{s}_{|g}\|_\infty, \quad (2) $$

where $\mathcal{G}$ defines the set of groups of neighboring pixels, and $\mathbf{s}_{|g} \in \mathbb{R}^p$ is a sparse vector with non-zero elements at the indices represented in a group $g \in \mathcal{G}$. $\eta_g$ specifies the weight for a group of the pixels. In this paper, we assume each group contributes equally and assign 1.0 to it, $\eta_g = 1.0, \forall g \in \mathcal{G}$. In LSD, no temporal prior or constraints on the foreground are considered, thus the structured sparse penalty over a sequence of frames is frame-wise separable. In this paper, the groups of spatially related pixels $\mathcal{G}$ is constructed by 3 × 3 grid scanning

over the foreground, as the moving targets in satellite videos are usually in small scales.

Inspired by [32], the equality constraint in Eq. (1) is removed, and we obtain a reformulated optimization problem

$$ (\mathbf{B}^*, \mathbf{S}^*) = \arg \min_{\mathbf{B},\mathbf{S}} \lambda_1 \|\mathbf{B}\|_* + \lambda_2 \sum_{\mathbf{s} \in \mathbf{S}} \|\mathbf{s}\|_{\ell_1/\ell_\infty} \\ + \frac{1}{2} \|\mathbf{D} - \mathbf{B} - \mathbf{S}\|_F^2, \quad (3) $$

in which $\lambda_1 > 0$ and $\lambda_2 > 0$ are the corresponding weights for the low-rank penalty and the structured sparsity penalty.

Guided by the trending reformulation by matrix factorization in [32], [40], [35], we replace the low-rank term $\|\mathbf{B}\|_*$ in Eq. (3) by its approximation, which makes use of the following lemma.

**Lemma II.1.** Given that $\mathbf{B}$ is factorized as $\mathbf{B} = \mathbf{L}\mathbf{R}$, $L \in \mathbb{R}^{p \times r}$, $\mathbf{R} \in \mathbb{R}^{r \times n}$, the nuclear norm of $\mathbf{B}$ is upper bounded by the sum of Frobenius norms of $\mathbf{L}$ and $\mathbf{R}$, as

$$ \|\mathbf{B}\|_* = \inf_{\mathbf{L} \in \mathbb{R}^{p \times r}, \mathbf{R} \in \mathbb{R}^{r \times n}} \left\{ \frac{1}{2} \|\mathbf{L}\|_F^2 + \frac{1}{2} \|\mathbf{R}\|_F^2 : \mathbf{B} = \mathbf{L}\mathbf{R} \right\} \quad (4) $$

When $r > \text{Rank}(\mathbf{B})$, the jointly non-convex quadratic optimization problem Eq. (4) is equivalent to minimize the nuclear norm of $\mathbf{B}$. <fnref n="1" />

By substituting $\|\mathbf{B}\|_*$ with its factorized approximation, we rewrite the optimization problem in Eq. (3) to

$$ (\mathbf{L}^*, \mathbf{R}^*, \mathbf{S}^*) = \arg \min_{\mathbf{L}, \mathbf{R}, \mathbf{S}} \frac{1}{2} \|\mathbf{D} - \mathbf{L}\mathbf{R} - \mathbf{S}\|_F^2 + \frac{\lambda_1}{2} \|\mathbf{L}\|_F^2 \\ + \frac{\lambda_1}{2} \|\mathbf{R}\|_F^2 + \lambda_2 \sum_{\mathbf{s} \in \mathbf{S}} \|\mathbf{s}\|_{\ell_1/\ell_\infty}, \quad (5) $$

where $\mathbf{L} \in \mathbb{R}^{p \times r}$ is considered as the subspace basis of the background matrix $\mathbf{B}$, and $\mathbf{R} \in \mathbb{R}^{r \times n}$ is the coefficients to reconstruct $\mathbf{B}$ with given $\mathbf{L}$. $r$ is the estimated dimension of the subspace that the background frames lay in.

With a pair of estimated $\mathbf{L}$ and $\mathbf{R}$, each column vector in $\mathbf{R}$ corresponds to an estimated background frame in $\mathbf{B}$. Let $\mathbf{d}_t$, $\mathbf{r}_t$ and $\mathbf{s}_t$ refer to the t-th column of $\mathbf{D}$, $\mathbf{R}$ and $\mathbf{S}$ respectively, this optimization problem in Eq. (5) is equivalent to minimizing an empirical cost function

$$ f_n(\mathbf{L}) = \frac{1}{n} \sum_{i=1}^{n} \ell(\mathbf{D}_i, \mathbf{L}) + \frac{\lambda_1}{2n} \|\mathbf{L}\|_F^2, \quad (6) $$

in which $\ell(\mathbf{d}_i, \mathbf{L})$ is the reconstruction cost evaluated with fixed $\mathbf{L}$ by

$$ \ell(\mathbf{d}, \mathbf{L}) = \min_{\mathbf{r}, \mathbf{s}} \hat{\ell}(\mathbf{d}, \mathbf{L}, \mathbf{r}, \mathbf{s}), \\ \hat{\ell}(\mathbf{d}, \mathbf{L}, \mathbf{r}, \mathbf{s}) = \frac{1}{2} \|\mathbf{d} - \mathbf{L}\mathbf{r} - \mathbf{s}\|_2^2 + \frac{\lambda_1}{2} \|\mathbf{r}\|_2^2 + \lambda_2 \|\mathbf{s}\|_{\ell_1/\ell_\infty}. \quad (7) $$

Minimization of the empirical cost function in Eq. (6) associates the sum of reconstruction costs $\ell(\cdot, \mathbf{L})$, where, for each frame, $(\mathbf{r}, \mathbf{s})$ is optimized with the optimization target $\mathbf{L}$ as a

<footnote n="1">¹Please refer to [39] for detailed proof.</footnote>
parameter, whose formulation fits to the max-min optimization problem.

## B. Online LSD

Through the above reformulation, it is still impossible to update $\mathbf{L}$ without re-estimating all pairs of $(\mathbf{r}, \mathbf{s})$, which obstructs processing in an online fashion. In order to promote online processing, we propose an online algorithm, named **Online Low-Rank and Structured Sparse Decomposition (O-LSD)**, where Foreground and Background Separation and Subspace Basis Update are sequentially conducted for each frame.

O-LSD is an online algorithm that processes an input frame at each time instance in an online manner. At each time instance $t$, we have obtained $\mathbf{L}_{t-1}$ estimated from previous time instance $t-1$. The foreground frame and background frame are separated by solving the following optimization problem

$$ (\mathbf{r}_t^*, \mathbf{s}_t^*) = \arg \min_{\mathbf{r}_t, \mathbf{s}_t} \hat{\ell}(\mathbf{d}_t, \mathbf{L}_{t-1}, \mathbf{r}_t, \mathbf{s}_t). \quad (8) $$

We term this procedure as Foreground and Background Separation, which is detailed Section II-C.

Then subspace basis updating is performed with all pair of $\mathbf{r}_i$ and $\mathbf{s}_i$, $i \in \{1, \dots, t-1\}$. Directly minimizing the empirical cost function defined in Eq. (6) requires re-estimations on all pairs of $\mathbf{r}_i$ and $\mathbf{s}_i$. Instead, the subspace basis $\mathbf{L}_t$ is updated by minimizing a surrogate function of the empirical cost function $g_t(\mathbf{L}_t)$, which provides an upper bound for $f_t(\mathbf{L}_t)$ so that $g_t(\mathbf{L}_t) > f_t(\mathbf{L}_t)$. We define the surrogate function as

$$ \begin{aligned} g_t(\mathbf{L}_t) &= \frac{1}{t} \sum_{t-1}^{\infty} \hat{\ell}(\mathbf{d}_t, \mathbf{L}_t, \mathbf{r}_t, \mathbf{s}_t) + \frac{\lambda_1}{2t} \|\mathbf{L}_t\|_F^2 \\ &= \frac{1}{t} \sum_{i=1}^{t} \left( \frac{1}{2} \|\mathbf{d}_i - \mathbf{L}_t \mathbf{r}_i - \mathbf{s}_i\|_2^2 + \frac{\lambda_1}{2} \|\mathbf{r}_i\|_2^2 \right) \quad (9) \\ &\quad + \lambda_2 \|\mathbf{s}_i\|_{\ell_1/\ell_\infty} + \frac{\lambda_1}{2t} \|\mathbf{L}_t\|_F^2. \end{aligned} $$

The minimization of $g_t(\mathbf{L}_t)$ with respect to $\mathbf{L}_t$ is termed as Subspace Basis Update, which is then explained in Section II-D. The entire O-LSD algorithm is summarized in Algorithm 1.

## C. Foreground and Background Separation

Foreground and Background Separation obtains a pair of $\mathbf{r}_t^*$ and $\mathbf{s}_t^*$ by solving the optimization problem defined in Eq. (8), where $\mathbf{L}_{t-1}$ is provided by the previous time instance $t-1$. As $[\mathbf{L} \mathbf{I}]^T [\mathbf{L} \mathbf{I}]$ is always positive semi-definite, the objective function of Eq. (8) is convex with respect to $(\mathbf{r}_t, \mathbf{s}_t)$. For solving this convex optimization problem, instead of solving $\mathbf{r}_t$ and $\mathbf{s}_t$ together, we adopt a Block Coordinate Descent (BCD) method [46], where $\mathbf{r}_t$ and $\mathbf{s}_t$ are alternately updated by fixing each other.

By fixing $\mathbf{s}_t$, $\mathbf{r}_t^*$ is obtained by solving

$$ \mathbf{r}_t^* = \arg \min_{\mathbf{r}_t} \frac{1}{2} \|\mathbf{d}_t - \mathbf{L}_{t-1} \mathbf{r}_t - \mathbf{s}_t\|_2^2 + \frac{\lambda_1}{2} \|\mathbf{r}_t\|_2^2, \quad (10) $$

## Algorithm 1 Proposed O-LSD Algorithm for MOD

**Input:** $\mathbf{d}_t \in \mathbb{R}^p$, $\mathbf{L}_{t-1} \in \mathbb{R}^{p \times r}$, $\mathbf{A}_{t-1}$ and $\mathbf{B}_{t-1}$

**Output:** $\mathbf{b}_t$, $\mathbf{r}_t$, $\mathbf{s}_t$ and $\mathbf{L}_t$

1. Separate the foreground and background:

$$ (\mathbf{r}_t^*, \mathbf{s}_t^*) = \arg \min_{\mathbf{r}_t, \mathbf{s}_t} \frac{1}{2} \left\| \mathbf{d}_t - \mathbf{L}_{t-1} \mathbf{r}_t - \mathbf{s}_t \right\|_2^2 + \frac{\lambda_1}{2} \left\| \mathbf{r}_t \right\|_2^2 + \lambda_2 \left\| \mathbf{s}_t \right\|_{\ell_1/\ell_\infty}, $$

which is solved by Algorithm 2.

2. Compute the background frame: $\mathbf{b}_t = \mathbf{L}_{t-1} \mathbf{r}_t$.

3. Update the accumulation matrices $\mathbf{A}_t$ and $\mathbf{B}_t$ by Eq. (17).

4. Update the subspace basis $\mathbf{L}_t$:

$$ \mathbf{L}_t^* = \arg \min_{\mathbf{L}_t} \mathrm{Tr}(\mathbf{L}_t^T (\lambda_1 \mathbf{I} + \mathbf{A}_t) \mathbf{L}_t) - 2 \mathrm{Tr}(\mathbf{L}_t^T \mathbf{B}_t), $$

whose solution is presented in Algorithm 3.

5. **return** $\mathbf{b}_t$, $\mathbf{r}_t$, $\mathbf{s}_t$ and $\mathbf{L}_t$.

which constructs a least-square problem, and its closed-form solution is given by

$$ \mathbf{r}_t^* = (\mathbf{L}_{t-1}^T \mathbf{L}_{t-1} + \lambda_1 \mathbf{I})^{-1} (\mathbf{d}_t - \mathbf{s}_t). \quad (11) $$

Using fixed $\mathbf{r}_t$, let $\mathbf{u} = \mathbf{d}_t - \mathbf{L}_{t-1} \mathbf{r}_t$, then the sub-problem for estimating the structured sparsity $\mathbf{s}_t$ is defined as

$$ \mathbf{s}_t^* = \arg \min_{\mathbf{s}_t} \frac{1}{2} \|\mathbf{u} - \mathbf{s}_t\|_2^2 + \lambda_2 \|\mathbf{s}_t\|_{\ell_1/\ell_\infty}, \quad (12) $$

whose solution is obtained by its dual problem that define a Quadratic Min-Cost Flow problem [42], [47], [48] as

$$ \begin{gathered} \xi^* = \arg \min_{\xi} \frac{1}{2} \left\| \mathbf{u} - \sum_{g \in \mathcal{G}} \xi^g \right\|_2^2, \\ \text{s.t.} \quad \forall g \in \mathcal{G}, \|\xi^g\|_1 \le \lambda_2 \text{ and } \xi_j^g = 0 \text{ if } j \notin g \end{gathered} \quad (13) $$

where $\xi^g \in \mathbb{R}^p$, $\forall g \in \mathcal{G}$ denotes the corresponding dual variables for the group of variables in $g$, and $\xi$ is the set of all $\xi^g$, $\forall g \in \mathcal{G}$. The primal solution $\mathbf{s}_t$ is then obtained by

$$ \mathbf{s}_t^* = \mathbf{u} - \sum_{g \in \mathcal{G}} \xi^{*g}. \quad (14) $$

The iteration of alternatingly estimation of $\mathbf{r}_t$ and $\mathbf{s}_t$ continues until the stop criterion is reached:

$$ \max\{\|\mathbf{r}'_t - \mathbf{r}''_t\|_2, \|\mathbf{s}'_t - \mathbf{s}''_t\|_2\} \le \tau, \quad (15) $$

where $(\mathbf{r}'_t, \mathbf{s}'_t)$ and $(\mathbf{r}''_t, \mathbf{s}''_t)$ are two pairs of estimation solutions at two consecutive iterations. Similar to [33], [35], the stop criterion is set as $\tau = 1.0e-5$ in this paper. The BCD algorithm for Foreground and Background Separation is summarized in Algorithm 2.
**Algorithm 2** Block Coordinate Descent Method for Foreground and Background Separation

**Input:** $\mathbf{L}_{t-1} \in \mathbb{R}^{p^{*}r}$, $\lambda_1 > 0$ and $\lambda_2 > 0$

**Output:** $\mathbf{r}_t$ and $\mathbf{s}_t$

1: $\mathbf{r}_t = \mathbf{0}$, $\mathbf{s}_t = \mathbf{0}$.

2: **while** not converged **do**

3: Estimate $\mathbf{r}_t$:

$$ \mathbf{r}_t = (\mathbf{L}_{t-1}^T \mathbf{L}_{t-1} + \lambda_1 I)^{-1}(\mathbf{d}_t - \mathbf{s}_t). $$

4: Estimate $\mathbf{s}_t$ by solving its dual problem on $\xi$ defined in Eq. (13), then

$$ \mathbf{s}_t = \mathbf{d}_t - \mathbf{L}_{t-1}\mathbf{r}_t - \sum_{g \in \mathcal{G}} \xi^{*g}. $$

5: Check the convergence using Eq. (15).

6: **end while**

7: **return** $\mathbf{r}_t$ and $\mathbf{s}_t$

### D. Subspace Basis Update

After estimating $(\mathbf{r}_t, \mathbf{s}_t)$, the subspace basis $\mathbf{L}_t$ is updated by minimizing the surrogate function of empirical cost function $g_t(\mathbf{L}_t)$, which defines an optimization problem as

$$ \begin{aligned} \mathbf{L}_t^* &= \arg\min_{\mathbf{L}_t} g_t(\mathbf{L}_t) \\ &= \arg\min_{\mathbf{L}_t} \mathrm{Tr}(\mathbf{L}_t^T(\lambda_1 \mathbf{I} + \mathbf{A}_t)\mathbf{L}_t) - 2 \mathrm{Tr}(\mathbf{L}_t^T \mathbf{B}_t), \end{aligned} \quad (16) $$

in which $\mathrm{Tr}(\cdot)$ denotes the trace of a matrix, and $\mathbf{A}_t$ and $\mathbf{B}_t$ are two auxiliary accumulation matrices that are introduced to remove duplicated calculations at each time instance,

$$ \left\{ \begin{aligned} \mathbf{A}_t &= \mathbf{A}_{t-1} + \mathbf{r}_t \mathbf{r}_t^T \\ \mathbf{B}_t &= \mathbf{A}_{t-1} + (\mathbf{d}_t - \mathbf{s}_t) \mathbf{r}_t^T. \end{aligned} \right. \quad (17) $$

Similar to [32], [33], [41], the optimization problem defined in Eq. (16) is solved by a Block Coordinate Descent Method for avoiding matrix inverse of large matrix. The Subspace Basis Update algorithm is illustrated in Algorithm 3.

**Algorithm 3** Block Coordinate Descent Method for Subspace Basis Update

**Input:** $\mathbf{L}_{t-1} = [\mathbf{l}_1, \dots, \mathbf{l}_r] \in \mathbb{R}^{p^{*}r}$, $\mathbf{A}_t = [\mathbf{a}_1, \dots, \mathbf{a}_r] \in \mathbb{R}^{r^{*}r}$, $\mathbf{B}_t = [\mathbf{b}_1, \dots, \mathbf{b}_r] \in \mathbb{R}^{p^{*}r}$ and $\lambda_1 > 0$

**Output:** $\mathbf{L}_t$

1: $\tilde{\mathbf{A}} = \mathbf{A}_t + \lambda_1 \mathbf{I}$.

2: **for** $i = 1$ to $r$ **do**

3: $\mathbf{l}_i = \frac{1}{\tilde{\mathbf{A}}_{i,i}}(\mathbf{b}_i - \mathbf{L}_{t-1}\mathbf{a}_i) + \mathbf{l}_i$.

4: **end for**

5: $\mathbf{L}_t = [\mathbf{l}_1, \dots, \mathbf{l}_r]$.

6: **return** $\mathbf{L}_t$

The subspace basis $\mathbf{L}_0$ is initialized before starting O-LSD. $\mathbf{L}_0$ can be initialized by either the first a few frames in the given sequence or their Principal Components [35], [34]. In satellite videos, moving objects move slowly, and choosing these initialization scheme risks including the slow moving foreground objects into the background, which thus influences the detection performance, or a more extended sequence for

initialization is required. Such a satellite video with adequate length is, however, not available technically yet. Therefore, in satellite videos, we recommend initializing $\mathbf{L}_0$ by random values instead, which performs pretty well in practice.

### E. Convergence Analysis

One technical contribution of this paper is to present the proposed O-LSD algorithm converges almost surely under mild condition.

**Assumption 1.** The observed data are uniformly bounded, and each data is independent.

As a widely-used assumption [42], [32], [41], this assumption on the boundedness of observation data is quiet natural for real videos. Based on the above assumption, we present our first conclusion on the convergence of the surrogate function $g_t(\mathbf{L}_t)$.

**Theorem II.2.** Let $\{\mathbf{L}_t\}_{t=1}^\infty$ be the sequence of solution obtained by Algorithm 1, the surrogate function $g_t(\mathbf{L}_t)$ converges almost surely.

Similarly, we obtain the convergence of two solutions obtained at two consecutive time instance by Algorithm 1.

**Theorem II.3.** For two solutions produced by Algorithm 1 at two consecutive time instances, $\|\mathbf{L}_t - \mathbf{L}_{t+1}\|_F = O(\frac{1}{t})$.

Then, we analyze the gap between the empirical cost function $f_t(\mathbf{L}_t)$ and its surrogate function $g_t(\mathbf{L}_t)$ with the estimated $\mathbf{L}_t$.

**Theorem II.4.** Note $f_t(\mathbf{L})$ is the empirical cost function defined in Eq. (6), and $g_t(\mathbf{L})$ is its surrogate function defined in Eq. (9). $\mathbf{L}_t$ is the solution obtained by Algorithm 1, when $t$ tends to infinity, $g_t(\mathbf{L}_t) - f_t(\mathbf{L}_t)$ converges to 0 almost surely.

In stochastic optimization, the expected cost function over $\mathbf{L}$ is defined as

$$ f(\mathbf{L}) = \mathbb{E}_{\mathbf{d}}[\ell(\mathbf{d}, \mathbf{L})] = \lim_{t \to \infty} f_t(\mathbf{L}), \quad (18) $$

then we present the convergence of the gap between the expected cost function $f(\mathbf{L}_t)$ and the surrogate function $g_t(\mathbf{L}_t)$.

**Theorem II.5.** As $t$ tends to infinity, given the $\mathbf{L}_t$ is obtained by Algorithm 1, $g_t(\mathbf{L}_t) - f(\mathbf{L}_t)$ converges to 0 almost surely.

Furthermore, the solution $\mathbf{L}_t$ obtained by Algorithm 1 is not a stationary point of expected cost function $f(\mathbf{L})$, when $t$ tends to infinity, which on contrary is proved true in [42], [32], [41]. Due to the existence of more than one solutions to Eq. (8), $\ell(\mathbf{d}, \mathbf{L}, \mathbf{r}, \mathbf{s})$ is no longer strictly convex (or strongly convex) with respect to $(\mathbf{r}, \mathbf{s})$, and the gradient of the expected cost function $\nabla_{\mathbf{L}}f(\mathbf{L})$ is no longer Lipschitz. Therefore, the gradient of the expected cost function $\nabla_{\mathbf{L}}f(\mathbf{L})$ would not become zero when $t$ tends to infinity, based on which we conclude the solution $\mathbf{L}_t$ may not be the stationary point of the expected cost function as $t$ tends infinity.

Please refer to the appendices for detailed proofs of the presented theorems.
### III. EXPERIMENTS

The detection performance of O-LSD was evaluated on a dataset of two satellite videos. This dataset is constructed from a satellite video captured over Las Vagas, USA on March 25, 2014, whose spatial resolution is 1.0 meter and the frame rate is 30 frames per second. Both videos contains 700 frame with boundary boxes for moving vehicles as groundtruth, and details on both videos are listed in Table. II <fnref n="2" />. In this paper, we used the first 200 frames in each video for the discussion on parameter selection, and the remaining frames were utilized for performance evaluation against existing state-of-the-art methods.

<table><caption>TABLE II
INFORMATION ON THE EVALUATION DATASETS</caption><thead><tr><th rowspan="2">Video</th><th rowspan="2">Frame Size</th><th colspan="2">Cross Validation</th><th colspan="2">Performance Evaluation</th></tr><tr><th>#Frames</th><th>#Vehicles</th><th>#Frames</th><th>#Vehicles</th></tr></thead><tbody><tr><td>001</td><td>400 × 400</td><td>200</td><td>9306</td><td>500</td><td>18167</td></tr><tr><td>002</td><td>600 × 400</td><td>200</td><td>13443</td><td>500</td><td>39362</td></tr></tbody></table>

The detection performance on moving object detection is evaluated on recall, precision and $F_1$ scores given by

$$ 
\begin{aligned}
\text{recall} &= \frac{TP}{TP + FN} \\
\text{precision} &= \frac{TP}{TP + FP}, \\
F_1 &= \frac{2 \times \text{recall} \times \text{precision}}{\text{recall} + \text{precision}}
\end{aligned}
\qquad (19)
$$

where *TP* denotes the number of correct detections, *FN* and *FP* are the numbers of missed detections and false alarms, respectively. In this paper, we define a correct detection with maximum Intersection over Union (IoU) against the groundtruth greater than a threshold. To complement the vehicles in small size in satellite videos, the threshold is set as 0.3 <fnref n="3" />. In this paper, we refer 5-Frame detection performance at each time instance to the metrics obtained from its 5 latest frames, which is used for observing the convergence, and the accumulated detection performance is measured on all frames before current time instances, which is used for comparing the overall performance over a sequence.

#### A. Parameter Setting

The performance of O-LSD is controlled by the dimension of the estimated subspace *r*, and two weights for the low-rank term and the structured sparsity penalty separably, $\lambda_1$ and $\lambda_2$. The following experiments are conducted on the cross-validation sequence from Video 001.

The weight $\lambda_1$ assigns the importance of the low-rank subspace term. With the fixed $\lambda_2$, a significantly small $\lambda_1$ would encourage more information encoded in the low-rank subspace factors and hurt the detection performance in both terms of recall and precision. Increasing $\lambda_1$ improves the

<figure><img src="image_2.png" /><figcaption>(a) Varying $\lambda_1$ with $\lambda_2 = 0.025$</figcaption></figure>

<figure><img src="image_3.png" /><figcaption>(b) Varying $\lambda_2$ with $\lambda_1 = 0.0025$</figcaption></figure>

<caption>Fig. 1. Moving object detection performance with different $\lambda_1$ and $\lambda_2$.</caption>

detection performance by the increased emphasis on the low-rank subspace modeling. After approaching the best detection performance, continuing increasing $\lambda_1$ would prevent the information encoded into the background. As illustrated in Fig. 1a, with the fixed $\lambda_2 = 0.025$ and *r* = 5, as $\lambda_1$ increases, the $F_1$ score gradually increases to about 75% from around 60%, then the detection performance starts dropping with continuously increasing $\lambda_1$.

Increasing $\lambda_2$ with fixed $\lambda_1$ would put more emphasis on the structured sparsity of the extracted foreground, which thus improves the precision of the detected moving objects by restraining the random noises. When $\lambda_2$ continuously increases, the weight for the structured sparsity norm tends too large to encode information into the foreground, which then decreases the detection performance. As illustrated in Fig. 1b, with fixed $\lambda_1 = 0.0025$ and *r* = 5, the $F_1$ score first approaches to the highest point as $\lambda_2$ increases, then the same metric drops when $\lambda_2$ tends too large.

Then we discuss the selection of the dimension of the subspace *r*. With fixed $\lambda_1$ and $\lambda_2$, the O-LSD with smaller *r* probably converges faster, however, it may fail in modeling the permutation of the background for a long video sequence. On the contrary, selecting a higher *r* would disadvantage the updating of subspace basis and require more frames before O-LSD converges. As presented in Fig. 2, for $r \le 10$, the 5-frame $F_1$ scores increase faster than those with $r \ge 25$. As the sequence length increases, 5-frame $F_1$ scores by the O-LSD with $r \le 10$ show a trend of dropping. For $r = 25$, this trend is negligible, and the same metric is still rising for $r > 25$, which means the O-LSD requires a longer sequence to converge. The highest $F_1$ score at Frame-200 is achieved by *r* = 25 in Fig. 2.

In the rest of the paper, based on cross-evaluation, we select $\lambda_1 = 1/\sqrt{p}$ and $\lambda_1/\lambda_2 = 0.1$ with *r* = 25 for evaluation, and further fine-tuning on the parameter selection would improve the detection performance by O-LSD.

<footnote n="2">²Moving vehicles are manually labeled by the Computer Vision Annotation Tool (CVAT), and a boundary box is provided for each moving object on each frame.</footnote>

<footnote n="3">³The estimated foreground is built by contiguous values rather than binary value, so we deploy threshold segmentation as post-processing for extracting the foreground mask and the moving objects [49].</footnote>
<figure><img src="image_4.png" /><figcaption>Fig. 2. 5-Frame performance evaluation of O-LSD with different *r* on cross-validation sequence from Video 001.</figcaption></figure>

<table><caption>TABLE III
DETECTION PERFORMANCE COMPARISON WITH ONLINE ALGORITHMS</caption><thead><tr><th rowspan="2">Video</th><th colspan="3">001</th><th colspan="3">002</th><th rowspan="2">Avg(F<sub>1</sub>)</th></tr><tr><th>Recall</th><th>Precision</th><th>F<sub>1</sub> score</th><th>Recall</th><th>Precision</th><th>F<sub>1</sub> score</th></tr></thead><tbody><tr><td>GRASTA</td><td><strong>76.96%</strong></td><td>31.76%</td><td>44.97%</td><td>72.22%</td><td>46.73%</td><td>56.75%</td><td>50.86%</td></tr><tr><td>OR-PCA</td><td>66.51%</td><td>41.50%</td><td>51.11%</td><td>71.94%</td><td>73.79%</td><td>72.86%</td><td>61.99%</td></tr><tr><td>GUSOS</td><td>59.35%</td><td>49.15%</td><td>53.77%</td><td>68.03%</td><td>62.61%</td><td>65.21%</td><td>59.49%</td></tr><tr><td><strong>O-LSD</strong></td><td>64.99%</td><td><strong>63.75%</strong></td><td><strong>64.36%</strong></td><td><strong>73.00%</strong></td><td><strong>90.21%</strong></td><td><strong>80.69%</strong></td><td><strong>72.48%</strong></td></tr></tbody></table>

## B. Comparison with Online Approach

To verify the effectiveness of O-LSD, detection results by O-LSD are compared against the state-of-the-art online approaches, GRASTA [31], OR-PCA [32] and GOSUS [34]. For all methods, the subspace is initialized by the random scheme, and their parameters are selected through cross-validation.

O-LSD boosts the detection performance in terms of the precision and $F_1$ scores. This improvement should be owned to the structured sparsity penalty, which suppress the random noises in estimated foreground frames. As present in Table. III, the O-LSD achieves the highest precision and $F_1$ scores on both videos, although there is a little drop of recall score on Video 001, compared with GRASTA and OR-PCA. In terms of online methods employing structured sparsity penalty, O-LSD outperforms GOSUS on both satellite videos. For common moving object detection tasks GOSUS is proved effective, however, it produces no improvement to OR-PCA on the satellite videos.

Another advantage of O-LSD is its faster convergence against other state-of-the-art online algorithms. As demonstrated in Fig. 3, on both videos, the O-LSD achieves the higher $F_1$ scores earlier than GRASTA, OR-PCA and GOSUS, implying that O-LSD converges faster by introducing the structured sparsity penalty. Similar trends can also be observed from the perspective of accumulated detection performance, as the accumulated detection performance of O-LSD is always better than the three existing methods as the length of sequence increases, as illustrated in Fig. 4.

Besides, as more frames are processed by O-LSD algorithm, the cleanness of the estimated background frame gradually improves, as shown in Fig. 5.

## C. Comparison with Batch-based Approach

Besides the comparison against the state-of-the-art online algorithms, we also compare O-LSD with the batch-based

algorithms, which are RPCA [50] and LSD [29]. O-LSD achieves slightly descent detection performance with significantly reduced delay in processing on both videos.

Compared with batch-based approaches, O-LSD achieves comparable performance with the batch-based approach RPCA, since it generates less false alarms, as visualized in Fig. 6. O-LSD fails in matching the detection performance by its batch-based counterpart, LSD, and Table. V presents a drop of about 10% in $F_1$ score by O-LSD. This little gap in detection performance between LSD- and O-LSD implies that O-LSD works pretty well, although it may not converge to the global optimum of LSD.

In term of the processing time for each frame, O-LSD significantly reduces this metric, compared with LSD. As presented in Table. V, the time cost per frame for O-LSD is ten times smaller than LSD. Compared with RPCA, O-LSD improves the detection performance with moderately increased time cost per frame. As the detection results by those batch-based approaches are not available until the entire optimization is completed, O-LSD significantly reduces the delay in moving object detection by the O-LSD.

<table><caption>TABLE IV
DETECTION PERFORMANCE BY O-LSD WITH TEMPORALLY DOWN-SAMPLING</caption><thead><tr><th>Video</th><th><em>T</em></th><th>Recall</th><th>Precision</th><th><em>F</em><sub>1</sub></th><th>Time per Frame</th></tr></thead><tbody><tr><td rowspan="4">001</td><td>1</td><td>64.99%</td><td>63.75%</td><td>64.36%</td><td>6.57s</td></tr><tr><td>3</td><td>67.85%</td><td>59.79%</td><td>63.57%</td><td>12.74s</td></tr><tr><td>5</td><td>69.22%</td><td>62.02%</td><td>65.42%</td><td>14.62s</td></tr><tr><td>10</td><td>67.72%</td><td>62.14%</td><td>64.91%</td><td>14.26s</td></tr><tr><td rowspan="4">002</td><td>1</td><td>73.00%</td><td>90.21%</td><td>80.69%</td><td>10.75s</td></tr><tr><td>3</td><td>74.21%</td><td>88.46%</td><td>80.71%</td><td>14.48s</td></tr><tr><td>5</td><td>74.41%</td><td>87.72%</td><td>80.52%</td><td>17.86s</td></tr><tr><td>10</td><td>73.46%</td><td>88.68%</td><td>80.36%</td><td>21.12s</td></tr></tbody></table>
<figure><img src="image_5.png" /><figcaption>Fig. 3. 5-Frame detection performance by different online algorithms.</figcaption></figure>

<figure><img src="image_6.png" /><figcaption>Fig. 4. Accumulated detection performance by different online algorithms.</figcaption></figure>

<table><caption>TABLE V
DETECTION PERFORMANCE COMPARISON AGAINST BATCH-BASED ALGORITHMS</caption><thead><tr><th rowspan="2">Video</th><th colspan="4">001</th><th colspan="4">002</th><th rowspan="2">Avg(F<sub>1</sub>)</th></tr><tr><th>Recall</th><th>Precision</th><th>F<sub>1</sub> score</th><th>Time Per Frame</th><th>Recall</th><th>Precision</th><th>F<sub>1</sub> score</th><th>Time Per Frame</th></tr></thead><tbody><tr><td>RPCA</td><td><strong>94.57%</strong></td><td>40.65%</td><td>56.86%</td><td><strong>3.16s</strong></td><td><strong>90.15%</strong></td><td>78.06%</td><td>83.67%</td><td><strong>5.45s</strong></td><td>70.27%</td></tr><tr><td>LSD</td><td>86.80%</td><td><strong>70.79%</strong></td><td><strong>77.98%</strong></td><td>68.48s</td><td>82.19%</td><td><strong>90.87%</strong></td><td><strong>86.31%</strong></td><td>119.50s</td><td><strong>82.15%</strong></td></tr><tr><td><strong>O-LSD</strong></td><td>64.99%</td><td><u>63.75%</u></td><td><u>64.36%</u></td><td>6.57s</td><td>73.00%</td><td><u>90.21%</u></td><td>80.69%</td><td>10.75s</td><td><u>72.48%</u></td></tr></tbody></table>

## D. Performance Evaluation with Temporally Down-sampling

Additionally, we evaluate the effects of temporally down-sampling on detection performance. In this set of experiments, one frame from every T frames, $T \in \{1, 3, 5, 10\}$, is fed to O-LSD.

As shown in Table. IV, with increasing T from 1 to 10, the $F_1$ scores fluctuate negligible, which implies that detection performance of O-LSD is almost not influenced by the temporally down-sampling. At the same time, the time costs for each frame are more or less the same with different temporally down-sampling frequencies. Since fewer frames need to be processed with large temporally down-sampling scales, the delay for processing is reduced. However, considering that it takes more than 1 second to obtain detection results for each frame by O-LSD, it is still hard to directly apply O-LSD for real-time applications without appropriate accelerations.

## IV. CONCLUSION

The main contribution of this paper is an effective algorithm, Online Low-rank and Structured Sparse Decomposition (O-LSD), which combines the stochastic optimization and structured sparsity penalty to improve online subspace estimation

method for moving object detection in satellite videos. We elaborate the model of O-LSD and its optimization method that is proved to converge almost surely under mild condition. The experiments on a dataset of two satellite videos validate the improvement of O-LSD to the existing state-of-the-art approaches. With temporal down-sampling scheme, O-LSD also reduces the processing delay with almost unchanged performance.

## ACKNOWLEDGMENT

This work is partially supported by China Scholarship Council. The authors would like to thank Planet Team for providing the data in this research [13].

## APPENDIX A
TECHNICAL LEMMA

**Lemma A.1** (Danskin's Theorem from [51]). Let $C \subset \mathbb{R}^m$ be a compact set. The function $\ell(\mathbf{x}, \mathbf{u}) : \mathbb{R}^n \times C \to \mathcal{R}$ is continuous, and $\ell(\cdot, \mathbf{u})$ is convex with regards to $\mathbf{x}$ for every $\mathbf{u} \in C$. Define $\ell(\mathbf{x}) = \min_{\mathbf{u} \in C} \ell(\mathbf{x}, \mathbf{u})$ and $C(\mathbf{x}) = \{\mathbf{u}^* | \mathbf{u}^* = \arg \min_{\mathbf{u} \in C} \ell(\mathbf{x}, \mathbf{u})\}$.
<figure><img src="image_7.png" /><figcaption>Fig. 5. Visualization of the estimated background and detection by O-LSD on Video 001. Images in the 1st row are the original input, and those in the 2nd and 3rd rows are the estimated background frames and detection results, respectively.</figcaption></figure>

<figure><img src="image_8.png" /><figcaption>Fig. 6. Detection results obtained by different algorithms.</figcaption></figure>

If $\ell(\mathbf{x}, \mathbf{u})$ is differentiable with respect to $\mathbf{x}$ for all $\mathbf{u} \in C$, and $\frac{\partial \ell(\mathbf{x}, \mathbf{u})}{\partial \mathbf{x}}$ is continuous with respect to $\mathbf{u}$ for all $\mathbf{x}$, then the sub-gradient of $\ell(\mathbf{x})$ is given by

$$ \partial_{\mathbf{x}} \ell(\mathbf{x}) = \operatorname{conv}\left\{\frac{\partial \ell(\mathbf{x}, \mathbf{u})}{\partial \mathbf{x}} \mid \mathbf{u} \in C(\mathbf{x})\right\} \quad (20) $$

where $\operatorname{conv}\{\cdot\}$ indicates the convex hull operator.

**Lemma A.2** (Lemma 2.6 from [52]). Let $f: \mathbf{X} \to \mathbb{R}$ be a convex function. Then, $f$ is L-Lipschitz over $\mathbf{X}$ with respect to a norm $\|\cdot\|$ if and only if for all $\forall \mathbf{w} \in \mathbf{X}$ and $\mathbf{z} \in \partial f(\mathbf{w})$ we have that $\|\mathbf{z}\|_* \le L$, where $\|\cdot\|_*$ is the dual norm.

**Lemma A.3** (Sufficient condition of convergence for a stochastic optimization from [52]). Let $(\Omega, \mathcal{F}, P)$ be a measurable probability space, $\mu_t$, for $t > 0$, be the realization of a stochastic process and $\mathcal{F}_t$ be the filtration by the past information at time $t$. Let

$$ \delta_t = \begin{cases} 1, & \text{if } \mathbb{E}[u_{t+1} - u_t | \mathcal{F}_t] > 0, \\ 0, & \text{otherwise.} \end{cases} \quad (21) $$

If for all $t$, $\mu_t \ge 0$ and $\sum_{t=1}^{\infty} \mathbb{E}[\delta_t(u_{t+1} - u_t)] < \infty$, then $\mu_t$ is a quasi-martingale and converges almost surely. Moreover,
$$ \sum_{t=1}^{\infty} |\mathbb{E}[u_{t+1} - u_t | \mathcal{F}_t]| < +\infty \text{ a.s.} \quad (22) $$

**Lemma A.4** (Corollary of Donsker theorem from [53]). Let $\mathbf{F} = \{f_\theta : \mathcal{X} \to \mathbb{R}, \theta \in \Theta\}$ be a set of measurable functions indexed by a bounded subset $\Theta$ of $\mathbb{R}^d$. Suppose that there exists a constant $K$ such that

$$ |f_{\theta_1} - f_{\theta_2}| \le K \|\theta_1 - \theta_2\|_2, \quad (23) $$

for every $\theta_1$ and $\theta_2$ in $\Theta$ and $\mathbf{x}$ in $\mathcal{X}$. Then $\mathbf{F}$ is P-Donsker. For any $f$ in $\mathbf{F}$, let us define $\mathbb{P}_n f$, $\mathbb{P} f$ and $\mathbb{G}_n f$ as

$$ \begin{aligned} \mathbb{P}_n f &= \frac{1}{n} \sum_{i=1}^{n} f(\mathbf{X}_i), \\ \mathbb{P} f &= \mathbb{E}[f(X)], \\ \mathbb{G}_n f &= \sqrt{n}(\mathbb{P}_n f - \mathbb{P} f). \end{aligned} \quad (24) $$

Let us also suppose that for all $\mathbb{P}f^2 \le \delta^2$ and $\|f\|_\infty \le M$ and that the random variables $\mathbf{X}_1, \mathbf{X}_2, \dots$ are Borel-measurable. Then, we have

$$ \mathbb{E} |\mathbb{G}_n f|_F = O(1), \quad (25) $$

where $|\mathbb{G}_n f|_F = \sup_{f \in F} |\mathbb{G}_n f|$.

**Lemma A.5** (Positive converging sums from [42]). Let $a_n, b_n$ be two real sequences such that for all $n, a_n \ge 0, b_n \ge 0, \sum_{n=1}^\infty a_n = \infty, \sum_{n=1}^\infty b_n < \infty, \exists K > 0 \text{ s.t. } |b_{n+1} - b_n| < Ka_n$. Then, $\lim_{n \to \infty} b_n = 0$.

# APPENDIX B

## PROPOSITION

**Proposition B.1.** Assume $\mathbf{d} \in \mathbb{D}$ is uniformly bounded, and $(\mathbf{r}^*, \mathbf{s}^*)$ is the minimizers of the reconstruction cost function $\hat{l}(\mathbf{d}, \mathbf{L}, \mathbf{r}, \mathbf{s})$ obtained by Algorithm 2. Then,

1) $\mathbf{r}^*$ and $\mathbf{s}^*$ is uniformly bounded;

2) $\frac{1}{t}\mathbf{A}_t$ and $\frac{1}{t}\mathbf{B}_t$ is uniformly bounded;

3) $\mathbf{L}_t$ is supported by a compact subset $\mathcal{L}$,

*Proof.* Given $(\mathbf{0}, \mathbf{d})$ is a non-trivial feasible solution to Eq. (7), for the optimal solution $(\mathbf{r}^*, \mathbf{s}^*)$,

$$ \begin{aligned} & \frac{1}{2} \|\mathbf{d} - \mathbf{L}\mathbf{r}^* - \mathbf{s}^*\|_2^2 + \frac{\lambda_1}{2} \|\mathbf{r}^*\|_2^2 + \lambda_2 \|\mathbf{s}^*\|_{\ell_1/\ell_\infty} \\ & \le \hat{l}(\mathbf{d}, \mathbf{L}, \mathbf{0}, \mathbf{d}) \le \lambda_2 \|\mathbf{d}\|_{\ell_1/\ell_\infty}, \end{aligned} \quad (26) $$

thus, we obtain that

$$ \begin{aligned} \|\mathbf{r}^*\|_2^2 &\le \frac{2\lambda_2}{\lambda_1} \|\mathbf{d}\|_{\ell_1/\ell_\infty}. \\ \|\mathbf{s}^*\|_{\ell_1/\ell_\infty} &\le \|\mathbf{d}\|_{\ell_1/\ell_\infty}. \end{aligned} \quad (27) $$

Based on the assumption that $\mathbf{d}$ is uniformly bounded, then $\mathbf{r}^*, \mathbf{s}^*$ is uniformly bounded.

Similarly, we show that the accumulation matrices $\mathbf{A}_t$ and $\mathbf{B}_t$ are also uniformly bounded, as

$$ \begin{aligned} \frac{1}{t} \mathbf{A}_t &= \frac{1}{t} \sum_{i=1}^{t} \mathbf{r}_i \mathbf{r}_i^T, \\ \frac{1}{t} \mathbf{B}_t &= \frac{1}{t} \sum_{i=1}^{t} (\mathbf{d}_i - \mathbf{s}_i) \mathbf{r}_i^T. \end{aligned} \quad (28) $$

The closed-from solution $\mathbf{L}_t$ is given as

$$ \begin{aligned} \mathbf{L}_t &= \mathbf{B}_t(\mathbf{A}_t + \lambda_1 \mathbf{I})^{-1} \\ &= \frac{1}{t}\mathbf{B}_t(\frac{1}{t}\mathbf{A}_t + \frac{\lambda_1}{t}\mathbf{I})^{-1}. \end{aligned} \quad (29) $$

in which $\frac{1}{t}\mathbf{A}_t$ and $\frac{1}{t}\mathbf{B}_t$ is uniformly bounded, therefore, $\mathbf{L}_t$ is uniformly bounded.

**Proposition B.2.** Let $\mathbf{r}, \mathbf{s}, \mathbf{L}_t$ be the solution obtained by Algorithm 1,

1) $\hat{l}(\mathbf{d}, \mathbf{L}, \mathbf{r}, \mathbf{s})$ and $l(\mathbf{d}, \mathbf{L})$ are uniformly bounded;

2) The surrogate function $g_t(\mathbf{L})$ is uniformly bounded and Lipschitz.

*Proof.* The first claim is proved by combining the definition of $\hat{l}(\mathbf{d}, \mathbf{L}, \mathbf{r}, \mathbf{s})$ and the uniform boundedness of $\mathbf{d}, \mathbf{L}, \mathbf{r}$ and $\mathbf{s}$. Similarly, we can show $g_t(\mathbf{L})$ is uniformly bounded.

To proof $g_t(\mathbf{L})$ is Lipschitz, we show that the gradient of $g_t(\mathbf{L})$ is uniformly bounded as

$$ \begin{aligned} \|\nabla g_t(\mathbf{L})\|_F &= \left\| \mathbf{L} \left( \frac{1}{t}\mathbf{A}_t - \frac{1}{t}\mathbf{I} \right) - \frac{1}{t}\mathbf{B}_t \right\|_F \\ &\le \|\mathbf{L}\|_F \left( \left\| \frac{1}{t}\mathbf{A}_t \right\|_F + \left\| \frac{1}{t}\mathbf{I} \right\|_F \right) + \left\| \frac{1}{t}\mathbf{B}_t \right\|_F \end{aligned} \quad (30) $$

where the terms on the right side of the inequality are uniformly bounded, $\|\nabla g_t(\mathbf{L})\|_F$ is uniformly bounded. According to Lemma. A.2, $g_t(\mathbf{L})$ is convex with respect to $\mathbf{L}$, the boundedness of the gradient implies that $g_t(\mathbf{L})$ is Lipschitz.

**Proposition B.3.** $\mathcal{X}_0(\mathbf{L})$ refers to the set of all minimizers to $\hat{l}(\mathbf{d}, \mathbf{L}, \mathbf{r}, \mathbf{s})$ as

$$ \mathcal{X}_0(\mathbf{L}) = \{ (\bar{\mathbf{r}}, \bar{\mathbf{s}}) | (\bar{\mathbf{r}}, \bar{\mathbf{s}}) = \arg \min_{\mathbf{r}, \mathbf{s}} \hat{l}(\mathbf{d}, \mathbf{L}, \mathbf{r}, \mathbf{s}) \}. \quad (31) $$

1) The sub-gradient of function $l(\mathbf{d}, \mathbf{L})$ with respect to $\mathbf{L}$ is given as

$$ \partial_L l(\mathbf{d}, \mathbf{L}) = \operatorname{conv}\{(\mathbf{L}\mathbf{r}^* + \mathbf{s}^* - \mathbf{d})\mathbf{r}^{*T} \\ \qquad \qquad \qquad \qquad |(\mathbf{r}^*, \mathbf{s}^*) \in \mathcal{X}_0(\mathbf{L})\}, \quad (32) $$

where $\operatorname{conv}\{\cdot\}$ is convex hull operator.

2) The subgradient $\partial_L l(\mathbf{d}, \mathbf{L})$ is uniformly bounded, and $l(\mathbf{d}, \mathbf{L})$ is uniformly Lipschitz.

*Proof.* To the best of our knowledge, there is no available necessary and sufficient condition for uniqueness of minimizer to the reconstruction cost function, which implies that more than one minimizers of $\hat{l}(\mathbf{d}, \mathbf{L}, \mathbf{r}, \mathbf{s})$ probably exist.

$\hat{l}(\mathbf{z}, \mathbf{L}, \mathbf{r}, \mathbf{s})$ is convex and differentiable with respect to $\mathbf{L}$ for every feasible $(\mathbf{r}^*, \mathbf{s}^*)$. Given $\frac{\partial \hat{l}(\mathbf{z}, \mathbf{L}, \mathbf{r}, \mathbf{s})}{\partial \mathbf{L}} = (\mathbf{L}\mathbf{r} + \mathbf{s} - \mathbf{d})\mathbf{r}^T$ is differentiable with respect to $(\mathbf{r}, \mathbf{s})$ for all $\mathbf{L}$, according to Lemma. A.1, the sub-gradient of $l(\mathbf{d}, \mathbf{L})$ is given as

$$ \partial_L l(\mathbf{d}, \mathbf{L}) \in \operatorname{conv}\{(\mathbf{L}\mathbf{r}^* + \mathbf{s}^* - \mathbf{d})\mathbf{r}^{*T} \\ \qquad \qquad \qquad \qquad |(\mathbf{r}^*, \mathbf{s}^*) \in \mathcal{Z}_0(\mathbf{L})\}. \quad (33) $$

Then we proof that $(\mathbf{L}\mathbf{r} + \mathbf{s} - \mathbf{d})\mathbf{r}^T$ is uniformly bounded

$$ \|(\mathbf{L}\mathbf{r} + \mathbf{s} - \mathbf{d})\mathbf{r}^T\|_2 \le \|\mathbf{r}\|_2 (\|\mathbf{L}\|_F \|\mathbf{r}\|_2 + \|\mathbf{s}\|_2 + \|\mathbf{d}\|_2), \quad (34) $$
in which every item on the right side of the inequality is uniformly bounded. Therefore $(\mathbf{L}\mathbf{r} + \mathbf{s} - \mathbf{d})\mathbf{r}^T$ is uniformly bounded. The convex hull of the bounded set is also bounded, thus the sub-gradient $\partial_L \ell(\mathbf{d}, \mathbf{L})$ is bounded.

By Lemma. A.2, $\ell(\mathbf{d}, \mathbf{L})$ is convex with respect to $\mathbf{L}$, and its sub-gradient is uniformly bounded, thus $\ell(\mathbf{d}, \mathbf{L})$ is uniformly Lipschitz.

**Proposition B.4.** The empirical cost function $f_t(\mathbf{L})$ is uniformly bounded and Lipschitz.

*Proof.* By checking the definition of the empirical cost function $f_n(\mathbf{L})$, $f_n(\mathbf{L})$ is also uniformly bounded.

For $\mathbf{g}_t \in \partial_L \ell(\mathbf{d}_i, \mathbf{L})$, the sub-gradient $\|\partial_L\|_F$ is uniformly bounded,

$$ \|\partial_{\mathbf{L}}\|_F = \left\| \frac{1}{n} \sum_{i=1}^{n} \mathbf{g}_i + \frac{\lambda_1}{2n} \mathbf{L} \right\|_F \le \frac{1}{n} \sum_{i=1}^{n} \|\mathbf{g}_i\|_F + \frac{\lambda_1}{2n} \|\mathbf{L}\|_F. \quad (35) $$

Because $f_n(\mathbf{L})$ is convex with respect to $\mathbf{L}$, and its sub-gradient $\|\partial_{\mathbf{L}}\|_F$ is uniformly bounded, $f_n(\mathbf{L})$ is Lipschitz.

## APPENDIX C
PROOF DETAILS

**Theorem C.1.** Let $\{\mathbf{L}_t\}_{t=1}^\infty$ be the sequence of solution obtained by Algorithm 1, the surrogate function $g_t(\mathbf{L}_t)$ converges almost surely.

*Proof.* In stochastic optimization, the expected cost function is defined all the samples,

$$ f(\mathbf{L}) = \mathbb{E}_{\mathbf{d}}[\ell(\mathbf{d}, \mathbf{L})] = \lim_{n \to \infty} f_n(\mathbf{L}). \quad (36) $$

$g_t(\mathbf{L}_t)$ is analyzed as a stochastic positive process, since each term in it is non-negative and samples are drawn randomly (independent).

Note $u_t = g_t(\mathbf{L}_t)$, the difference between two consecutive time instances is given as

$$ \begin{aligned} u_{t+1} - u_t &= g_{t+1}(\mathbf{L}_{t+1}) - g_t(\mathbf{L}_t) \\ &= g_{t+1}(\mathbf{L}_{t+1}) - g_{t+1}(\mathbf{L}_t) + g_{t+1}(\mathbf{L}_t) - g_t(\mathbf{L}_t) \\ &= g_{t+1}(\mathbf{L}_{t+1}) - g_{t+1}(\mathbf{L}_t) \\ &\quad + \frac{\ell(\mathbf{d}_{t+1}, \mathbf{L}_t) - f_t(\mathbf{L}_t)}{t+1} + \frac{f_t(\mathbf{L}_t) - g_t(\mathbf{L}_t)}{t+1}. \end{aligned} \quad (37) $$

The first two terms satisfy $g_{t+1}(\mathbf{L}_{t+1}) \le g_{t+1}(\mathbf{L}_t)$, and $f_t(\mathbf{L}_t) - g_t(\mathbf{L}_t) \le 0$, therefore,

$$ \begin{aligned} u_{t+1} - u_t &\le \frac{\ell(\mathbf{d}_{t+1}, \mathbf{L}_t) - f_t(\mathbf{L}_t)}{t+1} \\ &\le \frac{\ell(\mathbf{d}_{t+1}, \mathbf{L}_t) - \frac{1}{t} \sum_{i=1}^t \ell(\mathbf{d}_i, \mathbf{L}_t)}{t+1} \end{aligned} \quad (38) $$

The expectation conditioned on past information $\mathcal{F}_t$ is given as

$$ \begin{aligned} \mathbb{E}[u_{t+1} - u_t | \mathcal{F}_t] &\le \frac{\mathbb{E}[\ell(\mathbf{d}_{t+1}, \mathbf{L}_t) | \mathcal{F}_t] - \frac{1}{t} \sum_{i=1}^t \ell(\mathbf{d}_i, \mathbf{L}_t)}{t+1} \\ &\le \frac{f(\mathbf{L}_t) - \frac{1}{t} \sum_{i=1}^t \ell(\mathbf{d}_i, \mathbf{L}_t)}{t+1} \\ &\le \frac{\|f - f_t\|_{\infty}}{t+1}, \end{aligned} \quad (39) $$

where we define $f_t = \frac{1}{t} \sum_{i=1}^t \ell(\mathbf{d}_i, \mathbf{L}_t)$, $f = \mathbb{E}_{\mathbf{d}}[\ell(\mathbf{d}, \mathbf{L}_t)]$, and $\|f - f_t\|_{\infty} = \sup_{f \in \mathcal{F}} |f - f_t|$. $\mathcal{F} = \{\ell(\mathbf{d}, \mathbf{L}) : \mathcal{D} \to \mathbb{R}, \mathbf{L} \in \mathcal{L}\}$ defines a set of measurement function indexed by $\mathbf{L}_t$ from a compact subset $\mathcal{L}$, and P-Donsker. In addition, the boundedness of $\ell(\mathbf{d}, \mathbf{L})$ implies that $\mathbb{E}[\ell(\mathbf{d}, \mathbf{L})^2]$ is uniformly bounded. Then, the requirements of Lemma. A.4 are all satisfied such that

$$ \mathbb{E}[\sqrt{t} \|f - f_t\|_{\infty}] \le \kappa, \quad \kappa > 0. \quad (40) $$

Therefore,

$$ \begin{aligned} \mathbb{E}[\mathbb{E}[u_{t+1} - u_t | \mathcal{F}_t]]^+] &= \mathbb{E}[\max\{0, \mathbb{E}[u_{t+1} - u_t | \mathcal{F}_t]\}] \\ &\le \frac{\kappa}{t^{3/2}}, \end{aligned} \quad (41) $$

where $[\cdot]^+$ is the positive variation operator.

Then we deploy Lemma. A.3 to present the convergence of $g_t(\mathbf{x})$. We define that

$$ \delta_t = \begin{cases} 1, & \text{if } \mathbb{E}[u_{t+1} - u_t | \mathcal{F}_t] > 0 \\ 0, & \text{otherwise} \end{cases}. \quad (42) $$

We have

$$ \begin{aligned} \sum_{t=1}^{\infty} \mathbb{E}[\delta_t(u_{t+1} - u_t) | \mathcal{F}_t] &= \sum_{t=1}^{\infty} \mathbb{E}[[\mathbb{E}[u_{t+1} - u_t | \mathcal{F}_t]]^+] \\ &\le \sum_{t=1}^{\infty} \frac{\kappa}{t^{3/2}} < +\infty. \end{aligned} \quad (43) $$

Conclusively, $g_t(\mathbf{L}_t)$ is quasi-martingale and converges almost surely. Moreover,

$$ \sum_{t=1}^{\infty} |\mathbb{E}[u_{t+1} - u_t | \mathcal{F}_t]| < +\infty \text{ a.s.} \quad (44) $$

**Theorem C.2.** For two solutions produced by Algorithm 1 at two consecutive time instances,

$$ \|\mathbf{L}_t - \mathbf{L}_{t+1}\|_F = O\left(\frac{1}{t}\right). \quad (45) $$

*Proof.* The Hessian matrix of $g_t(\mathbf{L})$ is $\mathbf{H} = \mathbf{I} \otimes (\mathbf{A}_t + \lambda_1 \mathbf{I})$, where $\otimes$ is the Kronecker production operator. By the definition of $\mathbf{A}_t$, $\mathbf{A}_t$ is always semi-positive, therefore, the smallest eigenvalue of $\mathbf{H}$ is greater than $\lambda_1$, which implies that $g_t(\mathbf{L})$ is strictly convex (perhaps strongly convex) with respect to $\mathbf{L}$ and

$$ g_t(\mathbf{L}_{t+1}) - g_t(\mathbf{L}_t) \ge \lambda_1 \|\mathbf{L}_{t+1} - \mathbf{L}_t\|_F^2 \quad (46) $$
Because $\mathbf{L}_{t+1}$ minimizes $g_{t+1}(\mathbf{L})$, $g_{t+1}(\mathbf{L}_{t+1}) - g_{t+1}(\mathbf{L}_t) < 0$, which presents that

$$ 
\begin{aligned}
& g_t(\mathbf{L}_{t+1}) - g_t(\mathbf{L}_t) \\
&= g_t(\mathbf{L}_{t+1}) - g_{t+1}(\mathbf{L}_{t+1}) + g_{t+1}(\mathbf{L}_{t+1}) - g_t(\mathbf{L}_t) \\
&\le g_t(\mathbf{L}_{t+1}) - g_{t+1}(\mathbf{L}_{t+1}) + g_{t+1}(\mathbf{L}_t) - g_t(\mathbf{L}_t) \\
&\le (g_t(\mathbf{L}_{t+1}) - g_{t+1}(\mathbf{L}_{t+1})) - (g_t(\mathbf{L}_t) - g_{t+1}(\mathbf{L}_t))
\end{aligned} 
\quad (47) $$

We define $G_t(\mathbf{L}) = g_t(\mathbf{L}) - g_{t+1}(\mathbf{L})$, then gradient of $G_t(\mathbf{L})$ is extracted as

$$ 
\begin{aligned}
\nabla G_t(\mathbf{L}) &= \nabla g_t(\mathbf{L}) - \nabla g_{t+1}(\mathbf{L}) \\
&= \frac{1}{t} \mathbf{L} \hat{\mathbf{A}}_t + \frac{1}{t} \mathbf{B}_t - \frac{1}{t+1} \mathbf{L} \hat{\mathbf{A}}_{t+1} - \frac{1}{t+1} \mathbf{B}_{t+1} \\
&= \frac{1}{t} \mathbf{L} (\hat{\mathbf{A}}_t - \frac{t}{t+1} \hat{\mathbf{A}}_{t+1}) + \frac{1}{t} (\hat{\mathbf{B}}_t - \frac{t}{t+1} \hat{\mathbf{A}}_{t+1}),
\end{aligned} 
\quad (48) $$

in which $\hat{\mathbf{A}}_t = \mathbf{A}_t + \lambda_1 \mathbf{I}$.

Given that $\mathbf{A}_t, \mathbf{B}_t$ and $\mathbf{L}$ is uniformly bounded, the gradient $\nabla G_t(\mathbf{L})$ is also uniformly bounded,

$$ 
\begin{aligned}
& \|\nabla G_t(\mathbf{L})\|_F \\
&\le \frac{1}{t} (\|\mathbf{L}\|_F \|\hat{\mathbf{A}}_t - \frac{t}{t+1} \hat{\mathbf{A}}_{t+1}\|_F + \|\hat{\mathbf{B}}_t - \frac{t}{t+1} \hat{\mathbf{A}}_{t+1}\|_F).
\end{aligned} 
\quad (49) $$

Thus, we conclude that $G_t(\mathbf{L})$ is uniformly Lipschitz with respect to $\mathbf{L}$, and there exist a positive constant $\kappa_t = \frac{1}{t} (\|\mathbf{L}\|_F \|\hat{\mathbf{A}}_t - \frac{t}{t+1} \hat{\mathbf{A}}_{t+1}\|_F + \|\hat{\mathbf{B}}_t - \frac{t}{t+1} \hat{\mathbf{A}}_{t+1}\|_F)$ that satisfies

$$ G_t(\mathbf{L}_t) - G_t(\mathbf{L}_{t+1}) \le \kappa_t \|\mathbf{L}_{t+1} - \mathbf{L}_t\|_F. \quad (50) $$

Combining Eqs. (46), (47) and (50), we conclude that

$$ \|\mathbf{L}_{t+1} - \mathbf{L}_t\|_F \le \frac{\kappa_t}{\lambda_1}. \quad (51) $$

$$ \text{and } \|\mathbf{L}_{t+1} - \mathbf{L}_t\|_F = O(\frac{1}{t}). \quad \blacksquare $$

**Theorem C.3.** Note $f_t(\mathbf{L})$ is the empirical cost function, and $g_t(\mathbf{L})$ is its surrogate function. $\mathbf{L}_t$ is the solution obtained by Algorithm 1, when $t$ tends to infinity, $g_t(\mathbf{L}_t) - f_t(\mathbf{L}_t)$ converges to 0 almost surely.

*Proof.* This proof is originally presented by [32], for the completeness of this proof, we introduce it here. Combining Eq. (37) with $g_{t+1}(\mathbf{L}_{t+1}) - g_{t+1}(\mathbf{L}_t) \le 0$, we obtain

$$ 
\begin{aligned}
& \frac{g_t(\mathbf{L}_t) - f_t(\mathbf{L}_t)}{t+1} \\
&\le \frac{\ell(\mathbf{d}_{t+1}, \mathbf{L}_t) - f_t(\mathbf{L}_t)}{t+1} - (g_{t+1}(\mathbf{L}_{t+1}) - g_t(\mathbf{L}_t)) \quad (52) \\
&\le \frac{\ell(\mathbf{d}_{t+1}, \mathbf{L}_t) - f_t(\mathbf{L}_t)}{t+1} + [g_{t+1}(\mathbf{L}_{t+1}) - g_t(\mathbf{L}_t)]^-, 
\end{aligned} $$

in which $[\cdot]^-$ refers to the negative variation operator.

Similar to the proof of Theorem C.1, we take the expectation conditioned on past information $\mathcal{F}_t$

$$ 
\begin{aligned}
\mathbb{E}[\frac{g_t(\mathbf{L}_t) - f_t(\mathbf{L}_t)}{t+1} | \mathcal{F}_t] &= \frac{g_t(\mathbf{L}_t) - f_t(\mathbf{L}_t)}{t+1} \\
&\le \mathbb{E}[\frac{\ell(\mathbf{d}_{t+1}, \mathbf{L}_t) - f_t(\mathbf{L}_t)}{t+1} | \mathcal{F}_t] \\
&\quad + \mathbb{E}[[g_{t+1}(\mathbf{L}_{t+1}) - g_t(\mathbf{L}_t)]^- | \mathcal{F}_t].
\end{aligned} 
\quad (53) $$

Accumulating $\frac{g_t(\mathbf{L}_t) - f_t(\mathbf{L}_t)}{t+1}$ with $t$ tending to $\infty$, we have

$$ 
\begin{aligned}
& \sum_{t=1}^{\infty} \frac{g_t(\mathbf{L}_t) - f_t(\mathbf{L}_t)}{t+1} \\
&\le \sum_{t=1}^{\infty} \mathbb{E}[\frac{\ell(\mathbf{d}_{t+1}, \mathbf{L}_t) - \frac{1}{t} \sum_{i=1}^{t} \ell(\mathbf{d}_t, \mathbf{L}_i)}{t+1} | \mathcal{F}_t] \\
&\quad + \sum_{t=1}^{\infty} \mathbb{E}[[g_{t+1}(\mathbf{L}_{t+1}) - g_t(\mathbf{L}_t)]^- | \mathcal{F}_t] \\
&\le \sum_{t=1}^{\infty} \frac{|f - f_t|}{t+1} + \sum_{t=1}^{\infty} \mathbb{E}[[g_{t+1}(\mathbf{L}_{t+1}) - g_t(\mathbf{L}_t)]^- | \mathcal{F}_t].
\end{aligned} 
\quad (54) $$

According to central limit theorem, $\sqrt{t}|f - f_t|$ converges almost surely as $t$ tends to infinity. From Theorem C.1, we also obtain that

$$ \sum_{t=1}^{\infty} |\mathbb{E}[[g_{t+1}(\mathbf{L}_{t+1}) - g_t(\mathbf{L}_t)]^- | \mathcal{F}_t]| < +\infty. \quad (55) $$

Hence, we have the almost sure convergence of the positive sum

$$ 
\begin{aligned}
& \sum_{t=1}^{\infty} \frac{g_t(\mathbf{L}_t) - f_t(\mathbf{L}_t)}{t+1} \\
&\le \sum_{t=1}^{\infty} \frac{|f - f_t|}{t+1} + \sum_{t=1}^{\infty} \mathbb{E}[[g_{t+1}(\mathbf{L}_{t+1}) - g_t(\mathbf{L}_t)]^- | \mathcal{F}_t] \\
&< \infty.
\end{aligned} 
\quad (56) $$

As demonstrated in Propositions. B.2 and B.4, $g_t(\mathbf{L})$ and $f_t(\mathbf{L})$ are both Lipschitz with respect to $\mathbf{L}$, which implies there exists $\kappa_2$ such that

$$ 
\begin{aligned}
& |g_{t+1}(\mathbf{L}_{t+1}) - f_{t+1}(\mathbf{L}_{t+1}) - (g_t(\mathbf{L}_t) - f_t(\mathbf{L}_t))| \\
&= |g_{t+1}(\mathbf{L}_{t+1}) - g_t(\mathbf{L}_t) - (f_{t+1}(\mathbf{L}_{t+1}) - f_t(\mathbf{L}_t))| \\
&\le |g_{t+1}(\mathbf{L}_{t+1}) - g_t(\mathbf{L}_t)| + |f_{t+1}(\mathbf{L}_{t+1}) - f_t(\mathbf{L}_t)| \\
&\le |g_{t+1}(\mathbf{L}_{t+1}) - g_{t+1}(\mathbf{L}_t)| + |g_{t+1}(\mathbf{L}_t) - g_t(\mathbf{L}_t)| \\
&\quad + |f_{t+1}(\mathbf{L}_{t+1}) - f_{t+1}(\mathbf{L}_t)| + |f_{t+1}(\mathbf{L}_t) - f_t(\mathbf{L}_t)| \\
&\le \kappa_2 \|\mathbf{L}_{t+1} - \mathbf{L}_t\|_F.
\end{aligned} 
\quad (57) $$

According to Lemma A.5, combining with $\sum_{t=1}^{\infty} \frac{1}{t+1} = \infty$, we have that

$$ \lim_{t \to \infty} g_t(\mathbf{L}_t) - f_t(\mathbf{L}_t) = 0, \text{ a.s..} \quad (58) $$

**Theorem C.4.** As $t$ tends to infinity, given the $\mathbf{L}_t$ is obtained by Algorithm 1, $g_t(\mathbf{L}_t) - f(\mathbf{L}_t)$ converges to 0 almost surely.

*Proof.* From Theorem C.2, we have $\|f - f_t\|_{\infty}$ converges to 0 almost surely, which implies

$$ \lim_{t \to \infty} g_t(\mathbf{L}_t) - f(\mathbf{L}_t) = 0, \text{ a.s..} \quad (59) $$

■
REFERENCES

[1] G. Cheng and J. Han, "A survey on object detection in optical remote sensing images," *ISPRS Journal of Photogrammetry and Remote Sensing*, vol. 117, pp. 11–28, 2016.

[2] G.-S. Xia, X. Bai, J. Ding, Z. Zhu, S. Belongie, J. Luo, M. Datcu, M. Pelillo, and L. Zhang, "Dota: A large-scale dataset for object detection in aerial images," in *IEEE CVPR*, 2018.

[3] R. Girshick, J. Donahue, T. Darrell, and J. Malik, "Rich feature hierarchies for accurate object detection and semantic segmentation," in *Proceedings of the IEEE conference on computer vision and pattern recognition*, 2014, pp. 580–587.

[4] R. Girshick, "Fast r-cnn," in *Proceedings of the IEEE international conference on computer vision*, 2015, pp. 1440–1448.

[5] S. Ren, K. He, R. Girshick, and J. Sun, "Faster r-cnn: Towards real-time object detection with region proposal networks," in *Advances in neural information processing systems*, 2015, pp. 91–99.

[6] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, "You only look once: Unified, real-time object detection," in *Proceedings of the IEEE conference on computer vision and pattern recognition*, 2016, pp. 779–788.

[7] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and A. C. Berg, "Ssd: Single shot multibox detector," in *European conference on computer vision*. Springer, 2016, pp. 21–37.

[8] Y. Long, Y. Gong, Z. Xiao, and Q. Liu, "Accurate object localization in remote sensing images based on convolutional neural networks," *IEEE Transactions on Geoscience and Remote Sensing*, vol. 55, no. 5, pp. 2486–2498, 2017.

[9] K. Li, G. Cheng, S. Bu, and X. You, "Rotation-insensitive and context-augmented object detection in remote sensing images," *IEEE Transactions on Geoscience and Remote Sensing*, vol. 56, no. 4, pp. 2337–2348, 2017.

[10] P. Ding, Y. Zhang, W.-J. Deng, P. Jia, and A. Kuijper, "A light and faster regional convolutional neural network for object detection in optical remote sensing images," *ISPRS journal of photogrammetry and remote sensing*, vol. 141, pp. 208–218, 2018.

[11] W. Liu, L. Ma, and H. Chen, "Arbitrary-oriented ship detection framework in optical remote-sensing images," *IEEE Geoscience and Remote Sensing Letters*, vol. 15, no. 6, pp. 937–941, 2018.

[12] Y. Luo, L. Zhou, S. Wang, and Z. Wang, "Video satellite imagery super resolution via convolutional neural networks," *IEEE Geoscience and Remote Sensing Letters*, vol. 14, no. 12, pp. 2398–2402, 2017.

[13] P. Team, "Planet application program interface: In space for life on earth. san francisco, ca," 2016.

[14] L. Mou and X. X. Zhu, "Spatiotemporal scene interpretation of space videos via deep neural network and tracklet analysis," in *Geoscience and Remote Sensing Symposium (IGARSS), 2016 IEEE International*. IEEE, 2016, pp. 1823–1826.

[15] B. Du, Y. Sun, S. Cai, C. Wu, and Q. Du, "Object tracking in satellite videos by fusing the kernel correlation filter and the three-frame-difference algorithm," *IEEE Geoscience and Remote Sensing Letters*, vol. 15, no. 2, pp. 168–172, 2018.

[16] J. Zhang, X. Jia, J. Hu, and K. Tan, "Satellite multi-vehicle tracking under inconsistent detection conditions by bilevel k-shortest paths optimization," in *2018 Digital Image Computing: Techniques and Applications (DICTA)*. IEEE, 2018, pp. 1–8.

[17] B. Uzkent, A. Rangnekar, and M. J. Hoffman, "Tracking in aerial hyperspectral videos using deep kernelized correlation filters," *IEEE Transactions on Geoscience and Remote Sensing*, no. 99, pp. 1–13, 2018.

[18] T. Bouwmans and E. H. Zahzah, "Robust pca via principal component pursuit: A review for a comparative evaluation in video surveillance," *Computer Vision and Image Understanding*, vol. 122, pp. 22–34, 2014.

[19] T. Bouwmans, A. Sobral, S. Javed, S. K. Jung, and E.-H. Zahzah, "Decomposition into low-rank plus additive matrices for background/foreground separation: A review for a comparative evaluation with a large-scale dataset," *Computer Science Review*, vol. 23, pp. 1–71, 2017.

[20] T. Bouwmans, S. Javed, H. Zhang, Z. Lin, and R. Otazo, "On the applications of robust pca in image and video processing," *Proceedings of the IEEE*, vol. 106, no. 8, pp. 1427–1457, 2018.

[21] E. J. Candès, X. Li, Y. Ma, and J. Wright, "Robust principal component analysis?" *Journal of the ACM (JACM)*, vol. 58, no. 3, p. 11, 2011.

[22] Z. Lin, R. Liu, and Z. Su, "Linearized alternating direction method with adaptive penalty for low-rank representation," in *Advances in neural information processing systems*, 2011, pp. 612–620.

[23] J. Wright, A. Ganesh, S. Rao, Y. Peng, and Y. Ma, "Robust principal component analysis: Exact recovery of corrupted low-rank matrices via convex optimization," in *Advances in neural information processing systems*, 2009, pp. 2080–2088.

[24] T. Zhou and D. Tao, "Godec: Randomized low-rank & sparse matrix decomposition in noisy case," in *International conference on machine learning*. Omnipress, 2011.

[25] N. Wang, T. Yao, J. Wang, and D.-Y. Yeung, "A probabilistic approach to robust matrix factorization," in *European Conference on Computer Vision*. Springer, 2012, pp. 126–139.

[26] Y. Xu, Z. Wu, J. Chanussot, M. Dalla Mura, A. L. Bertozzi, and Z. Wei, "Low-rank decomposition and total variation regularization of hyperspectral video sequences," *IEEE Transactions on Geoscience and Remote Sensing*, vol. 56, no. 3, pp. 1680–1694, 2017.

[27] X. Zhou, C. Yang, and W. Yu, "Moving object detection by detecting contiguous outliers in the low-rank representation," *IEEE Transactions on Pattern Analysis and Machine Intelligence*, vol. 35, no. 3, pp. 597–610, 2013.

[28] R. Jenatton, J.-Y. Audibert, and F. Bach, "Structured variable selection with sparsity-inducing norms," *Journal of Machine Learning Research*, vol. 12, no. Oct, pp. 2777–2824, 2011.

[29] X. Liu, G. Zhao, J. Yao, and C. Qi, "Background subtraction based on low-rank and structured sparse decomposition," *IEEE Transactions on Image Processing*, vol. 24, no. 8, pp. 2502–2514, 2015.

[30] J. Zhang, X. Jia, and J. Hu, "Error bounded foreground and background modeling for moving object detection in satellite videos," *arXiv preprint arXiv:1908.09539*, 2019.

[31] J. He, L. Balzano, and A. Szlam, "Incremental gradient on the grassmannian for online foreground and background separation in subsampled video," in *2012 IEEE Conference on Computer Vision and Pattern Recognition*. IEEE, 2012, pp. 1568–1575.

[32] J. Feng, H. Xu, and S. Yan, "Online robust pca via stochastic optimization," in *Advances in Neural Information Processing Systems*, 2013, pp. 404–412.

[33] M. Shakeri and H. Zhang, "Corola: a sequential solution to moving object detection using low-rank approximation," *Computer Vision and Image Understanding*, vol. 146, pp. 27–39, 2016.

[34] J. Xu, V. K. Ithapu, L. Mukherjee, J. M. Rehg, and V. Singh, "Gosus: Grassmannian online subspace updates with structured-sparsity," in *Proceedings of the IEEE International Conference on Computer Vision*, 2013, pp. 3376–3383.

[35] S. Javed, A. Mahmood, S. Al-Maadeed, T. Bouwmans, and S. K. Jung, "Moving object detection in complex scene using spatiotemporal structured-sparse rPCA," *IEEE Transactions on Image Processing*, vol. 28, no. 2, pp. 1007–1022, 2019.

[36] P. Turaga, A. Veeraraghavan, and R. Chellappa, "Statistical analysis on stiefel and grassmann manifolds with applications in computer vision," in *2008 IEEE Conference on Computer Vision and Pattern Recognition*. IEEE, 2008, pp. 1–8.

[37] L. Balzano, R. Nowak, and B. Recht, "Online identification and tracking of subspaces from highly incomplete information," in *2010 48th Annual Allerton conference on communication, control, and computing (Allerton)*. IEEE, 2010, pp. 704–711.

[38] M. Harandi, C. Sanderson, C. Shen, and B. C. Lovell, "Dictionary learning and sparse coding on grassmann manifolds: An extrinsic solution," in *Proceedings of the IEEE international conference on computer vision*, 2013, pp. 3120–3127.

[39] B. Recht, M. Fazel, and P. A. Parrilo, "Guaranteed minimum-rank solutions of linear matrix equations via nuclear norm minimization," *SIAM review*, vol. 52, no. 3, pp. 471–501, 2010.

[40] P. Sprechmann, A. M. Bronstein, and G. Sapiro, "Learning efficient sparse and low rank models," *IEEE transactions on pattern analysis and machine intelligence*, vol. 37, no. 9, pp. 1821–1833, 2015.

[41] J. Shen, P. Li, and H. Xu, "Online low-rank subspace clustering by basis dictionary pursuit," in *International Conference on Machine Learning*, 2016, pp. 622–631.

[42] J. Mairal, F. Bach, J. Ponce, and G. Sapiro, "Online learning for matrix factorization and sparse coding," *Journal of Machine Learning Research*, vol. 11, no. Jan, pp. 19–60, 2010.

[43] E. Dohmatob, A. Mensch, G. Varoquaux, and B. Thirion, "Learning brain regions via large-scale online structured sparse dictionary learning," in *Advances in Neural Information Processing Systems*, 2016, pp. 4610–4618.

[44] R. Jenatton, J. Mairal, G. Obozinski, and F. R. Bach, "Proximal methods for sparse hierarchical dictionary learning." in *ICML*, vol. 1. Citeseer, 2010, p. 2.
[45] K. Jia, T.-H. Chan, and Y. Ma, "Robust and practical face recognition via structured sparsity," in *European conference on computer vision*. Springer, 2012, pp. 331–344.

[46] S. J. Wright, "Coordinate descent algorithms," *Mathematical Programming*, vol. 151, no. 1, pp. 3–34, 2015.

[47] J. Mairal, R. Jenatton, F. R. Bach, and G. R. Obozinski, "Network flow algorithms for structured sparsity," in *Advances in Neural Information Processing Systems*, 2010, pp. 1558–1566.

[48] J. Mairal, R. Jenatton, G. Obozinski, and F. Bach, "Convex and network flow optimization for structured sparsity," *Journal of Machine Learning Research*, vol. 12, no. Sep. pp. 2681–2720, 2011.

[49] Z. Gao, L.-F. Cheong, and M. Shan, "Block-sparse rpa for consistent foreground detection," in *European Conference on Computer Vision*. Springer, 2012, pp. 690–703.

[50] Z. Lin, M. Chen, and Y. Ma, "The augmented lagrange multiplier method for exact recovery of corrupted low-rank matrices," *arXiv preprint arXiv:1009.5055*, 2010.

[51] D. P. Bertsekas, "Nonlinear programming," *Journal of the Operational Research Society*, vol. 48, no. 3, pp. 334–334, 1997.

[52] S. Shalev-Shwartz et al., "Online learning and online convex optimization," *Foundations and Trends® in Machine Learning*, vol. 4, no. 2, pp. 107–194, 2012.

[53] A. Bensoussan, J.-L. Lions, and G. Papanicolaou, *Asymptotic analysis for periodic structures*. American Mathematical Soc., 2011, vol. 374.

<figure><img src="image_9.png" /></figure>

**Jiankun Hu** receive the Ph.D. degree in control engineering from the Harbin Institute of Technology, China, in 1993, and the masters degree in computer science and software engineering from Monash University, Australia, in 2000. He was a Research Fellow with Delft University, The Netherlands, from 1997 to 1998, and The University of Melbourne, Australia, from 1998 to 1999. He is a full professor of Cyber Security at the School of Engineering and Information Technology, the University of New South Wales at Canberra, Australia. His main re-

search interest is in the field of cyber security, including biometrics security, where he has published many papers in high-quality conferences and journals including the IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE. He has served on the editorial boards of up to seven international journals and served as a Security Symposium Chair of the IEEE Flagship Conferences of IEEE ICC and IEEE GLOBECOM. He has obtained nine Australian Research Council (ARC) Grants. He served at the prestigious Panel of Mathematics, Information and Computing Sciences, ARC ERA (The Excellence in Research for Australia) Evaluation Committee 2012.

<figure><img src="image_10.png" /></figure>

(DICTA 2018).

**Junpeng Zhang** received the B.Sci. degree from the China University of Mining and Technology, Xuzhou, China, in 2013 and the Master's degree in surveying engineering from the same university in 2016. He is currently pursuing the Ph.D. degree in electrical engineering from The University of New South Wales, Australia. His research interests include object detection and tracking in remote sensing imaginary. He was the winner of "DSTG Best Contribution to Science Award" in Digital Image Computing: Techniques and Applications 2018

<figure><img src="image_11.png" /></figure>

**Jocelyn Chanussot** (M04-SM04-F12) received the M.Sc. degree in electrical engineering from the Grenoble Institute of Technology (Grenoble INP), Grenoble, France, in 1995, and the Ph.D. degree from the Universit de Savoie, Annecy, France, in 1998. In 1999, he was with the Geography Imagery Perception Laboratory for the Delegation Generale de l'Armement (French National Defense Department). Since 1999, he has been with Grenoble INP, where he is currently a Professor of signal and image processing. He has been a Visiting Scholar

with Stanford University, Stanford, CA, USA; KTH, Stockholm, Sweden; and NUS, Singapore. Since 2013, he has been an Adjunct Professor with the University of Iceland, Reykjavik, Iceland. From 2015 to 2017, he was a Visiting Professor with the University of California at Los Angeles, Los Angeles, CA, USA. He is conducting his research at GIPSA-Lab. His research interests include image analysis, multicomponent image processing, nonlinear filtering, and data fusion in remote sensing.

<figure><img src="image_12.png" /></figure>

**Xiuping Jia** (M93-SM03) received the B.Eng. degree from the Beijing University of Posts and Telecommunications, Beijing, China, in 1982 and the Ph.D. degree in electrical engineering from The University of New South Wales, Australia, in 1996. Since 1988, she has been with the School of Engineering and Information Technology, The University of New South Wales at Canberra, Australia, where she is currently an Associate Professor. Her research interests include remote sensing, image processing and spatial data analysis. Dr. Jia has authored or

coauthored more than 200 referred papers, including over 100 journal papers with h-index of 34 and i10 of 102. She has co-authored of the remote sensing textbook titled Remote Sensing Digital Image Analysis [Springer-Verlag, 3rd (1999) and 4th eds. (2006)]. She is a Subject Editor for the Journal of Soils and Sediments and an Associate Editor of the IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING.

Dr. Chanussot was a member of the IEEE Geoscience and Remote Sensing Society AdCom from 2009 to 2010, in charge of membership development and Machine Learning for Signal Processing Technical Committee of the IEEE Signal Processing Society from 2006 to 2008. He is a member of the Institut Universitaire de France from 2012 to 2017. He was the General Chair of the first IEEE GRSS Workshop on Hyperspectral Image and Signal Processing, Evolution in Remote sensing. He was the Chair from 2009 to 2011 and the Co-Chair of the GRS Data Fusion Technical Committee from 2005 to 2008. He was the Program Chair of the IEEE International Workshop on Machine Learning for Signal Processing in 2009. He is the Founding President of IEEE Geoscience and Remote Sensing French Chapter from 2007 to 2010 which received the 2010 IEEE GRSS Chapter Excellence Award. He was a co-recipient of the NORSIG 2006 Best Student Paper Award, the IEEE GRSS 2011 and 2015 Symposium Best Paper Award, the IEEE GRSS 2012 Transactions Prize Paper Award, and the IEEE GRSS 2013 Highest Impact Paper Award. He was an Associate Editor for IEEE GEOSCIENCE AND REMOTE SENSING LETTERS from 2005 to 2007 and Pattern Recognition from 2006 to 2008. He was the Editor-in-Chief of the IEEE JOURNAL OF SELECTED TOPICS IN APPLIED EARTH OBSERVATIONS AND REMOTE SENSING from 2011 to 2015. Since 2007, he has been an Associate Editor for the IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING, and since 2018, he has also been an Associate Editor for the IEEE TRANSACTIONS ON IMAGE PROCESSING. He was the Guest Editor for the PROCEEDINGS OF THE IEEE in 2013 and IEEE SIGNAL PROCESSING MAGAZINE in 2014.