# Particle tracking with continuous energy minimization for the study of segregation in bedload transport

Philippe Frey, Christophe Ducottet

► To cite this version:

Philippe Frey, Christophe Ducottet. Particle tracking with continuous energy minimization for the study of segregation in bedload transport. Experiments in Fluids, 2025, 66 (8), pp.150. 10.1007/s00348-025-04072-3. ujm-05174246

HAL Id: ujm-05174246

https://ujm.hal.science/ujm-05174246v1

Submitted on 21 Jul 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Particle Tracking with Continuous Energy
Minimization for the Study of Segregation in
Bedload Transport

Philippe Frey¹* and Christophe Ducottet²

¹*Université Grenoble Alpes, INRAE, IGE, CNRS, Grenoble INP, IRD, 2 rue de la Papeterie-BP76, 38402, St-Martin-d'Hères, France.

²Université Jean Monnet Saint-Etienne, CNRS, Institut d'Optique Graduate School, Laboratoire Hubert Curien UMR 5516, F-42023, Saint-Etienne, France.

*Corresponding author(s). E-mail(s): philippe.frey@inrae.fr;

Contributing authors: ducottet@univ-st-etienne.fr;

Abstract

Bedload transport, the coarser component of sediment transport moving
in contact with the bed in stream channels, has major implications for
public safety, water resources, and environmental sustainability. Size seg-
regation is largely responsible for our limited ability to predict sediment
flux and river morphology, particularly in mountains where steep slopes
drive an intense transport of a wide range of grain sizes. Two-size exper-
iments were carried out in a dedicated 10% steep flume to study vertical
segregation at the grain scale. Particle tracking was used to obtain trajec-
tories of high concentration bedload mixtures of spherical particles but
it fails to correctly retrieve long trajectories due to strong grain-grain
interactions. In this paper we propose a new particle tracking algorithm
using a global optimization scheme based on a Continuous Energy func-
tion and a specific iterative Minimization (CEM). For the purpose of
evaluating this new algorithm named CEM-ST, we have designed two

<footnote>* available at https://gitlab.univ-st-etienne.fr/labhc-iscv/cem-st</footnote>
## 2 CEM-based Particle Tracking for the Study of Bedload Segregation

typical experimental reference sequences with corresponding full trajectory ground truths, made available to the community. Compared to online algorithms, which consider only previous time steps, this new CEM-ST algorithm is less sensitive to the quality of the detections and performs better both globally and in the details of the trajectories and the depth profiles of concentration, particle velocity and sediment transport rate. Application of CEM-ST has allowed us to gain a better insight into the influence of the fine particle rate on segregation, in particular unravelling the role of clusters in the bedload dynamics.

# 1 Introduction

Bedload transport in rivers refers to the water shear-driven transport of sediment particles close to the river mobile bed by sliding, rolling or saltating (Julien, 2012). It plays a crucial role in shaping the river channel and influencing the overall geomorphology of river systems. A major still unsolved issue in bedload transport is size segregation (Ancey, 2020) due to the large range of grain size in natural systems, especially in steep mountain streams. This complex phenomenon can be studied experimentally at the particle scale (Ferdowsi et al, 2017) using a two-size mixture of spherical glass beads transported by a turbulent steady water flow, filming from the side by a high speed camera and applying dedicated image analysis techniques (Frey et al, 2020).

To get in-depth insight into bedload segregation processes, and particularly to study vertical segregation, particle velocity and concentration depth profiles of the coarse particles have proved invaluable (Frey, 2014; Capart and Fraccarollo, 2011). This requires the individual tracking of all coarse grains of the channel over extended periods of time. Lagrangian Particle Tracking (LPT) methods have mostly been developed for the purpose of fluid flow studies. Their principle is to artificially incorporate small tracer particles that are illuminated and observed by one or more cameras. Significant progress has been made to propose relevant LPT algorithms either in 2D or 3D even in relatively high concentration (Schanz et al, 2016; Schröder and Schanz, 2023). However LPT methods are not adapted for bedload transport mainly because they consider small particles with almost no interactions between them.

This has motivated the development of tracking algorithms specifically dedicated to high concentration sediments with strong grain-grain interactions and occlusions making difficult both the detection and the tracking of individual grains. Figure 1 illustrates several issues with the tracking of 5 coarse particles over a mobile bed of fine particles. In frame 104, due to occlusion with fine particles, particle #2 is missed. In frame 106, it is detected again as a new particle with a new id #6. In frame 110, due to occlusion between particles #1 and #6, the identities of these two particles are switched.
<figure><img src="image_2.png" /><figcaption>**Fig. 1:** Image of bedload transport in our laboratory flume where coarse sed- iments are modelled with spherical glass beads of diameter 5 mm. Above: the entire image. Below: detailed example of tracking of the leftmost five beads in successive frames with missing detections and identity switch due to occlusions. See section 3 for experimental details. Flow from left to right.</figcaption></figure>

The common way to tackle this issue is to rely on the standard tracking by detection approach. A specific detector first gives the location of grains in the image sequence. Then, a data association problem is solved to build tra- jectories from the set of grain locations across time. Finding the optimal data association across the whole time period is of high complexity and sub-optimal solutions are generally proposed among which online tracking considers only past detections. A widely used strategy takes into account only the detections at time *t* − 1 and finding the particle pairing that minimizes a cost function rating each possible association between the detections of time *t* and *t* − 1 (Capart et al, 2002; Ouellette et al, 2006). This simple but yet efficient online approach has been specifically adapted for bedload analysis in (Böhm et al, 2006; Hergault et al, 2010) and was implemented in a toolbox named OL- ST by Lafaye de Micheaux et al (2018). The main drawback of the previous methods is that they strongly rely on the detector quality. False or missing detections make the association process fail, resulting in fragmented or over- crossing trajectories. To cope with these difficulties, probabilistic approaches that exploit the detector confidence as a graded observation model have been proposed (Breitenstein et al, 2011; Lafaye de Micheaux et al, 2018). How- ever, these methods still fail to produce relevant long-term trajectories in the presence of false detections and occlusions.

Another important limitation of the previous approaches is to consider a
limited number of time steps (only the previous one most of the time) to
solve the data association problem. When the number of objects to track is
high and there are a lot of interactions, the data association problem has to
be solved more globally, considering a large temporal window. Most state-
of-the-art general purpose tracking algorithms either propose a near global
optimization over large temporal windows (Berclaz et al, 2011; Henriques et al,
2011; Hofmann et al, 2013; Possegger et al, 2014; Milan et al, 2016) or con-
sider, as post-processing step, a global link model to associate short tracklets
into long trajectories (Wang et al, 2016, 2019; Du et al, 2021, 2023). The
specific approach proposed by Milan et al (2016) focused on the design of a
4 CEM-based Particle Tracking for the Study of Bedload Segregation

single global energy function taking into account physical constraints, such as object dynamics, mutual exclusion and track persistance, and proposed an efficient optimization scheme called Continuous Energy Minimization (CEM). The specificity is to define a global energy function in a continuous space which depends on the location and motion of all objects in all frames. Thanks to this formulation in closed form, the optimization can be computed efficiently by alternating gradient descent and trans-dimensional jump moves to explore the whole dimensionality of the search space. This kind of approach was primarily designed for human centered problems such as person or vehicle tracking. It obviously needs adaptation to account for the specific physical constraints of sediment tracking.

In this paper, we propose (i) a new algorithm dedicated to sediment track-
ing with the code available online<fnref n="1" />, (ii) a dataset of two real experiment-based
reference image sequences with their trajectory ground truth, (iii) a thor-
ough evaluation on the dataset and (iv) applications giving insights on the
bedload segregation dynamics. The new algorithm is built upon the con-
tinuous energy minimization principle, and is named CEM-ST, standing for
Continuous Energy Minimization Sediment Tracking.

This new approach has the following advantages:

* solving the multi-object tracking problem through a global optimization scheme,

* incorporating the physical constraints of sediment tracking,

• being robust to missing or false detections,

* can be used with any detection algorithm.

Following this introduction, section 2 describes this new CEM-ST algorithm, focussing on the energy function. Section 3 presents our bedload segregation experiments, specifically two series where distinct occlusion problems occurred, and details how we designed the two corresponding ground truth reference sequences. Section 4 is dedicated to the evaluation of CEM-ST, first globally using CLEAR MOT metrics and second with a detailed analysis of spatio-temporal trajectories. Section 5 deals with depth profiles of concentration, particle velocity and sediment rate. Improvement is shown by confronting our previous and new CEM-ST algorithms to Ground Truth depth profiles. Finally two specific applications illustrate the capabilities of this new algorithm.

2 Continuous Energy Minimization Sediment Tracking

2.1 Problem formulation

The context of our study is the analysis of sediment transport and vertical segregation in a laboratory flume where sediment transport is modeled by spherical glass beads entrained by a turbulent water flow. In addition to coarse sediment, a collection of finer beads is also transported by the flow, producing

<footnote n="1">¹https://gitlab.univ-st-etienne.fr/labhc-iscv/cem-st</footnote>
a mobile fluctuating bed. Videos are registered through the transparent sidewalls, producing images as those in Figures 1 and 4. The width of the channel is slightly greater than two times the diameter of the coarse beads such that only two beads can overlap. The purpose of the tracking is to recover the individual trajectory of each coarse bead along the entire video.

In the tracking by detection paradigm, detection and tracking are done in two successive steps. The detection step consists in detecting coarse beads in each frame $t$ of the video sequence. At this step, beads are detected independently in each frame and we denote $\mathbf{D}_g^t = (x_g^t, y_g^t)$ the coordinates of bead $g$ in frame $t$. The total number of beads detected in frame $t$ is denoted $D(t)$. As the detector is not perfect, some detections are missing (false negative) and some of them are wrong (false positive).

The purpose of the tracking step is to find the individual trajectory of each coarse bead along the whole video i.e. to determine a set of trajectories $\mathbf{X} = (\mathbf{X}^i)_{i=1,...,N}$ where $i$ is the index of a trajectory, $N$ the total number of trajectories and $\mathbf{X}^i = (x_i^i, y_i^i)_{i \in T_i}$ is the time series giving the 2D coordinates of the bead corresponding to trajectory $i$, $T_i$ being the time interval (set of frame numbers) for which trajectory $i$ exists. As the beads can enter or exit from the observation window, each trajectory has a starting time $s_i$ and an ending time $e_i$ such that we have $T_i = \{s_i, ..., e_i\}$.

Note that this formulation is general to any 2D tracking problem and can also be generalized to 3D if the optical setup gives access to the third dimension.

## 2.2 Algorithm overview

The general principle is to formulate the tracking algorithm as a global optimization problem where the goal is to determine the whole set of trajectories $\mathbf{X}$ given the detections $\mathbf{D}$. In this problem, not only the coordinates of beads are unknown but also the number of trajectories $N$ and the time interval $T_i$ of each trajectory. Moreover, because of the inaccuracy of the detector, there is no necessary correspondence between detections $\mathbf{D}_g^t$ in an image at a given time $t$ and the set of locations at that time which are solution of the optimization problem. For that reason, we distinguish the bead locations obtained after the detection step qualified as detected beads from the locations issued from the solution trajectories qualified as target beads. The purpose of the tracking is thus to determine the target beads which are the most likely in relation with the detections.

If we define an energy function $E(\mathbf{X})$ measuring how well the target trajectories $\mathbf{X}$ fit the detections and how well they satisfy some physical a priori properties linked to sediment dynamics, the problem boils down to a minimization problem i.e. finding the set of trajectories $\mathbf{X}^*$ which globally minimizes the energy function:

$$ \mathbf{X}^* = \arg \min_{\mathbf{X} \in \mathbb{R}^n} E(\mathbf{X}) \qquad (1) $$
To solve this problem, we rely on the continuous energy minimization algo-
rithm proposed by Milan et al (2016) which formulates the energy function
as a continuous function and proposes a specific iterative minimization algo-
rithm. The whole CEM-ST tracking algorithm, is based on three steps (see
figure 2): (i) all the beads are detected along the entire image sequence, (ii)
a set of initial trajectories is determined using a greedy approach, (iii) the
energy minimization algorithm is applied to find the optimal solution.

<figure><img src="image_3.png" /><figcaption>**Fig. 2:** Synopsis of the CEM-ST algorithm. The first step is the detection of beads in all images, the second step is the initial tracking where a first set of trajectories is determined in an online approach, the third step is the global energy minimization which gives the final set of trajectories.</figcaption></figure>

The steps are further presented in the following subsections.

2.2.1 Bead detection

Due to the fixed geometry of the beads which are transparent spheres of a
known diameter *d*, we can rely on the detection algorithm proposed by Her-
gault et al (2010); Lafaye de Micheaux et al (2018). The principle is to first
correlate the processed frame by the image model of the bead and then detect
the local maxima of the result that are greater than a given detection thresh-
old *th*<sub>D</sub>. In order to have a confident value associated with each detection, we
further define the confidence *c*<sub>*g*</sub><sup>*t*</sup> of detection *g* in frame *t* as the local maximum
value in the correlation results divided by *th*<sub>D</sub>. We thus have *c*<sub>*g*</sub><sup>*t*</sup> ≥ 1 with all
the greater this value the better the resemblance to the bead model.

2.2.2 Initial tracking

The algorithm has to be initialized with an initial state matrix X₀. The idea
is to start with a first solution that is not too far from the optimal one
and that will be refined thanks to the minimization algorithm. To obtain a
good initial solution, we use the deterministic online tracking algorithm imple-
mented in the BeadTracking<fnref n="2" /> toolbox by (Lafaye de Micheaux et al, 2018).
This tracking-by-detection algorithm, hereafter referred to as OL-ST (for On
Line Sediment Tracking), combines an efficient bead detector and a greedy

<footnote n="2">²https://github.com/hugolafaye/BeadTracking</footnote>
association approach based on a constant velocity motion model and a dedicated cost function (Böhm et al, 2006). This algorithm provides a suboptimal solution because the cost function is gradually optimized as the image sequence is processed without questioning previously made associations.

### 2.2.3 Energy minimization

The purpose of this step is to find the tracking solution $\mathbf{X}^*$ that minimizes the total energy (equation 1). The main difficulty of this minimization is that both the values and the dimension of the state matrix to be determined are unknown. We therefore rely on the approach proposed by (Milan et al, 2016) which alternates between gradient descent and greedy trans-dimensional jump moves. The purpose of the gradient descent is to adjust the values of the state matrix without changing the number of trajectories or the number of target beads along each trajectory. The jump moves are operations to change the morphology of the trajectories by either removing trajectories, adding new trajectories, splitting, merging, shrinking or growing some of them (see Figure 3). These operations are greedy in the sense that each of them is tested and kept if it reduces the global energy.

<figure><img src="image_4.png" /><figcaption>Fig. 3: Examples of jump moves applied to modify the morphology of trajectories (figure adapted from Milan (2013)).</figcaption></figure>

## 2.3 Formulation of the energy function

The purpose of the energy function is to evaluate the goodness of a solution $\mathbf{X}$ with respect to the detections and the sediment dynamics. It is defined as a loss function such that the lower the energy the better the solution. In our case, we define an energy function of six terms with a formulation inspired by Milan et al (2016). The detection term $E_{det}$ tries to force the target beads to be close to the detections, the dynamic term $E_{dyn}$ favors solutions where the bead velocity does not change too fast, the forward motion term $E_{for}$ penalizes solutions with beads moving backward with respect to the fluid flow, the mutual exclusion term $E_{exc}$ avoids having multiple target beads at the same location, the trajectory persistence $E_{per}$ term favors trajectories starting at the channel input and ending at the channel output, and the regularization term $E_{reg}$ favors solutions with a low number of trajectories.
The total energy function $E$ is then defined as the linear combination of all the terms:

$$E = E_{\text{det}} + k_{\text{dyn}}E_{\text{dyn}} + k_{\text{for}}E_{\text{for}} + k_{\text{exc}}E_{\text{exc}} + k_{\text{per}}E_{\text{per}} + k_{\text{reg}}E_{\text{reg}} \quad (2)$$

where $k_{\text{dyn}}$, $k_{\text{for}}$, $k_{\text{exc}}$, $k_{\text{per}}$, $k_{\text{reg}}$ are coefficients adjusted to balance the influence of each term. We have experimentally determined them on a small training set of several frames (see section 4).

Each energy term should incorporate the constraints specific to sediment tracking. The following subsections give their precise formulation.

### 2.3.1 Detection term

The first term $E_{\text{det}}$ is designed to keep the trajectories $\mathbf{X}^i$ close to the observations $\mathbf{D}_g^t$. It is defined as:

$$E_{\text{det}}(\mathbf{X}) = \sum_{i=1}^{N} \sum_{t=s_i}^{e_i} \left[ \lambda - \sum_{g=1}^{D(t)} c_g^t \frac{r^2}{\|\mathbf{X}_t^i - \mathbf{D}_g^t\|^2 + r^2} \right] \quad (3)$$

This definition tends to promote solutions that are close to at least one detection in each frame taking into account the confidence $c_g^t$ of this detection. The role of the radius $r = d/2$ is to ignore the effect of detections which are far compared to the size of the beads. Parameter $\lambda$ is tuned to penalize targets which are not associated to one detection. It has to be adjusted according to the missing detection rate. In case we have a lot of missing detections, we have to choose a low value. Conversely, if there are few missing detections, the value should be high.

### 2.3.2 Dynamic model term

The dynamic model term $E_{\text{dyn}}$ allows regularizing the shape of trajectories by assuming the velocity vector is smoothly varying along trajectories. Thus, this terms evaluates the distance between two successive velocity vectors. It is defined as:

$$E_{\text{dyn}}(\mathbf{X}) = \sum_{i=1}^{N} \sum_{t=s_i}^{e_i-2} \|\mathbf{X}_t^i - 2\mathbf{X}_{t+1}^i + \mathbf{X}_{t+2}^i\|^2 \quad (4)$$

### 2.3.3 Forward motion term

Due to the transport effect of the water, the beads must not flow backwards. We thus add an energy term which penalize targets with backward velocity along the $x$ direction. It is defined as:

$$E_{\text{for}}(\mathbf{X}) = \sum_{i=1}^{N} \sum_{t=s_i}^{e_i-1} \max(x_{t-1}^i - x_t^i, 0) \quad (5)$$
### 2.3.4 Mutual exclusion term

The mutual exclusion term $E_{exc}$ is intended to apply a penalty to configurations in which two beads come too close to each other. It is defined as:

$$E_{exc}(\mathbf{X}) = \sum_{t=1}^{F} \sum_{i,j \neq i}^{N(t)} \frac{r^2}{\|\mathbf{X}_i^t - \mathbf{X}_j^t\|^2 + r^2} \quad (6)$$

### 2.3.5 Trajectory persistence

In case of missing detections, a trajectory can apparently start after (respectively stop before) the beginning (respectively end) of the channel. However, this situation is not possible due to mass conservation. The trajectory persistence term tends to avoid this situation by measuring the horizontal distance $d_b(x_{s_i}^i)$ between the left border of the channel and the beginning of the trajectory and the horizontal distance $d_b(x_{e_i}^i)$ between the right border of the channel and the end of the trajectory. The corresponding energy term is defined as:

$$E_{per}(\mathbf{X}) = \sum_{\substack{i=1,\dots,N \\ t \in \{s_i, e_i\}}} \frac{1}{1 + \exp(-q \cdot d_b(x_t^i) + 1)} \quad (7)$$

where parameter $q$ is the soft entry margin taken as $q = 1/r$.

### 2.3.6 Regularization term

The role of the regularization term $E_{reg}$ is to penalize solutions with a too high a number of trajectories and solutions with short trajectories. It is defined as:

$$E_{reg}(\mathbf{X}) = N + \sum_{i=1}^{N} \frac{1}{F(i)} \quad (8)$$

where $F(i)$ is the length of trajectory $i$ defined by $F(i) = e_i - s_i + 1$.

## 3 Experimental data, reference sequences and ground truths

To assess the relevance and utility of the CEM-ST algorithm for sediment transport analysis, we use high frequency series from real sediment transport experiments. Most importantly, to quantitatively evaluate the performance of CEM-ST, we need to have, for a subset of images, hereafter named reference sequence, the full trajectory ground truth, i.e., the true trajectories of each sediment particle.

The purpose of this section is first to present the general context of experiments, the experimental arrangement and the summary of the results obtained at low image frequency. Second, following a preliminary unsatisfactory analysis with our initial algorithm OL-ST (Lafaye de Micheaux et al, 2018), we
will describe how we chose two series and devise, for each of them, a reference sequence with the corresponding full trajectory ground truth.

## 3.1 General context and segregation experiments

Grain size sorting or segregation in bedload transport is an important topic. Here we are interested in particular in the vertical segregation occurring with bidisperse mixtures leading to bed structurations such as armouring (Frey and Church, 2009). The experiments presented here are part of an effort to understand the increase in transport rate resulting from a fine sediment addition.

The objective of these experiments was to analyse how the response of the channel varies following an input of finer sediment which is a frequent occurrence in natural mountain streams. It is often but not always reported an increase of transport rate results from this fine sediment input. Both the size ratio and the rate of that input were varied in an experimental campaign comprising 26 experiments. Results especially on the slope variation were obtained from low frequency (1 fps) image series (Dudill et al, 2017, 2018).

However, in all experiments, data were also recorded at high image frequency (130fps) for further refined research at the grain scale.

## 3.2 Experimental Arrangement and image acquisition

Experiments were undertaken using spherical glass beads in a channel (refs) located at the Institut national de recherche en sciences et technologies pour l'environnement et l'agriculture (now Inrae), Grenoble, France (Figure 4). The slope of the channel (tangent of the inclination angle) was 10.1%. The width of the channel was 10.3 mm. To induce bed formation, a 64 mm-high obstacle was placed at the downstream end of the channel. Water was supplied from a constant head reservoir, controlled by an electromagnetic flowmeter. The water rate was kept constant for all experiments at 49.5 ml/s.

Two feeders were used to input the sediment into the channel, one for the large beads, one for the finer beads. The diameter of the larger beads forming the coarse component of the mixture was 5.0 +/- 0.3 mm. During all experiments, the large 5 mm sediment rate and the flow rate were constant. The input coarse particle rate was 2.22 g/s equivalent to 13.6 coarse beads/s.

In a first stage, equilibrium transport was obtained with the 5 mm beads only. The outgoing sediment rate was equal to the input rate with no more increase or decrease of the bed slope. At equilibrium, both the water and bed slope matched the channel slope. Since the flow rate was kept constant, an increase (resp. decrease) of the sediment rate would result in an increase (resp. decrease) of the slope. In a second stage, a given rate of finer sediment of a given diameter was introduced into the channel. The diameter of the fine component varied from 0.7 to 4 mm (size ratio from 7.14 to 1.25). Upon infiltration, the system transitioned to a new two-size equilibrium slope, either larger or smaller than the one-size slope. In this paper, we apply our new CEM-ST algorithm
to two experiments, more specifically to image series recorded during the final two-size equilibrium (Figure 4). One involves fine particles of diameter 1.5 +/- 0.2 mm, the other uses less uniform beads with a median diameter of 0.9 mm and an interquartile ratio (D75/D25) of 1.12. All the glass beads used in the experiments had a density of 2500 kg/m³ and were transparent. In all experiments, the beads moved as bed load, including rolling and saltation.

Experiments were recorded using a Photon Focus© high-speed camera, which was positioned perpendicular to the channel and tilted at the same angle. An LED backlight panel was positioned behind the channel to provide stable, uniform illumination. The camera recorded 130 frames per second and the resolution was 1024 x 500 pixels. The images were subsequently rotated depending on the two-size slope and cropped (Figures 1 and 4).

## 3.3 Summary of results at low image frequency

The results were analysed at low image frequency (one image per second) concentrating essentially on the bed slope evolution. For the smallest size ratios, whatever the added fine rate, the bed slope always increased, similar to the one size behaviour, with no effect of segregation. Above 2.5 (diameter less or equal than 2 mm) the bed slope strongly decreased for the smallest fine rate with a two-size slope typically equal to 7-8 % compared to 10%. However, when the fine particle rate was larger than the coarse rate the bed slope positively increased with the fine rate, the more so as the size ratio was small (the diameter larger). Above a certain size-ratio-dependent value of the fine rate, a slope value superior to the initial occured. More details are given in Dudill et al (2018). In many cases, and depending on infiltration conditions (see Dudill et al (2017)), a layer of fine particles of variable thickness developed above the coarse bed (Figure 4). Studying the dynamics of the coarse particles above this fine layer by analyzing their trajectories was the motivation prompting us to develop the CEM-ST algorithm.

## 3.4 Reference sequences and ground truth designing

Preliminary image analysis with our previous standard tracking algorithm OL-ST (Hergault et al, 2010; Lafaye de Micheaux et al, 2018) gave relatively poor results. Some series were characterized by a too large amount of False Negatives and False Positives (hereafter abbreviated by FN and FP). In particular, the effect of FN bore unfavorably on the mean sediment rate calculated with tracking results. The mean image sediment rate was often underestimated representing typically only 70-80 % of the mean input sediment rate.

Clearly, it was necessary to improve particle tracking before analyzing the entire high frequency dataset. Two problems were identified. Figure 4 gives image examples for each type of problem out of two typical 8,000 image series taken during the final two-size equilibrium (corresponding videos are given in suppMat). The first is due to the occurrence of clusters resulting in numerous occlusions (Figure 4a). The second is due to the noise generated by numerous
<figure><img src="image_5.png" /><figcaption>**Fig. 4:** Example sequence of 3 images for a) Low Fine Rate (LFR), images No. 53, 55 and 57. See the occlusion-prone cluster and b) High Fine Rate (HFR), images No. 20, 22 and 24. Numerous fine particles impair good initial tracking. The bedline and the free surface are drawn on the first image of each sequence. Flow from left to right.</figcaption></figure>

much smaller fine particles (Figure 4b). Below are details of these series and
how we designed the two corresponding ground truth reference sequences that
we will use to validate our new CEM-ST algorithm.

**3.4.1 Low Fine Rate**

The chosen series (Figure 4a) is taken out of experiment No. 15 in Dudill et al (2018) (fine diameter 1.5 mm, 42% fine feed in the total input rate). It consists of the last 8,000 images (about 1 minute) recorded during the two-size final equilibrium characterized by a slope of 7.3 %. In this paper, this experiment will be termed Low Fine Rate (LFR)

The LFR series is characterized by clusters of several beads travelling
much slowly than individual beads. This regime has been termed ‘congested’
by Iseya and Ikeda (1987). Since the width of the channel is twice the bead
diameter, occlusion occured frequently in clusters. Occlusion means that a
foreground bead conceals partially or entirely a background bead. Out of the
8,000 image LFR series, a reference sequence was taken consisting of the first
866 images characterized by two clusters of about 15 beads. The first cluster
travels approximately over the first 470 images from the center of the image to
the downstream end and the second appears at the upstream end from image
720. Note that over the sequence the number of beads in a cluster varies as some beads decelerate and aggregate usually upstream and conversely some beads depart from the cluster. To get the full ground truth trajectories, we used the initial algorithm OL-ST, choosing a threshold minimizing the number of false negatives and positives. Then each image was checked by naked eye, FN were added and FP removed in a semi-automatic manner. In several cases, it was necessary to scroll over several images to make sure that a concealed bead was not missed. In total, this ground truth encompasses about 90 single beads (consistent with an input sediment rate of 13.6 beads/s over a duration of 6.7s). A total of about 17,000 beads were reviewed on the 866 images, i.e., about 20 beads per image. This reference sequence was characterized by a relatively high rate of FN, predominantly in the clusters, which meant adding by hand on average about 3 missed detections by image. Only about 0.3 FP, i.e., 1 every 3 images was removed.

### 3.4.2 High Fine Rate

The chosen series (Figure 4b) is taken out of experiment No. 12 in Dudill et al (2018) (fine diameter 0.9 mm, 69% fine feed in the total rate, meaning that the fine rate was 2.2 times the coarse rate). It again consists of the last 8,000 images recorded during the final two-size equilibrium. This series is characterized by a distinct thick fine layer. This experiment was remarkable in that the two-size equilibrium slope did not change and remained equal to the initial one size slope (10%). In this paper, this experiment will be termed High Fine Rate (HFR).

Examining the images closely gave the impression that coarse beads were sometimes partly or entirely occluded by the fine particles. Indeed the fine particles acted as added noise, resulting in a high rate of FN with our previous OL-ST bead detector. It was possible to improve detections (less FN) by choosing a smaller threshold $th_D$ (see 2.2.1) but at the expense of a dramatic increase of FP which was not acceptable.

Our second ground truth was designed with a reference sequence of 300 images out of the HFR series. The coarse beads are much less concentrated and only about 8 beads per image, with a total of about 2,300 beads to review. On average 1.3 FN beads were added and 1.8 FP beads were removed in each image.

## 4 Evaluation of CEM-ST

Given the reference sequences and their corresponding ground truths, the CEM-ST algorithm can be fully evaluated and compared to the original OL-ST algorithm. To perform a quantitative evaluation, we must first select relevant evaluation metrics. In this section, the evaluation metrics and the implementation details of OL-ST and CEM-ST algorithms are first presented. Then, both a global quantitative evaluation and a detailed analysis of spatio-temporal trajectories are presented.
## 4.1 Evaluation metrics

To evaluate the tracking performance, we use the CLEAR MOT metrics MOTA and MOTP (Bernardin and Stiefelhagen, 2008). These metrics are computed in two steps. First, a match between the target beads of the tracking solution and the ground truth beads is performed. Target beads that are correctly matched are True Positives (TP), target beads that are not matched (extra beads) are False Positives (FP) and ground truth beads that are not matched (missing beads) are False Negatives (FN). Thus a good match requires the sum of the number of FP and FN to be low. Then, considering TPs, the relevance of the tracking is evaluated through Identity Switches (IDSW), i.e. when beads are assigned to an incorrect trajectory compared to the ground truth. If $N_B$ denotes the total number of beads along the sequence, the MOTA metrics corresponds to a matching score. It is defined as:

$$ \text{MOTA} = 1 - \frac{\text{FN} + \text{FP} + \text{IDSW}}{N_B} \qquad (9) $$

The match between target beads and ground truth beads is classically checked by defining a threshold on a similarity score. In our case, as the beads are all spherical with a known radius $r$, we choose to define our similarity score $S$ using the distance $D$ between the target and ground truth bead centers and the radius $r$ as:

$$ S = \max(1 - \frac{D}{2r}, 0) \qquad (10) $$

This value is between 0 and 1 and the closer the beads, the larger the similarity. We fix the similarity threshold to 0.5 thus the matching is considered correct if $S > 0.5$.

The MOTP metrics evaluates the average similarity over all TPs.

## 4.2 Implementation details

Both algorithms have adjustable parameters which depend on the experimental conditions. For OL-ST algorithm, we must determine the detection threshold $th_D$ and the maximum velocity considered for associations denoted $v_{max}$. For CEM-ST, we also have $th_D$ and $v_{max}$ that are used to determine the initial state and we additionally have all the energy related parameters namely $\lambda$, $k_{dyn}$, $k_{for}$, $k_{exc}$, $k_{per}$ and $k_{reg}$. All the parameters were determined for both reference sequence HFR and LFR by trial and error on a subset of images of these sequences. They are given in table 1.

<table><caption>Table 1: OL-ST and CEM-ST parameters</caption><thead><tr><th rowspan="2"></th><th colspan="3">OL-ST</th><th colspan="9">CEM-ST</th></tr><tr><th>th<sub>D</sub></th><th>v<sub>max</sub></th><th>th<sub>D</sub></th><th>v<sub>max</sub></th><th>λ</th><th>k<sub>dyn</sub></th><th>k<sub>for</sub></th><th>k<sub>exc</sub></th><th>k<sub>per</sub></th><th>k<sub>reg</sub></th></tr></thead><tbody><tr><th>HFR</th><td>0.14</td><td>0.83</td><td>0.12</td><td>0.83</td><td>1.25</td><td>1.5</td><td>2</td><td>0.45</td><td>2</td><td>2</td></tr><tr><th>LFR</th><td>0.16</td><td>0.83</td><td>0.15</td><td>0.3</td><td>0.5</td><td>2</td><td>2</td><td>0.75</td><td>2</td><td>1.5</td></tr></tbody></table>
Note that the detection threshold parameter $th_D$ is chosen to be lower for CEM-ST in order to consider more detected beads even false positives (FP), knowing that this algorithm is able to select the relevant ones, thanks to the constraints imposed by the different energy terms.

## 4.3 Global quantitative evaluation

We compare globally our new CEM-ST vs initial OL-ST algorithm. For LFR (Table 2) which was the most difficult sequence to improve, CEM-ST managed to decrease significantly the FN rate from about 14% to 10% at the expense of a slight increase of the FP rate which was already very low (of the order of 2%). As a result the MOTA improved from about 82% to 86% with a very small increase of MOTP.

CEM-ST applied to the HFR case showed spectacular improvement for both the FP and FN rates (Table 3), with FP decreasing from 23.7% to 5.6% and FN from 16.6% to 7.9%. As a result the MOTA dramatically increased from about 52% to 86% with the MOTP also significantly increasing. To conclude CEM clearly improved our previous algorithm, resulting in a near perfect tracking of HFR while LFR was significantly improved.

<table><caption>Table 2: LFR results for 866 images and a total of 16996 detected beads.</caption><thead><tr><th></th><th>FP</th><th>FP (%)</th><th>FN</th><th>FN (%)</th><th>MOTA</th><th>MOTP</th></tr></thead><tbody><tr><th>OL-ST</th><td>269</td><td>1.6</td><td>2396</td><td>14.1</td><td>82.1</td><td>93.7</td></tr><tr><th>CEM-ST</th><td>373</td><td>2.2</td><td>1798</td><td>10.6</td><td>86.5</td><td>94.3</td></tr></tbody></table>

<table><caption>Table 3: HFR results for 300 images with a total of 2321 detected beads.</caption><thead><tr><th></th><th>FP</th><th>FP (%)</th><th>FN</th><th>FN (%)</th><th>MOTA</th><th>MOTP</th></tr></thead><tbody><tr><th>OL-ST</th><td>551</td><td>23.7</td><td>385</td><td>16.6</td><td>51.8</td><td>97.0</td></tr><tr><th>CEM-ST</th><td>130</td><td>5.6</td><td>183</td><td>7.9</td><td>86.3</td><td>98.3</td></tr></tbody></table>

After this global evaluation, we will analyse some examples of spatio-temporal trajectories, thanks to a 3D visualization, comparing again CEM-ST vs OL-ST.

## 4.4 Detailed analysis of spatio-temporal trajectories

The 3D view uses as coordinate system the streamwise direction (X), the depth (Y with 0 at the top of the image) and the time (frame number). The trajectories are compared to the ground truth.

### 4.4.1 High Fine Rate case

Figure 5 shows a 3D view over 25 frames of bead trajectories with the frame No.32 given as a specific example. On the initial OL-ST algorithm (fig. 5a)
<figure><img src="image_6.png" /><figcaption>**Fig. 5:** Global comparison of trajectories between frames 25 and 50 on the HFR sequence, with example of frame No.32 using the initial OL-ST (a) and new CEM-ST (b) algorithms. Note that colors agree between the 3D view and the frame No.32 but not between OL-ST and CEM-ST since a different number of trajectories changes the ordering.</figcaption></figure>

several short trajectories are due to false positives (three of them appear on frame No.32) occurring somewhat randomly and sometimes remaining at the same location for several frames (see in fig. 5a the upper right vertical trajectory corresponding to the upper right corner in frame 32). CEM-ST (fig. 5b) in this case strikingly removes all false positive trajectories. There is also a trajectory affected by a false negative (in fig. 5b 2nd bead from the right on frame 32). The bead is detected by CEM-ST on frame 32 contrary to the initial OL-ST. However, it is not the case in later frames, showing here some limitation of CEM-ST.

### 4.4.2 Low Fine Rate case

We first compare the 3D global views of the initial OL-ST vs new CEM-ST algorithm focusing on the cluster of beads. We can notice both small erratic lines essentially due to FPs and broken trajectories essentially due to FNs (see fig. 6a). The CEM-ST algorithm (fig. 6b) works very well in removing practically all small erratic lines and quite fairly in reattaching broken trajectories though not perfectly. In addition, a video of trajectories corresponding to fig. 6b is given in supplementary information.

We focus (fig. 7) on a pattern involving three beads (numbered 1 to 3 from left to right) followed on six images, where both FP and FN have intertwined effects. Bead 2 in frame 178 is not detected and at the same time there is a FP appearing between beads 1 and 3. As a consequence, bead 2 connects erroneously with the FP bead (green small trajectory). Bead 2 remains not
<figure><img src="image_7.png" /><figcaption>**Fig. 6:** Global comparison of trajectories between frames 130 and 180 on the LFR sequence, with example of frame No.140, using the initial OL-ST (a) and new CEM-ST (b) algorithms. See also supplementary material. Note that colors agree between the 3D view and the frame No.140 but not between OL-ST and CEM-ST since a different number of trajectories changes the ordering.</figcaption></figure>

detected (FN) until reappearing on frame 182. CEM-ST (fig. 7d) is able to get rid of the FP while at the same time detecting bead 2 in all frames allowing correct calculation of the entire bead 2 trajectory. This example clearly illustrates the improvement provided by the new CEM-ST algorithm.

# 5 Depth profile analysis

Depth profiles of concentration, particle velocity and particle rate give invaluable insights into the kinematics and dynamics of segregation. We will first confront our previous OL-ST and new CEM-ST algorithms to Ground Truth depth profiles. Then we will apply CEM-ST to the two long duration entire series to improve our process knowledge.

## 5.1 Postprocessing

Results for each bead center location and velocity are used to calculate concentration, particle velocity and particle rate depth profiles. We partition each image into bins of equal size parallel to the *x* streamwise direction, comprising the full length of the image and the height of 1 pixel in the normal direction *y*, corresponding to about 1/20 th of the diameter. The procedure is similar to that of Hill and Fan (2008) and Maurin et al (2015). For each elevation, the volumetric solid concentration *C* is calculated as the sum of all bead volume slices present in the bin divided by the total volume of the bin. The volume of
<figure><img src="image_8.png" /><figcaption>**Fig. 7:** Occlusion and false detection case around frame 179. a) and c) In the initial OL-ST algorithm, trajectory of bead 2 is missed from frame 177 to frame 181 (FN) due to occlusion with bead 1. It reappears only in frame 182 as bead 4. A false detection (FP) appears at frame 178 connecting erroneously with bead 2. b) and d) The two close beads 1 and 2 are well detected by CEM-ST algorithm and there is no false detection.</figcaption></figure>

a sphere slice is simple to calculate since it can be evaluated analytically. The average particle velocity $U_{px}$ is calculated in the same way, with the velocity of each bead weighted by its volume in the bin. The temporal mean is calculated over all images in a given time sequence. The particle transport rate density per unit width at each elevation is defined as $qsx = U_{px}C$. In the following we simply refer to the particle rate.

The origin of the elevation y-axis was taken at the mean spatio-temporal bedline delimiting the quasi-static bed from the bedload layer. An example on a specific image is given in fig. 4 for each sequence. The bedline is located in the upper part of the layer of fine particles. Following Lafaye de Micheaux
(2017), the bedline was detected based on morphological operations, gradient calculations and watershed transformations (Vincent and Soille, 1991).

Results are presented following the usual convention for depth profiles in fluid mechanics. The variables of interest (C, Upx, qsx) are not plotted on the y-axis but rather on the x-axis and the elevation normalized by the large diameter (y*) on the vertical y-axis.

## 5.2 Comparing initial OL-ST and new CEM-ST algorithms to Ground Truth (GT) depth profiles

This comparison is made on both the HFR and LFR cases. Figure 8 gives the depth profiles in the High Fine Rate (HFR) case comparing OL-ST and CEM-ST algorithms vs the GT.

<figure><img src="image_9.png" /><figcaption>Fig. 8: Comparing depth profiles of the High Fine Rate case (HFR) based on initial OL-ST and new CEM-ST algorithms vs Ground Truth (GT). a) Particle concentration b) Streamwise particle velocity Upx c) Particle transport rate qsx</figcaption></figure>

CEM-ST gives spectacular results on HFR depth profiles. Although the detection threshold $th_D$ was actually optimized to minimise the number of FP detections, the initial algorithm still gave too many FPs especially in the upper elevations. It is obvious on the concentration where for example at $y^* = 1.5$, the initial OL-ST concentration is about twice that of the GT. There is also significant improving in reducing the number of FN for $0.7 \le y^* \le 1.2$ resulting in an increase of the peak value toward the GT. Even more spectacular is the particle velocity depth profile. We recall that in this configuration, beads are never stopping, moving relatively uniformly with a velocity ranging from 0.33 to 0.45 m/s, from bottom to top. The initial algorithm performs poorly with a systematic under estimation and a dramatic unphysical decrease of the velocity above $y^*=1$. On the contrary, CEM-ST gives almost perfect results on the entire range except a slight underestimation above $y^*=1.5$. Spectacular improving of both particle velocity and concentration results of course in the
20

sediment rate profile matching closely the GT with a significant increase of the peak, though residual FPs affect the sediment rate above y*=1.6 with low but non zero values around y*=2, whereas GT values are already zero. Note that the good matching of the initial OL-ST algorithm around y*=1.5 is spurious since the underestimation of the particle velocity compensates the overestimation of the concentration (remember that the sediment rate is obtained by multiplying the concentration by the particle velocity). This advocates the need to analyse not only the sediment rate, but also primarily both concentration and particle velocity profiles. This evaluation gives us confidence in this new CEM-ST algorithm which will be applied to the entire series of 8,000 images (about 1 min) in the next section.

<figure><img src="image_10.png" /><figcaption>**Fig. 9:** Comparing depth profiles of the Low Fine Rate case (LFR) based on initial OL-ST and new CEM-ST algorithms vs Ground Truth (GT). a) Particle concentration b) Streamwise particle velocity Upx c) Particle transport rate qsx</figcaption></figure>

LFR (low fine rate) is a more difficult sequence due to two clusters where numerous occlusions occur. The FPs are not a problem with a low rate of 1.6 %. In contrast the FN rate is high at 14.1% (see table 2). Figure 9 gives the depth profiles in the LFR case. In the upper region of the concentration profile (panel a) the initial OL-ST algorithm and hence CEM-ST performed well. In the bottom region up to y*=0.7, CEM-ST was not able to improve much the concentration profile. However the peak value region was improved increasing toward the truth. Overall the FN rate improved from 14% to 10%. Concerning particle velocity profiles, CEM-ST allows for a slight improvement in the lower region. More spectacular is the improvement in the upper part above y*=1.2 where the OL-ST algorithm underestimated the velocity while CEM-ST reproduces almost perfectly the ground truth profile. As a result of improvements of the concentration and velocity, the CEM-ST sediment rate profile almost perfectly follows the ground truth above y*=0.8. Globally, while the OL-ST algorithm only retrieved around 94% of the GT sediment rate,
CEM-ST increased this value to 98.5%, which is very good given the difficulty of this sequence.

## 5.3 Applications based on depth profiles obtained with CEM-ST

We propose two applications of CEM-ST. The first deals with the influence of the fine particle rate and the second is focused on the effect of clusters on bedload segregation.

### 5.3.1 Contrasting the low (LFR) vs the high (HFR) fine rate cases

<figure><img src="image_11.png" /><figcaption>**Fig. 10:** Contrasting the low (LFR) vs the high (HFR) fine rate depth profiles both with initial OL-ST and new CEM-ST algorithms. a) Particle concentration b) Streamwise particle velocity Upx c) Particle transport rate qsx.</figcaption></figure>

We use here the entire two series each made of 8,000 images taken during the final two-size equilibrium. Depth profiles for the entire series are shown in Figure 10 comparing HFR and LFR. Obviously LFR is characterized by a higher concentration and a lower velocity than HFR. The peak of HFR concentration is slightly lower which corresponds to the visual impression that beads are drowned in the thick fine layer. The spread of the LFR concentration is larger than HFR with particles present at higher elevation presumably saltating over clusters. While the LFR velocity increases quasi linearly, HFR velocity profile is remarkable in that it is fairly constant over most of the depth at about 0.4 m/s. The velocity is still high at the bedline which is related to the visual impression of the thick layer acting as a moving carpet.

The sediment rate profiles are Gaussian-like and the total sediment rate of both HFR and LFR are practically equal (area within the bell shape) as it should be since at equilibrium over a sufficient duration (here about one minute) both the outgoing and ingoing particle rates are constant and equal
for both configurations. This is a good improvement of the CEM-ST algorithm compared to the OL-ST, which gives non-equal unphysical rate values.

### 5.3.2 Contrasting cluster vs loose regions in the low fine rate (LFR) case

The LFR case is essentially characterized by several clusters but between clusters there is a more or less spatially extended region of beads moving individually named here loose region. Below we contrast the depth profiles of the two regions.

We tracked the cluster moving in the half right region of the 200 first images (see figure 4a). The loose region was analysed over 300 images beginning at image No. 450.

<figure><img src="image_12.png" /><figcaption>**Fig. 11:** Contrasting depth profiles in cluster vs loose region in the LFR case. The cluster is tracked in the half right region of the 200 first images. The loose region is analysed over 300 images beginning at image No. 450. a) Particle concentration b) Streamwise particle velocity $U_{px}$ c) Particle transport rate $q_{sx}$</figcaption></figure>

Figure 11 shows striking differences in the depth profiles of the cluster zone vs. the loose zone. Obviously, the concentration is larger and the velocity lower in the cluster than in the loose region. The concentration profile is Gaussian-like in the loose region while in the upper elevations the concentration tends more slowly to zero for the cluster region. This is certainly due to the odd loose particles joining or departing the cluster and to those saltating over the cluster. More strikingly, the peak in depth profiles of both the concentration and the sediment rate occurs at a lower elevation for the cluster. It means that comparatively a larger number of particles are moving in the upper elevations in the loose case while particles in the cluster are stuck in the lower part near the bedline. Looking at the sequence, a more subtle effect can be evidenced. In presence of a cluster the bedline is lower as if the cluster would compact the fine particle layer (see half right zone of figure 4a). So the peak in the
depth profiles of concentration and sediment rate is lower not only due to
fewer particles in the upper elevations but also because the entire cluster is
shifted downward compared to the surrounding loose particles. A systematic
analysis of the interaction of clusters with the fine particle layer, presumably
a granular physics effect, would maybe explain in the future the role of this
fine particle layer in the formation or not of clusters and more generally why
a fine sediment addition increases sediment transport.

Finally, we note that the integrated sediment rate is larger for the clus-
ter than for the loose case. Indeed the higher velocity in the loose case does
not compensate the much lower concentration. This may seem peculiar since
bedload is at equilibrium with a constant coarse particle rate. However previ-
ous studies (Böhm et al, 2004; Ancey and Heyman, 2014) have shown that in
bedload transport large fluctuations occur the more so over small timesteps.
A typical duration of one minute is actually necessary to achieve a converged
constant sediment rate. Here we have considered two-second samples so that
there is no reason that both sediment rates should have the same value. A
larger sediment rate for the cluster might even partly explain why fluctuations
occur. A systematic high frequency analysis of depth profiles within a sequence
would certainly in the future give new insights on this fluctuation issue.

6 Conclusion

This new CEM-ST algorithm thanks to a global optimization scheme performs
significantly better and is less sensitive to the quality of the detections than
online algorithms that only consider previous timesteps. Better performance
was demonstrated by a thorough evaluation based on our experimental refer-
ence sequences and corresponding full trajectory ground truths made available
to the community. The evaluation of CEM-ST was positive not only globally
but also in the details of the trajectories and the depth profiles. The evaluation
of the depth profiles of both concentration and particle velocity and not only
particle rate showed the full capabilities of this new CEM-ST algorithm, which
is able to eliminate spurious effects. Applying CEM-ST allowed us a better
insight into the influence of the fine particle rate on segregation, disentangling
in particular the role of clusters in the bedload dynamics.

In the future, this new algorithm will be applied to other high-frequency
experimental series to allow a more confident analysis. Improvements are still
possible along two research avenues: first, the use of other detectors, in par-
ticular those based on deep learning, and second, a better management of
occlusions, especially in the presence of a high concentration of clusters.

**Supplementary information.** A video is given consisting of raw images and superimposed trajectories, obtained by the new CEM-ST algorithm between frames 130 and 180 of the Low Fine Rate (LFR) sequence. It corresponds to Fig. 6b.
**Data availability.** The two ground truths (reference image sequences and files) are available in zenodo repository. The link is https://zenodo.org/records/14624102.

The CEM-ST Matlab code together with a tutorial and a subset of LFR images are available at https://gitlab.univ-st-etienne.fr/labhc-iscv/cem-st.

**Acknowledgments.** The authors are grateful to Frédéric Nouguier for helping designing the low fine rate ground truth. This research was funded by the French Agence Nationale de la Recherche Project No. ANR-16-CE01-0005 SegSed, “Size segregation in sediment transport.” The authors acknowledge the support of INRAE, IGE (formerly Irstea/ETNA), Labex Osug@2020 Investissements d'Avenir Grant No. ANR-10-LABX-0056 and Labex TEC21 Investissements d'Avenir Grant No. ANR-11-LABX-0030.

## References

Ancey C (2020) Bedload transport: a walk between randomness and determinism. part 2. challenges and prospects. Journal of Hydraulic Research 58(1):18–33. https://doi.org/10.1080/00221686.2019.1702595

Ancey C, Heyman J (2014) A microstructural approach to bed load transport: mean behaviour and fluctuations of particle transport rates. Journal of fluid mechanics 744:129–168. https://doi.org/10.1017/jfm.2014.74

Berclaz J, Fleuret F, Turetken E, et al (2011) Multiple object tracking using k-shortest paths optimization. IEEE Transactions on Pattern Analysis and Machine Intelligence 33(9):1806–1819. https://doi.org/10.1109/TPAMI.2011.21, publisher: IEEE

Bernardin K, Stiefelhagen R (2008) Evaluating multiple object tracking performance: the clear mot metrics. EURASIP Journal on Image and Video Processing 2008:246,309

Böhm T, Ancey C, Frey P, et al (2004) Fluctuations of the solid discharge of gravity-driven particle flows in a turbulent stream. Physical review E 69(6):061,307. https://doi.org/10.1103/PhysRevE.69.061307

Böhm T, Frey P, Ducottet C, et al (2006) Two-dimensional motion of a set of particles in a free surface flow with image processing. Experiments in fluids 41:1–11

Breitenstein MD, Reichlin F, Leibe B, et al (2011) Online multiperson tracking-by-detection from a single, uncalibrated camera. IEEE Transactions on Pattern Analysis and Machine Intelligence 33(9):1820–1833

Capart H, Fraccarollo L (2011) Transport layer structure in intense bedload. Geophysical Research Letters 38:L20,402. https://doi.org/10.1029/
2011gl049408

Capart H, Young D, Zech Y (2002) Voronoï imaging methods for the
measurement of granular flows. Experiments in Fluids 32(1):121–135

Du Y, Wan J, Zhao Y, et al (2021) Giaotracker: A comprehensive framework for mcmot with global information and optimizing strategies in visdrone 2021. In: Proceedings of the IEEE/CVF International conference on computer vision, pp 2809–2819

Du Y, Zhao Z, Song Y, et al (2023) StrongSORT: Make DeepSORT Great Again. IEEE Transactions on Multimedia 25:8725-8737. https://doi.org/10.1109/TMM.2023.3240881, conference Name: IEEE Transactions on Multimedia

Dudill A, Frey P, Church M (2017) Infiltration of fine sediment into a coarse mobile bed: a phenomenological study. Earth Surface Processes and Landforms 42(8):1171–1185. https://doi.org/10.1002/esp.4080

Dudill A, Lafaye de Micheaux H, Frey P, et al (2018) Introducing finer grains into bedload: The transition to a new equilibrium. Journal of Geophysical Research: Earth Surface 123(10):2602–2619. https://doi.org/10.1029/2018JF004847

Ferdowsi B, Ortiz CP, Houssais M, et al (2017) River-bed armouring as a granular segregation phenomenon. Nature Communications 8:1363. https://doi.org/10.1038/s41467-017-01681-3

Frey P (2014) Particle velocity and concentration profiles in bedload experiments on a steep slope. Earth Surface Processes and Landforms 39(5):646–655. https://doi.org/10.1002/esp.3517

Frey P, Church M (2009) How river beds move. Science 325(5947):1509–1510.
https://doi.org/10.1126/science.1178516

Frey P, Lafaye de Micheaux H, Bel C, et al (2020) Experiments on grain size segregation in bedload transport on a steep slope. Advances in Water Resources 136:103,478

Henriques JF, Caseiro R, Batista J (2011) Globally optimal solution to multi-object tracking with merged measurements. In: 2011 International Conference on Computer Vision. IEEE, pp 2470–2477, https://doi.org/10.1109/ICCV.2011.6126532

Hergault V, Frey P, Métivier F, et al (2010) Image processing for the study of bedload transport of two-size spherical particles in a supercritical flow. Experiments in fluids 49(5):1095–1107
Hill KM, Fan Y (2008) Isolating segregation mechanisms in a split-bottom cell. Physical Review Letters 101(8):088,001

Hofmann M, Wolf D, Rigoll G (2013) Hypergraphs for joint multi-view reconstruction and multi-object tracking. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 3650–3657

Iseya F, Ikeda H (1987) Pulsations in bedload transport rates induced by a longitudinal sediment sorting - a flume study using sand and gravel mixtures. Geografiska Annaler Series a-Physical Geography 69(1):15–27

Julien P (2012) Erosion and sedimentation. 2nd ed. Cambridge University press

Maurin R, Chauchat J, Chareyre B, et al (2015) A minimal coupled fluid-discrete element model for bedload transport. Physics of Fluids 27(11):113,302. https://doi.org/10.1063/1.4935703

Lafaye de Micheaux H (2017) Image processing for segregation in bedload sediment transport : morphology and tracking. Doctoral dissertation, University Jean Monnet of Saint-Etienne

Lafaye de Micheaux H, Ducottet C, Frey P (2018) Multi-model particle filter-based tracking with switching dynamical state to study bedload transport. Machine Vision and Applications 29(5):735–747

Milan A (2013) Energy minimization for multiple object tracking. Doctoral dissertation, Technical University of Darmstadt

Milan A, Schindler K, Roth S (2016) Multi-target tracking by discrete-continuous energy minimization. IEEE Transactions on Pattern Analysis and Machine Intelligence 38(10):2054–2068

Ouellette NT, Xu HT, Bodenschatz E (2006) A quantitative study of three-dimensional lagrangian particle tracking algorithms. Experiments in Fluids 40(2):301–313

Possegger H, Mauthner T, Roth PM, et al (2014) Occlusion geodesics for online multi-object tracking. In: proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 1306–1313

Schanz D, Gesemann S, Schröder A (2016) Shake-the-box: Lagrangian particle tracking at high particle image densities. Experiments in fluids 57(5):70

Schröder A, Schanz D (2023) 3d lagrangian particle tracking in fluid mechanics. Annual Review of Fluid Mechanics 55:511–540
Vincent L, Soille P (1991) Watersheds in digital spaces: An efficient algorithm based on immersion simulations. IEEE Transactions on Pattern Analysis and Machine Intelligence 13(6):583–598. https://doi.org/10.1109/34.87344

Wang B, Wang G, Chan KL, et al (2016) Tracklet association by online target-specific metric learning and coherent dynamics estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence 39(3):589–602. https://doi.org/10.1109/TPAMI.2016.2551245, publisher: IEEE

Wang G, Wang Y, Zhang H, et al (2019) Exploit the Connectivity: Multi-Object Tracking with TrackletNet. In: Proceedings of the 27th ACM International Conference on Multimedia. ACM, Nice France, pp 482–490, https://doi.org/10.1145/3343031.3350853