# Tracking of Intermittent and Moving Speakers: Dataset and Metrics

Taous Iatariene, Alexandre Guérin, Romain Serizel

► To cite this version:

Taous Iatariene, Alexandre Guérin, Romain Serizel. Tracking of Intermittent and Moving Speakers: Dataset and Metrics. Proceedings of the 11th Convention of the European Acoustics Association Forum Acusticum 2025, Jun 2025, Malaga, Espagne, Spain. hal-05058365

HAL Id: hal-05058365

https://hal.science/hal-05058365v1

Submitted on 10 Jun 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_2.png" /></figure>

# TRACKING OF INTERMITTENT AND MOVING SPEAKERS: DATASET AND METRICS

Taous Iatariene¹²*, Alexandre Guérin<fnref n="1" />, Romain Serizel<fnref n="2" />

## ABSTRACT

This paper presents the problem of tracking intermittent and moving sources, i.e, sources that may change position when they are inactive. This issue is seldom explored, and most current tracking methods rely on spatial observations for track identity management. They are either based on a previous localization step, or designed to perform joint localization and tracking by predicting ordered position estimates. This raises concerns about whether such methods can maintain reliable track identity assignment performance when dealing with discontinuous spatial tracks, which may be caused by a change of direction during silence. We introduce LibriJump, a novel dataset of acoustic scenes in the First Order Ambisonics format focusing on speaker tracking. The dataset contains speakers with changing positions during inactivity periods, thus simulating discontinuous tracks. To measure the identity assignment performance, we propose to use tracking association metrics adapted from the computer vision community. We provide experiments showing the complementarity of association metrics with previously used tracking metrics, given continuous and discontinuous spatial tracks.

**Keywords:** speaker tracking, dataset, tracking metrics, intermittent speakers, discontinuous tracks

## 1. INTRODUCTION

Sound source tracking is a subtask of acoustic scene analysis, which aims at estimating the spatial tracks of sound sources from multichannel audio recordings. It involves two main considerations: spatial, mainly through Direction of Arrival (DoA) estimation, and identity-related through track identity (ID) management techniques. The last decade has demonstrated interest in this research field, as shown by the organization of the LOCATA challenge [1], where efforts were made towards the construction of an open recorded dataset of moving speakers in various configurations [2], and the sharing of an evaluation framework to evaluate tracking systems<fnref n="1" />.

Most studies focus on tracking continuously active sources, setting aside the challenging yet realistic scenario of intermittent and moving sound sources, which may move while silent. To the best of our knowledge, only seldom works have addressed this issue, often by tracking both active and inactive sources and assuming continuous movement during inactivity [3], [4]. However, difficulties arise when change of direction occurs during silence and lead to discontinuous spatial tracks [4]. Indeed, track ID management in such cases becomes more challenging, increasing the risk of assigning multiple redundant identities to the same source. Another resulting issue lies in the careful choice of evaluation metrics for evaluating track ID assignment, which is the principal type of error induced by movement during silences.

In this paper, we focus on speaker tracking and propose the following contributions. First, we introduce LibriJump, an evaluation dataset of synthetic acoustic scenes in the First Order Ambisonics (FOA) format con-

<footnote n="1">¹ Orange Innovation, 35000 Rennes, France</footnote>

<footnote n="2">² Université de Lorraine, CNRS, Inria, LORIA, 54000 Nancy, France</footnote>

<footnote>*Corresponding author: taous.iatariene@orange.com.
Copyright: ©2025 Taous Iatariene et al. This is an open-access article distributed under the terms of the Creative Commons Attribution 3.0 Unported License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.</footnote>

<footnote n="1">¹https://github.com/cevers/sap_locata_eval</footnote>
<figure><img src="image_3.png" /></figure>

taining intermittent and moving speakers. In this dataset, speakers remain static while speaking but change position (jump) when silent, leading to discontinuous spatial tracks. Second, we propose to complement the LOCATA tracking metrics with the tracking association metrics [5], which are focused on evaluating the ID assignment performances of tracking systems on a global level rather than on a short context (frame) level. Initially proposed by Luiten et al. [5], these metrics have gained popularity in Multi-Object-Tracking. Finally, we analyze the complementarity of the proposed global association metrics with frame-level identity related tracking metrics.

## 2. PROBLEM FORMULATION

We consider an indoor acoustic scene in which one or several speakers are discussing, such as an office meeting between colleagues. In such scenarios, speakers may move around, either while speaking or when silent. Given a microphone array (multichannel) recording of such acoustic scene, the speaker's positions relatively to the microphone array (i.e, the DoA), can be extracted.

In the rest of this paper, we denote as *J* the number of speakers in the scene. We use the term “track” to refer to the time-varying DoA of each speaker. As illustrated by Fig. 1, each track is labeled with a track identity (ID) designing its corresponding speaker. Depending on the nature of movement during silence, a track can be either continuous or discontinuous. Continuous tracks occur when the speaker is static (Track 1 in Fig. 1) or when it moves continuously when silent (Track 2). Discontinuous tracks can be due to non-predictable movement occurring during silence, whether the speaker is static while speaking (Track 3), or moving (Track 4).

The goal of a tracking system is to reconstruct the speakers' spatial tracks given the multichannel recording. This implies two main considerations: spatial DoA estimation, and track ID management (i.e, assigning the correct ID to DoA through time). In the rest of this paper, we use the term “predictions” or “predicted tracks” to refer to the output of a tracking system. Similarly, “ground truths” or “ground truth tracks” will refer to the real speaker's time-varying position.

## 3. RELATED WORKS

### 3.1 Sound source tracking

A two-step approach of sound source localization for DoA estimation [6], followed by Bayesian filtering to link the

DoAs through time, is broadly adopted for tracking [1]. In such approaches, tracks' “states” (position, velocity) are updated at each new timestep given a new set of DoA observations, using a Bayesian filter like a particle filter [7].

In the more challenging multi-speaker case, observation-to-track association is usually performed before Bayesian filtering. Track ID management rely generally on the latter step, in which tracks are initiated, updated or deleted given the result of observation-to-track association. Thus, we observe that track ID management relies on DoAs (spatial observations). While this may be acceptable when speakers are continuously active or when their movement while silent is predictable (tracks 1 and 2 on Fig. 1), ID assignment errors may occur, especially in case of change of direction during silence like tracks 3 and 4 in Fig. 1.

Only several works have addressed tracking of intermittent and moving speakers. Quinlan et al. [3] and Li et al. [4] proposed to track a fixed number of simultaneously active speakers, whether active or inactive, by assuming continuous movement during silence for track position update. Quinlan et al. [3] proposed to estimate dynamically the number of active speakers $N_a$ and mark as active the $N_a$ tracks with the highest probability of existence. Li et al. [4] proposed to evaluate the speech activity of each track. Discontinuous tracks due to direction changes during silences were a limitation of their study.

Recent deep learning-based tracking approaches were proposed, by predicting ordered position estimates, through the permutation-aware training of localization networks [8], [9]. Emphasis has been given on the spatial track reconstruction, setting aside the track ID manage-

<figure><img src="image_4.png" /><figcaption>**Figure 1:** Example of continuous and discontinuous tracks. Each color designs a different track ID.</figcaption></figure>
<figure><img src="image_5.png" /></figure>

ment aspect. The number of predicted tracks is limited by the number of output branches of the networks. Also, the permutation-aware training usually lead to the *block permutation problem* described in speech separation [10], but still not addressed for deep-learning based audio tracking.

We recently proposed a novel approach, focused on speaker tracking, where DoAs and identity-related observations (speaker embeddings) were combined to improve the identity assignment performance of several tracking systems, in the occurring of discontinuous tracks [11].

## 3.2 Moving speakers datasets

We focus on this section on summarizing the existing datasets of moving speakers. A more exhaustive overview of existing sound source tracking datasets can be found in the works of Grumiaux et al. [6] and Yang et al. [12].

Due to the difficulty of acquiring metadata of the sources' positions, which is crucial for supervised training and evaluation of tracking systems, only a few datasets of real recordings are available. This is why it is often resorted to synthetic data generation, where room acoustics (spatialization and reverberation) is simulated through Spatial Room Impulse Responses (SRIRs), which are then convolved with so-called *dry* (clean) speech sources. We note in particular the works of Diaz-Guerra et al. [13] to simulate acoustic scenes with moving sound sources.

As for labeled recorded datasets of moving speakers, the LOCATA data corpus contains recordings (≈ 1 hour) of static and moving english speakers, using multiple microphone arrays [2]. The recent RealMAN dataset, is a larger scale recorded corpus (83 hours) of static and moving mandarin speakers, in various acoustic scenes [12].

All previously mentioned datasets provide recordings of continuously moving sources (whether static or moving). As a result, recordings containing speakers with discontinuous spatial tracks are scarce to find.

## 3.3 Tracking metrics

Tracking systems require metrics evaluating the two aforementioned aspects of tracking: spatial-related through localization metrics, and identity-related through the measurement of track ID assignment errors. The LOCATA challenge have provided a first set of metrics derived from Multi-Target-Tracking [1]. The CLEAR metrics from Multi-Object-Tracking (MOT) [14] have also been adapted to audio tracking more recently [8]. We provide here insights on the computation of those metrics.

Given ground truths and predicted tracks, a prior to metric computation is to perform a one-to-one matching between predictions and ground truths, on the frame level. Matching pairs form True Positives (TPs), while non-assigned predictions become False Positives (FPs), and non-assigned ground truths False Negatives (FNs).

Localization metrics are estimated on the set of TPs, often using the mean angular distance as an error measure [8], [9]. For identity related metrics, further distinction can be made between measuring detection errors (taking into account FPs and FNs) and association errors (ID mismatches within the set of TPs). We focus in the following on association metrics.

In LOCATA, track ID *swaps* on the set of TPs are defined, by comparing the consistency of the predicted track IDs given their matching ground truth track IDs, from frame to frame. Any change in predicted ID given the same ground truth ID result in a swap. *Broken* tracks are also measured by counting the number of times a ground truth track is associated then remains unassociated on the next frame (i.e, moving from TP to FN). Track Swap Rate (TSR) is defined as the number of swapped tracks, and Track Fragmentation Rate (TFR) as the number of broken plus swapped tracks. The CLEAR metrics defines Identity Switches (IDSW) similarly to track ID *swaps*.

Global metrics combining localization, detection and association errors, have been developed to evaluate the overall tracking efficiency. In LOCATA, the Optimal Sub-Pattern Assignment (OSPA) metric combines cardinality error and localization error, making it able to evaluate both detection and localization. The CLEAR metrics propose the Multi Object Tracking Accuracy (MOTA) score, in which IDSW is combined with FNs and FPs, thus measuring detection and association errors.

# 4. LIBRIJUMP DATASET

We present LibriJump, an evaluation dataset containing intermittent and moving speakers<fnref n="2" />.

## 4.1 Description

LibriJump is a synthetic dataset, divided into three subsets (LibriJump-1spk, LibriJump-2spk, LibriJump-3spk) for three different values of *J* (the number of speakers in the scenes). Each subset contains 150 mixtures of acoustic

<footnote n="2">² The dataset will be shared on zenodo.org upon publication.</footnote>
<figure><img src="image_6.png" /></figure>

scenes in the First Order Ambisonics (FOA) format, of 60 second each. The main objective of this dataset is to provide the scientific community with acoustic scenes containing speakers with discontinuous spatial tracks, in order to draw attention to the challenges raised by such scenarios in term of track ID assignment. In LibriJump, emphasis is placed on track discontinuity, and challenges induced by spatially dynamic tracks (i.e, movement during activity) are minimized. Indeed, the latter scenario is already addressed in previously mentioned datasets. Consequently, similar to track 3 in Fig. 1, speakers remain static while speaking but move when silent, thus simulating “jumps” in speakers positions. This allows easier spatial reconstruction, without taking anything away from the difficulty induced by discontinuities during silence in terms of identity assignment. LibriJump is derived from the LibriSpeech test-clean dataset [15].

## 4.2 Methodology

To simulate a mixture of *J* speakers, we: select *dry* (clean) speeches, generate SRIRs, spatialize the dry speech, and add background noise.

Dry speech sources are obtained by first selecting *J* random speakers from the LibriSpeech test-clean dataset. For each speaker, several utterances are chosen randomly, and concatenated to reach 60 second of dry speech.

We obtain SRIRs through the simulation of *Shoe-boxes* rooms. For each shoebox room, 6 SRIRs are generated, corresponding to 6 positions relatively to one microphone position. Positions are chosen randomly to achieve a minimal angular distance of 60° between each pair of positions. We rely on a custom FOA SRIR generator based on Pyroomacoustics [16]. Shoeboxes are generated by varying randomly the reverberation time (RT60) from 0.2 to 0.8 seconds, and the length and width from 2 to 10 m, and height from 2 to 3 m.

Each speaker is then spatialized individually. First, its corresponding dry speech is segmented using a voice activity detector<fnref n="3" />. Each segment of speech is then convolved with one of the 6 SRIRs to simulated changing positions from one period of activity to another. We add a random silence from 0.1 to 1 second between each segment to reduce overlap. Then, we adjust the level of one arbitrary speaker, and set the levels of the other speakers to be 2 to 6 dB lower. The summation of the *J* spatialized signals leads to the obtention of the unnoisy mixture.

A background noise is randomly selected from samples taken from Freesound [17]. It is then convolved with the diffuse part of two SRIRs to obtain a diffuse (non spatialized) noise. The resulting diffuse noise is added to the mixture with a fixed SNR of 15 dB.

# 5. TRACKING ASSOCIATION METRICS

## 5.1 Description

As with the CLEAR metrics, we propose to adapt to audio tracking the recent Higher Order Tracking Accuracy (HOTA) metrics proposed by Luiten *et al.* [5]. The HOTA metrics are divided into sub-metrics evaluating localization, detection and association errors.

We adapt the subset of HOTA measuring association errors: association accuracy, precision and recall (AssA, AssPr, AssRe). These association metrics provide a global evaluation of predictions against ground truths in term of track ID assignment, in contrast with IDSW, TFR and TSR which are measured at the frame level.

Tracking association metrics overcome numerous limits of CLEAR tracking metrics. As mentioned, IDSW evaluates association errors at the frame-level. This can penalize a tracker that has wrongly switched but corrects itself later on. Moreover, IDSW only focuses on the switches made on predicted IDs, setting aside the evaluation of switches that can be made over ground truth IDs. The MOTA score values equally FP, FNs (related to detection) and IDSW (related to association). As a result, MOTA gives twice as much priority to detection evaluation than association. More limits to CLEAR metrics can be found in the paper of Luiten *et al.* [5].

One limitation to association metrics is that it is not evaluating the degree of track fragmentation. Although a fragmentation aware HOTA was proposed [5], we choose to keep TFR to evaluate the degree of track fragmentation. Tracking association metrics are thus viewed as complementary to identity-related metrics.

AssRe focuses on predicted IDs errors, and measures how a single ground truth track may be spread into multiple predicted tracks, with a low score indicating a tendency to split a ground truth track into many predicted ones. AssPr focuses on ground truth IDs errors (not measured by IDSW), and measures how multiple ground truth tracks can be merged into a single predicted track, with a low score indicating that a single predicted track is an aggregate of multiple portions of ground truth tracks. AssA provides an overall score combining AssRe and AssPr.

<footnote n="3">³https://huggingface.co/speechbrain/vad-crdnn-libriparty</footnote>
<figure><img src="image_7.png" /></figure>

# FORUM ACUSTICUM
## EURONOISE 2025

## 5.2 Calculation

In addition to the previously defined concepts of TP, FP, FN, Luiten et al. [5] define a novel concept of TPA, FPA and FNA ("A" standing for Association).

More precisely, given a TP couple $c = (prID, gtID)$ denoting a matched prediction and ground truth ID, respectively, TPAs are the TPs sharing the same ID match as $c$ (i.e, the matched prID and gtID are the same). FPAs are the TPs couples with same prID, but with a different gtID, added to the FPs with same prID. Similarly, FNAs are the TPs with same gtID, but different prID, added to FNs having same gtID. The reader is referred to Fig. 2 in [5] for a visual representation of TPA, FPA and FNA.

AssA, AssPr, AssRe are then obtained by combining TPAs, FPAs and FNAs scores and averaging over TPs, as proposed by Luiten et al. [5]:

$$ \text{AssRe} = \frac{1}{|\text{TP}|} \sum_{c \in \{\text{TP}\}} \frac{|\text{TPA}(c)|}{|\text{TPA}(c)| + |\text{FNA}(c)|} \quad (1) $$

$$ \text{AssPr} = \frac{1}{|\text{TP}|} \sum_{c \in \{\text{TP}\}} \frac{|\text{TPA}(c)|}{|\text{TPA}(c)| + |\text{FPA}(c)|} \quad (2) $$

$$ \text{AssA} = \frac{1}{|\text{TP}|} \sum_{c \in \{\text{TP}\}} \mathcal{A}(c) \quad (3) $$

$$ \text{with } \mathcal{A}(c) = \frac{|\text{TPA}(c)|}{|\text{TPA}(c)| + |\text{FNA}(c)| + |\text{FPA}(c)|} \quad (4) $$

where $|\cdot|$ stands for the cardinal of the considered set.

## 5.3 Adaptation to audio tracking

The previous definitions of AssA, AssPr and AssRe are universal to any tracking system, as long as matching between ground truth and predicted tracks has been performed to obtain as set of TP, FP and FN. As such, the main difference between the initial HOTA metrics and our adaptation lies in the similarity measure used to compute the matches between predictions and ground truths: the Intersection-over-Union (IoU) used in MOT have been replaced by the angular distance for sound source tracking.

## 6. EXPERIMENTS

We propose experiments to highlight the complementarity of the proposed tracking association metrics with previous identity related metrics. We use two tracking systems on two datasets (LOCATA and LibriJump).

## 6.1 Tracking systems

The tracking systems we use in the experiments reflect the diversity of the aforementioned tracking approaches: a Bayesian particle filter (PF) [7], and a neural tracker (NN) adapted from the works of Subramanian et al. [18].

### 6.1.1 Bayesian tracker

The PF tracker consist in a two-step localization and particle filtering tracker. Localization is performed using a neural network trained to estimate for each frame azimuth and elevation angles on a discrete spherical grid, given FOA input features [19]. The particle filter's track ID management is based on a policy that takes into account the maximum number of simultaneously active sources (equal to the number of speakers in the scene $J$) and the maximum number of track IDs, noted $K_{max}$, which can be unbounded ($K_{max} = \inf$) [7]. Two different hyperparameter sets are used, depending on the dataset (LOCATA or LibriJump) over which tracking is performed.

### 6.1.2 Neural tracker

The NN tracker is a neural network inspired from the Source Splitting approach of Subramanian et al. [18]. It also predicts ordered DoAs on a discrete spherical grid, using the Permutation Invariant Training (PIT) strategy. It is designed with two output branches, which fixes the maximum number of predictable IDs to $K_{max} = 2$.

## 6.2 Datasets and metrics

The dataset used for training the localizer and NN tracker described in the previous section is formed similarly to what is described in Sec. 4.2, except, notably, for the use of separate subsets for SRIRs, noise and speech (LibriSpeech train-clean-100 and train-clean-360), the use of a random SNR (between 5 and 50 dB).

In addition to LibriJump, we use real recordings from Tasks 1 and 3 of the LOCATA evaluation dataset, containing several single speaker scenes, either static (Task 1) or moving (Task 3). In those recordings, all spatial tracks are continuous, similar to tracks 1 and 2 of Fig.1. To simulate spatial discontinuity, we create a subset 3*, by zeroing some voice activity periods on Task 3 recordings.

We focus on identity related metrics, and use TFR and TSR [$s^{-1}$], and the proposed AssPr, AssRe, AssA [%]. For LibriJump dataset, mean scores with 80-20% bootstrap are displayed, and standard deviation was below 1%. For LOCATA dataset, no bootstrap is performed due to the small number of recordings (13 for Task 1, 5 for Task 3).
<figure><img src="image_8.png" /></figure>

<figure><img src="image_9.png" /><figcaption>**Figure 2:** Identity tracking metrics on Task 1 and 3 (single speaker) of LOCATA evaluation dataset, using PF tracker ($K_{max}$ = inf). 3* refers to Task 3 with voice activity periods set to zero to induce discontinuity.</figcaption></figure>

## 6.3 Results

Metric scores using PF tracker on LOCATA (setting $K_{max}$ = inf) and LibriJump ($K_{max}$ = 3, 2J, 4J, inf) datasets are displayed on Fig. 2 and Fig. 3, respectively. Tab. 1 shows NN tracker performance on LibriJump ($K_{max}$ = 2 fixed). We first focus on single speaker scenes (LOCATA and LibriJump-1spk), then analyze identity performance in the more complex multi-speaker case (subsets 2spk and 3spk).

### 6.3.1 Single speaker scenes analysis

We analyze the results using TFR and TSR, then with association metrics to show their complementarity.

Single speaker metric scores using PF tracker can be found on Fig. 2 (LOCATA dataset), solid lines of Figs. 3a and 3b (LibriJump-1spk). The **1spk** line on Tab. 1 gives NN tracker's performances. AssA follows similar evolution to AssRe and is not displayed.

On both Figs. 2a and 3a, TFR is higher than TSR, since it counts the number of broken and swapped tracks, in comparison with TSR which measures only the swaps. For NN tracker, they are both equal to $0.43 s^{-1}$ in the 1spk subset. This indicates the absence of broken tracks, linked to excellent detection performance (broken tracks are linked to the presence of FNs).

AssPr is near 100% for all datasets and trackers. This is expected in the single speaker case with only one ground truth: merging errors are impossible, which leads to high AssPr. Thus, the only possible identity assignment error is the creation of redundant IDs for the same ground

truth, as measured by AssRe.

On LOCATA dataset, we compare identity assignment performances given continuous (Tasks 1 and 3) and discontinuous (Track 3*) tracks. We focus on AssRe and TSR, as AssPr is optimal and TFR is influenced by detection errors and not only association errors. On continuous tracks, PF tracker displays optimal performance in terms of TSR for all recordings of Task 1, and 4 out of 5 recordings of Task 3, showing the ability of PF tracker to maintain a consistent predicted ID when the single speaker track is continuous. Given discontinuous tracks (Task 3*), performance degrades so that only 1 out of 5 recordings remains optimal. Similar conclusions can be drawn from AssRe, since it measures the tendency to attribute new IDs, like *swaps* in TSR. Although, it does provide complementary information due to its computation at the global level. For example, in Task 3*, the worst recording has a TSR of $0.28 s^{-1}$ and AssRe of 47%. The TSR score translates into an ID change occurring approx. every 3 seconds, while the AssRe means that half the ground truth track is split across other predicted tracks. This gives a complementary understanding of PF tracker's behavior.

### 6.3.2 Multiple speaker scenes analysis

We now focus on more complex multi-speaker scenes, namely LibriJump-2spk and 3spk. PF tracker's results on these subsets are plotted on Fig. 3 (dotted lines), and those of NN tracker are on the corresponding lines of Tab. 1.

We first analyze the behavior common to both PF and NN tracker. Considering AssPr and AssRe, when $K_{max}$ = J (2spk line for NN tracker and black markers
<figure><img src="image_10.png" /></figure>

<figure><img src="image_11.png" /><figcaption>**Figure 3:** Mean identity tracking metrics using PF tracker on the three subsets of LibriJump dataset, for different $K_{max}$ values. Black markers in Fig. 3b is for highlighting $K_{max} = J$ scores.</figcaption></figure>

on Fig 3b for PF tracker), we notice that they are approximately equal for both trackers. This highlights the trackers limitations in term of ID assignment in such cases, and their tendency to interchange between same IDs over time. As a result, any counted FPA for a given TP is likely to be counted as a FNA for another TP, which balances AssPr and AssRe. Such behavior is not visible when looking at TSR and TFR, since TSR looks at when an ID change occur without further specifications, and TFR informs about the presence of FNs.

Looking now at NN tracker, TFR and TSR are equal on the 2spk subset, for the same reasons as in the single speaker case. On the 3spk subset, TFR becomes superior to TSR, which underlines the presence of FNs since NN tracker has only has 2 outputs for localizing 3 speakers. Looking at association metrics, AssPr (36.6%) is largely inferior to its corresponding AssRe (47.3%). This occurs for the first and only time and corresponds to the only case where $K_{max} < J$ in our experiments. With a number of possible ID predictions inferior to the number of ground truths, NN tracker is forced to merge multiple ground truths into one prediction, which lowers AssPr.

For PF tracker, similar to the single speaker case, TFR is superior to TSR on both 2spk and 3spk subsets. Also, they worsen when $K_{max}$ augments, showing the increase of swaps as the tracker has more freedom in ID assignment. Interestingly, Fig. 3 shows that AssRe and TSR degrade and AssPr improves when $K_{max}$ augments. As PF tracker has more flexibility in ID assignment, it becomes more likely to initiate new tracks upon discontinuities, rather than maintaining the same ID for longer

periods of time. This results in splitting a track into a multitude of predictions, which lowers AssRe, as well as TSR since each new ID change causes a swap. AssPr, linked to the opposite tendency to merge multiple tracks into a single predicted one, increases.

<table><caption>**Table 1:** Tracking metrics on the three subsets of LibriJump dataset using NN tracker ($K_{max} = 2$).</caption><thead><tr><th rowspan="2">Subset</th><th>TSR</th><th>TFR</th><th rowspan="2">AssA</th><th rowspan="2">AssPr</th><th rowspan="2">AssRe</th></tr><tr><th>[s<sup>-1</sup>] ↓</th><th>[%] ↑</th></tr></thead><tbody><tr><td><b>1spk</b></td><td>0.43</td><td>0.43</td><td>55.4</td><td>100</td><td>55.4</td></tr><tr><td><b>2spk</b></td><td>0.87</td><td>0.87</td><td>38.1</td><td>54.0</td><td>54.2</td></tr><tr><td><b>3spk</b></td><td>0.84</td><td>1.15</td><td>26.6</td><td>36.6</td><td>47.3</td></tr></tbody></table>

## 7. CONCLUSION

In this paper, we presented the problem of tracking intermittent and moving speakers, and the challenges raised movement during silence, in term of spatial discontinuity. We presented LibriJump, a synthetic dataset of acoustic scenes containing speakers with discontinuous spatial tracks, to promote further interest to this seldom explored, yet realistic scenario. We gave an overview of current tracking metrics used in sound source tracking, and proposed to complement them with tracking association metrics from computer vision. We finally analyzed the identity assignment performance of two tracking systems upon continuous and discontinuous tracks, and showed the complementarity of proposed and existing metrics.
<figure><img src="image_12.png" /></figure>

8. REFERENCES

[1] C. Evers, H. Loellmann, H. Mellmann, A. Schmidt, H. Barfuss, P. Naylor, and W. Kellermann, "The LOCATA Challenge: Acoustic Source Localization and Tracking," *IEEE/ACM Trans. on Audio, Speech, and Lang. Proc.*, vol. 28, pp. 1620–1643, 2020.

[2] H. W. Lollmann, C. Evers, A. Schmidt, H. Mellmann, H. Barfuss, P. A. Naylor, and W. Kellermann, "The LOCATA Challenge Data Corpus for Acoustic Source Localization and Tracking," in *2018 IEEE 10th Sensor Arr. and Multichannel Sig. Proc. Workshop (SAM)*, IEEE, 2018, pp. 410–414.

[3] A. Quinlan, M. Kawamoto, Y. Matsusaka, H. Asoh, and F. Asano, "Tracking Intermittently Speaking Multiple Speakers Using a Particle Filter," *EURASIP Jour. on Audio, Speech, and Music Proc.*, vol. 2009, no. 1, pp. 1–11, 2009.

[4] X. Li, Y. Ban, L. Girin, X. Alameda-Pineda, and R. Horaud, "Online Localization and Tracking of Multiple Moving Speakers in Reverberant Environments," *IEEE Jour. Sel. Top. in Sig. Proc.*, vol. 13, no. 1, pp. 88–103, 2019.

[5] J. Luiten, A. Osep, P. Dendorfer, P. Torr, A. Geiger, L. Leal-Taixe, and B. Leibe, "HOTA: A Higher Order Metric for Evaluating Multi-Object Tracking," *Int. Jour. Computer Vision*, vol. 129, no. 2, pp. 548–578, 2021.

[6] P.-A. Grumiaux, S. Kitić, L. Girin, and A. Guérin, "A survey of sound source localization with deep learning methods," *The Jour. the Acoustical Soc. of America*, vol. 152, no. 1, pp. 107–151, 2022.

[7] S. Kitić and A. Guérin, *TRAMP: Tracking by a Real-time AMbisonic-based Particle filter*, 2018.

[8] S. Adavanne, A. Politis, and T. Virtanen, "Differentiable Tracking-Based Training of Deep Learning Sound Source Localizers," in *2021 IEEE Workshop on App. of Sig. Proc. to Audio and Acoustics (WAS-PAA)*, 2021, pp. 211–215.

[9] D. Diaz-Guerra, A. Politis, and T. Virtanen, "Position Tracking of a Varying Number of Sound Sources with Sliding Permutation Invariant Training," in *2023 31st Eur. Sig. Proc. Conf. (EUSIPCO)*, 2023, pp. 251–255.

[10] S. Chen, Y. Wu, Z. Chen, J. Wu, J. Li, T. Yoshioka, C. Wang, S. Liu, and M. Zhou, "Continuous Speech Separation with Conformer," in *2021 IEEE Int. Conf. on Acoustics, Speech Sig. Proc. (ICASSP)*, 2021, pp. 5749–5753.

[11] T. Iatariene, C. Cui, A. Guérin, and R. Serizel, "Speaker Embeddings to Improve Tracking of Intermittent and Moving Speakers," 2025.

[12] B. Yang, C. Quan, Y. Wang, et al., "RealMAN: A Real-Recorded and Annotated Microphone Array Dataset for Dynamic Speech Enhancement and Localization," *Advances in Neural Information Proc. Systems*, vol. 37, 2024.

[13] D. Diaz-Guerra, A. Miguel, and J. R. Beltran, "Robust Sound Source Tracking Using SRP-PHAT and 3D Convolutional Neural Networks," *IEEE/ACM Trans. on Audio, Speech, and Lang. Proc.*, vol. 29, pp. 300–311, 2021.

[14] K. Bernardin and R. Stiefelhagen, "Evaluating Multiple Object Tracking Performance: The CLEAR MOT Metrics," *EURASIP Jour. on Image and Video Proc.*, no. 1, pp. 1–10, 2008.

[15] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, "Librispeech: An ASR corpus based on public domain audio books," in *2015 IEEE Int. Conf. on Acoustics, Speech and Sig. Proc. (ICASSP)*, 2015, pp. 5206–5210.

[16] R. Scheibler, E. Bezzam, and I. Dokmanić, "Pythonacoustics: A Python Package for Audio Room Simulation and Array Proc. Algorithms," in *2018 IEEE Int. Conf. on Acoustics, Speech and Sig. Proc. (ICASSP)*, 2018, pp. 351–355.

[17] F. Font, G. Roma, and X. Serra, "Freesound technical demo," in *Proc. 21st ACM Int. Conf. on Multimedia*, ACM, 2013, pp. 411–412.

[18] A. S. Subramanian, C. Weng, S. Watanabe, M. Yu, and D. Yu, "Deep learning based multi-source localization with source splitting and its effectiveness in multi-talker speech recognition," *Computer Speech & Lang.*, vol. 75, p. 101 360, 2022.

[19] P.-A. Grumiaux, S. Kitic, L. Girin, and A. Guerin, "Improved feature extraction for CRNN-based multiple sound source localization," in *2021 29th Eur. Sig. Proc. Conf. (EUSIPCO)*, 2021, pp. 231–235.