Mixture of Dynamical Variational Autoencoders for Multi-Source Trajectory Modeling and Separation
Authors: Xiaoyu Lin
Laurent Girin
Xavier Alameda-Pineda
Venue: Accepted by TMLR ()

Abstract: In this paper, we propose a latent-variable generative model called mixture of dynamical variational autoencoders (MixDVAE) to model the dynamics of a system composed of multiple moving sources. A DVAE model is pre-trained on a single-source dataset to capture the source dynamics. Then, multiple instances of the pre-trained DVAE model are integrated into a multi-source mixture model with a discrete observation-to-source assignment latent variable. The posterior distributions of both the discrete observation-to-source assignment variable and the continuous DVAE variables representing the sources content/position are estimated using the variational expectation-maximization algorithm, leading to multi-source trajectories estimation.  We illustrate the versatility of the proposed MixDVAE model on two tasks: a computer vision task, namely multi-object tracking, and  an audio processing task, namely single-channel audio source separation. Experimental results show that the proposed method works well on these two tasks, and outperforms several baseline methods.

Published in Transactions on Machine Learning Research (12/2023)
Mixture of Dynamical Variational Autoencoders for Multi-Source Trajectory Modeling and Separation
Xiaoyu Lin
rigoyu.ln@iinri
Inria @ Univ. Grenoble Alpes, LJK, CNRS, France
Laurent Girin
laurent. girin Sgrenoble-inp.f
Unio. Grenoble Alpes, Grenoble-INP, GIPSA-lab, Pronce
Xavier Alameda-Pineda
zovier. auseer..lameedd--piaaaaaaiinnn
Inria Univ. Grenoble Alpes, LJK, CNRS, France
Reviewed on OpenReview: https: // openreview. net/ forum? id= sbkZKBVC31
Abstract
In this paper, we propose a latent-variable generative model called mirture of dynamical variational autoencoders (MixDVAE) to model the dynamics of system composed of mul- tiple moving sources. A DVAE model is pre-trained source dynamics. Then, multiple instances of the pre-trained DVAE model are integrated into a multi-source mixture model with discrete observation-ton toooanttt--oooo source assignment latent variable. The posterior distributions of both the discrete tooevaation-o--surree assignment variable and the continuous DVAE variables representing the sources content/ position are estimated using variational expectationn-maxiiiaaa trajectories estimation. We illustrate the versatility of the proposed MixDVAE model on two tasks: computer vision task, namely multi-object tracking, and audio processing task, namely single-channel audio source separation. Experimental results show that the proposed method works well on these two lasks, and outperforms several baseline methods.
single-source dataset to capture the
content/position
algorithm, leading to multi-source
1 Introduction
1.1 Latent-variable generative models: From GMMs DVAES
Latent-variable generative models (LVGMs)
very general class of probabilistic models that introduce latent (unobserved) variables to model complex distributions over the observed variables. Depending the type of considered latent variables, different LVGM structures can be defined. Mixture models, such as the Gaussian Mixture Model (GMM), are LVGMs with a discrete latent variable (McLachlan & Basford 1988). They are widely used in various pattern recognition and signal processing tasks to model the distribution of signal that can take several different states, each state being encoded by different. value of the latent variable. The (marginal) distribution of the observed data modeled as a lincar combination of compo- nent distributions, which are conditioned the latent variable (in the of GMM, these are Gaussian distributions), each corresponding to different possible state. The mixing coefficients represent the prior distribution of the discrete latent variable and determine the relative weight of each component.
Simple mixture models are appropriate to model 'static' data since they do not consider possible temporal correlations data sequences. More sophisticated LVGMs can be designed to model the dynamics, ie. the temporal dependencies, of soquential data. First, mixture models can be generalized by applying Markov model on the latent variable different time steps, resulting in a hidden Markov model (HMM) (Rabiner & Juang 1986). For example, when applying first-order Markov model, the prior distribution of the latent

---

Published in Transactions on Machine Learning Research (12/2023)
variable at time t depends on the latent variable at time t- via a transition matrix. The mixture component, often called the observation model in this context, is conditioned on the latent variable at time 1. A variety of different conditional distributions can be used. For example, the observation model can be a Gaussian (as in the GMM) or it can be itself a GMM (i.c., one GMM for cach state). The latter HMM-GMM combination has been the state-of-the-art automatic speech recognition for a decade in the pre-deep-learning era (Yu & Deng 2016). Another notable extension of HMMs is the factorial HMMs (Ghahramani & Jordan 1995) which consider a set of parallel factorial discrete latent variables instead of single one. Each latent variable follows Markov model and has its own transition matrix. The conditional distribution of the observed variable at time step t is a Gaussian distribution with the mean being a weighted combination of the latent variables at the same time step, and with a common covariance matrix. Factorial HMMs are suitable for modeling sequential data generaled by the interaction of multiple independent processes.
A discrete latent variable can only be used to model categorical latent generative factors. Tf we consider that the latent state evolves continuously and is better represented by a continuous latent variable, we enter in the world of continuous dynamical models and stade space models (SSMs) (Aoki 2013). The simplest and most dynamical system (LDS), in which the distribution of the latent variable at time step t is Gaussian with the mcan being linear function of the latent variable at time step --1 (Ghahramani & Hinton 1996). If the observation model is also Gaussian with the mean being a linear function of the latent variable at time t, a famous analytical solution for the LDS is the Kalman filter (Kalman 1960). However, the Aiear-Gausssin for real-world signals with complex dynamics. A widely-used generalization of the Kalman filter to sequential data with non-linear dynamics is the extended Kalman filter (Finicke & White 1999 Zarchan 2005), which is a first-order Gaussian approximation to the Kalman filter based on local lincarization using the Taylor series expansion. Another interesting extension is to combine a set of LDSs with an HMM, resulting in the switching state space model (also named switching Kalman filter) (Murphy 1998 Ghahramani & Hinton, 2000). This model segments the sequential data into different regimes, each regime being modeled by an LDS, and the succession of regimes is ruled by the HMM. Recently, deep neural networks (DNNs), and in particular recurrent neural networks (RNNs), have been used within LVGM structures to model sequential data. In this line, the dynamical variational autoencoders (DVAEs) (Girin et al. [202]] are a family of powerful LVGMs that extend the famous variational autocncoder (VAE) (Kingma & Welling) 2014 Rezende et al. 2014) to model complex non-linear lemporal dependencies within a sequence of observed data vectors and corresponding (contimuous) latent vectors. DVAEs have been successfully applied on different types of sequential data such as speech signals (Bie et al. 2021 h and 3D human motion data (Bie et al. 2022a).
commonly used continuous dynamical model is the linear(-Gaussiin
assumption can be a significant limitation
1.2
Contribution: Multi-Source Mixture of DVAEs - Model and solution
All the LVGMs discussed so far have been used to model the distribution of single-source data, the source being either static and have several possible states (in the case of mixture models) or sequential with different types of underlying dynamics (e.g., locally linear). In real-life scenarios, we often encounter situations where a number of soquential source signals appear concurrently in a natural scene for a certain period of time and are observed jointly. Each underlying source can have its own dynamics, and the problem is to obtain an estimation of the content and/or the position along time of each source separately, which includes consistently recovering the identity of each source over time. In short, we want to estimate and separate each source's trajectory from a set of mixed-up observations.
In this paper, we propose to tackle this problem within a deep LVGM probabilistic framework, with a model combining the following two bricks: (a) a deep LVGM for modeling the dynamics of each source indepen- dently; in this work, we propose to use a DVAE to model each individual source. The DVAE-generated random vector represents the source vector, Le. the source content/position that we want to track over time, and the latent random vector reprosents the underlying (continuous) hidden state/factor that governs the source dynamics. (b) A discrete latent assignment variable which assigns each observation in the set of (mixed-up) observations to a source. We name the resulting model as Multi-Source Mixture of DVAEs (MixDVAE). In addition to the MixDVAE model, we propose a multi-source trajectory estimation algo- rithm (i.e., a solution to the MixDVAE model). Importantly, this estimation method does not require a
In the present context, model solution an algorithm for model parameters estimation and latent variables inference.
2

---

Published in Transactions on Machine Learning Research (12/2023)
massive multi-source annotated dataset for model training. Instead, we first pre-train the DVAE model on an unlabeled (synthetic or natural) single-source trajectory dataset, to capture the dynamics of an indi- vidual source type. Afterwards, the pre-trained DVAE is plugged into the MixDVAE model together with the obscrvation-to-soure assignment latent variable to solve the problem for each multi-source test data sequence to process. For each lest. data sequence, the (approximate) posterior distributions of both the observation-to-sourre assignment variable and the source vector of each source are derived using the varia- tional inference methodology more specifically, we propose a variational expectationnmmxxiizzee
(VEM)
algorithm (Jordan et al. 1999 Bishop 2006; Wainwright et al., 2008).
The proposed model and method are versatile in essence. They can be casily adapted and applied to a variety of estimation problems with multiple dynamiical sources with different configurations. For example, if all sources are assumed to have similar dynamics, a single DVAE model can be used to model all sources (more specifically, a different instance of the same DVAE model can be used for each source) and only one pre-training is made on a single single-source datasct. If different types of sourees are considered, with different dynamics, one can use different. instances of the same DVAE model, but pre-trained on different single-source datasets, or one use (different instances of) different DVAE models, also pre-trained on different single-source datascts. In any case, as stated above, there is no need for a massive datasct containing annotated mixtures of simultaneous sources, as would be the case with a fully-supervised approach. Labeled multi-source datasets must be much larger than single-source datasets, since the mixture process intrinsically multiplies the content diversity, and thus can be very costly and difficult to obtain. Therefore, our method can be considered as data-frugal and weakly supervised compared to fully-supervised method. One limitation of the proposed model, though, is that the VEM algorithm applied at test time is relatively costly in computation (this point is invostigated in our study). Another limitation is the fact that all sources are assumed to behave independently. In other words, the proposed MixDVAE does not explicitly model the possible interactions between the different sources. This is planned for future work. We illustrate the versatility of MixDVAE by applying it to two notably different tasks, in computer vision and in audio processing namely multi-object tracking (MOT) and single-channel audio source separation (SC-ASS) and we report corresponding experimental results.
In short, the contributions of this paper arc:
A generic latent-variable generative model called MixDVAE for the separation of mixed observations into independent sources with non-linear dynamics.
A learning and inference (variational EM) algorithm associated with MixDVAE, derived from the corresponding variational lower bound.
A set of experiments demonstrating the interest of MixDVAE for various tasks and using different
types of data.
MixDVAE is data-frugal and weakly supervised since it does not require a massive labeled multi- source dataset at training time, but only one or several single-source dataset of much moderate size.
1.3 Application to multi-object tracking
In multi-object tracking (MOT) (also called multi-target target tracking (MTT) depending on the context, scientific community and applications), a number of
appear simultancously in the same visual scene and each of them has its own moving trajectory, e.g., pedestrians/vehicles in videos or aircrafts in radar scans. The problem is to estimate the position of each object/targe at every time step, and track it along time by assigning a unique identity to it (Vo et al. 2015 Ciaparrone et al. 2020). In the popular 'tracking- by-detection' configuration, a set of detection bounding boxes (DBBs) are given at each time step by a front-end detection algorithmn, each of them potentially corresponding to one of the largets. These DBBs are then used as the observations. In Section [] we apply the MixDVAE model to the MOT problem in this configuration (i.c., using the set of DBBs as observations). We do that in a simplified scenario where the number of objects is assumed known and constant during the observation measurements. A complete and fully-operational MOT system would require to include module managing the 'birth' and 'death' of target
3

---

Published in Transactions on Machine Learning Research (12/2023)
tracks (i..,, objects disappearing of the scene and new objects appearing in the scene). We do not address this problem since the purpose of this work is not to propose a fully-operational MOT system but rather to focus on the problem of multi-source dynamics modeling with DVAEs. It must be noted however that even if we assume that the actual number of objects present in the scone is known and does not vary across the modeled sequence, at any time step I, it is not necessarily equal to the number of DBBs, since occlusions (leading to missod detections) may occur. We will see our reported experiments that MixDVAE is able to deal with these difficulties.
Related work in MOT /MTT. In the MOT/MTI literature, few works have considered the detection assignment problem (which is also called the data association problem) and the target dynamics modeling problem jointly in a unified probabilistic framework. In fact, the data association problem is usually solved by designing a complicated association algorithm (such as the joint probabilistic data association filter (JPDAF) or the multiple hypothesis tracking (MHT) algorithm) and the target trajectorics estimation is obtained by applying a post-processing track filtering algorithm (such as the Kalman filter, the extended Kalman filter, or particle filter) separately on each estimated track (Vo et al. 2015). Recent MOT algorithms use DNNs such as RNNs or convolutional neural networks (CNNs) to extract several features (e.g., visual features, motion features, targets interaction fcatures) from the videos and use these features for data association [Luo et al. 2021 Ciaparrone et al. 2020). The motion models are generally used for track refinement. The work that is the closest to ours is that of Ban et al. (2021), p, who proposed a unified probabilistic framework for audio-visual multi-speaker tracking. However, the dynamical model that was used in their work is a simple dynamical model. In this paper, we use DVAEs to model non- con-lincar objects dynamics.
1.4 Application to audio source separation
The second problem on which we apply the proposed MixDVAE model is the single-channel audio source separation (SC-ASS) task. Here, the recorded (observed) signal unique (digital) waveform that is assumed to result from the physical summation of individual source waveforms, such as several speakers speaking simultancously or several musical instruments playing together. The goal is to estimate the different source signals composing the mixture (Vincent et al.) 2011). At first sight, this scenario is not an appropriate configuration for MixDVAE because, for each discretized time, we do not have a set of observed samples to assign to one of the source signals. However, a widely-used approach in SC-ASS is to work in the time- frequency (TF) domain, most often using the short-time Fourier transform (STFT), and exploit the sparsity of audio signals in the TF domain (Yilmaz& & Rickard 2004). This means that at cach TF bin of the STFT, the observed mixture signal is assumed to be composed of one dominant source (with a power that is much larger than that of the other sources), so that the observation at that TF bin can be (totally or mainly) attributed to that dominant source. This can be done using an assignment variable, which is often referred to as a TF mask in the SC-ASS literature (Wang & Chen 2018). Therefore, we can apply the proposed MixDVAE model by combining this assignment variable with a set of DVAEs modeling the dynamics of the audio sources in the TF domain. The main difference compared to the MOT problem is that we have to consider the frequency dimension in addition to time, and at a given TF bin, we have here only one single observation to assign to a source, instead of a set of observations. We apply this principle and illustrate the use of MixDVAF for SC-ASS in Section G It can be noted that the dynamics of different types of audio source signals (speech, musical instruments, noises, etc.) can be very different. So, this is a typical use-case where we can pre-train different DVAE models on different single-source datascts to capture the dynamics of different types of source.
Related work in SC-ASS. State-of-the-art SC-ASS methods are based on the use of huge and sophisticated DNNs that directly map the mixture signal to the individual source signals to the TF masks (Chandna et al. 2017 Wang & Chen 2018). These methods obtain impressive separation performance, but they require to adopt fully-supervised approach using huge parallel datascts (that contain both the mixture and the aligned separate source signals). This contrasts with the spirit of MixDVAE which, again, can be considered as weakly supervised, and does not require a huge parallel dataset, but only a reasonable-size single-source datasct for any type of source to separate. Weakly-supervised (or at the extreme unsupervisod) methods for audio source separation are still quite rare and their development largely open topic. We can find some connection with the SC-ASS method of Ozerov et al. (2009) based on factorial HMMs. A
4

---

Published in Transactions on Machine Learning Research (12/2023)
Wase-based unsupervised speech enhancement method was recently presented by (Bie et al. 2022b but unique DVAE instance was used to model the speech signal and the noise (power spectrogram) was modeled with a nonnegative matrix factorization (NMF) model. To our knowledge, the present work is the first time a mixture of DVAEs is used for SC-ASS (and for audio processing in general).
1.5 Organization of the paper
In Section 2 we present the general methodological background for developing the MixDVAE model, in- cluding the variational inference principle and the DVAFs. In Section E we present the MixDVAF model and the general principle we used to derive its solution. The solution itself, Le. the MixDVAE algorithm, is presented in Section | Soction 3 and Section G illustrate the application of MixDVAE to the MOT and SC-ASS problems, respectively, including experiments. Section 7 concluuee the paper.
2 Methodological background
2.1 Latent-variable generative models and variational inference
An LVGM depicts the relationship between an observed o and a latent random (vector) variable from which o is assumed to be generated. We consider an LVGM defined via a parametric joint probability distribution po(o, h), where 0 denotes the set of parameters. closely related to each other: (i) estimate the parameters 0 that maximize the observed data (marginal) likelihood po),), and (ii) derive the posterior distribution po (hlo) to infer the latent variable h from the observation o.
a general manner, we are interested in two problems
A prominent tool to estimate 0 is the family of expectation-mmxxiissttie (EM) algorithms (Bishop) 2006; McLachlan & Krishnan 2007), that maximizes the following lower bound of the marginal likelihood, called the evidence lower-bound (ELBO):
0)) gb o) po h) log ghlo)) s log Pog),
(1)
where (h)o) is a distribution on h conditioned on o. The EM algorithm is an iterative alternate optimisation procedure that maximizes the ELBO w.r.t. the distribution (E-step) and the parameters 0 (M-step). Maximizing the ELBO w.r.l. is equivalent to minimising the Kullback-Leibler divergence (KLD) between (hlo)) and the exact posterior distribution po(l))) (Bishop) 2006 McLachlan & Krishnan 2007]. When po (olo)) is computationally tractable, is optimal choose g(( = (h)h)) then the ELBO is tight and the EM is called "exact." Otherwise, the optimization w.r.l. q is constrained within a given family of computationally tractable distributions and the bound is not tight anymore. We then have to step in the variational inference (VI) framework (Jordan et al. 1999 Wainwright et al. 2008).
A first family of VI approaches is the structured mean field method (Parisi & Shankar 1988 which consists in splitting h into a set of disjoint variables = (h,... (h,...,,y h,). The approximate posterior distribution g is thus assumed to factorize over this set, i.e. g(hlo) = I gi (h,lo)) which leads to the following optimal factor, given the parameters computed at the previous M-step, gold;
mean-field
q (h,lo) x exp
E (IT,,
(hylo))) log Pools (o. h)))
(2)
Since each factor is expressed as a function of the others, this formula itcratively applied in the E-stop of the EM algorithm until some convergence criterion met, leading to the VEM family of algorithms (Bishop) 2006).
A sccond approach is to rely on amortized inference (Hoffman et al. 2013 where set of shared parameters is used to compute the parameters of the approximate posterior distribution. A very well-known example is the variational autoencoder (VAE) (Kingma & Welling 2014 Rezende et al. 2014). For a reason that will become clear Section 3.1 let us here denote by the observed variable and by z the latent onc. In a VAE, the joint distribution po(s, z) po(s)z)p(z) defined via the prior distribution on z, generally chosen the standard Gaussian distribution p(z) AZ2z 0, I), and via the conditional distribution
5

---

Published in Transactions on Machine Learning Research (12/2023)
generally chosen as Gaussian with diagonal covariance matrix po(s)z) (sss: (z)) diag( (o (z))) The mean and variance vectors pz( and (z()) are nonlinear functions of 2 provided by DNN, called the decoder network, taking z as input. 0 is here the (amortized) set of parameters of the DNN. The posterior distribution (z corresponding to this model does not have an analytical expression and it is approximated by a Gaussian distribution with diagonal covariance matrix 1s (z)s) = N NN t&g (s), (s))( where the mean and variance vectors (s)(s) and (s(s) are non-linear functions of implemented by another DNN called the encoder network and parameterized by d. The ELBO is here given by:
C(0, d) = (xs))ss) [log po(s, log 90 (z|s)]
(3)
In practice, the ELBO is jointly optimized w.r.t. 0 and á on training dataset using combination of stochastic gradient descent (SGD) and sampling (Kingma & Welling 2014). This is in contrast with the EM algorithm where and are optimized alternatively.
2.2 From VAE to DVAE
In the VAE, cach observed data vector s is considered independently of the other data vectors. The dynamical variational autoencoders (DVAEs) are a class of models that exlend and generalize the VAE to model sequences of data vectors correlated in time (Girin et al. 2021). Roughly speaking, DVAE models combine a VAE with temporal models such as RNNs and/or SSMs.
Let $1.7- = [s: and 21:T { be a discrete-time sequence of observed and latent vectors, respectively, and let P. ( 1 21:--1 denote the set of past observed and latent vectors at time t. Using the chain rule, the most general DVAE generative distribution can be written as the following causal generative process:
T
Po ZI.T) = II 1=1
po. (s: 2,
(4)
where Po, (s: (s. z:) and po, (2 (2, pt) are arbitrary generative distributions, which parameters are provided sequentially by RNNs taking the respective conditioning variables as inputs. A common choice is to use Gaussian distributions with diagonal covariance matrices:
pa, (st Pr. z() (ssi me. (P:, zz), diag vo, (pr, z.)),
(5)
Pa_ (z (p:) = N (zri nea (p:), di (p:))).
(6)
It can be noted that the distribution of 2, is more complex than the standard Gaussian used in the vanilla VAE. Also, the different models belonging to the DVAE class differ in the possible conditional independence assumptions that can be made in (4)
Similarly to the VAE, the exact posterior distribution po (0.17 (s:T) corresponding to the DVAE generative model not analytically tractable. Again, an inference model qo, (Z1:T ($1:) is defined to approximate the exact postcrior distribution. This inference model factorises as:
go (ZIT(SLT))
II 96. (ze (:),
where q: (1rr denotes the set of past latent variables and all observations. Again, the Gaussian distribution with diagonal covariance matrix is generally used:
go. (2.(2z/)) )q)) = NZz Ps. (qr).. diag(vs.(q)))
(8)
where the mean and variance vectors are provided by an RNN (the encoder network) taking q. as input and parameterized by ó. With the most general generative model defined in (4), the conditional distribution in cannot be simplified. However, if conditional independence assumptions are made (4). the dependencies
Ce (z )q) can be simplified using the D-separation method (Bishop 2006 Geiger et al. 1990). see (Girin)
al. 2021 for details. In addition, we force the inference model be causal by replacing $1.7 with

---

Published in Transactions on Machine Learning Research (12/2023)
Table 1: Summary of the variable notations.
Variable notation
Definition
T, E {I,..... T
Sequence length and frame index Total number of sources and source index Number of observations at i, and obs. index True /content of source n at time t Latent variable of source n at time L Observation at time Assignment variable of observation it time Source vector sequence for source n Set of all source vectors at time t Set of all source vectors Analogous for the latent variable Set of all observations Set of all assignment variables
N, E {1, ,I,...,,N N
K2, h E (1, Ki}
Stn E RRS
Zen E RL
Ock E RO
Wik E (I....,N}}
Sn S1.T.
St.: S1,1:N
S1.T,I:N
m. Zt,: Z
0 = 01.T.1:K
W = WL.T.I.K.
S1.t in q.. This is particularly suitable for on-line processing. In the rest of the paper, we will use the causal inference model, i.c. q. (11 in and (8).
Similar to the VAF, and following the general VI principle, the DVAE model is also trained by maximizing the ELBO with a combination of SGD and sampling, the sampling being here recursive. The ELBO has here the following general form (Girin et al. 2021
C(Os, Oz S1.T) = E 4o (31))) log ZIrT) - log 9o, (21:((11T)]
(9)
3 MixDVAE model
3.1 Problem formulation and notations
Let us consider sequence containing N sources or targets that observe over time. Let n (1, N} denote the source index and let Stn E S be here the true (unknown) n-th source vector at time frame t. At every time frame t, we gather K2 observations, and this number can vary over time. We denote by Osk E RO, k (I, Kr), the k-th observation at frame The problem lackled in this paper consists estimating the sequence hidden source vectors S1:T,x (stm) for each source n, from the complete set of obscrvations O1.T,1KK = {ox*) Je
To solve this problem, we define two additional sets of latent variables. First, for each source n at time frame t, we define latent variable Ztn R associated with Sta through a DVAE model. This DVAE model, which might be identical for all sources not, is used to model the dynamics of each individual source and is plugged into the proposed probabilistic MixDVAE model. Second, for each observation Otk, define a discrete observation-to-souie that observation k at time frame is assigned to to/was was generated by source n. This results in per-source
assignment variable Wis taking its value in {1,..., N). Wix means
sequences of assigned observations.
Hereinafter, to simplify the notations, we use 4." as a shortcut subscript for the set of values of the corresponding index. For example, S.x S1.T, a is the complete trajectory of source n and st.: St,1:N the set of all source vectors at time frame t. All notations summarized in Table
3.2 General principle of the propased model and solution
The general methodology of MixDVAE is to define parametric joint distribution of all variables (o, S, z, w), then estimate its parameters and (an approximation of) the corresponding posterior distribu- tion (s, 2, wlo), from which deduce estimate of s for each source n. The proposed MixDVAE

---

Published in Transactions on Machine Learning Research (12/2023)
Z
7
1
S
1
S
W 1
WE
W W
w
0
O++1
Folded graphical model
Extended graphical model time frames
Figure 1: Graphical representation of the proposed MixDVAE model.
generalive model po(o. S, 2, w) is presented in Section 3.3 As briefly stated above, it integrates the DVAE generative model (4) () for modeling the sources dynamics and does not use any auman-annotater data for training. As is usually the case in (D)VAE-based generative models, both the exact posterior distribution po(s, z, w) and the marginalization of the joint distribution po(o, 8, 2, w) w.r.t. the latent variables are analytically intractable. Therefore we cannot directly use an exact EM algorithm and we resort to VI. We propose the following strategy, inspired by the structured mean-field method that we summarized in Sec- tion 2.1 with being here equal (s, z, w). In Section 3.4 we define an approximate posterior distribution (o z, w/o) that partially factorizes over (s, w). Just. like the proposed MixDVAF generative model includes the DVAE generative model, the approximate posterior distribution includes the DVAE inference model one of the factors. This factorization makes possible the derivation of a model solution in the form of a VEM algorithm, as detailed in Section
3.3 Generative model
Let us now specify the joint distribution of observed and latent variables po(o, w, s, z). We assume that the observation variable only depends and s, while the assignment, variable of the other variables. The graphical representation of MixDVAE is shown in Figure 1 Applying the chain rule and these conditional dependency assumptions, the joint distribution can be factorised as follows:
priori independent
(o w, pa_ (o w, s)pe (w)pe (s, 2).
(10)
Observation model. We assume that the observations are conditionally independent through time and independent of each other, that is to say, at any time frame 4, the observation Ozk only depends on its corresponding assignment wi and source vector at the same time frame. The observation model po., (o w) can thus factorised as
pe. (o)w,s)
II I Pe. (o Wiks St.))
(11)
[=1==
Given the value of the assignment variable, the distribution p(ork) Wen, st.)) is modeled by a Gaussian distri- bution:
Pe. (Orz) Wer n., n,, No S LIx).
(12)
This equation models only the observation noise via the covariance P E OxO and thus assumes that the assigned observation lies close the true source vector
St.; and not Sta since the value of wys is not specified. For simplicity of presentation, state the in which the observation and vector dimensions the same, general case where O S, consider the of projection matrix Pş Boxs and define
2In this equation,
L.. 0 S. In Pa. (or /wik = n,,t)) (ooo PySin, P1a)- Again, simplicity, consider P in the rest the paper. All derivations and results generalizable Pt #

---

Published in Transactions on Machine Learning Research (12/2023)
Assignment model. Similarly, assume that, priori, the assignment variables are independent across time and observations:
Pow (w) IIII Paw (Wex).
(13)
rm k=1
For each time frame t and each observation k, the assignment variable Wie is assumed to follow a uniform prior distribution:
pow (wek)
(14)
N
Dynamical model. Finally, Paae (8, z) modeled with a DVAE. The different sources are assumndd to be independent of each other. This implies that in the present work we do not consider possible interactions among sources. More complex dynamical models including source interaction are beyond the scope of this paper. With this assumption, the joint distribution of all source vectors and corresponding latent variable pa,e (s can be factorized across sources as:
po (s, 2) II po (S., 2i,n);
(15)
where 20,, (8. 2:n is the DVAE model defined in 6 and applied to 8., and 2 (defining Pt.x
1m 1m )nn }) As mentioned before, the DVAE model can be either the same architecture for all
sources, pre-trained on unique single-source dataset, the same architecture but pre-trained on different single-source datasets for different sources, or completely different architectures for each source.
Overall, the parameters in the generative model to be estimated are =0o { ) 0g 02) (note that Ow = 0).
3.4 Inference model
The exact posterior distribution corresponding to the MixDVAE generative model described in Section 3.3 is neither analytically nor computationally tractable. Therefore, we propose the following factorized approx- imation that leads to computationally tractable inference model:
(s, z, w/o) yow (w/o) (s)))
(16)
where qd. (z|s) corresponds the inference model of the DVAE and the optimal distributions 4a, (s|o) and gow (w]o) are derived below in the E-steps of the MixDVAE algorithm. The factorization [16 inspired by the structured mean-field method PParii & Shankar 1988 since break the posterior dependency between and {s,z), z}. However, we keep the dependency between and at inference time since it is the essence of the DVAF. In addition, we assume that the posterior distribution of the DVAE latent variable is independent for each source, so that we have:
to (z)S) = II
Jos (Z.,n (s.n): ==1
(17)
where lo. (z..))..n) is given by 7 and g applied to {z.,n, ). This is coherent with the generative model, where we assumed that the dynamics of the various sources are independent of each other.
4 MixDVAE solution: A variational expectation--maiiiiatooo algorithm
Let now present the proposed algorithm for jointly deriving the terms of the inference model (other than the DVAE terms) and estimating the parameters of the complete MixDVAE model, based on the maximization of the corresponding ELBO. The inference done directly on each multi-source test sequence
4 Here denote the DVAF, parameters by do instead of 8 differentiate the DVAF parameters from the other parameters.

---

Published in Transactions on Machine Learning Research (12/2023)
to process and does not require previous supervised training with labeled multi-source dataset. It only requires to pre-train the DVAE model on synthetic or natural single-source sequences.
As discussed in Section 2 in many generative models, the optimization of the ELBO is done either following the structured mean-field method (2) or using amortized inference as in (D)VAEs. In our case, we cannot directly use the generic structured mean-field inference procedure, since the proposed approximation (16) does not factorize completely in a set of disjoint latent variables (e.g., 9o, (z|s) is conditioned on s). Al- ternatively, one could resort to purely amortized inference and conceive deep encoder that approximates the distributions in 166, leading to looser approximation bound. We propose strategy that is middle ground between these two worlds. We use the structured mcan-field principles that provide a tighter bound since they do not impose a distribution lamily for dew and J0, and we use the philosophy of amortized inference for qo, so as to exploit the pre-trained DVAE encoder.
To do so, we have to go back to the fundamentals of VI and iteratively maximize the MixDVAE model ELBO defined by:
[(O, o; = E... (/o pe (o (,, w) log (s w/o)].
(18)
By injecting 10 and
16 into 18 we can develop L(A, di o) follows:
C(0, Φ: 0) E Ego (wlo)as (s|a)
log Po, (o/w. s)] + E now (wla)
Pow Pa (w) llog gdw (wlo)]
+ Bosr (slo) Bosx (x1s) [log lo.g po.. (s, z) - log 9o (z s)]] E (8)0) log qs (s|o)].
(19)
The ELBO maximization is done by alternatively and iteratively maximizing the different terms correspond- ing to the various posterior and generative distributions. In case, obtain series of variational E and M steps. While the E steps associated to tow and 9s. follow the structured mean-field principle, the E step associated to go is based on the principle of amortized inference commonly used in (A).
4.1 E-S step
We first consider the computation of the optimal posterior distribution 9a. (s|o). To this aim, we first select the terms 199 that depend on 8, the other terms being here considered as a constant:
(0) = (slo) Byow (w]o) log Po. (o/ s)] + E (x|s) logg po.. (s, 2) - logg Toz (2|s)] - log Jow (810)
(20)
Let us define:
p(s|o) = exp (Bgon (w/a) lo. (o/w, + (x( log pe.. (s,z) log 9d. (z|s)))
Cxe
(21)
where C" > is the appropriate normalisation constant.
20 rewrites:
Ca(0, 0) -Dr, (s|o) || p(slo)) + +C,
(22)
where (--(-I) denotes the Kullback-Leible divergence (KLD). Therefore, the optimal distribution is the one minimising the above KLD:
lo. (s/o) = p(s]o) exp Eggo o) po, (o/w, w.wws) s)] + Eye, (m|s) [oo po... (s,z) log qo, (z]s))
(23)
Since for any pair (t, k), the assignment variable win follows discrete posterior distribution, we can denote the corresponding probability values by Vetkn yow (Wex n (ox))) These values will be computed in the E-W step below. The expectation with respect to quw (wlo) in (23) can be calculated using these values. However, the expectation with respect to Zo, (z)s) cannot be calculated in closed form. As usually done the VAE methodology, is thus replaced by Monte Carlo estimate using sampled sequences drawn from the DVAF inference model at the previous iteration (see Section 4.5). Replacing the distributions in
10

---

Published in Transactions on Machine Learning Research (12/2023)
23
with 11
15 and 17 and calculating the expectations with respect to qew (w/o) and g0. (z|s),
find that Tá. (s|o) lactorizes with respect to as follows:
ao (s|o)
II n=1
qd, (s.,
(24)
Each of these factors corresponds to the posterior distribution of the n-th source vector. Given (23 and the DVAE generative and inference models, dependencies w.r.t. the previous and current DVAE latent variables 2itt and the previous source vectors $14-1,n- These non-linear dependencies impede to obtain an efficient closed-form solution. We resort to point sample estimates obtained using samples of &iæ, and of 1,n, at the current iteration, denoted Ziar, and 1.r- Using these samples, the posterior distribution is approximated with (details can be found in Appendix A.1
that at given time t, the distribution over Stn has non-linear
(8) where each term of the product is shown to be Gaussian:
do n 0)
(0) II qo (Stn 11 Lit,n 0),
(25)
qs. Stt i lat,a o) Ns min: Vin),
(8)
(26)
with covariance matrix and mean vector given by:
K
+ diag ( -1
Vin
Σ
Trkn & Lk
(27)
m Vtn
Tekn 4 t Osk + diag(vo 1 b
(2)
(28)
where tn and agtt are simplified notations for ve, ($14-1.n 1i.tn) and ne, ( ii,,)), respectively denoting the variance and mean vector provided by the DVAE decoder network for source at time frame t. As we have to sample both S.n. and Z:,n: we need to pay attention to the sampling order. This will be discussed in detail in Section 4.5 Importantly, in practice, mWn is used as the estimate of Stn.
(
(
Eq. 28 shows that the estimated n-th source vector obtained by combining the observations Ou and (t)
the mean source vector Ma,.on predicted by the DVAE generative model. The balance between these two terms depends on the assignment variable Mtkn: the observation model covariance matrix 4u and the source variance predicted by the DVAE generative model Vg,, Ideally, the model should be able to appropriately
(2)
balance these two terms as to optimally exploit both the observations and the DVAE predictions.
4.2
E-Z step
In the E-Z step, we consider the DVAE inference model qo, (zls), defined by 17 and In the corresponding term is the third one, which denote by Lu (fs, O2 Φz; o) and which factorizes sources as follows (see Appendix A.2
Cz(0s, 02 Σ La Oz:
(29)
with
Cz, (0,, 02, 02 Bys (e.,. 10) Eggn (z... le..)) log pen. (S., m) Egg, n... log you (z.,n [s..
(30)
Inside the expectation Bys bojlh we recognize the DVAE ELBO defined in 9 and applied source This suggests the following strategy. Previously to and independently of the MixDVAE algorithm,
11

---

Published in Transactions on Machine Learning Research (12/2023)
pre-train the DVAE model on dataset of synthetic or natural unlabeled single-source sequences (this is detailed in Sections 5.1 and 6.1p. This is done only once, and the resulting DVAE is then plugged into the MixDVAE algorithm to process multi-source sequences. This provides the E-Z step with very good initial values of the DVAE paramoters fg. 8₂ and Φz. As for the following of the E-Z step, the expectation over 300 is not analytically tractable. A Monte Carlo estimate is thus used instead, using samples of both z and s, similarly to what was done in the F-S step. Finally, SGD used maximize (the Monte Carlo estimate of) C2(Os, Oz, (2; 0); jointly updating 0s, O2 and Φz; that is, we fine-tune the DVAE model within the MixDVAE algorithm, using the observations Note that in experiments, also consider the case where we neutralize the fine-tuning, i.e. remove the F-Z step and use the DVAE model as provided by the pre-training phase.
4o (s.n (o)
4.3 E-W step
Thanks to the separation of w from the two other latent variables in (16), the posterior distribution gow (w/ can be calculated closed form by directly applying the optimal structured mean-field update equation (2)
to our model. It can be shown that this equivalent to maximizing (19) w.r.t. qow (wlo). We obtain (see
A Appendix A for details):
T K. W O/ x
Now (wek O0;
(31)
(=1
with
Bekn
9dw (wik n/o) -]tkn
(32)
Σ Beni
where
1 TT V..))
Bukn Noouk mmn, P(k) exp
(33)
2
The parameters Man and Vm in the above equation have been defined in 28 and 27 respectively,
4.4 M step
As discussed in Section 2 the maximization step generally consists estimating the parameters of the generative model by maximizing the ELBO over f. We recall that 0 = {6. ( 02). In this work, the parameters of the DVAE decoder 0g and 0g are first estimated (offline) during the pre-training of the DVAE and then fine-tuned in the E-Z step in amortized way, all this jointly with the parameters of the encoder dh Therefore, in the M-step, we only need to estimate the observation model covariance T.K, matrices 0, = P jt,=1- In (19,, only the first term depends on lo Setting its derivative with respect to Pik to zero, we obtain (sce Appendix A for details):
T.K.
Lk: Σ Mekn (oo Mo m V.n).
(34)
In practice, is difficult to obtain a reliable estimation using only a single observation. We address this issue in Sections 5.2 and 6.2
4.5 MixDVAE complete algorithm
As already mentioned in Section 4.1 must pay attention to the sampling order of s and when running
the iterations of the E-S and F-Z steps. As indicated in the pseudo-code of Algorithm 1 in practice, the
(0) E-S and E-Z steps are processed jointly. We start with the initial source vectors sequence S1.,I::N and initial
(0) mean source vectors sequence m mI At any iteration i of the E-Z and E-S steps, for each source n and
each time frame sample in the following order:
12

---

Published in Transactions on Machine Learning Research (12/2023)
Algorithm MixDVAE algorithm
Input:
Observation vectors o 01:7,1:K,i
Output:
Jen= (the estimated n-th source veclor al time frame / is m Values of the assignment variable (1) T.N.K.
Parameters of dd (s) {m V
(2)
ta
1: Initialization
2: See Sections 5.2 and 6.2
3: for i to do
E-W Step
for to N do
for + to T do for k: + to Kg do
7:
8:
Compute Mekn using 32)
and (33);
9:
end for end for end for
10:
11:
12:
E-Z and E-S Step
13:
for +- to N do
14:
for t + to T do
15:
Encoder;
Compute n on an with input (i-1) S1.T, and (
(1)
16:
Sample zin from as (zm)))()))) 1,n)1,n) N (Zzni H Mo diag( (6) (2 (n))
(0)
(i-1) (0)
(1)
17:
18:
Decoder;
Compute (6)
Po and (1)
on with input $1t-11n and (t)
(6)
19:
Zia-1nn
20:
Compute Mga.tn and Ug with input s 1,m
(i)
(i)
(4)
and Zist.ni
(i)
21:
E-S update;
22:
Compute min in using 28 and 27
23:
Sample Stn. from (Smi m )m
(6)
24:
end for
25:
E-Z update;
26:
Compute (0,, Oz, dzi using 35
27:
end for
Compute E(0g, Oz 0z = EN (Os, Onr ozi
28:
Fine-tune the DVAE parameters (0x, On, da) by applying SGD Z(0,, Oz, Oxi 0);
29:
30:
M Step
31:
Compute using 34 following Sections 5.2 and 6.2
32:
end for
1. Compute the parameters p tn and v
dr of the posterior distribution of z, using the DVAE encoder (6) network with inputs $ sampled at the previous iteration and Zi sampled the current iteration.
(6)
(2)
Then, sample 2 from go, (Zmn (i..s I.n)
(2)
2.
Compute the parameters M and V of the generative distribution of 2, using the corresponding
(6)
DVAE decoder network with inputs S1.t-1,- 1. and z (i)
(8)
1, both sampled at the current iteration.
3. Compute the parameters Ma,t and ug of the generative distribution of s, using the corresponding (5)
(5)
DVAE decoder network with inputs $1t-1,0 and zit. both sampled at the current iteration. Compute
(6)
(0) to and idgg are shortcuts for Ho, (x(1)) (t) (1)11))) and vos (19-1) (t) 2110-1..)) respectively.
6 AAnalooous definitions hold.
13

---

Published in Transactions on Machine Learning Research (12/2023)
E-W Step
Assignment variable the
Update Φ #
M Step
Observations at frame 0,1:K
MixDVAE algorithm
Estimated source
mean vector and
E-Z Step
Update Bg 0
E-S Step
covariance matrix m ( VO
Update dx
Source vectors estimated itcration 1-1 (G-1) 1:7.1:M
LEN A
Φ (
DVAE predicted mean and variance vectors
Mg ( (((1))
1=-1,12 (rr Va (s 12- (1.12))
T frumes
DVAE model
Offline pre-training
7 frames
frames
Overview of the proposed MixDVAE algorithm at given time frame t. The DVAE model is pretrained offline using a (synthetic or natural) single-source dataset. It takes as input the sequence of source vectors, encodes them into a sequence of latent vectors, which are then decoded into the reconstructed sequence of source vectors. For given time frame I, the MixDVAE algorithm takes as input the observations time t as well as the mean and variance vectors estimated by the DVAE model. By iterating the E-S, E-Z, E- W and M steps, we obtain estimates of the assignment variable and of each source vector.
Figure 2:
( the parameters m and ((6 in of the posterior distribution of with 27 and 28 and sample Six from
(+)
it.
Note that with the above sampling order, the Monte Carlo estimate of the ELBO term maximized in the E-Z step (30) is given by (for source n):
T
En,n Oz O2i
Σ log po, (5 111-1111110 18 lit,n
(i-1) (8) DKa. (yon (2tn (1:T (i:t-1,))llo soalsff21111
(elellll22aa (Zen 11
211--1n))
(8)
(35)
The whole MixDVAE algorithm, taking into account these practical aspects, is summarized the form of pseudo-code in Algorithm 1 In addition, Figure shows a schematic overview of the algorithm.
4.6 Choice the DVAE model
We recall that the DVAE is a general class of models that differ by adopting different conditional independence assumptions for the generative distributions in the hid of (4). In Girin al. (2021), seven DVAE the
models from the literature have been extensively discussed, and six of them have been benchmarked analyeis-essyntheii task (on speech signals and 3D human motion data). We chose to use here the stochastic recurrent neural network (SRNN) model initially proposed in Fraccaro et al. (2016), because it was shown in Girin et al. (021] to provide very good trade-off between model complexity and modeling power. The probabilistic dependencics of the SRNN generative model are definod as follows:
Pe., (SITT.ZTT)
II pe. (s. (z). 2-11. 1=1
(36)
To perform online estimation, we use the following causal SRNN inference model:
T
go (ZLT//1)))) = II
go, (zr (1,1, Zz-1).
(37)
1=1
As illustrated in Sections 5.2 and 6.2 in practice, we choose different VEM step orders. Here we present the algorithm
with the order E-S/F-Z Step. E.W Step and M Step.
14

---

Published in Transactions on Machine Learning Research (12/2023)
The implementation details of the SRNN model can be found in Appendix D
5 Application of MixDVAE to multiple object tracking
configuration, the objective of the MOT task is to estimate the trajectories of moving objects from set of given DBBs. In this case, the source vector Stn represents the position of object n at time frame I, which is given in practice by the coordinates of the (top- left and bottom-right points of the) "true" corresponding bounding box, i.e. Stn = (sin sin in stn) E IR., The observation vector Otk (o'x Oe oik o(k) E R contains the coordinates of the (top-left and bottom- right points of the) k-th DBB at frame 1. In VAE or DVAE, the dimension I. of the latent vector Zin is usually smaller than the dimension of the observed vector, in order to obtain compact data representation. Since in the MOT task the data dimension is already small (O = S 4), we also set L = 4. The sequence of estimated source position vectors is given directly by (28) for n 1 to N and t = 1 to T, directly forming source trajectories, with no further post-processing.
As mentioned in Section 1.3 under the baakknggby-detoctiing
5.1 DVAE pre-training
Dataset. We consider pedestrian tracking for the MOT task and assume that all the moving sources have similar dynamical patterns. We thus pre-train a single DVAE model on a synthetic single-source trajectory dataset. This dataset contains synthetic bounding box trajectories in the form of T'frame sequences (1' 60) of 4D vectors {(x], xT, wf; =;))²=-- These trajectories are generated using piece-wise combinations of several elementary functions, namely: static a(t) do. constant velocity a(t) = ait + do, constant acceleration ao, and sinusoidal (allowing for circular trajectories) a(t) = a so) The parameters 01, 02 w, and do are sampled from some pre-defined distributions, whose parameters are estimated from the detections on the training subset of the MOT17 dataset (Dendorfer et al., 2021 b. which is a widely-used pedestrian tracking dataset (rapidly described it in the next subsection). The two remaining parameters, do and a, are set to the values that ensure continuous trajectorics. More details about the single-souree synthetic trajectories generation can be found in Appendix E. Overall, we generated 12, 105 sequences for the training dataset and 3,052 sequences for the validation dataset.
a(t) azt² + aa1++ ao,
Training details. The SRNN model used in our experiments is an auto-regressive model, i.c., it uses the past source vectors S1+-1 to predict the current one 8,- In practice, the est imated past vectors are used for this prediction, rather than the ground-truth past vectors. To make the model robust to this problem, we trained the model in the scheduled sampling mode (Bengio et al. 2015). This means that during training, we gradually replace the ground-truth past values with the previously generated ones to predict the current value (see (Girin et al. 2021 Chapter 4) for discussion on this issue). The model was trained using the Adam optimizer (Kingma & Ba, 2014) with a learning rate set to 0.001 and a batch size set to 256. An early-stopping strategy was adopted, with a patience of 50 epochs.
5.2 MixDVAE evaluation set-up
Dataset. For the evaluation of the proposed MixDVAE algorithm, we used the training set of MOT17. MOT17 contains pedestrian scenes filmed in different places such as in a shopping mall or in a street, with static or moving cameras. The motion patterns of the pedestrians in these videos are quite diverse and challenging. The MOT17 training set contains seven sequences with length varying from twenty seconds to one minute, with different frame rates (14, 25, and 30 fps). The ground-truth bounding boxes are provided, as well as the detection results obtained with three customized detectors, namely DPM (Felzenszwalb et al,, 2010), Faster-RCNN (Ren et al., 20155,, and SDP (Yang et al. 2016). As briefly stated in the introduction, we focus our study on modcling the source dynamics for multiple-source tasks. Therefore, we leave aside the problem of appearing/ petearinng/dsspparriii disappearing sources (usually referred to as birth/death processes) and consider a fixed mumber of N = 3 tracks. We have thus designed a new dataset from the MOT17 training set, which we call the MOT17-3T datasct. The MOT17-3T dataset uses the publicly-relcased DBBs of the MOT17 datasot. We split a complete video sequence into subsequences of sequence length T. Three values of T are evaluated in our experiments: 60, 120, and 300 frames (respectively corresponding to 2, 4, and 10 seconds at 30 fps).
15

---

Published in Transactions on Machine Learning Research (12/2023)
Each test sequence contains three source trajectories with possible occlusions and detection absences, see example in Fig. 3 More details on the design of the MOT17-3T dataset can be found in Appendix F.2 We have finally created 1,712, 1,161, and 1,137 3-source test sequences of length T = 60, 120, and 300 frames, respectively. Notice that the pre-trained DVAE is not fine-tuned on these test sequences.
Algorithm initialization. Before starting the iterations of the proposed VEM algorithm, we need to initialize the values of several parameters and variables. Theoretically, there is no preference in the order of the three E-steps. In practice, however, for initialization convenience, we followed the order E-W Step, E-Z/E-S Step. Indeed, starting with E-W Step requires the initialization of the mean vector and covariance matrix of the source vector posterior distribution Min: Vtn, the input vectors of the DVAE encoder $1.7, and the observation covariance matrices Dta. For MOT, men can be easily initialised over a short sequence by assuming that the source does not move too much. Indeed, the initial values of mnn can be set to the value of the observed bounding box at the beginning of the sequence møn- While this strategy very straightforward to implement, it is too simple for many tracking scenarios, especially for long sequences. We thus propose to split long sequence into sub-sequences. For each sub-sequence, we initialise myz to the value at the beginning of the sub-sequence. After this initialisation, we run a few iterations of the VEM algorithm over the sub-sequence, allowing us to have an estimate of the source position at the end of the sub-sequence. This value is then used to provide a constant initialisation for the next sub-sequence. At the end, all these initializations are concatenated, providing a pieco-wise constant initialization for mtn over the entire long sequence, More implementation details, as well as the pseudo-code of this cascade initialization strategy, are provided in Appendix C The input vectors of the DVAE encoder are initialized with the same values as the ones used for m
Observation covariance matrix. In our experiments, we observed that the estimated values of both Pik and vn in (28) increased very quickly with the VEM algorithm iterations. This caused instability and unbalance between these two terms, which finally conducted the whole model to diverge. To solve this problem, we set Pe to a given fixed value, which is constant on the whole analyzed T-frame sequence and not updated during the VEM iterations. Specifically, for the MOT lask, 4e is set to a diagonal matrix, (o)x - - where ræ is a
and the diagonal entries are set to r [(o o1x)², (o - of factor lower than 1. In common terms, Pu is set to fraction of the (squared) size of the corresponding observation at frame 1. The covariance matrices Vin are initialized with the same values PiA-
-ff))
Hyperparameters. The VEM algorithm of MixDVAE has four hyperparameters to be set. The observation covariance matrix ratio T& is set to 0.04, the initialization subsequence length J is set to 30, and the initialization iteration number Io is set to 20. The MixDVAE algorithm itself is run for I = 70 iterations, which was experimentally shown to lead to convergence.
Baselines. We compare our model with two recent state-of-the-art probabilistic MOT methods: The Autoregressive Tracklet Inpainting and Scoring for Tracking (ArTIST) model of (aleh et al (2021) and the Variational Kalman Filter (VKF) of Ban et al.] (2021). In addition to that, in order to demonstrate the advantage of using a DVAE model for modeling the dynamics of single-source trajectorics, we consider replacing the DVAE model with a simpler deep auto-regressive (Deep AR) model. ArTIST is a supervised stochastic autoregressive model that learns the discretized multi-modal distribution of human motion using annotated MOT sequences. It can assign detections to tracks by scoring tracklef proposals with their likelihood. And it can also generate continuations of the source trajectories and inpaint those containing missing detections. We have reused the trained models as well the tracklet scoring and inpainting code provided by the author:aand and reimplemented the object tracking part according to the paper, as this part was not provided. Implementation details can be found in Appendix F Alike the proposed MixDVAE algorithm, the VKF algorithm for MOT (Ban et al. 2021) is also based on the VT hotoodyyggy to combine source position estimation and snrection-to-source assignment. However, a basic one-step linear dynamical model is used in VKF instead of the DVAE model in the proposed MixDVAE algorithm. In short, the dynamical model we use in VKF is Pa, (s)(s))))1)) = I N Ds;-1.n: Atn), where D is assumed to be the identity matrix and Atn is estimated in the M step. Hence, the VKF MOT algorithm is combination of VI and Kalman filter update equations. In (Ban et al. 2021 1, the method was proposed in audiovisual set-up. The
8 A tracklet indicates a sequence of estimated position vectors consistent over time and assigned to the same object. 9 available at https:// wgw.tht...co//hhege httpa//.oo//attteees////AAAAA
16

---

Published in Transactions on Machine Learning Research (12/2023)
Table 2: MOT results for short (T' = 60), medium (T' = 120), and long (T' = 300) sequences.
Dataset
Method
MOTA
MOTP!
IDF11
#ID$I
%IDS!
MT'
MI.
#FP4
%FP4
#FN4
%FNG
ArTIST
63.7
84.1
48.7
86371
28.0
4684
0
9962
3.2
15525
5.0
Short
VKF
56.0
82.7
77.3
5660
1.8
3742
761
64945
21.1
64945
21.1
Doep AR
67.4
76.1
83.1
5248
1.7
3670
129
49595
16.0
49595
16.0
MixDVAE
79.1
81.3
88.4
4966
1.6
4370
50
29808
9.7
29808
9.7
ArTIST
61.0
84.2
43.9
102978
24.6
2943
25388
6.1
34812
8.3
Medium
VKF
57.5
83.3
77.6
7657
1.8
2563
487
85053
20.3
85053
20.3
Deep AR
65.3
76.0
81.8
5387
1.3
2435
149
71775
17.0
71775
17.0
MixDVAE
78.6
82.2
88.0
6107
1.5
2907
120
41747
9.9
41747
9.9
ArTIST
53.5
84.5
40.7
205263
20.1
2513
4
1354101
13.2
135401
13.2
Long
VKF
74.4
86.2
84.4
30069
2.9
2756
100
116160
11.4
116160
11.4
Deep AR
75.5
76.6
87.1
26506
2.6
2555
18
123262
12.1
123262
12.1
MixDVAE
83.2
90.0
23081
2.3
2890
74550
7.3
74550
7.3
observations contain not only the DBB coordinates, but also appearance features and multichannel audio rocordings. For a fair comparison with MixDVAE, we use here the same observations, i.c., we simplified VKF by using only the DBB coordinates. For both ArTIST and VKF, the tracked sequences are initialized using the DBBs at the first frame, as what we have done for MixDVAE. For VKF, similarly to MixDVAE, applied the same cascade
need to provide initial values for Mmn and Vtn- tn- For fair comparison, initialization the one presented above, except that linear dynamical model is used in place of the DVAE to ensure the transition between two consecutive subsequences. The covariance matrices Vtn are initialized with pre-defined values that stabilize the EM algorithm. The covariance matrices Ptk are fixed to the same values as for MixDVAE. The covariance matrices of the linear dynamical model (denoted Atn in (Ban et al.] 202T)) are initialized with the same values as V Finally the simpler Deep AR baseline model deep generative model without stochastic latent variables. In this baseline, the dynamical model becomes Pa, (s:(S;-1) II (Srr, Ma, (,,) diag(ve, (,)))-1,n)))- In practice, the Deep AR model is implemented with an LSTM layer. The hidden dimension of the LSTM layer is set to match that of the LSTM layers employed in the DVAE model, i.c. it is equal to 8.
Evaluation metrics. We use the standard MOT metrics (Bernardin & Stiefelhagen 2008 Ristani et al. 2016) to evaluate the tracking performance of MixDVAE and compare it to the baselines, namely: multi- object tracking accuracy (MOTA), multi-object tracking precision (MOTP), identity Fl score (IDF1), mum- ber of identity switches (IDS), mostly tracked (MT), mostly lost (ML), false positives (FP) and false negatives (FN). The three test subsets contain different number of test sequences, with different. sequence length T. Therefore, for IDS, FP and FN, report both the number of occurrences and the corresponding percentage, Among them, MOTA is considered to be the most representative metric. It is defined by aggregating the frame versions of the metrics FP,, FN,, and ID, over frames:
MOTA 1 EFFN,, + FP, + +DS.)
(38)
E.
where GT, denotes the number of ground-truth tracks at frame t. Higher MOTA values imply less errors (in terms of FPs, FNs, and IDS), and henee better tracking performance. MOTP defines the averaged overlap between all correctly matched sources and their corresponding ground truth. Higher MOTP implies more accurate position estimations. IDF1 is the ratio of correctly identified detections over the average number of ground-truth and computed detections. IDS reflects the capability of the model to preserve the identity of the tracked sources, especially in case of occlusion and track fragmentation. MT and ML represent how much the trajectory is recovered by the tracking algorithm. A source track is mostly tracked (resp. mostly lost) if it covered by the tracker for at least 80% (resp. not more than 20%) of its life span.
5.3 Experimental results
Quantitative analysis. We now present and discuss the tracking results obtained with the proposed MixDVAE algorithm and compare them with those obtained with the baselines. In these experiments, the
17

---

Published in Transactions on Machine Learning Research (12/2023)
value of the observation variance ratio r is set to 0.04 and no fine-tuning is applied to SRNN in the E-Z step. Ablation study on these factors is presented in Appendix !|
The values of the MOT metrics obtained on short, medium and long sequence subsets (T 60, 120, and 300 frames, respectively) are shown in Table 2 We see that the proposed MixDVAE algorithm obtains the best MOTA scores for the three subsets (i.e., for the three different sequence length values). This is remarkable given that ArTIST was trained on the MOT17 training dataset, whereas MixDVAE never saw the ground- truth sequences before the test. Furthermore, we notice that both VKF and MixDVAE have much less IDS and much higher IDF1 scores than ArTIST, which implies that the observation- to-sonut------eeeeee assignment based on the VT method is more efficient than direct estimation of the position likelihood distribution to preserve the correct source identity during tracking. Besides, the MixDVAE model also has better scores than the VKF model for these two metrics, which implies that the AVAE-based dynamical model performs better on identity preservation than the linear dynamical model of VKF. For the 60- and 120-frame sequences, the ArTIST model has lower FP and FN percentages and higher MOTP scores (though the MOTP scores of all three algorithms are quite close for every value of T). This is reasonable because, again, ArTIST was trained the same dataset using the ground-truth sequences while our model is unsupervised. Overall, the adverse effect caused by frequent identity switches is much greater than the positive effect of lower FP and FN for the ArTIST model. That explains why MixDVAE has much better MOTA scores than ArTIST. For the long (300-frame) sequences, MixDVAF obtains an overall much better performance than the ArTIST model, since it obtains here the best scores for 6 metrics out of 8, including FP and FN. This shows that MixDVAE is particularly good at tracking objects on the long term (we recall that T 300 represents 10 s of video at 30 fps).
Besides, MixDVAE also globally exhibits notably better performance than VKF on all of the three datasets. This clearly indicates that the modeling of the sources dynamics with a DVAE model outperforms the use of simple Shear-GGussinn also notice that the VKF algorithm globally performs much better on 300-framc sequences than on 60- and 120-frame sequences. One possible explanation for this phenomenon is that the dynamical patterns of long sequences are simpler than those of short and medium sequences. In fact, the data statisties show that the average velocity in long sequences is much lower than that in short and medium sequences. In this case, the linear dynamical model can perform quite well allthough not as well as the DVAE.
dynamical model and can greatly improve the tracking performance. We can
Finally, we can see in Table 2 that MixDVAE with SRNN as dynamical model has an overall significantly better performance than MixDVAE with the baseline Deep AR dynamical model. This demonstrates the important role of the latent variables in SRNN for the dynamical modeling of sequential data. We remind that the latent vector Z is assumed to efficiently encode the generative factors of source n's trajectory.
Qualitative analysis. To illustrate the behavior of MixDVAE and the baseline models, we present an example of tracking result in Fig. & More examples can be found in Appendix G In the example of Fig. 3 the detection for Source 3 (os in the figure) is absent from t = 2 and reappears after t = 20. But we limit the plot to t = 10 for better visualization. This is a case of forg-term detection absence. An immediate identity switch occurs at t= 2 for the ArTIST model. Then, the track obtained by ArTIST is no longer stable. We speculate the reason for the frequent identity switches made by ArTIST is that the estimated distributions do not correspond well to the truc sequential position distributions, which is possibly due to the way these distributions are discretized. In addition to the identity switches, the estimations generated by ArTIST at t = 5, 8, and 10 are not accurate. This causes decrease of the tracking performance. For the VKF model, the estimated bounding boxes for Sources 2 and 3 (m2 and ma in the figure) overlap each other. This means that the two observations are both assigned to the same source, which is Source 2. From (32) and 33 we know that the value of the assignment. variable depends on the posterior mean and variance vectors Min and Vtn, which themselves depend on the dynamical model. With a linear dynamical model, VKF is not able to correctly predict distinct m2 and my trajectories. The Deep AR model succeed to predict distinct m2 and my trajectories. However, the trajectory my is not accurate due to the absence of og. In contrast, the very good dynamical modeling capacity of the DVAE makes MixDVAE able to keep tracking despite of the long-term detection absence and generate reasonable my estimations, which correspond well to the ground-trun truth trajectory of Source 3 (53 in the figure).
18

---

Published in Transactions on Machine Learning Research (12/2023)
Groadd Tiutt
SP
Detession
Figure 3: Example of tracking result obtained with the proposed MixDVAE algorithm and the two baselines. For clarity of presentation, the simplified notations $1, 01, and denote the ground-truth source position, the observation, and the estimated source position, respectively (for Source 1, and the same for the two other sources). Best seen in color.
6 Application of MixDVAE to single-channe audio source separation
When applying MixDVAE to the SC-ASS task, we work the short-time Fourier transform (STFT) domain. This implics that both the source and observation vectors are complex-valuete More precisely, the n-th source vector Sta = f f= E IF is the short-time spectrum of audio source at time frame & (ʃ denotes the frequency bin and s E C is the complete STFT spectrogram). The number frequency bins, F, is typically set 256, 512 or 1024 (a power of is preferred use the fast. Fourier transform). As is usually complex Gaussian prior distribution (Févotte al. 2009 Liutkus al., 2011 Girin et al., 2019), i.c. (5) becomes Pe, (s. Pc; P., (s;; (e, (Pr.ze)) The latent space dimension L is typically set to a value significantly lower than F. Also, here, o {ors 1t E CFx denotes the STFT spectrogram of the observed mixture signal. We define the k-th observation variable at frame as O,k = ot E C, which the STFT coefficient of the mixture signal at TF bin (t. f). In other words, the observation index k is identified with the frequency bin/index f, and the total number of observations for cach t. We note that in this casc, Stn and Otk do not have the same dimension, even though Stn and ot,: mentioned in Footnote 3, we need to define projection matrix Pa E CIxF, which is here
adopted in audio processing, Stn is assumed to follow a zero-mean circularly-symmeet
any frame equals the number of frequency bins, iee. K,
do. Therefore, the transposed one-hot vector activated at the k-th index. Finally, the observation Osk Orf is modeled with a conditional cireularlysymmeetrie complex Gaussian, centered at the corresponding source coefficient, and 12 becomes Pa. (0u/wue n. Sta) = NN PASt. L.x). We can see that the assignment variable wa associates each TF-bin of the observed mixture spectrogram to one of the sources, and thus implicitly defines a TF mask. All these adaptations yield minimal changes the MixDVAE derivation and solution. These changes are provided in Appendix B including the final source vector estimate. Note that the estimated n-th source waveform is obtained by applying the inverse STFT mI.T,n
6.1 DVAE pre-training
Dataset. We illustrate the application of MixDVAE to the SC-ASS problem with the separation of a speech signal and a musical instrument, in the present case the Chinese bamboo flute (CBF). These audio sources have very different spectral and dynamical patterns. So, we choose here to pre-train instances of the same DVAE model separately two single-source datasets, speech dataset. and CBF dataset.
19

---

Published in Transactions on Machine Learning Research (12/2023)
For the speech dataset, we used the Wall Street Journal (WSJO) dataset (Garofolo et al. 1993 which is composed of 16-kHz monophonic speech signals, with three subsets: si Ir si di 05 and si el 05, used for model training, validation and test, and containing 24.9, 2.2 and 1.5 hours of speech recordings,
respectively. For the musical instrument datasct, we used the CBF dataset of Wang et al. (2022), which contains CBF performances recorded by 10 professional CBF performers. The datasel comprises recordings of both isolated playing techniques and full-length pieces. We only used the full pieces recordings in our experiments. The original recordings are stereo and at a sampling frequency of 44.1kHz.. In our experiments, we used only one channel and downsampled the signals to 16-kHz, to match the speech signals rate. We selected the second half pieces recordings of player 1 and 2 as the validation set, the second half pieces recordings of player 3, 4 and 5 as the test set, and use all other recordings for DVAE pre-training. The total duration of the training, validation and lest sets are 2.1, 0.2 and 0.3 hours respectively. For both datasets, we used the training set for DVAE pre-training and the validation set for early stopping.
Pre-processing For both the speech and CBF datasct, we pre-processod the raw audio signals in the following way. First, the silence al the beginning and end of each signal are trimmed with voice activily detection threshold of 30 dB. Then, the waveform signals are normalized by dividing their absolute maximum value. The STFT coefficients are computed with a 64-ms sine window (1024 samples) and a 75%-ovcraap (256-sample hop length), resulting in sequences of 513-dimensio ---dimensionall discrete Fourier coefficient vectors (for positive frequencies). Note that in practice, the DVAE model will input speech power spectrograms, i.e., the squared modulus of s, instead of the complex-valued STFT spectrograms (Girin et al.) 2021 Chapter 13), and these STFT power spectrograms are split into smaller sequences of length 50 frames (corresponding to audio segments of 0.8s) for training.
Training details. We also used the SRNN model with scheduled sampling training (see Section 5.1 The model was trained with the Adam optimizer with learning rate set to 0.002 and a batch size set to 256. The latent space dimension L was set to 16. The early-stopping patience was set to 50 epochs for the WSJO datasct and 200 epochs for the CBF datasct.
6.2 MixDVAE evaluation set-up
Dataset. To generate the test mixture signals, we first randomly selected two signals from the WSJO test set and the CBF test set, respectively. Then, we removed the silence at the beginning and end in the same way as for the pre-processing. The clipped speech and CBF signals were mixed together with several different specch-to-music (power) ratios, namely 100, -5, 0 and 5 dB. The waveform mixture signals were theu normalized and transformed to STFT spectrograms in the same way as in the pre-processing. Similar to the MOT scenario, we tested MixDVAE with different test sequence length values. To this aim, the mixed signal STFT spectrograms were split into subsequences of length 50, 100 and 300 frames (respectively corresponding to audio segments of 0.8, 1.6 and s.8s). Overall, we generated 878, 491, and 372 mixed test signals of length T = 50, 100 and 300 frames, respectively.
Algorithm initialization. As for MOT, we need to initialize the values of several parameters and variables. For SC-ASS, it is more difficult to obtain a reasonable initialization for m.n the observed mixture signal. We thus choose the following VEM iteration order: F-Z//ES Step, F-W Step. In this case, we have to initialize the posterior distribution of the assignment variable (i.e. all the values of mAn,, the input vectors of the DVAE encoder S1.,,> (for the two sources), and the observation model covariance matrices Lik- We initialize Tikn with a discrete uniform distribution. As for the DVAE encoder, we first input the power spectrogram of the mixture signal (recall that the two DVAEs were pre-trained on different natural single-source datasets). We then use the reconstructed output power spectrograms as the initialization of the DVAE encoders input.
valued) using directly
Observation variance. Similar to the MOT case, T is not estimated in the M Step, but fixed to r louel? 12 In plain words, Pu: is set to a fraction of the observation power. This setting turned out to stabilize the VEM iteration process and finally led to very satisfying estimation results.
Hyperparamcters. Regarding the hyperparameters of the MixDVAE VEM algorithm for the SC-ASS task, the observation covariance matrix ratio re is set to 0.01. The total number of iterations I is set to 70. And the DVAE model is not fine-tuned in the F-Z step for the reported experiments.
20

---

Published in Transactions on Machine Learning Research (12/2023)
Baselines. As mentioned in Section 1.4 the state-of-the-art SC-ASS methods are mostly fully supervised, thus requiring a very large amount of paired (aligned) mixture signals and individual source signals for training. Very few methods are under the weakly-supervised or unsupervised settings. Thus, it is difficult to find a fairly comparable baseline model. In the presented experiments, we have compared the proposed MixDVAE method with the unsupervised audio source separation method called MixIT (Wisdom et al., 2020) and with two weakly-supervised methods based on NMF, namely a vanilla NMF model (Févotte et al. 2018) and an NMF model with temporal extensions (Virtanen 2007). MixIT is deep-learning- baase unsupervised single-channel source separation method. It is trained on dataset constructed by mixing up the existing mixture audio signals. The model separates them into a variable number of latent source signals that can be remixed to approximate the original mixtures. In a totally unsupervised setting, MixIT does not require having the separated source signals for training. For the implementation of the MixIT model, we have reused the code provided by the authors and adapted it for the speech-CBF source separation task. In the weakly-supervisedd NMF bascline mothods, an NMF model first pre-trained on cach single-source dataset separately, resulting in a dictionary of non-negative spectral templates W,, for each of the sources to separate. Such pre-training is similar in spirit to the pre-training stage of the MixDVAE method. After that, the obtained spectral template dictionaries of all sources are fixed and concatenated together as to learn the temporal activation matrix H for the test mixture signal. Then the H entries corresponding to the spectral templates in W, are used to separate source n (in practice, Wiener filters are build to separate the sources in the STFT domain, see (Févotte et al. 2018) and (Virtanen 2007) for details). We have re-implemented both NMF-based baselines according to the formula given in the corresponding papers. The latent dimension K of the NMF model for both speech and CBF data set to 128, which is determined by grid search. To demonstrate the interest of using DVAE model for modeling the audio source dynamics in MixDVAE, we made additional experiments with replacing the DVAE model in MixDVAE with two other dynamical models: a results in baseline models similar to the VKF model and the Deep AR model that we have already used in our MOT experiments (see Section 5.2). For the VKF model, we initialize the values of Takn in two ways: the ground-truth assignment mask, which is also named as ideal binary mask (IBM) in the audio source separation literature (we call the resulting model VKF-oracle) and the mask defined from the outputs of the pre-trained DVAEs when inputing the mixture signal spectrogram (we call the resulting model VKF-DVAF- init). Note that VKF-oracle provides an (unrealistie) upper bound of separation performance with linear dynamical model, whereas VKF-DVAB-init uses the same initial information as MixDVAF. 4.x are fixed to the same values as for MixDVAE and Am are initialized with the identity matrix multiplied by a scalar. For the Deep AR model, it is implemented with an LSTM layer, with the hidden dimension set equal to that of the LSTM layers employed in the DVAE model. Finally, to investigate the effects of the VEM algorithm in the MixDVAE model, we also compared our model with the direct reconstruction of the source signals from the output of the pre-trained DVAEs when using the mixture spectrogram as the input, i.e. the information used to intialize both MixDVAE and VKF-DVAE-init (we call this baseline method DVAE-init). As these output spectrograms are power spectrograms, we combined their square root (amplitude spectrogram) with the phase spectrogram of the mixture signal to reconstruct the waveform of the bascline separated signals.
dynamical model and deep auto-regressive dynamical model, which
Evaluation metrics. We used four source separation performance metrics widely-used in speech/aue audio processing. The root mean squared error (RMSE), the ocale-invariant signal-to-distortion ratio (SI-DDR) [Roux et al. 2019 in dB, and the perceptual evaluation of speech quality (PESQ) score (Rix et al., 2001 (values in 0.55 4.5))
I0 For all metrics, the higher the better.
6.3 Experimental results
Quantitative analysis. We report the speech-CBF separation results on the short, medium and long test sequence subsets (T'= 50, 100, 300 frames, respectively) in Table 3 In addition to the results obtained by the different models, we also report the values of the evaluation metrics when applied on the mixture signal, for reference.
The PESQ objective mcasure was developed mostly for evaluating the quality of speech signals, but since it is largely based on a model of Human auditory perception, we assume we can also use on the CBF sounds to avoid complicate the evaluation protocol.
21

---

Published in Transactions on Machine Learning Research (12/2023)
Table 3: SC-ASS results for short (T = (0), medium (T" =100), and long (T' 300) sequences.
Speech SI-SDR
Chinese bamboo flute
Dataset
Method
RMSE 4
PESQ t
RMSE
SI-SDR
PESQ
Mixture
0.016
-4.94
1.22
0.016
4.93
1.09
VKF-Oracle
0.004
14.83
2.00
0.004
20.15
2.33
DVAE-inf init
0.013
-0.51
1.20
0.019
3.04
1.44
Short
IKF-DVAE-ine init
0.012
2.24
1.21
0.012
8.06
1.33
Deep AR
0.009
5.32
1.29
0.018
5.19
1.48
MixIT
0.011
3.26
0.009
7.15
Vanilla NMF
0.011
3.01
1.40
0.012
9.09
1.37
Temporal NMF
0.009
4.99
1.53
0.011
10.26
1.53
MixDVAE
0.006
9.23
1.73
0.007
13.50
2.30
Mixture
0.016
-4.44
1.17
0.016
4.44
1.08
VKF-Oracle
0.004
14.88
1.88
0.003
20.24
2.41
DVAE-init
0.014
0.10
1.15
0.020
2.42
1.27
Medium
VKF-DVAE-init
0.013
1.25
1.12
0.013
7.42
1.26
Deep AR
0.010
4.88
1.21
0.017
5.17
1.35
MixIT
0.009
4.75
0.009
8.74
Vanilla NMF
0.011
3.28
1.41
0.011
8.88
1.35
Temporal NMF
0.010
5.12
1.48
0.011
9.96
MixDVAE
0.007
9.32
1.65
0.007
13.05
2.16
Mixture
0.016
-4.52
1.19
0.016
4.53
1.10
0.004
14.65
1.89
0.003
20.45
2.60
DVAE-ini
0.013
0.20
1.15
0.020
2.29
1.22
Long
VKF-DVAE-init
0.013
0.34
1.10
0.013
7.35
1.24
Deep AR
0.010
3.87
1.17
0.017
4.74
1.27
MixIT
0.006
10.2
0.007
11.76
Vanilla NMF
0.011
3.31
1.40
0.011
8.98
1.35
Temporal NMF
0.010
5.01
1.47
0.011
10.06
1.42
MixDVAE
0.007
9.06
1.64
0.007
12.92
2.06
We observe that on the short and medium sequence subsets, MixDVAE achieves the best performance for all of the evaluation metrics. While on the long sequences subsets, MixIT obtains slightly better results than MixDVAE for the speech. This demonstrates that the proposed method works well on the SC-ASS task. Unsurprisingly, VKF-Oracle obtains the best scores all metrics because it initialized with the ground-truth mask. When comparing MixDVAE with the methods of different. dynamical models, we find that MixDVAE obtains overall better performance than both VKF-DVAE-init and Deep AR on all of the metrics for all of the three subsets. It is clear that the non-linear DVAE model with stochastic latent variables is much more efficient than the SinaarGaussiaan model and the deep auto-regressive model without latent variables for modcling the audio source dynamics. Besides, with the increase of the sequence length, the performance of MixDVAE dropped quite moderately (less than 0.6 dB and less than 0.3 dB in SI-SDR gain decrease speech and CBF respectively), while the performance of VKF dropped by 2.32 dB speech and by 0.71 dB on CBF, and the performance of Deep AR dropped by 1.45 dB on speech and by 0.45 dB on CBF.
Compared to DVAE-init, both MixDVAE and VKF-DVAE-init exhibit better separation performance (at least in terms of SI-SDR for VKF-DVAE-init). This indicates that the multi-source dynamical model with the obscrvation-to-ee to-source assignment latent variable plays an important role in separating the content of different audio sources.
22

---

Published in Transactions on Machine Learning Research (12/2023)
Speech
4
Flute
(a) Mixture
(b) Ground Truth
(c) VKF-Oracle
Speech
Flute
(d) DVAE-init
(e) VKF-DVAE-Init
(f) MiXDVAE
Figure 4: An example of audio source separation result obtained with the proposed MixDVAE algorithm and the basclines (speech and CBF power spectrograms). Best seen in color.
Although MixIT obtains slightly better performance than MixDVAE for speech on the long sequences subsets, its performance on short and medium sequences subsets is quite bad (in terms of S1-SDR, only 3.26 dB for speech and 7.15 dB for CBF on the short sequences subset, and 4.75 dB for speech and 8.74 dB for CBF on the medium sequences subset). As for the NMF based models, though adding temporal extensions the vanilla NMF model indeed improves the model performance, the obtained results remain significantly inferior that obtained by MixDVAE.
Qualitative analysis. To illustrate the behavior of the different models, selected an audio source separation example and plotted the spectrograms in Fig. g More examples can be found in Appendix H In the given example, the sequence length of the spectrograms is 300 frames. It is obvious that the ground-truth spectrograms of both the speech and CBF have spectral components with non-linear trajectories over time. Even though VKF-Oracle achieves the best performance, we observe in Subfigure (c) that there are several stationary traces (artifactual horizontal spectral lines) in the spectrograms caused by the inappropriate linear dynamics hypothesis. This phenomenon becomes even worse when VKF is initialized with the mask defined by the outputs of the pre-trained DVAEs (VKF-DVAB-init). In Subfigure (d), we clearly see the stationary traces, especially for the scparated speech spectrogram. We believe that this is the reason why VKF-DVAE- init showed poor separation performance in general. When looking at the outputs of the pre-trained DVAE models, we find that the pre-trained DVAE models can provide a relatively good initialization for the VEM algorithm, even if we are still far from separated sources. In fact, since the SC-ASS task, we pre-trained separately two DVAE models on the speech dataset and on the CBF dataset, the pre-trained DVAE models already have some prior information about. the single-source dynamics. Though only give the mixture
23

---

Published in Transactions on Machine Learning Research (12/2023)
spectrogram as imput, the pre-trained DVAE models can, to some extent, enhance the information of the source used in pre-training and attenuate the information of the other source. However, this kind of filtering is not very efficient. As we can see in the top figure of Subfigure (e), the output spectrogram provided by the DVAE model pre-trained on the speech datasct still keeps a significant amount of information on the CBF. Finally, even the initialization not that accurale, we see in Subfigure (f) that MixDVAE achieved a good separation of the two sources after running the VEM iterations.
7 Conclusion and future work
In this paper, we introduce MixDVAE, an LVGM designed to model the dynamics of multiple, jointly observed, sources. MixDVAE involves two main modules: A DVAE model for capturing the dynamics of each individual source and discrete latent assignment variable that assign observations to sources, thus enabling us to form complete trajectories. The model learning process consists of two stages. During the first stage, the same or different DVAE model(s) is/are pre-trained on the synthetic or natural single-source trajectory dstaset(s) to obtaín prior information about the sources dynamics. During the second stage, the pre-trained DVAE model(s) is/are integrated into the general MixDVAE model. The entire MixDVAE model is solved using the VI framework with a VEM algorithm that combines the structured mean-field approximation and the amort inference principles. The VEM algorithm is run directly on each multi- source test data sequence to process and the entire method does not require massive multi-source annotated datasets for training, which are difficult to obtain, especially for natural data. Hence, we consider it as weakly-superrised, as opposed to the fully-supervised approaches most commonly used in many multi-source processing applications. We illustrate the versatility of MixDVAF by applying it to two distinct scenarios: the MOT task and the SC-ASS task. Experimental results demonstrate that MixDVAE performs well on both tasks. Specifically, thanks to the strong dynamical modeling capacity of DVAE, MixDVAE shows be more efficient than the combination of linear dynamical model with the assignment variable. In addition, MixDVAE can generate reasonable predictions of the source vector even in the absence of observations, resulting in smooth and robust trajectories, as demonstrated in the MOT task. Our experiments demonstrate the generalization capability of MixDVAE trained on a synthetic single-target dataset, and evaluated in a multiple-target dataset. Finally, we believe that MixDVAE has a very strong potential for modeling the dynamics of multiplo-source systems in general, and can be applied to various other tasks. However, we acknowledge that MixDVAE also has certain limitations, such as the assumption that each source behaves independently and the lack of consideration for interactions among them. We leave this as challenging topic for future rescarch.
Acknowledgments
This research was partially funded by the Horizon 2020 SPRING project funded by the European Commission (under GA #871245), by the French Research Agency Young Researchers Program ML3RI project (under GA #ANR-19-CI333-000-.0 and by the Multidisciplinary Institute of Artificial Intelligence (under GA 1ANR-19-P3LA-00
References
Masanao Aoki. State space modeling of time series. Springer Science & Business Media, 2013.
Yutong Ban, Xavier Alameda-Pineda, Laurent Girin, and Radu Horaud. Variational Bayesian inference for audio-visual tracking of multiple speakers. IEEE Trans. Pottern Anal. Mach. Intell., 4366):1761-1776, 2021.
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. In Advances in Neural Inform. Process. Systems (NewrIPS), pp. 1171-1179, 2015.
Keni Bernardin and Rainer Sticfelhagen. Evaluating multiple object tracking performance: The CLEAR MOT metrics. FURASIP J. Image Video Process., 2008:10, 2008.
24

---

Published in Transactions on Machine Learning Research (12/2023)
Xiaoyu Bie, Laurent Girin, Simon Leglaive, Thomas Hueber, and Xavier Alameda-Pineda. A benchmark of dynamical variational autoencoders applied to speech spectrogram modeling. In Proc. Interspecch Conf- Brno, Czech Republic, 2021.
Xiaoyu Bie, Wen Guo, Simon Leglaive, Lauren Girin, Francesc Moreno-Noguer, and Xavier Alamoda- Pineda. Hit-DVAE: Human motion generation via hicrarchical transformer dynamical VAE. arXiv preprint wrXiu:2204.01666 2022a.
Xiaoyu Bie, Simon Leglaive, Xavier Clameda-Pinedae and Laurent Girin. Unsupervised speech enhancement using dynamical variational autoencoders. JEEE/ACM Trans. Audio, Speech, Lang. Process., 30:2993- 3007, 2022b.
Christopher M. Bishop. Puttern recognition and machine learning. Seringer-Verlage Berlin, 2006.
Pritish Chandna, Marius Miron, Jordi Janer, and Emilia Gómez. Monoaural audio source scparation using deep convolutional neural networks. In Proc. Latent Variable Analysis and Signal Separation (LVA/ICA), Grenoble, France, 2017.
Gioele Ciaparrone, Francisco Luque Sánchez, Siham Tabik, Luigi Troiano, Roberto Tagliaferri, and Francisco Herrera. Deep learning in video multi-object tracking: A survey. Neurocomputing, 381:61-88, 2020.
Patrick Dendorfer, Aljoša Ošep, Anton Milan, Konrad Schindler, Daniel Cremers, Ian Reid, Stefan Roth, and Laura Leal-Taixé. MOTChallenge: A benchmark for single-camera multiple target tracking. Int. J. Comput. Vis., 129:845 881, 2021.
Garry A Einicke and Langford B White. Robust extended Kalman filtering. IEFE Trans. Signal Process.. 47(9):2596-22)) 1999.
Pedro F. Felzenszwalb, Ross B. Girshick, David McAllester, and Deva Ramanan. Object detection with discriminatively trained part-based models. IEEE Trans. Pattern Anal. Mach. Intell., 129):61277 2010.
Cédric Févotte, Emmanuel Vincent, and Alexey Ozerov. Single-channel audio source separation with nmf: Divergences, constraints and algorithms. Audio Source Separation, pp. 24, 2018.
Marco Fraccaro, Spren Kaae Sonderby, Ulrich Paquet, and Ole Winther. Sequential neural models with stochastic layers. In Advances in Neural Inform. Process. Systems (NeurIPS), pp. 2207-2215, 2016.
Cédric Févotte, Nancy Bertin, and Jean-Louis Durrieu. Nonnegative matrix factorization with the Itakura- Saito divergence: With application to music analysis. Newral Comp., 21(3):993-82) -800, 2009.
John S. Garofolo, David Graff, Doug Paul, and David Pallett. CSR-I (WSJO) Sennheiser LDC93S6B. Philadelphia: Linguistic Data Consortium, 1993.
Dan Geiger, Thomas Verma, and Judea Pearl. Identifying independence in Bayesian networks. Networks, 20(5):507-534, 1990.
Zoubin Ghahramani and Geoffrey E Hinton. Parameter estimation for lincar dynamical systems. Technical Report, University of Toronlo, 1996.
Zoubin Ghahramani and Geoffrey E. Hinton. Variational learning for switching state-space models. Neural Comp., 12(4):831-86) 864, 2000.
Zoubin Ghahramani and Michael Jordan. Factorial hidden Markov models. In Advances in Neural Inform Process. Systems (NewrIPS), 1995.
Laurent Girin, Fanny Roche, Thomas Hueber, and Simon Leglaive. Notes on the use of variational autoen- coders for speech and audio spectrogram modeling. In Proc. Int. Conf. Digital Audio Effects (DAFx), Birmingham, UK, 2019.
25

---

Published in Transactions on Machine Learning Research (12/2023)
Laurent Girin, Simon Leglaive, Xiaoyu Bie, Julien Diard, Thomas Hueber, and Xavier Alameda-Pineees Dynamical variational autoencoders: A comprehensive review. Found. Trends Mach. Learn., 15(1-2): 1 1-175, 2021.
Matthew D. Hoffman, David M. Blei, Chong Wang, and John Paisley. Stochastic variational inference. J. Mach. Learn. Res., 14(4):1303 1347, 2013.
Michael I Jordan, Zoubin Ghahramani, Tommi S Jaakkola, and Lawrence K Saul. An introduction variational methods for graphical models. Mach. Learn., 37(2):183 233, 1999.
Rudolf Emil Kalman. A new approach to linear filtering and prediction problems. J. Basic Eng., 82(1): 45, 1960.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1xxiv:112.600 2014.
Diederik P. Kingma and Max Welling. Auto-encoding variational Bayes. In Proc. Int. Conf. Learn. Repres. (ICLR), 2014.
Harold W Kuln. The Hungarian method for the assignment problem. Nav. Res. Logist Q. 2(1-2):83-9) 97, 1955.
Antoine Liutkus, Roland Badeau, and Gäel Richard. Gaussian processes for underdetermined source sepa- ration. IEFE Trans. Signal Process., 59(7):3155-3166 2011.
Wenhan Luo, Junliang Xing, Anton Milan, Xiaoqin Zhang, Wei Liu, and Tae-Kyun Kim. Multiple object tracking: A literature review. Artif. Intell., 293:103448, 2021.
Santiago Manen, Michael Gygli, Dengxin Dai, and Lue Van Gool. Pathtrack: Fast trajectory annotation
with path supervision. In Proc. IEEE Int. Conf. Computer Vision (ICCV), pp. 290-299, 2017.
Geoffrey J McLachlan and Kaye Basford. Mizture models: Inference and applications clustering. M. Dekker New York, 1988.
Geoffrey J McLachlan and Thriyambakam Krishnan. The EM algorithm and extensions. John Wiley & Sons, 2007.
Kevin Murphy. Switching Kalman Filters. Citeseer, 1998.
Alexey Ozerov, Cédric Févotte, and Maurice Charbit. Factorial scaled hidden Markov model for polyphonic audio representation and source separation. In Proc. IEEE Workshop Appl. Signal Process. Audio Acoust. (WASPAA), pp. 121-124, 2009.
Giorgio Parisi and Ramamurti Shankar. Statistical field theory. Phys. Today, 41(12):110, 1988.
Lawrence Rabiner and Biinghwang Juang. An introduction to hidden Markov models. IEEE ASSP Magazine, 3(1):4-10) 16, 1986.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. In Advances in Neural Inform. Process. Systems (NeurIPS). 2015.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. Tn Proc. Int. Conf. Mach. Learn. (TCML), pp. 1278-1286 2014.
Ergys Ristani, Francesco Solera, Roger Zou, Rita Cucchiara, and Carlo Tomasi. Performance measures and data set for multi-target, multi-camera tracking. In Proc. Europ. Conf. Computer Vision (ECCV), pp. 17-35. Springer, 2016.
Antony Rix, John Becrends, Michael Hollier, and Andries Hekstra. Perceptual evaluation of speech quality (PESQ) A new method for speech quality assessment of telephone networks and codecs. In Proc. IEEE Trl. Conf. Acoust., Speech, Signal Process. (ICASSP), Salt Lake City, UT, 2001.
26

---

Published in Transactions on Machine Learning Research (12/2023)
Jonathan Le Roux, Scott Wisdom, Hakan Erdogan, and John R.. Hershey, SDR - Half-baked or well done? In Proc. IEEE Tnl. Conf. Acoust., Speech, Signal Process. (ICASSP), Brighton, UK, 2019.
Fatemeh Saleh, Sadegh Aliakbarian, Hamid Rezatofighi, Mathieu Salzmann, and Stephen Gould. Proba- bilistic tracklet scoring and inpainting for multiple object tracking. Tn Proc. IEEE Int. Conf. Computer Vision Pattern Recogn. (CVPR), pp. 14329-14339, 2021.
Emmanuel Vincent, Maria Jafari, Samer Abdallah, Mark Plumbley, and Mike E Davies. Probabilis- tic modcling paradigms for audio source separation. In Machine Audition: Principles, Algorithms and Systems, pp. 162 185. IGI global, 2011.
Tuomas Virtanen. Monaural sound source separation by nonnegative matrix factorization with temporal continuity and sparseness criteria. IEEE Trans. Audio, Speech, Lang. Process., 15(3):1066-1)) 1074, 2007.
Ba-ngu Vo, Mahendra Mallick, Yaakov Bar-Shalom, Stefano Coraluppi, Richard Osborne, Ronald Mahler, and Ba-tuong Vo. Multitarget tracking. Wiley encyclopedia of electrical and electronics engineering, 2015.
Martin J Wainwright, Michael I Jordan, al. Graphical models, exponential families, and variational inference. Found. Trends Mach. Learn., 111-2):1-305, 2008
Changhong Wang, Emmanouil Benetos, Vincent Lostanlen, and Elaine Chew. Adaptive scattering transforms for playing technique recognition. IEEE Trans. Audio, Speech, Lang. Process., 30:1407 1421, 2022.
De Liang Wang and Jitong Chen. Supervised speech separation based on deep learning: An overview. IEEE/ACM Trans. Audio, Speech, Lang. Process., 26(10):1702 1726, 2018.
Scott Wisdom, Efthymios Tzinis, Hakan Erdogan, Ron Weiss, Kevin Wilson, and John Hershey. Unsuper- vised sound separation using mixture invariant training. In Advances in Neural Inform. Process. Systems (NeurIPS), volume 33, pp. 3846-3857, 2020.
Fan Yang, Wongun Choi, and Yuanqing Lin. Exploit all the layers: Fast and accurate CNN object detector with scale dependent pooling and cascaded rejection classifiers. In Proc. IEEE Int. Conf. Computer Vision 2016.
Pattern Recogn. (CVPR), pp.
Ozgur Yilmaz and Scott Rickard. Blind separation of speech mixtures via time-frequency masking. IEEE Trans. Signal Process., 52(7))1830-18)) 1847, 2004.
Dong Yu and Li Deng. Automatic speech recognition. Springer, 2016.
Paul Zarchan. Progress in astronautics and aeronautics: fundamentals of Kalman filtering: a practical approach, volume 208. Aiaa, 2005.
27

---

Published in Transactions on Machine Learning Research (12/2023)
A MixDVAE algorithm calculation details
A.1
E-S Step
Here detail the calculation of the posterior distribution qo (s/o). in 23 can be developed
Using 11
the first expectation term
K
Byn (wlo) loggo, (olw, = Bgow (w/o)
log pe. (Otn WeRk st,1,))
T
Kr
E Baow (walow))
po. po. (ouk (eek St.1:))]
(39)
Since for any pair (1, the assignment variable wee follows discrete posterior distribution, we denote its values by
aew (Was n/o:)) = Qekn
which will be calculated later in the E-W Step. With this notation, we have:
T K. ΣΣ>
E Ey (w)o) pog s)]
Mekn log Po. (Oes/wre n,, St,8
(40)
The second expectation 23 cannot be computed analytically as a distribution on because of the non- linearity in the decoder and in the encoder. In order to avoid a tedious sampling procedure and obtain computationally efficient solution, we further approximate this term by assuming 4e, (z]s) & Qo (z|s = m(-1)) where (i-1) is the mean value of the posterior distribution of estimated at the previous iteration. By using this approximation, the term Ey (z|s) log qa, (z|s)] now considered constant.
In addition, observe that the second term of 23 can be rewritlen as:
Eeg (2|s) log
Po.. Pr (s, [s,2)]
Σ
E qos (R.../mm (t-1) log po (S:, 2,,a)]:
(41)
since both the DVAE joint distribution and posterior distribution factorise over the sources, as formalizod in 15 and 17). As consequence, the posterior distribution of factorises over the sources too:
N
go. (s|o) = II
9a (s.,nlo), n=1
(42)
and therefore:
T K.
90 ().. exp
E [ You (z.,,. (m ()
log Po., (s. m))
III (=1k=
exp (Nenn log pa, (Ouk/wee n2, St))
In the above equation, the expectation term cannot be calculated in closed form. As usually done in the DVAE methodology, it is thus replaced by Monte Carlo estimate using sampled sequences drawn from the DVAE inference model. Let us denote by 2. goz (z..n (((1))) such sampled sequence. In the present work, use single point estimate, thus obtaining:
K+ II exp (Mtkn log po, (ou luee n, Stx))
Lo. (s),)()) x Po.. ms z
T
K.
Pe. (Stm $1.1 1. Zi )po, (z (1 1,1 1:: 1.n II
exp (Neen log Pe. (ou Wis = spm)
[=1
(43)
28

---

Published in Transactions on Machine Learning Research (12/2023)
We observe that the t-th element of the previous factorisation is distribution over Stn conditioned by 81.t-1nn As for te. (8., the dependency with $1:1-1,8 is non-linear and therefore would impede to obtain a computationally efficient closed-form solution. In the same attempt of avoiding costly sampling strategies, we approximate the previous expression replacing $1.--1, with (4) 11 obtaining:
9o. go. (8.n
II
go. (Sen |s $i:--1, 0),
(44)
with
Kr II k=1
( qán (Sta (11 1 0) pa, (Sen [t] 1, innn
exp (Nun Po. Pe. (on wis n, n,
(45)
(1) (
(1)
since the term po, (Zin 11:t-1,0 11:--.n) becomes constant.
Another interesting consequence sampling $14-11, that the dependency with the future observations of 9o. (st o) disappears. Indeed, since we are sampling every time step, the future posterior do not depend on Stn; and therefore the posterior distribution of St will not depend the future observations.
distributions qo. (Sc+kkm/$i:+++-1..
(2)
The two distributions in the above equation are Gaussian distributions defined in and 12 Therefore, it
can be shown that the variational posterior distribution of St, is Gaussian distribution: 4s. (Stn 1 1, o)
(( M Von) with covariance matrix and mean vector provided in (77 and (28) respectively, and recalled here for completeness:
Kr Σ
V
Thkn
$ --1 + diag(ve
tk
(46)
min
Vin
Ukkn 1 ik Osk + diag(v (
(47)
(1) where tn and t are simplified notations for va, (s iit.m) and ne, (sid-1. z respectively,
(5)
(5
denoting the variance and mean vector estimated by the DVAE for source n time frame
A.2 E-Z Step
Here we detail the calculation of the ELBO term 29
C(Os, Oz 0) = Egg, (x|o)
Eges (m/>s) [og Pe.. (s,
log qd. [z|s)]
E N
E v
Σ
log pr.. (s,,,, mim
EE
Σ
log 4o (2..n(1..u))
II
au (8.
II mmi
aox (n (8
II aus (a. 18
N
Σ
Egen (o))
Boon (a.n log
Po..o.. Po.. (s., 2,,n)]
Egen (#... (s..)) log T0, (Zi,n (8,,,)]
==1
(0s, Oz (2;
(48)
==1
with
Lm Lm,(0s, Oz, 2z; 0) = Eggr (0.,10)
E Bges (a. (.,))
(B.).))) log pa,e (s.,, 2.,a]] - E Eggn (z. (.,)) log an (2..n [s.)
(49)
29

---

Published in Transactions on Machine Learning Research (12/2023)
A.3 E-W Step
Here detail the calculation of the posterior distribution low (w/o). Applying the optimal update equa- tion 2 to w, we have:
Now (w)o) xx exp (R... (olo)es (ele) (o,,
w, s,,,)]]
(50)
Using
10
we derive:
Now (w)o) x pa (w) exp (Eg,,, (slo) (olw, s]]]
(51)
Using the expectation term can be developed as
Egg, (=1o) [log Pv. (olw, s)] E Egs (x|o)
log pe. (orx werks st.:)
Bus (o)
log Pao (ork/ s...)
(52)
1=1
Combining
13 and the previous result, have:
gow (w)o)
po (wix) exp Enoo (o).
[log po, (oek Wek: st.)))
(53)
1= km
which we can rewrite
K. ahh (wlo) x I
Now (wek lo:
(54)
(=1 k=1
with
Vew (we) pow (Wex) exp (Ryes (o). loog po. (ouk Wek: st.)))
(55)
The assignment variable follows discrete distribution and we denote:
Mkkn = Tow (wik = n)o x Pow (wix = n) exp Ey (s... (o) log Pa, (ow//// = n, stn))
(56)
Using the fact that both Pe. (Ois) Wyy = n, Stn) and 4o, (smm/o) are multivariate Gaussian distributions (defined
in 12
and 266
28
respectively), the previous expectation can be calculated in closod form:
Ego, (sm. [o]
log Pe. (Otk |wek n2, Stn) (Stnn Men V.n) log (ors) Stn, P. )dsen
log P + (or men) T )
(ou min) + Tr ( (V.''Ven))
Ven)
(57)
By using 14h.
the previous result, and normalizing to 1, finally get:
Bekn
Meka
ΣN Bekt
(58)
where
Bokn = Nou myn. P.:) exp
T
(59)
2
In fact, the posterior distribution 9s, (o) also conditioned 81 and Zis::- We this abuse notation for concision.
30

---

Published in Transactions on Machine Learning Research (12/2023)
A.4 M Step
Here detail the calculation of L.k- In the ELBO expression
19
only the first. term depends on 0,:
0) = Baow (wlo) (slo) Po, Po. (o/w,s)
Kn
(Stnn min: V log.V Stn, Pek Lx:) )d
Mekn
N
K2
Thkn
1es| Deal + (Oix mmn)² Pi (ou mmn + Tr(2, (V))
(60)
By computing the derivative of L(o,: o) with respect to a and setting it to 0, we find the optimal value of l that maximizes the ELBO:
Σ Tekn (O1
mor) m V.n).
(61)
tk
B Formulas for SC-ASS
With the adaptations in the model mentioned at the beginning of Section G for the SC-ASS lask, the solution formulas are as following. In the E-S Step, 27 and 28
become:
V bn
Makn P
T
diag(u (8)
(n) )-)
-1
(62)
Min
Vin
Mekn PT 1 1 Ock
(63)
The E-Z Step is not changed. In the E-W Step,
33
become:
1 T (P
Bekn = Ne PamIn
12x) cxp
(64)
Finally, in the M Step, 34
become:
Σ
Makn (ook - Pamm,)(Ou - Pamm) PAV P.)..
(65)
n=1
C Cascade initialization in MOT
For the initialization of the source (position) vector, we first split the long sequence indexed by E f1,2,..., T} into smaller sub-sequences indexed by {f1,..,, 4,, (t1 (....,,)).... {fy-1 I,..., T]). For the first sub- for & times with arbitrary order of assignment. Thus, there i.c. this implicitly sets N = Kj. The subsequence of source position vectors S1:h, is initialized with the same values as for the mean vector. Then, we run the MixDVAE algorithm on the first. subsequence for To iterations. Next, we initialize the mean vector sequence +1 of the second subsequence with m repeated for to -, times (and the same for St, +n)) And so on for the following subsequences. Finally, the initialized subsequences are concatenated together to form the initialized whole sequence. The pseudo-code of the cascade initialization can be found in Algorithm g
sequence, the mean vector sequence m is initialized the detected vector at the first frame ОΙд repeated
many tracked sources as initial detections,
31

---

Published in Transactions on Machine Learning Research (12/2023)
Algorithm 2 Cascade initialization of the position vector sequence
Input:
Detected bounding boxes at the first frame 01,1:K;3
Pre-trained DVAE parameters (0x, 8z, Φx);
Initialized observation model covariance matrices { {
(0),KK 1t
Initialized covariance matrices (V()) Output:
T.N ( It,==
Initialized mean position vector sequence {m (0) T.N It,n=
Initialized sampled position vector sequence {s (0) I.NN
Split the whole observation sequence into J sub-sequences indexed by {to tit. {&r 1,..,tt) .... {t--1 ..., t. T);
for j == 1 do
3: for k + 1 to K1 do
4:
n + k;
5: for t + to f do
(0)
(0)
m 8tn -
end for
end for
end for
10: for
+- 2 to J do
11: for n n 1 N do
12:
for t 1--1--1 + to ty do
(0) (0)
(T))
13:
min Stn m
14:
end for
15:
(m
(1n) f
Jt 1+1 MixDVAE(To, {(Os, O2 6z).
( (0)
(0)
K..
16:
m in Sin --=1--1-1,, -.
17: end for
18: end for
19: m
[
Lib
1:N1
m, - (0)
20: $1.TI:N
PLU.1:N:
D SRNN implementation details
The SRNN generative model is implemented with forward LSTM network, which embeds all the past information of the sequence s. Then, a dense layer with the tanh activation function plus a linear layer provide the parameters μe,, Vo.- Similarly, the parameters μe,, VØ, are computed with two dense layers with tanh activation function plus linear layer appended to the LSTM as well. The inference model shares the hidden variables of the forward LSTM network of the generative model and uses two dense layers with the tanh activation function plus linear layer compute the parameters p Vo..
In the MOT set-up, both S, and z, of dimension 4. While in the SC-ASS set-up, is of dimension 513 and 2, is of dimension 16. The SRNN generative distributions in the right-hand side of (36) are implemented as:
h, dr dn (s: h;-1);
(66)
[1o, vo.1 dz (h,, 24 2z-1);
(67)
pa, (Ze|S1:t 21-1) = N Ma diag (a)))(
(68)
[MO_ Vo. (he, Zt).
(69)
po, (Sr 2,) N Nst; nra diag(vo,))
(70)
32

---

Published in Transactions on Machine Learning Research (12/2023)
Decoder
MLP
pa, 11.1
S
MLP
LSTM
h2
MLP
Po, (s,(5,22
Sample
9s, (1,12-111111)
S
MLP
Encoder
7-11
Figure 5: Schema of the SRNN model architecture. The "plus" symbol represents the concatenation of the input vectors.
where the function d, in 66 is implemented by forward RNN and h, denotes the RNN hidden state vector, the dimension of which is set to 8 for MOT and 128 for SC-ASS, In practice, LSTM networks are used. The function d, in 69) is implemented by dense layer of dimension 16 for MOT and of dimension 256 for SC-ASS, with the tanh activation function, followed by a linear layer, which outputs are the parameters po,: v0.-- The function d, in (67 is implemented by two dense layers dimension 8, 8 respectively for MOT and of dimension 64, 32 respectively for SC-ASS, with the tanh activation function, followed by a linear
layer, which outputs are the parameters p 10..
The SRNN inference model in the right-hand side of 37 is implemented as:
(ebr Voy1 = ez (he, St, Z1-1),
(71)
qo. (z. 22 11 = [2 Mor diag(vs.)).
(72)
where the function 6x in 77 implemented by two dense layers of dimension 16 and 8 respectively for MOT and of dimension 64 and 32 respectively for SC-ASS, with the tanh activation function, followed by linear layer, which outputs are the parameters Moo, Vaz.
The SRNN architecture is schematized in Figure It can be noted that the RNN internal state h, cumulating
the information on S1a-1 is shared by the encoder and the decoder, see GGrin et al. 2021 Chapter 4) for a discussion on this issue.
E
MOT dataset processing
E.1
Synthetic trajectory dataset generation
To generate bounding boxes with reasonable size, generate the coordinates of the top-left point (noted as øi and xF-) plus the height (noted as az) and width (noted as by) of the bounding boxes and deduce the coordinates of the bottom-right point. The width-height, ratio is sampled randomly, and kept constant during the trajectory. While the trajectory of one coordinate is generated using piece-wise combinations of elementary functions, which are: static a(t) dg, constant velocity a(t) = a,t + ag, constant acceleration a(t) azt² +att+al an, and sinusoidal (allowing for circular trajectories) a(t) = a sin((w + Φo). That is say, split the whole sequence into several segments, and each segment. is dominated by certain elementary
33

---

Published in Transactions on Machine Learning Research (12/2023)
Algorithm 3 Syuthetic trajectories generation
Input:
Total sequence length T;
Maximum sub-sequence number Smazi
Distribution parameters Mbo Tbe pr. dr, May, O0i Pog: O0g1 Muy Ows Mooo Tboi
Discrete probability distribution of different elementary trajectory function types [p1, P2. Pr]:
Output:
Synthetic bounding box position sequence gen_seg ((1), 21, 21,;
1: function GENSEQ(IO, 8, laplit, params prob, p)
2: start = -o;
3: for t 0 to 8 do
4: Sample function_typp type using p;
Sample trajectory function parameters params list using porams_ prob;
t. = teptaal]]]
sus, GenTraj(start, fune_type, params lisl);
8:
start = x =x_sub [ty]:
10: end for
11: x = (z_suby, x subb-1l;
12: return x;
13: end function
14: Sample wn, yo from 4(0, 1);
15: Sample by from log. N (Pbo Abul;
16: Sample Fab from M(Hr, Ty);
17: Randomly sample in {O,..., Smaz);
18: Randomly sample Epplt = (oo, ..., t--1) } in {I...., T);
19: x GenSeq(xo, 5, taplit, params prob, p);
20: y GenSeq (No: s, tapise, params_ prob, p):
21: b GenSeq (Wi; 8, laplit, params prob, p);
22: b Tabi
23: gen seq b,
function. An example of a 3-segment combination could be:
au azt² + an + do ag sin(wt do)
1 1
a(t) -
h 5 <22
(73)
ti st.T,
where the segments length is sampled from some pro-defined distributions to generate reasonable and con- tinuous trajectories. The number of segments is first uniformly sampled in the set {1...., Smax). We then sample segment lengths that sum up to T. This defines the segment, boundaries 44, For each segment, one of the four elementary functions is randomly selected. The function parameters are sampled as follow: a1 ~ [(H,,, o ), 02 ~ ((aa o2g), w ~ (Mu,, o.) and do ~ N(Hou 222). The two remaining parameters, ag and a, are set to the values needed to ensure continuous trajectories, thus initialising the trajectorics at every segment, except for the first one. The very initial trajectory point is sampled randomly from 4(00 1). And the initial width sampled from log-normal distribution by log xh,). Finally, the ratio between the height and width is supposed to be constant with respect to time. It is sampled from log-normal distribution lab & ~ (ur, of) and the height is obtained by multiplying the width and the ratio. More implementation details can be found in Algorithm 3
In our experiments, the total sequence length of the generated trajectories for DVAE pre-training equals T = 60 frames. And the maximum number of segments set to Smaz = The parameters of the a 02, w, Φu, Wo, and The distributions are determined by estimating the statistical characteristics publicly published detections of the MOT17 training dataset. More precisely, we estimated the empirical mean and
34

---

Published in Transactions on Machine Learning Research (12/2023)
standard deviation of the speed and acceleration for all matched detection sequences (i.c., the first and second order differentiation of the position sequences).
E.2 MOT17-3T dataset construction
To construct the MOT17-3T dataset, first, we matched the detected bounding boxes to the ground-truth bounding boxes using the Hungarian algorithm (Kuhn, 1955) and retained only the matched detected bound- ing boxes (i.e., the detected bounding boxes that were not matched to any ground ground-truth bounding boxes were discarded). The cost matrix were computed according to the the Intersoction-over- Union (IoU) dis- lance between bounding boxes. We split each complete video sequence into subsequences of length T (three different values of T are tested in our experiments, as detailed below) and only kept the tracks with length shorter than T' For each subsequence, we randomly chose three tracks that appeared in this subsequence from the beginning to the end. The detected bounding boxes of these three tracks form one test data sample. We have tested three values for the sequence length T to evaluate its influence on the tracking performance of our algorithm: 60, 120, and 300 frames (respectively corresponding to 2, 4, and 10 seconds at 30 fps). Among the three public dctection results provided with the MOT17 datasct, SDP has the best detection performance. So, we used the detection results of SDP create our dataset.
F MOT baselines implementation details
ArTIST Salch et al. (2021) is a probabilistic auto-regressive model which consists of two main blocks: MA- Net and the ArTIST model. MA-Net is recurrent autoencoder that is trained to learn a representation of the dynamical interaction between all agents in the scene. ArTIST is an RNN that takes as input a 4D velocity vector of the current frame for one object as well as the corresponding 256-dimensional interaction representation learned by MA-Net, and outputs a probability distribution for each dimension of the motion velocity for the next frame. As indicated in Saleh et al.] (2021), the models are trained on the MOT17 training set and the Track Manen et al. (2017) datasot. We have reusod the trained models as well as the tracklet scoring and inpainting code provided by the authors and reimplemented the object tracking part according to the paper, this part was not provided. The tracklets are initialized with the bounding boxes detected in the first frame. For any time frame t, the score of assigning a detection O&k to tracklet n is obtained by evaluating the likelihood of this detection under the distribution estimated by the ArTIST model. The final assignment is computed using the Hungarian algorithm. For any tentatively alive tracklet whose last assignment is prior to t-1 with a non-zero gap (implying that there exists a detection absence), the algorithm first performs tracklet inpainting to fill the gap up to t- 1, then computes the assignment score with the inpainted tracklet. As described in (aaleh et al. (2021), the inpainting is done with multinominal sampling, and tracklet rejection scheme (TRS) is applied to select the best inpainted trajectory. In order to eliminate possible inpainting ambiguities, the Hungarian algorithm is run twice, once only for the full sequences without gaps and the second time for the inpainted sequences. The number of candidates for multinominal sampling is set to 50. For the TRS, the IoU threshold used in Saleh et al. (2021) is 0.5. In our test scenario, there are less tracklets and the risk of false negative is much greater than that of false positive. So, we decreased the threshold to 0.1, which provided better results than the original value.
G More MOT tracking examples
The first example plotted in Fig. G Gillustrates the case where two persons cross each other. This is one of the most complicated situations that may cause an identity switch and even lead to tracking loss. Considering the limited space for the figure, of crossing. For t = 60, when the ground-truth bounding boxes of Sources 2 and 3 (32 and 83 in the figure) strongly overlap, Detection 02 disappears. Again, ArTIST exhibits frequent identity switches. Besides, at f = 20, the estimated bounding box mi is totally overlapped with that of m3. And at t = 90, the estimated bounding boxes for all of the three sources are getting very close to each other. This indicates that the identity switches can cause unreasonable trajectories estimation. For VKF, the observations for both Sources 2 and 3 are assigned to the same target 83 all along the sequence, due to 82 and 83 being close
display the bounding boxes every ten frames to view the whole process
35

---

Published in Transactions on Machine Learning Research (12/2023)
$87
SEP
8
$12
Groad Iist
4
Detestion
R/z
/0l2
<
(a) Example 1: Crossing sources.
Truth
MALVAE
(b) Example 2: Crossing sources with frequent detection absence.
Figure 6: Examples of tracking result obtained with the proposed MixDVAE algorithm and the two baselines. For clarity of presentation, the simplified notations $1, 01, and m, denote the ground-trun -rrth source position, the observation, and the estimated position, respectively (for Source 1, and the same for the two other sources). Best seen in color.
to each other, that the estimated bounding boxes m2 and ma overlap completely. For the Deep AR, the estimation of ma becomes inaccurate from 70 and it disappears t 80 and t = 90 (the estimation is out of the frame). In contrast, MixDVAE displays a consistent tracking of the three sources. For f = 50, 60, and 70, the estimated bounding boxes my and my overlap due to the ground-truth bounding boxes 82 and 83 strongly overlap each other. However, the tracking is correctly resumed at 1 = 80, with no identity switch (i.e., the crossing of Sources 2 and 3 is correctly captured by the model).
The second example displayed in Fig. 0 another more complicated situation with two sources very close to each other and frequent detection absence. At. 20 when observation 03 disappears, both ArTIST and
36

---

Published in Transactions on Machine Learning Research (12/2023)
Speech
Flute
(a) Mixture
(b) Ground Truth
(c) VKF-Oracle
Speech
Flute
(d) DVAE-init
(e) VKF-DVAE-irg init
(1) MiXDVAE
Speech
Flute
(a) Mixture
(b) Ground Truth
(c) VKF-Oracle
Speech
Flute
(d) DVAE-init
(e) VKF-DVAE-init
(f) MiXDVAE
Figure 7: Examples of audio source separation results obtained with the proposed MixDVAE algorithm and the baselines. Best seen in color.
37

---

Published in Transactions on Machine Learning Research (12/2023)
Table 4: Results obtained by MixDVAE on MOT17-3T (short sequences subset) for different values of rap The values on the left (resp. right) side of the slashes are obtained without (resp. with) the iinetuning of SRNN in the E-Z Step.
MOTA!
MOTPI
IDFT
#1Ds
%IDs,
MTT
MLJ
#PP
%PPL
#FN1
XFN,
0.01
35.9/32.8
84.5/84.8
06.6/05.5
4914/3216
1.6/1.0
2946/2711
916/913
96433/ 1020622
31.3/33.1
96435/ 102062
06435/102062
31.3/33.1
0.02
65.5/6618
84.2/84.7
813/77.88
5319/3073
1.7/1.0
3932/3652
407/379
50596/57291
16.4/18.6
50596/87291
16.4//8.6
0.03
74.9/70.0
83.1/84.3
86.1/84.4
5088/2885
1.7/0.9
4232 /3931
155/160
36165/4377
11.7/14.2
36165/43777
11.7/14.2
0.04
79.1/75.1
813//83.5
88.4/86.7
1966/2862
1.6/0.9
4370/ 4U67
50/66
29808 (6699)
9.7/11.9
29808 /360000
9.7 9.7/11.9
0.05
78.4/75.6
79.2/82.6
$7.1/87.1
4982/2919
1.6/0.9
4268 4066
4268/4066
42/53
339/44 36088
11.0/11.7
33924, /36088
11.0/11.7
09.2/70.1
76.9/82.0
83.3/814
1.7/1.0
T3/
14.3/1155
44793;
14.5/44.
0.07
09,8,06688
74.8/80.3
78.9/82.9
6146/3000
1.7/L.0
3688 /3776
188 (88/288
50348 19646
19.2/16.1
69348/49666
19.2/16.1
0.08
18.5/6006
73.1/79.4
13.3/79.9
5087/3119
1.7/1.0
3303 /3637
337/ 132
76865/59220
24.9/19.2
TG865/592220
21.9/19.2
r_phi value
0.01
0.02
0.03
0.04
0.05
0.06
0.07
0.08
0.7
MOTA values
0.6
0.5
0.4
0.3
60
Iterations
Figure 8: MOTA score obtained by MixDVAE as function of the number of VEM iterations, for different values of rq-
VKF lose of the tracks, whereas MixDVAE keeps reasonable tracking of the three tracks. From / 60 to 80, both 02 and 03 absent. The tracks inpainted by ArTIST are not consistent anymore and VKF still misses one track. The estimations of Deep AR are inaccurate when the detections are absent. However, even in this difficult scenario, MixDVAE keeps on providing three reasonable trajectories.
More SC-ASS examples
In Figure plot two other SC-ASS examples.
Ablation study
In order to better understand the MixDVAE model, we conducted ablation studies on the influence of the pre-trained DVAE model quality, the influence of fine-tuning the DVAE, and the influence the observation variation matrix ratio 74.
1.1 Influence of the pre- pre-trained DVAE model quality
We have conducted an ablation study on the influence the pre-trained DVAE model quality on the whole MixDVAE algorithm, for both the MOT task and the SC-ASS task. Specifically, we have pre-trained the DVAE model different data scales and tested the performance of MixDVAE using these different pre- trained models.
38

---

Published in Transactions on Machine Learning Research (12/2023)
Table 5: Capacity the SRNN model pre-trained at three data scales the synthetic trajectories dataset.
SRNN-full, SRNN-half, and SRNN--quater stand for SRNN pre-trained on the totality, half of and quarter of our original training set, respectively.
Model name
Training loss
Validation loss
SRNN-full
-40.77
-40.15
SRNN-half
-39.36
-38.86
SRNN-quarter
-36.55
-35.27
Table 6: MOT results obtained by MixDVAE with SRNN pre-trained at the three data scales, The results are reported for the short sequence test subset (T = 60 frames).
Model name
MOTA
MOTP1
IDF1
# #TDs
%IDs|
MTT
MT.
#FP/
%FP4
#FN
%FN,
SRNN-full 79.1
81.3
88.4
4966
1.6
4370
50
29808
9.7
29808
9.7
SRNN-half
74.7
84.4
86.6
5624
1.8
4039
94
38153
12.4
38153
12.4
SHNN-quarter 75.2
84.4
86.9
5598
1.8
4040
91
37443
12.2
37443
12.2
Table 7: Capacity of the SRNN model pre-trained at three data scales of the WSJO and the CBF datasets. SRNN-full, SRNN-half, and SRNN-quarter stand for SRNN pre-trained on the totality, half of and quarter of our original training set, respectively.
WSJO
CBF
Model name
Training loss Validation loss
Training loss
Validation loss
SRNN-full
353.89
373.61
521.76
779.69
SRNN-half
358.13
389.58
489.53
949.11
SRNN-quarter
361.58
383.64
646.55
1106.27
MOT task. We conducted pre-training of the SRNN model three separate datasets with different scales: the full synthetic trajectories training set used in Section 5 consisting of 12,105 trajectorics, dataset containing half of these synthetic trajectories, randomly selected (6,052 trajectories). and another dalaset with a quarter of these synthetic trajectories, randomly selected (3,026 trajectories). We use the ELBO loss to represent the quality of the resulting pre-trained SRNN models. The ELBO loss values are reported in Table 5] As expected, we observe that by decreasing the training data size, the performance of the SRNN model drops (with higher training and validation loss).
We run the MixDVAE inference algorithm with the three pre-trained SRNN models the short sequence test subset (T 60 frames), and the obtained results are reported in Table 6 We see that the overall performance of the MixDVAF algorithm with SRNN-half and SRNN-quarter drops compared to that with SRNN-full, but this drop is relatively limited, at least for some of the metrics, including the key MOTA metric. Moreover, the difference between the performance of MixDVAE with SRNN-half and with SRNN- quarter is quite small. Therefore, even it hard to draw a general conclusion from a single experiment with three dataset sizes, this scems to indicate some robustness of MixDVAE w.r.t. the DVAE training datasct size, and confirm its interest as data-frugal weakly supervised method (here for the MOT application).
SC-ASS task. Similar to the MOT task, also generated two additional subsets of the training data for both the WSJO and CBF datascts, comprising half and quarter of our original training datasct (used in Section 6 randomly selected. The two new subsets of WSJO contains 12.45 and 6.29 hours of speech recordings respectively. And the two new subsets of CBF contains 1.07 and 0.55 hours of CBF recordings respectively. The performance of the SRNN model pre-trained on these different datasets reported Table 7 For the WSJO dataset, we observe that the training and validation losses are relatively close each other, and both increase when decreasing the training data size, but the increase is moderate. Therefore,
39

---

Published in Transactions on Machine Learning Research (12/2023)
Table 8: SC-ASS results obtained by MixDVAE with SRNN pre-trained at the three data scales. The results are reported for the short sequence test subset (T 50).
Speech SI-SDR t
Chinese bamboo flute
Model name
RMSE
PESQ t
RMSE
SI-SDR 1
PESQ t
SRNN-full 0.006
9.23
1.73
0.007
13.50
2.30
SRNN-half 0.006
9.66
1.82
0.009
12.29
2.28
SRNN-quarter 0.007
8.83
1.79
0.011
10.29
2.13
Table 9: MOT results obtained by MixDVAE with and without the fine-tuning of SRNN. The results are
reported for the short, medium and long sequence test subsets (T = 60, 120, and 300 frames, respectively).
Dataset
Fine-tuning
MOTAI
MOTP!
IDFIt
#IDs|
%IDs]
MT1
MLJ
#FP1
%FP.
#FN
%FN.
Yes
75.1
83.5
86.7
2862
0.9
4067
64
36990
11.9
36990
11.9
Short
No
79.1
81.3
88.4
4966
1.6
4370
50
29808
9.7
29808
9.7
Yes
73.1
84.0
85.9
3044
0.7
2705
136
54604
13.1
54604
13.1
Medium
No
78.6
82.2
88.0
6107
1.5
2907
120
41747
9.9
41747
9.9
Yes
65.6
84.9
81.6
8670
2286
67
171515
13.8
171515
13.8
Long
No
83.2
82.4
90.0
23081
2.3
2890
12
74550
7.3
74550
7.3
the capacity of SRNN drops, but quite slightly. However, for the CBF dataset, the gap between the training and validation losses is higher, and the training loss of SRNN-half decreases compared to SRNN-full while the validation loss increases significantly, increasing the gap. Both the training loss and the validation loss of SRNN--qarrer are higher than that of SRNN-full and SRNN-halt and the gap between training and validation is also relatively large. This shows that the size of the (full) CBF dataset may be a bit too limited, and reducing this dataset may harm the generalization capacity of SRNN.
The SC-ASS results obtained by MixDVAE with SRNN pre-trained at the three different data scales are reported in Table [ The experiments conducted the short sequence subset (T 50). We find that, surprisingly, the separation performance of MixDVAE with SRNN-half on the speech signals has been slightly improved over SRNN-full, whereas (much less surprisingly) the performance on the CBF signals has decreased. This may be caused by the lower generalization ability of SRNN-half on the CBF dataset. For SRNN-quarter, the performance of MixDVAE on both the speech and the CBF decrease, but the decrease for the speech quile moderate (0.4 dB SI-SDR w.r.l. SRNN-full; the PESQ value even slightly better), whereas the CBF loosing about 3.2 dB SI-SDR. Again, even is difficult to draw general conclusion from this single experiment, those results seem indicate relative robustness of MixDVAE to the limitation of the DVAE training dataset size, provided that the DVAE keeps sufficient generalization capability.
1.2 Influence of the DVAE fine-tuning
As mentioned in Section 4.2
the DVAE model can either be fine-tuned or not in the MixDVAE algorithm. We have studied the effect of fine-tuning SRNN on both MOT and SC-ASS tasks.
MOT task. Table D shows the MOT scores obtained by MixDVAE on the three test subsets with and without the finc-tuning of SRNN in the E-Z step. We observe that for all three datasets, not finc-tuning the DVAE model leads to the best overall performance (as measured by MOTA in particular). Though Ene-tuning the DVAE model can indeed increase the MOTP score and decrease the number of identity switches, it does not improve the overall tracking performance. Indeed, fine-tuning increases the FP and FN numbers/ proportions, and thus decreases the MOTA scores. Especially on the long sequence dataset, the MOTA score drops from 83.2 to 65.6.
SC-ASS task. Table 10 shows the performance of MixDVAE on the three test subsets with and without fine-tuning SRNN in the E-Z step. Similar to the MOT task, for all three datasets, not fine-tuning SRNN leads to the best overall source separation performance (on all of the evaluation metries).
40

---

Published in Transactions on Machine Learning Research (12/2023)
Gruna A
+
+
4
Desection
Wilwet
A
APF
A
Fine
F02
FB3
With
AP
(8%
(a) Example
S
Grond
52
50
Detection
03
au
Tha sadin
m2
rR:
m3
<8:
(8)
012
m2
Wi
m0
01
702
702
m83
M3
my (b) Example
m
Figure 9: Two examples of tracking result obtained with the proposed MixDVAE algorithm, with and without fine-tuning during the E-Z step. For clarity of presentation, the simplified notations $1, 01, and my denote the ground-truth source position, the observation, and the estimated position, respectively (for Source 1, and the same for the two other sources). Best in color.
Table 10: SC-ASS results obtained by MixDVAE with and without the fine fine-tuning SRNN. The results are reported for the short (T 50), medium (T 100) and long (T 300) test sequence subsets.
Speech
Chinese bamboo flute
Dataset
Finetuning
RMSE 4
SI-SDR
PESQ t
RMSE + SI-SDR 1
PESQ 1
Yes
0.007
8.00
1.63
0.007
12.73
2.15
Short
No
0.006
9.23
1.73
0.007
13.50
2.30
Yes
0.008
8.00
1.55
0.008
12.23
2.02
Medium
No
0.007
9.32
1.65
0.007
13.05
2.16
Yes
0.008
7.02
1.49
0.008
11.40
1.88
Long
No
0.007
9.06
1.64
0.007
12.92
2.06
41

---

Published in Transactions on Machine Learning Research (12/2023)
Speech St- SI-SDR values
MOTA values
Flute 51-SDR
VEM iterations
VEM iterations
VEM iterations
Figure 10: Evolution of the performance of MixDVAR as a function of the number of VEM iterations (MOTA score for the MOT task and S1-SDR scores for the SC-ASS task).
We therefore observe that for both tasks, fine-tuning the DVAE model results in performance degradation. The possible reason is that fine-tuning could make the model more semsible to observation noise, and lead to a generative model with worse performance. To verify this conjecture and to better understand the effect of fine-Luning, we have plotted in Figure g two examples for the MOT lask, extracted from the long sequence test subset ( = 300 frames). To make possible the display of a long sequence in a limited space, the first example is plotted every 5 frames, whereas the sccond example is plotted every 10 frames. In Example 1, we observe that the detection for source 8y is missed for t 0, t 10, 15, t 25 and t = 45. At these frames, MixDVAE without SRNN fine-tuning can still make good estimation of sy's position, whereas MixDVAF with SRNN fine-tuning can not make an accurate prediction. In the latter case, this caused a large егror between the estimated source position and the ground truth. We can see a similar phenomena in Example 2. At frame = 30, 40, 50 and 60, when the detection bounding box for source 83 is absent, the estimation obtained by MixDVAE with SRNN fine-tuning bad (it is particularly bad for t = 40). This phenomenon confirms our conjecture that the observation noise, particularly the lack of observations, can introduce unforeseen effects during fine-tuning, resulting in model with degraded performance.
1.3 Influence of the observation variance ratio
Table reports the MOT scores obtained with MixDVAE as a function of ræ- These experiments are conducted on the subset of short sequences. We report the results for both with and without fine-tuning SRNN in the E-Z step. Apart from the value of ræ and the fine-tuning option, all other conditions are exactly the same across experiments. Table | shows that, whether fine-tuning SRNN in the E-Z step not, the MOT scores first globally increase with re reach their optimal values for T = 0.04 0.05 have also computed the
(for most metrics), and then decrease for greater r4 values. For confirmation, averaged empirical ratio f& of the detected bounding boxes (with the SDP detector), which is calculated as This value equals to 0.053, 0.053 and 0.047 respectively for the short, medium and long sequence datasct. These values, which are close to cach other because we used the same delector, correspond well to the r value for the best performing model in Table 4 We can conclude that the model has better performance if the value of r corresponds (empirically) to the detector performance. Besides, we have also observed that the value of r& has an impact on the convergence of the MixDVAE algorithm. Fig. g displays the MOTA score as function of the number of MixDVAE iterations (here with the fine-tuning of the DVAE model). Tt appears clearly that for too high values of re: the model exhibits lower and more hectic performance than for the optimal value.
+ H (1 - |sj - - 162--0 04-0
oi - oi
-
E'xeeept for the MOTP score, which continually decreases with the increase of rg. This can be explained follows. MOTP measures the precision of the position estimation for the matched bounding boxes. The estimated position in 28 is weighted combination of the observation and the DVAE prediction. When a increases, the contribution of the observation decreases and man is closer to the DVAE prediction. Since the error of the DVAE prediction may accumulatc over time, this finally decreases the position estimation accuracy.
Note that here 8ik denotes the position of the target matched with the observation on time frame We omit the target. positions that are not matched with any observation.
42

---

Published in Transactions on Machine Learning Research (12/2023)
Table 11: Averaged processing time per sequence for the MOT task.
Sequence length (frames)
60
120
300
# sources
3
3
6
Computation time per sequence (s)
23.01
57. 27.29
45.05
110.41
112.93
272.94
Table 12: Pre-training computational cost on different datasets at different scales.
Task
Data set
Data scale
One epoch training time (s)
Full
15
MOT
Synthetic trajectories
Half
7.8
Quarter
4.8
Fall
190.8
SC-ASS
WSJO
Half
121.2
Quarter
63
J Discussion on the computational complexity
The proposed method is based on two parts: (i) the pre-training of a DVAE model on a single-source dataset, and (ii) the MixDVAE variational EM algorithm for source tracking. The computational cost for the pre- training stage mainly depends on the data type and data size of the single-rraeectory dataset. To give a general idea, we measured the average training time required for a single epoch (iteration over the whole training set) on both the synthetic trajectories dataset for MOT and the WSJO dataset for SC-ASS. The measurement is conducted on an NVIDIA Quadro RTX 8000, in machine with an Intel(R)) Xeon(R) Gold 5218R CPU @ 2.10GHz. The obtained results for different data scales as mentioned in Section L. is reported in Table 12 We have observed that doubling the size of the training data results in almost a doubling of the training time. On the other hand, the computation complexity of the MixDVAE E algorithm mainly depends on three factors: the number of VEM iterations, the number of sources to track and separate, and the sequence length. Typically, the performance of MixDVAE exhibits an initial rapid increase over the VEM iterations, followed by stabilization towards plateau. In Figure 10] we plot. the evolution of the averaged performance of MixDVAE over the medium sequence test dataset performance is represented by the MOTA score for the MOT task and by the S1-SDR score for the SC-ASS task). We observe that for the MOT task, the performance of MixDVAE has been stabilized from around 10 iterations, whereas for the SC-ASS task, the performance has been stabilized from around 20 iterations. In practice, we run the algorithm for more itcrations to guarantee the convergence. Taking computational time optimization into account, it possible to identify an optimal number of iterations by applying a grid search, for a specific task and dataset. To quantify the computational time of the MixDVAE algorithm, we compute the averaged processing time for one sequence the MOT task, for the three considered values of the sequence length, and for the case of 3 and 6 sources. This average processing time is measured NVIDIA Quadro RTX 4000 GPU, in machine with an Intel(R) Xeon(R) W-2145 CPU@3.70GH4, and it is averaged on 10 test sequences. The results are reported in Table II We observe a linear increase of the computation time as a function of the sequence length. As the number of sources to track doubles, the computation time exhibits more than a twofold increase. The computation complexity can be bottleneck for the MixDVAE especially for long sequences with large number of sources. However, further algorithm and code optimization might be possible, since did not focus on this aspect of the problem so far.
function of the number of VEM iteration (the
43


Reviews:
[{"review_id": "HRUjGjkad6", "reviewer": "Reviewer_UTAB", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The method is scalable (weakly supervised) as one can train on independent source data and combine the instances of models. The results are very convincing since the methodology compares (on certain metrics) and proves superior (on other metrics) to supervised alternatives. \n\nIt is slightly difficult for the reader to discern whether a simpler framework could have done equally well. The study could become much stronger if the authors included an ablation study on the latent variables of the model.\n\nSince it is a rather complex methodology proposed, it would be beneficial to the reader to understand training complexity. It would also be good to understand the practical way of choosing the variation matrix ratio.\n\nWhen fine-tuning the SRNN it seems like performance has a tendency to decrease. You write:\n\"The possible reason is that fine-tuning could make the model more sensible to detection noise, and lead to a generative model with worse performance.\"\nIt would be nice to understand this better, and it would be interesting to see it visualized through experiments.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Finding scalable methods to track individual temporal sources in a mixed signal is still an unsolved problem. The authors exemplify this through two applications: audio source separation and multi-object tracking. There exists a trove of other applications that would be interesting to analyze further with this method. The introduced method adds a DVAE, in this study an SRNN, into a larger framework optimized through EM. The true power of the method is that it allows for only training on the distinct sources, hence, there is no need for a dataset that holds both the separate sources in combination with the mixed sources.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Ablation study of the model and the training method (where possible)\n2. An explanation of the training complexity\n3. An analysis of the problem with fine-tuning the SRNN during optimization", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "5PfunYh6wk", "reviewer": "Reviewer_PS4D", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\n— The method is a natural extension of the sequential VAE, and being able to model the complicated multi-source mixture of trajectories. \n\n— The proposed variational expectation maximization algorithm looks sound.\n\n— The proposed methods show overall better or comparable performance compared to the baseline systems that were compared with.\n\n\nWeakness:\n\n\n\n— The impact of the DVAE is not carefully studied. How critical is the pre-trained DVAE? If pre-trained DVAE is of relatively low quality, will it heavily affect the whole system? I would suggest the authors study DAVE pre-training in different data scales, and show their effect on the whole system. Also, I would recommend the authors compare the whole system with/without fine-tuning the pre-trained DVAE. \n\n\n\n— The complexity of the algorithm should be discussed. Overall, the algorithm’s computation positively relates to the number of iterations of the EM, number of sources, number of assignments, and also length of the sequence. How’s the performance evolve as the number of EM iterations goes on? What’s the computation given a different number of sources and sequence length? \n\nThe authors would need some more ablation study to demonstrate the proposed method can scale up to large datasets, or to state computation cost is one bottleneck. The pre-training of DVAE should also be considered as part of the computation cost.\n\n\n\n— It’s unclear to me if the presented approach is a ‘plug-in’ system, or needs some adaptation for using a different sequential VAE system. \n\n\n\n— The generalization to unknown sources is unknown. Basically, the whole system assumes the number of sources are already known. What if there is a new type of source that is unknown during inference, or the number of sources is beyond N? What if there are some unseen types of data that during inference but not in training?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose a latent-variable generative approach for modeling the multi-source trajectory. The proposed approach is called MixDVAE, which extends the existing sequential variational models (DVAEs as mentioned by the authors) with the ability to model multiple sources. The optimization is done first by pre-training a DVAE system for modeling single source data, and then by using the variational expectation-maximization algorithm to estimate the assignment variable and the representation of each source. \n\nThe authors demonstrate the effectiveness of their methods in two tasks — multi-object tracking and single-channel audio source separation.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Overall, the paper is easy to follow, and the overall idea sounds reasonable.\n\nI would recommend the authors to address the bullet points regarding the potential weakness I listed in the ‘strength/weakness’ section. \n\nFinally, as a generic method described in a long paper, the authors could also consider applying their task to one more domain (e.g., processing text), and see how things are going on.", "broader_impact_concerns": "I did not find anything that could have potential negative impact to the society.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "BsrbHYrrIX", "reviewer": "Reviewer_Bi6u", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n- The paper describes the learning and the inference algorithm in great detail. \n- Experimental results indicate slight performance improvement over the other methods that the authors compare against. Though I am not sure if this comparison is exhaustive / extensive. \n\nWeaknesses\n- I am not an expert on multi-target object tracking. But on audio/speech source separation the proposed algorithm seems rather restrictive. The proposed method requires pretraining on specific source types (similar to generative source separation with NMF), which limits the general applicability of the method. These days we are able to train source separation methods in a source-indepent way using techniques such permutation invariant loss, or embedding methods such as Deep=clustering based source separation. And, even if the models are trained with source independent methods on speech separation it is possible to obtain SI-SNR values above 20dB. Given this state-of-the-art I am not sure what is the benefit of the proposed approach on speech separation.\n- In absolute sense the source separation results do not seem very good. (Below 2.00 PESQ, 10dB for speech) \n- The paper seems very long and verbose. I might be possible to compress it and make it more to-the-point. \n- I am not very sure about how novel the proposed model is. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors propose a multiple source variational encoder model. In this proposed model, each source is assigned in it's own VAE, and there is a pretraining step for each source separately, which if I understand correctly restricts the model to the specific type of source it was trained on. \n\nThe authors propose an EM type algorithm for learning the model parameters. The evaluation is done on multi-source tracking and audio/speech source separation. \n\n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- It would be good to compare with state-of-the-art speech/audio separation models. This can include supervised source separation models such as Sepformer, or unsupervised models such as Mixit. At its current state, I don't understand the benefit of using the proposed method in source separation. For generative source separation, you can also compare with vanilla NMF, or it's time dependent versions. \n- Overall, the contributions are not very clear to me. It would be good to include an itemized contributions list at the end of section 1.2. \n- If possible you can shorten the paper. It would be better to highlight the advantages of the approach rather than focusing on the technical details. ", "broader_impact_concerns": "Nothing very obvious stands out to me to point out as a concern. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors engaged with the reviewers in the discussion period providing a very thorough rebuttal, addressing most of the raised concerns. Specifically, apart from adding several clarification notes, the author incorporated baselines (e.g. deep AR, LSTM, Mixit and NMF with variants), included ablation studies showing how the performance of the algorithm changes with the quality of the pre-trained DVAEs and detained how fine-tuning the DVAE at inference time affects performance. The updated version of the paper is clearer describing the strengths and limitations of the approach. The quality of the manuscript has been significantly improved.  

Both reviewers PS4D and UTAB lean to accept the paper and consider that all their concerns were addressed by the authors. Reviewer Bi6u leans to rejecting it, their main concern being the relevance of the results in source separation.  

Specifically, the reviewer pointed out that the performance of the proposed method is very low compared to the state of the art, even considering source-independent methods. One could add that the problem tackled here (separating speech from a musical instrument) is not a particularly challenging task, as the spectral characteristics and dynamics of the two sources are quite different. The authors acknowledged this point in their response, and stated that the purpose of this example is to show the versatility of the approach, and to show that the method works beyond MOT. While the AE agrees with the points raised by the reviewer, it also finds the explanation provided by the authors satisfactory.

It is certainly positive that the model requires a small amount of data. However, it is hard to make this case when using speech signals for which huge amounts of (cheap) data is available. Also, many methods work by synthetically generating the mixture signals requiring the same type of data than models training source-specific models (but more computational cost). This method however could be used in applications for which training data is not that easily available. The paper would make a stronger case in one of these scenarios. Having said that, the baselines included by the authors operate in the low data regime and are the strongest methods in it.Thus the results are not insignificant and certainly serve the purpose of showing that the MixDVAE is able to model the dynamics of complex signals and perform the separation (beyond MOT).

Below some other aspects discussed in the reviewing process.

One limitation pointed out by Reviewer PS4D is that the model cannot handle unknown types of signals, meaning a DVAE needs to be pre-trained using available data similar to the one that will be encountered at inference time. This is an natural assumption for both applications presented in this work, and seems reasonable to the AE.

Reviewers PS4D wanted an experimental study showing how the performance of the model degrades with the quality of the pre-trained DVAE(s). The authors included a detailed ablation study addressing this point by pre-training the DVAE model at different data scales, for both examples presented in this work. Results indicate that the degradation is very smooth, showing robustness with respect to the source models.  

Another concern raised by reviewers PS4D and UTAB is the effect of fine-tuning the DVAE model during the E-Z step of the algorithm. The original manuscript already included some initial results on this showing a reduction in overall performance when including the fine-tuning step. The authors further added experiments to study the root causes of this effect. Given that this step is certainly costly and makes the algorithm less practical (e.g. harder to deploy), the authors should make clear that this is the recommended way of using it. Particularly in Section 4.

Reviewer PS4D noted that when using a specific DVAE model, small adaptations are required in the MixDVAE algorithm. However, they don’t find this a reason to reject the work. The AE agrees with this view.

The computational complexity is one of the limitations of the proposed method. This is clearly acknowledged in the paper. In response to inquiries by Reviewer UTAB the authors revised the manuscript to include a study that shows how the performance of the proposed MixDVAE changes with the computational complexity (mainly the number of EM iterations, the number of sources to track and the sequence length at inference time). The AE finds the analysis insightful and Reviewer UTAB found it satisfactory.

Given the above, the AE recommends accepting the work and considers the paper a good contribution that will be of interest to many of the readers of TMLR.
Decision: Accept (poster)