## 3 Methodology
### DoA Estimation  
In a real-world environment, the sound source of interest is often out of the FOV of the camera. To locate and track such a target, the robot should first estimate its DoA and then rotate to find it. In our AV-SSLT, the DoA estimation is based on the SEVD-MUSIC algorithm [18] using another microphone array. MUSIC takes the transfer function from sound to each microphone as a prior. Consider a microphone array with \(N_{m}\) microphones (which is 4 in our system), the multichannel transfer function can be expressed as,  
\[\mathbf{H}(\theta,\omega)=[h_{1}(\theta,\omega),\dots,h_{N_{m}}(\theta,\omega)]^{ T}, \tag{1}\]  
where \(\theta\) is the azimuth of the sound source relative to the microphone array in the 2D case, \(\omega\) represents the frequency and \(h_{i},i\in(1,N_{m})\) is the transfer function from the sound to the \(i\)-th microphone. To perform sound source localization, we have to calculate the correlation matrix among the input signal channels. To achieve this, the signal vector in the frequency domain is firstly obtained via STFT of the input acoustic signal in \(N_{m}\) channels as,  
\[\mathbf{X}(\omega,f)=[X_{1}(\omega,f),\dots,X_{N_{m}}(\omega,f)]^{T}, \tag{2}\]  
Figure 2: The overall framework of AV-SSLT. To identify a target sound source of interest, the robot continuously collects sound signals and waits for keyword detection. Once a keyword is detected, acoustic features extracted from the sound signals and face features obtained from RGB-D images are compared with the database to identify the target. To localize and track the target speaker, the motion module controls the robot’s movement using the directions of arrival (DoA) estimated from the audio data. where \(f\) expresses the frame index. With \(\mathbf{X}(\omega,f)\), the correlation matrix among its channels can be further defined as follows for every frame and every frequency,  
\[\mathbf{R}(\omega,f)=\mathbf{X}(\omega,f)\mathbf{X}(\omega,f)^{*}, \tag{3}\]  
where \((\cdot)^{*}\) denotes the conjugate transpose operator. In order to be robust against noise, in AV-SSLT, \(\mathbf{R}(\omega,f)\) is time-averaged over \(T_{r}\) frames via,  
\[\mathbf{R}^{\prime}(\omega,f)=\frac{1}{T_{r}}\sum_{i=0}^{T_{r}-1}\mathbf{R}(\omega,f+i). \tag{4}\]  
To separate the speech signal and noise signal from the mixed signal, standard eigenvalue decomposition of \(\mathbf{R}^{\prime}(\omega,f)\) is conducted to decompose the signal space into the speech and noise subspaces,  
\[\mathbf{R}^{\prime}(\omega,f)=\mathbf{E}(\omega,f)\mathbf{\Lambda}(\omega,f)\mathbf{E}^{-1}( \omega,f), \tag{5}\]  
in which \(\mathbf{\Lambda}(\omega,f)=\text{diag}(\lambda_{1}(\omega,f),\ldots,\lambda_{N_{m }}(\omega,f))\) is a diagonal matrix composed of the eigenvalues of \(\mathbf{R}^{\prime}(\omega,f)\) in descending order and the \(i\)-th column of \(\mathbf{E}(\omega,f)=[e_{1}(\omega,f),\ldots,\)\(e_{N_{m}}(\omega,f)]\) is the eigenvector of \(\mathbf{R}^{\prime}(\omega,f)\) associated with \(\mathbf{R}^{\prime}(\omega,f)\)'s eigenvalue \(\lambda_{i}(\omega,f)\). Subsequently, the MUSIC spectrum \(\zeta\) for SSL can be written as follows,  
\[\zeta(\theta,\omega,f)=\frac{|\mathbf{H}^{*}(\theta,\omega)\mathbf{H}(\theta,\omega)| }{\sum_{i=N_{+}1}^{N_{m}}|\mathbf{H}^{*}(\theta,\omega)e_{i}(\omega,f)|}, \tag{6}\]  
where \(\mathbf{H}(\theta,\omega)\) is the multichannel transfer function defined in Equation (1) and \(N_{s}\) is an empirical parameter considered as the number of sound sources in the SSL process to remove the noise from the time-averaged correlation matrix \(\mathbf{R}^{\prime}(\omega,f)\). Note that the MUSIC spectrum defined by Equation (6) is for every frequency.