## 2 Proposed Approach
### Model Architecture
#### 2.3.2 Decoder  
The decoder part of the transformer is modified to efficiently decode the information in the embedded features for the prediction of sound events or source directions. In particular, the prediction can be improved by exploiting the related information from the other embedded features using the DCMA in the decoder. The overall schematic diagram of the decoder is represented in Figure 1.  
Similar to the encoder, the decoder is also split into SED and DOAE streams, each of which is based on the decoder part of the transformer, which was first proposed by [PERSON] et al. [20]. Although the transformer composed of attention mechanisms without convolution or recurrent layers has used the previous decoder output as the current decoder input to analyze sequence data [20], our decoder uses the embedded features for either SED or DOAE as the decoder input to analyze the current sound directly. As shown in Figure 1, the decoder in each stream is composed of a stack of identical layers. Each layer has two multi-head attention blocks followed by an FC network. The first attention block is the masked multi-head self-attention block using the common input embedding as the query, key, and value, while the second attention block is the multi-head attention based on the DCMA with embedded features for SED and DOAE (corresponding to the outputs of the first attention block) used as the queries and keys/values. Residual connections are employed around each of the blocks and the FC network, followed by layer normalization. Multi-head attention is a module for attention mechanisms which runs through an attention mechanism several times in parallel [20]. Then, independent attention outputs are concatenated and linearly transformed into the original dimension. Therefore, multiple attention heads focus on different aspects of input features during the attention process, and the outputs are concatenated to enable effective attention. In addition, since each attention mechanism is performed in a small dimension, it is possible to benefit from the amount of computation. After passing through the multi-head self-attention that is also employed in the EINV2 [13], we exchange the related information from the other features embedded through the multi-head self-attention in the two streams by applying the DCMA [21] instead of the encoder-decoder multi-head attention in the transformer. Although the DCMA was introduced to utilize both an audio context vector using video query and a video context vector using audio query in audio-visual speech recognition [21], the related information between the embedded features for SED or DOAE can be enhanced in learning the association between the two features by employing the DCMA architecture with the two different features used as the queries and keys/values. Using the scaled dot-product attention, attention outputs in the \(i\)-th head, \(\mathbf{A_{SD}}\) and \(\mathbf{A_{DS}}\), can be expressed as  
\[\mathbf{A_{SD}} = \text{Attention}(\mathbf{E_{S}},\mathbf{E_{D}},\mathbf{E_{D}}) \tag{9}\] \[= \text{softmax}\left(\frac{\mathbf{E_{S}W_{S}^{Q_{i}}(W_{S}^{\text {KV}_{i}})^{T}\mathbf{E_{D}^{T}}}}{\sqrt{1/d_{k}}}\right)\mathbf{E_{D}W_{S}^{ \text{KV}_{i}}},\] \[\mathbf{A_{DS}} = \text{Attention}(\mathbf{E_{D}},\mathbf{E_{S}},\mathbf{E_{S}})\] (10) \[= \text{softmax}\left(\frac{\mathbf{E_{D}W_{D}^{Q_{i}}(W_{D}^{\text {KV}_{i}})^{T}\mathbf{E_{S}^{T}}}}{\sqrt{1/d_{k}}}\right)\mathbf{E_{S}W_{D}^{ \text{KV}_{i}}},\]  
where \(\mathbf{E_{S}}\in\mathbb{R}^{N_{T}\times D_{F}}\) and \(\mathbf{E_{D}}\in\mathbb{R}^{N_{T}\times D_{F}}\) are SED and DOAE features used as inputs with \(N_{T}\) and \(D_{F}\) denoting the number of time steps and the feature dimension, respectively. \(\mathbf{W_{S}^{Q_{i}}}\) and \(\mathbf{W_{S}^{\text{KV}_{i}}}\) represent learnable query and key/value matrices for the \(i\)-th head in the SED stream, respectively. In the same way, \(\mathbf{W_{D}^{Q_{i}}}\) and \(\mathbf{W_{D}^{\text{KV}_{i}}}\) denote the corresponding matrices in the DOAE stream.