## 3 Proposed Fast Multichannel Parallel Acoustic Score Computation  
Using the baseline acoustic score computation, the number of concurrent clients is restricted due to the frequent data transfer between GPU and CPU and the low parallelization of the GPU [39]. To support more concurrent clients in real time, this section proposes a fast multichannel parallel acoustic score computation method by accelerating the GPU parallelization and reducing the transmission overhead. \begin{table}
\begin{tabular}{l c c} \hline \hline
**Thread** & **Name** & **Device** & **Size** \\ \hline Child\({}_{i}\) & \(\mathcal{M}_{feat\_vec}^{CPU}\) & CPU & \(\mathcal{N}_{bat}\) \\ Child\({}_{i}\) & \(\mathcal{M}_{feat\_mat}^{CPU}\) & CPU & \(\mathcal{N}_{win}\times\mathcal{N}_{seg}\) \\ Child\({}_{i}\) & \(\mathcal{M}_{prob\_vec}^{CPU}\) & CPU & \(\mathcal{N}_{node}\) \\ \hline Child\({}_{i}\) & \(\mathcal{M}_{feat\_mat}^{GPU}\) & GPU & \(\mathcal{N}_{win}\times\mathcal{N}_{seg}\) \\ Child\({}_{i}\) & \(\mathcal{M}_{prob\_vec}^{GPU}\) & GPU & \(\mathcal{N}_{node}\) \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Summarization of the allocated run-time memories at CPU and GPU of the baseline multichannel acoustic score computation. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Transmission** & **Frequency** & **Size** \\ \hline from CPU to GPU & \(\mathcal{N}_{channel}\) & \(\mathcal{N}_{win}\times\mathcal{N}_{seg}\) \\ from GPU to CPU & \(\mathcal{N}_{channel}\) & \(\mathcal{N}_{node}\) \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Summarization of the frequency and transmission sizes between CPU and GPU for each feature with a duration of \(\mathcal{T}_{bat}\) if the number of concurrent clients is \(\mathcal{N}_{channel}\), when using the baseline multichannel acoustic score computation. As shown in Figure 4, the proposed fast multichannel parallel acoustic score computation is performed with one decoding thread per client and an additional worker thread, whereas the baseline method is performed with no worker thread. When an online ASR server is launched, the server creates a worker thread for a GPU parallel decoding and initializes the maximum number (\(\mathcal{N}_{parallel}^{CPU}\)) of concurrent clients and that (\(\mathcal{N}_{parallel}^{GPU}\)) of the GPU parallel decodings. Once the worker thread is initialized, the run-time memories are allocated, as shown in Figure 3(a). These memories comprise (a) three types of CPU memories--\(\mathcal{M}_{feat\_vc\_f}^{CPU(W)}\) for input feature vectors, \(\mathcal{M}_{feat\_vc}^{CPU(W)}\) for feature vectors to be decoded, and \(\mathcal{M}_{prob\_vc}^{CPU(W)}\) for acoustic scores, and (b) three types of GPU memories, \(\mathcal{M}_{feat\_pc}^{GPU}\) for feature vectors to be decoded, \(\mathcal{M}_{feat\_mat}^{GPU}\) for a CSC-based matrix, and \(\mathcal{M}_{prob\_vc}^{GPU}\) for acoustic scores. As shown in Figure 3(b), one run-time CPU memory is also allocated when a decoder thread for a client is initialized. The run-time CPU memory \(\mathcal{M}_{prob\_vc}^{CPU}\) is for the acoustic scores. Moreover, the sizes of the run-time memories are summarized in Table 3.  
Whenever a decoder thread is in idle state and \(\mathcal{R}_{feat}\) contains speech features more than \(\mathcal{N}_{bat}\), the decoder thread obtains a speech feature vector of length \(\mathcal{T}_{bat}\) and stores it in the CPU memory of  
Figure 4: Main procedure of the proposed fast multichannel parallel acoustic score computation using a CSC BLSTM AM. The green- and blue-shaded boxes indicate a CPU memory and a GPU memory, respectively.