Accelerating hierarchical acoustic likelihood computation on graphics processors.

pavel.kveton@cz.ibm.com, miroslav@us.ibm.comABSTRACTThe paper presents a method for performance improvements ofa speech recognition system by moving a part of the computation -acoustic likelihood computation - onto a Graphics Processor Unit(GPU). In the system, GPU operates as a low cost powerful co-processor for linear algebra operations. The paper compares GPUimplementation of two techniques of acoustic likelihood computa-tion: fullGaussian computation of all components and a signiﬁcantlyfaster Gaussian selection method using hierarchical evaluation. Thefull Gaussian computation is an ideal candidate for GPU implemen-tation because of its matrix multiplication nature. The hierarchicalGaussian computation is a technique commonly used on aCPU sinceit leads to much better performance by pruning the computation vol-ume. Pruning techniques are generally much harder to implementon GPUs, nevertheless, the paper shows that hierarchical Gaussiancomputation can still be done using a GPU with much better perfor-mance than full Gaussian computation.Index Terms— Speech recognition, GPU, Parallelization1. INTRODUCTIONSpeech recognition is a CPU-intensive tasks. Acoustic likelihoodevaluation usually represents 30-50% of the computation, whichmakes it a good candidate for optimization. A lot of work has beendedicated to increase the performance of acoustic scoring on CPU.One of the successful methods is a hierarchical Gaussian selection[1, 2] leading to 3-4x speed up in comparison with full Gaussianevaluation without a noteworthy accuracy loss. Signiﬁcant perfor-mance increase can be also achieved by the use of Streaming SIMDExtension (SSE) instruction set [2, 3].Graphical processor units (GPUs) represent a low-cost computa-tional power that has been reserved for image/video processing tasksfor a long time. An introduction of General Purpose computationson GPU (GPGPU, e.g. CUDA from NVIDIA [4], ATI Stream fromAMD [5]) made it feasible to use GPUs also for other tasks. Re-cently, an OpenCL [6] standard for cross-platform parallelprogram-ming has been developed in cooperation of many industry-leadingcompanies and institutions. Both NVIDIA and AMD provide anOpenCL SDK and drivers for their GPUs.With the dawn of GPGPU environments the graphics processorsbecome an affordable and powerful coprocessors for linear algebraoperations also in speech recognition. Recent works show contribu-tion of GPUs mainly to full Gaussian evaluation, which is straight-forward to implement on a GPU due to its matrix multiplication na-ture. Dixon at al. [7] presents a GPU based full Gaussian evalua-tion that is 4–6x faster than on-demand CPU evaluation on a largevocabulary continuous speech recognition task (LVCSR). Cardinalat al. [3] reports full Gaussian evaluation on GPU 5x faster thanan SSE-optimized CPU on-demand evaluation, which leads to 35%speed up of the LVCSR task. Chong et al. [8] implemented a model-optimized on-demand Gaussian evaluation together with pruning theViterbi search showing the GPU approx. 9x faster than CPU, gener-alized later by You et al. [9] with comparable results. Shi et al. [10]employs a GPU in ﬁnding clusters of Gaussians for fMPE discri mi-native training achieving 17x speed-up on this task.This paper presents an implementation of hierarchical Gaussianevaluation on GPU and compares it with both full Gaussian evalua-tion on GPU and with SSE optimized CPU implementations. Al-though the Gaussian selection requires global pruning of clusterswhich is not very efﬁcient on GPU, acceptable combination of GPUand CPU usage has been found that leads to approx. 2x speed up incomparison with full Gaussian computation on GPU.2. SOFTAW RE AND HARDAW RE ENVIRONMENTIn the paper we have used NVIDIA GPUs with CUDA architecture[4] based on SIMT (Single Instruction Multiple Thread) computa-tional model. Among several options, we have chosen OpenCL [6]as the programming framework because it is platform-independent.The paper hence uses the OpenCL terminology.An OpenCL program - akernel - is a code segment that is exe-cuted on a GPU in N-Dimensional range(NDrange) of work-items(all work-items execute the same code). The NDrange is splitintoequally-sizedwork-groups. The work-items in one work-group runtogether on one streaming multiprocessor where they can be syn-chronized and share a small and low-latencylocal memory (typi-cally 16kB). There is no communication available between differentwork-groups except by the termination of all the work-itemsin theNDrange.In addition to the local memory, all work-items have access toglobal memory (large and high-latency, typically 1GB). A small por-tion of the global memory can be declared as constant (read-only,typically 64kB) and then it is cashed .Each work-group is split intohalf-warps(two half-warps forma warp) of work-items that are executed together. Optimization ofthe memory access of the work-items within half-warps is crucial toachieve the best performance on GPU. Basically, it is recommendedto use local or constant memory as much as possible in favor ofglobal memory, however, all memory spaces have their optimal ac-cess strategy on the CUDA-enabled chips — if this strategy is fol-lowed, all work-items in half-warp access memory in parallel re-sulting in the best instruction throughput. Otherwise, the access isserialized with a serious performance impact. The local memory isorganized into banks and it is ideal to avoid bank conﬂicts; the globalmemory requires coalescent access by the work-items within a half-