Implementation of speech feature extraction for low‐resource devices

IET Circuits, Devices & SystemsVolume 13, Issue 6 p. 863-872 Research ArticleFree Access Implementation of speech feature extraction for low-resource devices Sa'ed Abed, Corresponding Author Sa'ed Abed s.abed@ku.edu.kw orcid.org/0000-0003-1849-9316 Computer Engineering Department, Kuwait University, KuwaitSearch for more papers by this authorBassam Jamil Mohd, Bassam Jamil Mohd Computer Engineering Department, Hashemite University, Zarqa, JordanSearch for more papers by this authorMohammad H. Al Shayeji, Mohammad H. Al Shayeji Computer Engineering Department, Kuwait University, KuwaitSearch for more papers by this author Sa'ed Abed, Corresponding Author Sa'ed Abed s.abed@ku.edu.kw orcid.org/0000-0003-1849-9316 Computer Engineering Department, Kuwait University, KuwaitSearch for more papers by this authorBassam Jamil Mohd, Bassam Jamil Mohd Computer Engineering Department, Hashemite University, Zarqa, JordanSearch for more papers by this authorMohammad H. Al Shayeji, Mohammad H. Al Shayeji Computer Engineering Department, Kuwait University, KuwaitSearch for more papers by this author First published: 20 September 2019 https://doi.org/10.1049/iet-cds.2018.5225Citations: 1AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract Feature extraction is an important function in the speech recognition system. Employing a speech recognition system in low-resource devices (LRDs) have increased significantly in recent years. Implementing feature extraction, which involves complex computations, in LRD is very challenging because LRD has limited energy, storage, and processing power. The optimum design must carefully balance performance metrics, including speed, area, and energy. The objective of this research is to implement and model speech feature extraction design in a field-programmable gate array (FPGA) platform, and to identify the optimum implementation for low-resource devices (LRDs). The novelty of this research is optimising feature extraction implementations using design options such as word size; and developing accurate performance models to enhance future designs. The authors study extensively examines the effect of fixed-point n-bit word size on the design of Mel frequency cepstral coefficients feature extraction in the FPGA implementation. The results show that the performance metrics (area, power, and energy) increase at a slower pace compared with n because the dependency of some blocks (e.g. logarithm) on n is non-linear. For example, increasing n by 50% increases the resource utilisation by 38%, power by 41% and energy by 41%. Models for resources, power, and energy are developed with accuracies of 5.1, 4.5, and 4.3%, respectively. Furthermore, n has a weak impact on timing results and therefore speed is almost similar across implementations. Each bit (in n) costs 690 logical elements in the area, 2.35 mW in power and 0.55μJ in energy. For LRD, the 32-bit design demonstrates the most optimum design, followed by 48-bit and 24-bit designs. 1 Introduction In recent years, there has been significant growth of systems and devices equipped with automatic speech recognition capabilities. Example of such devices is internet of things (IoT), which are smart devices connected to the internet (e.g. IoTs) with some including speech recognition capabilities [1]. Concurrently, more applications were developed to utilise systems with speech recognition capabilities. For example, in medical applications, speech recognition systems are used in enhancing learning performance, attention, and meditation [2]; speech therapy [3]; communicating with elderly [4]; and patient assessment documentation [5]. In education, speech recognition systems assist non-native English-speakers [6], provide deaf-hearing interactions at school and work [7], and help visually impaired children to learn Braille [8]. At home, speech recognition systems control home appliances, personalised educational toys and voice-based multi-player gaming [9]. Furthermore, speech recognition has been used in cars [10, 11] as well as mobile phones [12]. Many of the devices equipped with speech recognition can be described as a low-resource device (LRD). A LRD is characterised by limited battery capacity, low computing performance, a small area (or gate count) and small memory requirements [13]. The fundamental challenge in designing LRDs is to carefully balance performance metrics: speed, area, power, and energy. The designer must satisfy the speed and area requirements; with optimum power dissipation and energy consumption [13]. Adding to the design challenge is the fact that speech recognition includes complex signal computations [1, 9]. As a result, balancing performance metrics in LRD with speech recognition function is rather more challenging. Generally, the speech recognition system includes four steps: pre-processing, feature extraction, classification, and language model [14]. The pre-processing step conditions the audio signal for speech recognition processing. Typical signal conditioning includes noise removal, endpoint detection, pre-emphasis, and framing. Feature extraction generates a feature vector containing essential information to classify sampled speech. Feature vectors of speech signals are typically extracted using spectral analysis techniques such as Mel-frequency cepstral coefficients (MFCC), linear predictive coding (LPC) and discrete wavelet transforms (DWTs) [14-16]. The language model captures the syntax and semantics of the spoken language to produce a meaningful representation of speech [17, 18]. The classification step recognises the speech by processing extracted features in association with the language model. Examples of classifiers include a hidden Markov model (HMM) and Gaussian mixture model [14]. The speech recognition system includes two operation phases: training and recognition. In the training phase, the features are extracted and associated with training words. While in the recognition phase, the features are extracted and then processed by speech model. Numerous methods are used to realise speech recognition systems, such as field-programmable gate array (FPGA) platform, application-specific integrated circuit (ASIC), digital signal processors (DSPs) and general-purpose processor platform. Overall, FPGA is flexible and can be reconfigurable as well as provides good performance. Therefore, FPGA is an attractive option for hardware implementation [19]. Feature extraction is an essential step in the training and recognition phases of the speech recognition system. However, it involves complex computations [e.g. logarithm operation and discrete Fourier transform (DFT)]. If feature extraction is implemented in LRD, performance metrics must be carefully balanced to mitigate the impact on device resources. As such, feature extraction is implemented using fixed-point design since the floating-point design is typically slower, larger and consumes more power. Floating-point and fixed-point adder designs were studied and compared on FPGA platform [20]. It was concluded that floating-point design is larger in the area by 7–15×, higher in power by 5–10× and slower by 0.8–1× [20]. The benefits of fixed-point design diminish for higher word size, i.e. 64-bit. In LRD, a fundamental design issue is the selection of word size (i.e. the number of bits in data) because it impacts directly performance, area, power, and energy. Another important issue is modelling the performance metrics to select optimum implementations. Numerous recent studies focused on optimising and modelling hardware designs in LRDs, such as the work in [21-24]. However, to our knowledge, we do not know of any published reports that optimise and model the performance metrics of feature extraction on the FPGA platform targeted for LRDs. 1.1 Objectives and main contributions Optimising design options, such as the word size, for feature extraction implementation targeted for LRDs, presents several challenges. First, complex computations used in feature extraction (i.e. logarithm) are non-linear and hard to scale and model. Such operations complicate the analysis of performance metrics. Second, there are several options for extraction algorithms, transformation functions, and implementation platforms. Determining the optimum algorithm and implementation for LRDs is complex and requires comprehensive analysis. The objective of this research is to implement, optimise, and model speech feature extraction design targeted for LRDs in FPGA platform; and to identify the optimum design, which balances energy, area, and speed. Differently, from the published researches, the novelty of this research is optimising feature extraction implementations using design options such as word size; and developing simple and accurate performance models to predict and enhance future feature extraction designs. Our contributions in this research are summarised as follows: Determine the most appropriate speech feature extraction method and hardware implementation platform. Design and implement the feature extraction method in FPGA, emphasising optimisation techniques for functions such as fast Fourier transform (FFT), logarithm, and square-root functions. Implement, optimise, and examine the feature extraction with different word size (i.e. n). Derive representative performance models for frequency, area, power, and energy metrics based on the implementation results. Determine the best implementation, which balances energy, area, and speed. 1.2 Paper organisation This paper is organised as follows. Section 2 explains the research methodology. Section 3 discusses the methods and implementations of speech feature extraction. Section 4 discusses the FPGA implementations of feature extraction. Section 5 examines and models implementation results. Finally, section 6 presents concluding remarks. 2 Design methodology The research work was conducted in methodical steps, which are illustrated in Fig. 1. The methodology is similar to other methodologies used in research works such as [25-27]. The flow consists of three main phases: investigation, design, and analysis. The initial step is to determine which feature extraction method to be examined; and what is the hardware method to be implemented on. This step requires examining the pros and cons of various methods, as well as investigating literature work. Fig. 1Open in figure viewerPowerPoint Research methodology The next phase is to design feature extraction hardware. Initially, the design is developed at the system level to validate the concept. Once validated, the design is translated to register-transfer level (RTL). RTL design typically implemented using one hardware description language. In our research, RTL is designed using Verilog™ language. RTL design is validated by conducting functional verification simulations to check RTL design against system design. Functional verification simulations were performed using ModelSim simulator [28]. Verified RTL design is then processed by the Altera FPGA software tool, which is Quartus-II software [29]. The RTL design is initially checked against synthesis rules, as some Verilog™ constructs are not supported. Next, the RTL design synthesised to logic primitives and then mapped to FPGA hardware resources. The synthesis was performed using Cyclone-II device family at 50 MHz frequency. The choice of FPGA family does not impact the research outcome as one study concluded [30]. Also, input/output signals assumed a timing setup of a half-clock cycle. Finally, the resources are placed and connected via routing wires. At this stage, various reports are produced to inform the design of details regarding performance metrics. The reports include: Timing results which are the summary of longest and shortest timing paths. It provides the maximum frequency for the design. Resource utilisation report details utilised logic elements (LEs). LEs are basic building units in FPGA design. LE typically includes programmable combinational logic and/or flip-flop. Resource utilisation indicates design size. Therefore, it is always desirable to reduce the number of employed LEs to reduce cost and power consumption. Power report illustrates the power dissipation of various blocks and units in the design. For accurate results, node activity must be available during power analysis. Node activity is computed from ModelSim simulations and saved in value-change-dump files, which are fed into power analysis. The computed power is primarily the core dynamic power, which includes logic (i.e. combinational) power, clock power, and register (i.e. sequential) power. The next step is to develop models for timing, resources, power, and energy. Such models would indicate general trends and simplify system performance. Furthermore, it is important to decide the optimum implementation. 3 Feature extraction: methods and implementations In this section, we present a summary of the main feature extraction methods and their implementations. 3.1 Feature extraction methods There are several feature extraction methods presented in the literature. Since it is similar to human ear system, MFCCs are the most common feature extraction method for speech recognition systems [14, 31, 32]. MFCC resolves frequency in a technique called Mel-scale. Mel-scale resolves frequencies linearly below 1 kHz, and non-linearly (i.e. logarithmically) above 1 kHz as shown in (1) [33, 34] (1) MFCC ability to emulate the human auditory system and provide excellent discrimination features contributed to its wide use in recognition systems [14, 31, 35]. However, the main issue with MFCC is low robustness to noise [35, 36]. Other techniques include LPC and linear predictive cepstral coefficients (LPCCs). LPC computes linear predictive model using a time domain voice sample. The model approximates each current sample as a linear combination of past samples [14, 31, 35]. LPCC is computed by applying cepstral analysis on LPC [31]. The main drawbacks of LPC and LPCC are: linear scale is not an appropriate representation for voice information [35], and both cannot include a priori information on the speech signal [31]. Another feature extraction method is a DWT. DWT considers temporal information apart from the frequency spectrum; and capable of performing efficient time and frequency localisation [32, 35]. Also, DWT is capable of de-noising and compressing without major degradation [14, 35]. The main problem of DWT is the stringent requirement of applying the exact basic wavelets for all speech signals [35]. Additionally, other techniques for feature extractions were presented in the literature including wavelet packet transform and perceptual linear prediction In conclusion, the most popular method is MFCC, which is the mostly renowned feature extraction method for speech recognition systems [14], and hence will be used in our research work. 3.2 Implementation methods The speech recognition system is implemented in various methods, which can be classified as a software approach or hardware approach. In the software approach, automatic speech recognition (ASR) system is implemented in software code to be executed on a general purpose processor (GPP) or DSP. In the hardware approach, the design is realised in FPGA or ASIC technology [19]. Earlier implementations were based entirely on the software approach executed using GPP, also referred to it as a central processing unit (CPU). However, modest performance struggled to support real-time applications. The main drawback of the software approach is limited parallelism could be exploited to enhance performance [19]. The SPHINX recognition system was implemented completely in software [37]. SPHINX was based on discrete HMMs with LPC-derived parameters. It attained word accuracy of 96% on a 997-word vocabulary. Another method of software implementations was based on DSP platforms. DSP processors have wider data buses and capable of performing concurrent complex math operations [38]. Enhanced performance by DSPs supports real-time applications. Yan , et al. [39] presented implementation based on fixed-point DSP. It implemented feature extraction using the MFCC method [39]. Unfortunately, to realise the full potential of the DSP processor, the system must be implemented in the DSP assembly language, a time-consuming task [19]. The FPGA platform provides an attractive environment for system developers since it provides soft-cores, hard-cores, and configurable hardware to implement specialised hardware. Such capabilities provide designers with various options to realise a speech recognition system. Options include implementing part/all of the system in software to run on a platform core or realise system components using FPGA hardware resources. Numerous speech recognition systems using FPGA were reported in the literature. The implementations fall into three major categories [19]: Soft-core to run software-based system. This option is best suitable when speech recognition is used as a secondary module. In this approach, the entire system can be run on a small core, which is capable of running other tasks. Soft-core FPGA was used to implement feature extraction in multiple types of research such as the work in [40, 41]. Implementation of parts of the system in the FPGA hardware to enhance computational speed. Vu , et al. [42] presented a small footprint MFCC design targeted for low-cost speech recognition systems. The implemented design utilises only 10% total resources of a low-cost and modest-size FPGA device. Ahadi , et al. [43] described another good example of partial implementation, where feature extraction is implemented with different algorithms to improve recognition in a noisy environment. Implementation of the entire speech recognition system in the FPGA platform, i.e. system on chip. Example of this approach is the work of [44], which implemented the continuous speech recondition system with medium-sized vocabulary. Elmisery , et al. discussed FPGA implementation for isolated Arabic word recognition, where feature extraction was implemented using LPC [45]. Other examples include the work in [46-48]. ASIC technology provides better performance compared with FPGA, but with less flexibility. Kim , et al. presented the ASIC implementation of a speech recognition system targeted for small-scale isolated speech recognition applications [49]. The design extracts LPC cepstral coefficients and computes the spectral distances using a DWT technique. The system recognises 1000 isolated words per second with an accuracy of 90.3% [49]. The drawback of ASIC design is the expensive development time it requires. 3.3 Discussion An important part of a successful ASR design is to determine the optimum implementation method. This task is not trivial for various reasons: There is no method without challenges. Balancing methods’ trade-offs is the key to select the best approach. Comparing the performance of the methods is difficult and not accurate. For a fair comparison, compared implementations should use the following: Same process and technology. It is not accurate to compare CPU manufactured in the 14 nm process with FPGA manufactured in 28 nm. Same or equivalent software setups. For example, compiled ASIC designs are sensitive to the compiler version and constraints. Similar optimisations. For example, John [50] showed that code optimisation achieved 12% speed up and 11% energy. Despite the stated issues, we can still examine some of the published reports to draw general trends and guidelines. Table 1 presents a sample of published articles comparing methods speed in speech systems and similar applications. For each report, numbers are normalised to slowest design. Furthermore, Bouraoui et al. [59] presented a comprehensive and detailed study and comparing implementation methods for ASR systems. Table 1 presents a summary of the study. Examining Tables 1 and 2, we can draw general guidelines: DSP is faster than GPP provided that DSP executes its intended application. John [50] reported that running support vector machine algorithm on DSP processor TMS320C6713 resulted in 28× slow down compared with Intel i7-2720QM@2.2 GHz. Also, DSPs do not fulfil real-time requirements [59]. Hardware methods are faster than software. Software methods are more flexible. ASIC method has the highest cost and is the least flexible. It is quite clear that the FPGA method combines speed, reconfigurability, flexibility and reduced cost, which make it the most popular hardware method for the ASR system [59]. Reconfigurability is an efficient solution for minimising production and evolution costs [59]. Table 1. Comparing performance implementation methods Devices CPU DSP FPGA ASIC — CPU = RISC processor@2.2 GHz — — — — [51] DSP = acoustic coprocessor 1 2 — — — CPU = IBM RS/6000@2.2 GHz — — — — [52] DSP = Spert-II 1 6 — — — CPU = ARM Cortex-A57 — — — — [53] ASIC = 28 nm design 1 — — 70.7 — CPU = 2.4 GHz Pentium 4 — — — — [54] ASIC = 0.25 μm design 1 — — 2 — CPU = i7-2600@3.4 GHz — — — — [55] FPGA = Xilinx Stratix V 1 — 53 — — FPGA = Xilinx Virtex6 — — — — [56] ASIC = UMC-90 nm design — — 1 2 — CPU = Intel Xeon — — — — — FPGA = Altera Arria 10 — — — — [57] ASIC = 14 nm design 1 — 22 100 — CPU = Intel Xeon — — — — — FPGA = Altera Arria 10 — — — — [58] ASIC = 14 nm design 1 — 10 70 Table 2. Summary of comparison in [59] Pros Cons CPU most flexible lowest performance easy to programme DSP flexible and programmable high cost better performance than CPU large size high power consumption FPGA flexibility and reconfigurability slower than ASIC better performance than DSP and CPU short time to market low cost ASIC fastest not flexible lowest power/energy consumption high cost As such, we will use the FPGA platform in investigating and optimising feature extraction designs. Furthermore, we will adopt fixed-point implementation, as it requires less hardware area and consumes less power and energy, required for LRDs. 4 Feature extraction: algorithm and implementation In this section, we discuss the feature extraction algorithm and design implementation. 4.1 Feature extraction algorithm The features, extracted by the algorithm, are the Mel-frequency cepstralcoefficients (MFCCs). The coefficients are computed by taking inverse cosinetransform of the logarithm of the signal frequency spectrum. The frequency bandsare equally spaced on the Mel scale to approximate the human auditory system.The FFT is an essential part of the algorithm, as it provides fast techniques tocompute frequency spectrum to filter and process the signal. Fig. 2Open in figure viewerPowerPoint Feature extraction algorithm Fig. 3Open in figure viewerPowerPoint Design block diagram The algorithm flow is illustrated in Fig. 2. In thepre-processing step, the input signal is passed through a pre-emphasis filterand appropriate window function (e.g. hamming window). It is assumed that theinput signal is sampled at 8 kHz. The audio signal is partitioned into frames,where each frame is 25 ms. Frames are overlapped by 10 ms. Each frame contains200 samples. It is assumed that signals have been already pre-processed. For each frame, 13 features are calculated. The MFCC steps are as follows: The first feature is computed separately, by taking the logarithm of sample squares, as expressed in (2) (2) FFT block transforms time domain samples into the frequency domain. It processes one frame at a time. The block implements a 256-point FFT algorithm, which is a fast technique to compute DFT. A 256-point DFT is expressed as (3) is referred to as twiddle factor. To eliminate the imaginary part of the FFT results, the absolute values are calculated. Absolute value calculations involve multiplication and square-root operations. A simpler approximation used in research [60, 61] is expressed in (4) (4) Filter bank block computes the results of 26 filters, which mimic the human auditory system. The filter pass-bands are triangular and spaced in the Mel scale. For each filter, the FFT coefficients (from previous steps) are multiplied by triangular filter parameters and summed. Log block computes the logarithm of Mel-frequencies from the previous step. This is implemented by a log of base-2 since it is easier to implement in hardware. discrete cosine transform (DCT) unit transforms coefficients back to the time domain by computing inverse DCT. The output is a cepstral vector, , where k = 1,…, L, and L is the cepstral order. The operation is expressed in (5) (5) whereN is the number of filter bank channels, is the logarithm of filter bank result, and are cepstral coefficients. Lifter unit produces power amplification in some segments of the signal. The lifter unit multiplies the cepstrum vector by lifter function as shown in (6) and (7) [62]. h is set to value between L/2 and L (6) (7) 4.2 MFCC unit design Fig. 3 illustrates the MFCC unit block diagram. The designmainly consists of Feature extraction unit implements algorithm steps discussed earlier.It includes blocks: FFT, filter bank, logarithm, DCT, andlifter. On-chip memory contains buffers used to save temporary results duringthe MFCC calculations, including FFT, Mel-spectrum, andfeatures. Off-chip memory contains the initial speech data and final MFCCresults. Static random-access memory interface provides translate transactionsbetween off-chip and on-chip buses. Several design issues/challenges that should be addressed: Feature design is fixed-point with a data width of n-bits. The main objective of our work is to determine optimum n. In this work, we investigate n values: 16, 24, 32, and 48-bit. 64-bit (and larger word size) was not considered as fixed-point implementation benefits diminish for large word size [20]. Varying n does mean simple scaling of the design. The reason is that some blocks are complex and do not scale linearly, e.g. logarithm block. Such blocks present a substantial challenge in designing and predicting their area and power. Our results and analysis will be focused on the feature extraction unit. The reset of the section presents details of hardware blocks in the feature extraction unit. Our emphasis is to optimise the design for LRDs. 4.3 First feature block The first feature is calculated by (2), which can be implemented using the function. Also, the division can beimplemented as subtraction. Equation (8) provides a simplified expression (8) Fig. 4 illustrates theblock diagram of the first feature design. It consists mainly ofmultiply-accumulate logic, followed by logarithm base-2 block. Additional multiplexers andsubtractors are implemented to compute the constants in (8). The log block is explained inthe next subsection. Fig. 4Open in figure viewerPowerPoint First feature implementation Fig. 5Open in figure viewerPowerPoint Logarithm block 4.4 Logarithm block Several methods are used to implement the algorithm function. The main goal is to optimise area and accuracy [63]. The most popular methods are the digit-recurrence and lookup table (LUT). Digit-recurrence suffers long delays and convergence problems. The LUT approach, on the other hand, is simpler and faster; and hence widely used. Examples of the LUT approach research are in [63-65]. Our research implements the LUT method. Fig. 5 illustrates the block diagram of the algorithmfunction. The operation can be explained as follows: The input and output have integerand fraction parts. The LUT provides a pre-computed logarithm result for . In this case, can be expressed as , where b is 0or 1. is the fraction part of theoutput. In case the , then Input is right shifted tohave format . The number of right shiftpositions is determined by the priority encoder circuit. The right shift operation is performed by shifter logic. The number of right shift positions is the integer part of theoutput. 4.5 FFT block FFT blocks implement 256-point (DFT) expressed in (3). Numerous implementations exist to designN-point FFT. For largeN, it is more efficient to decomposeN-point FFT into smallerV-FFTs, where . In this work, 256-point FFT is decomposedinto 16-points FFTs, and hence, computation in (3) can be computed as (9) As shown in (9), 256-point FFT is decomposed into three steps: First stage 16-point FFT: points are processed by16-point FFT. The processing is performed column by column: column to column . Twiddle factor multiplication: results of first stage FFT aremultiplied by twiddle factor . Second stage 16-point FFT: the results of the twiddle stage areprocessed by 16-point. The processing is performed row by row: fromrow to row . Fig. 6 illustrates a block diagramof the FFT block. It includes two 16-point FFT units to perform column and row16-point computations. The twiddle block implements twi