CONVERTING SPEECH SIGNALS TO PHONOLOGICAL FEATURES

The FUL speech recognition system [1] uses underspecified phonological features for the lexical access of words. These features are extracted from the acoustic signal by standard LPCanalysis and by the computation of rough spectral shape parameters. The obtained streams of formants and spectral shape parameters are corrected on a local and global basis by an algorithm that mimics the visual perception of the formant/shape traces. The corrected traces are converted by simple logical decisions into sets of features. Short disturbances are removed from these feature sets and the individual features are time-aligned to give sets of synchronously changing features. These aligned feature sets are used to access possible word candidates from a lexicon of 50 000 base word forms. 1. THEORY Most automatic speech recognition systems use spectral templates of short-time windowed speech signal to compare the incoming speech signal with sequences of templates for stored words. The identification of sequences of computed templates with stored template sequences of words is usually performed with a Hidden Markov Model. These models are trained with large sets of acoustic material. Any change of the acoustic characteristics that influences the spectral templates has to be corrected. Consequently, different microphone or line conditions, different speakers or dialectal variants require an adjustment of the system. This adjustment is normally performed by a training session or an online estimation for the corrections the system has to make. The FUL system operates on a completely different principle. No spectral templates are computed from the speech signal to access the lexicon, nor is the signal analysed in great detail for acoustic evidence of individual segments and their boundaries. Instead, the signal is converted into speaker independent sets of phonological features. These features are compared with the feature sets stored in the lexicon using a ternary logic [1]. The task of the acoustic front-end described here is (a) to remove linguistically irrelevant information, (b) to use speaker independent acoustic characteristics to compute the features, and (c) not to exclude potential word candidates due to computational faults or poor signal quality. The general design principle of the system is to use simple and only rough measures that through cooperation form a stable system. This behaviour could be compared to beavers building a dam Ð they use many roughly cut stems of trees und combine them in a way that forms a robust dam. The idea is to be able to handle massively faulty data rather than to expect clean data with a high signal-to-noise ratio. 2. SYSTEM DESCRIPTION The conversion of the speech signal to phonological features is performed in two steps with one correction applied in each step. First, the signal undergoes a spectral analysis that delivers LPC formants and some rough spectral shape parameters. A heuristic filter is applied to this computed sequence of formant and spectral shape data to convert the raw (formant) data into (formant) tracks as they would be drawn by a human investigator of the visually presented data. In the second step the formant and spectral shape data are converted by simple heuristic filters into phonological features. The feature data again undergoes a heuristic filter to remove irregularities in the stream of features. Additionally, the individual tracks of features are time aligned so that changes in the feature sets occur in all features at the same time. These steps and the corrections are described below. 2.1 . Spec tra l a n a l y s i s A standard LPC analysis and a standard FFT power spectrum are computed from the speech signal from 20 ms Hamming windowed stretches of speech. The first five formants are taken from the LPC analysis, the overall spectral slope, the spectral slope below, and the slope above 2.5 kHz are computed from page 1733 ICPhS99 San Francisco the FFT spectra. The window step rate is 1 ms. That is, in the time domain a very dense sequence of quite roughly computed spectra is available. The intention is to convert the ÔonlineÕ speech signal stream into an online stream of spectral data. This temporal density might not be necessary for the system at every part of the signal but it is currently used as redundant information to separate accidental glitches in the data from systematic variations due to properties of the signal. The outcome of the first step, the spectral analysis, is a set of 5 formants and 3 spectral slope parameters computed every millisecond from a 20 ms window. 2.2. Spectra l correc t ion The spectral analysis is performed for each window individually without considering the preceding and following windows. The lowest five roots of the autocorrelation function or the first five maxima in the LPC spectrum are numbered from 1 to 5. (Here I describe only the correction procedure for formants but the algorithm operates similar for the spectral slope parameters. Furthermore, the algorithm is described as if the signal is stored in the computer; the algorithm can operate online but the description becomes rather awkward). If a formant is ÔmissingÕ in one or more consecutive windows, either because it is not part of the acoustic signal, because it is camouflaged by a zero or merged with an adjacent formant, or because the spectral analysis failed to register it, the numbering of the formants changes whereas the formant tracks are continuing. There is no way to absolutely decide what is the ÔcorrectÕ number of a formant but in general a human, looking at formant tracks, will identify stretches of formant tracks and can label them. The correction step described here tries to mimic this behaviour. For each formant of a given frame (i.e., the set of formants computed from a windowed signal part) the directly left and right adjacent frames (i.e. the formants of the preceding and following 1 ms) are investigated. Each adjacent formant that is less than 50 Hz away and does not have another formant within a range of 200 Hz is taken as a direct neighbour of a formant. If a direct neighbour cannot be found on either the left or the right of a formant, the next but one and the next but two frames are investigated. If a neighbouring formant can be found there formants in the intermediate frames are generated by linear interpolation. This first operation of the formant correction algorithm leads to stretches of connected formant tracks that cannot cross each other. In a second pass, those formants are investigated that have two adjacent formants within 200 Hz. In these cases, the determination of the sequence of formants and the interpolation procedure ensures that two formant tracks do not cross. The first two passes link formants according to their temporal behaviour. A third pass investigates the formants in terms of their similarity within one frame. Formants are connected to their adjacent tracks if the frequencies of all formants of a frame are shifted relative to the preceding or following frames by a similar amount. This operation ensures that formant shifts due to disruption of the speech signal, e.g. at a plosion, are treated as sporadic formant values. The fact that all formants of a frame change in a similar way is used to treat them as formants that bear some information. After this procedure, all singular formants or formant tracks that are shorter than 5 ms are removed in a fourth pass. Finally, the formant tracks are numbered from low to high frequency with 1 to 5 with the constraint that formant 1 must be below 1 kHz. If the first formant track is above 1 kHz the counting of formants starts with Ô2Õ. 2.3. Convert ing spectral parameters to p h o n o l o g i cal features The corrected spectral parameters are used to compute phonological features. These features should reflect important properties of the speech signal and they should be relatively independent of the speaker and acoustic line properties. It is not intended that the features exactly match the features that are stored in the lexicon; only features that are impossible for a certain sound (e.g. a low vowel should never be high) should not be computed from the signal. That is, a low vowel should preferably have the feature [low] in the acoustic signal, but a missing low computed from the signal does not exclude it from the subsequent processing. Only if the feature [high] is computed from the signal a low vowel is excluded in the lexical access. The system uses very broad acoustic characteristics to define the phonological features. We investigated two databases [2, 3] and found that it is possible to characterise page 1734 ICPhS99 San Francisco individual segments for individual speakers with more complex acoustic characterisations, but these characteristics could not be extended to different speakers. We also tested some speaker normalisation procedures but we found a rather broad specification of acoustic characteristics for the phonological features for the raw spectral data more efficient than the application of a speaker normalisation procedure prior to the classification. We observed an improvement by the speaker normalisation procedures with carefully measured formants of speech signals recorded under optimal conditions, but we observed a deterioration with the normalisation procedure with automatically extracted formant data (see [4] for detail). Most of the 13 phonological features we use (consonantal, vocalic, continuant, RTR, voice, strident, abrupt, nasal, labial, coronal, dorsal, high, low) are defined by simple acoustic characteristics. For example, the feature [high] is defined by F1 < 450 Hz. These very simple characteristics are often true for sounds that do not belong to a particular feature (e.g., some other sound that is not [high] might have an F1 below 450 Hz) but crucially, this acoustic characteristic seldom misses any sound that is [high]. That is, the acoustic ch