# Harmonic Differences Method for Robust Fundamental Frequency Detection in Wideband and Narrowband Speech Signals

Authors: Cevahir Parlak; Yusuf Altun
Journal: Mathematical Problems in Engineering
 (2021)

---

## Abstract

In this article, a novel pitch determination algorithm based on harmonic differences method (HDM) is proposed. Most of the algorithms today rely on autocorrelation, cepstrum, and lastly convolutional neural networks, and they have some limitations (small datasets, wideband or narrowband, musical sounds, temporal smoothing, etc.), accuracy, and speed problems. There are very rare works exploiting the spacing between the harmonics. HDM is designed for both wideband and exclusively narrowband (telephone) speech and tries to find the most repeating difference between the harmonics of speech signal. We use three vowel databases in our experiments, namely, Hillenbrand Vowel Database, Texas Vowel Database, and Vowels from the TIMIT corpus. We compare HDM with autocorrelation, cepstrum, YIN, YAAPT, CREPE, and FCN algorithms. Results show that harmonic differences are reliable and fast choice for robust pitch detection. Also, it is superior to others in most cases.


---

## Body

## 1. Introduction

Pitch is an extraordinarily complicated and distinct feature of human speech and plays a major role in the perception of human conversations as well as in human-computer interactions. Pitch detection has a strong and disputed background spanning more than a century. Myriad of methods have been proposed, but it is still a formidable task especially in narrowband (telephone), noisy, multipitch, and multitalker speech with reasonable resolution and fast implementation due to extremely complicated structure of frequency spectrum. Pitch helps us to identify some of the important cues about the speaker, such as the identity, gender, emotional state, or about the tones of a musical instrument. It has a wide range of applications in emotion and gender recognition, speech synthesis, human-computer interaction, and detection of symptoms of pathological disorder at early stages. Fundamental frequency is the quantity of pitch and is measured on the periodic signals (musical tones) or quasiperiodic signals (speech). Pitch detection algorithms (PDA) and pitch tracking algorithms are extensively used to extract the fundamental frequency of a person’s speech or of a musical tone. Fundamental frequency is the frequency of vocal cord oscillation, and it can highly vary among the men, women, boys, and girls. Therefore, exact calculation is a crucial factor in variety of applications spanning from human-computer interaction to early detection of pathological symptoms.This article is organized as follows: in Section1, we discuss the foundations and importance of pitch detection and tracking, Section 2 deals with literature overview, historical background, algorithms, novelties of this article, datasets, ground truth methods, error measures, difficulties, application areas, and related algorithm domains, Section 3 describes the novel HDM algorithm, Section 4 delineates datasets used in this article and experimental setup, Section 5 presents the results of wide and narrowband experiments, Section 6 is devoted to gender detection results, and finally, in Section 7, we conclude this work with the evaluations and future studies.


## 2. Literature Review

### 2.1. Historical Background

The saga of pitch determination and tracking begins with the dispute between August Seebeck and George Simon Ohm on the mysterious missing fundamental concept. In 1841, August Seebeck showed that the pitch of a sound did not depend on the tone having a fundamental frequency component of the pitch frequency. The debate over missing fundamental began shortly after this [1–3]. In 1843, Ohm severely rejected this idea and stated that the quality of a tone depends solely on the number and relative strength of its partial simple tones [4]. This law is championed by Helmholtz in his masterpiece work [5].Rayleigh, in his famous study, The Theory of Sound (1877), claimed that pitch could not be simply associated with period using his experiments with sirens. But similar to loudness and timbre, pitch is not a thing to be measured directly [6]. In 1924, Harvey Fletcher proved that even if several lower harmonics of a waveform were removed, the pitch remained the same. The pitch was very closely related to the difference in frequency, even though that frequency did not exist in the sound source [7]. This idea is one of the bases of this study to exploit these differences to extract the fundamental frequency.In 1938, Schouten proved that the missing fundamental effect could not be explained as a nonlinear difference as Helmholtz had claimed and strongly supported the Seebeck [8]. Schouten’s theory is known as the residue theory of pitch, periodicity pitch, or virtual pitch [9–11]. Today, scientists agreed that Seebeck disproved the Ohm’s idea. Missing fundamental has a practical implementation in our telephone conversation. In telephone signals, the frequency spectrum is limited between 300 Hz and 4000 Hz. But we can still perceive the voice clearly and discriminatively to capture the talker’s identity, gender, and even emotional state. For instance, if we have a signal with harmonics at 400 Hz, 600 Hz, 800 Hz, 1000 Hz, and 1200 Hz, the pitch of the signal is still perceived as 200 Hz, and autocorrelation of both signals remains nearly same as depicted in Figure 1 [12].Figure 1
(a) A signal with multiple harmonics and (b) its autocorrelation function. (c) Same signal with the first harmonic removed and (d) its autocorrelation function.
(a)(b)(c)(d)The autocorrelation model of pitch perception dates back to Licklider’s “duplex” and “triplex” models. Licklider solved the dilemma between Seebeck and Ohm definitely in favor of Seebeck. Periodicity pitch theory succeeded place pitch theory after Licklider’s “duplex” and “triplex” theories [13–16]. This theory was investigated deeply by Ritsma, and some of its limitations were shown [17]. Today, the debate is far from over, and two mainstream theories compete against each other paving the way for Place Code of Ohm & Helmholtz and Temporal Code Theory of Seebeck & Wever of hearing.


### 2.2. Algorithms

First works started around 1950s with AMDF because in the early days of computers, the multiplication of autocorrelation function was replaced with subtraction due to the high computational cost of multiplication. The fierce debate on missing fundamental led to the calculation of fundamental frequency, and many algorithms flourished to extract and track the pitch of a signal. Among the most notables are AMDF [18–20], autocorrelation [13–16, 21–25], cepstrum [26–28], harmonic product spectrum, period histograms [29–31], parallel processing methods [32–34], simplified inverse filter tracking (SIFT) [35], comb filters [36], data reduction [37], LPC-based spectral equalization (unpublished), spectral sieves [38], harmonic spacings and structures [39–41], LPC inverse filtering [20], feature based [42], IPTA [43], harmonic pattern recognition [44], envelop analysis, threshold-crossing analysis (ZXABE, TABE, TTABE) [45, 46], subharmonic summation [47], subband processing [48], superresolution [49], two-way mismatch [50], resolution improvement [51], TEMPO [52], RAPT (NCCF) [53], instantaneous frequency [54–57], STRAIGHT-NDF [58, 59], wavelet based [60–62], joint time-frequency analysis (JTFA) [63], PRAAT [64], FIR filter, subharmonic-to harmonic ratio [65], YIN [66], YAAPT [67, 68], HMM for multipitch tracking, PEPSI_LITE, ESACF, PEBS (block sparsity), PEARLS [69–78], chirp transform [79], sawtooth waveform inspired pitch estimator (SWIPE) [80], perturbation spectrum (TANDEM STRAIGHT) [81], ANLS [82], vocal fold vibration (DIO) [83], ensemble empirical mode decomposition and entropy [84, 85], event-based instantaneous fundamental frequency with impulse-like characteristics of excitation in glottal vibrations [86], tandem algorithm [87], summation of residual harmonics (SRH) [88], glottal closure instants, EGG, glottal opening instants (GOI, GCI) [89–91], PEFAC [92], SAFE [93], SAcC [94], BANA [95, 96], adaptive harmonic model (aHM) with adaptive iterative refinement (AIR) [97], MPM [98], BSURE-IR [99], time-warping for fast f0 changes [100], robust harmonic features with multilayer perceptron [101], HARVEST [102], Pyin [103], off-grid [104], harmonics of impulse-like excitation [105], phase space [106], normalized squared difference function, LASSO regression based [107], adaptive total variation penalty, single frequency filtering and temporal envelopes [108, 109], CNN and DNN (CREPE, Deep Pitch, FCN) [110–117] are among these methods.Schroeder histogram, Brown comb filter, modified Maher–Beauchamp pitch detector, Meddis–Hewitt model, McAulay–Quatieri’s method, and cepstral detector are used in pitch detection and tracking of musical signals. The challenges in multipitch tracking are overlapped harmonics (octaves, fifths), noise, threshold of detection, scarcity of labeled datasets, timbre complexity (one pitch played by multiple instruments), and efficiency. F0 is also an important feature in emotional speech recognition [118–120]. There are many reviews about the methods and datasets of pitch detection and pitch tracking [12, 121–134].


### 2.3. Innovations in This Article

Pitch detection is an extensively studied research field. Today pitch detection methods are successful in wideband, clean human speech. But there are still too many formidable obstacles that need to be resolved particularly in narrowband (telephone speech) [67, 68], multipitch [69–78], multitalker, and noisy speech signals [69, 95, 96, 108, 109]. Various techniques have been implemented including autocorrelation, cepstrum, phase based, and lastly, CNN methods. However, an interesting clue for pitch determination is highly neglected and rarely used. Harmonic spacings are strong pieces of evidence for pitch determination even though human speech is extraordinarily quasiperiodic. This method was first used by Seneff [39] for narrowband speech on real-time data between 210 Hz and 1050 Hz band by iteratively assigning weight factors to the pitch candidates obtained from harmonic spacings with LDVT (Lincoln Digital Voice Terminal). Seneff’s work is too old, and dataset and results are not clearly presented. There are other works till 21st century. Wu [40] tried harmonic spacings in guitar sound using GCD (Greatest Common Divisor of largest peak) with a table, and Dziubiński and Kostek in various musical instruments with an Artificial Neural Network on a matrix of sets of harmonics. We will apply this method both for wideband and narrowband speech (400 Hz–3400 Hz) signals on large speech datasets, eliminating some of the limitations while spurring the accuracy and speed via determining the most repeated difference between the harmonics using a histogram. This article will revive this technique and demonstrate that harmonic spacings can reliably be used to obtain state-of-the-art results fast and efficiently both in wideband and narrowband telephone speech samples. More detailed explanations are presented in Section 3.


### 2.4. Datasets

As usual with the other experiments, creation and use of proper datasets is essential in pitch tracking experiments. There are numerous datasets used in this field and among them Keele Studio, PTDB-TUG [135], Keele Telephone [136], TIMIT [137], NTIMIT [138], CSTR [139], FARSDAT [140], Mocha TIMIT [141], RWC Music Database [142], MedleyDB [143], Vowel-CVC [144], NOIZEUS [145], and SPEECON [146] datasets. Some authors used CMU ARCTIC, KED TIMIT [147], APLAWD [148], BACH10 [149], SyncRWC60, Saarland Music Data [150], and Mazurka Dataset [151]. Vowel datasets can also be used for pitch determination. Because f0 tracking is specifically important in music transcription, many music datasets are available in ISMIR (https://www.ismir.net/resources/) web pages.Pitch detection can be implemented in clean wideband, narrowband telephony, noisy speech, and multipitch musical sounds. In telephony speech, signal is usually band-passed between 300 Hz and 4000 Hz to save the bandwidth. In this study, we applied band-pass filter twice to remove all remains between 400 Hz and 3400 Hz. There are many types of noises that can be added to the pitch datasets, including babble noise, exhibition noise, HF (high-frequency) channel noise, restaurant noise, street noise, white noise, pink, brown, and pub noises. NOISEX [153] dataset is a publicly available noise dataset. Pitch datasets can be recorded in studio, office, living rooms, and car interior environments.In pitch datasets, speaker profiles, distribution of gender, distribution of age, mother tongue, distribution of dialect, distribution of profession or education, pathologies, number of speakers, contents, speaking style, read speech, answering speech, command and control speech, descriptive speech, nonprompted speech, spontaneous speech, neutral and emotional speech, general recording setup (telephone, on-site, field, wizard-of-oz), annotation, technical specifications (sampling rate, sample type, number of channels, file formats), corpus structure, release plan and validation procedure, meta data, recording protocol (session id, speaker id, recording date, environmental conditions, technical recording conditions), postprocessing, pronunciation dictionary, and validation are important issues [135].


### 2.5. Ground Truth Determination of Pitch Datasets

Another important issue for pitch datasets is the exact determination of ground truth. Hand-editing, EGG, and Laryngograph are widely used as ground truth methods. SIGMA, Hilbert Envelope-based detection (HE), the Zero Frequency Resonator-based method (ZFR), the Dynamic Programming Phase Slope Algorithm (DYPSA), the Speech Event Detection using the Residual Excitation And a Mean-based Signal (SEDREAMS), and Yet Another GCI Algorithm (YAGA) are some other methods used for finding ground truth of pitch samples [154–158]. They have some advantages and disadvantages, and we usually need hand-editing by an expert. In some cases, detecting f0 manually by human experts can be quite difficult. EGG, laryngogram, particularly differentiated laryngogram provides a signal, which makes automatic f0 calculation easier using available methods as shown in Figure 2.Figure 2
Speech signal, laryngogram, and differentiated laryngogram [156].


### 2.6. Error Measures and Pitch Refinement Techniques

Gross Pitch Error (GPE), Fine Pitch Error (FPE), Voicing Decision Error (VDE), f0 Frame Error (FFE) [88, 159, 160], MOS (Mean Opinion Score) [161], Diagnostic Rhyme Test (DRT) [162], Perceptual Evaluation of Speech Quality (PESQ) [163], and Perceptual Objective Listening Quality Analysis (POLQA) [164] are prominent metrics used to evaluate pitch detection algorithms.Pitch estimation derived from the discrete Fourier spectrum can be improved using techniques, such as Spectral Reassignment (phase based) [165–167], super sampling, interpolation (Grandke, quadratic, Gaussian) [168–171], Matching pursuit [172], Synchrosqueezing [173–176], and Empirical Mode Decomposition [177], to compensate for the resolution problem of FFT.Vocal Tract Model, Uniform Lossless Tube, and Two-Tube Model can be used to model the f0 and other formant frequency structures [178–186]. Using the laws of conservation of mass, momentum, and energy, it can be proved that sound wave propagation inside a lossless tube satisfies the equations:(1)−∂p∂x=ρ∂u/A∂t,−∂u∂x=1ρc2∂pA∂t+∂A∂t,where p = p (x, t) is the pressure of sound at position x and time t, u = u (x, t) is the velocity at position x and time t, ρ is air density inside the tube, c is sound velocity, and A = A (x, t) is the cross-sectional area function of the distance and time.


### 2.7. Difficulties in Pitch Tracking

It may be helpful at this point to summarize the difficulties in f0 tracking and voiced detection:(i)
F0 may change in each glottal period(ii)
Subharmonics and spurious harmonics(iii)
Inharmonicity and quasiharmonicity(iv)
Doubling or halving of f0(v)
Formants and band-limiting(vi)
Irregularity of voicing at voice onset and offset(vii)
Even experts cannot always agree on the locations of voice onset and offset(viii)
Narrowband filtering of unvoiced excitation may create periodic signals(ix)
The amplitude of voiced speech may vary in a wide range(x)
Background noise(xi)
Some voiced speech may have very little glottal pulse duration(xii)
Determining the start and end of each pitch period in voiced speech parts(xiii)
Shimmer: amplitude variation from one cycle to the next(xiv)
Jitter: frequency variation from one cycle to the next(xiv)
Breathy voice(xvi)
Inherent quasiperiodicity of human speech(xvii)
Multiple periodicities in music signals(xviii)
Multitalker signals(xix)
Transient regions(xx)
Estimation of pitch with low energy can be difficult(xxi)
Low formant F1 may interfere with high f0 of females and children(xxii)
In telephony systems, nonlinear effects such as phase distortion, crosstalk, clipping of high-level sounds, and amplitude modulation


### 2.8. Application Areas

Today determination of fundamental frequency is an important factor used in a wide range of solutions:(i)
Emotion recognition and human-computer interaction(ii)
Gender determination (male/female/boy/girl)(iii)
Detection of the pathological characteristics of the voice(iv)
Speech understanding systems(v)
Prosody analysis(vi)
Speaker identification and separation(vii)
Data-driven speech synthesis(viii)
Digital hearing prostheses(ix)
Computer-aided intonation teaching


### 2.9. Domains of Methods

Pitch detection algorithms can be classified as time, frequency, and hybrid methods. Some of these algorithms use temporal smoothing (median filter, HMM, Viterbi etc.).


#### 2.9.1. Time-Domain Methods

Many methods rely only on the time-related information for pitch calculation.(i)
Threshold-crossing pitch detectors (ZXABE, TABE, and TTABE, Zero-Crossing Rate, Peak rate, Slope event rate)(ii)
Parallel processing method(iii)
Average Magnitude Difference Function (AMDF)(iv)
Envelope analysis methods(v)
Autocorrelation(vi)
YIN(vii)
Phase space(viii)
Deep learning and convolutional neural networks (end to end).


#### 2.9.2. Frequency-Domain Methods

Frequency spectrum is widely used to detect the pitch value of a signal.(i)
Optimum comb filter(ii)
Tunable IIR filter(iii)
FIR filter(iv)
Cepstrum(v)
Harmonics(vi)
Multiresolution methods(vii)
Statistical frequency-domain methods(viii)
Neural networks(ix)
Maximum likelihood estimators


#### 2.9.3. Time- and Frequency-Domain (Hybrid) Pitch Detectors

Some algorithms combine time and frequency domain to calculate fundamental frequency:(i)
Schroeder histogram(ii)
Brown comb filter method(iii)
Maher–Beauchamp pitch detector(iv)
Meddis–Hewitt model(v)
McAulay–Quatieri method(vi)
Neural networks(vii)
YAAPT


## 2.1. Historical Background

The saga of pitch determination and tracking begins with the dispute between August Seebeck and George Simon Ohm on the mysterious missing fundamental concept. In 1841, August Seebeck showed that the pitch of a sound did not depend on the tone having a fundamental frequency component of the pitch frequency. The debate over missing fundamental began shortly after this [1–3]. In 1843, Ohm severely rejected this idea and stated that the quality of a tone depends solely on the number and relative strength of its partial simple tones [4]. This law is championed by Helmholtz in his masterpiece work [5].Rayleigh, in his famous study, The Theory of Sound (1877), claimed that pitch could not be simply associated with period using his experiments with sirens. But similar to loudness and timbre, pitch is not a thing to be measured directly [6]. In 1924, Harvey Fletcher proved that even if several lower harmonics of a waveform were removed, the pitch remained the same. The pitch was very closely related to the difference in frequency, even though that frequency did not exist in the sound source [7]. This idea is one of the bases of this study to exploit these differences to extract the fundamental frequency.In 1938, Schouten proved that the missing fundamental effect could not be explained as a nonlinear difference as Helmholtz had claimed and strongly supported the Seebeck [8]. Schouten’s theory is known as the residue theory of pitch, periodicity pitch, or virtual pitch [9–11]. Today, scientists agreed that Seebeck disproved the Ohm’s idea. Missing fundamental has a practical implementation in our telephone conversation. In telephone signals, the frequency spectrum is limited between 300 Hz and 4000 Hz. But we can still perceive the voice clearly and discriminatively to capture the talker’s identity, gender, and even emotional state. For instance, if we have a signal with harmonics at 400 Hz, 600 Hz, 800 Hz, 1000 Hz, and 1200 Hz, the pitch of the signal is still perceived as 200 Hz, and autocorrelation of both signals remains nearly same as depicted in Figure 1 [12].Figure 1
(a) A signal with multiple harmonics and (b) its autocorrelation function. (c) Same signal with the first harmonic removed and (d) its autocorrelation function.
(a)(b)(c)(d)The autocorrelation model of pitch perception dates back to Licklider’s “duplex” and “triplex” models. Licklider solved the dilemma between Seebeck and Ohm definitely in favor of Seebeck. Periodicity pitch theory succeeded place pitch theory after Licklider’s “duplex” and “triplex” theories [13–16]. This theory was investigated deeply by Ritsma, and some of its limitations were shown [17]. Today, the debate is far from over, and two mainstream theories compete against each other paving the way for Place Code of Ohm & Helmholtz and Temporal Code Theory of Seebeck & Wever of hearing.


## 2.2. Algorithms

First works started around 1950s with AMDF because in the early days of computers, the multiplication of autocorrelation function was replaced with subtraction due to the high computational cost of multiplication. The fierce debate on missing fundamental led to the calculation of fundamental frequency, and many algorithms flourished to extract and track the pitch of a signal. Among the most notables are AMDF [18–20], autocorrelation [13–16, 21–25], cepstrum [26–28], harmonic product spectrum, period histograms [29–31], parallel processing methods [32–34], simplified inverse filter tracking (SIFT) [35], comb filters [36], data reduction [37], LPC-based spectral equalization (unpublished), spectral sieves [38], harmonic spacings and structures [39–41], LPC inverse filtering [20], feature based [42], IPTA [43], harmonic pattern recognition [44], envelop analysis, threshold-crossing analysis (ZXABE, TABE, TTABE) [45, 46], subharmonic summation [47], subband processing [48], superresolution [49], two-way mismatch [50], resolution improvement [51], TEMPO [52], RAPT (NCCF) [53], instantaneous frequency [54–57], STRAIGHT-NDF [58, 59], wavelet based [60–62], joint time-frequency analysis (JTFA) [63], PRAAT [64], FIR filter, subharmonic-to harmonic ratio [65], YIN [66], YAAPT [67, 68], HMM for multipitch tracking, PEPSI_LITE, ESACF, PEBS (block sparsity), PEARLS [69–78], chirp transform [79], sawtooth waveform inspired pitch estimator (SWIPE) [80], perturbation spectrum (TANDEM STRAIGHT) [81], ANLS [82], vocal fold vibration (DIO) [83], ensemble empirical mode decomposition and entropy [84, 85], event-based instantaneous fundamental frequency with impulse-like characteristics of excitation in glottal vibrations [86], tandem algorithm [87], summation of residual harmonics (SRH) [88], glottal closure instants, EGG, glottal opening instants (GOI, GCI) [89–91], PEFAC [92], SAFE [93], SAcC [94], BANA [95, 96], adaptive harmonic model (aHM) with adaptive iterative refinement (AIR) [97], MPM [98], BSURE-IR [99], time-warping for fast f0 changes [100], robust harmonic features with multilayer perceptron [101], HARVEST [102], Pyin [103], off-grid [104], harmonics of impulse-like excitation [105], phase space [106], normalized squared difference function, LASSO regression based [107], adaptive total variation penalty, single frequency filtering and temporal envelopes [108, 109], CNN and DNN (CREPE, Deep Pitch, FCN) [110–117] are among these methods.Schroeder histogram, Brown comb filter, modified Maher–Beauchamp pitch detector, Meddis–Hewitt model, McAulay–Quatieri’s method, and cepstral detector are used in pitch detection and tracking of musical signals. The challenges in multipitch tracking are overlapped harmonics (octaves, fifths), noise, threshold of detection, scarcity of labeled datasets, timbre complexity (one pitch played by multiple instruments), and efficiency. F0 is also an important feature in emotional speech recognition [118–120]. There are many reviews about the methods and datasets of pitch detection and pitch tracking [12, 121–134].


## 2.3. Innovations in This Article

Pitch detection is an extensively studied research field. Today pitch detection methods are successful in wideband, clean human speech. But there are still too many formidable obstacles that need to be resolved particularly in narrowband (telephone speech) [67, 68], multipitch [69–78], multitalker, and noisy speech signals [69, 95, 96, 108, 109]. Various techniques have been implemented including autocorrelation, cepstrum, phase based, and lastly, CNN methods. However, an interesting clue for pitch determination is highly neglected and rarely used. Harmonic spacings are strong pieces of evidence for pitch determination even though human speech is extraordinarily quasiperiodic. This method was first used by Seneff [39] for narrowband speech on real-time data between 210 Hz and 1050 Hz band by iteratively assigning weight factors to the pitch candidates obtained from harmonic spacings with LDVT (Lincoln Digital Voice Terminal). Seneff’s work is too old, and dataset and results are not clearly presented. There are other works till 21st century. Wu [40] tried harmonic spacings in guitar sound using GCD (Greatest Common Divisor of largest peak) with a table, and Dziubiński and Kostek in various musical instruments with an Artificial Neural Network on a matrix of sets of harmonics. We will apply this method both for wideband and narrowband speech (400 Hz–3400 Hz) signals on large speech datasets, eliminating some of the limitations while spurring the accuracy and speed via determining the most repeated difference between the harmonics using a histogram. This article will revive this technique and demonstrate that harmonic spacings can reliably be used to obtain state-of-the-art results fast and efficiently both in wideband and narrowband telephone speech samples. More detailed explanations are presented in Section 3.


## 2.4. Datasets

As usual with the other experiments, creation and use of proper datasets is essential in pitch tracking experiments. There are numerous datasets used in this field and among them Keele Studio, PTDB-TUG [135], Keele Telephone [136], TIMIT [137], NTIMIT [138], CSTR [139], FARSDAT [140], Mocha TIMIT [141], RWC Music Database [142], MedleyDB [143], Vowel-CVC [144], NOIZEUS [145], and SPEECON [146] datasets. Some authors used CMU ARCTIC, KED TIMIT [147], APLAWD [148], BACH10 [149], SyncRWC60, Saarland Music Data [150], and Mazurka Dataset [151]. Vowel datasets can also be used for pitch determination. Because f0 tracking is specifically important in music transcription, many music datasets are available in ISMIR (https://www.ismir.net/resources/) web pages.Pitch detection can be implemented in clean wideband, narrowband telephony, noisy speech, and multipitch musical sounds. In telephony speech, signal is usually band-passed between 300 Hz and 4000 Hz to save the bandwidth. In this study, we applied band-pass filter twice to remove all remains between 400 Hz and 3400 Hz. There are many types of noises that can be added to the pitch datasets, including babble noise, exhibition noise, HF (high-frequency) channel noise, restaurant noise, street noise, white noise, pink, brown, and pub noises. NOISEX [153] dataset is a publicly available noise dataset. Pitch datasets can be recorded in studio, office, living rooms, and car interior environments.In pitch datasets, speaker profiles, distribution of gender, distribution of age, mother tongue, distribution of dialect, distribution of profession or education, pathologies, number of speakers, contents, speaking style, read speech, answering speech, command and control speech, descriptive speech, nonprompted speech, spontaneous speech, neutral and emotional speech, general recording setup (telephone, on-site, field, wizard-of-oz), annotation, technical specifications (sampling rate, sample type, number of channels, file formats), corpus structure, release plan and validation procedure, meta data, recording protocol (session id, speaker id, recording date, environmental conditions, technical recording conditions), postprocessing, pronunciation dictionary, and validation are important issues [135].


## 2.5. Ground Truth Determination of Pitch Datasets

Another important issue for pitch datasets is the exact determination of ground truth. Hand-editing, EGG, and Laryngograph are widely used as ground truth methods. SIGMA, Hilbert Envelope-based detection (HE), the Zero Frequency Resonator-based method (ZFR), the Dynamic Programming Phase Slope Algorithm (DYPSA), the Speech Event Detection using the Residual Excitation And a Mean-based Signal (SEDREAMS), and Yet Another GCI Algorithm (YAGA) are some other methods used for finding ground truth of pitch samples [154–158]. They have some advantages and disadvantages, and we usually need hand-editing by an expert. In some cases, detecting f0 manually by human experts can be quite difficult. EGG, laryngogram, particularly differentiated laryngogram provides a signal, which makes automatic f0 calculation easier using available methods as shown in Figure 2.Figure 2
Speech signal, laryngogram, and differentiated laryngogram [156].


## 2.6. Error Measures and Pitch Refinement Techniques

Gross Pitch Error (GPE), Fine Pitch Error (FPE), Voicing Decision Error (VDE), f0 Frame Error (FFE) [88, 159, 160], MOS (Mean Opinion Score) [161], Diagnostic Rhyme Test (DRT) [162], Perceptual Evaluation of Speech Quality (PESQ) [163], and Perceptual Objective Listening Quality Analysis (POLQA) [164] are prominent metrics used to evaluate pitch detection algorithms.Pitch estimation derived from the discrete Fourier spectrum can be improved using techniques, such as Spectral Reassignment (phase based) [165–167], super sampling, interpolation (Grandke, quadratic, Gaussian) [168–171], Matching pursuit [172], Synchrosqueezing [173–176], and Empirical Mode Decomposition [177], to compensate for the resolution problem of FFT.Vocal Tract Model, Uniform Lossless Tube, and Two-Tube Model can be used to model the f0 and other formant frequency structures [178–186]. Using the laws of conservation of mass, momentum, and energy, it can be proved that sound wave propagation inside a lossless tube satisfies the equations:(1)−∂p∂x=ρ∂u/A∂t,−∂u∂x=1ρc2∂pA∂t+∂A∂t,where p = p (x, t) is the pressure of sound at position x and time t, u = u (x, t) is the velocity at position x and time t, ρ is air density inside the tube, c is sound velocity, and A = A (x, t) is the cross-sectional area function of the distance and time.


## 2.7. Difficulties in Pitch Tracking

It may be helpful at this point to summarize the difficulties in f0 tracking and voiced detection:(i)
F0 may change in each glottal period(ii)
Subharmonics and spurious harmonics(iii)
Inharmonicity and quasiharmonicity(iv)
Doubling or halving of f0(v)
Formants and band-limiting(vi)
Irregularity of voicing at voice onset and offset(vii)
Even experts cannot always agree on the locations of voice onset and offset(viii)
Narrowband filtering of unvoiced excitation may create periodic signals(ix)
The amplitude of voiced speech may vary in a wide range(x)
Background noise(xi)
Some voiced speech may have very little glottal pulse duration(xii)
Determining the start and end of each pitch period in voiced speech parts(xiii)
Shimmer: amplitude variation from one cycle to the next(xiv)
Jitter: frequency variation from one cycle to the next(xiv)
Breathy voice(xvi)
Inherent quasiperiodicity of human speech(xvii)
Multiple periodicities in music signals(xviii)
Multitalker signals(xix)
Transient regions(xx)
Estimation of pitch with low energy can be difficult(xxi)
Low formant F1 may interfere with high f0 of females and children(xxii)
In telephony systems, nonlinear effects such as phase distortion, crosstalk, clipping of high-level sounds, and amplitude modulation


## 2.8. Application Areas

Today determination of fundamental frequency is an important factor used in a wide range of solutions:(i)
Emotion recognition and human-computer interaction(ii)
Gender determination (male/female/boy/girl)(iii)
Detection of the pathological characteristics of the voice(iv)
Speech understanding systems(v)
Prosody analysis(vi)
Speaker identification and separation(vii)
Data-driven speech synthesis(viii)
Digital hearing prostheses(ix)
Computer-aided intonation teaching


## 2.9. Domains of Methods

Pitch detection algorithms can be classified as time, frequency, and hybrid methods. Some of these algorithms use temporal smoothing (median filter, HMM, Viterbi etc.).


### 2.9.1. Time-Domain Methods

Many methods rely only on the time-related information for pitch calculation.(i)
Threshold-crossing pitch detectors (ZXABE, TABE, and TTABE, Zero-Crossing Rate, Peak rate, Slope event rate)(ii)
Parallel processing method(iii)
Average Magnitude Difference Function (AMDF)(iv)
Envelope analysis methods(v)
Autocorrelation(vi)
YIN(vii)
Phase space(viii)
Deep learning and convolutional neural networks (end to end).


### 2.9.2. Frequency-Domain Methods

Frequency spectrum is widely used to detect the pitch value of a signal.(i)
Optimum comb filter(ii)
Tunable IIR filter(iii)
FIR filter(iv)
Cepstrum(v)
Harmonics(vi)
Multiresolution methods(vii)
Statistical frequency-domain methods(viii)
Neural networks(ix)
Maximum likelihood estimators


### 2.9.3. Time- and Frequency-Domain (Hybrid) Pitch Detectors

Some algorithms combine time and frequency domain to calculate fundamental frequency:(i)
Schroeder histogram(ii)
Brown comb filter method(iii)
Maher–Beauchamp pitch detector(iv)
Meddis–Hewitt model(v)
McAulay–Quatieri method(vi)
Neural networks(vii)
YAAPT


## 2.9.1. Time-Domain Methods

Many methods rely only on the time-related information for pitch calculation.(i)
Threshold-crossing pitch detectors (ZXABE, TABE, and TTABE, Zero-Crossing Rate, Peak rate, Slope event rate)(ii)
Parallel processing method(iii)
Average Magnitude Difference Function (AMDF)(iv)
Envelope analysis methods(v)
Autocorrelation(vi)
YIN(vii)
Phase space(viii)
Deep learning and convolutional neural networks (end to end).


## 2.9.2. Frequency-Domain Methods

Frequency spectrum is widely used to detect the pitch value of a signal.(i)
Optimum comb filter(ii)
Tunable IIR filter(iii)
FIR filter(iv)
Cepstrum(v)
Harmonics(vi)
Multiresolution methods(vii)
Statistical frequency-domain methods(viii)
Neural networks(ix)
Maximum likelihood estimators


## 2.9.3. Time- and Frequency-Domain (Hybrid) Pitch Detectors

Some algorithms combine time and frequency domain to calculate fundamental frequency:(i)
Schroeder histogram(ii)
Brown comb filter method(iii)
Maher–Beauchamp pitch detector(iv)
Meddis–Hewitt model(v)
McAulay–Quatieri method(vi)
Neural networks(vii)
YAAPT


## 3. HDM Algorithm

The novel harmonic differences method (HDM) tries to exploit the differences between the harmonics of power spectrum of the signal with the goal of finding the most repeating difference. Harmonic spacings have been tried by Seneff [39] for telephony speech on real-time data with LDVT, Wu on guitar sound [40], and Dziubiński and Kostek [187] for various musical instrument sounds. Seneff used the area under the hump in frequency spectrum, creating a list of 8 estimates and applies temporal smoothing using median filter. We try to find the most repeating difference without temporal smoothing. We will extend these works by eliminating many steps in the implementation and generalizing the method to all kinds of sound waves, improving the speed and the accuracy. Matlab 2019 student version is used for HDM, autocorrelation, cepstrum, YIN, and YAAPT. Python 3.6.5 is used for CREPE and FCN implementations. General outline and pseudocode of the algorithm is as follows:(1)
Take the discrete Fourier transform and run the peak picking algorithm between the minimum f0 and maximum threshold frequencyFth. Length of the samples is taken as 1024 and Hamming windowed before the processing to smooth the signal and avoid the edge effects. Windowing is necessary due to the DFT’s vulnerability to discontinuities. Windowed Fourier Transform is given as follows:(2)Hk=∑i=0N−1xiwie−j2πki/N.Sampling rate is 16000 Hz yielding a frequency resolution of 15.625 Hz. Despite this low FFT resolution, the results of HDM are very promising. During this search, the minimum power of a partial should be at least1/γ times the magnitude of the largest partial. Here, γ is a prespecified constant such that(3)Hk>Hmaxγ,α,β,γ,f0min,Fth (upper-limit threshold frequency) are determined empirically with more than 7 million experiments on Hillenbrand dataset. There is no limit in the number of peaks to collect, but we set a threshold frequency value Fth, and we collect all peaks below that threshold. This threshold frequency is also determined empirically.(2)
To decide a true peak, its amplitude must be larger than the previous and next Fast Fourier coefficient.J⟵0iff^i>f0minandf^i<FththenifAi−1<Ai>Ai+1thenifAi>Hmax/γthenℱ^i⟵f^i,J=J+1endifendifendif(3)
In this list, remove the unnecessary peaks that are closer than minimum f0 value to one another.iff^i<f^i−1+f0minandA^i<A^i−1thenremovef^ifromℱ^,J⟵J−1endififf^i>f^i+1−f0minandA^i<A^i+1thenremovef^ifromℱ^,J⟵J−1endif(4)
In this list, remove the spurious peaks that are weaker than a predefined empirical ratio of the previous and next spectrum value.ifAi<Ai+1/αorifAi<Ai−1/αthenremovef^ifromℱ^,J⟵J−1endif(5)
Handle some special cases such as frequencies below 110 so that if the amplitude of first entry is less than 1/5 of the second entry remove first entry.iff^1<110Hz andA1<A2/βthenremovef^1fromℱ^,J⟵J−1endif(6)
Find the differences between the adjacent entries in the list(4)D^:d^i|d^i=f^i−f^i+1,d^i>f0min,i=1,2,…,N.(7)
Sort the list according to the repetition counts with a histogram.//f0wb: wideband fundamental frequency.//f0nb: narrowband fundamental frequency.//A0wb: amplitude of wideband fundamental frequency.//A0nb: amplitude of narrowband fundamental frequency.(5)list=sortlist,repeat count,f0nb=list1,1,D^=ℱ^−f0nb,D^=ℱ^,D^,d^i>0.(8)
Sort the final list according to the differences. First entry is the most repeating difference//remove zero elements if any(6)list=sortlist,d^i,f0wb=list1,1,A0wb=list1,2,A0nb=A0wb,ifabsf0wb−f0nb<f0min/2thenf0wb=f0nbendifelsef0wb⟵0,f0nb⟵0returnf0wb,f0nbHDM does not use temporal smoothing. Although it is possible to obtain better results using different parameters for each dataset, we use the same parameter set for all 3 datasets and for all wide and narrowband experiments. Our goal is to find a global parameter set that can achieve best results for wideband and narrowband telephone speech in all datasets.The relation between GPE andα is shown in Figure 3. In the small values of α, GPE goes too high, and after 5, it remains nearly constant. A typical value for α is between 6 and 12. This parameter is used to eliminate the spurious peaks in the list ℱ^ of candidate pitches. Elimination of redundant peaks is a key point in capturing the correct spacing between the harmonics.Figure 3
GPE vs.α.β parameter is used to handle some special cases such as frequencies below 110 Hz. Due to nonlinearity and complexity of equal loudness curves, different regions of human speech spectrum have different loudness. Therefore, we used β parameter for handling low-frequency components. The effect of α and β parameters on GPE is depicted in Figure 4. Linear region of human auditory system is usually considered between 20 and 1000 Hz (we do believe that a value between 1100 and 1200 Hz is a better choice for upper limit), and logarithmic region is the rest of the audible spectrum [188–190].Figure 4
GPE vs.αandβ.Another important parameter is the value of minimum f0, and it heavily affects the GPE as shown in Figure5. Hillenbrand and Texas datasets provide ground truth value for f0, and minimum fundamental frequency value is 82 Hz.Figure 5
GPE vs.αand minimumf0.


## 4. Datasets and Experimental Setup

In this study, we employ 3 vowel datasets: Hillenbrand dataset (http://homepages.wmich.edu/∼hillenbr/voweldata.html) [191–193], Texas Vowel dataset (https://personal.utdallas.edu/∼assmann/KIDVOW1/North_Texas_vowel_database.html) [194, 195], and vowel part of TIMIT [137] dataset including the SA samples. Hillenbrand dataset is a collection of 1668 vowels, including boy, girl, man, and woman speech samples. Total length of the samples is 223.878 seconds and length of files ranges from 2049 to 6872 with mean 2147.515 and standard deviation 489.218. Texas dataset is a collection of 3314 vowels with kid, female, and male vowels. Total length is 698.4711 seconds and length of files ranges from 1110 to 24257 with mean 3372.22 and standard deviation 1010.025. TIMIT is a large collection of words and phones. Usually, it is divided into training, development, and test sets. In this work, we use all vowels, including SA samples. SA samples are not used in classification applications due to overfitting concerns; however, this is not relevant in our case. TIMIT vowel set comprises 78374 samples, including 24017 female and 54357 male voices. Total length is 7521.265 seconds, length of files ranges from 74 to 7735 with mean 1535.461, and standard deviation of 778.7452. Detailed information is shown in Table 1.Table 1
Datasets used in this work.
BoyGirlMaleFemaleHillenbrand324228540576Texas12329721110TIMIT5435724017In Hillenbrand and Texas datasets, there are 12 American English vowel classes /i/ (heed), /ɪ/ (hid), /ε/ (head), /æ/ (had), /e/ (hayed), /ɑ/ (hod), /ɔ/ (hawed), /o/ (hoed), /ʌ/ (hud), /ʊ/ (hood), /u/ (who’d), /ɝ/ (heard). Vowels are obtained from the /hVd/ syllable context. In TIMIT dataset, 20 vowels are included: /i/ (beet), /ɪ/ (bit), /ɛ/ (bet), /æ/ (had), /e/ (hayed), /ɑ/ (bott), /ɔ/ (bought), /o/ (boat), /ʌ/ (but), /ʊ/ (hood), /u/ (boot), /ɝ/ (herd) with additional /aɪ/ (bite), /ə/ (about), /ɔɪ/ (boy), [ü] (toot), [ɪ] (debit), [əh] (suspect), [aw] (bout), and [ɚ] (butter). TIMIT vowel classes can be reduced to 14 from 20 by eliminating the stress and semivowel intonations [137, 196, 197].In Hillenbrand dataset, fundamental frequency ground truth is calculated using autocorrelation followed by hand-editing. In Texas Vowel dataset, fundamental frequency ground truth is calculated by visual inspection together with semiautomatic LPC analysis. In the TIMIT corpus, transcriptions have been hand-verified. Transcriptions are obtained using the program SPIRE of MIT and then hand-verified by experienced acoustics phoneticians. But there is no fundamental frequency ground truth for TIMIT vowels. Therefore, we used average f0 values found by HDM, autocorrelation, cepstrum, YAAPT, CREPE, and FCN methods in a consistent manner in conjunction with the gender labels provided by the TIMIT dataset. In some samples of TIMIT dataset, finding the ground truth is very difficult even by visual expert inspection on the frequency spectrum. The harmonics and spacing between them can be spread nearly randomly. Such a sample from TIMIT dataset is depicted in Figure6.Figure 6
The near random distribution of harmonics and the spacing between them can make extraction of fundamental frequency too complicated.Autocorrelation of a signal with the symmetry property can be obtained using the following equation:(7)Aack=∑i=kN−1xixi−k,where k is the lag number.Cepstrum has the complexity ofONlogN, and power cepstrum can be calculated using the following formula:(8)Cp=ℱ−1logℱft22.As can be seen from Table2, HDM is the fastest method followed by cepstrum and autocorrelation. For autocorrelation and cepstrum, we used Naotoshi SEO’s (http://note.sonots.com/SciSoftware/Pitch.html) implementations, but in cepstrum, we changed the maximum pitch value as 500 Hz to obtain better results. Timings are average values of 10 consecutive runs on Hillenbrand dataset for a single speech sample. YIN, YAAPT (version 2016), FCN (2019), and especially CREPE (2018) are too far away from HDM, cepstrum, and autocorrelation in terms of speed. YIN [66], YAAPT [67, 68], and FCN [113] employ temporal smoothing, whereas HDM, autocorrelation, cepstrum, and CREPE [110] do not use temporal smoothing. HDM produces nearly identical results without hamming windowing, but the rest of the algorithms have worse results without hamming window, particularly, cepstrum doubles the error margin in TIMIT dataset. No other preprocessing was applied to the waveforms. In cepstrum and autocorrelation, we imposed an upper-limit frequency of 500 Hz. Without this limit, their performances are going only worse. This makes them unsuitable for the detection of high pitch values. In angry emotional speech samples, it is possible to see up to 700 Hz pitch values.Table 2
Average speeds of used methods on the Hillenbrand dataset in microseconds.
HDM (2021)AC (2008)CEPS (2008)YIN (2002)YAAPT (2016)CREPE (2018)FCN (2019)10819213214544832726396582522


## 5. Results

Many error measures are used to evaluate the pitch detection algorithms. Gross Pitch Error (GPE) is the average error, voicing detection error (VDE) if applicable are among the most used. Additionally, we introduce two different error measures. The first is thee10, which denotes the number of samples with more than 10% sway from the ground truth value. Another useful application of pitch estimation is the gender detection. Gender detection error can be very useful for the evaluation of these algorithms. GPE and e10 error are, respectively, defined as(9)GPE=1N∑i=1Nf^i−fi,e10=#i|i=1,…,N,f^i−fi>fi10.In Hillenbrand and Texas Vowel datasets, ground truth pitch values are given, and we can make a solid comparison with our predictions. In Figure7, such a comparison is depicted for Hillenbrand dataset. Boy, girl, man, and woman samples can clearly be seen to make an intuition over the frequency regions of these samples. Male voices have definitely lower f0 values compared with the boys, girls, and woman. It is nearly impossible to separate boy, girl, and woman voices using only f0 values. In Hillenbrand dataset, average f0 is 236.0 Hz for boys, 238.35 Hz for girls, 131.21 Hz for man, and 220.40 Hz for woman. Maximum f0 for boy, girl, man, and woman are 320, 303, 224, and 307 Hz, respectively. Minimum f0 for boy, girl, man, and woman are 183, 188, 90, and 149 Hz, respectively. There is no boy or girl sample with f0 value of lower than 180 Hz. Only 7 males have pitch values of greater than 190, and only 5 women have pitch values of lower than 160 Hz. In Texas dataset, there are kid, man, and woman classes. In Texas dataset, the difference between woman and kid classes is clearer than Hillenbrand dataset. In Texas dataset, we have 3-, 5-, and 7-year-old kid samples. This suggests that as age decreases, pitch values tend to be higher. Maximum f0 for kid, man, and woman samples are 392, 202, and 271 Hz, respectively. Minimum f0 for kid, man, and woman samples are 105 (sample k7bree05, this is a strange value for a kid so that we felt to check it manually), 82, and 141 Hz, respectively. In Texas Vowel dataset, only 1 kid sample has f0 value of lower than 180 Hz (abovementioned sample k7bree05), and only 3 males have f0 value of greater than 140 Hz. Only 7 woman samples have f0 value of lower than 160 Hz. In TIMIT dataset, we have only male and female classes, and average values conform well to the other two datasets as shown in Table 3. In TIMIT dataset, there are no supplied ground truth pitch values.Figure 7
f 0 ground truth (red) and HDM estimations (blue) in Hillenbrand Vowel dataset.Table 3
Averagef0 values for Hillenbrand, Texas, and TIMIT datasets by gender.
BoyGirlManWomanHillenbrand236238.35131.21220.40Texas245.76110.86217.23TIMIT119.07207.21As seen from Table4, in wideband Hillenbrand dataset, the proposed HDM has the smallest GPE, and AC has the smallest e10 error. Although autocorrelation is an old technique, it is highly successful in this dataset. In Texas dataset, FCN has the smallest GPE error, and AC has the smallest e10 error. In TIMIT dataset, FCN is the best performing method followed closely by HDM and cepstrum in GPE and the proposed HDM is the best method in e10 error. YIN produces too many outliers, and for YIN, we tried many different parameters to find better results. A box plot of the abovementioned algorithms is depicted in Figure 8. We need to emphasize that in the AC and cepstrum implementations, we imposed an upper-limit frequency of 500 Hz; otherwise, these methods produce worse results. No upper limit is applied in the remaining methods. Convolutional neural network methods are quite successful in wideband implementations, but as we will see right now, they are nearly blind in narrowband telephone speech data.Table 4
Experimental results on wideband Hillenbrand, Texas, and TIMIT datasets.
HillenbrandTexasTIMITGPEe10GPEe10GPEe10HDM7.656.419.4611.416.394.68AC8.174.6810.367.0922.748.79CEPS7.866.0014.5311.3811.277.45YIN16.718.6316.678.9043.8812.58YAAPT17.1813.5512.8714.7616.0015.13CREPE8.979.538.458.1212.336.84FCN9.108.758.177.425.825.29Bold values denote the best performance in the specified dataset and error type.Figure 8
Box plot of all algorithms for wideband TIMIT dataset.From now on, we extend our experiments to the telephony speech. For this purpose, we will apply band-pass filter to our datasets twice to completely remove the frequencies below 400 Hz and above 3400 Hz. In some telephony speech, this bandwidth can be applied between 300 Hz and 4000 Hz. We selected the low frequency as 400 Hz because the highest f0 value is 392 Hz in our datasets, and by selecting 400 Hz as threshold value, we remove the fundamental frequency from all samples in all datasets. This is one of the objectives of this algorithm.In band-passed Hillenbrand dataset, cepstrum remarkably is the most successful algorithm in all error types as shown in Table5. HDM is the second in GPE, and AC is the second in e10 error measures. Performance of CREPE and FCN is very disappointing in band-passed speech. In narrowband Texas dataset, cepstrum is the most successful algorithm in all error types as shown in Table 5. HDM is the second in GPE, and AC is the second in e10 error type.Table 5
Experimental results on narrowband Hillenbrand, Texas, and TIMIT datasets.
HillenbrandTexasTIMITGPEe10GPEe10GPEe10HDM34.7219.3039.4229.0615.2617.88AC37.3315.5355.7122.2469.4723.30CEPS11.237.3737.8917.5075.1725.36YIN72.6425.12130.2934.85160.9740.83YAAPT51.9350.1845.8342.1827.1028.14CREPE111.5242.51134.8746.11162.4450.66FCN185.5370.08191.8171.73194.7967.28Bold values denote the best performance in the specified dataset and error type.In narrowband TIMIT dataset, our novel HDM algorithm is superior to all other methods in GPE ande10 error measures as seen in Table 5. YAAPT is the second best in GPE, and AC is the second in e10 error. YAAPT is primarily designed for telephone speech. In TIMIT dataset, 22670 samples are shorter than 1024 in length. This may explain the failure of cepstrum in narrowband TIMIT dataset. To further clarify the underlying essence of this failure, we removed the samples that are shorter than 1024 and rerun cepstrum on narrowband TIMIT data. In this scheme, cepstrum GPE error dropped to 49.85 from 75.17 and e10 error to 17.41 from 25.36. Arguably, we can conclude that zero padding can severely affect the performance of the cepstrum. Box plot of these algorithms for narrowband TIMIT dataset are shown in Figure 9. CREPE and FCN are nearly useless in narrowband speech. This may be due to the fact that their training is done only on the wideband speech. They need to be trained for narrowband speech as well. Convolutional neural networks are quite successful in wideband speech data, however, we must keep in our mind that they are extremely slow compared with HDM, AC, and cepstrum as shown in Table 2. HDM is the fastest method in these experiments.Figure 9
Box plot of all algorithms for narrowband TIMIT dataset.


## 6. Gender Detection Implementations

Gender detection is an application of fundamental frequency detection. Although gender is not restricted to pitch value, it is highly related to its value. Pitch has specific ranges between men, women, boys, and girls. Therefore, gender evaluation of thef0 algorithms is a good measure for the robustness. Here, we present the gender detection errors for wideband and band-passed TIMIT dataset. In TIMIT dataset, the gender information is given with the first letter of the name of the speech sample.As seen from Table6, in wideband TIMIT dataset, HDM is the best method in gender detection by a significant margin, FCN is the second, and cepstrum is the third. Cepstrum’s success is well known in male speech samples. In the TIMIT dataset, there are 24017 female and 54357 male samples. This may explain the success of cepstrum in this large dataset. In Table 7, we can clearly conclude that HDM has no match specifically in male samples.Table 6
Gender detection results on wideband TIMIT dataset.
Male incorrectFemale incorrectError (%)HDM83214762.94AC533124039.86CEPS299932197.93YIN8136267413.79YAAPT5475248310.15CREPE343522957.31FCN130217243.86Bold value denotes the best performance.Table 7
Gender detection results on narrowband TIMIT dataset.
Male incorrectFemale incorrectError (%)HDM36365618.83AC15091391924.25CEPS16258424826.16YIN24393735940.51YAAPT6129512414.35CREPE291551033150.38FCN343981771766.49Bold value denotes the best performance.


## 7. Conclusions

The experimental results show that proposed harmonic differences can safely be used to detect fundamental frequency in wideband and narrowband telephony speech. The new algorithm shows great success particularly in the large TIMIT dataset. Fast Fourier Transform has a natural resolution problem, but in this article, despite the low resolution of the implementation, the results are satisfactory. It is robust to band-limiting and moderate inharmonicity. HDM algorithm is the fastest method, and further speed improvements can be expected. FCN and CREPE are performing remarkably well in wideband data, but they are too slow compared with the other methods. Therefore, they cannot be used for real-time applications, but they can be helpful in ground truth determination. An interesting finding is the highly disappointing results of the FCN and CREPE algorithms in narrowband speech. Although they are quite successful in wideband speech datasets, they produced low success rates in all band-passed datasets. We should bear in mind that FCN and CREPE are end-to-end algorithms, and they take the raw waveform as input without using the frequency-domain descriptors. Most of the useful pitch information is hidden in the low part (0–400 Hz) of band-passed signal and without this data, FCN and CREPE are unable to extract the necessary features for pitch determination. The cepstrum algorithm is very old compared to YIN, YAAPT, CREPE, and FCN, but in some cases, it can present better predictions. FCN and CREPE are CNN-based methods; FCN is using temporal smoothing, whereas CREPE does not use temporal smoothing but FCN is still much faster than CREPE.In the future works, we plan to implement temporal smoothing in HDM. Temporal smoothing can be quite efficient in f0 detection and is used by many algorithms, including YIN, YAAPT, and FCN. Another future direction is testing the ability of HDM in noisy environments and musical sounds that needs to be handled. Pitch refinement is another technique that can be incorporated inside HDM.



---

*Source: 6658951-2021-10-19.xml*