Automatic detection of speech breath events using deep neural
networks

Younes El Belghiti¹, Khalid Daoudi², Khalid Minaoui¹, Alexis Deighton MacIntyre³

¹ Laboratory for research in computer science and telecommunications (LRIT), Faculty of Sciences, Mohammed V University in Rabat, Morocco

² INRIA centre at the University of Bordeaux, France

³ MRC Cognition and Brain Sciences Unit, University of Cambridge, Cambridge, UK

Abstract

Respiration and speech are intrinsically related, in fact the latter is a typical example of the respiratory apparatus functioning. Breath sounds are the most common nonverbal realizations in speech and their automatic identification can be useful in many speech processing applications, such as speech synthesis/recognition or voice biomarkers. Despite this fact, the acoustic analysis/identification of breath events in speech remain under-explored. In this paper, we propose a novel highly accurate algorithm for breath events detection from read and spontaneous speech which is based on feature extraction using the VGGish pre-trained model and an autoencoder model for classification. We evaluate the performance of our algorithm on a state-of-the-art dataset of speech and respiratory signals and compare it, in particular, to the most recent state-of-the-art method in the field. The results show that our algorithm achieves a substantially higher accuracy with a better model stability.

*Keywords:* Speech breathing, breath events detection, deep neural networks, VGGish model.
# 1. Introduction

Respiration is an integral part of human speech and involves a complex interaction of physiological processes (Fant, 1960; Laver, 1994; Levelt, 2001; Fuchs and Rochet-Capellan, 2021). These processes facilitate a continuous flow of air that not only supports basic respiratory functions but also enables speech production through phonation, articulation, and the modulation of prosody (Hixon, 1987). During speech, individuals often take a quick inhalation followed by a controlled and prolonged exhalation. This pattern allows for the articulation of sentences before the need to inhale again, optimizing clarity and effectiveness in communication (Fuchs and Rochet-Capellan, 2021). Before beginning speech, individuals might inhale more deeply than normal. This deeper inhalation ensures that there is sufficient air to complete sentences without unnecessary pauses, thus maintaining the natural flow of dialogue (Rochet-Capellan and Fuchs, 2013b; Winkworth et al., 1994, 1995; Rochet-Capellan and Fuchs, 2013a). In alignment with linguistic requirements, the timing of breaths during speech is typically synchronized with syntactic boundaries, such as pauses at commas. This natural synchronization helps in maintaining the fluency and rhythm of speech (Conrad et al., 1983; Winkworth et al., 1994; Grosjean and Collins, 1979). Engaging in complex dialogue increases cognitive demand, which may accelerate respiratory rates as the body requires more oxygen for increased brain activity (Shea, 1996; Ben-Tal et al., 2019). Emotional states also influence breathing during speech; for instance, anxiety can lead to quicker breaths, impacting speech pace and fluency (Boiten, 1998; Ainsworth, 1939).

Conversely, the capabilities and limitations of the respiratory system can influence speech quality. A larger lung capacity allows for longer phrases without taking a breath, which influences the rhythm and flow of speech (Boucher and Lalonde, 2015; Sperry and Klich, 1992). Respiratory ailments like asthma can reduce lung capacity, leading to more frequent breaths and potentially disrupting the smooth delivery of speech (Lee et al., 1993; Wiechern et al., 2018). Conditions that impair breathing, such as Chronic obstructive pulmonary disease (COPD) or asthma, force shorter and more frequent breaths, thereby affecting speech fluency and reducing vocal loudness and clarity (Lee et al., 1993; Wiechern et al., 2018; Hoit et al., 1990).

Early foundational work on speech breathing by Hoit and Hixon explored how variables such as age, gender, and body type affect speech breathing, providing a baseline for subsequent studies (Hoit and Lohmeier, 2000; Hoit et al., 1990; Watson and Hixon, 1985). Further investigations into the dynamics of the chest and airflow during speech, such as those by Hixon et al., highlighted the intricate control of breathing during speech production (Hixon, 1973; Hixon and Weismer, 1995). Analysis of the variations in lung volumes during reading was carried out in (Winkworth et al., 1994), showing the influence of speech intensity and linguistic factors. Other works discussed the differences in breathing patterns between read and spontaneous speech, highlighting cognitive load effects (Henderson et al., 1965; Mitchell et al., 1996; Wang et al., 2010b). It has also been shown that speech breathing differs significantly from quiet breathing, with speech breathing being characterized by shorter inhalations and longer exhalations to accommodate speech flow (Hixon, 1987; Włodarczak and Heldner, 2017).

Early techniques to measure respiratory activity during speech utilized devices such as the pneumotachograph and the Rothenberg mask (Rothenberg, 1977; Dromey et al., 1995;
Koenig, 2000). Recent technological advancements have led to the development of biofeed-back systems that monitor breathing during speech (Frey et al., 2018; Mathew et al., 2012). However, the most commonly used technique is inductance plethysmography which measures thoracic and abdominal movements, using two wearable flexible belts, to derive respiratory signals (McKenna and Huber, 2019). Despite the effectiveness of these techniques, their complexity and cost have limited their widespread application (Shea et al., 1987; Frey et al., 2018).

From this perspective, the use of the acoustic speech signal to characterize the respiration activity is naturally an appealing technique, because of its simple use and cost-effectiveness. This area, however, remains under-explored. In this paper, we present a novel technique for the automatic detection of breath events from speech signals. This technique is based on the use of the foundational model VGGish (Hershey et al., 2017) as a feature extractor and an autoencoder model as a classifier. We evaluate our technique using a state-of-the-art database of respiratory and speech signals. We manually annotated the read and spontaneous speech of the latter to provide ground truth data for the evaluation. We compare the performance of the most commonly used features in the field, (log-)Mel and MFCC features, to the one achieved using features extracted by VGGish. This evaluation is carried out using two distinct approaches: firstly, with a convolutional neural network (CNN), and secondly, with the autoencoder classifier. We then comprehensively assess our results with respect to a recent state-of-the-art technique (Yang et al., 2024) for which the code<fnref n="1" /> was made available. We could not compare to the most recent technique (Mohamed Ismail Yasar Arafath and Routray, 2025) because the code<fnref n="2" /> made available by the authors cannot be executed. We also make the data<fnref n="3" /> and the implementation code<fnref n="4" /> of our method accessible to the community for full reproducibility and to serve as an additional baseline in future studies.

The paper is organized as follows. The next section presents the positioning of our work with respect to the literature. Section 3 describes the data and the methodology we adopted for breath events detection. Section 4 presents the results and their analysis. The conclusion of our work is given in Section 5.

## 2. Related work

One approach that has been recently investigated to study the acoustics of speech breathing is the prediction of the respiratory signal from the speech signal. In this approach, measurement of the respiration signal during speech is used to train regression models to predict that signal from the speech audio. For instance, this constituted one the tasks of the ComParE challenge (Markitantov et al., 2020). Different deep architectures and processing methods have been proposed in this framework (MacIntyre et al., 2020; Härmä et al., 2023; Nallanthighal et al., 2021; Nallanthighal and Strik, 2019; Nallanthighal et al., 2020).

Another approach is to automatically detect breath events during speech. This task is less demanding because it comes back to a classification/segmentation task rather than inferring

<footnote n="1">¹https://github.com/ydqmkkx/Respiro-en</footnote>

<footnote n="2">²https://huggingface.co/spaces/kmiyasar/BreathSoundDetection</footnote>

<footnote n="3">³The UCL-SBM database and the manual annotations will be shared upon request</footnote>

<footnote n="4">⁴The code will be accessible online once the paper is published</footnote>
the entire respiratory signal (with a regression model for instance). Still, this approach can be
very useful in studying speech breathing dynamics, for example, to assess how often a patient
needs to breathe during conversational speech in clinical settings. The contribution of this
paper falls within this framework where relatively few studies have been reported.
(Ruinskiy and Lavner, 2007) proposed a template-matching approach, based on MFFC, to
detect the beginning and the end of inspiration segments. In (Igras and Ziólko, 2013), wavelet
analysis was used to estimate the number of breaths in a speech signal. In (Fukuda et al.,
2018), a two-step algorithm to detect breath events in Japanese telephone conversations was
proposed. A Gaussian mixture model was first used to detect breath events candidates,
then an SVM with various acoustic cues was applied to eliminate false alarms. In (Trouvain
et al., 2019, 2020; Werner et al., 2021), an analysis of some acoustico-phonetic features of
inspirations was reported, such as intensity, duration and spectral properties. In (Mohamed
Ismail Yasar Arafath and Routray, 2019b), the work of (Ruinskiy and Lavner, 2007) served as
a basis to combine an SVM classifier with the cepstrogram matrix to identified the inspiration
segments. Then, the same authors added a voice activity detector to improve the performance
and to reduce the execution time of their method (Mohamed Ismail Yasar Arafath and
Routray, 2019a). Later, they extended their work by employing deep networks (BiLSTM
and CNN-dense) for frame-base classification (Mohamed Ismail Yasar Arafath and Routray,
2025). In (Székely et al., 2019), for the purpose of improving text-to-speech synthesis (TTS),
frame-level speech breath detection was performed using Mel-spectrogram and zero-crossing
rate as inputs to a CNN. Very recently, also in the TTS framework, (Yang et al., 2024)
proposed a self-training method for frame-wise breath detection, aiming to enhance breath
naturalness in TTS systems. Their approach combines rule-based initial annotation with
iterative pseudo-labeling, allowing the model to learn from a large speech corpus without
extensive manual labeling. Using a Conformer-based architecture with down/up-sampling
layers, their model outperforms previous methods, including (Székely et al., 2019), in breath
detection accuracy. In our contribution, we naturally considered (Yang et al., 2024) as the
baseline reference method and we comprehensively compared our method's performance to
theirs, including by re-training their model using our annotated data.

3. MATERIALS AND METHODS

3.1. *Speech data*

We used a state-of-the-art dataset, the UCL Speech Breath Monitoring (UCL-SBM) database (MacIntyre and Werner, 2023; Schuller et al., 2020), which consists in parallel recordings of speech and respiratory signals captured in a controlled office environment. The speech tasks include read passages, automated dialogue, poetry recitations, resting-state and spontaneous monologues. The sponataneous speech subset of the UCL-SBM database served as the evaluation set in the Breathing Sub-Challenge (BSC) of the ComParE'2000 challenge (Markitantov et al., 2020). The speech recordings were captured via an AKG C555L microphone and supplemented by respiratory signals measured by dual piezoelectric belts (MLT1132, ADInstruments, Castle Hill, Australia) which sense inhalation/exhalation activity. The speech and respiratory signals are sampled at 40 kHz. The database is structured into multiple channels, ensuring a comprehensive framework for the detailed examination of the diverse speech types and their associated respiratory patterns (Schuller et al., 2020). In
this paper, we analyzed the read and spontaneous speech of 52 speakers. The dataset consists
of 53 speakers but we discarded one speaker because of the quality of its recordings.

We carried out a pre-processing step on all the speech signals to improve the acoustic
quality of the recordings. We first filtered the signals by a 0.5 Hz high-pass filter to remove
low-frequency noise. Then, we applied a min-max normalization function to rescale the signal
amplitudes within the range [-1, 1], enhancing the contrast between low- and high-amplitude
components. This pre-processing yielded a better distinction between breath and non-breath
events.

3.2. *Basic acoustic characteristics*

Figure 1 shows an example of an utterance waveform, its spectrogram and the correspond-
ing respiratory signal. It can be seen that inspiration generally produces a wide-band noise
with energy spanning over a large band, from low to high frequencies, with a formant-like
structure generally around 2 kHz. Previous studies on the characteristics of speech breath
events (Ruinskiy and Lavner, 2007) have indicated, in particular, that (a) breath sounds do
not exhibit harmonics due to the lack of vocal cord vibration, (b) the duration of the breathing
sound is usually longer than phonemes in a sentence, (c) very short silence intervals are
present immediately before and after the breathing sound. All these characteristics were ob-
served in our data. In particular, the short silence intervals before and after of the breathing
sound are consistently present in our data, as illustrated in Figure 1. The inspiration sound
has a stable duration with no harmonics and the silence intervals clearly observed. Ruin-
skiy and Lavner (Ruinskiy and Lavner, 2007) referred to these silence intervals as "edges" of
breath events, which can serve as strong markers for event detection. In this paper, we call
"breath event" the segment consisting of both the inspiration sound and its edges.

3.3. *Manual annotation of speech breath*

The first step in our methodology was to generate ground truth data to evaluate the
performance of the breath detection algorithms. We used the Audacity software (Audacity,
2023) to manually annotate the speech signals, thus generating the time-boundaries of breath
events during speech. The process of manual annotation was performed using 3 modalities
in parallel: audition, respiratory signals and time/frequency representations, as illustrated in
Figure 1. The audio tracks were carefully listened to identify segments that potentially con-
tain inspiration events. The latter were characterized by audible cues that correspond to the
intake of breath during speech. Once a potential inspiratory event was detected by auditory
analysis, we carried out a visual inspection of the respiratory signal, the waveform and the
Mel-spectrogram. The latter was computed over 25ms sliding windows with a 10ms hop size
and 64 Mel filter banks. The breath events were identified using the acoustic characteristics
described in the previous section along with the respiratory signal information to confirm
and adjust the boundaries. The start (resp. end) of an inspiration was confirmed/adjusted
according to the time instant of the corresponding local minimum (resp. maximum) in the
respiratory signal, as illustrated in Figure 1. The use of the respiratory signals modality
allows a more accurate annotation than with the time/frequency representation or/and au-
dition alone. In particular, it allowed the differentiation between true inspiration events and
similar-sounding phonetic components which are not inhalations, such as the high-energy
consonants /s/ and /sh/. Respiratory signals are also more accurate that the thermal videos
<figure><img src="image_1.png" /><figcaption>Figure 1: Example of an utterance waveform (top), the corresponding respiratory signal (middle) and its Mel-spectrogram (bottom)</figcaption></figure>

used in (Mohamed Ismail Yasar Arafath and Routray, 2025; Mohamed Ismail Yasar Arafath and Routray, 2019b,a).

Consequently, this approach of cross-referencing using the 3 modalities ensured a high level of precision in identifying breath events, yielding the breath events boundaries as well as the labeling of each 25ms frame, 0 for non-breath frames and 1 for breath frames.

### 3.4. Breath event statistics

Table 1 (resp. Table 2) presents the basic statistics of breath duration and rate of read (resp. spontaneous) speech, categorized by gender. Figure 2 shows the normalized distribution of the durations of breath events in read and spontaneous speech. The duration of breath events spans a large range in both read and spontaneous speech, but the mode of both distributions is the same duration of 400 ms. For read speech, whilst the minimum breath duration observed is approximately 103.3 ms, the maximum recorded breath duration reaches up to 1414.7 ms. The average breath duration across all participants is 469 ms, with men exhibiting slightly longer breath events compared to women. The average breath rate is 0.31 and is similar for men and women. For spontaneous speech, the minimum (resp. maximum) breath duration 189.9 ms (resp. 1925.2 ms) are greater than their read speech counterparts. The average breath duration 553.8 ms is also longer, with men exhibiting slightly longer breath events compared to women as in read speech. As expected, the average breath rate in spontaneous speech is naturally lower than in read speech, for both men and women. Note also that the values of the average breath durations are consistent with those reported in (Wang et al., 2010a).
<table><caption>Table 1: Breath events statistics of read speech</caption><thead><tr><th></th><th>All</th><th>Men</th><th>Women</th></tr></thead><tbody><tr><td>Number of speakers</td><td>52</td><td>23 (44.3%)</td><td>29 (55.7%)</td></tr><tr><td>Mean audio duration (s)</td><td>40.8</td><td>41.3</td><td>40.3</td></tr><tr><td>Number of breaths</td><td>521</td><td>224</td><td>297</td></tr><tr><td>Min breath duration (ms)</td><td>103.3</td><td>103.3</td><td>144.8</td></tr><tr><td>Max breath duration (ms)</td><td>1414.7</td><td>1274.5</td><td>1414.7</td></tr><tr><td>Average breath duration (ms)</td><td>469</td><td>500.3</td><td>445.4</td></tr><tr><td>Min breath rate (breath/s)</td><td>0.07</td><td>0.07</td><td>0.12</td></tr><tr><td>Max breath rate (breath/s)</td><td>1.17</td><td>1.14</td><td>1.17</td></tr><tr><td>Average breath rate (breath/s)</td><td>0.31</td><td>0.31</td><td>0.32</td></tr></tbody></table>

<table><caption>Table 2: Breath events statistics of spontaneous speech</caption><thead><tr><th></th><th>All</th><th>Men</th><th>Women</th></tr></thead><tbody><tr><td>Number of speakers</td><td>52</td><td>23 (44.3%)</td><td>29 (55.7%)</td></tr><tr><td>Mean audio duration (s)</td><td>235.09</td><td>216.55</td><td>252.66</td></tr><tr><td>Number of breaths</td><td>3877</td><td>1667</td><td>2210</td></tr><tr><td>Min breath duration (ms)</td><td>189.9</td><td>189.9</td><td>200.3</td></tr><tr><td>Max breath duration (ms)</td><td>1925.2</td><td>1754.5</td><td>1925.2</td></tr><tr><td>Average breath duration (ms)</td><td>553.8</td><td>590.4</td><td>525.5</td></tr><tr><td>Min breath rate (breath/s)</td><td>0.15</td><td>0.15</td><td>0.25</td></tr><tr><td>Max breath rate (breath/s)</td><td>1.05</td><td>0.95</td><td>1.05</td></tr><tr><td>Average breath rate (breath/s)</td><td>0.21</td><td>0.20</td><td>0.23</td></tr></tbody></table>

## 3.5. Feature extraction

Feature extraction was based on the Mel-spectrogram of each utterance. We considered two sets of features, a set of standard features and the features extracted by the VGGish foundation model (Hershey et al., 2017).

### 3.5.1. Standard features

The first standard features were the raw Mel and log-Mel spectral features, resulting in 64-dimensional feature vectors. The second features was the Mel-Frequency Cepstral Coefficients (MFCCs) where we considered the first 13 MFCCs augmented by their first and second order derivatives (Δ and ΔΔ), resulting in 39-dimensional feature vectors. The choice of these features was motivated by their wide use in several speech processing application, in particular they were the backbone features in related works.
<figure><img src="image_2.png" /><figcaption>Figure 2: Normalized distribution of breath events durations in read and spontaneous speech</figcaption></figure>

### 3.5.2. VGGish features

VGGish (Hershey et al., 2017) is a pre-trained deep learning model developed by Google for audio feature extraction. It was trained on Google's AudioSet (Gemmeke et al., 2017) —a vast collection of over 2 million human-labeled, 10-second sound clips from YouTube videos covering 632 audio event classes. VGGish transforms audio signals, segmented into 96x64 windows with a 20% hop size, into 128-dimensional embeddings, computing salient features capable of capturing audio content at a high abstraction level.

The main motivation behind the choice of VGGish is the diversity of sound types, besides speech, used in its training. This model has thus learned from speech and non-speech data. Moreover, the speech data therein are more naturalistic and likely to capture breathing sounds and other noises that are often edited in the training corpora of speech-specialized pre-trained models. The latter corpora may thus be sub-optimal for breath sensing because the acoustic correlates of speech breathing are often deliberately removed or attenuated in such recordings. This makes VGGish more suitable to our downstream task (speech sounds vs breath sounds) than popular pre-trained models, such as Wav2Vec (Baevski et al., 2020) or WavLM (Chen et al., 2021), which have been trained on speech data only. An additional motivation was the observation that VGGish yielded a better separation between breath and
non-breath events than Mel-features and MFCC, as illustrated in Fig 3. The latter represents
the 2-dimensional projection of the features using the t-SNE dimension reduction algorithm
(van der Maaten and Hinton, 2008) which is known to be more adequate to nonlinear data
than classical linear methods such as PCA.

In the VGGish original implementation, the preprocessing computes Mel spectrograms
with a window length of 25ms and a 10ms hop size. In order to make the duration of the VG-
Gish analyzing window adequate to our task, we changed the hop size to 1ms. Consequently,
each 128-dimensional output vector corresponded to a speech segment of 121ms (25+96) du-
ration. This was slightly above the minimal breath duration of read speech (103ms) but had
no impact on the performance evaluation, given the rare occurence of the breath event with
the minimal duration and the small resolution loss for that event.

Given that the time-resolution of VGGish features is lower than the standard features,
we used the manual annotation to generate a specific annotation of the VGGish features.
This annotation served as the ground truth in training the VGGish-based classifiers. This
was done as follows. Each of the 96 frames forming a VGGish input frame has a 0 or 1 label
given by the manual annotation. Then the average over the 96 frames was computed and we
labeled the corresponding output VGGish feature as a breath (1) or a non-breath (0) feature
if the average was higher or lower than 0.5, respectively.

3.6. *CNN frame-based classification*

The CNN employed in our method was designed to classify input (standard or VGGish
features) frames as breath or non-breath frames. As illustrated in Figure 4, the architecture
consisted of a sequence of three 1D convolutional layers, each with a kernel size of K=3
and an increasing number of filters (F=32, 64, 128), enabling hierarchical feature extraction.
Each convolutional layer was followed by batch normalization to stabilize learning and max
pooling to reduce dimensionality. We used the Adam optimizer and 40 as the number of
epochs.

Following the convolutional feature extraction, the output was passed through a flatten
layer to convert the multi-dimensional feature maps into a one-dimensional vector suitable for
classification. This vector was then processed through a series of fully connected dense layers
with progressively decreasing node sizes (N=64, 32, 16), incorporating dropout regularization
to prevent overfitting. The final output layer consisted of a single neuron (N=1) with a
sigmoid activation function, producing a probability score for breath occurence. This model
was trained separately for both feature representations (standard and VGGish) to evaluate
their respective effectiveness in distinguishing breath and non-breath events.

3.7. *Autoencoder segment-based classification*

We additionally trained an autoencoder classifier composed of two parts, an encoder and a
decoder both incorporating a CNN architecture. The encoder's role is to compress the input
data into a lower-dimensional space (latent space). It was constructed with 1D convolutional
layers with kernel sizes of 3, designed to capture fine details in the data. These layers had
increasing filter counts (64, 128, 256), enabling the model to recognize more complex patterns.
Batch normalization followed each layer to prevent overfitting, and max pooling was used
to further reduce dimensionality by keeping only the highest values. We used the Adam
optimizer and 40 as the number of epochs.
<figure><img src="image_3.png" /><figcaption>Figure 3: t-SNE 2d-projection of the standard and VGGish features</figcaption></figure>

The decoder part of the autoencoder works on reconstructing the segmented data from the compressed representation. It employed 1D convolutional transpose layers, also with kernel sizes of 3, to upsample the input. These layers also had progressively decreasing filter counts (256, 128, 64) to efficiently reconstruct from the compressed form. Each layer was followed by a 1D convolutional layer with the same kernel size of 3 and filters of 128 and 64, respectively. Finally, the model included a 1D convolutional layer with a kernel size of 3 and a single filter, using a sigmoid activation function. Figure 5 illustrates the model architecture.

The autoencoder takes as input non-overlapping segments of 52 feature vectors. Thus, the inputs for the (log)Mel-spectrogram, MFCC and VGGish features are 52\*64, 52\*39 and 52\*128 matrices, respectively. The output is a 52\*1 vector, where each element is labeled 0 or 1 in training, using the manual annotation for the standard features and the specific annotation for the VGGish features.
<figure><img src="image_4.png" /><figcaption>Figure 4: CNN architecture</figcaption></figure>

### 3.8. Identification of breath segments boundaries

The identification of breath segment boundaries in a speech signal began with classifying each frame as either a breath (1) or non-breath (0). A sliding window approach was then applied over the resulting binary vector $V$, with a window size of $n$ samples and a hop size of 1. The indices corresponding to breath frames ($V(i) = 1$) were extracted into a set $I$. If two consecutive indices in $I$ differed by exactly 1, they were considered as part of the same breath segment, otherwise as the start of a new segment. This process continued over the entire signal, yielding the start and end indices of breath segments in $V$. Finally, the temporal locations of these indices were determined using the frame index, hop size, and sampling rate. In our implementation, we used $n = 10$ for the standard features and $n = 5$ for the VGGish features. The pseudo-code implementation of this identification process is presented in Algorithm 1.

## 4. Results

To assess the performance of our model, we employed standard classification metrics, including Precision, Recall, F1-score, and Intersection over Union (IoU). These metrics collectively provide a comprehensive evaluation of the performance of each model.

All scores were computed using a 10-fold cross validation and are given with their standard deviation (STD) between parenthesis. The performances of the frame-level binary classification and the breath segments detection algorithm using our method on read and spontaneous
**Algorithm 1** Identification of breath segments boundaries

**Require:** Input list *L*, window size *n*

**Ensure:** List of inspiration intervals with corresponding timestamps

1: $B \leftarrow []$ ▷ Binary list after thresholding

2: **for** $i = 1$ to $|L| - n + 1$ **do**

3:     $mean \leftarrow \frac{1}{n} \sum_{j=0}^{n-1} L[i + j]$

4:     **if** $mean > 0.5$ **then**

5:         Append 1 to *B*

6:     **else**

7:         Append 0 to *B*

8:     **end if**

9: **end for**

10:     $indices \leftarrow \{i|B[i] = 1\}$ ▷ Get indices of ones

11:     $segments \leftarrow []$ ▷ List to store start and end indices

12: **if** $indices$ is not empty **then**

13:         $start \leftarrow indices[0]$

14:         **for** $k = 1$ to $|indices| - 1$ **do**

15:             **if** $indices[k] - indices[k - 1] \neq 1$ **then**

16:                 Append $(start, indices[k - 1])$ to *segments*

17:         $start \leftarrow indices[k]$

18:     **end if**

19: **end for**

20:     Append $(start, indices[|indices| - 1])$ to *segments*

21: **end if**

22: **return** $time$ ▷ Convert indices to time values **return** *segments* with corresponding timestamps
<figure><img src="image_5.png" /><figcaption>Figure 5: Autoencoder architecture</figcaption></figure>

speech are reported in Table 3 and 5, respectively. We do not report the frame-level binary classification scores when using VGGish features because the comparison with the 25ms-frame original manual annotation would be biased by the label averaging in the specific manual annotation of VGGish frames. In Table 4 (resp. 6), we report the performance of the baseline method (Yang et al., 2024) on read (resp. spontaneous) speech using two implementations. Indeed, the code provided by (Yang et al., 2024) allows to classify frames, also computed on a 25ms window with a 10ms hop size, as breath or non-breath and to estimate breath segment boundaries (we call it "Baseline"). Additionally, it also allows to retrain their model using a given annotation of speech data. This allows full reproducibility and precise comparison to their method. We thus used the manual annotation of our speech data to retrain their model and perform the frame-level binary classification and breath segments detection (we call it "Retrained").
## 4.1. Evaluation of the CNN and autoencoder performance

### 4.1.1. Read speech

For the CNN classifier, the MFCC features yield the best IoU performance among the standard features, with about 20% relative improvement over (log)Mel features. On the other hand, VGGish features yield the highest IoU performance, with 18.6% relative improvement compared to MFCC. Note also that the IOU standard deviation (STD) of VGGish is much lower, showing a better stability of the CNN model when fed by the VGGish features.

For the autoencoder classifier, for every feature, the performance is substantially higher in all metrics. Moreover, STD of all metrics is 1%, showing stability of the autoencoder as compared to the CNN. MFCC yield again the best performance among the standard features, but in all metrics this time. This suggests that MFCC might have a better ability to characterize breath event than (log)Mel features. On the other hand, VGGish features achieved a substantially higher performance, reaching an IoU as high as **0.86**, which constitutes 34.3% relative improvement than MFCC.

### 4.1.2. Spontaneous speech

The behavior of all the models on spontaneous speech was globally similar to read speech, with generally a slight decrease in performance. In particular, for the autoencoder classifier, VGGish features achieved the highest performance, reaching an IoU as high as **0.83**, which constitutes 33.8% relative improvement than MFCC.

## 4.2. Comparison with the baseline

### 4.2.1. Read speech

Compared to the CNN, the baseline yields the best IoU performance, with about 28% and 8% relative improvement over MFCC and VGGish features, respectively. Note however that the STD of the latter is lower than the baseline STD. On the other hand, the autoencoder classifier achieves a higher performance than the baseline, for all features and metrics. In particular, the standard features yield more than 16% relative improvement for MFCC. More importantly, the VGGish features achieve an even more substantial relative IoU improvement of more than **56%**.

As for the baseline-retrained, it achieves a substantially better performance than the CNN for all features and metrics, with more than 46% and 28% IoU relative improvement than MFCC and VGGish features, respectively. This shows that the baseline model, when trained in a particular setting, is substantially more accurate than a simple CNN model trained using the features we considered. On the other hand, the autoencoder classifier yields a better performance than the baseline-retrained for all features and metrics. In particular, the frame-based binary classification using the standard features is more accurate in terms of average scores and stability, but the IoU scores are comparable. The later observation shows the competitive behavior of the baseline-retrained as compared to the autoencoder using standard features. However, using the VGGish features, the autoencoder classifier achieves an IoU of more than **36%** relative improvement than the baseline-retrained. This shows that our model is highly more accurate in detecting breath segments boundaries than both the baseline and the baseline-retrained.
### 4.2.2. Spontaneous speech

The behavior of all the models on spontaneous speech was globally similar to read speech, with generally a slight decrease in performance. The autoencoder classifier with VGGish features substantially outperformed the baseline and baseline-retrained by a relative IoU improvement of **56.6%** and **36.6%**, respectively. It is worth noting that the autoencoder model is more stable in spontaneous speech also.

There are several reasons that could explain the substantial higher performance of our model as compared to the baseline-retrained. First, the baseline model relies on standard acoustic features: Mel-spectrogram, Zero-Crossing Rate (ZCR), and Variance of Mel-Spectrogram (VMS). This might explain its similar behavior as compared to the autoencoder classifier based on standard features, thought it uses a different classifier (Conformer-BiLSTM). While these features capture spectral characteristics, they lack high-level semantic information about breath events. On the other hand, VGGish embeddings provide a more abstract, context-aware representation, allowing our model to distinguish breath events from speech more effectively. Second, BiLSTMs can learn sequence relationships but may overfit to speech patterns, making breath detection less precise. Indeed, breath events are sparse in speech, making them harder to detect with traditional sequence models. On the other hand, autoencoders were designed for anomaly detection, (Neloy and Turgeon, 2024) making them ideal for detecting rare breath events. Moreover, autoencoders are known to handle unbalanced data, which is the case in our setting as breath events are rare compared to speech. Additionally, conformer blocks are computationally demanding and may struggle with relatively short events like breaths.

<table><caption>Table 3: Performance of our proposed method on read speech</caption><thead><tr><th>Features</th><th>Precision</th><th>Recall</th><th>F1-Score</th><th>IoU</th></tr></thead><tbody><tr><td colspan="5" style="text-align: center; font-weight: bold;">CNN</td></tr><tr><td>Mel</td><td>0.76 (0.08)</td><td>0.70 (0.08)</td><td>0.72 (0.08)</td><td>0.35 (0.06)</td></tr><tr><td>log-Mel</td><td>0.74 (0.09)</td><td>0.74 (0.09)</td><td>0.74 (0.09)</td><td>0.36 (0.07)</td></tr><tr><td>MFCC</td><td>0.70 (0.07)</td><td>0.71 (0.07)</td><td>0.70 (0.07)</td><td>0.43 (0.06)</td></tr><tr><td>VGGish</td><td>-</td><td>-</td><td>-</td><td>0.51 (0.01)</td></tr><tr><td colspan="5" style="text-align: center; font-weight: bold;">Autoencoder</td></tr><tr><td>Mel</td><td>0.83 (0.01)</td><td>0.83 (0.01)</td><td>0.83 (0.01)</td><td>0.61 (0.01)</td></tr><tr><td>log-Mel</td><td>0.84 (0.01)</td><td>0.83 (0.01)</td><td>0.83 (0.01)</td><td>0.62 (0.01)</td></tr><tr><td>MFCC</td><td>0.85 (0.01)</td><td>0.85 (0.01)</td><td>0.85 (0.01)</td><td>0.64 (0.01)</td></tr><tr><td>VGGish</td><td>-</td><td>-</td><td>-</td><td><b>0.86 (0.01)</b></td></tr></tbody></table>

<table><caption>Table 4: Performance of the baseline system (Yang et al., 2024) on read speech</caption><thead><tr><th>Method</th><th>Precision</th><th>Recall</th><th>F1-score</th><th>IoU</th></tr></thead><tbody><tr><td>Baseline</td><td>0.72 (0.13)</td><td>0.70 (0.13)</td><td>0.71 (0.13)</td><td>0.55 (0.05)</td></tr><tr><td>Retrained</td><td>0.81 (0.06)</td><td>0.80 (0.06)</td><td>0.80 (0.06)</td><td>0.63 (0.04)</td></tr></tbody></table>
<table><caption>Table 5: Performance of our proposed method on spontaneous speech</caption><thead><tr><th>Features</th><th>Precision</th><th>Recall</th><th>F1-Score</th><th>IoU</th></tr></thead><tbody><tr><td colspan="5" style="text-align: center; font-weight: bold;">CNN</td></tr><tr><td>Mel</td><td>0.73 (0.08)</td><td>0.70 (0.08)</td><td>0.71 (0.08)</td><td>0.35 (0.06)</td></tr><tr><td>log-Mel</td><td>0.70 (0.09)</td><td>0.74 (0.09)</td><td>0.72 (0.09)</td><td>0.36 (0.07)</td></tr><tr><td>MFCC</td><td>0.70 (0.07)</td><td>0.70 (0.07)</td><td>0.70 (0.07)</td><td>0.41 (0.06)</td></tr><tr><td>VGGish</td><td>-</td><td>-</td><td>-</td><td>0.55 (0.01)</td></tr><tr><td colspan="5" style="text-align: center; font-weight: bold;">Autoencoder</td></tr><tr><td>Mel</td><td>0.81 (0.01)</td><td>0.81 (0.01)</td><td>0.81 (0.01)</td><td>0.59 (0.01)</td></tr><tr><td>log-Mel</td><td>0.85 (0.01)</td><td>0.84 (0.01)</td><td>0.84 (0.01)</td><td>0.63 (0.01)</td></tr><tr><td>MFCC</td><td>0.80 (0.01)</td><td>0.86 (0.01)</td><td>0.83 (0.01)</td><td>0.62 (0.01)</td></tr><tr><td>VGGish</td><td>-</td><td>-</td><td>-</td><td><b>0.83 (0.01)</b></td></tr></tbody></table>

<table><caption>Table 6: Performance of the baseline system (Yang et al., 2024) on spontaneous speech</caption><thead><tr><th>Method</th><th>Precision</th><th>Recall</th><th>F1-score</th><th>IoU</th></tr></thead><tbody><tr><td>Baseline</td><td>0.70 (0.15)</td><td>0.70 (0.13)</td><td>0.70 (0.14)</td><td>0.53 (0.05)</td></tr><tr><td>Retrained</td><td>0.80 (0.05)</td><td>0.82 (0.05)</td><td>0.81 (0.06)</td><td>0.61 (0.05)</td></tr></tbody></table>

## 5. Conclusion

The automatic detection of speech breath events from the acoustic signal can be relevant in many speech technology applications such as, text-to-speech synthesis, speech recognition and the development of voice biomarkers of respiratory disorders. This area of research has been however under-explored. In this paper, we proposed a novel highly accurate algorithm for the automatic detection of breath events during read and spontaneous speech. This algorithm leverages the discriminative power of the acoustic features extracted using the VGGish foundation model and uses an autoencoder model for the classification of breath events. For evaluation, we first manually annotated all the speech signals of a state-of-the-art dataset of read and spontaneous speech recorded along with respiration activity signals. We performed a high precision annotation by using 3 modalities: audition, the respiratory signals and the time/frequency representations. We first compared the performance of our algorithm to a CNN classifier fed by standard acoustic features, (log)Mel and MFCC. We then carried out a comprehensive comparison with the latest state-of-the-art method for breath events detection. The results showed that the performance of our algorithm was substantially higher, both in term of accuracy and model stability. We made the code available to the community to (potentially) serve as a baseline in future studies. A limitation of our algorithm is that it was evaluated using data recorded in a controlled noise-free environment. The latter is indeed sufficient for our next objective, the analysis of the impact of respiratory disorders on speech (breathing) collected in controlled clinical settings. However, in other applications, noise robustness can be an important factor. We did not (yet) analyze the behavior of our algorithm in the presence of noise, this will be the purpose of a future work.
## References

Ainsworth, S., 1939. Studies in the psychology of stuttering: Xii: emphatic breathing of auditors while listening to stuttering speech. Journal of Speech Disorders 4, 149-156.

Audacity, T., 2023. Audacity. URL: https://www.audacityteam.org/.

Baevski, A., Zhou, Y., Mohamed, A., Auli, M., 2020. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural information processing systems 33, 12449-12460.

Ben-Tal, A., Wang, Y., Leite, M.C., 2019. The logic behind neural control of breathing pattern. Scientific reports 9, 9078.

Boiten, F.A., 1998. The effects of emotional behaviour on components of the respiratory cycle. Biological psychology 49, 29-51.

Boucher, V.J., Lalonde, B., 2015. Effects of the growth of breath capacities on mean length of utterances: How maturing production processes influence indices of language development. Journal of Phonetics 52, 58-69.

Chen, S., Wang, C., Chen, Z., Wu, Y., Liu, S., Chen, Z., Li, J., Kanda, N., Yoshioka, T., Xiao, X., Wu, J., Zhou, L., Ren, S., Qian, Y., Qian, Y., Zeng, M., Wei, F., 2021. Wavlm: Large-scale self-supervised pre-training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing 16, 1505-1518.

Conrad, B., Thalacker, S., Schönle, P., 1983. Speech respiration as an indicator of integrative contextual processing. Folia Phoniatrica et Logopaedica 35, 220-225.

Dromey, C., Ramig, L.O., Johnson, A.B., 1995. Phonatory and articulatory changes associated with increased vocal intensity in parkinson disease: A case study. Journal of Speech, Language, and Hearing Research 38, 751-764.

Fant, G., 1960. Acoustic theory of speech production. Mouton, The Hague.

Frey, J., Grabli, M., Slyper, R., Cauchard, J.R., 2018. Breeze: Sharing biofeedback through wearable technologies, in: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, pp. 1-12.

Fuchs, S., Rochet-Capellan, A., 2021. The respiratory foundations of spoken language. Annual Review of Linguistics 7, 13-30.

Fukuda, T., Ichikawa, O., Nishimura, M., 2018. Detecting breathing sounds in realistic japanese telephone conversations and its application to automatic speech recognition. Speech Communication 98, 95-103.

Gemmeke, J.F., Ellis, D.P.W., Freedman, D., Jansen, A., Lawrence, W., Moore, R.C., Plakal, M., Ritter, M., 2017. Audio set: An ontology and human-labeled dataset for audio events, in: Proc. IEEE ICASSP.
Grosjean, F., Collins, M., 1979. Breathing, pausing and reading. *Phonetica* 36, 98–114.

Härmä, A., Grossekathöfer, U., Ouweltjes, O., Nallanthighal, V.S., 2023. Sensing of inspiration events from speech: comparison of deep learning and linguistic methods. *arXiv preprint arXiv:2305.11683*.

Henderson, A., Goldman-Eisler, F., Skarbek, A., 1965. Temporal patterns of cognitive activity and breath control in speech. *Language and Speech* 8, 236–242.

Hershey, S., Chaudhuri, S., Ellis, D.P.W., Gemmeke, J.F., Jansen, A., Moore, C., Plakal, M., Platt, D., Saurous, R.A., Seybold, B., Slaney, M., Weiss, R., Wilson, K., 2017. Cnn architectures for large-scale audio classification, in: Proc. *IEEE ICASSP*.

Hixon, T.J., 1973. Respiratory function in speech. Normal aspects of speech, hearing, and language 80, 73–125.

Hixon, T.J., 1987. Respiratory function in speech and song. *College-Hill Press*.

Hixon, T.J., Weismer, G., 1995. Perspectives on the edinburgh study of speech breathing. *Journal of Speech, Language, and Hearing Research* 38, 42–60.

Hoit, J.D., Banzett, R.B., Brown, R., Loring, S.H., 1990. Speech breathing in individuals with cervical spinal cord injury. *Journal of Speech, Language, and Hearing Research* 33, 798–807.

Hoit, J.D., Lohmeier, H.L., 2000. Influence of continuous speaking on ventilation. *Journal of speech, language, and hearing research* 43, 1240–1251.

Igras, M., Ziólko, B., 2013. Wavelet method for breath detection in audio signals, in: 2013 IEEE International Conference on Multimedia and Expo (ICME), IEEE. pp. 1–6.

Koenig, L.L., 2000. Laryngeal factors in voiceless consonant production in men, women, and 5-year-olds. *Journal of Speech, Language, and Hearing Research* 43, 1211–1228.

Laver, J., 1994. Principles of phonetics. Cambridge university press.

Lee, L., Loudon, R.G., Jacobson, B.H., Stuebing, R., 1993. Speech breathing in patients with lung disease. *American Review of Respiratory Disease* 147, 1199–1199.

Levelt, W.J., 2001. Relations between speech production and speech perception: Some behavioral and neurological observations. *Language, brain and cognitive development: Essays in honor of Jacques Mehler*, 241–56.

van der Maaten, L., Hinton, G., 2008. Visualizing data using t-SNE. *Journal of Machine Learning Research* 9, 2579–2605.

MacIntyre, A.D., Rizos, G., Batliner, A., Baird, A., Amiriparian, S., Hamilton, A., Schuller, B.W., 2020. Deep attentive end-to-end continuous breath sensing from speech, in: Proc. of Interspeech.
MacIntyre, A.D., Werner, R., 2023. An automatic method for speech breathing annotation, in: Proc. of ESSV, TUDpress, Dresden.

Markitantov, M., Dresvyanskiy, D., Mamontov, D., Kaya, H., Minker, W., Karpov, A., et al., 2020. Ensembling end-to-end deep models for computational paralinguistics tasks: Compare 2020 mask and breathing sub-challenges., in: INTERSPEECH, pp. 2072–2076.

Mathew, J., Semenova, Y., Farrell, G., 2012. A miniature optical breathing sensor. Biomedical optics express 3, 3325–3331.

McKenna, V.S., Huber, J.E., 2019. The accuracy of respiratory calibration methods for estimating lung volume during speech breathing: A comparison of four methods across three adult cohorts. Journal of Speech, Language, and Hearing Research 62, 2632–2644.

Mitchell, H.L., Hoit, J.D., Watson, P.J., 1996. Cognitive-linguistic demands and speech breathing. Journal of Speech, Language, and Hearing Research 39, 93–104.

Mohamed Ismail Yasar Arafath, K., Routray, A., 2019a. Automatic detection of breath using voice activity detection and svm classifier with application on news reports, in: INTERSPEECH, pp. 609–613.

Mohamed Ismail Yasar Arafath, K., Routray, A., 2019b. Automatic measurement of speech breathing rate, in: 27th European Signal Processing Conference (EUSIPCO), pp. 1–5.

Mohamed Ismail Yasar Arafath, K., Routray, A., 2025. Detection of breath sounds in speech: A deep learning approach. Engineering Applications of Artificial Intelligence 141, 109808.

Nallanthighal, V.S., Härmä, A., Strik, H., 2020. Speech breathing estimation using deep learning methods, in: ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE. pp. 1140–1144.

Nallanthighal, V.S., Mostaani, Z., Härmä, A., Strik, H., Magimai-Doss, M., 2021. Deep learning architectures for estimating breathing signal and respiratory parameters from speech recordings. Neural Networks 141, 211–224.

Nallanthighal, V.S., Strik, H., 2019. Deep sensing of breathing signal during conversational speech, in: Proc. of Interspeech.

Neloy, A.A., Turgeon, M., 2024. A comprehensive study of auto-encoders for anomaly detection: Efficiency and trade-offs. Machine Learning with Applications 17.

Rochet-Capellan, A., Fuchs, S., 2013a. Changes in breathing while listening to read speech: the effect of reader and speech mode. Frontiers in psychology 4, 906.

Rochet-Capellan, A., Fuchs, S., 2013b. The interplay of linguistic structure and breathing in german spontaneous speech, in: Interspeech 2013-14th Annual Conference of the International Speech Communication Association, p. 1228.

Rothenberg, M., 1977. Measurement of airflow in speech. Journal of Speech and Hearing Research 20, 155–176.
Ruinskiy, D., Lavner, Y., 2007. An effective algorithm for automatic detection and exact demarcation of breath sounds in speech and song signals. IEEE transactions on audio, speech, and language processing 15, 838-850.

Schuller, B.W., Batliner, A., Bergler, C., Messner, E.M., Hamilton, A., Amiriparian, S., Baird, A., Rizos, G., Schmitt, M., Stappen, L., et al., 2020. The interspeech 2020 computational paralinguistics challenge: Elderly emotion, breathing & masks, in: Proc. of Interspeech.

Shea, S., Walter, J., Pelley, C., Murphy, K., Guz, A., 1987. The effect of visual and auditory stimuli upon resting ventilation in man. Respiration Physiology 68, 345-357.

Shea, S.A., 1996. Behavioural and arousal-related influences on breathing in humans. Experimental Physiology: Translation and Integration 81, 1-26.

Sperry, E.E., Klich, R.J., 1992. Speech breathing in senescent and younger women during oral reading. Journal of Speech, Language, and Hearing Research 35, 1246-1255.

Székely, É., Henter, G.E., Gustafson, J., 2019. Casting to corpus: Segmenting and selecting spontaneous dialogue for tts with a cnn-lstm speaker-dependent breath detector, in: ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE. pp. 6925-6929.

Trouvain, J., Möbius, B., Werner, R., 2019. On acoustic features of inhalation noises in read and spontaneous speech, in: 1st International Seminar on the Foundations of Speech: Breathing, Pausing and Voice.

Trouvain, J., Werner, R., Möbius, B., 2020. An acoustic analysis of inbreath noises in read and spontaneous speech, in: Proceedings of Speech Prosody, pp. 789-793.

Wang, Y.T., Green, J., Nip, I., Kent, R., Kent, J., 2010a. Breath group analysis for reading and spontaneous speech in healthy adults. Folia phoniatrica et logopaedica: official organ of the International Association of Logopedics and Phoniatrics (IALP) 62, 297-302.

Wang, Y.T., Green, J.R., Nip, I.S., Kent, R.D., Kent, J.F., 2010b. Breath group analysis for reading and spontaneous speech in healthy adults. Folia Phoniatrica et Logopaedica 62, 297-302.

Watson, P.J., Hixon, T.J., 1985. Respiratory kinematics in classical (opera) singers. Journal of Speech, Language, and Hearing Research 28, 104-122.

Werner, R., Fuchs, S., Trouvain, J., Möbius, B., 2021. Inhalations in speech: Acoustic and physiological characteristics., in: Interspeech, pp. 3186-3190.

Wiechern, B., Liberty, K.A., Pattemore, P., Lin, E., 2018. Effects of asthma on breathing during reading aloud. Speech, Language and Hearing 21, 30-40.

Winkworth, A.L., Davis, P.J., Adams, R.D., Ellis, E., 1995. Breathing patterns during spontaneous speech. Journal of Speech, Language, and Hearing Research 38, 124-144.
Winkworth, A.L., Davis, P.J., Ellis, E., Adams, R.D., 1994. Variability and consistency in speech breathing during reading: Lung volumes, speech intensity, and linguistic factors. Journal of Speech, Language, and Hearing Research 37, 535–556.

Włodarczak, M., Heldner, M., 2017. Respiratory constraints in verbal and non-verbal communication. Frontiers in psychology 8, 708.

Yang, D., Koriyama, T., Saito, Y., 2024. Frame-wise breath detection with self-training: An exploration of enhancing breath naturalness in text-to-speech, in: Interspeech 2024, pp. 4928–4932.