SILENCE: Protecting privacy in offloaded speech understanding on resource-constrained devices
Authors: DONGQI CAI
Shangguang Wang
Zeling Zhang
Felix Xiaozhu Lin
Mengwei Xu
Venue: NeurIPS 2024 poster (2024)
Keywords: spoken language understanding
resource-constrained devices
privacy-preserving
Primary area: speech_and_audio
TL;DR: We provide a lightweight, privacy-preserving encoder that can be efficiently embedded into low-power audio devices.

Abstract: Speech serves as a ubiquitous input interface for embedded mobile devices. 
Cloud-based solutions, while offering powerful speech understanding services, raise significant concerns regarding user privacy. 
To address this, disentanglement-based encoders have been proposed to remove sensitive information from speech signals without compromising the speech understanding functionality. 
However, these encoders demand high memory usage and computation complexity, making them impractical for resource-constrained wimpy devices.
Our solution is based on a key observation that speech understanding hinges on long-term dependency knowledge of the entire utterance, in contrast to privacy-sensitive elements that are short-term dependent. 
Exploiting this observation, we propose SILENCE, a lightweight system that selectively obscuring short-term details, without damaging the long-term dependent speech understanding performance.
The crucial part of SILENCE is a differential mask generator derived from interpretable learning to 
automatically configure the masking process.
We have implemented SILENCE on the STM32H7 microcontroller and evaluate its efficacy under different attacking scenarios. 
Our results demonstrate that SILENCE offers speech understanding performance and privacy protection capacity comparable to existing encoders, while achieving up to 53.3$\times$ speedup and 134.1$\times$ reduction in memory footprint.

SILENCE: Lightweight Protection for Privacy in Offloaded Speech Understanding
Dongqi Cai Beiyou Shenzhen Institude Shenzhen, Guangdong cdq@bupt. edu. cn
Shangguang Wang Beiyou Shenzhen Institude Shenzhen, Guangdong sgwang&bupt. edu. cn
Zeling Zhang
Beiyou Shenzhen Institude Shenzhen, Guangdong marovlo@bupt. edu.
Felix Xiaozhu Lin University of Virginia Charlottesville, VA, 22904 felixlin@virginia edu
Mengwei Xu
Beiyou Shenzhen Institude Shenzhen, Guangdong mwx@bupt edu. cn
Abstract
Speech serves ubiquitous input interface for embedded mobile de- vices. Cloud-based solutions, while offering powerful speech understanding services, raise significant concerns regarding disentanglemenntbaseed encoders have been proposed to remove sensitive infor- mation from speech signals without compromising the speech understanding fune- tionality. However, these encoders demand high memory usage and computation complexity, making them impractical for resource-constrained wimpy devices. Our solution is based on a key observation that speech understanding hinges on long-term dependency knowledge of the entire utterance, in contrast to privacy- sensitive elements that are short-term dependent. Exploiting this observation, we propose SILENCE, lightweight system that selectively obscuring short-term de- tails, without damaging the long-term dependent speech understanding perfor- mance. The crucial part of SILENCE differential mask generator derived from interpretable learning to automatically configure the masking process. We have implemented SILENCE on the STM32H7 microcontroller and evaluate its efficacy under different attacking scenarios. Our results demonstrate that SILENCE offers speech understanding performance and privacy protection capacity comparable existing encoders, while achieving up to 53.3x speedup and 134. 14 reduction in memory footprint.
privacy. To address this,
Introduction
Privacy concern for cloud speech service The volume of speech data uploaded to the cloud for spoken language understanding (SLU) steadily increasing wimpy devices where textual input is inconvenient [ 5
2 3), particularly in ubiquitous 61. c.g., home automation devices [7], smartwatches [8]. telehealth sensors g and smart factory sensors [T0] However, exposing raw speech signal to the cloud raises privacy concerns [TT]. It revealed that contractors regularly listened to confidential details in Siri recordings to improve its accuracy 12 This included private
discussions, medical information, and even intimate moments.
There are many aspects potential privacy leakage in cloud-based SLU. Among them: biometric or contextual privacy leakage have been well studied and somewhat solved by removing information relevant to such tasks without compromising the SLU accuracy [13 14] transcript protection (espe- cially sensitive entities) is more challenging since it is deeply entangled with the SLU task itself. As
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Trust-worthy embedded devices
Semi-honest cloud
Wimpy
Raw
Biometric, Sensitive entity. Sermantics, Transcript...
Audio
Device
Signal
Intent
(a). Offloaded speech understanding on wimpy devices
Resource-intensive
Signal
Raw
Disentangled Encoder
Hidden States
Intent
Obscured sensitive information
(b). Previous approaches protect speech privacy
Light-weight
...
Raaw Signal
SILENCE
Obscured
Signal
Intent
Obscured sensitive information
(c). Our solution: novel asymmetric dependency- encoder
Figure Illustration of offloaded speech understanding on resource-constrained devices and its privacy protection.
shown in Figure this paper focus on ensuring that cloud-based systems could efficiently classify the intent of SLU task (e.g., scheduling appointments controlling home devices) while refraining from identifying the concrete entities (c.g.. unintended names or passwords) in the spoken utterance, i.., high word eπrоr rate (WER) of Automatic Speech Recognition (ASR) task. This is also setting
commonly used in speech privacy protection [1 16
18]
Prior approaches A prevalent method for private speech processing is employing encoders based on disentanglement representation learning 15
19 20 as illustrated in Figure [[b). Those en- coders extract the speech representations using pre-trained acoustic models, e.g., wav2vec [21 [6],
16
conformer [22 20 and Preformer
23
[5]. Furthermore, they promote representation disentangle-
ment through adversarial training 24 For example, PPSLU [[]] uses a 12-layer transformer-based
Preformer as its encoder.
As a result, disentanglementt-basee encoders still demand considerable computational resources, often exceeding tens of GFLOPs, achieve effective disentanglement [25). They are also memory- intensive, often comprising tens of millions of parameters. Consequently, they are unsuitable for embedded devices with limited memory. Moreover, it takes time-consuming adversarial training to disentangle the encoded representation for each specific SLU task. This aspect limits the flexibility and scalability for emerging SLU tasks. More motivating details will presented in 22.
In this paper, we aim to achieve the real-time, crivacy-rreservigg offloading of speech understanding task on wimpy devices like STM32H7 microcontroller (26) with only 1MB RAM. This goal neces- sitates a novel encoder design that must be both lightweight and effective in filtering out sensitive information, illustrated in Figure Tc).
Our solution We therefore present SILENCE, a SImpLe ENCodEr designed for efficient privacy- preserving SLU offloading. It is based the asymmetric dependency observation: SLU intent extraction (e.g., scenario identification) typically requires only long-term dependency knowledge across the entire utterance, while ASR task (e.g., recognizing individual words or phrases) needs short-term dependency, as confirmed by our experiments in I3.1 Based on it, SILENCE strategically partitions the utterance into several segments, selectively masking out the majority to enhance pri- vacy by obscuring short-term details, without significantly damaging the long-term dependencies. The processed audio waveform is then transmitted to the cloud for SLU intent analysis. Addition- ally. we integrate differential mask generator, inspired by interpretable learning methods [27], to optimize performance by automatically identifying how many and which segments mask.
Results We deploy SILENCE on the STM32H7 microcontroller (26) and assess its performance using the SLURP dataset (28) in both black-box and white-box attack environments. SILENCE achieves 81.2% intent classification accuracy on SLURP, surpassing previous privacy-preserving SLU systems by up to 8.3%. Regarding privacy protection, SILENCE offers comparable security
Note that these encoders are not specifically transformer encoders; rather, they can be implemented using any NNs encode speech signals.

---

to earlier systems. with a word enror rate of up to 81.6% and an entity enror rate of 90.7% under malicious ASR attacks. Even against white-box attacks, where attackers are strongly assumed to have the same encoder structure and weights as SILENCE, plus partial data from malicious clients, SILENCE maintains 67.3% word error rate and 64.3% entity error rate. Additionally, SILENCE proves to be resource-efficient and feasible for wimpy devices, using only 394.9KB of memory and taking just 912.0ms to encode a 4-second speech signal. Integrated with RPI-4B for a fair comparison, SILENCE uses up to 134.1x less memory and operates up to 53.3x faster than prior systems. The accuracy of SILENCE only 7% lower than unprotected SLU systems.
Contribution We have made the following contributions.
Based on the observation of asymmetric dependency between SLU and ASR tasks, we propose SILENCE, simple yet effective encoder system for privacy-preserving SLU of- floading.
We are the first to retrofit interpretable learning methods to automatically configure the masking process for a better balance between privacy and utility in speech understanding tasks.
We evaluate SILENCE on wimpy microcontroller unit and demonstrate its effectiveness under various attack scenarios.
2 Related Work and Background
2.1 Privacy-preserving SLU
Spoken Language Understanding (SLU) is critical component of modern voice-activated systems, responsible for interpreting human speech and translating it into structured, actionable commands. For instance, when user says, "Set meeting for tomorrow 10 AM," " the SLU system might map this to a structured intent such as (scenario: Calendar, action: dependent intend classification is currently the main objective of SLU understanding literature and has a wide range of application scenarios 22 30 31
Create_ent ceatte_entry.... Long-
32
33
28
34
35 36).
Evolution of SLU Systems The evolution of SLU systems has seen shift from traditional two- component systems, comprising ASR and Natural Language Understanding (NLU), to modern end- to-end neural networks 38). These advanced systems bypass the intermediate textual represen- tation and directly map speech signals to their semantic meaning, enhancing efficiency and reducing error propagation. A typical end-to-end SLU model features encoder, often with convolution and attention- based elements, and a decoder, including a transformer decoder and a connectionist tem- poral classification decoder. Many SLU systems incorporate encoders from pre-trained ASR models like HuBERT [39] replacing the original ASR decoder with one tailored for SLU tasks.
Threat Model Our threat model aligns with prior work [15 16] where users (the victims) actively offloads their audio data to the cloud server (the adversary) for intended SLU tasks. Upon receiving the data, the adversary may employ automatic speech recognition to transcribe the audio and identify private entities [17
II [8] Note that the transcriptions are often exceedingly detailed, containing much more information than the users intend to disclose. The goal of this paper is to ensure that the victims can reliably obtain the predefined SLU intent from the adversary, while preserving the adversary from discerning sensitive details private entities in the transcript.
For instance, home pods might capture recordings of confidential daily interactions alongside ex- plicit commands, presenting paradigmatic case for SILENCE, Without SILENCE, over 80% of our private daily conversations could be automatically recognized and stored for unforeseen usage as will be analyzed in $5.1
2.2 Inefficiency of Existing Approaches
-rrccy-preserving methods Crypto-based approaches, such as HE 40 and MPC [1 have been proposed to provide encrypted computation. Unfortunately, they are technically slow and thus im- practical for deployment on resource-constrained audio devices due to the significant increase in computation and communication complexity. For example, MPC-based PUMA 42 takes 5 min-
3

---

utes to complete one token inference, which is far too slow for real-time. Voice conversion is another method protect speech content. Presch [43] integrates voice conversion with GPT-based gener- ated noise protect privacy, but it is far from feasible for deployment on wimpy devices. Traditional peripheral devices, such as ultrasonic microphone jammers (UMJ), are designed to obscure raw speech by inserting non-linearity noise, thereby preventing illegal eavesdropping 44, [8]; however, they also corrupt speech semantics as well. emerging and prevailing strategy is disentangling- based encoders [16, [1 [1]; they aim to create a disentangled and hierarchical representation of the speech signal devoid of sensitive data, But we reveal their performance issue next.
OnDevice
Encoder+ Cloud
AllOffload
Memory (MB)
10a
plad time
30184
pladd time (ms
15:00
3530
1580
60OM
12766
(ms)
1000
907
101
500
101
RPI-4B
RPI-4B
TX2
Figure 2: Cost of disentangling- based encoders 115 for 4-sonond audio inference.
We conduct preliminary experiments to measure the resource consumption of the disentangling- based encoder of pre-trained SLU model Raspberry Pi 4B (RPI-4B) 45 and Jetson TX2 (TX2) [46]. Our key observation is that disentangling- resource-intensive for practical deployment. As illustrated in Figure 2 disentanglement encoder consumes 648.7N88 7MB memory and 12.8s for complete one inference on RPI-4B. Even in the strong TX2 with GPU. the encoder still takes 593.0ms Oms to complete one inference. Considering the network latency, the end-to-end latency of the disentangling- based SLU offloading system only saves 0.7% wall-clock time compared to the OnDevice inference without offloading, with a similar memory footprint over 500M.
privacy-preserving SLU system is too
Disentangling- -baiid encoders is slow and eeemory-intensivee due to the complex coder structure designed to separate sensitive information from the speech signal. Given the limited resource of wimpy devices, it is not practical for common privacy- privacy-preeerving SLU scenarios. To enable practical privacy-preserving SLU, the encoder structure and the inference process need to be simplified.
Implications
SILENCE Design
3.1 System Design and Rationales
We introduce SILENCE to efficiently scrub raw audio for privacy-preseree preserving SLU, depicted in Figure 3 The key idea of SILENCE is simple and novel: it masks out a portion of audio segments before sending them to the cloud for SLU tasks. This design is based
unique observation shown in Figure 4cc): when a portion of audio segments is masked out, the ASR model becomes incapable to recognize the phonemes in the masked frames, while the SLU model can still recognize the intent.
Intest: Cscanario: Calender, action: Create
SILENCE
Mask generator
Cloud
Intent: (scenarão:
Calondar,
action:
Create_ _outtyy
Figure 3: SILENCE overview. Red hard line represents the long-term dependency, while the green dotted line represents the short-term dependency.

---

ASR
SLU
aotaree ning <blank mo
ACC-SL.U (96)
60
WER SEU (80)
40
40
(a) Peaky phoneme is short-dependent
20
[Scenario: Calendar, Action: Setj
[Sconorio: Action: Sed
20
0
50
Mask ratio
(c) Empirical performance under different ratios of masked portion.
Attention seeks intent globally
Figure 4: Foundation of SILENCE: asymmetrical dependency. (a). ASR task is short-term dependent on the peaky phoneme probability. (b). SLU task is long-term dependent knowledge from the whole utterance. (c). Empirical results.
Design rationale Why is SILENCE able protect the sensitive entity privacy while maintaining SLU accuracy? This capability is rooted in the asymmetrical dependency between the ASR and SLU task.
Speech composed many meta phonemes, and the generation of single meta phoneme depends on its adjacent frame [I]] Dependency is defined the length of frame that model's output depends on. Figure(4(a) shows each phoneme is mainly dependent on a few frames, indicating short- term dependency. This phenomenon is referred to "peaky behavior" in the ASR literature [47]. In contrast, an SLU model utilizes an attention-basee decoder (39 to capture the relationship between the entire utterance and the intent, implying that the intent is long-term dependent the whole utterance.
Formally, SILENCE is simple encoder based asymmetrical dependency-basee masking. This simple masking encoder is defined as: â x C Z. where x is the input audio signal, @ represents the element-wise multiplication, & is the masked audio signal and Z is the binary masking vector with the same dimension as æ. Z consists of k uniform portion, with all or is in one portion to mask-out preserve the complete adjacent frames, respectively. This simple encoder forms the basis of SILENCE's efficiency and privacy-rreeevatte capacity, enabling secure offloading of speech understanding tasks wimpy devices.
The configuration challenges: Figure 4ec) demonstrates that the ratio of masked portion plays crucial role in balancing the privacy (WER-ASR) and utility (ACC-SLU). Currently, SILENCE employs a trivial masking mechanism, necessitating clients undertake time-intensive hyper- parameter adjustment about the extent and location of masking. Incorrect masking configurations can result significant loss of global long-term dependency, negatively affecting SLU accuracy, insufficient masking of sensitive information, thus compromising privacy. Therefore, we face critical questions: how many and which portions should be masked?
3.2 Online Configurator for SILENCE
To address these challenges, we derive differential mask generator from the interpretable learn- ing (27) as online configurator for SILENCE. This automatically generate the masking vector Z. The mask generator is trained to identify how many and which portions to mask, optimizing the privacy-utility balance.
Differentiable mask generator The configurator model aims to minimize the discrepancy between masked and original output by generating a mask Σ. Formally, we define the number of unmasked portions as Cg loss:
n
Cold, x) > pa (Z,)
where e the mask generator, 1(-) the indicator function. We minimize Lo for dataset D. ensuring that predictions from masked inputs resemble those from the origin model:
5

---

Edge
Cloud
<ky>
(1a)
Mask Generator
Gate
Masked x
SLU
(1b
Generator
Gate
Mesked x
SLU
(2)
predicted
Generator
Figure 5: SILENCE workflow. (1) Offtine phase: (la) Training mask generator and (1b) adapting cloud SLU model it; (2) Online phase: masked input audio x and insensitive intent label are exposed to the cloud.
Conducting could inference with the masked x. Only
min
Σ z)(
(2)
FED
s.t. D.uulel Sy
Vx E
(3)
where y = f()),, is the tokenized label, D. [vlbb]] is the KI. divergence and the margin E R is a hyperparameter.
Given that Lo is discontinuous and has zero derivative almost everywhere, and the mask generator ø requires discontinuous output activation (like a step function) for binary masks, we utilize sparse relaxation to binary variables [48] 49 instead of the binary mask during training.
Holistic workflow As shown in Figure SILENCE encompasses two phases:
(1) Offfine phase: (1a) First, SILENCE trains a differentiable mask generator. The client selects
mask generator model, potentially a submodule of a pre-trained ASR model, such as HuBERT's CNN feature extractor. small gate model is then integrated with this submodule. The combined model processes the input audio and generates mask. This mask selectively conceals parts of the input, ensuring retention of only vital SLU information while hiding sensitive data. The masked input is then forwarded to either trusted cloud service or local SLU model for obtaining masked output. The mask generator is fine-tuned to minimize the discrepancy between masked output logits and the original intent, defined in Equation (1-3).
(1b) Second, SILENCE adapts the cloud model. Here, the client forwards the masked input and specific SLU intent (e.g., "set alarm") to the cloud-based SLU model. The model undergoes fine- tuning to adapt to the masked inputs. This process includes adjusting the model parameters for accurate recognition and response to SLU commands based on the masked input.
(2) Online phase: In online speech understanding, the client sends the masked input to the cloud SLU model. Using the adapted model, the cloud-based SLU accurately identifies and executes the intended SLU action or response.
Configurator cost analysis Training the differentiable mask generator is affordable for the client. Our experiments indicate that convergence is achieved with approximately 200 audio samples, equivalent to 600 seconds of audio. This process takes up to 30 seconds on an A40 GPU, Adapting the SLU model to each mask generator is one-pass effort. This adaptation is relatively trivial, espe- cially when starting from a fine-tuned SLU model rather than building from scratch. This aspect of the process incurs minimal cost compared to the training of the cloud SLU model. Moreover, these costs can be amortized over a large number of edge users in the long run, making it an economically viable solution.
Remark Note that mask generator not developed for tagging sequences semantic level. Rather, its design focuses on identifying segments that are more relevant to the SLU task. This task is essentially relatively straightforward binary classification problem, which is proven be effective in prior interpretable learning literature [27] 49] and light- enough for real-time inference.

---

Implementation and Methodology
We have fully implemented the SILENCE prototype atop SpeechBrain (50, a Py PyTorch-bseed and unified speech toolkit. As prior work [39, SpeechBrain to train the differential mask gener- ator and simulate the cloud training process. After that, we deploy the trained mask generator into the embedded devices and evaluate the end-to-end performance.
Hardware and environment Offline training is simulated on a server with NVIDIA A40 GPUs. The trained mask generator is deployed into the STM32H7 (26) Raspberry PI 4 (RPI-4B) 145) STM32H7 is resource-constraime microcontroller with IMB RAM. RPI-4B is a popular develop- ment board with 4GB RAM. We embed the approaches not feasible to fit in the STM32H7 into the RPI-4B.
Models We design four types of mask generator structures: (1) Random: random binary vector generator with 50% portion masked; (2) SILENCE-S: learnable mask generator with only one MLP gate; (3) SILENCE-KS -M: learnable mask generator with one HuBERT encoder layer and the gate; (4) SILENCE-La a learnable mask generator with three HuBERT encoder layers and the gate. As for cloud SLU model, we simulate it using the SoTA end-to-end SLU model [39] It replaces the ASR decoder of pre-trained HuBERT with SLU attentional decoder.
Dataset and Metrics We run our experiments on SLURP (28) and FSC [1]] FSC is widely used dataset for spoken language understanding research. SLURP's utterances are complex and closer to daily human speech, We select scenario classification accuracy to measure the SLU understanding performance (ACC-SLU). Following prior work [.5] we choose large-scale English reading corpus LibriSpeech [52] for multi-task protection scenario. In the multi-task protection scenario, not only the subsequent utterance (LibriSpeech) are uploaded to the cloud. WER is used to measure the attack performance. More specifically, we utilize WER-SLU to measure
the SLU command utterance (SLURP/FSC) but also the background
word infor- mation in the uploaded SLU audio itself, and WER-ASR as the WER of recognized accompanying audio, i.c.. LibriSpeech dataset. We also report the private entity recognition error rate (EER) to ensure that the cloud model is not able to recognize the private information in the speech signal. As for latency, we sequentially fed test audios into the local model without any window processing" and recorded the average forward time as the local execution time.
attacker's capacity to recognize
Baselines
We compare SILENCE to the following alternatives: (1) OnDevice means the cloud SLU model is downloaded and run locally on the client device. (2) Al10ffload means the raw audio uploaded to the cloud for SLU inference.
(3) VAE [16] the vanilla variational auto-encoder method that uses adversarial training to disentangle the private information from speech signal. (4) PPSLU [5] the state-of-the-ae disentangling- baaaadd SLU privacy-presers system, which uses 12 transformer layers to separate the SLU information into part of the hidden layer and only sends those hidden layers to the cloud for SLU inference.
passive attack
active inpainting
active predicting
Diffusion
Azure
Whisper
/Unet
whisper
Transcript
Learn
oblanio
To be unmasked
Masked
Masked
Masked
Masked
Masks
Masks
Masks
Masks
Mask generator
Mask generator
Mask generator
Waveform rm
Waveforn
Wave-Form
Kavefform
Nornal users
Malicious users
Normal users
Malicious users
Figure 6: Mask generator and different attack scenarios, including both passive and active attacks.
'The average duration of test SLU snippets is 2.8 seconds, with a maximum of 21.5 seconds, which is shorter than the maximum imput window of speech models (e.g., 30 seconds for Whisper [53]).

---

Attack scenarios. As illustrated in Figrue 6 we use five attacks encompassing both active and pas- sive attacks: (1) Azure represents passive black-box attacker scenario, in which the masked audio is transmitted to Azure [54] for automatic speech recognition. (2) Whisper simulates SoTA cloud- based ASR model. This passive black-box attacker uses the pre-trained W Whisper.mediung model [53] directly downloaded from HuggingFace [55]. (3) Whisper (White-box) constitutes an active white-box attack. Here, we hypothesize that certain users are malicious and disclose the mask generator's structure and weights, along with their own audio data, to the Whisper attack model. Whisper (ooite-box) then utilizes this collected data from malicious users to adapt the pre-trained W hhhhspe..mdddumm inpainting model based on convolutional U-Net structure, commonly used in literature to actively reconstruct missing audio signals [56]] [57]. We utilize the SLURP training and their masked coun- terparts to train the inpainting model from scratch to reconstruct the missing audio. (5) CQT-Diff is neural diffusion model with an invertible Constant-Q Transform leverage pitch-equivariant symmetries (58), allowing it to effectively reconstruct audio without retraining.
model to the specific masking pattern. (4) U-Net is traditional
Sypee--prramtert During the offline phase in Figure 5 we use the Adam optimizer with learning rate of le-5 and batch size of 4. For the inference step, we use the batch size of to simulate the real streaming audio input scenario. The end-to-end cloud SLU latency is measured by invoking Azure APIs following previous work [39]. KL threshold À is set as 0.15 for all mask generators. Attack model is set Whisper without special declaration. We have illustrative example of the generated masks on audios selected randomly from SLURP in supplimentary material.
5 Evaluation
5.1 End-to-end performance
SILENCE achieves comparable accuracy performance and privacy protection capacity to pre- vious encoders. As shown Figure 7 we compare the accuracy of SILENCE with all baselines. OnDevice offloads no signals to the cloud and thus has the best privacy protection (WER=100). observed that SILENCE could achieve up to 81. accuracy, with less than 7% accuracy loss compared to unprotected AllOffload and local OnDevice ce SLU model. Its rationale is that we mainly mask the short-dependent frames that does not significantly affect the SLU performance. We also compare the performance of SILENCE with the SoTA privacy-preservee PPSLU [5). SILENCE achieves 7.2% higher accuracy than PPSLU which tries to apply complex non- linear transformation to the hidden layer to prevent malicious re-construction, but this might also damage part of the SLU information. In terms privacy preservation, our learnable mask generator achieves up to 78.6% WER using SILENCE-L, indicating a privacy-preseevvie capacity par with PPSLU. The same benefits exist in FSC dataset as well. SILENCE demonstrates more than 99% in- tent understanding accuracy, similar to all the baselines, while effeetively defends against sensitive word recognition attacks, achieving more than 80% WER, outperforming all disentanglemenntbban protections. Furthermore, we complete the inference with much lower delays and memory footprint as will be shown in Figure 10
SLU system, i.e.,
PPSLU
Random
Pladn
WER-ARR
70-
Gaboriia
SILENCE M
SILENCE
WER-ASR (96)
WER-SUU (96)
EER (96)
Mi Millpeer
60
SILENCE-L
Azuze
VAE
15
Donsltanden
PPSIU
10
72.5
75.0
775
82.5 ACC-SU
86.0
85.0 87.5
Multi task
Single-
Single task
Figure 7: Performance of different
Figure 8: SILENCE priivacy-preeervingg capacity under dif- ferent attack models.
privacy-preserving SLU approaches.
SILENCE is resistant to different attack models. As illustrated in Figure g SILENCE increases the SLU-WER from 14.7% to 78.6% under the attack model Whisper. As for the online attack model Azure, SILENCE increases the SLU-WER from 14.7% to 81.6%. According to our returned service details, we find that over 50% of the sent audios are tagged as "Rsulll Rew.on.. means audios are recognized as null utterances by the Azure ASR model. Whisper (White-box) is white-box attack model, which means the attacker has the same mask generator structure and
which
8

---

RPH-4B
Mask rato (50
1.00
0.75
Selcctod
10'
253
405
351
STM3IN7
Eapsed trme (nei)
20*
12784
STM31HH
0.50
Memory
107
4950
SILENCE-S SILENCE-M SILENCE-L
1810
0.25
10-2
0.05
10 0.15 0.20 0.25 0.30
Local
PPSU
SILENEE
SIEECT-
50 SLENCCE
Rendem
Ontine
Local
PPSL
SILENCE-4
SEEENCE
SILENCE
Random
Online
Threshold: KL Divergence
Figure 9: Effect of threshold with different mask generators
(a) Memory footprint
(b) End-to latency
Figure 10: Comparison of resource cost in different SLU ap-
proaches. Ours are highlighted in red.
weights as the SILENCE. We still achieve more than 50% SLU-WER under this attack model. This because even Whisper (White-box) is fine-tuned to fill some of the missing frames, it still could not recover the private missing frames. Because masking the short-dependent frames fundamentally destroys the raw audio signal. It is not possible to re-construct the phoneme without knowing any speech information. In the last subfigure, we show the high entity error rate to demonstrate that the private entity is not leaked.
SILENCE can defend against the active inpainting attack as well. As shown in Table I U-Net can rarely reconstruct the masked audio. Even worse, it introduces incorrect noisy signals, degrading the attack success rate. CQT-Diff inpainting can fill the missing waveforms but cannot success- fully reconstruct the content because designed to reconstruct background music, such as piano concertos. SLU audio, which includes human intent and conversation, is difficult to reconstruct.
Text
Azure
Naive Whisper
U-Net
CQT-Diff
Whisper predict (white box)
WER-SLU (%)
14.7
81.6
78.6
82.5
74.3
67.3
WER-ASR (%)
12.3
71.6
681.
71.4
65.9
64.4
Table 1: Potential attack Word Error Rate (WER) under different attack scenarios.
SILENCE scales to better privacy-accuracy trade-off with a larger mask generator. We explore the impact of the threshold 7 of SILENCE under different mask generator structures. As shown in Figure / the threshold "y controls the trade-off between the privacy and utility. When y small, the mask generator is more conservative, leading higher the utility lower the masking portion. As we have discussed in Section 3 lower rate of masking portions leads to higher possibility of privacy entity leakage. When 7 is large, the mask generator more aggressive, enhancing privacy. Another way to achieve more practical privacy-utility balance is using a more complex mask gener- ator structure, e.g., SILENCE-L. It achieves higher utility with the same privacy level compared to SILENCE-S, albeit with less efficiency, as shown in $ 5.2
5.2
System cost
SILENCE protects the private entities efficiently shown in Figure 10 Different from prior en- coders using complex disentanglement model, SILENCE only requires a wight mask generator to scrub the private information. The size of this generator varies according to different mask gen- erator structures. For the smallest mask generator, SILENCE-S. S, it only requires 394.9KB memory footprint, and could successfully embed into the resource-constraing constrained STM32H7 with 2MB RAM. SILENCE is efficient not only in terms of memory footprint but also in latency. SILENCE-S com- pletes the local encoding with only 912.2ms on the resource-constrained STM32H7. For a fair comparison, we embed SILENCE-S into RPI-4B and find that it is 18.1x faster and 134.1x less memory footprint than PPSLU. Even with the strong mask generator SILENCE-L, SILENCE achieves up to 7.5x lower encoding latency and consumes 1.9x less memory compared to OnDevice.

---

Conclusion and Discussion
SILENCE is an efficient and Krivacy-reesevvin end-to-end SLU system based on the asymmetrical dependency between ASR and SLU. SILENCE selectively mask the short-dependent sensitive words while retaining the long-dependent SLU intents. Together with the differentiable mask generator, SILENCE shows superior end-to-end inference speedup and privacy protection under different attack scenarios.
Limitations: While for the first time, SILENCE provides a feasible privacy-preserving solution for resource-constrained audio devices, it introduces huge design space for mask generator structures. The mask generator is akin to a lock; a genius lock design can protect privacy in the smallest of spaces, but poor lock design can be bulky and easily broken. In this work, we simply inherit the SLU model structure and instantiate three sub-models from it to demonstrate better efficiency than previous encoders. Researchers can explore other structures for better sareey-accurayy-ffiiieie trade-off. We will open-source all the code and checkpoints to facilitate further research in this direction.
Some other potential limitations about lossy privacy-preserving capacity, the need for fine-tuning the cloud SLU model, the scope of defended threat model and the extension offline scenarios thoroughly discussed below for further clarification.
Is current crrivcc--peseeve enough, as previous encoders have strived to reach that level [T5] [6]. And some SLU transcripts contain the intent word, so the successfully inferred word might be non-private intent word. For instance, in one test audio transcript, "I want some jazz music to play". the intent is 'scenario': "play', 'action': 'music'. The interpretation of the malicious cloud ASR, "all subjects were used to play", is acceptable since the predicted phrase "to play" contains no private information. This scenario is typical for most audios; we managed to preserve 90% of the private entities in Figure This achievement matches the SoTA in privacy-preserving capacity, with up to 30x lower latency and 100x memory reduction.
capacity enough? The quantitative WER 80% is considered secure
Why and how to fine-tune the cloud SLU Model? Initially, the cloud SLU is a generic pre-trained speech model lacking the capability to accurately understand personalized user intent. It is crucial to fine-tune the cloud SLU for better personalized intent understanding") Secondly, while short- dependent masking does not eliminate intent information, it does impact specific details within the attention map, as depicted in Figure 4b). Fine-tuning the cloud SLU model helps mitigate this impact and enhances the understanding of the user's intent. Currently, cloud service providers have already offered APIs that allow users to fine-tune their personalized cloud speech model (60).
Could private semantic detection attack be prevented? We clarified that detecting short- dependent key phrases or specific commands is not the focus of this work. For example, eaves- dropping specific financial words and political framing out-of-scope However, we can offer defense capabilities against them. The mask generator, controlled by the user, is trained to scrub utterances unrelated to the public intent. Private entities not predefined by the user are almost never included in the masked audio. Therefore, even an attacker possesses a well-defined semantic and the mask generator, training the detection threat model is challenging because the synthetic masked audio lacks clear representations of the private semantic.
Extesion to offline scenarios: Offline conditions occur periodically for resource-come conttrrc--ont de- vices. SILENCE can be easily integrated into an orchestration of small on-device SLU models and robust cloud models. This orchestration has been officially adopted by many off-the-shed products, such as Apple Intelligence in iOS 18. Our system remains indispensable in such circumstances be- cause small on-device SLU models may not generate satisfactory intent understanding due to their restricted model size. Even when on-device SLU models produce correct intent understanding, they cannot always operate due to limited device energy. As result, online procedures are still the main components of current SLU solutions. The on-device functionality can be used as an alternative in offline conditions. With our system, the cloud- based SLU component is both privacy-preserving and efficient.
$ "Nte that general speech model is sufficient for training the local mask generator in Sigure5ssep (la), as the focus is not generating precise intent but rather on obtaining coarse coarse-grained distribution of numerical logits to facilitate mask generator training.
10

---

Acknowledgments and Disclosure of Funding
The authors thank the anonymous reviewers for their insightful feedbacks. This work was sup- ported by the National Key Research and Development Program of China under Grant No. 2021ZD01 13001. the Youth Program of the National Natural Science Foundation of China under Grant No. 62102045, the National Science Fund for Distinguished Young Scholars under Grant No. 62425203, and the CCF-Sangfor "Yuanwang" Research Fund. Dongqi Cai was supported by the China Scholarship Council under Grant No. 202406470054.
References
[1]
-ww----an--------eeeeee-eee----- and-speak
[2] Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: framework for self-supervised learning of speech representations. Advances in neural information processing systems, 11112244-1246// 2020.
[3]
[4] Suranga Seneviratne, Yining Hu, Tham Nguyen, Guohao Lan, Sara Khalifa, Kanchana Thi- lakarathna, Mahbub Hassan, and Aruma Seneviratne. A survey of wearable devices and chal- lenges. IEEE Communications Surveys & Tutorials, 19(4):253322200 2017.
[5] Leigh Clark, Philip Doyle, Diego Garaialde, Emer Gilmartin, Stephan Schlögl, Jens Edlund, Matthew Aylett, João Cabral, Cosmin Munteanu, Justin Edwards, et al. The state of speech in hei: Trends, themes and challenges. Interacting with computers, 3144):349-371, 2011.
2019.
[6]
[7] Nombulelo CC Noruwana, Pius Adewale Owolawi, and Temitope Mapayi. Interactive iot- based speech-controlled home automation system. In 2020 2nd International Multidisciplinary Information Technology and Engineering Conference (IMITEC), pages 1-8. IEEE, 2020.
[8] Joel M Raja, Carol Elsakr, Sherif Roman, Brandon Cave, Issa Pour Bour-Ghaz, Amit Nanda, Miguel Maturana, and Rami N Khouzam. Apple watch, wearables, and heart rhythm: where do stand? Annals of translational medicine, 7(17), 2019.
[9] Lloyd E Emokpae, Roland N Emokpae, Wassila Lalouani, and Mohamed Younis. Smart mul- timodal telehealth-iot system for covid-19 patients. IEEE Pervasive Computing, 20(2):73-80, 2021.
[10] Naveen Kumar and Seul Chan Lee. Human- machine interface in smart factory: A systematic literature review. Technological Forecasting and Social Change, 174:121284, 2022.
[11] Ke Sun, Chen Chen, and Xinyu Zhang. alexx, stop spying on me!" speech privacy protection against voice assistants. In Proceedings of the 18th conference embedded networked sensor systems, pages 298-311, 2020.
for forizzeenn-----ttttiniii to--toi-----i-
conversatiion.htanl
[13] Trung Dang, Om Thakkar, Swaroop Ramaswamy, Rajiv Mathews, Peter Chin, and Françoise Beaufays. A method to reveal speaker identity in distributed asr training, and how to counter it. In ICASSP 2022-2022 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 4338-4342. IEEE, 2022.
[14] Jianwei Qian, Haohua Du, Jiahui Hou, Linlin Chen, Tacho Jung, and Xiang Yang Li. Hide- behind: Enjoy voice input with voiceprint unclonability and anonymity. In Proceedings of the 16th ACM Conference on Embedded Networked Sensor Systems, pages 82-94, 2018.
[15] Yinggui Wang, Wei Huang, and Le Yang. Privacy-preserving end-to-end spoken language un- derstanding. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, pages 5224-5232, 2023.
11

---

[16] Ranya Aloufi, Hamed Haddadi, and David Boyle, entangled representations. Proceedings of the 2020 ACM SIGSAC Conference on Cloud Computing Security Workshop, pages 1-14, 2020.
voice analysis via dis-
[17] Peng Cheng and Utz Roedig. Personal voice assistant security and privacy-a survey. Proceedings of the IEEE, 10(4))47-5507, 2022.
[18] Yike Chen, Ming Gao, Yimin Li, Lingfeng Zhang, Li Lu, Feng Lin, Jinsong Han, and Kui Ren. Big brother is listening: An evaluation framework on ultrasonic microphone jammers. In IEEE INFOCOM 2022-IEEE Conference on Computer Communications, pages 1119-1128. IEEE, 2022.
[19] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 2013.
[20] Cal Peyser, Ronny Huang Andrew Rosenberg Tara N Sainath, Michael Picheny, and Kyunghyun Cho. Towards disentangled speech representations. arXiv preprint arXiv:2208.1312 13191, 2022.
[21] Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli. wav2vec: Unsuper- vised pre-training for speech recognition. arXiv preprint SEEi::194400585 2019.
[22] Anmol Gulati, James Qin, Chung-Chemn Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, et al. Conformer: Convolution-augmentt transformer for speech recognition. arXiv preprint arXiv:2005.08 100, 2020.
[23] Keqi Deng, Songjun Cao, Yike Zhang, and Long Ma. Improving hybrid cic/attention end-to- end speech recognition with pretrained acoustic and language models. In 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 76 82. IEEE, 2021.
[24] Ian Goodfellow, Jean Pouget-Abadic Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
[25] Siddhant Arora, Siddharth Dalmia, Xuankai Chang, Brian Yan, Alan Black, and Shinji Watan- abe. Two-pass low latency end-to-end spoken language understanding. arXiv preprint arXiv:2207.06670, 2022.
series.html
[27] Nicola De Cao, Michael Schlichtkrull, Wilker Aziz, and Ivan Titov. How do decisions emerge across layers in neural models? interpretation with differentiable masking. arXiv preprint arXiv:2004.14992 2020.
(28) Emanuele Bastianelli, Andrea Vanzo, Pawel Swietojanski, and Verena Rieser. Slurp: A spoken language understanding resource package. arXiv preprint arXiv:2011.13205 2020.
[29] Rongxiang Wang and Felix Xiaozhu Lin. Turbocharge speech understanding with pilot infer- ence, 2024.
[30] Soham Deshmukh, Benjamin Elizalde, Rita Singh, and Huaming Wang. Pengi: An audio lan- guage model for audio tasks. Advances in Neural Information Processing Systems, 18090- 18108, 2023.
[31] Jixuan Wang, Martin Radfar, Kai Wei, and Clement Chung. End-to-end spoken language un- derstanding using joint ete loss and self-supervised, pretrained acoustic encoders. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pages 1-5. IEEE, 2023.
[32] Bhuvan Agrawal, Markus Müller, Samridhi Choudhary, Martin Radfar. Athanasios Mouchtaris, Ross McGowan, Nathan Susanj, and Siegfried Kunzmann. Tie your embeddings down: Cross-modal latent spaces for end-to-end spoken language understanding. In ICASSP 2022-2022 IEFE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7157-7161. IEEE, 2022.
12

---

[33] Libo Qin, Tianbao Xie, Wanxiang Che, and Ting Liu. A survey on spoken language under- standing: Recent advances and new frontiers. arXiv preprint arXiv:2103.03095, 2021.
[34] Shinji Watanabe, Takaaki Hori, Suyoun Kim, John R Hershey, and Tomoki Hayashi. Hybrid ctc/attention architecture for end-to-end speech recognition. IEEE Journal of Selected Topics in Signal Processing, (1(8):1240) 240--123, 2017.
[35] Jan K Chorowski, Dzmitry Bahdanau, Dmitriy Serdyuk, Kyunghyun Cho, and Yoshua Bengio. Attention-bars models for speech recognition. Advances in neural information processing systems, 28, 2015.
[36] Renato De Mori. Spoken language understanding: A survey. 2007 IEEE Workshop on Automatic Speech Recognition & Understanding (ASRU), pages 365-376. IEEE, 2007.
[37] Subendhu Rongali, Beiye Liu, Liwei Cai, Konstantine Arkoudas, Chengwei Su, and Wael Hamza. Exploring transfer learning for end-to-end spoken language understanding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 13754-
13761. 2021.
[38] Parisa Haghani, Arun Narayanan, Michiel Bacchiani, Galen Chuang, Neeraj Gaur, Pedro Moreno, Rohit Prabhavalkar, Zhongdi Qu, and Austin Waters. From audio to semantics: Approaches to end-to-end spoken language understanding. In 2018 IEEE Spoken Language Technology Workshop (SLT), pages 720-726. IEEE, 2018.
[39] Yingzhi Wang, Abdelmoumene Boumadane, and Abdelwahab Heba. A fine-tuned wav2vec 2.0/hubert benchmark for speech emotion recognition, speaker verification and spoken lan- guage understanding. arXiv preprint arXiv:2111.02755 2021.
[40] Chengliang Zhang, Suyi Li, Junzhe Xia, Wei Wang. Feng Yan, and Yang Liu. Batcherypt: Efficient homomorphic encryption for cross-silo federated learning. In 2020 USENIX annual technical conference (USENIX ATC 20), pages 493-506, 2020.
[41] Oded Goldreich. Secure multi-party computation. 78(110):1-108, 1998.
Manuscript. Preliminary version,
[42] Ye Dong, Wen-jie Lu, Yancheng Zheng, Haoqi Wu, Derun Zhao, Jin Tan, Zhicong Huang, Cheng Hong, Tao Wei, and Wenguang Cheng. Puma: Secure inference of Ilama-7b in five minutes. arXiv preprint arXiv:2307.12533 2023.
[43] Shimaa Ahmed, Amrita Roy Chowdhury, Kassem Fawaz, and Parmesh Ramanathan. Presch: A system for preeeeein speech transcription. arXiv preprint arXiv:1909.04199 v2, 2019.
[44] Ming Gao, Yike Chen, Yajie Liu, Jie Xiong, Jinsong Han, and Kui Ren. Cancelling Speech Signals for Speech Privacy Protection against Microphone Eavesdropping. Association for Computing Machinery, New York, NY, USA, 2023.
[46]
[47] Albert Zeyer, Ralf Schlüter, and Hermann Ney. Why does etc result in peaky behavior? arXiv preprint arXiv:2105. 14849, 2021.
[48] Christos Louizos, Max Welling, and Diederik P Kingma. Learning sparse neural networks through |_0 regularization. arXiv preprint arXiv:1712.01312, 2017.
[49] Jasmijn Bastings, Wilker Aziz, and Ivan Titov. Interpretable neural predictions with differen- tiable binary variables. arXiv preprint arXiv: 190.00866, 2019.
[50] Mirco Ravanelli, Titouan Parcollet, Peter Plantinga, Aku Rouhe, Samuele Cornell, Loren Lugosch, Cem Subakan, Nauman Dawalatabad, Abdelwahab Heba, Jianyuan Zhong, et al. Speechbrain: A general-purpose speech toolkit. arXiv preprint arXiv:2 106, 04624, 2021.
13

---

[51] Loren Lugosch, Mirco Ravanelli, Patrick Ignoto, Vikrant Singh Tomar, and Yoshua Bengio. Speech model pre-training for end-to-end spoken language understanding. arXiv preprint arXiv:1904.03670 2019.
[52] Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an asr corpus based on public domain audio books. 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 5206-5210. IEEE, 2015.
[53] Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In International Conference Machine Learning, pages 28492-28518. PMLR, 2023.
[54] Microsoft. Azure ast. http::///aaeeiiiicoooooooooer/rd opeech-to-text/
[55] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et Huggingface's trans- formers:
State-of-the-art natural language processing, arXiv preprint arXiv: 1910.3771,.0777, 2019.
[56] Xiang Hao, Xiangdong Su, Shixue Wen, Zhiyu Wang, Yiqian Pan, Feilong Bao, and Wei Chen. Masking and inpainting: A two-stage speech enhancement approach for low snr and non-stationary noise. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages (959-6963.. IEEE, 2020.
[57] Mikolaj Kegler, Pierre Beckmann, and Milos Cernak. Deep speech inpainting of time- frequency masks. arXiv preprint arXiv:1910.0900 1910.09058. 2019.
[58] Eloi Moliner, Jaakko Lehtinen, and Vesa Välimäki. Solving audio inverse problems with diffusion model. In ICASSP 2023-2023 IEEE International Conference on Acousties, Speech and Signal Processing (ICASSP), pages 1-5. IEEE, 2023.
[59] Rongxiang Wang and Felix Lin. Efficient deep speech understanding at the edge, arXiv preprint arXiv:2311.17065, 2023.
2ith-azure-custom-ppeeeel
14

---

NeurIPS Paper Checklist
(1) Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Our contribution is outlined as a seperated paragraph in I
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
(2) Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We thoroughly discuss the limitations of our work in $6
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The au- thors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the ap- proach. For example, a facial recognition algorithm may perform poorly when image resolution is low images taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
(3) Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theo- rems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
(4) Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide detailed instructions on how to reproduce the main experimental results in $4 We will open-source the code and data upon acceptance.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or the contribution is a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data also be provided via
often one good way accomplish this, but reproducibility detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of model checkpoint, other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or a way to re- produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case au- thors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models. it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
55) Open access to data and code
16

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification:
We will open-source the code and data upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
ptaipa.c//ubbiicguuueess Soo.edeeeeeooeSumbimceee onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
(6) Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide detailed instructions on how to reproduce the main experimental results in $4
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
(7) Experiment Statistical Significance
Question:
Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Answer: [No]
Justification: Error bars are not reported because of the time limit. We will attempt to add them in the camera-ready version.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
17

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors). should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK report I-sigma error bars, but one should state The authors should prefer- ably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
(8) Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide detailed hardware information and the intended runtime in
$32 and $5
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental rums well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't t make it into the paper).
(9) Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httph::tpp://t
Answer: [Yes]
Justification: We have reviewed the NeurlPS Code of Ethics and believe that our research conforms to
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No. they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
(10) Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:
[Ye]]
Justification: We have discussed and provided real-world examples of both positive and negative societal impacts in I and $2
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
18

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
(11) Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper is intended for privacy protection and does involve high-risk data or models.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards allow for controlled use of the model, for example by quiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
(12) Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We have properly cited the original code, data and models in $4
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
19

---

If assets are released, the license, copyright information, and terms of use in the pack- age should be provided. For popular datasets, paperswithcoee com datasets has curated licenses for some datasets. Their licensing guide can help determine the li- cense of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
(13) New Assets
Question: Are new assets introduced in the paper well documented and the documenta- tion provided alongside the assets?
Answer: [No]
Justification:
We will provide detailed documentation for the new assets upon acceptance.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
(14) Crowdsoureing and Research with Human Subjects
For crowdsourcing experiments and research with human subjects, does the pa- per include the full text instructions given to participants and sereenshots, applicable, well as details about compensation (if any)?
Question:
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, cura- tion, or other labor should be paid at least the minimum wage in the country of the data collector.
(15) Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper deseribe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equiva- lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
20

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
21


Reviews:
[{"review_id": "k9E05kLl3F", "reviewer": "Reviewer_rbWb", "summary": "This paper proposed a private speech processing system that selectively obscures short-term details to reduce privacy leakage in cloud-based Spoken Language Understanding (SLU). A differential mask generator is learned to automatically mask out portions of audio signals along with online cloud inference with the generated masks. Empirical results show that the proposed system offers comparable speech understanding performance and privacy protection capacity with high memory efficiency.", "questions": "What types of devices can be defined as 'wimpy' devices?", "limitations": "N/A", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "\\+ Tackled a practical and crucial topic of cloud-based SLU privacy protection.\n\n\\+ detailed system illustration and rationale interpretation.\n\n\\+ Comprehensive experiment details.", "weaknesses": "Could provide more background info for attack scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments and questions. Here are the answers to the points:\n\n> *W1. Could provide more background info for attack scenarios.*\n\nAttacks may involve passive and active adversaries attempting to automatically recognize private entities in uploaded command audio files.\nPassive adversaries use off-the-shelf automatic speech recognition (ASR) models to conduct recognition on the masked audio.\nActive adversaries may collect data from malicious users to build reverse or predictive models that can either reconstruct the raw audio waveform or directly reconstruct the correct final transcript.\n\n**We have provided an illustration of several attack scenarios in Figure 1 of the global rebuttal PDF. \nPlease refer to it for a more detailed description.**\n\n> *Q1. What types of devices can be defined as 'wimpy' devices?*\n\nWimpy devices refer to devices that either lack the power to run robust transformer-based spoken language understanding (SLU) models, such as the STM32H7 microcontroller, or those IoT devices that have the capability to run these models but have limited power, such as the Raspberry Pi 4. \nOffloading computation to the cloud can help these devices save energy.\nAs suggested by Reviewer BMwq, we will change the term to \"resource-constrained conditions\" to improve clarity.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gsampTL0S4", "reviewer": "Reviewer_BMwq", "summary": "This paper presents a lightweight speech intent understanding paradigm for wimpy devices, with heavy concern on the privacy. It targets the recently-popular disentanglement-based approaches for speech processing. It reached a decent balance between efficiency and privacy preservation, and make the SLU system work on wimpy devices.", "questions": "1. Compared to conventional SLU systems, do you think end-to-end systems preserve the nature of enhanced privacy? Do you think your algorithm will work for both the end-to-end system and the conventional modularized system?\n2. The reviewer has problem with the definition of wimpy devices, since raspberry PI shall not be considered as \"wimpy\". Chips such as ARM cortex V7 has much lower memory and a lot of start-up are still using it for their offline solutions. The reviewer thinks \"resource-constrained conditions\" might be a better term. Of if there is official or your clear definition of \"wimpy\" devices with detailed stats, the reviewer is happy to keep head down and learn.\n3. The reviewer are interested into the proposed framework with conventional methods.", "limitations": "The reviewer does not see any limitation or ethical concern about the paper.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 4, "strengths": "1. The paper addresses real-time problem with its own novelty, which is quite rare for recent papers who mostly prefer large-scale solution with massive and massive data. The novelty here is thus quite high.\n2. The paper has a lot of measurements and concerns about hardware-wise perspective, which echoes (1).", "weaknesses": "Since this is more like a engineer-oriented work, the reviewer does not have strong problem or weakness from technical point of view.\n\nBut the reviewer does have a big problem on the architecture of SILENCE. Seems like from Figure 5, it still relies on cloud technology to maintain part of the framework. However, the wimpy devices sometimes (if not mostly) will meet offline conditions. In such case, the paper does not have an intention of making the system work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments, and we greatly appreciate your high praise regarding the novelty of our work. \nWe hope that our rebuttal response will further enhance the soundness of our research. \nBelow are the answers to your concern and questions:\n\n> *W1. The paper does not have an intention of making the system work under offline conditions.*\n\nWe completely agree that offline conditions occur periodically for resource-constrained devices. \nHowever, we want to emphasize that our system can be easily integrated into an orchestration of small on-device SLU models and robust cloud models. \nThis orchestration has been officially adopted by many off-the-shelf products, such as Apple Intelligence in iOS 18.\n\nOur system remains indispensable in such circumstances because small on-device SLU models may not generate satisfactory intent understanding due to their restricted model size. \nEven when on-device SLU models produce correct intent understanding, they cannot always operate due to limited device energy. \nAs a result, online procedures are still the main components of current SLU solutions. \nThe on-device functionality can be used as an alternative in offline conditions. \nWith our system, the cloud-based SLU component is both privacy-preserving and efficient.\n\n> *Q1.1. Compared to conventional SLU systems, do you think end-to-end systems preserve the nature of enhanced privacy?*\n\nWe think the end-to-end system preserves enhanced privacy during both training and inference. \nDuring training, the end-to-end system does not require raw transcripts to improve SLU performance, as illustrated in Figure 5 of the paper. \nDuring inference, skipping the generation of intermediate transcripts can also enhance privacy and directly improve the final SLU performance.\n\n> *Q1.2. Do you think your algorithm will work for both the end-to-end system and the conventional modularized system?*\n\nYes, our algorithm can correctly detect SLU intent in both the end-to-end system and the conventional modularized system. \nThe conventional modularized system can recognize intent with nearly 90\\% accuracy because it uses extra intermediate correct transcripts to enhance the first ASR module.\nAdded implementation and experiments results are detailed in the **Q5**.\n\n> *Q2. The reviewer has problem with the definition of wimpy devices, since raspberry PI shall not be considered as \"wimpy\". Chips such as ARM cortex V7 has much lower memory and a lot of start-up are still using it for their offline solutions. The reviewer thinks \"resource-constrained conditions\" might be a better term. Of if there is official or your clear definition of \"wimpy\" devices with detailed stats, the reviewer is happy to keep head down and learn.*\n\nWe agree that the term \"resource-constrained conditions\" is more explicit to a wider range of readers. \nOur initial use of \"wimpy\" was intended to include the following types of devices:\n- Devices that do not have enough power to run the model, such as the STM32H7 microcontroller. \n- Devices that have enough power to run the model but can save energy by offloading computations to the cloud, e.g., Raspberry Pi.  Reasons to include those devices have been discussed in **W1**.\n\nThank you again for helping us select a clearer term.\n\n> *Q3. The reviewer are interested into the proposed framework with conventional methods.*\n\nWe applied our algorithm to conventional modularized SLU models. \nThe experimental results demonstrate that when both the ASR and NLU modules are fine-tuned as required, the conventional modularized SLU model can recognize intent correctly when fed with masked audio. \nThe detailed results are summarized in the table below:\n\n|                   | Plaintext |   VAE   |  PPSLU  | NLU only (Ours) | Decoupled SLU (Ours) | E2E SLU (Ours) |\n|:-----------------:|:---------:|:-------:|:-------:|:------------------:|:-----------------------:|:-----------------:|\n| **SLU-ACC (%)**       |   87.2    |  72.5   |  74.5   |        12.6        |          89.1           |        81.1       |\n\n*Table: System performance on conventional modularized SLU. Plaintext equals to AllOffloaded or OnDevice.*\n\nHere, the ASR model of the modularized SLU is Whisper.medium.en, and the NLU model is with two LSTM layers. \nFor the column labeled \"NLU only,\" the ASR module uses pre-trained weights downloaded directly from Hugging Face without any fine-tuning. \nDuring training, the real transcript is used as input, and audio intent is used as a label to reduce the loss. \nDuring inference, the masked audio is processed through the pre-trained ASR and fine-tuned NLU model sequentially to obtain the final intents. \nHowever, it cannot achieve accurate intent understanding because the intermediate transcripts are corrupted.\n\nFor the column labeled \"Decoupled SLU,\" the ASR module is also fine-tuned to better match the masked audio and real transcript as the first step. \nAs a result, the final intent recognition is hugely improved because it obtains real transcripts from users and preserves the intend information beforehand.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JN3ea4H46R", "reviewer": "Reviewer_ubkn", "summary": "The paper presents SILENCE, a method designed to address privacy concerns in cloud-based speech services by selectively obscuring short-term dependencies in speech signals. This technique preserves speech understanding functionality while protecting sensitive information. Implemented on the STM32H7 microcontroller, SILENCE significantly enhances speed and memory efficiency compared to existing solutions, effectively protecting privacy against various attack scenarios. Key contributions of the paper include an innovative encoder design based on asymmetric dependencies, the integration of automated masking configuration, and the demonstration of SILENCE's practical feasibility on low-resource devices.", "questions": "- How does SILENCE perform against adaptive adversaries using advanced reverse-engineering techniques?\n- How does mask granularity affect both privacy protection and SLU performance at various levels of speech granularity?\n- How does SILENCE handle SLU tasks that require short-term dependencies?\n- How does SILENCE perform on additional datasets to ensure its generalizability and robustness across different speech scenarios?", "limitations": "The authors adequately addressed the limitations and potential negative societal impact of their work.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Innovative use of differential mask generators for privacy-preserving speech understanding on constrained devices.\n- Thorough experimental validation demonstrating significant performance and efficiency gains.\n- Clear explanations of the key concepts and methodology, though some technical details could be more detailed.", "weaknesses": "- While the paper evaluates the system under black-box and white-box attacks, it does not thoroughly address the robustness of the masking approach against adaptive adversaries who might use advanced techniques to reverse-engineer the mask patterns.\n\n- Masking short-term dependent frames to protect privacy could lead to losing crucial phonetic information necessary for accurate SLU. There is a lack of detailed analysis on how the granularity of the mask affects both privacy and SLU performance at different levels of speech granularity.\n\n- The method relies heavily on the assumption that SLU tasks are primarily long-term dependent while ASR tasks are short-term dependent. This binary distinction might not hold for all SLU tasks, especially those involving contextually rich and intricate utterances.\n\n- The experiments are primarily conducted on a single dataset. Evaluating the method on a broader range of datasets would better demonstrate its generalizability and robustness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments and questions. Here are the answers to the questions:\n\n> *Q1. How does SILENCE perform against adaptive adversaries using advanced reverse-engineering techniques?*\n\nOur system can still preserve content privacy under advanced reconstruction attacks, achieving over 64\\% recognized word error rates.\n\nWe have implemented two more active reconstruction adversaries and demonstrated our efficiency in defending against them. \n- `U-Net` is a traditional inpainting model based on convolutional U-Net structure, commonly used in literature to reconstruct missing audio signals [1,2]. \nWe utilize the SLURP training set and their masked counterparts to train the inpainting model from scratch to reconstruct the missing audio. \n- `CQT-Diff` is a neural diffusion model with an invertible Constant-Q Transform (CQT) to leverage pitch-equivariant symmetries [3], allowing it to effectively reconstruct audio without retraining. \n\nThe reconstructed audio is sent to Whisper for automatic recognition. The visualizations of reconstructed waveforms have also been included in the global PDF for your reference.\n\nThe updated evaluation results under attacks are summarized in the table below. \n|                   | **PlainText** | **Azure** | **Whisper** | **U-Net** | **CQTdiff** | **Whisper (white box)** |\n|:-----------------:|:-------------:|:---------:|:-----------:|:---------:|:-----------:|:-----------------------:|\n| **WER-SLU (%)**   |      14.7     |    81.6   |     78.6    |    82.5   |     74.3    |           67.3          |\n| **WER-ASR (%)**   |      12.3     |    71.6   |     68.1    |    71.4   |     65.9    |           64.4          |\n\n*Table: Potential attack Word Error Rate (WER) of different attack scenarios. PlainText means AllOffloaded without any privacy protection.*\n\nAs shown in the table, `U-Net` can rarely reconstruct the masked audio. \nEven worse, it introduces incorrect noisy signals, degrading the attack success rate. \n`CQT-Diff` inpainting can fill the missing waveforms but cannot successfully reconstruct the content because it is designed to reconstruct background music, such as piano concertos. \nSLU audio, which includes human intent and conversation, is difficult to reconstruct. \n\n> *Q2. How does mask granularity affect both privacy protection and SLU performance at various levels of speech granularity?*\n\nWe included two more fine-grained speech understanding tasks: action and the combined intent (scenario_action) recognition. \nFor your reference, there are 18 different scenarios and 46 defined actions, resulting in 828 possible combinations for intend.\n\n|                   | AllOffloaded | VAE  | PPSLU | OnDevice | Ours |\n|:------------:|:------------:|:----:|:-----:|:--------:|:----:|\n| ACC-Scenario (\\%) | 88.2         | 72.8 | 73.9  | 88.2     | 80.2 |\n| ACC-Action (%)    | 77.1         | /    | /     | 77.1     | 76.4 |\n| ACC-Intent (%)    | 83.3         | /    | /     | 83.3     | 76.8 |\n| WER-SLU (%)       | 14.7         | /    | /     | 100      | 68.6 |\n| WER-ASR (\\%)      | 12.3         | 69.3 | 75.3  | 100      | 68.1 |\n\n*Table: Comparison between Privacy-preservation and SLU performance at different speech granularities. ‘/’ means not supported. Local leaks no words as nothing is uploaded.*\n\nIt can be seen that our method can recognize speech intent at different granularities. \nFor example, we can correctly recognize 76.8\\% of the combined intent.\n\nIn comparison, disentanglement-based methods need to re-entangle representations for different semantic granularities. \nThus, the classifier used for scenario classification cannot be applied to other intents, and these methods are not designed to preserve the sensitive information within command audios. \nThis emphasises a significant advantage of our approach, as it does not require retraining the model for different intent granularities.\n\n> *Q3. How does SILENCE handle SLU tasks that require short-term dependencies?*\n\nWe admit that our work primarily focuses on long-term dependent SLU classification tasks. \nIn most of our experiments, we treat entities requiring short-term dependencies as sensitive words to preserve.\n\nHowever, we want to emphasize that long-term dependent intent recognition is a major objective of SLU understanding literature [4] and has a wide range of applications. \nFor example, it can be used for command recognition, sensor activation monitoring, and more. \nThese applications are particularly relevant for resource-constrained devices.\n\n> *Q4. How does SILENCE perform on additional datasets to ensure its generalizability and robustness across different speech scenarios?*\n\nWe conducted further experiments on the Fluent Speech Commands (FSC) dataset [5], another widely used dataset for spoken language understanding research. \nThe FSC dataset includes 97 speakers and 30,043 relevant utterances. \nWe split the data, using 20\\% for testing and the remaining 80\\% for training. \nThe results are shown below.\n\n|               | AllOffloaded | VAE  | PPSLU | Local | Random | SILENCE |\n|:-------------:|:------------:|:----:|:-----:|:-----:|:------:|:-------:|\n| ACC-SLU (\\%)  | 99.7         | 98.3 | 99.2  | 99.7  | 86.4   | 99.1    |\n| WER\\-ASR (\\%) | 1.2          | 65.5 | 78.5  | 100   | 76.6   | 81.4    |\n\n*Table: Evaluation of privacy preservation and SLU performance on FSC dataset.*\n\nOur system demonstrates accurate intent understanding (more than 99\\%, similar to all the baselines) and effectively defends against sensitive word recognition attacks (achieving more than 80\\% WER, outperforming all disentanglement-based protections).\nAdditionally, our method significantly outperforms existing baselines in terms of computational efficiency and memory usage, allowing for a wider range of deployment scenarios.\n\nReferences are listed in the global rebuttal.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "f3leTu8Fd7", "reviewer": "Reviewer_YHpd", "summary": "This paper investigates the perturbation of audio signals to impair the accuracy of automatic speech recognition (ASR) models, thus enhancing privacy protection, while ensuring that spoken language understanding (SLU) models maintain high accuracy for interpreting user intentions. The key insight is that ASR models depend on local utterances, whereas SLU models utilize global features. The authors propose masking certain utterances to break local dependencies and explore the development of a learnable mask generator.", "questions": "1. Could the authors elaborate on the potential ASR accuracy when facing an active adversary that trains a model to reconstruct raw signals from the masked signals?\n2. Is it possible to visualize some masks generated by the trained mask generator to verify that they effectively disrupt local utterances, aligning with the main motivation of the study?", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The proposed method outperforms existing baselines significantly in terms of computational efficiency and memory usage, which are based on disentangled encoders. \n2. The evaluation includes both black-box adversaries, who only have access to the masked signal, and white-box adversaries, who also have access to the encoder. \n3. The paper is well-organized, making it an easy read.", "weaknesses": "1. My primary concern is the paper's limitation to passive adversaries. An active adversary might develop models specifically to reconstruct the raw signal from the masked signal. Such a threat is not investigated in this paper.\n2. While the disruption of local dependencies does not affect the performance of user intent classification, this approach may not apply to all scenarios. Certain tasks, such as detecting key phrases or specific commands such as “set my clock at 10 AM” rely on local utterances.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for comments and questions. Here are the answers to them:\n\n> *W1. Concern about limitation to passive adversaries.*\n\nWe clarify it and add evaluation of defending against reconstruction adversaries in **Q1**.\n\n> *W2.  This approach may not apply to all scenarios*\n\nWe admit that *detecting key phrases or specific commands* is not the focus of this work. \nIn most of the experiments, we treat these as sensitive words to preserve.\n\nHowever, we want to emphasize that long-term dependent intent classification remains a major objective of SLU understanding literature [4] and has a wide range of applications. \nFor example, it can be used for command recognition, sensor activation monitoring, and more. \nThese applications are particularly relevant for resource-constrained devices.\n\nWe have added this clarification into the updated manuscript.\n\n> *Q1. Could the authors elaborate on the potential ASR accuracy when facing an active adversary that trains a model to reconstruct raw signals from the masked signals?*\n\nCertainly, we conducted further experiments and offer additional clarification to better elaborate on the system's performance when facing active adversaries. We also have Figure 1 in the uploaded pdf to better illustrate the different attack scenarios.\n\nFirst of all, we want to clarify a possible misunderstanding: the white-box adversary, `Whisper (White-box)`, not only has access to the encoder but also collects enormous data from malicious users to learn how to predict the lost manuscript. \nThus, it might be considered a form of active predictive attack. \nThe raw description can be found in lines 251-253 of the manuscript:\n\n```\nWhisper (White-box) then utilizes this collected data from malicious users to adapt the pre-trained Whisper.medium.en model to the specific masking pattern.\n```\n\nBesides, we have implemented two more active reconstruction adversaries and demonstrated our efficiency in defending against them. \n- `U-Net` is a traditional inpainting model based on convolutional U-Net structure, commonly used in literature to reconstruct missing audio signals [1,2]. \nWe utilize the SLURP training set and their masked counterparts to train the inpainting model from scratch to reconstruct the missing audio. \n- `CQT-Diff` is a neural diffusion model with an invertible Constant-Q Transform (CQT) to leverage pitch-equivariant symmetries [3], allowing it to effectively reconstruct audio without retraining. \n\nThe reconstructed audio is sent to Whisper for automatic recognition. The visualizations of reconstructed waveforms have also been included in the global PDF for your reference.\n\nThe updated evaluation results under attacks are summarized in the table below. \n|                   | **PlainText** | **Azure** | **Whisper** | **U-Net** | **CQTdiff** | **Whisper (white box)** |\n|:-----------------:|:-------------:|:---------:|:-----------:|:---------:|:-----------:|:-----------------------:|\n| **WER-SLU (%)**   |      14.7     |    81.6   |     78.6    |    82.5   |     74.3    |           67.3          |\n| **WER-ASR (%)**   |      12.3     |    71.6   |     68.1    |    71.4   |     65.9    |           64.4          |\n\n*Table: Potential attack Word Error Rate (WER) of different attack scenarios. PlainText means AllOffloaded without any privacy protection.*\n\nAs shown in the table, `U-Net` can rarely reconstruct the masked audio. \nEven worse, it introduces incorrect noisy signals, degrading the attack success rate. \n`CQT-Diff` inpainting can fill the missing waveforms but cannot successfully reconstruct the content because it is designed to reconstruct background music, such as piano concertos. \nSLU audio, which includes human intent and conversation, is difficult to reconstruct. \n\n`Whisper (white-box)` is fine-tuned to predict the missing transcript directly and shows the best attack performance among all attack methods. \nHowever, with our system enabled, it still results in over a 60\\% word error rate, which is 50\\% higher than without our preservation. \nThus, our system can successfully protect the private content in command audio under a wide range of attacks, including passive and active attacks.\n\n> *Q2. Is it possible to visualize some masks generated by the trained mask generator to verify that they effectively disrupt local utterances, aligning with the main motivation of the study?*\n\nSure! We have visualized some masks generated by the trained mask generator. \nThe figure has been added to the uploaded PDF in the global rebutta. \n\nIt can be seen that the mask generator can dispatch suitable mask granularity to proper speech granularity to some extent.\nWith more semantics utterance around, the mask becomes more meticulous, with the slices being distributed accordingly.\n\nReferences are listed in the global rebuttal.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers,

**Please see the attached PDF for a one-page summary with an illustrative description of different attack scenarios, visualizations of generated masks and reconstructed waveforms, and additional experiment results against advanced active reconstruction attacks.**

We would like to thank all reviewers for providing constructive feedback that helped us improve the paper. We are encouraged that reviews think our paper:

- “addresses real-time problem with its own novelty, which is quite rare for recent papers who mostly prefer large-scale solution with massive and massive data. The novelty here is thus quite high.” (Reviewer BMwq)
- provides “comprehensive experiment details” (Reviewer rbWb) with “significant performance and efficiency gains” (Reviewer ubkn) and “outperforms existing baselines significantly” (Reviewer YHpd)
- provides “detailed system illustration and rationale interpretation” (Reviewer rbWb), “making it an easy read.” (Reviewer YHpd)

We have been working diligently on improving the paper in several aspects, addressing your concerns and problems. Below, we summarise the changes that we have made in an updated draft.

**1. Experiment results against advanced reconstruction attacks**
- We first illustrated passive and active adversaries in the uploaded pdf, including the advanced reconstruction attacks.
- We implemented two advanced inpainting techniques to reconstruct the missing waveforms. (1) U-Net: a traditional inpainting method based on convolutional U-Net [1, 2]. (2) CQT-Diff: a neural diffusion model preconditioned with an invertible Constant-Q Transform (CQT) [3].
- The reconstructed waveforms are visualised in the attached PDF to see their effects.
- We conduct experiments to show that our system can still preserve content privacy under those advanced reconstruction attacks, with over 64% recognised word error rates. Detailed results are summarised in the attached PDF.

**2. Evaluation of the additional dataset**
- We have conducted further experiments on Fluent Speech Commands (FSC) dataset [5], another widely used dataset for spoken language understanding research. 
- Our system can still give accurate intent understanding (more than 99%, similar level with all the baselines) and defend against the sensitive word recognition attack (more than 80% WER, better than all the disentanglement-based protections). 

|               | AllOffloaded | VAE  | PPSLU | Local | Random | SILENCE |
|:-------------:|:------------:|:----:|:-----:|:-----:|:------:|:-------:|
| **ACC-SLU (\%)** | 99.7         | 98.3 | 99.2  | 99.7  | 86.4   | 99.1    |
| **WER-ASR (\%)** | 1.2          | 65.5 | 78.5  | 100   | 76.6   | 81.4    |

*Table: Evaluation of privacy preservation and SLU performance on FSC dataset.*

**3. Evaluation of the conventional modularised SLU**
- We implemented a conventional modularised SLU system with Whisper.medium.en as the ASR modular and two LSTM layers as the NLU modular.
- We conducted experiments on both `NLU only` and `Decouple SLU` settings to demonstrate that our system can still generate accurate intent understanding if we fine-tune the ASR modular.

|                   | Plaintext |   VAE   |  PPSLU  | NLU only (Ours) | Decoupled SLU (Ours) | E2E SLU (Ours) |
|:-----------------:|:---------:|:-------:|:-------:|:------------------:|:-----------------------:|:-----------------:|
| **SLU-ACC(%)**       |   87.2    |  72.5   |  74.5   |        12.6        |          89.1           |        81.1       |

*Table: System performance on conventional modularized SLU. Plaintext equals to AllOffloaded.*

**4. Evaluation on different speech granularity**
- We select two more speech granularities including Action (46 classes) and Intent (828 classes) classification.
- We conducted experiments to show that our system can understand the speech intent on different speech granularities.

|                   | AllOffloaded | VAE  | PPSLU | OnDevice | Ours |
|:------------:|:------------:|:----:|:-----:|:--------:|:----:|
| **ACC-Scenario (\%)** | 88.2         | 72.8 | 73.9  | 88.2     | 80.2 |
| **ACC-Action (%)**    | 77.1         | /    | /     | 77.1     | 76.4 |
| **ACC-Intent (%)**    | 83.3         | /    | /     | 83.3     | 76.8 |
| **WER-SLU (%)**       | 14.7         | /    | /     | 100      | 68.6 |
| **WER-ASR (\%)**      | 12.3         | 69.3 | 75.3  | 100      | 68.1 |

*Table: Comparison between privacy-preservation capacity and speech understanding performance at different speech granularities. ‘/’ means not supported. OnDevice leaks no words as nothing is uploaded.*

**5. More helpful visualisation and analysis**

We added visualisation of generated masks in the uploaded pdf to verify that they effectively disrupt certain local utterances.

**6. More clear statements and insightful discussion**
- We added a clear definition of the wimpy devices.
- We added a discussion about the integration with small on-device SLU models to address the occasional offline conditions.
- We clarified that detecting short-dependent key phrases or specific commands is not the focus of this work. But we also emphasised the importance of long-dependent intend classification. It is currently the main objective of SLU understanding literature [4] and has a wide range of application scenarios.

Please see our reviewer-specific feedback for more information.

---

[1]. Masking and inpainting: A two-stage speech enhancement approach for low SNR and non-stationary noise, ICASSP’23

[2]. Deep speech inpainting of time-frequency masks, interspeech’19

[3]. CQTDiff: Solving audio inverse problems with a diffusion model, ICASSP’23

[4]. A Survey on Spoken Language Understanding: Recent Advances and New Frontiers, IJCAI’21.

[5]. Fluent Speech Commands: A dataset for spoken language understanding research, Fluent.ai.

Meta-review: The paper presents a  speech processing system that selectively obscures short-term details to reduce privacy leakage.  The proposed solution, a differential mask, is is based on the observation that speech understanding hinges on long-term dependency knowledge of the entire utterance, in contrast to privacy-sensitive elements that are short-term dependent. The authors propose a differential mask generator that automatically masks out portions of audio signals during online cloud inference. Empirical results show that the proposed system offers comparable speech understanding performance and privacy protection capacity with high memory efficiency.

The extensive additional experiments that that the authors have conducted during the rebuttal phase should be included in the paper. These include the illustrative description of different attack scenarios, visualizations of generated masks and reconstructed waveforms, and additional experiment results against advanced active reconstruction attacks, as well as the evaluation on the Fluent Speech Commands (FSC) dataset, the evaluation of the conventional modularised SLU and the different speech granularity.

The authors are also encouraged to replace "wimpy" with "resource-constrained", as it would be easier for the community to infer what they mean. Regardless, they should explain precisely and clearly what resources are deemed constrained, to what extent (i.e., in a quantifiable fashion), and why devices such as, e.g., Rasberry PIs pass/fail the definition test. Finally, active adversaries should also be explicitly addressed, both in the paper's presentation but also through the inclusion of the conducted additional experiment results against advanced active reconstruction attacks.
Decision: Accept (poster)