Fine-grained Control of Generative Data Augmentation in IoT Sensing
Authors: Tianshi Wang
Qikai Yang
Ruijie Wang
Dachun Sun
Jinyang Li
Yizhuo Chen
Yigong Hu
Chaoqi Yang
Tomoyoshi Kimura
Denizhan Kara
Tarek F. Abdelzaher
Venue: NeurIPS 2024 poster (2024)
Keywords: Generative models
data augmentation
Internet of Things
signal processing
Primary area: generative_models

Abstract: Internet of Things (IoT) sensing models often suffer from overfitting due to data distribution shifts between training dataset and real-world scenarios. To address this, data augmentation techniques have been adopted to enhance model robustness by bolstering the diversity of synthetic samples within a defined vicinity of existing samples. This paper introduces a novel paradigm of data augmentation for IoT sensing signals by adding fine-grained control to generative models. We define a metric space with statistical metrics that capture the essential features of the short-time Fourier transformed (STFT) spectrograms of IoT sensing signals. These metrics serve as strong conditions for a generative model, enabling us to tailor the spectrogram characteristics in the time-frequency domain according to specific application needs. Furthermore, we propose a set of data augmentation techniques within this metric space to create new data samples. Our method is evaluated across various generative models, datasets, and downstream IoT sensing models. The results demonstrate that our approach surpasses the conventional transformation-based data augmentation techniques and prior generative data augmentation models.

Fine-grained Control of Generative Data Augmentation in IoT Sensing
Tianshi Wang, Qikai Yang, Ruijie Wang.' Dachun Sun, Jinyang Li, Yizhuo Chen, Yigong Hu, Chaoqi Yang, Tomoyoshi Kimura, Denizhan Kara, Tarek Abdelzaher
University of Illinois Urbana-Champaign, USA (tianshi3, qikaiy2, ruijiew2 dsun18, jinyang7, yizhuoc, yigongh2, chaoqiy2, tkimura4, kara4, zaher) @illinois. edu
Abstract
Internet of Things (IoT) sensing models often suffer from overfitting due to data distribution shifts between training dataset and real-world scenarios. To address this, data augmentation techniques have been adopted to enhance model robustness by bolstering the diversity of synthetic samples within defined vicinity of existing samples. This paper introduces novel paradigm of data augmentation for loT sensing signals by adding fine-grained control to generative models. We define metric space with statistical metrics that capture the essential features of the short-time Fourier transformed (STFT) spectrograms of IoT sensing signals. These metrics serve as strong conditions for generative model, enabling us tailor the spectrogram characteristics in the time-frequency domain according to specific application needs. Furthermore, within this metric space to create new data samples. Our method evaluated across various generative models, datasets, and downstream IoT sensing models. The results demonstrate that our approach surpasses the conventional transformation- based data augmentation techniques and prior generative data augmentation models.
propose set of data augmentation techniques
Introduction
IoT sensing applications aim to detect physical world phenomena in specific environments through time-series data captured by sensors, such as inertial, acoustic, and bioclectrical signals. Given the wide variety of real-world conditions, IoT sensing models often face potential domain shifts and
unpredictable variations when deployed [23]. Additionally, collecting and labeling IoT sensing data
expensive because sensing signals are harder for annotators interpret compared to images, videos,
and natural language 322 34
21 22]. Consequently, enhancing model robustness across diverse
scenarios with minimal collection overhead becomes a key research focus in the IoT field [55 56)
spectrum of data augmentation techniques has been proposed reduce the need for extensive data collection. The objective to diversify existing datasets while preserving a plausible data distribution. Conventional data augmentation methods for IoT sensing signals generally apply manually-crafiee transformations to create different perspectives of the same sample (58) [2]. Typical transformations include operations the time domain (such as jittering, rescaling, rotation, and cropping) 12 29 39
50 and the frequency domain (such as spectral flipping, warping, and masking) [58] 11, 38 They are designed modify the original data to reflect possible real-world variations, guided by domain knowledge in IoT. However, those transformations can be overly simplistic, failing to capture the
"Corresponding author. Email: ruijjer2@illinois.ee
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

nuanced low-level features of realistic IoT signal variance. The resultant artifacts may compromise the authenticity of the synthetic samples, thus offering limited benefits for downstream tasks.
Generative data augmentation techniques have recently been introduced to synthesize highly realistic loT sensing data [9]. such as variational autoencoders (VAEs) (24), generative adversarial networks (GANs) [3]. and diffusion models [46]. These techniques use inherent labels, such as activity types in human activity recognition (HAR) and vehicle types in vehicle detection tasks as conditions, for generating data. This process creates new samples with diverse characteristics under the same label, thereby enriching the variety of the training dataset. Despite advances in generating high quality signals, most existing works, if not all, fail to incorporate domain knowledge to guide the generation process, often treating generative models as a black-box.
Real Data
Potential Variations
Statistical Metric Space
Augmented Data
Contrast
Energetic Runninng
FO Amplitude
Flatness
Conditional rateratve
Sansing Domain Knowledge
1
Flatness
Moderate Running
Cautious Running
FO Amplitude
NewieeeNuuu Noural Network
Entropy
Temporal Amplitude 4
Smartwatch sensor)
Varied Terrain Running
Range
Figure An Application Example of Our Method. The insights of the potential variations in the metric space of running signals can guide the generation process of augmented data, thereby enriching the diversity that is missing in the original dataset.
In this paper, we introduce novel data augmentation paradigm for 1oT sensing data that combines domain knowledge control with generative data augmentation methods to advance the synthesis capabilities. We argue that generative model with fine-grained control over the human-interpretable characteristics of IoT sensing signals can enhance the data augmentation effectiveness on downstream tasks. Our approach begins by leveraging domain expertise in loT sensing extract set of statistical metrics in the time domain, frequency domain, and timeffrequency domain, which represent the essential aspects of IoT sensing signals. We call the resultant space determined by those statistical metrics as metric space. Subsequently, we develop a suite of data augmentation techniques designed to operate within the metric space.
In Figure we illustrate an application example our method. Given real data of moderate running in human activity recognition task, we speculate the potential real-world variations of running behaviors and leverage domain knowledge to derive the changes of statistical metrics accordingly. These metrics serve conditions for generative neural network to guide the synthetic data generation of other running styles like energetic, cautious and on varied terrain. The generated data is then utilized to augment the original dataset, thereby improving the robustness of downstream 1oT sensing model. The strength of our paradigm comes from the precise control over the generation process via the metric space, while ensuring the authenticity of the generated samples through the generative model.
Empirically, we validate the versatility of our augmentation approach with various generative model backbones including a diffusion model and a VAE. We test our approach in three distinct loT sensing applications: wearable-base human activity recognition, seismic and acoustic-based vehicle detection, and EEG-based harmful brain activity recognition. Our results demonstrate that our method outperforms conventional and generative data augmentations across various downstream models in all the evaluated sensing tasks.
2 Related Work
Traditional Data Augmentation for IoT Sensing. Many traditional data augmentation methods rely on manually crafted transformations to enhance dataset diversity, As IoT sensing signals typically

---

involve temporal dynamics while exhibiting periodical patterns, transformations are designed in both time domain [58]
T2 29
39
50 and frequency domain [58] 38). Implementing these
augmentation techniques requires specific domain knowledge to determine the variation range that can enhance data diversity without compromising relevance the intended task. Another data augmentation approach involves similarity interpolation, where new samples are created by interpolating between existing samples, maintaining task invariance the generated outputs. Data space interpolation techniques, which interpolate directly between raw data samples, have been proposed [ 63 37 Additionally, there has been research on feature space interpolation, which occurs after data has been embedded into latent representations [TO B 35 A primary challenge for this line work lies in the synthetic data quality. The transformations often represent straightforward, linear variations and may introduce artifacts that reduce the realism of the generated data.
Generative Data Augmentation for IoT Sensing. Generative data augmentation utilizes generative These models, including GANs [49 31 6. VAEs [66 14
models to create varied data 48 60 64
53 and diffusion models downstream models and significantly lower the costs associated with collecting real data. However, these prior work treat the generative model as a black box, using it to sample from the learned data distribution with minimal control, typically only conditioned by the inherent labels from the dataset. The lost of control makes the generated results unpredictable, and loses the chance to benefit from domain insights and prior experience. This work is inspired by prior research on conditional generation for human speech and singing signals [25 16 33]. In these studies, fine- grained conditions in vocal and/or linguistic features applied to control acoustic signal generation. However, unlike human speech, IoT sensing time series data carries entirely different semantics and encompasses broader range of modalities. Condition space interpolation involves the application of interpolation techniques within a condition space, which is constituted by statistical metrics These augmented metries are then fed into generative model to synthesize varied data. Our work significantly advances prior research by redesigning the metric space and enhancing data augmentation techniques within it. We also perform extensive evaluations across diverse generative models and IoT sensing tasks, resulting in a more general data augmentation framework.
generate authentic samples that improve the performance of
3 Method
3.1
Overview
Stage 1. Metric Extraction
Stage Training Generative Model
Metrics
Data x
Metric
Metrics
Condition
Generative model
Generated Data
Label
Data x
Stage Generating Augmented Data
Metric Space
Augmented
Metrics m
Augmentation
Metrics 171'
Condition
Generative
Augmented Data
Label y
Noise e
model
Figure 2: An overview of the three-stage pipeline. Given input data x and label Stage 1. Extract statistical metrics m. Stage 2. Train generative model G, conditioned on m (encapsulated within condition vector c). Stage 3. Generate new data using the augmented metrics m'.
As shown in Figure 2 our fine-grained controllable generative data augmentation has three stages.
In Stage we extract the predefined spectrogram metries that reflect the critical characteristics of IoT sensing spectrograms. The selection of these metrics is based on domain knowledge in general loT sensing, capturing essential facets in the IoT signal semantics. These metries together define the metric space that conditions the subsequent generative model.
In Stage 2, the metrie values, along with the class label, are embedded into condition tensor. This tensor then conditions the training of the generative model, guiding it to synthesize spectrograms that adhere to the specified metric values.
In Stage 3, we utilize the fine-grained controllable generative model to augment the original dataset within the metric space. The metrics in this space act control knobs, allowing for the directional
3

---

generation of IoT sensing signals with specific characteristies in the spectrogram semantics. This enables the application of domain knowledge about IoT sensing and downstream tasks to strategically augment the original dataset. We introduce several data augmentation techniques that vary in their reliance on domain knowledge. The augmented conditions (metric values) are input into the trained generative model to produce synthetic samples. These samples, combined with the original dataset, form the augmented dataset used to train the downstream IoT sensing models.
We introduce each stage in details in Section 3.2 3.3 and 3.4
3.2 Construction of Metric Space
We aim to design a set of statistical metrics which can effectively serve as fine- sreegrained control knobs for data augmentation and easy to be manipulated by the domain experts in loT sensing.
As IoT sensing signals typically exhibit strong patterns in both time and frequency domain, we convert the original time-series data into 2D spectrograms using short-time Fourier transformation (STFT). We propose metrics as below three domains: time domain, frequency domain, and frequenyy domain.
Time domain metric: Temporal amplitude oscillation commonly exist in loT sensing signals, which is a strong indicator for the dynamics of the observed physical phenomena. To measure this essential characteristic, we propose using the temporal amplitude range a spectrogram as metric for controlling the volatility of the generated signal. It straightforwardly informs on the operational limits of the loT sensing signal, which essential for a robust generation that must operate within specific amplitude constraints.
Frequency domain metric: We introduce Fundamental frequency (FO) amplitude as the metric to reflect the frequency domain characteristics of 1oT sensing spectrograms. The fundamental frequency pervasive across various IoT sensing signals. It dictates the dominant periodicity of the signal and is thus essential for capturing the signal's 's primary oscillatory properties. For example, in a signal captured by the accelerometer of a wearable device on the wrist when the human subject is walking. the fundamental frequency corresponds to the walking cycle.
Timeencyy domain metrics: The information distribution across the time- fime-frequency domain is strongly correlated to phenomena such as environmental noises, signal interference, and the harmonic characteristics of the signal source. To effectively capture these aspects, we introduce metrics including contrast, flatres, and entropy that quantify the amplitude distribution and information density within the spectrogram.
Contrast quantifies the difference in amplitude between peaks and valleys within the spectrogram. This metric is critical for identifying the dominant signal patterns, allowing for a more nuanced generation of spectrograms where both signal and noise components are realistically balanced.
Flatness measures how noise-like a spectrogram is by comparing the geometric mean to the arithmetic mean of the power spectrogram values. high flatness indicates a uniform distribution of power across the spectrogram, typical of stochastic processes and ambient environmental noises. A low flatness indicates concentration of energy regions. This results in spectrogram with interleaving of peaks and valleys, reflecting the presence of tones or harmonics.
Entropy measures the randomness of the power distribution within the spectrogram. High entropy suggests a complex, less predictable signal, while low entropy indicates a more structured and tonal signal. This metric is crucial for generating data that mimics the variable information content found in real-world IoT signals, from highly predictable periodic signals to complex, non- nor-periodii disturbances.
The formal definitions of the metrics above can be found in Appendix B
3.3 Conditional Generative Model Training
Given the original dataset, we extract the metric values vector m from each data sample and embed with the class label as the condition tensor c. During the training of a conditional generative model G, takes the original data sample x and the condition tensor as inputs. To enforce the alignment of the generated data x' with the characteristics defined by the given metrics m, loss term Cieetri

---

is taken as penalty on the difference between the metric values r calculated from x' and m:
M
Emetric
wm(mm -m;)²
(1)
where is the total number metrics, and w; is a hyperparameter as the weight for specific metric loss. The total loss L consists of the original generative model loss La and the metric loss:
= La + Lmetrio-
In practice, to balance the two loss terms, we empirically adjust the weight w of each metric so that LG Cmetric approximately equals 10:1 at the end of the first training epoch.
3.4 Data Augmentation Techniques in Metric Space
To fully leverage the fine-grained controllable generative model to synthesize diverse data samples, effective data augmentation techniques are required. Different from prior approaches, our data augmentation happens in the metric space where the extracted metric values from the original data are manipulated to create variant data, As shown in Figure we propose three data augmentation techniques which require varying levels of IoT domain knowledge, ranging from low to high.
Realistic Distribution Realistic Date
Augmented Data
Interpolation
Disturbance
Domain Knowledge Instruction
Requirement of Domain Knowledge
Figure 3: Data Augmentation Techniques in Metric Space. In the interpolation method, the green dotted line represents the intermediate value range between two real samples. In the disturbance method, blue circles indicate the range of disturbance. In the domain knowledge instruction method, the red outline denotes specific metric value ranges derived from domain knowledge.
Interpolation: A vast amount of prior research has demonstrated that interpolation is an effective data augmentation approach [5] 63 37 3.. The intuition is that the model can correctly classify not only the original samples but also these in- in-between points, it is likely to perform better on unseen data that fall within the same general manifold. Given pair randomly selected samples under the same label and j, we extract their metric values m, and m,. The interpolated metric value minterp is defined as:
Minterp
fm, B)
where 3 is random number between 0 and 1, independently sampled from Gaussian distribution for each interpolation. One advantage of interpolation that does not require specific domain knowledge create new data samples.
Disturbance: Inheriting from the design of conventional transformation-basel data augmentation methods, we disturb the original metric value of data sample within a predefined varying range [a, 6). where a and are the minimum and maximum percentages of the original metric value. During the metric disturbance, random number is sampled from the uniform distribution ((aa, b). Given all the metrics as control knobs for creating new samples, users of this augmentation technique can to define the valid disturbance range grounded by domain insights.
Domain knowledge instruction: For many IoT sensing applications, deeper insights are available about the downstream model and the potential diversity of the realistic data distribution. In contrast disturbance, these domain knowledge instructions provide more precise scope for metric manipu- lation. This accuracy allows for the synthesis of data that not only enhances the downstream model but also avoids the production of unnecessary variants. To showcase the benefits of incorporating domain knowledge in IoT sensing data augmentation, we present two case studies in Section
5

---

4 Experiments
We evaluate our data augmentation approach on three 1oT sensing applications where domain shift issues are prevalent: human-activityy recognition, vehicle detection, and harmful brain activity recognition.
4.1 Experimental Setup
Datasets: (1) Human-Activity Recognition (HAR): We use RealWorld-HAR dataset [47], which aims at classifying 5 human activities based on the accelerometer and gyroscope signals. (2) Vehicle detection: We detect 5 vehicle types by deploying seismic and an acoustic sensor on the ground, monitoring the sound and vibration caused by passing vehicles. This dataset is collected
by ourselves [30] (3) Harmful brain activity recognition: We adopt Harmful Brain Activity
Classificatton [18 19 dataset which comprises electroencephalogruphh (EEG) signals from critically ill patients in hospitals. The objective to classify types of harmful brain activities.
Technically, our fine-grained controllable generative data augmenta- tion approach is compatible with any conditional generative model. We utilize two types of generative model backbones to validate our approach: (1) Diffusion Model and (2) VAE [51].
Generative Model Backbones:
42 (referred to as Diff' hereafter),
We compare two categories of data augmentation methods as baselines. Traditional data augmentation: This category includes (1) data space interpolation [63], (2) feature space interpolation [10]. and (3) time and frequency domain transformation -bssdd method (38] Label-
Baselines:
only conditional generative data augmentation: We compare with 3 generative models that are only conditioned by the inherent label in the dataset, including (4) Conditonal Diffusion Model 42
(cDiff), (5) Conditional VAE (51) (cVAE) and (6) Conditional GAN [1] (cGAN).
Downstream Models:
We choose the following two downstream models: (1) DeepSense 61
deep learning model designed for analyzing IoT signals in the time-frequency domain, featuring multiple convolutional and recurrent layers. (2) Transformer (Encoder-Only) (52), which uses self- attention mechanisms to process sequential data and can capture complex dependencies. We only use the encoder component of the original Transformer architecture for classífication tasks.
Data Augmentation Setup: Each method produce 5 times of synthetic samples comparative the number of the original training samples. For our methods that utilize disturbance the augmentation technique, we set the adjustable range for each metric in between -20% to 20%. This range is reasonably chosen to encompass substantial variance while maintaining invariance critical for the downstream task.
More details about the experimental setup can be found in the Appendix C
D
and F
4.2 Overall Performance
Table I reports the performance of the 3 variants of our fine-grained controllable generative aug- mentation comparing with the baselines. Interp uses interpolation for augmentation, Disturb applies metric disturbance, and Reconst generates augmented data by using the actual metric values from the validation set as conditions, showcasing the potential upper bound performance of our method when validation data metrics are perfectly predicted. We highlight the gighest-performing cell in the table, excluding the Reconst variant.
Across all datasets and two downstream models, our fine- fine-grained controllable generative augmentation consistently outperforms other methods. Typically, the performance improves with the integration of more domain knowledge into the augmentation process. Interp operates on the premise that interpolating between real samples enhances dataset diversity, requiring no specific domain knowledge. Disturb, on the other hand, involves domain expertise to define the disturbance range, leading to better performance than Interp. Reconst, which utilizes ground truth metric values from the validation set, achieves the highest performance, illustrating the significant benefits of incorporating domain knowledge about loT sensing signal characteristics through our designed metrics into the generative augmentation process.
Using the same augmentation technique, the diffusion model generally surpasses the VAE as backbone. As illustrated in Appendix G.I VAE generates spectrograms that are less crisp and

---

Table General Performance. Average accuracy and F-1 score of 5 independent runs are reported.
Activity
Homaa Activity Xecognition
Vatuele Detection
Augmentaiion
Augmentation Method
Recognition
Categury
DeepSense
Transfomeer
DeepSense
Transtormer
DeepSente
Transformer
Acc
F1
Ace
F1
Ace
F1
Acc
F1
Acc
F1
Ace
F1
Augmeentttion
NIA
0.7752
0.7961
0.8044
0.7875
0.7768
0.7745
0,7631
0.7001
0.6895
00.842
0.6633
Diff-interp
0.5090
0,7912
0.7150
Fine-grained
Diff-disturh
0.8077
0.833
0.8305
0.8352
0.8300
0.8159
0.5008
0.7944
0.7511
0,7410
0.7355
0,7210
Conirollable
Dill reconst
0.8618
0.8664
0.859]
0,8600
0,8305
0.5054
0.8118
0.8104
0.7588
0,7429
0.7398
0,7220
Generative
VAL interp
0.8120
0.8123
0.8099
0,8292
0.7836
0.7733
0.7792
0,7653
0.6987
0.0653
0.8991
0.6838
Augmentation
VAF-disturb
0.8289
0,7901
0.7016
0.6801
VAE recomet
0.8372
0.8493
0.8242
0.8377
0.8227
0.8123
0,7897
0.7796
0,7033
0,6837
0.7060
0,6886
Data Space
Tradiiional
0.7894
0.7759
0.7745
0.7392
0.7119
0.7225
0.7125
Interpolation
Data
Feature Space
Augmentation
0.7656
0.7708
0.7534
0.7205
0.7730
0.7615
0.7688
0,7510
0.6903
0,7059
0.6822
0.6774
Interpelation
Time and
Prequesey
0.8011
0.8117
0.8155
1.81776
0.8003
0.7922
0.7721
0.7662
0.7222
0,7079
0.7099
0.6849
Drensin
Traasfomaaiin
Label-unly Conditionl
eDid
0.8034
0.8130
0.7921
0,8100
0,7833
0.7796
0,7660
0,7451
0.7374
0,7198
0.7225
0,7107
Generative Data
.VAF
0.8016
0.8124
0.7928
0.8150
0.7685
0.7538
0.7720
0.7615
0.6022
0.6772
0.69979
0.6810
Augmentation
GGAN
0.7531
0.7431
0.7221
0.7354
0.7323
0.7110
0.7082
0.6885
0.6721
0.6630
0.6514
0.6330
POTTeparrllAne Amplitude te Range 125%
(4) Contrast +25%
(b) Flatness
Entropy +25%
(e) Amplitude
F0
Original
1.8
1.5
20
18
15
20
232 (un]
50
juc]
Time (ec
(a) Contrast -25%
(b) Flatness -2%
Entropy 25%
(d) TeTTarraa Amplltdt Range 25%
(o) Amplitude -25%
(82
[s8c]
1900: (m)
1996 [mn
Figure 4: Visualization of Metric Control Effects on Accelerometer Spectrogram Generation from HAR Dataset. The inherent label of this signal is "running". We altered single metric condition while retaining the remaining metrics. (a) Increasing contrast enhances the distinction between peak and valley values, while decreasing contrast reduces the differences, making high and low energy areas less pronounced. (b) Higher flatness results in a uniform and consistent energy distribution across the spectrum; lower flatness leads to pronounced interleaving between peaks and valleys, cxhibiting stronger harmonic patterns. (c) Higher entropy creates a more random or chaotic energy distribution, whereas lower entropy results in simpler and more predictable spectrogram appearance. (d) Increasing the temporal amplitude range introduces greater dynamics over time, whereas reducing it leads more uniform amplitude distribution throughout the timeline. (e) Adjusting the FO amplitude emphasizes the intensity of the fundamental frequency without affecting the relative amplitudes of other harmonics. Additional visualizations available in Appendix G
realistic compared to those produced by the diffusion model, finding supported by numerous previous studies (28 2] We hypothesize that the inferior generative quality of the VAE limits the effectiveness of data augmentation. This suggests that the success of our data augmentation method depends significantly on the underlying generative model's ability to synthesize authentic data.
To validate the fine-grained control over the generation process, we visualize examples of the generation results in Figure 4 Results demonstrate the precise fine-grained control of our approach over the IoT sensing signal generation.
7

---

4.3 Data Augmentation with Domain Knowledge Instruction
We highlight the effectiveness of integrating domain knowledge enhance data augmentation through three case studies. In each case, we create domain shifting scenario and aim to increase the downstream model's robustness to variations outside the training set. We utilize the diffusion model as the generative model backhone and DeepSense as the downstream model.
Case study 1: Various Running Styles in HAR dataset. We the Real World-HAK dataset for our analysis. For the training set, we select 3 human subjects who demonstrated cautious style during the data collection of the "running" activity. For the validation set, we choose other 3 human subjects who exhibited energetic running patterns. The distinct running patterns of the subjects were assessed by reviewing video footage recorded by an observer holding a camera and following the runners. Based on IoT sensing domain knowledge, we hypothesize that the more forceful arm swings of energetic runners produce more intense and varied acceleration signals. The dominant arm swing frequency and its harmonics are also more pronounced. In spectrograms, these characteristies should manifest as higher contrast, lower flatness, and stronger FO amplitude compared to the cautious runners. As shown in Figure 5a our visual inspections of accelerometer spectrograms from each participant confirmed these assumptions.
We then simulate scenario where we only have the data from the cautious runners, and need to generalize the activity recognition the running activity of the energetic runners. In our training set, we have the data of all activities from the cautious runners, while the validation data comprises rumning activities from the energetic runners. Leveraging our domain insights, we directionally augment the data by intensifying the contrast and fundamental frequency amplitude and reducing the flatness in the spectrograms of the cautious runners. We set the varying range of contrast, flatness, and fundamental frequency amplitude at [10%, 40%]. [-40%. -10%]. and [10%, 40%] respectively, while keeping the other metrics unchanged. Having these ranges, we employ the perturbation augmentation method to create the augmented data. In Figure 5b) results show that our domain knowledge instruction (noted as knowledge) substantially improves the results, closely approaching the upper performance bound set by Reconst. This demonstrates that the generated data can more precisely reflect the true characteristics of the diverse running styles.
Cautious Runners (Training Set)
Energetic Runners (Validation Set)
6.02
Accuracy
6.45
Subjoct
Subject No.
Supject
SunjecrNo.2 years
Sudjeot
Sunject
yoars
124 years
(18 yaszza ofti tecmatee
(28 yuava
[25 yours old maley
male)
(emolle
muvi)
mase)
6.75
N6 Km
Dituth chonlsdyy Renwie
(a) Visual Comparison of Spectrograms between Cautious and Energetic Runners.
(b) Performance Comparison.
Figure 5: Case Study 1. Leveraging our insights into different running styles, our method enables more precise data augmentation and enhances the downstream model's performance on "running" activity recognition.
Case Study 2: Various Road Types in Vehicle Detection Dataset. In the vehicle detection dataset, the same vehicle driving on different road types exhibit various signal patterns. Our domain expertise suggests that the uneven and shifting surface gravel road generates sound and vibration profile with higher entropy compared to smoother concrete surface. As presented in Figure [a for the same vehicle type, spectrograms exhibit more complex and disordered patterns when on gravel than on concrete.
In this case study, simulate scenario where the training set only includes data collected the concrete road while the validation set is the data collected on the gravel ground. By applying our knowledge about the effect of gravel surfaces, we generate synthetic data by increasing entropy and flatness and decreasing contrast. We set the varying range of these 3 metrics as [10%, 30%] J. [10%, 30%], and [-30%, 0] respectively, while keeping the other metrics unchanged. Then we use perturbation augmentation to synthesize the data. In Figure 6b] the results show that our data
Public information and video footage about the human subjects can be found at: https://www.nii

---

augmentation informed by precise domain knowledge, significantly enhances the generalizability of the downstream model from concrete road to gravel road.
(Training
0.70
CompactSUV SUV
300m Car
Roadsder
Gravel Road (Validation Set)
0.65
0.60
Compact SUV
Sports Car
Roadotor
MNd size SUV
No Aug
cDiff
Interp
Disturb Knowledge Reconst
(a) Visual Comparison of Spectrograms between Concrete Road and Gravel Road.
(b) Performance Comparison
Figure 6: Case Study 2. Enhanced by method with domain knowledge instruction, the down- stream model more effectively manages domain shift in road types.
Case Study 3: Various Seizure Patterns in Harmful Brain Activity Dataset. We observe that the seizure signals from different patients can divided into two categories regarding fundamental frequency patterns. One is with clear fundamental frequency, which might be associated with the repetitive firing patterns of neurons in a specific brain region. We call it Clear Seizure. The other category has chaotic, noisy signals and blury fundamental frequency, which might be related to different brain regions contribute varying frequencies and patterns simultancously. We call it Chaotic Seizure. In this case study, manually select 10 patients of clear seizure as the traing set, and select 10 patients of chaotic seizure as the validation set. To augment the training data, we set the varying range of contrast, entropy. and fundamental frequency amplitude [10%, 25%], [10% 40%]. and [-25%, -10%] respectively, while keeping the other metrics unchanged. We show example spectrograms Figure 7a The experiment results shown in Figure 7b) Among all the baselines, our domain knowledge instruction achieves the best performance, again proving the effectiveness and generalizability of our approach.
(Training Set)
0.80
Acturacy
Score
0.75
Chaotic Seizure (Validation Set)
Prdeent 11
Pretent 1.2
Pesterr
Interp
Knowledge
(a) Visual Comparison of Spectrograms between Clear Seizure and Chaotic Seizure.
(b) Performance Comparison
Figure 7: Case Study 3. With domain knowledge instruction, our model achieves the best on bridging the gap between seizure patterns between two groups of patients.
4.4 Number of Augmentation Data
We investigate the influence of the number of augmentation data on the downstream model per- formance. Here we use diffusion model as the generative model backbone, and DeepSense as the downstream model. As depicted in Figure even though the ideal augmentation ratio varies in different datasets, steep climb in performance occurs from 50% to 500%. Among the 3 datasets, human activity recognition has the smallest data volume, which below 5000 training samples. We envision that the generative capability is fully expressed at lower augmentation ratio. Harmful brain

---

activity recognition dataset has the largest number of training samples (~ 80,000), where tendeney of performance growing remains even a higher augmentation ratio.
Accuracy
0.00
0.825
0.825
0.75
0.800
0.800
0.775
Accuracy
0.775
Accuracy
F1 Score
F1 Score
0.750
0.750
50% 100% 500%
750% 1000% Augmentation Ratio
50%
750% 1000% Augmentation Ratio
100% 500%
50% 100% 500% 750% 1000% Augmentation Ratio
(a) Human Activity Recogni- tion
(b) Vehicle Detection
(c) Harmful Brain Activity Recognition
Figure 8: Impact of Augmentation Ratio on Downstream Model Performance. Augmentation ratio = Number of generated samples Number of real samples in the training set
4.5
Ablation Studies
Table 2: Ablation Studies. Results with 95% confidence interval.
Humfal Brain
Autivity Rocogsition
Vehicle
Activity
Metries
Detection
Recognition
Ace
metries
+-0.011
08833-0000145
0,7925 0,00088
0.7748- 00,099
+00016
0.7430+00014
No time damain metrie
0.850-0.0000
00.0122
0.7745-50:1122
0.7691 20004
0.72
0.2058 0268
frequency meenie
0,0016
+0,0151
No tiqe-fiquuecy domain metrri:
00001220001666
0.8 081462000122
0.78812000113
0.7753::00119
0.7133
0,7021 $0,007
To investigate the impact of different metrics our data augmentation approach, ablate each category of metrics and evaluate their effects using the disturb augmentation method with diffusion model as the generative model backbone. Performance was evaluated using DeepSense as the downstream model. As depicted in Table excluding any metrics consistently results in diminished performance, validating the effectiveness of our selected metrics. The results also indicate varying importance of metrics across different datasets.
5 Conclusion
In this paper, we present method of fine-grained control generative augmentation in IoT sensing applications. Our novel data augmentation technique merges the advantages of leveraging domain knowledge specific with the production of highly authentic synthetic samples. Compared traditional transformation-basee and generative data augmentation methods, our approach demonstrates superior performance, particularly when domain-specific knowledge available, which is common scenario in many IoT sensing tasks. Furthermore, the versatility of our method enables its application across different generative models, allowing benefit from the inherent generative capabilities of these models.
Acknowledgments
Research reported in this paper was sponsored in part by DEVCOM ARL under Cooperative Agree- ment W91 N-F-172-019. NSF CNS 38878 and the Boeing Company. It was also supported in part by ACE, one of the seven centers in JUMP 2.0, a Semiconductor Research Corporation (SRC) program sponsored by DARPA. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Army Research Laboratory, DARPA, or the U.S. Government. The U.S. Government authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.
10

---

References
[1] Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International conference on machine learning, pages 214-223 PMLR, 2017.
[2] Jianmin Bao. Dong Chen, Fang Wen, Houqiang Li, and Gang Hua. CVAE-GAN: Fine-grained image generation through asymmetric training. In Proceedings of the IEEE international conference on computer vision, pages 2745-2755 2017.
[3] Yoshua Bengio, Grégoire Mesnil, Yann Dauphin, and Salah Rifai. Better mixing via deep representations. In International conference machine learning, pages 552-560. PMLR, 2013.
[4] Roi Benita, Michael Elad, and Joseph Keshet. DiffAR: Denoising diffusion autoregressive model for raw speech waveform generation. arXiv preprint arXiv:2310.0138// 2023.
[5] Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and w Philip Kegelmeyer. SMOTE: Synthetic minority over-sampling technique. Journal of artificial intelligence research, 16:321- 357, 2002.
[6] Wen-Hui Chen and Po-Chuan Cho. GAN-based data augmentation approach for sensor- based human activity recognition. International Journal of Computer and Communication Engineering, 10(4):75-84, 2021.
[7] Rewon Child, Very deep VAEs generalize autoregressive models and can outperform them on images. arXiv preprint arXiv:2011.10650 2021.
[8] Salvatore Cuomo, Vincenzo Schiano Di Cola, Fabio Giampaolo, Gianluigi Rozza, Maziar Raissi, and Francesco Piccialli. Scientific machine learning through networks: Where are and what's next. Journal of Scientific Computing, 92(3):88, 2022.
neural
[9] Suparna De, Maria Bermudez-Fdoe Honghui Xu, and Zhipeng Cai. Deep generative models in the industrial Internet of Things: A survey. IEEE Transactions on Industrial Informatics, 18(9):5728- 5737, 2022.
[10] Terrance Veieie and Graham W Taylor. Dataset augmentation in feature space. arXiv preprint 772 2017.
[11] Jingkun Gao, Xiaomin Song, Qingsong Wen, Pichao Wang, Liang Sun, and Huan Xu. Ro- bustTAD: Robust time series anomaly detection via decomposition and convolutional neural networks. arXiv preprint arXiv:2002.09545, 2020.
[12] Zijun Gao, Lingbo Li, and Tianhua Xu. Data augmentation for time-series classification: An extensive empirical study and comprehensive survey. arXiv preprint arXiv:2310. 10060, 2023.
[13] Ian Goodfellow, Jean Pouget- Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
[14] Maxime Goubeaud, Philipp Joußen, Nicolla Gmyrek, Farzin Ghorban, Lucas Schelkes, and
Anton Kummert. Using variational autoencoder to augment sparse time series datasets. In 202/ 7th international conference on optimization and applications (ICOA), pages 1-6. IEEE, 2021.
[15] Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.044 0444444 2023.
[16] Jiarui Hai and Mounya Elhilali. Diff-Pitcher: Diffusion- singing voice pitch correction. In 2023 IEEE Workshop Applications of Signal Processing to Audio and Acoustics (WASPAA), pages 1-5, IEEE, 2023.
[17] Jiashu He, Charilaos Kanatsoulis, and Alejandro Ribeiro. Network alignment with transferable graph autoencoders. arXiv preprint arXiv:2310.03272 2023.
[18] Chaoqi Yang Ashley Chow Sohier Dane Jimeng Sun Brandon Westover Jin Jing, Zhen Lin. HMS harmful brain activity classification, 2024.

---

[19] Jin Jing, Wendong Ge, Shenda Hong. Marta Bento Fernandes, Zhen Lin, Chaoqi Yang, Sungtae An, Aaron F Struck, Aline Herlopian, Ioannis Karakis, et al. Development of expert-level classification of seizures and rhythmic and periodic patterns during eeg interpretation. Neurology, 100(17)e1750-<2172, 2023.
[20] Jin Jing, Wendong Ge, Aaron F Struck, Marta Bento Fernandes, Shenda Hong, Sungtae An, Safoora Fatima, Aline Herlopian, Ioannis Karakis, Jonathan J Halford, et al. Interrater reliability of expert eletinenneephaloggapher identifying seizures and rhythmic and periodic patterns in EEGss Neurology, 100(17))e1737-=1174) 2023.
[21] Denizhan Kara, Tomoyoshi Kimura, Shengzhong Liu, Jinyang Li, Dongxin Liu, Tianshi Wang. Ruijie Wang. Yizhuo Chen, Yigong Hu, and Tarek Abdelzaher. Freqmae: Frequency-aware masked autoencoder for multi-modal iot sensing, In Proceedings of the ACM Web Conference 2024, WWW '24, page 2795-2806, 2024.
[22] Tomoyoshi Kimura, Jinyang Li, Tianshi Wang, Yizhuo Chen, Ruijie Wang, Denizhan Kara, Maggie Wigness, Joydeep Bhattacharyya, Mudhakar Srivatsa, Shengzhong Liu, Mani Srivastava, Suhas Diggavi, and Tarek Abdelzaher. Vibrofm: Towards miero foundation models for robust multimodal iot sensing. pages 10-18, 09 2024.
[23] Tomoyoshi Kimura, Jinyang Li, Tianshi Wang, Denizhan Kara, Yizhuo Chen, Yigong Hu, Ruijie Wang, Maggie Wigness, Shengzhong Liu, Mani Srivastava, Suhas Diggavi, and Tarek Abdelzaher. On the efficiency and robustness of vibration- based foundation models for iot sensing: A case study, 2024.
[24] Diederik P Kingma and Max Welling. Auto-encoding variational Bayes. arXiv:1312.6114, 2013.
arXiv preprint
[25] Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Dive A versatile diffusion model for audio synthesis. arXiv preprini arXiv:2009.09761 2020.
[26] Timo Korthals, Marc Hesse, Jürgen Leitner, Andrew Melnik, and UIrich Rückert. Jointly trained variational autoencoder for multi-modal sensor fusion. In 20/9 22th International Conference on Information Fusion (FUSION), pages 1-8, 2019.
[27] Vivian Lai, Chacha Chen, Q Vera Liao, Alison Smith-Renner, and Chenhao Tan. Towards a science of human-Al decision making: A survey of empirical studies. arXiv preprint 12. 2021.
[28] Anders Boesen Lindbo Larsen, Søren Kaae Sønderby, Hugo Larochelle, and Ole Winther. Autoencoding beyond pixels using a learned similarity metric. In International conference on machine learning, pages 1558-1566. PMLR, 2016.
[29] Arthur Le Guennec, Simon Malinowski, and Romain Tavenard. Data augmentation for time series classification using convolutional neural networks. In ECML/KKDD workshop on advanced analytics and learning on temporal data, 2016.
[30] Jinyang Li, Yizhuo Chen, Tomoyoshi Kimura, Tianshi Wang, Ruijie Wang. Denizhan Kara, Yigong Hu, Li Wu, Walid A Hanafy, Abel Souza, et al. Acies-OS: A content-centric platform for edge ai twinning and orchestration. In 2024 33rd International Conference Computer Communications and Networks CactoorsssCCCC (ICCCN) pages 1-9. IEEE, 2024.
[31] Xi' ang Li, Jinqi Luo, and Rabih Younes. ActivityGAN: Generative adversarial networks for data augmentation in sensor-based human activity recognition. In Adjunct proceedings of the 2020 ACM international joint conference on pervasive and ubiquitous computing and proceedings of the 2020 ACM international symposiun on wearable computers, pages 249 449-254, 2020.
[32] Dongxin Liu, Tianshi Wang, Shengzhong Liu, Ruijie Wang, Shuochao Yao, and Tarek E. Abdelzaher. Contrastive self-supervised representation learning for sensing signals from the fime-frequency perspective. In ICCCN, pages 1-10. IEEE, 2021.
[33] Jinglin Liu, Chengxi Li, Yi Ren, Feiyang Chen, and Zhou Zhao. DiffSinger: Singing voice synthesis via shallow diffusion mechanism. In Proceedings of the AAAI conference on artificial intelligence, volume 36, pages 11020-11028, 2022.
12

---

[34] Shengzhong Liu, Tomoyoshi Kimura, Dongxin Liu, Ruijie Wang. Jinyang Li, Suhas Diggavi, Mani Srivastava, and Tarek Abdelzaher. Focal: Contrastive learning for multimodal time-series sensing signals in factorized orthogonal latent space, 2023.
[35] Xiaofeng Liu, Yang Zou, Lingsheng Kong, Zhihui Diao, Junliang Yan, Jun Wang, Site Li, Ping Jia, and Jane You. Data augmentation via latent space interpolation for image classification. In 2018 24th International Conference on Pattenn Recognition (ICPR), pages 728-733, 2018.
[36] Yun Luo, Li-Zhen Zhu, Zu Wan, and Bao-Liang Lu. Data augmentation for enhancing EEG-bbaed emotion recognition with deep generative models. Journal of Neural Engineering, 17(5):05021,, 2020.
[37] Cheolhwan Oh, Seungmin Han, and Jongpil Jeong. Time-series data augmentation based on interpolation. Procedia Computer Science, 175:64-71, 2020.
[38] Daniel S Park, William Chan, Yu Zhang, Chung-Chenn Chiu, Barret Zoph, Ekin D Cubuk, and Quoc V Le. SpecAugment: A simple data augmentation method for automatic speech recognition. arXiv preprint arXiv:1904. 08779, 2019.
[39] Gautier Pialla, Maxime Devanne, Jonathan Weber, Lhassane Idoumghar, and Germain Forestier. Data augmentation for time series classification with deep learning models. In International Workshop Advanced Analytics and Learning Temporal Data, pages 117-132. Springer, 2022.
[40] Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint
2015.
[41] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted interrention-MICCAA 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings. part III 18, pages 234 241. Springer, 2015.
[42] Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models, 2022.
[43] Shuai Shao and Victor Sanchez. A study diffusion modelling for sensor-based human activity recognition. In 2023 11th International Workshop on Biometrics and Forensics (IWBF), pages 1-7. IEEE, 2023.
[44] Kai Shu, Yuchang Zhao, Le Wu, Aiping Liu, Ruobing Qian, and Xun Chen. Data augmentation for seizure prediction with generative diffusion model. arXiv preprint arXiv:2306.08256 2023.
[45] Ramzi Snoussi and Habib Youssef. VAE-based latent representations learning for botnet detection in IoT networks. Journal of Network and Systems Management, (1):4, 2023.
[46] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. In International conference on machine learning, pages 2256-2265. PMLR, 2015.
[47] Timo Sztyler and Heiner Stuckenschmidt. On-body localization of wearable devices: An investigation of position-aware activity recognition. 2016 IEEE International Conference on Pervasive Computing and Communications (PerCom), pages 1-9. IEEE, 2016.
[48] Toan Tran, Trung Pham, Gustavo Carneiro, Lyle Palmer, and Ian Reid. A Bayesian data augmentation approach for learning deep models. Advances in neural information processing systems, 30, 2017.
[49] Maximilian Ernst Tschuchnig, Cornelia Ferner, and Stefan Wegenkitul. Sequential IoT data augmentation using generative adversarial networks. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4212-4216. IEEE, 2020.
13

---

[50] Terry T Um, Franz MJ Pfister, Daniel Pichler, Satoshi Endo, Muriel Lang, Sandra Hirche. Urban Fietzek, and Dana Kulić. Data augmentation of wearable sensor data for Parkinson' 's disease monitoring using convolutional neural networks. In Proceedings of the 19th ACM international conference on multimodal interaction, pages 216-220, 2017.
[51] Arash Vahdat and Jan Kautz. NVAE: A deep hierarchical variational autoencoder. In Neural Information Processing Systems (NeurIPS), 2020.
[52] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[53] Ruijie Wang, Zijie Huang, Shengzhong Liu, Huajie Shao, Dongxin Liu, Jinyang Li, Tianshi Wang, Dachun Sun, Shuochao Yao, and Tarek Abdelzaher. Dydiff-vaat vae: A dynamic variational framework for information diffusion prediction. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '21, page 163-172, 2021.
[54] Tianshi Wang, Yizhuo Chen, Qikai Yang, Dachun Sun, Ruijie Wang, Jinyang Li, Tomoyoshi Kimura, and Tarek Abdelzaher. Data augmentation for human activity recognition via condition space interpolation within a generative model. In 2024 33rd International Conference on Computer Communications and Networks (ICCCN), pages 1-9, 2024.
[55] Tianshi Wang, Denizhan Kara, Jinyang Li, Shengzhong Liu, Tarek Abdelzaher, and Brian Jalaian. The methodological pitfall of dataset-driven research on deep learning: An loT example. In MILCOM 2022 2022 IEEE Military Communications Conference (MILCOM), pages 1082-1087, 2022.
[56] Tianshi Wang, Jinyang Li, Ruijie Wang, Denizhan Kara, Shengzhong Liu, Davis Wertheimer, Antoni Viros Martin, Raghu Ganti, Mudhakar Srivatsa, and Tarek Abdelzaher. SudokuSens: Enhancing deep learning robustness for IoT sensing applications using generative approach. In Proceedings of the 21 ACM Conference on Embedded Networked Sensor Systems, pages 15-27, 2023.
[57] Yilin Wang, Sha Zhao, Haiteng Jiang, Shijian Li, Benyan Luo, Tao Li, and Gang Pan. DifIMDD: diffusion- based deep learning framework for MDD diagnosis using EEG. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2024.
[58] Qingsong Wen, Liang Sun, Fan Yang, Xiaomin Song, Jingkun Gao, Xue Wang, and Huan Xu. Time series data augmentation for deep learning: A survey, arXiv preprint 2002 2020.
[59] Ruimin Xie, Nabil Magbool Jan, Kuangrong Hao, Lei Chen, and Biao Huang. Supervised variational autoencoders for soft sensor modeling with missing data. IEEE Transactions on Industrial Informatics, 16(4):2820-282) 2020.
(60) Shin'y ya Yamaguchi, Sekitoshi Kanai, and Takeharu Eda. Effective data augmentation with multi-domain learning GANs. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pages 6566-6574, 2020.
[61] Shuochao Yao, Shaohan Hu, Yiran Zhao, Aston Zhang, and Tarek Abdelzaher. DeepSense: A unified deep learning framework for time-series mobile sensing data processing. In Proceedings of the 26th international conference on world wide web, pages 351-360, 2017.
[62] Kuang Yuan, Shuo Han, Swarun Kumar, and Bhiksha Raj. DeWinder: noise reduction using ultrassund sensing. arXiv preprint arXiv:2409.061.77 2024.
wind
[63] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv: 171009942, 2017.
[64] Chenyu Zheng, Guoqiang Wu, and Chongxuan Li. Toward understanding generative data augmentation. Advances in Neural Information Processing Systems, 36, 2024.
14

---

Appendix
A Limitations
Metric Selection: One major limitation of our current approach that the metrics are selected em- pirically rather than through quantitative method. This means our choices are based on observation and experience on the limited IoT sensing datasets, which might introduce bias or overlook potential correlations within certain metrics and the data. Also, when new sensing applications are involved or sensing modalities are introduced, the functionality of these metrics need to be empirically validated again, which is inefficient. We envision the development of an improved method for assessing metric importance to mitigate this issue. For instance, we could identify broader set potential metrics and analyze their correlations with the discriminative features identified by the downstream model. The strength of these correlations could then be used to rank the importance each metric among all candidates.
Introduction of Domain Knowledge: Even after we obtained the fine-grained control over the generation process, deciding the proper augmentation strategy still requires deliberate considerations. The introduction of domain knowledge not straightforward and involves a nuanced understanding of the underlying mechanisms of the target domain. Currently, this process relies heavily on the expertise practitioners who understand both technical aspects of generative models and the specific characteristics of the domain. This is an issue that pervasively exist in all the domain knowledge-guine neural network design 27]. We believe this is highly challenging and we look forward to make advancements in future research.
B Metrics Design
We formally define the statistical metrics we proposed in constructing the metric space.
Let e R represent the STFT spectrogram, where f and are the dimensions of frequency and time, respectively.
B.1 Temporal Amplitude Range
The temporal amplitude range measures the variability in signal amplitude across time in a spectro- gram after averaging over the frequency dimension. It is defined mathematically as:
Temporal Amplitude Range max (A()) min (A(t))
(4)
where A(I) + S(i, 1) represents the average amplitude at time / over all frequency bins f in the spectrogram S. This metric captures the total temporal variation in signal strength and provides insights into the operational limits of the loT sensing signal.
B.2 Fundamental Frequency (FO) Amplitude
We define the fundamental frequency amplitude as mean amplitude at the fundamental frequency across all time bins. To calculate that, we first estimate the fundamental frequency fo for each time frame in the spectrogram using the Harmonic Product Spectrum (HPS) method.
This process involves the following steps:
Step Compute the HPS for Each Time Frame
For each time frame t in the spectrogram S(S, f), the HPS is computed by:
1. Generating downsampled versions of the spectrum for the given time frame, where the downsampling occurs on the frequency dimension only:
(5) where 4 d denotes downsampling the frequency index ʃ by factor of d, and D is the maximum downsampling factor. This approach reduces the frequency resolution while maintaining the temporal resolution intact.
Xa(f.t) S(f dd for = 1,2,....
15

---

In practice, we set D for various datasets to accommodate different sampling rates and harmonic patterns of the modalities. Specifically, for the HAR dataset, D 2 is used for both accelerometer and gyroscope signals. In the vehicle detection dataset, D = 3 is set for acoustic signals and D for seismic signals. For harmful brain activity recognition, D = 2 is applied to EEG signals.
2. Multiplying these downsampled spectra together to emphasize the fundamental frequency:
H(f,t) = II xa(f.t)
d=1
Step 2: Identify fo for Each Time Frame
The fundamental frequency fo()) at each time frame is identified as the frequency corresponding to the maximum value in H(ftt):
fo(t) = arg max H(f,t)
Step 3: Calculate Average FO Amplitude
Once the fundamental frequencies are determined for each time frame, the average amplitude at these frequencies is computed by averaging the amplitudes fo(t) over all time frames:
FO Amplitude
Σ S(f((t),
(8)
where T is the total number of time frames.
B.3
Contrast
Contrast calculates the distance between the amplitude peaks and valleys in the spectrogram. We define the sets of peaks and valleys:
Let be the set of values in that are in the top 5% of all values in S.
P= (S(,.) (ii, quantile(S, 0.95))
(9)
Let be the set of values in that are in the bottom 5% of values in S.
V (S(i,) ((i,) quantile(S, 0.05))
(10)
Contrast is defined as:
Contrast
Σ
Σ
(11)
P
PEP
PEV
Where PP and [V] represent the number of elements in the sets P and V, respectively.
B.4 Flatness
We first define the power spectral density, denoted as P(f,t) t), of the spectrogram S(f, t):
P(f [S(ftt)
(12)
Flatness of a spectrogram is defined as:
I Σt= Σイ= log log P(S,t)
exp TF
Flatness
(13)
TF E P(f.t)
where 7 is the total number of time bins, the total number of frequency bins, and F represents the total number of elements in the spectrogram.
16

---

B.5 Entropy
To calculate entropy, we first normalize the spectrogram to create a probability distribution:
S(f.t)
t(f.t) = Σ S(f.t)
(14)
Here, p(f,t) represents the probability of observing the power value at time and frequency f. The entropy of the spectrogram is then defined as:
F T
Entropy = t(s.
(15)
f=1t=1 t=1
where the sum is taken over all time and frequency bins, and the logarithm is base 2.
C
Datasets
We describe the details of each dataset in the following.
C.1 Human-Activity Recognition (HAR)
We utilize the Real-World HAR dataset 47 which consists of synchronized readings from multiple wearable sensors on 15 human subjects. Specifically, we select data from the accelerometer and gyroscope mounted on the upper arm, both of which record at 100 Hz sampling rate. Each subject performed five dynamic activities for approximately 10 minutes each, except for jumping, which lasted about 1.7 minutes. These activities include climbing down, climbing up, jumping, running, and walking. We divide the dataset randomly by subjects, assigning 10 subjects to the training set and the remaining 5 to the validation set. We segmented the entire recording into 2.5-second segments. This partition results in 4,959 training samples and 2,309 validation samples. Due to the individual physical variations among subjects during the same activity, domain shift occurs when the validation dataset comprises human subjects not included in the training set.
C.2 Vehicle Detection
We created our own dataset by deploying 5 IoT sensing nodes, each equipped with a seismic sensor (100 Hz sampling rate) and an acoustic sensor (800 Hz sampling rate), placed on the ground to monitor the sounds and vibrations from passing vehicles. The goal is to develop model that can classify vehicle types based on this sensor data, with each segment being seconds long. The data was collected across 3 different locations: city parking lot with a concrete surface, state park parking lot with gravel surface, and a country road with a dirt surface. This dataset includes the footage of 5 different vehicle types: compact-size SUV, a mid-size SUV, sports car, and convertible roadster. Each data collection session lasted for 40 minutes, during which single vehicle type traversed the testing field at speeds between 5 and 30 miles per hour along a random path. The initial 30 minutes data are used for training, while the final 10 minutes serve as validation. Data segments where the vehicle was more than 65 feet from any sensor were excluded based on GPS distance calculations. This process results in 8,015 training samples and 2,763 validation samples.
C.3 Harmful Brain Activity Recognition
We adopt Harmful Brain Activity Classification 18 19 dataset which comprises EEG signals obtained from hospitalized patients who underwent EEG monitoring as part of their clinical care. The sampling rate of the EEG sensors is 100 Hz. Each original recording. approximately 10 minutes long, is segmented into 40-second intervals. The objective is to classify 5 types of harmful brain activities using these data segments, including seizure (SZ), generalized periodic discharges (GPD), lateralized periodic discharges (LPD), lateralized rhythmic delta activity (LRDA), and generalized rhythmic delta activity (GRDA). The ground truth labeling was done by human experts manually 20) We allocate 80% of the original EEG recordings to the training set and 20% to the testing set. This partition results in 78,548 training samples and 19,532 validation samples.
17

---

D
Generative Models
D.1 Diffusion Model
Our diffusion model utilizes the architecture proposed in 42 which incorporates the U-Net as the backbone [41] a design prevalent across many diffusion models. U-Net comprises multiple convolu- tional layers, pooling, and up-sampling operations, all interconnected through skip connections that maintain spatial hierarchies for detailed segmentation. When 2D spectrogram gets into the U-Net, it is first progressively downsampled through convolutional and pooling layers to extract features, and then incrementally upsampled using transposed convolutions. Skip connections integrating details from corresponding downscaled layers to the upscaled layers for precise localization. In our implementation, we have 3 layers for downsampling and 3 layers for upsampling. Each downsam- pling/upsampling rescales both the time and frequency dimension of the spectrogram by 2. We set the intermediate channel dimension as 128 for the HAR and vehicle detection datasets, while 64 for the harmful brain activity recognition dataset. Both the HAR and vehicle detection datasets have 45.5 million parameters, while the harmful brain activity recognition dataset has 1 11.32 million.
Our conditioning embedder begins by taking each metric value m, with channel dimension d,, reshaping to d, x 1 x 1. This reshaped tensor is then replicated T times along the third dimension to align with the spectrogram's 's time dimension. The tensor subsequently passes through a individual linear layer which increases its channel dimension to F, altering its shape to de x F x T. This process is consistently applied to each metrie value and the one-hot embedding of the data label. Following this, all transformed tensors are stacked together on the channel dimension, resulting in a tensor of dimensions Σ; d, x F x T, where M represents the total number of metrics and the label. We then compute the average over the first dimension of this stacked tensor, yielding a final dimension of 1 x F x T. This resulting tensor, denoted as C, serves as the condition tensor and is concatenated along the channel dimension with either the input spectrogram x or noise c, which is in shape of D X F x T, where D is the feature channel of the sensor signals. The concatenated tensor is in shape of 1) F and then fed into the U-Net described above.
During the training of the diffusion model, we compute the metric loss using the generated spectro- grams. To accelerate the generation process, we employ diffusion step of 10, which offers balance between moderate quality and rapid generation. For the data generation phase, however, we utilize a diffusion step of 100 to produce augmented data of higher quality.
We train the diffusion model using an Adam optimizer with a learning rate of 0.0001, paired with cosine annealing learning rate scheduler. The model is trained for 1,000 epochs the HAR and vehicle detection datasets, and for 200 epochs on the harmful brain activity recognition dataset. The batch size is set at 200 for the HAR and vehicle detection datasets, and at 64 for the harmful brain activity recognition dataset.
We handle the multi-modal inputs from HAR (accelerometer + gyroscope) and vehicle detection (acoustic + seismic) by employing two separate U-Nets, each dedicated to processing one modality. Although we anticipate strong correlations between the modalities, no modality fusion occurs within between the two U-Net architectures. We consider integrating modality fusion as a potential direction for future research to enhance the diffusion model's performance with multi-modal IoT sensing data.
Our model was trained on a desktop with Intel(R) Core(TM) 19-9960X CPU 3.10GHz and Nvidia GeForce RTX 2080 Ti. The training of the diffusion model on single GPU lasts for around 2 days for HAR and vehicle detection datasets, and around days for harmful brain activity recognition dataset.
D.2 VAE
VAEs have been extensively applied to various IoT tasks in prior research. 45 the hierarchical features of the spectrogram, our VAE model developed based on the NVAE [5I]. The effectiveness of such hierarchical structure has been tested and verified in multiple prior studies 15 17 The original NVAE model constitutes of a 3-layer encoder and 3-layer decoder, where each layer is residual cell, in order to model the long-range correlations in data(5ī]. Unlike the naive VAE model 24), NVAE designs its reparameterization process as a 3-step dependent sampling where each step is completed right before each layer of the decoder. Our conditional NVAE model remains
26 59 62
To capture
18

---

most of the architecture and settings of the original NVAE with only few changes. We replace the prior trainable parameters in the first layer of NVAE's decoder with our condition embedder, following the same settings used in our diffusion model. Similarly, we also add a condition embedder the encoder to concatenate our conditioning embedding to the input data.
During the training process, we feed our conditions and ground truth data to the encoder to generate the latent variables of each layer. Then we feed the conditions the beginning of the decoder and the latent variables to each layer's s sampling process respectively, to get the generated data. We calculate loss as the mean squared error (MSE) between the ground truth and generated data, added by a weighted KL divergence loss. The weighting for each dataset is as follows: 1 for the HAR dataset, 10 for the vehicle detection dataset, and 8 for the harmful brain activity recognition dataset.
We train the NVAE model using an Adam optimizer with a learning rate of 0.0005, paired with a cosine annealing learning rate scheduler. Our model is trained for 200 epochs on the HAR dataset, 300 epochs on the vehicle detection dataset, and 600 epochs on the harmful brain activity recognition dataset. The bacth size of all datasets is set as 64.
The total number parameters for each dataset is 19 million. We trained our VAE model on the same machine our diffusion model. The training time of each dataset is hour for the HAR dataset, 2 hours for the vehicle detection dataset, and 16 hours for the harmful brain activity recognition dataset.
E
Baselines
We follow the method called mixup proposed in 663. Mixup conducts interpolation within pairs of examples and their labels. The intuition is that mixup regularizes the neural network to favor simple linear behavior in- beteeween training examples, which reduces undesirable oscillations when predicting outside the training examples.
Data Space Interpolation:
Given pair of randomly selected data samples (I;; Vi) and (ırj Mj); where xi and x are data vectors, while and Vy are one-hot label encodings, the interpolated data sample (x',y") is constructed by:
x' Ar; + (1 - A)rr
(16)
y/ Ay + (1 A)ys
In our implementation, we follow the original authors' setup by randomly sampling À from beta distribution:
X d~ Beta
(17)
Feature Space Interpolation: As suggested by [10], first train an autoencoder using an unsupervised approach, followed by a linear interpolation of the intermediate feature maps from two randomly chosen samples of the same activity generate synthetic feature maps. The architecture of the autoencoder largely follows our VAE design described in Appendix D.2] but only keep the reconstruction loss during the training. These synthetic alongside the realistic feature maps are subsequently utilized to train the downstream models.
Time and Frequency Domain Transformation:
To augment data sample, first randomly apply 2 out of time domain transformations to the data in time series form. Subsequently, we employ short-time Fourier transform to convert the serieseries data into 2D spectrograms, and apply out of 2 frequency domain transformations. We adopt the time domain transformations described in 50, including jittering, scaling, negation, permutation, time warping, magnitude warping, rotation, and cropping. For frequency domain transformations, we incorporate frequency masking 38 and frequency perturbation IT).
This diffusion model has the same design depicted in Appendix D. with the distinction that the condition vector comprises only the embedding of the inherent label. During the generation phase, synthetic samples for each class are created by the trained diffusion model using the respective class label conditions. This approach facilitates direct comparison to assess the benefits of the fine-graiin grained control and metric space augmentation we propose.
Conditional Diffusion Model:
This method follows the design described in Appendix D.2 but differs in that the condition vector includes only the embedding of the inherent label. This setup is designed to evaluate the impact of label embedding on the generation process within the VAE framework.
Conditional VAE:
19

---

We utilize the deep convolutional generative adversarial network (DCGAN) architecture 40 for this model. To enhance the stability and balance between the generator and discriminator, we implement optimization strategies on the loss terms as suggested by []]. Similar to the other models, this GAN also uses only the inherent label as a condition for generation
Conditional GAN;
F Downstream Models
This is a deep neural network tailored for handling time series data from the loT sensing applications. This network operates the time-frequency domain, transforming the data using the short-time Fourier transform. To process data from multi-modalities, the model includes two separate convolutional layers for each sensor, with the resulting intermediate feature maps being merged subsequently. After that, it employs 2 2D-convolutional layers to extract spatial features from the data. Following this, Gated Recurrent Unit (GRU) layer is used to learn temporal patterns. Next, 2 linear layers used for refining the feature dimensions and generate the final classification logits.
DeepSense: 66
Transformer (Encoder-Only): [52] In the Transformer network, each input spectrogram from different modalities is fed into standard Transformer encoder layer, incorporating self-attention layer along with two linear layers. Here, the attention is applied to along the time dimension, while the frequency and feature dimensions are multiplied, serves the embedding dimension. Multimodal features are subsequently integrated through concatenation followed by processing via a linear layer. The classification mechanism the end consists of linear layer followed by softmax function.
G
Additional Visualization
G.1 Diffusion Model vs. VAE
We compare the generation quality of our method when using the diffusion model versus the VAE as the generative model. For this analysis, we use accelerometer signals from the human activity recognition dataset. Synthetic samples are generated under the same conditions as the original data samples. We select two groups of the original spectrograms based on their complexity in patterns: the first group has simpler and more pronounced pattems,, while the second group has more complex and chaotic pattern..
As illustrated in Figure 9 the VAE-generated results comparable to those from the diffusion model in the first group of the selected samples. Both the diffusion model and VAE successfully capture the fundamental frequency and its harmonic patterns. This group of selected samples exhibits temporal consistency with pronounced harmonic patterns that are straightforward to replicate. However, despite the relatively simple structure of these realistic samples, the spectrograms produced by the VAE less detailed than those from the diffusion model, particularly in the higher frequency regions.
For the second group, which features generation quality of the VAE significantly degrades. The findings indicate that the VAE struggles to replicate the intricate textures present in the original spectrograms, whereas the diffusion models continue to effectively capture and reflect these characteristics.
complex original spectrograms shown in Figure 10 the
20

---

Original
Diffusion Model
VAE
Freturncy [NN)
40
Frequanccy [122)
10
20
8
8.
6.
Time [soc]
Time [sec]
Time
Running
(h2)
Peeqeercy [Ho)
20
Frequerccy
%
2.0
25
0.5
20
35
200
Time [9ec]
Time (sec]
Running
10
Frequarcy [M2))
Frequeniyy [1tc)
Frerg uerico [FCC)
8
0.5
1.0
[Soc]
1.5
1.0
20
Time [s05]
Walking
Figure 9: Simpler Cases of Generation. The original spectrograms exhibit strong and consistent patterns that are easier to capture. Both models capture the fundamental frequency and harmonics. But VAE exhibits more blurry patterns.
The observation that VAE generates data with lower quality has been recognized in many prior research[28] 2). Consistent with these findings, our experiments also demonstrate that the VAE, in its current implementation, exhibits lesser generative capabilities compared to our diffusion model.
21

---

Original
Diffusion Model
VAE
Innouenno (1ac)
Reequer cy (H3)
3
50
Timsc [504]
Time [pec]
Time [sec]
Jumping
Freq serty [1xl
40
Resque (HK)
Time [sec]
Climbing Down
12
Trequency (H2)
Froq eeroy [He)]
3
2
0.5
1.0
[s0c]
1.5
2.0
60
Time [sec]
%
1.5
Walking
Figure 10: More Complex Cases of Generation. The original spectrograms are more chaotic and textured. VAE fails in capturing those details and variations.
G.2 Visualization of Vehicle Detection Dataset
In Figure randomly select acoustic sample from the vehicle detection dataset and visualize the generated spectrograms when manipulating the metrics as conditions. Again, we change a single metric condition while retaining the remaining metries to show control effects.
22

---

Original
173
Requunccy
105
[asc]
(d) Amplitude
(a) Contrast +25%
(b) Flatness +25%
(c) Entropy +25%
(e) FO Amplitude +25%
175
115
115
123
125
17%
100
30
[sec]]
Time (soc)
Time [unc]]
(d) Temporal Amplitude
(a) Contrast -25%
(b) Flatness -25%
(c) Entropy -25%
25%
(e) Amplitude -25%
150
303
100
100
322
2
Time [102]]
Tiol [Sac]
Timel [secc
Temm [vec]
Figure 11: Visualization of Metric Control Effects on Acoustic Spectrogram Generation from Vehicle Detection Dataset.
G.3 Visualization of Harmful Brain Activity Recognition
we randomly select an EEG sample from the harmful brain activity recognition dataset and visualize the generated spectrograms. Like above, when manipulating a certain metric condition, we keep the remaining metrics unchanged.
In Figure 12]
23

---

Original
20
20
Temporal Amplitude Range +255
Contrast +25%
(b) Flatness +25%
(c) Entropy +25%
(o) FO Amplitude +25%
Requency (HQ)
Frecuencyy [11))
[ML)
prepuenc (arr
Tine [see]]
Time [sac]
Time [scc]
Time [ssc]]
Temporal Amplitude
(8) Contrast -25%
(b) Flatness 25%
(c) Entropy -25%
Range -25%
Amplitude -2%%
Prequeecy Ltte)
Pruquency (Hc)
Prqquencc (eet
mequeroy (42)
20
[1Nc]
Time (sc
Time [sec]
Time [sec
Time Isecl
Figure 12: Visualization of Metric Control Effects on EEG Spectrogram Generation from Harmful
Brain Activity Recognition Dataset.
24

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our claims are clarified by detailed descriptions of the methodology and validated by comprehensive experiments.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: We put a discussion on the limitations of the work in Appendix A
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: We do not have any theoretical result in this paper.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We have detailed description on our proposed method as well as the model implementation. We will release the code and our collected dataset upon paper accep- tance.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We do not plan to release our code or data before paper acceptance.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: We have specified the necessary details for training and testing in Appendix D
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We report the average results of 5 independent runs.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: We specify the resources requirements in Appendix D
25

---

9. Code Of Ethics
Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httpppss//aaarpppc.cccccaaachcta
Question:
Answer: [Yes]
Justification: Our research conforms with the NeurlPS Code of Ethics.
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: We do not anticipate that the work presented in this paper will be directly applicable negative societal uses.
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We do not foresee need restrict access to our model. The datasets utilized are either publicly available or consist of anonymized IoT sensing data related to vehicles, posing no risk of misuse.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cited all the directly relevant papers and datasets used this work.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: do not plan to release any asset from this paper before acceptance. 14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification: The two public datasets (Human Activity Recognition and Harmful Brain Activity Recognition) employed in this paper are both thoroughly documented and accessible online.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Our paper does not directly conduct research human subjects involve crowdsourcing. The two human related datasets we used in our work are fully public
resources.
26


Reviews:
[{"review_id": "ccfKNYgTCI", "reviewer": "Reviewer_7DMe", "summary": "This paper is a study on data augmentation techniques for IoT sensing applications. The authors first present work on the most effective data augmentation techniques in the field of IoT sensing data. Based on this, they propose that domain knowledge can be combined with these data augmentation techniques to guide data generation. Specifically, they also propose a approach of fine-grained controllable generative augmentation, which uses data augmentation techniques based on the metric space of statistical metrics defined by human experience to improve data augmentation for downstream tasks. Experimental results from various scenarios validate the effectiveness of this method, and also explore the benefits of incorporating domain knowledge in data augmentation techniques.", "questions": "Please see the Weaknesses above. Considering my limited expertise in this specific domain, I am not in a position to raise specific questions.", "limitations": "Yes", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The authors make the point that domain knowledge can be integrated into data augmentation techniques is interesting.\n2. Experimental results indicate that the proposed fine-grained controllable generative augmentation method is simple and effective.\n3. Various scenarios are fully considered in this study. Various data augmentation methods are thoroughly evaluated in IoT sensing applications in different domains using different downstream task models.", "weaknesses": "The motivation of this paper to propose a fine-grained control approach, incorporating domain knowledge into the data enhancement approach is not clear enough. They are not based on a specific existing problem and are a little bit heuristic.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your positive comments on the paper idea and the evaluations. We would like to make the following responses and hope they can address your concerns.\n\n> W1: The motivation of this paper to propose a fine-grained control approach, incorporating domain knowledge into the data enhancement approach is not clear enough. They are not based on a specific existing problem and are a little bit heuristic.\n\n**Response:** Thank you for raising this point! We would like to elaborate on the distinct nature of IoT sensing to clarify our motivation. IoT sensing signals typically capture physical phenomena that exhibit strong patterns in both time and frequency domain. Therefore, a common practice is to transfer the time-series data to 2D spectrograms using short-time Fourier transformation. Even though the spectrograms are 2D matries just like images, they hold unique semantic meanings distinct from ordinary images. For example, an object's vibrations are characterized by its fundamental frequency (F0), appearing as the lowest bright band across the spectrogram's frequency range. The amplitude of F0 is a strong indicator to the physical movement. In human activity recognition task, it reprsents the intensity of arm swings during running. Even though as domain experts, we clearly understand that adjusting F0s amplitude allows us to generate varied yet representative signals of running; however, without fine-grained control over it, we cannot enforce the generative model to produce meaningful variations related to the intensity of arm movements. This requirement has driven the development of our paper.\n\nTo support our motivation, we conduct comprehensive evaluations on three different IoT datasets in Section 4.2. We also present two real-world case studies in Section 4.3, demonstrating how integrating fine-grained control and domain knowledge significantly enhances IoT task performance.\n\nFurthermore, the incorporation of domain knowledge into neural network design for IoT sensing tasks is an area of active research, as evidenced by the success of many prior studies [1][2][3]. These successes also inspire us on incorporating domain knowledge into the data augmentation.\n\n[1] Physics-Informed Data Denoising for Real-Life Sensing Systems\n\n[2] From Physics-Based Models to Predictive Digital Twins via Interpretable Machine Learning\n\n[3] PhyAug: Physics-Directed Data Augmentation for Deep Sensing Model Transfer in Cyber-Physical Systems", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bXfAagqX80", "reviewer": "Reviewer_8SZC", "summary": "This paper aims to synthesize IoT data to augment sensing models with generative models. The authors designed metric spaces to model the conditions which control the generative models. Experiments involving different IoT tasks, generative models, and sensing models have been conducted, proving the effectiveness of the proposed method.", "questions": "1. It is reasonable that the synthesized data can improve the generalization of sensing models. Can you show the results of adding Gaussian noise to samples as augmentation? This can be the simplest baseline, proving that your generative model can  work because of the synthesized data, but not because of using more samples that help tackle overfitting.\n2. According to the appendix, in the first dataset, you separated different domains, while in the last two datasets, you did not. Can you provide further experiments where domains for training and validation are separated for the last two datasets?\n3. The authors introduced the weight $w$ in the loss $\\mathcal{L}$. Can you provide hyperparameter sensitivity analysis for this?\n4. There are indeed latest works using spectrograms as conditions for generative models to augment models [1,2]. Can you further discuss your strengths compared to them? It seems that your work has covered more IoT modalities.\n\n[1] DiffWave: A Versatile Diffusion Model for Audio Synthesis\n\n[2] DiffAR: Adaptive Conditional Diffusion Model for Temporal-augmented Human Activity Recognition", "limitations": "1. Despite the effectiveness, the paper has not discussed the efficiency of generative models. As mentioned in the papers, it took several days to train generative models. I think this is a minor issue but can be discussed in the limitations.\n2. This paper mainly discusses models for classification, while in IoT there may also be regression problems. For example, it is typical to track subjects using IoT devices.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The organization of the paper is clear, and the writing is easy to understand.\n2. The motivation of the paper is clear. It is laborious to collect IoT data, while the limited data hinder the performance of sensing models. Therefore, this paper uses generative models to synthesize IoT data.\n3. One of the novelties is the design of metric spaces, which can contribute to data augmentation in different aspects.\n4. The authors have provided experimental results using different IoT datasets towards different sensing tasks.", "weaknesses": "1. Although I understand the novelty of using the proposed metrics as conditions for generative models, I am not sure about using the term “metric”. From my point of view, it seems that calling it “indicator” or other terms is more suitable. Otherwise, it may confuse with the metrics (accuracy, F1) in the experiments.\n2. A transformer typically includes an encoder and a decoder. It seems that you were using  an encoder as the classifier, which is fine. However, you should clearly state this to avoid ambiguity.\n3. For generative models, there are actually specific metrics to evaluate their performance, such as FID etc. I understand that this paper aims to augment sensing models so it is reasonable to measure the accuracy and F1, but specific metrics for generative models may be also beneficial to the quality of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **Author Reply to Reviewer 8SZC (Part 1/4)**\n\nThank you for your constructive suggestions and insightful questions! We would like to add further clarification and evaluation to address your concerns.\n\n> W1: Although I understand the novelty of using the proposed metrics as conditions for generative models, I am not sure about using the term “metric”. From my point of view, it seems that calling it “indicator” or other terms is more suitable. Otherwise, it may confuse with the metrics (accuracy, F1) in the experiments\n\n**Response**: We agree that changing \"metric\" to \"(statistical) indicator\" would be a better way to define the conditions we used for controlling the generation. \n\n> W2: A transformer typically includes an encoder and a decoder. It seems that you were using an encoder as the classifier, which is fine. However, you should clearly state this to avoid ambiguity.\n\n**Response**: Thanks for the correction. Indeed, the classifier we used is only the encoder of a conventionally defined Transformer. We will clarify that in the paper to eliminate any confusion.\n\n> W3: For generative models, there are actually specific metrics to evaluate their performance, such as FID etc. I understand that this paper aims to augment sensing models so it is reasonable to measure the accuracy and F1, but specific metrics for generative models may be also beneficial to the quality of the paper.\n\n\n**Response**: Thank you for the valuable comment! Conventionally, FID is based on the Inception-v3 model that was trained on ImageNet. It can only compare the difference of data distribution between generated images and real-world object photos. To quantitively measure the quality of the generated spectrograms, we build FID using our own dataset and DeepSense as the neural network backbone. For each IoT sensing dataset, we train DeepSense on the both the training and validation set. Then, we use DeepSense as an encoder, taking the intemediate outputs from the linear layer after its GRU layer as the latent representation, which is a 128 length vector. To calculate the distance between two datasets, we compute the mean and covariance of the latent representations from DeepSense encoder of each set. Then we calculate the Fréchet Inception Distance between the two distributions.\nTo illustrate that the generated samples can approximate the validation set, we first calculate the FID between training set and validation set (noted as \"No Aug\"). Then, for each data augmentation baseline, we calculate the FID between training set + augmented data and validation set. For baselines, we select the conventional time and frequency domain transformation, conditional diffusion model (cDiff), and Diff-disturb (ours). We want to show that by adding augmented data, the distance between training set and validation set should be reduced. We report the results in the table below:\n\n|                                          | Human Activity Recognition | Vehicle Detection | Harmful Brain Activity Recognition |\n|------------------------------------------|----------------------------|-------------------|------------------------------------|\n| No Aug |   9.26  |  6.77   |   10.22  |\n| Time and Frequency Domain Transformation |  10.44   |    8.32 |  11.90  |\n| cDiff   |  10.17  |  7.30  |  13  |\n| Diff-disturb (ours)    |  **7.61**   |  **6.00**   | **8.99** |\n\nDiff-disturb effectively reduces the FID between the augmented training set and the validation set. This result directly prove the generation quality of our method. \n\n\n> Q1: It is reasonable that the synthesized data can improve the generalization of sensing models. Can you show the results of adding Gaussian noise to samples as augmentation? This can be the simplest baseline, proving that your generative model can work because of the synthesized data, but not because of using more samples that help tackle overfitting.\n\n**Response**: We add Gaussian noise to the samples by a random ratio between 0.1~0.4 to the mean amplitude of the data sample in the time domain. The random ratio is drawn from a uniform distribution. We report the accuracy and F1-score of the downstream model based on DeepSense downstream classifier: \n\n1. *Human Activity Recognition*\n\n|                     | Accuracy | F1     |\n|---------------------|----------|--------|\n| No Augmentation     | 0.7752   | 0.7961 |\n| Add Guanssian Noise | 0.7629   | 0.7723 |\n| Diff-disturb (Ours) | **0.8307**   | **0.8333** |\n\n\n2. *Vehicle Detection*\n \n|                     | Accuracy | F1     |\n|---------------------|----------|--------|\n| No Augmentation     | 0.7875   | 0.7768 |\n| Add Guanssian Noise | 0.7911   | 0.7822 |\n| Diff-disturb (Ours) | **0.8200**   | **0.8159** |\n\n3. *Harmful Brain Activity Recognition*\n \n|                     | Accuracy | F1     |\n|---------------------|----------|--------|\n| No Augmentation     |  0.7001  | 0.6893 |\n| Add Guanssian Noise |  0.7029  | 0.6953 |\n| Diff-disturb (Ours) |  **0.7511**  | **0.7410** |\n\nAs the results demonstrate, simply adding Gaussian noise can hardly regularize the downstream model. This indicates that effective data augmentation indeed requires synthetic samples with plausible real-world variations rather than just noisy data.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HSInSab8nM", "reviewer": "Reviewer_eCvh", "summary": "The paper addresses the challenge of overfitting in Internet of Things (IoT) sensing models caused by data distribution shifts between training datasets and real-world scenarios. To enhance model robustness, the authors introduce a novel data augmentation paradigm that adds fine-grained control to generative models. By defining a metric space with statistical metrics that capture the essential features of short-time Fourier transformed (STFT) spectrograms of IoT sensing signals, the method enables tailored augmentation of spectrogram characteristics in the time-frequency domain. The proposed data augmentation techniques are evaluated across various generative models, datasets, and IoT sensing applications, demonstrating superior performance compared to conventional transformation-based and prior generative data augmentation techniques.\n\nThe major contributions of this paper are as follows. It empirically validates the versatility of the proposed augmentation approach using different generative model backbones, including diffusion models and VAEs, across three distinct IoT sensing applications: wearable-based human activity recognition, seismic and acoustic-based vehicle detection, and EEG-based harmful brain activity recognition. The results show that this method outperforms traditional and generative data augmentations across various downstream models in all evaluated sensing tasks. The novelty of the approach lies in its fine-grained control over generative augmentation, leveraging domain-specific knowledge to produce highly authentic synthetic samples. The significance of this work is highlighted by its ability to enhance IoT sensing model robustness, particularly in scenarios where domain-specific knowledge is available, and its adaptability across different generative models to benefit from their inherent capabilities.", "questions": "Three datasets are mentioned in the paper. There are case studies reported related to HAR and Vehicle Detection. No case study related to Brain Activity? A related dataset is mentioned.\n\nUnexplained trends: Regarding Figure 7. Performance increases from 50% to 500% augmentation but decreases slightly at 750% before stabilizing or slightly increasing at 1000%. Why is that the case? It is difficult to ascertain whether this drop is statistically significant or merely a result of random fluctuations in the data.\n\nChecklist Q5: how can the reviewers have confidence in the reported results if the code or data is not released — please argue to justify.", "limitations": "Statistical Significance and Robustness: The inclusion of statistical significance tests, such as p-values or confidence intervals, for the performance differences reported in Table 2 is currently lacking. This omission weakens the robustness of the claims.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Originality: The paper introduces a novel data augmentation paradigm for IoT sensing with fine-grained control over generative models using a metric space defined by statistical metrics. This innovative approach departs from traditional methods and is applied across various generative models and IoT tasks, showcasing its originality.\n\nQuality:  The research is supported by extensive experiments across human activity recognition, vehicle detection, and brain activity monitoring. The proposed method outperforms traditional and generative augmentation techniques, demonstrating high reliability and robustness in its findings.\n\nClarity: The paper is well-structured, clearly explaining the problem, methodology, and results. Despite minor typographical errors, the detailed explanations and clear presentation of results make the findings accessible and easy to understand.\n\nSignificance: The work significantly enhances IoT sensing model robustness by addressing data distribution shifts between training datasets and real-world scenarios. Its fine-grained control and adaptability across generative models have broad applications, advancing state-of-the-art data augmentation techniques in IoT sensing.", "weaknesses": "Table 2 (Section 4.5) \nStatistical Significance: Including statistical significance tests (e.g., p-values) for the performance differences would strengthen the claims and provide more confidence in the results. You could provide Confidence Intervals for the performance metrics at each augmentation level. \n\nTypographical and Formatting Issues: There are minor typographical errors and formatting issues, such as lack of space before citation (line 202) and inconsistent capitalization in references. Ensuring meticulous proofreading and adherence to formatting guidelines would improve the paper's presentation and readability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for all the comments! We would like to add the following clarifications to address your concerns.\n\n> W1: Table 2 (Section 4.5) Statistical Significance: Including statistical significance tests (e.g., p-values) for the performance differences would strengthen the claims and provide more confidence in the results. You could provide Confidence Intervals for the performance metrics at each augmentation level.\n\n**Response**: Thanks for your constructive suggestion. We add 95% confidence interval to the results in Table 2. The updated table is presented in the **uploaded pdf file** as Table 3.\n\n\n> W2: Typographical and Formatting Issues: There are minor typographical errors and formatting issues, such as lack of space before citation (line 202) and inconsistent capitalization in references. Ensuring meticulous proofreading and adherence to formatting guidelines would improve the paper's presentation and readability.\n\n**Response**: Thank you for pointing out the issues. We definitely will fix them and proofread the draft more carefully to ensure readability.\n\n> Q1: Three datasets are mentioned in the paper. There are case studies reported related to HAR and Vehicle Detection. No case study related to Brain Activity? A related dataset is mentioned.\n\n**Response**: We add the case study for the Brain Activity data as follows: we observe that the seizure signals from different patients can be divided into two categories regarding the fundamental frequency patterns. One is with clear fundamental frequency, which might be associated with the repetitive firing patterns of neurons in a specific brain region. We call it ***Clear Seizure***. The other category has chaotic, noisy signals and blury fundamental frequency, which might be related to different brain regions contribute varying frequencies and patterns simultaneously. We call it ***Chaotic Seizure***. In this case study, we manually select 10 patients of clear seizure as the traing set, and select 10 patients of chaotic seizure as the validation set. To augment the training data, we set the varying range of ***contrast***, ***entropy***, and ***fundamental frequency amplitude*** at [10\\%, 25\\%], [10\\%, 40\\%], and [-25\\%, -10\\%] respectively, while keeping the other metrics unchanged. We show example spectrograms in Figure 13.a in the **uploaded pdf file**. The experiment results are shown in Figure 13.b. Among all the baselines, our domain knowledge instruction (noted as ***knowledge***) achieves the best performance, again proving the effectiveness and generalizability of our approach.\n\n> Q2: Unexplained trends: Regarding Figure 7. Performance increases from 50% to 500% augmentation but decreases slightly at 750% before stabilizing or slightly increasing at 1000%. Why is that the case? It is difficult to ascertain whether this drop is statistically significant or merely a result of random fluctuations in the data.\n\n**Response**: We envision that the minor fluctuations in performance, both increases and decreases, may be attributed to the sensitivity of the downstream model to outliers within the augmented data samples. As indicated in Figure 7, these variations are more pronounced in the human activity recognition dataset (Figure 7.a) and the vehicle detection dataset (Figure 7.b). Given the smaller sizes of these datasets (4,959 and 8,015 samples respectively), there is a smaller number of augmented samples available, which could lead the downstream model to overfit on outliers in the augmented data, resulting in decreased performance. Conversely, as depicted in Figure 7.c, the harmful brain activity dataset, with its substantially larger volume of training data (78,548 samples), demonstrates a smoother and more stable trend in its curve. This suggests that given the same augmentation ratio, larger datasets can lead to more consistent augmentation effects, as observed.\n\n> Q3: Checklist Q5: how can the reviewers have confidence in the reported results if the code or data is not released — please argue to justify.\n\n**Response**: Thank you for bringing up this concern! As stated in the footnote on page 5 of our draft paper, we promise to release the code and datasets after paper acceptance.\n\n> L1: Statistical Significance and Robustness: The inclusion of statistical significance tests, such as p-values or confidence intervals, for the performance differences reported in Table 2 is currently lacking. This omission weakens the robustness of the claims\n\n**Response**: Thanks again for the constructive suggestion. We address this limitation in our response in W1.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "o3S6yLO1Oc", "reviewer": "Reviewer_eQAt", "summary": "This work introduces fine-grained control to generative models for IoT data augmentation by defining a metric space with statistical metrics. \n\nThese metrics are essential for capturing the key features of the spectrograms and serve as strong constraints for the generative model, allowing for customization of spectrogram characteristics in the time-frequency domain tailored to specific application requirements. \n\nThe evaluation performs better than traditional transformation-based data augmentation techniques and previous generative data augmentation models.", "questions": "1. What kind of outputs will be if we choose different weights between metric loss and generative loss? It seems this is not an easy job to adjust every weight corresponding to each metric according to the loss during the model training process.\n\n2. Why do not authors choose other metrics? Are the selected metrics general to all IoT tasks? What if some data types are not sensitive to the proposed metrics?\n\n3. How large the dataset is required to train the diffusion model for the generation task?\n\n4. Is it possible the interpolation, and disturbance in the metric space change the metric to respond to a different label, which conflicts with the original label? If so, can authors provide such low-quality generation examples?\n\n5. Will there be a domain shift between generated samples and raw samples?\n\n6. Could authors provide the magnitude of the spectrogram?", "limitations": "The authors have discussed the limitations in metric selection and the introduction of domain knowledge.\n\nThe authors have kindly answered my questions and I appreciate their efforts for the research community.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. A novel constraint design for generative models by defining a metric space with statistical metrics.\n\n2. An interesting metric space augmentation method is proposed.", "weaknesses": "1. How to balance the original generative model loss and the designed metric loss is not well discussed. As mentioned by the authors, the two loss terms are ratioed by 10:1. It seems the metric loss contributes marginally during the generation process. Considering the generation process is a probabilistic process, without deep analysis, it is hard to convince readers that high-quality outputs are random or consistent. \n\n2. The metrics selection is not well-founded. Though the authors have discussed its limitation in the metric selection, the metrics are data-type and task-oriented. The inappropriate selection of the metrics could introduce extra noise. Which or how these metrics contribute to the better generative process should be further analyzed. Overall, the contribution is limited as in-depth analysis is missing, and complex hyperparameter (e.g., metric selection, weight selection) tunning is involved.\n\n3. The evaluation setting is trivial, which can hardly demonstrate the efficacy of high-quality data generation. For example, the HAR dataset has only 5 simple activities, which is not a data-hungry task. The authors are encouraged to check more complex real-world datasets, considering the data and label heterogeneity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **Author Reply to Reviewer eQAt (Part 1/3)**\n\nThank you for your approvals, critiques and insightful questions! We would like to address your concerns and provide extra evaluation results as follows.\n\n> W1: How to balance the original generative model loss and the designed metric loss is not well discussed. As mentioned by the authors, the two loss terms are ratioed by 10:1. It seems the metric loss contributes marginally during the generation process. Considering the generation process is a probabilistic process, without deep analysis, it is hard to convince readers that high-quality outputs are random or consistent.\n\n**Response**: We set the ratio of the two loss terms to 10:1 by default, intending to prioritize the optimization of the generative model itself using $L_G$ in the early stages of training, and then gradually shift focus to enforcing control over the generation using $L_{metric}$ in the later stages. In Figure 12 in the ***uploaded pdf file***, we plot the curves of the two loss terms when training on the HAR dataset with ratios of 10:1 and 5:1, respectively. We observe that with a 10:1 ratio, $L_G$ decreases rapidly and eventually matches the scale of $L_{metric}$ after a few hundred epochs. Thus, the 10:1 ratio does not make the contribution of $L_{metric}$ trivial. We also notice that decreasing this ratio delays the intersection of the curves of the two loss terms, causing $L_{metric}$ to converge at a slightly higher value. Therefore, we fix the ratio at 10:1 in our implementation.\n\nWe further profile the influence of weight ratio $L_G:L_{metric}$ on the performance as below. Due to time limit, we only show results for Human Activity Recognition dataset using Diff-disturb as the data augmentation method.\n\n*DeepSense:*\n|          | 1:1 | 5:1 | 10:1 | 20:1 |\n|----------|-----|-----|------|------|\n| Accuracy |  0.8211 | 0.8307 |**0.8324**| 0.8249  |\n| F1-score |  0.8209 | 0.8333 |**0.8422**| 0.8360  |\n\n*Transformer:*\n|          | 1:1 | 5:1 | 10:1 | 20:1 |\n|----------|-----|-----|------|------|\n| Accuracy | 0.8199  | 0.8105  |  0.8205 | **0.8280**   |\n| F1-score | 0.8239  | 0.8195  |  **0.8352** | 0.8333   |\n\nAs shown in the results above, when $L_G / L_{metric}$ is between 1~20, the performance generally stays stable. These findings indicate the robustness of the loss term design.\n\n> W2: The metrics selection is not well-founded. Though the authors have discussed its limitation in the metric selection, the metrics are data-type and task-oriented. The inappropriate selection of the metrics could introduce extra noise. Which or how these metrics contribute to the better generative process should be further analyzed. Overall, the contribution is limited as in-depth analysis is missing, and complex hyperparameter (e.g., metric selection, weight selection) tunning is involved.\n\n**Response**: Before the widespread adoption of advanced deep learning models, researchers often used statistical metrics as features for conducting IoT sensing tasks [1][2][3][4]. The metrics selected in our paper have been widely adopted in previous works and are generally applicable across different modalities, sampling rates, and physical phenomena, making them versatile for IoT sensing tasks. While we acknowledge that inappropriate metric selection can introduce noise, our choice was guided by their proven utility in existing IoT sensing literature and our own evaluation across three different IoT tasks.\n\nNevertheless, we understand the need for a more in-depth analysis of how these metrics contribute to the generative process. As part of our future work, we plan to explore this further. This exploration will aim to refine the framework and potentially reduce the complexity, ensuring a more streamlined and efficient application in diverse IoT scenarios.\n\n[1] Detection of posture and motion by accelerometry: a validation study in ambulatory monitoring\n\n[2] Activity Recognition from Accelerometer Data\n\n[3] Vibration-based Terrain Classification Using Support Vector Machines\n\n[4] Epileptic activity recognition in EEG recording\n\n\n> W3: The evaluation setting is trivial, which can hardly demonstrate the efficacy of high-quality data generation. For example, the HAR dataset has only 5 simple activities, which is not a data-hungry task. The authors are encouraged to check more complex real-world datasets, considering the data and label heterogeneity.\n\n**Response**: In IoT sensing, unlike in vision or language-related tasks, data scarcity presents a significant challenge. Obtaining large labeled datasets is often difficult and costly. Even though the HAR dataset features only five activities, the variance among individuals' behaviors introduces substantial variability. This introduces the potential domain shift between the training and validation sets, making the task data-hungry and challenging. Our proposed method can effectively generalize from small datasets and enhance downstream tasks demonstrates its high data efficiency. This capability shows the practical value of our method, particularly in scenarios where labeled data is limited, showcasing its potential for real-world IoT sensing applications.\n\n\n> Q1: What kind of outputs will be if we choose different weights between metric loss and generative loss? It seems this is not an easy job to adjust every weight corresponding to each metric according to the loss during the model training process.\n\n**Response**: Thank you for the question. As discussed in our response to **W1**, different weights between the two loss terms $L_G$ and $L_{metric}$ can slightly influence the convergence speed of each loss. However, our profiling results indicate that the performance generally remains stable within the tested weight ratio range.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
## General Responses

We greatly appreciate the detailed feedback provided by the reviewers. In response to your comments and suggestions, we have made the following major modifications:

### Reviewer 7DMe
- **Clarification of Motivation**:
  - Clarified our motivation by elaborating on the unique nature of IoT sensing and the necessity of fine-grained control in data augmentation.

### Reviewer 8SZC
- **Terminology and Conceptual Clarifications**:
  - Changed the term "metric" to "(statistical) indicator" to avoid confusion with performance metrics like accuracy and F1-score.
  - Clarified that the classifier used is only the encoder of a Transformer, avoiding any ambiguity.
  
- **Evaluation Improvements**:
  - Introduced a custom FID (Fréchet Inception Distance) metric using our own dataset and DeepSense model to evaluate the quality of generated spectrograms.
  - Provided results of adding Gaussian noise as a baseline to demonstrate the effectiveness of our generative model.
  - Conducted hyperparameter sensitivity analysis for the weight ratio between the generative model loss and metric loss.
  - Conducted further experiments with domain-separated training and validation sets for vehicle detection and harmful brain activity recognition datasets.
  - Added a new regression task for vehicle distance prediction to validate the generalizability of our approach to regression problems.

- **Discussion of Related Work**:
  - Compared our work with related works such as DiffWave and DiffAR, highlighting differences in motivation, modality, conditions, and contributions.
  - Discussed efficiency and computational requirements as a limitation in the revised paper.


### Reviewer eCvh
- **Statistical Significance and Robustness**:
  - Included 95% confidence intervals for performance metrics in our evaluations.
  
- **Typographical and Formatting Corrections**:
  - Addressed minor typographical errors and formatting issues to improve readability.

- **Additional Case Study**:
  - Added the case study for the Brain Activity data, explaining different seizure signal types and their augmentation.

- **Analysis of Augmentation Ratio Influence**
  - Incorporated analysis of performance trends under different augmentation ratios.

- **Code and Data Release**
  - Confirmed that the release of code and data will be after the acceptance of the paper.

### Reviewer eQAt
- **Analysis on Loss Terms Weights**
  - Added new evaluations on the influence of different loss term weights.
  - Provided insights on the loss term weights fine-tuning. 

- **Additional Analysis and Low-Quality Generation Examples**:
  - Provided examples of low-quality generation to illustrate the importance of appropriate metric selection.
  - Analyzed the potential domain shift between generated and raw samples, emphasizing the importance of diversity in training data.

- **Metrics Selection and Generalization**:
  - Provided analysis on the selection of metrics and their general applicability to various IoT tasks.

- **Dataset Size and Magnitude**:
  - Listed the number of data samples in each dataset, demonstrating the effectiveness of our approach even with relatively small datasets.
  - Showed the statistics regarding the magnitude of spectrograms of different modalities in each dataset.

***We have uploaded the pdf file with all the newly added figures (Figures 12 - 14) and table (Table 3).***

Meta-review: The paper presents a data augmentation paradigm for IoT sensing by introducing a fine-grained controllable generative model using a metric space defined by statistical metrics, which enhances model robustness across various tasks. Supported by experiments in various IoT applications, the method outperforms traditional techniques, demonstrating its effectiveness and adaptability. The discussion mainly focused on various technical details in the work. After the discussion, all the reviewers recommended acceptance.
Decision: Accept (poster)