Fine-grained Control of Generative Data Augmentation in IoT Sensing

Authors: tianshi  wang
qikai  yang
ruijie  wang
dachun  sun
jinyang  li
yizhuo  chen
yigong  hu
chaoqi  yang
tomoyoshi  kimura
denizhan  kara
tarek  abdelzaher
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-12 03:45:05

Abstract: Internet of Things (IoT) sensing models often suffer from overfitting due to data distribution shifts between training dataset and real-world scenarios. To address this, data augmentation techniques have been adopted to enhance model robustness by bolstering the diversity of synthetic samples within a defined vicinity of existing samples. This paper introduces a novel paradigm of data augmentation for IoT sensing signals by adding fine-grained control to generative models. We define a metric space with statistical metrics that capture the essential features of the short-time Fourier transformed (STFT) spectrograms of IoT sensing signals. These metrics serve as strong conditions for a generative model, enabling us to tailor the spectrogram characteristics in the time-frequency domain according to specific application needs. Furthermore, we propose a set of data augmentation techniques within this metric space to create new data samples. Our method is evaluated across various generative models, datasets, and downstream IoT sensing models. The results demonstrate that our approach surpasses the conventional transformation-based data augmentation techniques and prior generative data augmentation models.

Main paper: 
 introduction: 
 IoT sensing applications aim to detect physical world phenomena in specific environments through time-series data captured by sensors, such as inertial, acoustic, and bioelectrical signals. Given the wide variety of real-world conditions, IoT sensing models often face potential domain shifts and unpredictable variations when deployed [23] . Additionally, collecting and labeling IoT sensing data is expensive because sensing signals are harder for annotators to interpret compared to images, videos, and natural language [32, 34, 21, 22] . Consequently, enhancing model robustness across diverse scenarios with minimal collection overhead becomes a key research focus in the IoT field [55, 56] .A spectrum of data augmentation techniques has been proposed to reduce the need for extensive data collection. The objective is to diversify existing datasets while preserving a plausible data distribution. Conventional data augmentation methods for IoT sensing signals generally apply manually-crafted transformations to create different perspectives of the same sample [58, 12] . Typical transformations include operations in the time domain (such as jittering, rescaling, rotation, and cropping) [12, 29, 39, 50] and the frequency domain (such as spectral flipping, warping, and masking) [58, 11, 38] . They are designed to modify the original data to reflect possible real-world variations, guided by domain knowledge in IoT. However, those transformations can be overly simplistic, failing to capture the In this paper, we introduce a novel data augmentation paradigm for IoT sensing data that combines domain knowledge control with generative data augmentation methods to advance the synthesis capabilities. We argue that a generative model with fine-grained control over the human-interpretable characteristics of IoT sensing signals can enhance the data augmentation effectiveness on downstream tasks. Our approach begins by leveraging domain expertise in IoT sensing to extract a set of statistical metrics in the time domain, frequency domain, and time-frequency domain, which represent the essential aspects of IoT sensing signals. We call the resultant space determined by those statistical metrics as metric space. Subsequently, we develop a suite of data augmentation techniques designed to operate within the metric space.In Figure 1 , we illustrate an application example of our method. Given real data of moderate running in human activity recognition task, we speculate the potential real-world variations of running behaviors and leverage domain knowledge to derive the changes of statistical metrics accordingly. These metrics serve as conditions for a generative neural network to guide the synthetic data generation of other running styles like energetic, cautious and on varied terrain. The generated data is then utilized to augment the original dataset, thereby improving the robustness of downstream IoT sensing model. The strength of our paradigm comes from the precise control over the generation process via the metric space, while ensuring the authenticity of the generated samples through the generative model.Empirically, we validate the versatility of our augmentation approach with various generative model backbones including a diffusion model and a VAE. We test our approach in three distinct IoT sensing applications: wearable-based human activity recognition, seismic and acoustic-based vehicle detection, and EEG-based harmful brain activity recognition. Our results demonstrate that our method outperforms conventional and generative data augmentations across various downstream models in all the evaluated sensing tasks. 
related work: 
 Traditional Data Augmentation for IoT Sensing. Many traditional data augmentation methods rely on manually crafted transformations to enhance dataset diversity. As IoT sensing signals typically involve temporal dynamics while exhibiting periodical patterns, transformations are designed in both time domain [58, 12, 29, 39, 50] and frequency domain [58, 11, 38] . Implementing these augmentation techniques requires specific domain knowledge to determine the variation range that can enhance data diversity without compromising relevance to the intended task. Another data augmentation approach involves similarity interpolation, where new samples are created by interpolating between existing samples, maintaining task invariance in the generated outputs. Data space interpolation techniques, which interpolate directly between raw data samples, have been proposed [5, 63, 37] . Additionally, there has been research on feature space interpolation, which occurs after data has been embedded into latent representations [10, 3, 35] . A primary challenge for this line of work lies in the synthetic data quality. The transformations often represent straightforward, linear variations and may introduce artifacts that reduce the realism of the generated data.Generative Data Augmentation for IoT Sensing. Generative data augmentation utilizes generative models to create varied data [48, 60, 64] . These models, including GANs [49, 31, 6] , VAEs [36, 14, 53] , and diffusion models [43, 44, 57] , generate authentic samples that improve the performance of downstream models and significantly lower the costs associated with collecting real data. However, these prior work treat the generative model as a black box, using it to sample from the learned data distribution with minimal control, typically only conditioned by the inherent labels from the dataset. The lost of control makes the generated results unpredictable, and loses the chance to benefit from domain insights and prior experience. This work is inspired by prior research on conditional generation for human speech and singing signals [25, 4, 16, 33] . In these studies, fine-grained conditions in vocal and/or linguistic features are applied to control acoustic signal generation. However, unlike human speech, IoT sensing time series data carries entirely different semantics and encompasses a broader range of modalities. Condition space interpolation involves the application of interpolation techniques within a condition space, which is constituted by statistical metrics [54] . These augmented metrics are then fed into a generative model to synthesize varied data. Our work significantly advances prior research by redesigning the metric space and enhancing data augmentation techniques within it. We also perform extensive evaluations across diverse generative models and IoT sensing tasks, resulting in a more general data augmentation framework. As shown in Figure 2 , our fine-grained controllable generative data augmentation has three stages.In Stage 1, we extract the predefined spectrogram metrics that reflect the critical characteristics of IoT sensing spectrograms. The selection of these metrics is based on domain knowledge in general IoT sensing, capturing essential facets in the IoT signal semantics. These metrics together define the metric space that conditions the subsequent generative model.In Stage 2, the metric values, along with the class label, are embedded into a condition tensor. This tensor then conditions the training of the generative model, guiding it to synthesize spectrograms that adhere to the specified metric values.In Stage 3, we utilize the fine-grained controllable generative model to augment the original dataset within the metric space. The metrics in this space act as control knobs, allowing for the directional generation of IoT sensing signals with specific characteristics in the spectrogram semantics. This enables the application of domain knowledge about IoT sensing and downstream tasks to strategically augment the original dataset. We introduce several data augmentation techniques that vary in their reliance on domain knowledge. The augmented conditions (metric values) are input into the trained generative model to produce synthetic samples. These samples, combined with the original dataset, form the augmented dataset used to train the downstream IoT sensing models.We introduce each stage in details in Section 3.2, 3.3, and 3.4. 
construction of metric space: 
 We aim to design a set of statistical metrics which can effectively serve as fine-grained control knobs for data augmentation and easy to be manipulated by the domain experts in IoT sensing.As IoT sensing signals typically exhibit strong patterns in both time and frequency domain, we convert the original time-series data into 2D spectrograms using short-time Fourier transformation (STFT).We propose metrics as below in three domains: time domain, frequency domain, and time-frequency domain.Time domain metric: Temporal amplitude oscillation commonly exist in IoT sensing signals, which is a strong indicator for the dynamics of the observed physical phenomena. To measure this essential characteristic, we propose using the temporal amplitude range in a spectrogram as a metric for controlling the volatility of the generated signal. It straightforwardly informs on the operational limits of the IoT sensing signal, which is essential for a robust generation that must operate within specific amplitude constraints.Frequency domain metric: We introduce Fundamental frequency (F0) amplitude as the metric to reflect the frequency domain characteristics of IoT sensing spectrograms. The fundamental frequency is pervasive across various IoT sensing signals. It dictates the dominant periodicity of the signal and is thus essential for capturing the signal's primary oscillatory properties. For example, in a signal captured by the accelerometer of a wearable device on the wrist when the human subject is walking, the fundamental frequency corresponds to the walking cycle.Time-frequency domain metrics: The information distribution across the time-frequency domain is strongly correlated to phenomena such as environmental noises, signal interference, and the harmonic characteristics of the signal source. To effectively capture these aspects, we introduce metrics including contrast, flatness, and entropy that quantify the amplitude distribution and information density within the spectrogram.Contrast quantifies the difference in amplitude between peaks and valleys within the spectrogram. This metric is critical for identifying the dominant signal patterns, allowing for a more nuanced generation of spectrograms where both signal and noise components are realistically balanced.Flatness measures how noise-like a spectrogram is by comparing the geometric mean to the arithmetic mean of the power spectrogram values. A high flatness indicates a uniform distribution of power across the spectrogram, typical of stochastic processes and ambient environmental noises. A low flatness indicates concentration of energy at regions. This results in a spectrogram with interleaving of peaks and valleys, reflecting the presence of tones or harmonics.Entropy measures the randomness of the power distribution within the spectrogram. High entropy suggests a complex, less predictable signal, while low entropy indicates a more structured and tonal signal. This metric is crucial for generating data that mimics the variable information content found in real-world IoT signals, from highly predictable periodic signals to complex, non-periodic disturbances.The formal definitions of the metrics above can be found in Appendix B.Given the original dataset, we extract the metric values vector m from each data sample and embed it with the class label as the condition tensor c. During the training of a conditional generative model G, it takes the original data sample x and the condition tensor c as inputs. To enforce the alignment of the generated data x ′ with the characteristics defined by the given metrics m, a loss term L metric is taken as a penalty on the difference between the metric values m calculated from x ′ and m:EQUATION)where M is the total number of metrics, and w i is a hyperparameter as the weight for a specific metric loss. The total loss L consists of the original generative model loss L G and the metric loss:EQUATIONIn practice, to balance the two loss terms, we empirically adjust the weight w of each metric so that L G : L metric approximately equals to 10:1 at the end of the first training epoch.To fully leverage the fine-grained controllable generative model to synthesize diverse data samples, effective data augmentation techniques are required. Different from prior approaches, our data augmentation happens in the metric space where the extracted metric values from the original data are manipulated to create variant data. As shown in Figure 3 , we propose three data augmentation techniques which require varying levels of IoT domain knowledge, ranging from low to high.Figure 3 : Data Augmentation Techniques in Metric Space. In the interpolation method, the green dotted line represents the intermediate value range between two real samples. In the disturbance method, blue circles indicate the range of disturbance. In the domain knowledge instruction method, the red outline denotes specific metric value ranges derived from domain knowledge.Interpolation: A vast amount of prior research has demonstrated that interpolation is an effective data augmentation approach [5, 63, 37, 10, 3] . The intuition is that if the model can correctly classify not only the original samples but also these in-between points, it is likely to perform better on unseen data that fall within the same general manifold. Given a pair of randomly selected samples under the same label x i and x j , we extract their metric values m i and m j . The interpolated metric value m interp is defined as:EQUATIONwhere β is a random number between 0 and 1, independently sampled from a Gaussian distribution for each interpolation. One advantage of interpolation is that it does not require specific domain knowledge to create new data samples.Disturbance: Inheriting from the design of conventional transformation-based data augmentation methods, we disturb the original metric value of a data sample within a predefined varying range [a, b], where a and b are the minimum and maximum percentages of the original metric value. During the metric disturbance, a random number r is sampled from the uniform distribution U (a, b). Given all the metrics as control knobs for creating new samples, users of this augmentation technique can to define the valid disturbance range grounded by domain insights.For many IoT sensing applications, deeper insights are available about the downstream model and the potential diversity of the realistic data distribution. In contrast to disturbance, these domain knowledge instructions provide a more precise scope for metric manipulation. This accuracy allows for the synthesis of data that not only enhances the downstream model but also avoids the production of unnecessary variants. To showcase the benefits of incorporating domain knowledge in IoT sensing data augmentation, we present two case studies in Section 4.3. 
experiments: 
 We evaluate our data augmentation approach on three IoT sensing applications where domain shift issues are prevalent: human-activity recognition, vehicle detection, and harmful brain activity recognition.Datasets: (1) Human-Activity Recognition (HAR): We use RealWorld-HAR dataset [47] , which aims at classifying 5 human activities based on the accelerometer and gyroscope signals.(2) Vehicle detection: We detect 5 vehicle types by deploying a seismic and an acoustic sensor on the ground, monitoring the sound and vibration caused by passing vehicles. This dataset is collected by ourselves [30] .(3) Harmful brain activity recognition: We adopt Harmful Brain Activity Classification [18, 19] dataset which comprises electroencephalography (EEG) signals from critically ill patients in hospitals. The objective is to classify 5 types of harmful brain activities.Generative Model Backbones: Technically, our fine-grained controllable generative data augmentation approach is compatible with any conditional generative model. We utilize two types of generative model backbones to validate our approach: (1) Diffusion Model [42] (referred to as Diff hereafter), and 2) Transformer (Encoder-Only) [52] , which uses selfattention mechanisms to process sequential data and can capture complex dependencies. We only use the encoder component of the original Transformer architecture for classification tasks.Data Augmentation Setup: Each method produce 5 times of synthetic samples comparative the number of the original training samples. For our methods that utilize disturbance as the augmentation technique, we set the adjustable range for each metric in between -20% to 20%. This range is reasonably chosen to encompass substantial variance while maintaining invariance critical for the downstream task.More details about the experimental setup can be found in the Appendix C, D, E, and F.Table 1 reports the performance of the 3 variants of our fine-grained controllable generative augmentation comparing with the baselines. Interp uses interpolation for augmentation, Disturb applies metric disturbance, and Reconst generates augmented data by using the actual metric values from the validation set as conditions, showcasing the potential upper bound performance of our method when validation data metrics are perfectly predicted. We highlight the highest-performing cell in the table, excluding the Reconst variant.Across all datasets and two downstream models, our fine-grained controllable generative augmentation consistently outperforms other methods. Typically, the performance improves with the integration of more domain knowledge into the augmentation process. Interp operates on the premise that interpolating between real samples enhances dataset diversity, requiring no specific domain knowledge. Disturb, on the other hand, involves domain expertise to define the disturbance range, leading to better performance than Interp. Reconst, which utilizes ground truth metric values from the validation set, achieves the highest performance, illustrating the significant benefits of incorporating domain knowledge about IoT sensing signal characteristics through our designed metrics into the generative augmentation process.Using the same augmentation technique, the diffusion model generally surpasses the VAE as a backbone. As illustrated in Appendix G.1, VAE generates spectrograms that are less crisp and realistic compared to those produced by the diffusion model, a finding supported by numerous previous studies [28, 2] . We hypothesize that the inferior generative quality of the VAE limits the effectiveness of data augmentation. This suggests that the success of our data augmentation method depends significantly on the underlying generative model's ability to synthesize authentic data.To validate the fine-grained control over the generation process, we visualize examples of the generation results in Figure 4 . Results demonstrate the precise fine-grained control of our approach over the IoT sensing signal generation.We highlight the effectiveness of integrating domain knowledge to enhance data augmentation through three case studies. In each case, we create a domain shifting scenario and aim to increase the downstream model's robustness to variations outside the training set. We utilize the diffusion model as the generative model backbone and DeepSense as the downstream model.Case study 1: Various Running Styles in HAR dataset. We use the RealWorld-HAR dataset for our analysis. For the training set, we select 3 human subjects who demonstrated cautious style during the data collection of the "running" activity. For the validation set, we choose other 3 human subjects who exhibited energetic running patterns. The distinct running patterns of the subjects were assessed by reviewing video footage recorded by an observer holding a camera and following the runners. 2 Based on IoT sensing domain knowledge, we hypothesize that the more forceful arm swings of energetic runners produce more intense and varied acceleration signals. The dominant arm swing frequency and its harmonics are also more pronounced. In spectrograms, these characteristics should manifest as higher contrast, lower flatness, and stronger F0 amplitude compared to the cautious runners. As shown in Figure 5a , our visual inspections of accelerometer spectrograms from each participant confirmed these assumptions.We then simulate a scenario where we only have the data from the 3 cautious runners, and need to generalize the activity recognition to the running activity of the energetic runners. In our training set, we have the data of all activities from the cautious runners, while the validation data comprises running activities from the energetic runners. Leveraging our domain insights, we directionally augment the data by intensifying the contrast and fundamental frequency amplitude and reducing the flatness in the spectrograms of the cautious runners. We set the varying range of contrast, flatness, and fundamental frequency amplitude at [10%, 40%], [-40%, -10%], and [10%, 40%] respectively, while keeping the other metrics unchanged. Having these ranges, we employ the perturbation augmentation method to create the augmented data. In Figure 5b , results show that our domain knowledge instruction (noted as knowledge) substantially improves the results, closely approaching the upper performance bound set by Reconst. This demonstrates that the generated data can more precisely reflect the true characteristics of the diverse running styles. Case Study 2: Various Road Types in Vehicle Detection Dataset. In the vehicle detection dataset, the same vehicle driving on different road types exhibit various signal patterns. Our domain expertise suggests that the uneven and shifting surface of a gravel road generates sound and vibration profile with higher entropy compared to smoother concrete surface. As presented in Figure 6a , for the same vehicle type, spectrograms exhibit more complex and disordered patterns when on gravel than on concrete.In this case study, we simulate a scenario where the training set only includes data collected on the concrete road while the validation set is the data collected on the gravel ground. By applying our knowledge about the effect of gravel surfaces, we generate synthetic data by increasing entropy and flatness and decreasing contrast. We set the varying range of these 3 metrics as [10%, 30%], [10%, 30%], and [-30%, 0] respectively, while keeping the other metrics unchanged. Then we use perturbation augmentation to synthesize the data. In Figure 6b , the results show that our data augmentation informed by precise domain knowledge, significantly enhances the generalizability of the downstream model from concrete road to gravel road. We investigate the influence of the number of augmentation data on the downstream model performance. Here we use diffusion model as the generative model backbone, and DeepSense as the downstream model. As depicted in Figure 8 , even though the ideal augmentation ratio varies in different datasets, a steep climb in performance occurs from 50% to 500%. Among the 3 datasets, human activity recognition has the smallest data volume, which is below 5000 training samples. We envision that the generative capability is fully expressed at a lower augmentation ratio. Harmful brain activity recognition dataset has the largest number of training samples (≈ 80,000), where a tendency of performance growing remains even at a higher augmentation ratio. To investigate the impact of different metrics on our data augmentation approach, we ablate each category of metrics and evaluate their effects using the disturb augmentation method with diffusion model as the generative model backbone. Performance was evaluated using DeepSense as the downstream model. As depicted in Table 2 , excluding any metrics consistently results in diminished performance, validating the effectiveness of our selected metrics. The results also indicate varying importance of metrics across different datasets. 
conclusion: 
 In this paper, we present a method of fine-grained control on generative augmentation in IoT sensing applications. Our novel data augmentation technique merges the advantages of leveraging domain knowledge specific with the production of highly authentic synthetic samples. Compared to traditional transformation-based and generative data augmentation methods, our approach demonstrates superior performance, particularly when domain-specific knowledge is available, which is a common scenario in many IoT sensing tasks. Furthermore, the versatility of our method enables its application across different generative models, allowing it to benefit from the inherent generative capabilities of these models. 
 
 Appendix: Even after we obtained the fine-grained control over the generation process, deciding the proper augmentation strategy still requires deliberate considerations. The introduction of domain knowledge is not straightforward and involves a nuanced understanding of the underlying mechanisms of the target domain. Currently, this process relies heavily on the expertise of practitioners who understand both the technical aspects of generative models and the specific characteristics of the domain. This is an issue that pervasively exist in all the domain knowledge-guided neural network design [8, 27] . We believe this is highly challenging and we look forward to make advancements in future research.We formally define the statistical metrics we proposed in constructing the metric space.Let S ∈ R f ×t represent the STFT spectrogram, where f and t are the dimensions of frequency and time, respectively.The temporal amplitude range measures the variability in signal amplitude across time in a spectrogram after averaging over the frequency dimension. It is defined mathematically as:EQUATIONwhere A(t) = 1 f f i=1 S(i, t) represents the average amplitude at time t over all frequency bins f in the spectrogram S. This metric captures the total temporal variation in signal strength and provides insights into the operational limits of the IoT sensing signal.We define the fundamental frequency amplitude as the mean amplitude at the fundamental frequency across all time bins. To calculate that, we first estimate the fundamental frequency f 0 for each time frame in the spectrogram using the Harmonic Product Spectrum (HPS) method. This process involves the following 3 steps:Step 1: Compute the HPS for Each Time Frame For each time frame t in the spectrogram S(f, t), the HPS is computed by: 1. Generating downsampled versions of the spectrum for the given time frame, where the downsampling occurs on the frequency dimension only:EQUATIONwhere f ↓ d denotes downsampling the frequency index f by a factor of d, and D is the maximum downsampling factor. This approach reduces the frequency resolution while maintaining the temporal resolution intact.In practice, we set D for various datasets to accommodate different sampling rates and harmonic patterns of the modalities. Specifically, for the HAR dataset, D = 2 is used for both accelerometer and gyroscope signals. In the vehicle detection dataset, D = 3 is set for acoustic signals and D = 2 for seismic signals. For harmful brain activity recognition, D = 2 is applied to EEG signals.2. Multiplying these downsampled spectra together to emphasize the fundamental frequency:EQUATIONStep 2: Identify f 0 for Each Time FrameThe fundamental frequency f 0 (t) at each time frame t is identified as the frequency corresponding to the maximum value in H(f, t):EQUATIONStep 3: Calculate Average F0 AmplitudeOnce the fundamental frequencies are determined for each time frame, the average amplitude at these frequencies is computed by averaging the amplitudes at f 0 (t) over all time frames:EQUATIONwhere T is the total number of time frames.Contrast calculates the distance between the amplitude peaks and valleys in the spectrogram. We define the sets of peaks and valleys:• Let P be the set of values in S that are in the top 5% of all values in S. P = {S(i, j) | S(i, j) > quantile(S, 0.95)}• Let V be the set of values in S that are in the bottom 5% of all values in S.V = {S(i, j) | S(i, j) < quantile(S, 0.05)}Contrast is defined as:EQUATIONWhere |P | and |V | represent the number of elements in the sets P and V , respectively.We first define the power spectral density, denoted as P (f, t), of the spectrogram S(f, t):EQUATIONFlatness of a spectrogram is defined as:EQUATIONwhere T is the total number of time bins, F is the total number of frequency bins, and T F represents the total number of elements in the spectrogram.To calculate entropy, we first normalize the spectrogram to create a probability distribution:EQUATIONHere, p(f, t) represents the probability of observing the power value at time t and frequency f .The entropy of the spectrogram is then defined as:EQUATIONwhere the sum is taken over all time and frequency bins, and the logarithm is base 2.We describe the details of each dataset in the following.We utilize the Real-World HAR dataset [47] , which consists of synchronized readings from multiple wearable sensors on 15 human subjects. Specifically, we select data from the accelerometer and gyroscope mounted on the upper arm, both of which record at a 100 Hz sampling rate. Each subject performed five dynamic activities for approximately 10 minutes each, except for jumping, which lasted about 1.7 minutes. These activities include climbing down, climbing up, jumping, running, and walking. We divide the dataset randomly by subjects, assigning 10 subjects to the training set and the remaining 5 to the validation set. We segmented the entire recording into 2.5-second segments. This partition results in 4,959 training samples and 2,309 validation samples. Due to the individual physical variations among subjects during the same activity, a domain shift occurs when the validation dataset comprises human subjects not included in the training set.We created our own dataset by deploying 5 IoT sensing nodes, each equipped with a seismic sensor (100 Hz sampling rate) and an acoustic sensor (800 Hz sampling rate), placed on the ground to monitor the sounds and vibrations from passing vehicles. The goal is to develop a model that can classify vehicle types based on this sensor data, with each segment being 5 seconds long. The data was collected across 3 different locations: a city parking lot with a concrete surface, a state park parking lot with a gravel surface, and a country road with a dirt surface. This dataset includes the footage of 5 different vehicle types: a compact-size SUV, a mid-size SUV, a sports car, and a convertible roadster. Each data collection session lasted for 40 minutes, during which a single vehicle type traversed the testing field at speeds between 5 and 30 miles per hour along a random path. The initial 30 minutes of data are used for training, while the final 10 minutes serve as validation. Data segments where the vehicle was more than 65 feet from any sensor were excluded based on GPS distance calculations. This process results in 8,015 training samples and 2,763 validation samples.We adopt Harmful Brain Activity Classification [18, 19] dataset which comprises EEG signals obtained from hospitalized patients who underwent EEG monitoring as a part of their clinical care. The sampling rate of the EEG sensors is 100 Hz. Each original recording, approximately 10 minutes long, is segmented into 40-second intervals. The objective is to classify 5 types of harmful brain activities using these data segments, including seizure (SZ), generalized periodic discharges (GPD), lateralized periodic discharges (LPD), lateralized rhythmic delta activity (LRDA), and generalized rhythmic delta activity (GRDA). The ground truth labeling was done by human experts manually [20] . We allocate 80% of the original EEG recordings to the training set and 20% to the testing set. This partition results in 78,548 training samples and 19,532 validation samples.Our diffusion model utilizes the architecture proposed in [42] , which incorporates the U-Net as the backbone [41] , a design prevalent across many diffusion models. U-Net comprises multiple convolutional layers, pooling, and up-sampling operations, all interconnected through skip connections that maintain spatial hierarchies for detailed segmentation. When a 2D spectrogram gets into the U-Net, it is first progressively downsampled through convolutional and pooling layers to extract features, and then incrementally upsampled using transposed convolutions. Skip connections integrating details from corresponding downscaled layers to the upscaled layers for precise localization. In our implementation, we have 3 layers for downsampling and 3 layers for upsampling. Each downsampling/upsampling rescales both the time and frequency dimension of the spectrogram by 2. We set the intermediate channel dimension as 128 for the HAR and vehicle detection datasets, while 64 for the harmful brain activity recognition dataset. Both the HAR and vehicle detection datasets have 45.5 million parameters, while the harmful brain activity recognition dataset has 11.32 million.Our conditioning embedder begins by taking each metric value m i with a channel dimension d i , reshaping it to d i × 1 × 1. This reshaped tensor is then replicated T times along the third dimension to align with the spectrogram's time dimension. The tensor subsequently passes through a individual linear layer which increases its channel dimension to F , altering its shape to d i × F × T . This process is consistently applied to each metric value and the one-hot embedding of the data label. Following this, all transformed tensors are stacked together on the channel dimension, resulting in a tensor of dimensionsM i d i × F × T, where M represents the total number of metrics and the label. We then compute the average over the first dimension of this stacked tensor, yielding a final dimension of 1 × F × T . This resulting tensor, denoted as c, serves as the condition tensor and is concatenated along the channel dimension with either the input spectrogram x or noise ϵ, which is in shape of D × F × T , where D is the feature channel of the sensor signals. The concatenated tensor is in shape of (D + 1) × F × T , and then fed into the U-Net described above.During the training of the diffusion model, we compute the metric loss using the generated spectrograms. To accelerate the generation process, we employ a diffusion step of 10, which offers a balance between moderate quality and rapid generation. For the data generation phase, however, we utilize a diffusion step of 100 to produce augmented data of higher quality.We train the diffusion model using an Adam optimizer with a learning rate of 0.0001, paired with a cosine annealing learning rate scheduler. The model is trained for 1,000 epochs on the HAR and vehicle detection datasets, and for 200 epochs on the harmful brain activity recognition dataset. The batch size is set at 200 for the HAR and vehicle detection datasets, and at 64 for the harmful brain activity recognition dataset.We handle the multi-modal inputs from HAR (accelerometer + gyroscope) and vehicle detection (acoustic + seismic) by employing two separate U-Nets, each dedicated to processing one modality. Although we anticipate strong correlations between the modalities, no modality fusion occurs within between the two U-Net architectures. We consider integrating modality fusion as a potential direction for future research to enhance the diffusion model's performance with multi-modal IoT sensing data.Our model was trained on a desktop with Intel(R) Core(TM) i9-9960X CPU @ 3.10GHz and 4 Nvidia GeForce RTX 2080 Ti. The training of the diffusion model on a single GPU lasts for around 2 days for HAR and vehicle detection datasets, and around 5 days for harmful brain activity recognition dataset.VAEs have been extensively applied to various IoT tasks in prior research. [45, 26, 59, 62] . To capture the hierarchical features of the spectrogram, our VAE model is developed based on the NVAE [51] . The effectiveness of such hierarchical structure has been tested and verified in multiple prior studies [7, 15, 17] . The original NVAE model constitutes of a 3-layer encoder and a 3-layer decoder, where each layer is a residual cell, in order to model the long-range correlations in data [51] . Unlike the naive VAE model [24] , NVAE designs its reparameterization process as a 3-step dependent sampling where each step is completed right before each layer of the decoder. Our conditional NVAE model remains most of the architecture and settings of the original NVAE with only a few changes. We replace the prior trainable parameters in the first layer of NVAE's decoder with our condition embedder, following the same settings used in our diffusion model. Similarly, we also add a condition embedder at the encoder to concatenate our conditioning embedding to the input data.During the training process, we feed our conditions and ground truth data to the encoder to generate the latent variables of each layer. Then we feed the conditions to the beginning of the decoder and the latent variables to each layer's sampling process respectively, to get the generated data. We calculate loss as the mean squared error (MSE) between the ground truth and generated data, added by a weighted KL divergence loss. The weighting for each dataset is as follows: 1 for the HAR dataset, 10 for the vehicle detection dataset, and 8 for the harmful brain activity recognition dataset.We train the NVAE model using an Adam optimizer with a learning rate of 0.0005, paired with a cosine annealing learning rate scheduler. Our model is trained for 200 epochs on the HAR dataset, 300 epochs on the vehicle detection dataset, and 600 epochs on the harmful brain activity recognition dataset. The bacth size of all datasets is set as 64.The total number of parameters for each dataset is 19 million. We trained our VAE model on the same machine as our diffusion model. The training time of each dataset is 1 hour for the HAR dataset, 2 hours for the vehicle detection dataset, and 16 hours for the harmful brain activity recognition dataset.Data Space Interpolation: We follow the method called mixup proposed in [63] . Mixup conducts interpolation within pairs of examples and their labels. The intuition is that mixup regularizes the neural network to favor simple linear behavior in-between training examples, which reduces undesirable oscillations when predicting outside the training examples.Given a pair of randomly selected data samples (x i , y i ) and (x j , y j ), where x i and x j are data vectors, while y i and y j are one-hot label encodings, the interpolated data sample (x ′ , y ′ ) is constructed by:EQUATIONIn our implementation, we follow the original authors' setup by randomly sampling λ from a beta distribution:EQUATIONFeature Space Interpolation: As suggested by [10] , we first train an autoencoder using an unsupervised approach, followed by a linear interpolation of the intermediate feature maps from two randomly chosen samples of the same activity to generate synthetic feature maps. The architecture of the autoencoder largely follows our VAE design described in Appendix D.2, but only keep the reconstruction loss during the training. These synthetic alongside the realistic feature maps are subsequently utilized to train the downstream models.To augment a data sample, we first randomly apply 2 out of 7 time domain transformations to the data in time series form. Subsequently, we employ short-time Fourier transform to convert the time-series data into 2D spectrograms, and apply 1 out of 2 frequency domain transformations. We adopt the time domain transformations described in [50] , including jittering, scaling, negation, permutation, time warping, magnitude warping, rotation, and cropping. For frequency domain transformations, we incorporate frequency masking [38] and frequency perturbation [11] .This diffusion model has the same design depicted in Appendix D.1, with the distinction that the condition vector comprises only the embedding of the inherent label.During the generation phase, synthetic samples for each class are created by the trained diffusion model using the respective class label as conditions. This approach facilitates a direct comparison to assess the benefits of the fine-grained control and metric space augmentation we propose.Conditional VAE: This method follows the design described in Appendix D.2, but differs in that the condition vector includes only the embedding of the inherent label. This setup is designed to evaluate the impact of label embedding on the generation process within the VAE framework.Conditional GAN: We utilize the deep convolutional generative adversarial network (DCGAN) architecture [40] for this model. To enhance the stability and balance between the generator and discriminator, we implement optimization strategies on the loss terms as suggested by [1] . Similar to the other models, this GAN also uses only the inherent label as a condition for generationDeepSense: [61] This is a deep neural network tailored for handling time series data from the IoT sensing applications. This network operates in the time-frequency domain, transforming the data using the short-time Fourier transform. To process data from multi-modalities, the model includes two separate convolutional layers for each sensor, with the resulting intermediate feature maps being merged subsequently. After that, it employs 2 2D-convolutional layers to extract spatial features from the data. Following this, 1 Gated Recurrent Unit (GRU) layer is used to learn temporal patterns. Next, 2 linear layers are used for refining the feature dimensions and generate the final classification logits.Transformer (Encoder-Only): [52] In the Transformer network, each input spectrogram from different modalities is fed into a standard Transformer encoder layer, incorporating a self-attention layer along with two linear layers. Here, the attention is applied to along the time dimension, while the frequency and feature dimensions are multiplied, serves as the embedding dimension. Multimodal features are subsequently integrated through concatenation followed by processing via a linear layer.The classification mechanism at the end consists of a linear layer followed by a softmax function.We compare the generation quality of our method when using the diffusion model versus the VAE as the generative model. For this analysis, we use accelerometer signals from the human activity recognition dataset. Synthetic samples are generated under the same conditions as the original data samples. We select two groups of the original spectrograms based on their complexity in patterns: the first group has simpler and more pronounced patterns, while the second group has more complex and chaotic patterns.As illustrated in Figure 9 , the VAE-generated results are comparable to those from the diffusion model in the first group of the selected samples. Both the diffusion model and VAE successfully capture the fundamental frequency and its harmonic patterns. This group of selected samples exhibits temporal consistency with pronounced harmonic patterns that are straightforward to replicate. However, despite the relatively simple structure of these realistic samples, the spectrograms produced by the VAE are less detailed than those from the diffusion model, particularly in the higher frequency regions.For the second group, which features more complex original spectrograms as shown in Figure 10 , the generation quality of the VAE significantly degrades. The findings indicate that the VAE struggles to replicate the intricate textures present in the original spectrograms, whereas the diffusion models continue to effectively capture and reflect these characteristics. In Figure 11 , we randomly select an acoustic sample from the vehicle detection dataset and visualize the generated spectrograms when manipulating the metrics as conditions. Again, we change a single metric condition while retaining the remaining metrics to show to control effects. In Figure 12 , we randomly select an EEG sample from the harmful brain activity recognition dataset and visualize the generated spectrograms. Like above, when manipulating a certain metric condition, we keep the remaining metrics unchanged. Public information and video footage about the human subjects can be found at: https://www.unimannheim.de/dws/research/projects/activity-recognition/dataset/dataset-realworld/ 


Decision: The paper presents a data augmentation paradigm for IoT sensing by introducing a fine-grained controllable generative model using a metric space defined by statistical metrics, which enhances model robustness across various tasks. Supported by experiments in various IoT applications, the method outperforms traditional techniques, demonstrating its effectiveness and adaptability. The discussion mainly focused on various technical details in the work. After the discussion, all the reviewers recommended acceptance.

Reviews: review: paper_summary: paper_summary: This paper is a study on data augmentation techniques for IoT sensing applications. The authors first present work on the most effective data augmentation techniques in the field of IoT sensing data. Based on this, they propose that domain knowledge can be combined with these data augmentation techniques to guide data generation. Specifically, they also propose a approach of fine-grained controllable generative augmentation, which uses data augmentation techniques based on the metric space of statistical metrics defined by human experience to improve data augmentation for downstream tasks. Experimental results from various scenarios validate the effectiveness of this method, and also explore the benefits of incorporating domain knowledge in data augmentation techniques.
strength_weakness: strength_weakness: The motivation of this paper to propose a fine-grained control approach, incorporating domain knowledge into the data enhancement approach is not clear enough. They are not based on a specific existing problem and are a little bit heuristic.
questions: questions: Please see the Weaknesses above. Considering my limited expertise in this specific domain, I am not in a position to raise specific questions.
limitations: limitations: Yes
paper_summary: paper_summary: This paper aims to synthesize IoT data to augment sensing models with generative models. The authors designed metric spaces to model the conditions which control the generative models. Experiments involving different IoT tasks, generative models, and sensing models have been conducted, proving the effectiveness of the proposed method.
strength_weakness: strength_weakness: 1. Although I understand the novelty of using the proposed metrics as conditions for generative models, I am not sure about using the term “metric”. From my point of view, it seems that calling it “indicator” or other terms is more suitable. Otherwise, it may confuse with the metrics (accuracy, F1) in the experiments.
2. A transformer typically includes an encoder and a decoder. It seems that you were using  an encoder as the classifier, which is fine. However, you should clearly state this to avoid ambiguity.
3. For generative models, there are actually specific metrics to evaluate their performance, such as FID etc. I understand that this paper aims to augment sensing models so it is reasonable to measure the accuracy and F1, but specific metrics for generative models may be also beneficial to the quality of the paper.
questions: questions: 1. It is reasonable that the synthesized data can improve the generalization of sensing models. Can you show the results of adding Gaussian noise to samples as augmentation? This can be the simplest baseline, proving that your generative model can  work because of the synthesized data, but not because of using more samples that help tackle overfitting.
2. According to the appendix, in the first dataset, you separated different domains, while in the last two datasets, you did not. Can you provide further experiments where domains for training and validation are separated for the last two datasets?
3. The authors introduced the weight $w$ in the loss $\mathcal{L}$. Can you provide hyperparameter sensitivity analysis for this?
4. There are indeed latest works using spectrograms as conditions for generative models to augment models [1,2]. Can you further discuss your strengths compared to them? It seems that your work has covered more IoT modalities.

[1] DiffWave: A Versatile Diffusion Model for Audio Synthesis

[2] DiffAR: Adaptive Conditional Diffusion Model for Temporal-augmented Human Activity Recognition
limitations: limitations: 1. Despite the effectiveness, the paper has not discussed the efficiency of generative models. As mentioned in the papers, it took several days to train generative models. I think this is a minor issue but can be discussed in the limitations.
2. This paper mainly discusses models for classification, while in IoT there may also be regression problems. For example, it is typical to track subjects using IoT devices.
paper_summary: paper_summary: The paper addresses the challenge of overfitting in Internet of Things (IoT) sensing models caused by data distribution shifts between training datasets and real-world scenarios. To enhance model robustness, the authors introduce a novel data augmentation paradigm that adds fine-grained control to generative models. By defining a metric space with statistical metrics that capture the essential features of short-time Fourier transformed (STFT) spectrograms of IoT sensing signals, the method enables tailored augmentation of spectrogram characteristics in the time-frequency domain. The proposed data augmentation techniques are evaluated across various generative models, datasets, and IoT sensing applications, demonstrating superior performance compared to conventional transformation-based and prior generative data augmentation techniques.

The major contributions of this paper are as follows. It empirically validates the versatility of the proposed augmentation approach using different generative model backbones, including diffusion models and VAEs, across three distinct IoT sensing applications: wearable-based human activity recognition, seismic and acoustic-based vehicle detection, and EEG-based harmful brain activity recognition. The results show that this method outperforms traditional and generative data augmentations across various downstream models in all evaluated sensing tasks. The novelty of the approach lies in its fine-grained control over generative augmentation, leveraging domain-specific knowledge to produce highly authentic synthetic samples. The significance of this work is highlighted by its ability to enhance IoT sensing model robustness, particularly in scenarios where domain-specific knowledge is available, and its adaptability across different generative models to benefit from their inherent capabilities.
strength_weakness: strength_weakness: Table 2 (Section 4.5) 
Statistical Significance: Including statistical significance tests (e.g., p-values) for the performance differences would strengthen the claims and provide more confidence in the results. You could provide Confidence Intervals for the performance metrics at each augmentation level. 

Typographical and Formatting Issues: There are minor typographical errors and formatting issues, such as lack of space before citation (line 202) and inconsistent capitalization in references. Ensuring meticulous proofreading and adherence to formatting guidelines would improve the paper's presentation and readability.
questions: questions: Three datasets are mentioned in the paper. There are case studies reported related to HAR and Vehicle Detection. No case study related to Brain Activity? A related dataset is mentioned.

Unexplained trends: Regarding Figure 7. Performance increases from 50% to 500% augmentation but decreases slightly at 750% before stabilizing or slightly increasing at 1000%. Why is that the case? It is difficult to ascertain whether this drop is statistically significant or merely a result of random fluctuations in the data.

Checklist Q5: how can the reviewers have confidence in the reported results if the code or data is not released — please argue to justify.
limitations: limitations: Statistical Significance and Robustness: The inclusion of statistical significance tests, such as p-values or confidence intervals, for the performance differences reported in Table 2 is currently lacking. This omission weakens the robustness of the claims.
paper_summary: paper_summary: This work introduces fine-grained control to generative models for IoT data augmentation by defining a metric space with statistical metrics. 

These metrics are essential for capturing the key features of the spectrograms and serve as strong constraints for the generative model, allowing for customization of spectrogram characteristics in the time-frequency domain tailored to specific application requirements. 

The evaluation performs better than traditional transformation-based data augmentation techniques and previous generative data augmentation models.
strength_weakness: strength_weakness: 1. How to balance the original generative model loss and the designed metric loss is not well discussed. As mentioned by the authors, the two loss terms are ratioed by 10:1. It seems the metric loss contributes marginally during the generation process. Considering the generation process is a probabilistic process, without deep analysis, it is hard to convince readers that high-quality outputs are random or consistent. 

2. The metrics selection is not well-founded. Though the authors have discussed its limitation in the metric selection, the metrics are data-type and task-oriented. The inappropriate selection of the metrics could introduce extra noise. Which or how these metrics contribute to the better generative process should be further analyzed. Overall, the contribution is limited as in-depth analysis is missing, and complex hyperparameter (e.g., metric selection, weight selection) tunning is involved.

3. The evaluation setting is trivial, which can hardly demonstrate the efficacy of high-quality data generation. For example, the HAR dataset has only 5 simple activities, which is not a data-hungry task. The authors are encouraged to check more complex real-world datasets, considering the data and label heterogeneity.
questions: questions: 1. What kind of outputs will be if we choose different weights between metric loss and generative loss? It seems this is not an easy job to adjust every weight corresponding to each metric according to the loss during the model training process.

2. Why do not authors choose other metrics? Are the selected metrics general to all IoT tasks? What if some data types are not sensitive to the proposed metrics?

3. How large the dataset is required to train the diffusion model for the generation task?

4. Is it possible the interpolation, and disturbance in the metric space change the metric to respond to a different label, which conflicts with the original label? If so, can authors provide such low-quality generation examples?

5. Will there be a domain shift between generated samples and raw samples?

6. Could authors provide the magnitude of the spectrogram?
limitations: limitations: The authors have discussed the limitations in metric selection and the introduction of domain knowledge.

The authors have kindly answered my questions and I appreciate their efforts for the research community.

Comments: title: Author Reply to Reviewer eQAt (Part 3/3)
Author Reply to Reviewer eQAt (Part 2/3)
Author Reply to Reviewer 8SZC (Part 4/4)
Author Reply to Reviewer 8SZC (Part 3/4)
Author Reply to Reviewer 8SZC (Part 2/4)
comment: Thank you for your feedback! We apologize for referencing the wrong paper [1], which has a similar title to DiffAR [2]. We will ensure that the correct comparison with DiffAR [2] is included in the camera-ready version.

The key differences between our work and DiffAR [2] are as follows:

1. **Conditions**: DiffAR uses different levels of spectrograms as conditions for guiding CSI generation at various diffusion steps. Specifically, low-frequency features guide the generation of global patterns in the earlier steps, while high-frequency features synthesize local patterns in later steps. In contrast, our work employs statistical indicators as conditions consistently throughout the diffusion process.

2. **Modality Coverage**: DiffAR is focused on generating CSI signals specifically for human activity recognition (HAR) tasks, whereas our method is designed to cover a broader range of IoT sensing tasks. Exploring the application of our approach to CSI signals in HAR could be an interesting direction for future research.

[1] Diffar: Denoising Diffusion Autoregressive Model for Raw Speech Waveform Generation

[2] DiffAR: Adaptive Conditional Diffusion Model for Temporal-augmented Human Activity Recognition
DiffAR is not generating human speech waveforms, but generating radio signals from WiFi (a kind of IoT signals) for augmenation in human activity recognition. Anyways, thanks for your detailed response. Good luck.
We sincerely appreciate your thorough review of our paper. Thank you for raising the rating to a weak accept!

We agree that assessing the statistical significance of the differences between baselines is crucial. In the camera-ready version, we will include paired t-tests between the ablated baselines and the "All metrics" baseline in the Appendix.

Once again, thank you for your valuable suggestions!
I acknowledge reading the rebuttal. The response looks fine. 

It is good that you have added 95% CI. Please consider also describing if the difference is statistically significant? Such tests would help determine whether the differences in performance between the various metrics (e.g., "All metrics" vs. "No time domain metric") are statistically significant or could have occurred by random chance.

Based on the response to the reviews, I have modified the ranking to weak accept.
We appreciate your thorough review of our paper. Great thanks for your positive feedback and for raising your rating!
Thank you for the clarifications provided. Taking in account the rest of the responses thus far, I will raise my score to weak accept and wish the authors best of luck for the remaining review process.
We really appreciate your dedication in reviewing our paper, which has greatly assisted us in the paper review. Also great thanks for raising the rating!
Thanks for your further question! When the metric values are controlled within a reasonable range, we observe that the generated spectrograms from our data augmentation approach are mostly visually authentic and experimentally effective to the downstream models. In this case, we envision that the domain shift is reflected more on the limited generative capability: the generated data only mirrors a subset of real-world variations, thus the downstream model still cannot perform perfectly in the whole validation set. This is especially true for **interpolation** augmentation method, as it is a more conservative augmentation, aiming at producing intermediate data samples within the known data distribution. 

For augmentation methods with higher freedom, including **disturbance** and **domain knowledge instruction**, to strictly prevent generated data with metric values that are out of the designed boundary, we propose a simple filtering method.
Specifically, we filter the generated dataset by the designed metric value boundary. For example, if our intention was to manipulate the contrast between -20%~20%, any generated data samples with contrast out of this range comparing to its original data sample will be excluded. Such outliers could exist as the control over metrics is not strictly guaranteed but through conditioning. By filtering out these outliers, the augmented data samples are further refined on their statiscal qualities.

We conduct a simple example to demonstrate the effectiveness of the filtering method. We adopt the same setting described in Section 4.3 Case study 1, where we apply **domain knowledge instruction**-based data augmentation to the human activity recognition task. We compare the number of augmented samples before and after the filtering, as well as the corresponding downstream model performance using DeepSense.

|          | Unfiltered | Filtered |
|----------|-----|-----|
|Augmented Sample Number | 24,795 | 22,922 (-7.55%) |
| Accuracy | 90.31%  | 91.22% (+0.91%)  | 
| F1 | 0.9027 | 0.9175 (+0.0148) |

Our findings are twofold. First, the metric conditions are generally well controlled as only 7.55% of the augmented samples requiring filtering. Second, the performance of the filtered augmented dataset slightly outperforms the unfiltered one, demonstrating the potential benefits of this further refinement.
Thank you so much for your detailed reply. Another question is how the synthetic data affects the model training process as there are distribution shift issues. Is there a way to filter out low-quality synthetic samples based on your proposed approach?
> Q5: Will there be a domain shift between generated samples and raw samples?

**Response**: As with any generative model, there will inevitably be a domain shift between generated samples and real samples, as the model can only approximate the natural data distribution rather than perfectly replicate it. This inherent limitation means that the generated samples may not fully capture all nuances of the original data. However, the primary goal of using these augmented samples is to introduce variety and expand the training set, thereby enhancing the robustness and generalizability of the downstream model. As long as the augmented samples cover a substantial range of the data's possible variations, they can provide valuable additional training data. This can help the downstream model better understand the underlying data distribution and improve its performance, even in the presence of a domain shift.



> Q6: Could authors provide the magnitude of the spectrogram?

**Response**: We list the mean and standard deviation of the magnitude of spectrograms by modalities in each dataset as below:

|                             | Human Activity Recognition |                     | Vehicle Detection         |                     | Harmful Brain Activity Recognition |
|-----------------------------|----------------------------|---------------------|---------------------------|---------------------|-------------------------------------|
|                             | Accelerometer              | Gyroscope           | Accelerometer             | Microphone          | EEG                                 |
| Mean                        | 24.8311                    | 5.3611              | 0.9683                    | 0.5752              | 0.7792                              |
| Standard Variance           | 91.3622                    | 13.4403             | 2.0264                    | 1.6248              | 0.7153                              |


The three datasets have various spectrogram magnitudes, illustrating the diversity of data formats in IoT sensing tasks. The positive results achieved with these datasets demonstrate the generalizability of our approach.
> Q2: Why do not authors choose other metrics? Are the selected metrics general to all IoT tasks? What if some data types are not sensitive to the proposed metrics?
 
**Response**: As discussed in our response to W2, the metrics selected in our paper have been widely used in previous literature as critical features for depicting signals in various IoT sensing applications. However, we acknowledge that there is a broader range of metrics that we have not incorporated. We recognize this as a limitation and plan to explore this area in future work. Specifically, we aim to consider additional potential metrics and develop a mechanism for building a custom metric set tailored to specific IoT sensing tasks. For example, we could construct a decision tree for the IoT task based on all potential metrics and select them using the impurity-based importance score of each metric.

> Q3: How large the dataset is required to train the diffusion model for the generation task? 

**Response**: We list the number of data samples in each dataset as below:
|          | Human Activity Recognition | Vehicle Detection | Harmful Brain Activity Recognition
|----------|-----|-----|------|
| Train |  4,959 |  8,015 |78,548 |  
| Validation | 2,309 | 2,763  | 19,532  | 
| Total | 7,268 | 10,778 | 98,080 |

Both the human activity recognition and vehicle detection datasets are relatively small, especially compared to large language or vision datasets. However, our evaluations demonstrate that these datasets are sufficient to support the training of our diffusion model for the purpose of data augmentation. In IoT sensing tasks, where labeled data can be expensive to obtain, data efficiency in training generative models is crucial. The effectiveness of our approach on these small IoT datasets highlights its great data efficiency.

> Q4: Is it possible the interpolation, and disturbance in the metric space change the metric to respond to a different label, which conflicts with the original label? If so, can authors provide such low-quality generation examples?

**Response**: Indeed, low-quality generation can occur if augmentation techniques are not properly applied. Figure 14 in the **uploaded pdf file** illustrates poor generation results. In these examples, we selected a metric for each original signal and manipulated its value by multiplying it by 0, 10, and 100, resulting in highly unnatural metric values. These modified metrics were then used as conditions for the generative model to synthesize the corresponding spectrograms. We observed that these out-of-distribution metric values led to unrealistic generation results. This demonstrates that augmentation methods, including interpolation and disturbance, must be used appropriately. The interpolation and extrapolation capabilities of the generative model are not limitless, and the designed metric conditions must remain within a reasonable range. However, we did not observe any cases where a sample, after modification in the metric space, explicitly exhibit patterns under a different label in our tested IoT tasks.
> L2: This paper mainly discusses models for classification, while in IoT there may also be regression problems. For example, it is typical to track subjects using IoT devices.

**Response**: We agree that IoT sensing tasks can also be regression problem. To validate the generalizability of our approach, we construct a vehicle distance prediction task using the vehicle detection dataset we described in our paper. Apart from the sensor signals, during the data collection of the dataset, we also recorded on-vehicle GPS traces sampling at 1 Hz. This was originally used for filtering out samples when vehicles are out of the sensing range (as mentioned in Appendix C.2, line 556). In this new vehicle distance prediction task, the objective of the downstream model is to predict the distance between the vehicle and sensor node. The input data is still a 5-second long segment. We use the vehicle to sensor distance of the last second as the label. We modify the original DeepSense model into a regression model, by replacing its classification head (the final linear layer with output length equals to the number of classes) with a linear layer with output length equal to one. The training loss function of the modified DeepSense is also changed from cross-entropy loss to mean squared error loss. 

When training the conditional diffusion model, we drop the vehicle type label and use distance label as a condition. To generate augmented data, we apply ***disturbance*** technique on metrics including ***contrast***, ***flatness***, ***entropy***, and ***F0 amplitude*** by [-15%, 15%]. The augmentations are supposed to reflect the variance brought by background noise and vehicle dynamics (instant speed, acceleration, turn rate, etc.). We report the results as follows:

|                                          | Mean Error (feet) |
|------------------------------------------|--------------------|
| No Aug                                   | 28.35              |
| Time and Frequency Domain Transformation | 32.33                   |
| cDiff                                    | 28.12                   |
| Diff-disturb                             | **25.27**                 |

The time and frequency domain transformation brings negative influence to the distance prediction, as transformations like amplitude scaling and time warping tend to disturb the original distance related patterns. The results of cDiff shows a constrained improvement. Diff-disturb most effectively reduces the mean error, again proving the superiority of fine-grained controlled generative augmentation.  The result of this case study shows that our approach can also be generalized to regression tasks in IoT sensing.
> Q4: There are indeed latest works using spectrograms as conditions for generative models to augment models [1,2]. Can you further discuss your strengths compared to them? It seems that your work has covered more IoT modalities.
> [1] DiffWave: A Versatile Diffusion Model for Audio Synthesis
[2] DiffAR: Adaptive Conditional Diffusion Model for Temporal-augmented Human Activity Recognition

**Response**: Thank you for bringing up the two related works! We list the differences between our work and DiffWave and DiffAR in the following aspects:
1. **Motivation**: For DiffWave and DiffAR, the ultimate goal is to generate high fidelity human speech waveforms. They do not explicitly address the data augmentation problem. As a matter of fact, many large public human speech datasets are available, which can sufficiently support general high-performance speech models. In contrast, our paper positions in data augmentation for IoT sensing applications, which typically have much less data available. The ultimate goal is to produce more diverse data samples that complement the originally constrained dataset thereby enhance the downstream models.
2. **Modality**: In DiffWave and DiffAR, the modality is human speech acoustic signals in waveform. Human speech data are recored in high sampling rate (both papers were evaluated on LJ-speech dataset, 22.05 kHz),  Spectrograms are used as intermediate results the generation pipeline. In our paper, we focus on IoT sensing modalities, which include accelerometer, gyroscope, seismic, acoustic, and EEG. Even though these are also time series data and can be represented in waveforms, they are much lower in sampling rate (100-800Hz) and measure various physical phenomena (human activities, car movements, and human brain activites). 
3. **Conditions**: In DiffWave and DiffAR, conditions taken by the diffusion model are linguistic features and mel spectrograms. These conditions provide detailed information and are very concrete indicators to the corresponding speech waveforms. In our paper, the conditions are statistical metrics (indicators), which only depict the high level features of the IoT sensing signals. The goal of using these conditions is to allow human designers to control the style of the generation results based on their domain expertise.
4. **Contributions**: DiffWave and DiffAR both advance the design of diffusion models in human speech synthesis. In our work, we propose a general generative data augmentation framework for IoT sensing. Our framework is not strictly coupled with either diffusion model or any specific type of generative model (although in our evaluations, the diffusion model implementation outperforms the VAE implementation). The intellectual contribution of our paper lies in the enforcement of fine-grained control over the generation, and how to utilize the control to precisely produce high-quality synthetic data for enhancing IoT sensing tasks.

Due the large difference between DiffWave, DiffAR and our work, we cannot directly compare them as baselines in the evaluations. But we will discuss them in the Related Work section in the camera ready version.

> L1: Despite the effectiveness, the paper has not discussed the efficiency of generative models. As mentioned in the papers, it took several days to train generative models. I think this is a minor issue but can be discussed in the limitations.

**Response**: We agree that the efficiency could be a drawback of generative data augmentation when the computational resources or time budget is constrained. We will add it in the Limitations section in the camera ready version.
> Q2: According to the appendix, in the first dataset, you separated different domains, while in the last two datasets, you did not. Can you provide further experiments where domains for training and validation are separated for the last two datasets?

**Response**: Thanks for the question. We apply different data splitting on the two datasets to introduce domain shifts between the training and validation set. Due to the time and space limitation, we only report results from the following baselines: time and frequency domain transformation, cDiff, and Diff-disturb (ours). We use DeepSense as the downstream model.

For the vehicle detection dataset, we utilize data gathered from a state park parking lot with a gravel surface as the validation set, while the remainder (data from the other two locations) serves as the training set. This division of data introduces a domain shift due to variations in ground surface types. The evaluation results are reported as below:

|      | Accuracy | F1-Score |
|------|----------|----------|
|No Aug| 0.6332   |   0.6553 |
| Time and Frequency Domain Transformation |   0.6410     |   0.6694       |
| cDiff |   0.6223   |      0.6320    |
| Diff-disturb (ours) |   **0.6712**    |  **0.6842**  |


For the harmful brain activity dataset, we randomly select 66 out of 668 patients (about 10%) to form the validation set, with data from the remaining patients comprising the training set. This approach to data splitting introduces a domain shift due to differences among the patients. The evaluation results are reported as below:

|      | Accuracy | F1-Score |
|------|----------|----------|
|No Aug| 0.5531   |   0.5102 |
| Time and Frequency Domain Transformation |   0.5677     |   0.5429       |
| cDiff |   0.5331   |    0.5209      |
| Diff-disturb (ours) |   **0.5977**    |  **0.5945**   |

As the above results show, our method achieves the best performance in addressing the domain shift in both the two datasets. 

> Q3: The authors introduced the weight $w$ in the loss $L$. Can you provide hyperparameter sensitivity analysis for this?

**Response**: Thanks for the comment. Due to the limited computational power and time schedule, we profile different weights in loss $L$ just for Human Activity Recognition dataset using Diff-disturb as the data augmentation method. 
We report four different ratios of $L_G$ (generative model loss) / $L_{metric}$ (metric loss) in loss $L$ as the sensitivity analysis.

DeepSense:
|          | 1:1 | 5:1 | 10:1 | 20:1 |
|----------|-----|-----|------|------|
| Accuracy |  0.8211 | 0.8307 |**0.8324**| 0.8249  |
| F1-score |  0.8209 | 0.8333 |**0.8422**| 0.8360  |

Transformer:
|          | 1:1 | 5:1 | 10:1 | 20:1 |
|----------|-----|-----|------|------|
| Accuracy | 0.8199  | 0.8105  |  0.8205 | **0.8280**   |
| F1-score | 0.8239  | 0.8195  |  **0.8352** | 0.8333   |

We observe that when the ratio of $L_G/L_{metric}$ is between 1 and 20, the performance generally remains stable. Figure 12 in the **uploaded pdf file** illustrates the changing curves of each loss term during model training for ratios of 10:1 and 5:1. Our approach to weighting these two loss terms prioritizes $L_G$ in the early training epochs to enhance the diffusion model's denoising quality. As training progresses, we gradually increase the emphasis on metric control, with $L_{metric}$ outweighing $L_G$ at around 200 epochs. Notably, a higher weight on $L_{metric}$ can lead to its convergence at a higher value, as shown by the comparison between Figures 1.a and 1.b. Consequently, we default to a $L_G: L_{metric}$ ratio of 10:1 in our evaluations.