Dataset transformations trade-offs to adapt machine learning methods across domains
Authors: Napoleon Costilla-Enriquez
Yang Weng
Venue: ICLR 2022 Submitted (2022)
Keywords: Datasets
multiple domains
cyber-attacks
optimal transport

Abstract: Machine learning-based methods have been proved to be quite successful in different domains. However, applying the same techniques across disciplines is not a trivial task with benefits and drawbacks. In the literature, the most common approach is to convert a dataset into the same format as the original domain to employ the same architecture that was successful in the original domain. Although this approach is fast and convenient, we argue it is suboptimal due to the lack of tailoring to the specific problem at hand. To prove our point, we examine dataset transformations used in the literature to adapt machine learning-based methods across domains and show that these dataset transformations are not always beneficial in terms of performance. In addition, we show that these data transformations open the door to unforeseen vulnerabilities in the new applied different domain. To quantify how different the original dataset is with respect to the transformed one, we compute the dataset distances via Optimal Transport. Also, we present simulations with the original and transformed data to show that the data conversion is not always needed and exposes the new domain to unsought menaces.

Under review as a conference paper at ICLR 2022
DATASET TRANSFORMATIONS TRADE-OFFS TO ADAPT MACHINE LEARNING METHODS ACROSS DOMAINS
Anonymous authors
Paper under double-blind review
ABSTRACT
Machine learning-based methods have been proved to be quite successful in dif- ferent domains. However, applying the same methods across domains is not a trivial task. In the literature, the most common approach is convert a dataset into the same format as the original domain to employ the same architecture that successful in the original domain. Although this approach is fast and conve- nient, we argue it is suboptimal due to the lack of tailoring to the specific problem at hand. To prove our point, (1) we exhaustively examine the dataset transfor- mations that are used in the literature to adapt machine learning- barning-bbsed methods across domains; (2) we show that these dataset transformations are not always beneficial; and (3) we show the drawbacks of converting the dataset to adapt a Machine sarming-based method
different domain. To quantify how different the original dataset is with respect to the transformed one, we compute the dataset distances via Optimal Transport. Also, we present simulations with the original and transformed data to show that the data conversion is not always needed beneficial.
INTRODUCTION
Machine learning (ML) methods are revolutionizing and transforming a wide range of disciplines. This is being boosted by unprecedented integration of new real-time monitoring, sensing, control, and communication devices, which give us a wealth of data to propel the ML techniques. However, ML research usually done in computer science to solve problems in computer vision (CV), audio recognition, among others. The same techniques used in these domains are adapted to many other different fields and areas. For example, many of these techniques are used to monitor, control, and operate cyber-physical systems (CPSs) such tracking object systems, water distribution systems, power systems, among others. Therefore, nowadays, data-driven monitoring and control play an important role for Phys-cl Systems (CPSs), c.g., smart grids, autonomous automobile sys- tems, robotic systems, etc. Jazdi (2014). For example, Duchesne et al. (2020) gives an overview of MI. methods for reliability and management in energy systems. Most of such applied methods fall into classification and regression problems using supervised and unsupervised learning. For exam- ple. Marot et (2018) automatically segments large-scale power grids into coherent zones for the management of the grid for control operators with ML techniques. However, little research has been done in maximizing the adaptation of ML techniques across domains in unified framework.
In this work, we claim that while adopting ML techniques solves specific problems, the way to adapt these techniques also open the door on CPSs vulnerability with dire consequences Alguliyev et al. (2018). For example, one can intentionally distort inputs according to how one applies ML method cause an incorrect control action due to an estimation mistake. Known as adversarial attacks Kurakin et al. (2018), many studies show that outputs from classifiers can be easily changed with imperceptible changes to the input in the computer science domain Carlini & Wagner (2017); Szegedy et al. (2013). In the CPS context, this is known where an attacker intercepts and maliciously changes the system measurements to cause harm in the real world. For instance, cyber-attack in power system could cause system operator to take wrong control actions causing a blackout. Similarly, a cyber-attack could cause fatal autopilot crashes in autonomous automobile systems Banks et al. (2018) Dikmen & Burns (207))
False Data Injection Attack (FDIA),

---

Under review as a conference paper at ICLR 2022
Adapting the ML methods from the originally proposed area to one (CV to CPSs, for example) is a non-trivial task that has not been thoroughly analyzed in the literature. The general consensus in the ML community says that there are standard recipes to apply given method across multiple domains. The dataset examples are different across domains; for example, whereas the CV dataset samples are images that represented three-dimensional tensors, in CPSs the dataset samples are one-dimensional tensor measurements from multiple devices with a timestamp. Then, in the literature, there are two main approaches to adapt methods specific domain. (1) Modify the ML learning architecture to fit the dataset samples format. For example, if the original domain is CV in which a convolutional neural network (CNN) in employed, the adapted ML technique will use as a base fully connected neural network to handle the one mie-dimensional data in the targeted domain. (2) Transform the new dataset samples into the same format as the original domain to use the same ML. architecture. For example, if in the original domain CNN with a three-dimensional as input, and the targeted domain a CPS with dine-dieesional tensors, then the one-dimensional is transformed into a grid-like format to use CNN as in the original domain. This begs the question, What are the benefits and downsides of using the first or second approach to adapt ML. techniques across different domains?
To better adapt ML to different domains such as CPS systems, it is important to understand the ben- efits and possible downsides for different methods using particular architectures and dataset types. In this paper, we explore the benefits and downsides of using different approaches to adapt MI. methods across disciplines. There are multiple ML methods. To simplify our analysis, we choose to study classifiers. This will allow us to quantify the benefits of the classifier's accuracy. To quantify or evaluate the downsides, we analyze how different datasets and ML architecture are impacted by adversarial attacks. We will show the trade-offs of using different dataset formats and model archi- tectures. In specific, this paper demonstrates that while CNN improves model accuracy. they are more sensitive to adversarial examples than fully connected neural networks (FCNN).
2 RELATED WORK
Computer vision (CV) is an area that studies how computers can gain high-level understanding from digital images or videos, which utilizes 2- and 3-dimensional data with multi-channels. Con- volutional neural networks (CNN) are the most important architecture for CV and can be 2- and 3- dimensional (CNN with higher dimensions are possible but rarely used). These CNN are designed to learn filters from data where there is translation invariant. CNN can pull out this translational invari- that images inherently have. For example, if you want recognize cat in image, it should not matter the cat's location in the picture. Computer vision also deals with 3-dimensional images, for example, CT and MRI scans. To exploit these 3-dimensional data, 3-d CNN are employed.
Adversarial attacks. Deep Convolutional Neural Networks (CNNs) models are highly susceptible to adversarial examples Goodfellow et al. (2014); Moooavi-Dezzoooli et al. (2017); (Szegedy et al. (013). Research on this topic has been done mainly for image classifiers and face recognition systems. These adversarial attacks or adversarial samples have drawn the community' attention; in 2018, there was an Adversarial Attacks and Defences Competition Kurakin et al. (2018).
A lot of work create adversarial samples has been done, including techniques such as the L-BFGS attack (Szegedy et al. (2133). FGSM Goodfellow et al. (2014), and the CW attack Carlini & Wagner (2016) In specific for classifiers in the white-box context, in addition to the model knowledge, the attacker knows the set of classes Y and the set of valid inputs X to the classifier. These methods do not explicitly make use of the latent features to create adversarial examples. For example, the conventional way to create these adversarial examples is by maximizing the log-likelihood of the target class yt over a e-radius ball around the original input (which is usually represented as a vector of d pixels in the range [0, 1) Athalye al. 2017))
arg max log P (uel=') 2'
subject to ||~' - x <E
(I)
x' E'
False data injection in power systems. Past work in the power systems area shows the vulner- ability of the state estimation functionality. It is based the observability of the system state.
2

---

Under review as a conference paper at ICLR 2022
Some works show different attacking approaches Mohammadpourfard al. (2017). Based on such analysis, the impact an attack on the system is also evaluated Mohammadpourfard et al. (2019) Mathematically, the FDIA can be expressed as:
min m. - h (v ?
subject m. + a,
where and m, the set of original and comrupted measurements, respectively; Ver is vector of the state variables, h is the state estimator non-linear model, and a is the attack vector to alter the original set of measurements.
BACKGROUND
This section presents the Optimal Transport (OT) problem that will allow us quantify how differ- ent similar two distributions or datasets are from each other.
3.1 BACKGROUND ON OPTIMAL TRANSPORT
Optimal transport (OT) is compares probability distributions (Villani 2003 2008). OT leverages the geometry of the underlying space, making them ideal for comparing distributions, shapes and point clouds (Peyré & Cuturi 2019).
The OT problem considers a complete and separable metric space X, along with probability mea- sures
E P(X), B E P(X).
The Kantorovich formulation Kantorovitch 1942) of the transportation problem can writeen as:
/ cla, y) dı( y).
OT(a, A
min xEll(a, xxx
where : X + R- is a cost function (the "ground" cost), and the set of couplings IT(a, 3) consists of joint probability distributions over the product space X' x X with marginals a and B, that is,
[a, 3) A (N E P(XxX) PI = a, P2Y T = 3}-
(5)
Whenever X is equipped with a metric dx, it natural to use as ground cost, e.g.,
c(x, v) dx(x,1)
(6)
for some p 2 1. In such case,
W, Wa 3) OT(a, 3/1/2
called the p- Wasserstein distance. The case p = also known as the Earth Mover's Distance (Rubner et al. 2000).
The measures a and are rarely known in practice. Instead, one access to finite samples
fx E X, fy Ex X.
(8)
In that case, one can construct discrete measures
a55y(0
(9)
and
3 =
b,d ())
(10)
1=1
3

---

Under review as a conference paper at ICLR 2022
where b are vectors in the probability simplex, and the pairwise costs can be compactly repre- sented as an n x m matrix C, i.e.
C c(x(0) ,
(11)
In this case, Equation equation becomes a linear program. Adding an entropy regularization, namely
(T, () & min TEll(a, x2
clz, y) dr( v) + (((T[a
(12)
where
H(T 69 3) (00))= log((dTT/ / da d3) dr
(13)
the relative entropy that results into a problem that can be solved more efficiently (Cutuni 2013 Altschuler et al. 2017) and with better sample complexity (Genevay al., 2019) than the original
one.
We will use OT to measure the distance between datasets done by Alvarez-Melis & Fusi 2020 see Fig. In specific, we'll measure the distance between regular and adversarial datasets.
Regular samples
Regular samples Perturbod samples
Perturbed samples
Smaller distance
Larger distance
Figure 1: Illustration of OT for regular and perturbed datasets.
TRADE-OFFS OF DATASET TRANSFORMATIONS
Machine learning (ML) approaches have been extensively applied on different domains to solve unsupervised and supervised tasks Olowononi al. (2020); Duchesne et al. (2020). Unsupervised tasks mainly encompassed clustering. and supervised tasks primarily included classification prob- lems. In the last section, the different types of data and the most NN architectures were described. Choosing a specific NN design will give you prior or impose a bias on the model. For example, you use vanilla MLP NN, each input will interact with each output. This means either you assume that all the input data is correlated or have no idea of structure that you might exploit. If you use CNN, you are assuming that your grid-like data are locally correlated but further data Xu al. Jeong et al. (2017); Xu et al. (2017). In this way, you are imposing bias or prior over the model. which in this case, will be reflected in the learned kernels by the CNN. On the other hand, your data comes in sequence, you can choose to use RNN that will exploit this prior knowledge of the problem.
ML techniques have been used across different domains. In CPSs power systems to do classifi- cation or regression, for example. In classification for stability, the work Alimi et al. (2020) gives an overview of the work in this area. The takeaway is that the applied methods are in supervised ML techniques to classify an event. For example, the work Wang et al. (2016) assesses the power system stability after the occurrence of disturbance. They
feedforward NN. For regression,
ML methods have been applied to forecast. For example, the work Xu et al. (018a) tries to predict power load with deep-belief network.
4

---

Under review as a conference paper at ICLR 2022
Chen al. 2018) used time-series data such wind and solar generation with 5- and 60-minute resolution. They separate the data by day and then stack the time series into a matrix. So, for hourly data, they form a 24 x 24 matrix. Then convolutional neural network is employed. [Guptal et al. (2018) monitored three time-series with each PMU at every generator unit: voltage magniude, voltage angle, and rate of change of voltage angle. The time series of all generators form 2-D matrix for each of the three variables. This results in three 2-D matrices. Then, these three images are combined form a three-channel, 2-D image, which they call heatmap. Finally, convolutional neural network is trained on the heat maps to classify the power system state. Cai Hill (2022) take voltage (V), active (P), and reactive (Q) power measurements at each bus. With each V. P. and Q. matrix is formed. Then the three matrices V, P. and Q are combined to create an "RGB" image. Shi et al. al. (2020) converted the data as follows. Given buses PMU phasor voltage time series with length 7: two real matrices N x T are formed to get two-channel image. Then, this is the imput to the convolutional neural network. Ren et al. (2020) converted the frequency time- series to matrix with wavelet decomposition and polar coordinate Gramian Angular Field. Then, this grid-like data can be used as input to the convolutional neural network. All these applications of given model architecture and dataset format without analyzing the implications
stacked to
ML methods of different alternatives.
In summary, we can list the main ideas principles by which data is transformed:
1. Single time-series
The most straightforward and least principled way to convert one-dimensional data into grid-like (or image-like) format is to resize your time-series with length T to a matrix with size n X m = T, as Wang et al. (2017) did.
(ii) A more principled way convert mne-dimesiional time-series data into grid-like format is to resize your time-series, exploiting the features of your signal. The time series can be segmented in chunks of one day, and then these chunks can stacked into a matrix, Chen et al. (2018) followed this approach. For hourly data, for example, they formed 24 x 24 matrix.
(iii) Given a one-dimensional time-series, transform be applied to every sample obtain grid-like data from the time series. In speech recognition, they use the Gabor transform to obtain a spectrogram, for example. In power systems, |enn et al. (2020) converted the frequency time-series to matrix with wavelet decomposition and polar coordinate Gramian Angular Field.
Multiple time-series
When multiple time-series are available, the data conversion is carried out in different way. A set of time series of the same variable from many sources can be stacked into matrix. This process can be applied to every variable resulting the same number of matrices (c). Then, these formed matrices can be stacked in tensor to form an "image" with e channels. Examples these approach are Gupta et al. (2018); Cai Hill (2022); Shi et al. (2020).
4.1 DIFFERENT DATASET FORMATS
Given all the dataset transformations previously described, in this part we introduce a classification for different dataset types. The use of each dataset type will dictate which neural network (NN) architecture to use. Starting with the most simple dataset format, let Dr, be a dataset type one defined as
N =((= m m = [M]] = [B]
Dr, =
(14)
n=1
(n)) where x and E T is the n time-series sample of variable m at location with length T, AM is the number of different measured variables, [M] = (1...., M). B is the number of different locations, [B] = [I....,B), is the number of samples in the dataset, and ((() is the label associated with the n sample.
5

---

Under review as a conference paper at ICLR 2022
As shown before, in the literature, they convert vector into matrix. To do so, stack the B samples (in) measuring the same variable from different locations as
(n)
a m,1
V(n)
E R
BxT
(15)
m
(u)
m.
Then, we can define a dataset type two Dr,
N Dr² = (v m= m-][M]
(16)
n=1
Similarly, in the literature they go further and create datasets with tensors. To achieve this from a dataset type two, we stack the Vn matrices a tensor obtaining I(m) = MxBT. This results in a dataset type three
Dr, = - (1r)),
(17)
From the last part, some important points and questions are: (1) What are the advantages or dis- advantages of using Dr, Dr., or D (if any). (2) Do different dataset types have the same vulnerabilities? (3) Can we quantify the difference between dataset types by using Optimal Trans- port?
5 EXPERIMENTAL RESULTS
To explore the advantages and disadvantages of using different dataset types, we carry trained dif- ferent classifiers with diverse dataset types and models. specific, we use the MNIST and Fashion- MNIST datasets. Although these datasets are not from different domain (e.g., CPSs), they are helpful to assess what are the implications of using them with other formats (i.e., Dr, or DT,) To assess the advantages, we measure the accuracy of the test regular data. To evaluate the disadvan- tages, we also measure the accuracy but in perturbed test data. The perturbations are done with the fast gradient sign method (Goodfellow al. 20144). All datasets consists in examples x E 28x28 According to our previous classifications, they are dataset type DMNIST and D Fashion respec- tively.
T2
These datasets can be transformed into datasets type 1 by stacking the columns into a vector resulting in examples æ 784 This would result in datasets type 1, D2, DAMNST and D DFsshion respectively. We create classifiers for both dataset types Dr, and Dr, based on fully connected neural networks (Dense) and convolutional neural networks (Conv), respectively. The fully connected neural network (Dense) has two dense layers of size 784 and 128, resulting in 101, 770 parameters. We use a CNN consisting of two convolutional layers with 32 and 62 filters, respectively, each followed by 2x2 max max-pooling, and fully connected layer of size 256 resulting in 84, 553 parameters (Convl), Also, see that the CNN model, the number of parameters does not affect the classifier output, we use second CNN (Conv2) with two fully connected layers of size 256 and 128 resulting in 1 116, 168 parameters.
We train these classifiers in two ways: (1) we train them regular data, and (2) we also adversary train the networks (AT or Adv) with regular and adversarial samples, with the method proposed by Madry et al. (007).
5.1 ACCURACY AND DISTANCE BETWEEN ORIGINAL AND ADVERSARIAL EXAMPLES
The results for the MNIST-FFashinn dataset are shown in Table (with = 0.05). We perturbed both datasets Dr, and Dr, with the fast gradient descent method (FGDM) (Goodfellow et al. 2014 As result, we obtain adversarial or perturbed datsets that DP, and DF, Then, we measure the distance between the regular and perturbed datasets with the optimal transport framework. The results are
OT, (Dr, DP, 1.05, OT. OT. DP, 0.84.
(18)

---

Under review as a conference paper at ICLR 2022
In this case
(D,, > (Dr,,,,
(19)
The effect of this is that adversarial examples look more similar for Dr,. We hyphotesize that this is correlated with the classifier accuracy on the perturbed dataset. If the distance between the regular and perturbed dataset is larger, then the accuracy on the adversarial trained model will be better shown in Table [ We repeat the same experiment for different ranging from 0 to 0.3 with increments of 0.05. The results are shown in Fig. 2 In this Figure, we make two observations. (I) The accuracy for the adversary trained model with dataset type for the perturbed dataset is higher than its counterpart with dataset type 2. (2) The OT distance between the regular and perturbed datasets type is bigger than the distance for dataset type
The results for the MNIST dataset (with = 0.05) are shown in Table regular and perturbed datasets are
The OT distances between
OT, (Pr; D 16.24, OT, Dj,)) - 9.64.
(20)
We can see similar effect, the distance for the dataset type larger than the distance for the dataset type 1. This results in bigger increase of accuracy in the adversary train classifier as shown in Table We repeat the same experiment for different e ranging from 0 to 0.3 with increments of 0.05. The results are shown in Fig. 3] In this Figure, we make two observations. (1) The accuracy for the adversary trained model with dataset type for the perturbed dataset higher than counterpart with dataset type 2. (2) The OT distance between the regular and perturbed datasets type is bigger than the distance for dataset type 2.
This shows that we must be aware when selecting the dataset transformation ML model archi- tecture across different domains. While using datasets type 2 improves the accuracy, it also makes models more sensible to adversarial examples.
Accuracy (%)
Model
No. of parameters
Regular
Perturbed
Regular (AT)
Perturbed (AT)
Conv1 1 (Dr,)
90
39
88
84
84,552
Conv2 (Dr,)
90
44
89
85
116,168
Linear (Dy,
86
59
87
97
101,770
Table 1: Model comparison for MNIST-Fashion dataset with 0.05.
1.0
OT(Dr, Df.) (Dense)
0.8
40
OT,D,,,P)) (Conv1)
Accuracy
Dense Perturbed
(T DP)
0.6
Convl Perturbed
30
Dense Ady Perturbed
0.4
Convl Adv Perturbed
20
0.2
0.0
0.0
0.1
0.2
0.3
0.0
0.1
0.2
0.3
(a) Accuracy for perturbed examples for the regular (b) OT distance between the regular and perturbed test and adversarial trained classifiers.
set.
Figure 2: Accuracy and OT distance for the MNIST-Fashion dataset.
5.2 COMPARISON OF DISTANCES BETWEEN SAMPLES FOR DIFFERENT MODELS
To further investigate the effect of adversarial examples for different models, we compute the dis- tances among the original and pertubed datasets for both the FCN and the CNN. Suppose have

---

Under review as a conference paper at ICLR 2022
Accuracy (%)
Model
No. of parameters
Regular
Perturbed
Regular (AT)
Perturbed (AT)
Convl (Di;)
99
47
99
99.22
84,552
Conv2 (Drg)
99
65
99
98
116,168
Linear (Dr,)
98
5
98
99.88
101,770
Table 2: Model comparison for dataset
1.0
OT(Pr, Pf)) (Dense)
OT(D,, D., (Convl)
Dense Perturbed
OT(D, D")
Accuracy
0.6
Convl Perturaed
20
Deace Adv Perturbed
0.4
Convt Adv Perturbed
0.2
0.0
0.0
0.1
0.2
0.3
0.0
0.1
0.2
0.3
(a) Accuracy for perturbed examples for the regu- lar and adversarial trained classifiers.
(b) OT distance between the regular and perturbed test set.
Figure 3: Accuracy and OT distance for the MNIST dataset.
N dataset with N samples D = { n= Then, we can compute the distance of an example with all the remaining dataset with the OT framework. Mathematically, is expressed follows
emd; OT; (Ii, {an E : zn + x;)), for I,...,N.
(21)
With the computed distances for every sample, define the as EMD femd,,) y ==1- We can interpret this as an empirical PDF of distances. The results for the MNIST dataset for the FCN and the CNN are shown in Fig. 4 We original and perturbed examples barely move. Fig. 4a on the other hand, shows that interestingly the perturbed distances are smaller than the original ones. This could be the reason why the CNN is more vulnerable to adversarial attacks than the FCN. In other words, the FCN is more resilient to adversarial attacks because the distribution of distances between the original and perturbed examples is not entirely overlapping as in the case of the CNN case. We can see similar results for the MNIST- Fasion dataset in Fig.
in Fig. 4b that the distances between the
CONCLUSION
This paper showed that using different dataset types and neural network architectures has benefits and downsides. When adapting MI techniques across domains, practitioners must be aware of the trade-offs on selecting a given dataset type format or model architecture. While using a convo- lutional neural network improves the model's accuracy, making more vulnerable to adversarial examples. We showed that it is possible to infer how vulnerable is a given model with respect another one by computing their corresponding OT distances between the regular and perturbed datasets. It was proven that a lower OT would result in more vulnerable model against adversarial examples.
REFERENCES
Rasim Alguliyev, Yadigar Imamverdiyev, and Lyudmila Sukhostat. Cyber-physich systems and their security issues. Computers in Industry, 100:212-223, 2018.
8

---

Under review as a conference paper at ICLR 2022
MNIST dataset, Fully-comeeted model
MNIST datese,, Conclutionnl model
Origina
0.040
Originll
Perturbed
Pecturbed
0.000
0.085
B1020
0.055
0.015
0.010
0.010
0.005
0,005
0.000
0.000
100
120
(a) Distance's PDF with FCN model.
(b) Distance's 's PDF with CNN model.
Figure 4: PDFs of MNIST dataset distances.
Fushion datrect,, F bully-connected model
Fashion ristzert, Convolational model
Pecturhod
Pecturbod
0.000
0.0150
0.0125
8.01.00
0.005
0.0025
01000
0,0000
(a) Distance's PDF with a FCN model.
(b) Distance's PDF with CNN model.
Figure 5: PDFs of Fashion dataset distances.
Oyeniyi Akeem Alimi, Khmaies Ouahada, and Adnan M Abu-Mahfouz. A review of machine learning approaches to power system security and stability. IEEE Access, 8:113512-113531, 2020.
Jason Altschuler, Jonathan Niles Weed, and Philippe Rigollet. Near-linear time approximation al- gorithms for optimal transport via sinkhorn iteration. In Guyon, Luxburg, Bengio, Wal- lach, R Fergus, S Vishwanathan, and R Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 1964-1974, Curran Associates, Inc., 2017.
David Alvarez-Melis and Nicolo Fusi. Geometric dataset distances via optimal transport. arXiv preprint arXiv:2002.02923, 2020.
Anish Athalye, Logan Engstrom, Andrew Ilyas, and Kevin Kwok. Synthesizing robust adversarial examples. arXiv preprint 17Xiv:1707.07397, 2017.
Victoria A Banks, Katherine L Plant, and Neville A Stanton. Driver error designer error: Using the perceptual cycle model to explore the cireumstances surrounding the fatal tesla crash on 7th may 2016. Safety science, 008:278-885.. 2018.

---

Under review as a conference paper at ICLR 2022
Huaxiang Cai and David Hill. A real-time continuous monitoring system for long-temn voltage stability with sliding 3d convolutional neural network. International Journal of Electrical Power & Energy Systems, 134:107378, 2022. 2022.
Nicholas Carlini and David Wagner. Defensive distillation is robust to adversarial examples. arXiv preprinl arviv: /607.04311, 2016.
Nicholas Carlini and David Wagner. Adversarial examples not easily detected: Bypassing ten detection methods. In Proceedings of the ACM Workshop on Artificial Intelligence and Security, pp. 3-14, 2017.
Yize Chen, Yishen Wang, Daniel Kirschen, and Baosen Zhang. Model-free renewable scenario generation using generative adversarial networks. IEEE Transactions on Power Systems, 33(3): 2018.
Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. In CJ C Burges, Bottou, M Welling. Ghahramani, and Q Weinberger (eds.), Advances Neural Information Processing Systems 26, pp. 2292-222222222 2300. Curran Associates, Inc., 2013.
Murat Dikmen and Catherine Burns. Trust in autonomous vehicles: The of tesla autopilot and summon. In International Conference on Systems, Man, and Cybernetics, pp. 1093-1098 IEEE, 2017.
Laurine Duchesne, Efthymios Karangelos, and Louis Wehenkel. Recent developments in machine learning for energy systems reliability management. Proceedings of the IEEE, 16569):1656-167, 2020.
Aude Genevay, Léna' "ic Chizat, Francis Bach, Marco Cuturi, and Gabriel Peyré. Sample complex- ity of sinkhorn divergences. Kamalika Chaudhuri and Masashi Sugiyama (eds.). Proceedings of Machine Learning Research, volume 89 of Proceedings of Machine Learning Research, pp- 3574-1584 PMLR, 2019.
lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint 4rXiv:1412.6572, 2014.
Ankita Gupta, Gurunath Gurrala, and PS Sastry. An online power system stability monitoring system using convolutional neural networks. IEEE Transactions on Power Systems, (402):864-872,2011
Nasser Jazdi. Cyber physical systems in the context of industry 4.0. In International conference automation, quality and testing, robotics, pp. 1-4. IEEE, 2014.
Il- Young Jeong, Subin Lee, Yoonchang Han, and Kyogu Lee. Audio event detection using multiple- input convolutional neural network. Detection and Classification of Acoustic Scenes and Events (DCASE), 2017.
Kantorovitch. On the translocation of masses. Dokl. Akad. Nauk SSSR, 37(7-8):227-22,, 1942. ISSN 0002-3264.
Alexey Kurakin, Ian Goodfellow. Samy Bengio, Yinpeng Dong, Fangzhou Liao, Ming Liang, Tianyu Pang, Jun Zhu, Xiaolin Hu, Cihang Xie, et al. Adversarial attacks and defences competition. In The NIPS' 17 Competition: Building Intelligent Systems, pp. 195-231. Springer, 2018.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv: 1xXiv:770.000883 2017.
Antoine Marot, Sami Tazi, Benjamin Donnot, and Patrick Panciatici. Guided machine learning for power grid segmentation. In 2018 IEEE PES Innovative Smart Grid Technologies Conference Europe (ISGT-Europe), pp. 1-6. IEEE, 2018.
Mostafa Mohammadpourfard, Ashkan Sami, and Yang Weng. Identification of false data injection attacks with considering the impact wind generation and topology reconfigurations. JEEE Transactions Sustainable Energy, 93):11349-1364 2017.
10

---

Under review as a conference paper at ICLR 2022
Mostafa Mohammadpourfard, Yang Weng, and Mohsen Tajdinian. Benchmark of machine learning algorithms on capturing future distribution network anomalies. IET Generation, Transmission & Distributtio,, 13(8):1441-1433 2019.
Seyed-Mohsen Densavi--ezfooli, Alhussein Fawzi, Omar Fawzi, and Pascal Frossard. Universal adversarial perturbations. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1765-1773, 2017.
Felix O Olowononi, Danda B Rawat, and Chunmei Liu. Resilient machine learning for networked cyber physical systems: A survey for machine learning security to securing machine learning for cps. IEEE Communications Surveys & Tutorials, 23(1):524-555 2020.
Gabriel Peyré and Marco Cuturi. Computational optimal transport. Foundations and Trends@ in Machine Learning, (15-6):355-600) 2019. ISSN 1935-8237. doi: 10.1561/2200000073..
Huiying Ren, Z Jason Hou, Bharat Vyakaranam. Heng Wang, and Pavel Etingov. Power system event classification and localization using a convolutional neural network. Frontiers in Energy Research, 8:327, 2020.
Yossi Rubner, Carlo Tomasi, and Leonidas Guibas. The earth mover's distance as metric for image retrieval. Int. J. Comput. Vis., 40(2):99-121, November 2000. ISSN 0920-5691, 1573- 1405. doi: 00.1033/A:002644300000
Zhongtuo Shi, Wei Yao, Lingkang Zeng, Jianfeng Wen, Jiakun Fang. Xiaomeng Ai, and Jinyu Wen. Convolutional neural network-based power system transient stability assessment and instability mode prediction. Applied Energy, 263:114558 14586, 2020.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv: 13/2.6 2.6199, 2013.
Cédric Villani. Topics in Optimal Transportation. American Mathematical Soc., 2003. ISBN 9780821833124.
Cédric Villani. Optimal transport, Old and New, volume 338. Springer Science & Business Media, 2008. ISBN 9783540710493.
Bo Wang, Biwu Fang, Yajun Wang, Hesen Liu, and Yilu Liu. Power system transient stability assessment based on big data and the core vector machine. IEEE Transactions on Smart Grid, (5):2561-2570 2016.
Huaizhi Wang, Haiyan Yi, Jianchun Peng, Guibin Wang, Yitao Liu, Hui Jiang, and Wenxin Liu. Deterministic and probabilistic forecasting of photovoltaic power based on deep convolutional neural network. Energy conversion and management, 153:409 422,, 2017.
Daoqiang Xu, Zhixin Li, Shihai Yang, Zigang Lu, Haowei Zhang, Wenguang Chen, and Qingshan Xu. A classified identification delieb network for predicting electric-power load. In 2018 2nd IEEE Conference on Energy Internet and Energy System Integration (EI2), pp. 1-6. IEEE, 2018a. 8a.
Yong Xu, Qiuqiang Kong, Qiang Huang, Wenwu Wang, and Mark D Plumbley. Convolutional gated recurrent neural network incorporating spatial features for audio tagging. 2017 International Joint Conference on Neural Networks (IJCNN) pp. 3461-3466. IEEE, 2017.
Yong Xu, Qiuqiang Kong, Wenwu Wang, and Mark D Plumbley. Large-scale weakly supervised audio classification using gated convolutional neural network. In 2018 IEEE international con- ference on acousties, speech and signal processing (ICASSP), pp. 121-125. IEEE, 2018b.
11


Reviews:
[{"review_id": "y-kuAlzILgo", "reviewer": "Reviewer_GQQh", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors find that different representations and models give different performance, which is not surprising. This is a known fact - in practice, domain specific models are generally trained tailored to the task at hand. These models encode implicit biases needed for the task. For vision tasks, convnets are trained and for NLP, transformers are used. Researchers are aware of this, and the conclusions of this paper add no value in my opinion. Furthermore, the experiments are performed only on MNIST with minimal architectural choices, and the results are not conclusive in any form. I recommend strongly rejecting the paper.", "summary_of_the_paper": "This paper attempts to study the effect of different transformations of input datasets, and the performance of the resulting models thereof. In particular, authors consider time-series data which is then represented in (1) time-series format, (2) vectorized form, (3) tensorized form. For each of the representation, a different neural network is trained and the performance is compared. Experiments are shown on MNIST dataset.", "summary_of_the_review": "The conclusions of the paper add no value, and the experiments are not performed well.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "oHHftYxrBqb", "reviewer": "Reviewer_57HV", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- The draft fails to support the claims strongly. For instance, they do not consider data from CPS, instead they consider well known datasets from computer vision on which various DNN architectures have already been explored. Instead, authors should have worked with a different domain as claimed.\n\n- Also, the datasets MNIST and Fashion MNIST are similar in some of the important aspects: both gray scale, same number of classes, and number of samples (complexity), etc. This does not support the draft's claim that it works with diverse dataset types.\n\n- Experimental analysis presented in the paper is very weak. Only a couple simplest datasets are considered to train simple CNN and MLP classifier architectures. This kind of analysis is not very convincing to generalize from.\n\n- It is not very clear why authors consider adversarial robustness to investigate the disadvantage of the dataset conversion.\n\n- The take-away (according to the abstract) is that  the data conversion is not always needed or beneficial. This is not very informative or novel, since it is understandable that the data formats need to be understood and domain knowledge also is required to avoid any structure or information loss while processing,", "summary_of_the_paper": "- Draft attempts to understand the effect of converting dataset format while applying an ML technique. It argues that converting the target data into a format on which the specific ML technique was shown to be successful is suboptimal. Via simple experiments it shows that the data sample format conversion need not be advantageous.", "summary_of_the_review": "- Draft is incomplete in multiple aspects. It promises much in the abstract (or goals) but fails to deliver it. It is supposed to be a comprehensive empirical analysis, but the provided experiments fall short of it.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "7Ee_5N0l7NQ", "reviewer": "Reviewer_JxT2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n\n- In general, the authors clearly deliver their ideas and experimental procedure. \n\n- While people are introducing ML methods to more and more real-world problems, the topic covered in this paper is certainly becoming more attractive. \n\nCons and feedbacks:\n\n- The problem is not well formulated. The idea of studying different input formates could be helpful when solving practical problems, but I would suggest the authors formulate this problem by starting from even simple NN models and assumptions on datasets. Is it possible to use an objective to describe this discrepancy caused by data transformations?\n\n- Lack experiments. The authors are inspired by ML problems on CPSs datasets, however, the only experiment in this paper is on the MNIST and Fashion-MNSIT datasets, and the experiment on these part (MNIST, fashionMNIST) are already provided by public codebases [0]. Experiments on datasets from power systems, robotics, or autonomous driving are expected. There are multiple public datasets from CPSs mentioned above [1, 2, 3]\n\n- The definition of “domain” and “adapt”. This paper defines the domain as a broad area of applications such as CV, audio recognition, and others, and the term “adapt” means changing the data formate. However, in many existing studies, the term domain is usually associated with the distribution of data [4]. Consider an autonomous driving case where we collect two datasets (image and vehicle trajectory) for each weather (sunny and raining) and thus have four datasets in total, how should we divide these four datasets into different domains? \n\n- The usage of the Optimal Transport Data Distance. While this is a promising prior study on dataset geometric distances, one assumption is that the label-feature distributions are Gaussian. Does that still hold for the CPSs dataset that may contain continuous signals? I would like to hear more from the authors on this point.\n\n- It is a good idea to use adversarial attacks to evaluate the transformations. It would be interesting to utilize the “Wasserstein adversarial attack” [5] and see if this method correlates with the optimal transport data distance. \n\n\n[0]: Link: github.com/microsoft/otdd\n\n[1]: Link: www.kaggle.com/robikscube/hourly-energy-consumption\n\n[2]: Geiger, Andreas, et al. \"Vision meets robotics: The kitti dataset.\" The International Journal of Robotics Research 32.11 (2013): 1231-1237. \n\n[3]: Mandlekar, Ajay, et al. \"Roboturk: A crowdsourcing platform for robotic skill learning through imitation.\" Conference on Robot Learning. PMLR, 2018. \n\n[4]: Courty, Nicolas, et al. \"Optimal transport for domain adaptation.\" IEEE transactions on pattern analysis and machine intelligence 39.9 (2016): 1853-1865.\n\n[5]: Wong, Eric, Frank Schmidt, and Zico Kolter. \"Wasserstein adversarial examples via projected Sinkhorn iterations.\" International Conference on Machine Learning. PMLR, 2019.", "summary_of_the_paper": "This paper aims to study the effects of different transformation methods when processing a dataset for supervised deep learning tasks. The authors conducted one experiment to show that different data arranging methods leads to different vulnerability for adversarial attacks. In the experimental procedure, an optimal transport based method is utilized to measure the distance between a dataset and its perturbed version. ", "summary_of_the_review": "In general, this paper raised an interesting problem and studied the obstacle when people want to apply ML methods to CPSs data. The authors conducted experiments with existing methods on OT, deep learning, and adversarial attack. However, I feel the results and analysis are still in their preliminary form. This paper could meet the acceptance requirements with a clear problem formulation, more extensive experimental results with CPSs data, and theoretical analysis. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Unfortunately, reviewers unanimously agreed that this paper does not meet the ICLR acceptance standards, citing generally unpolished experiments. I would recommend substantially expanding the experimental results in the future.
Decision: Reject