Year: 2021

Abstract:
Machine learning-based methods have been proved to be quite successful in different domains. However, applying the same methods across domains is not a trivial task. In the literature, the most common approach is to convert a dataset into the same format as the original domain to employ the same architecture that was successful in the original domain. Although this approach is fast and convenient, we argue it is suboptimal due to the lack of tailoring to the specific problem at hand. To prove our point, (1) we exhaustively examine the dataset transformations that are used in the literature to adapt machine learning-based methods across domains; (2) we show that these dataset transformations are not always beneficial; and (3) we show the drawbacks of converting the dataset to adapt a Machine learning-based method to a different domain. To quantify how different the original dataset is with respect to the transformed one, we compute the dataset distances via Optimal Transport. Also, we present simulations with the original and transformed data to show that the data conversion is not always needed or beneficial.

text: Machine learning-based methods have been proved to be quite successful in different domains. However, applying the same methods across domains is not a trivial task. In the literature, the most common approach is to convert a dataset into the same format as the original domain to employ the same architecture that was successful in the original domain. Although this approach is fast and convenient, we argue it is suboptimal due to the lack of tailoring to the specific problem at hand. To prove our point, (1) we exhaustively examine the dataset transformations that are used in the literature to adapt machine learning-based methods across domains; (2) we show that these dataset transformations are not always beneficial; and (3) we show the drawbacks of converting the dataset to adapt a Machine learning-based method to a different domain. To quantify how different the original dataset is with respect to the transformed one, we compute the dataset distances via Optimal Transport. Also, we present simulations with the original and transformed data to show that the data conversion is not always needed or beneficial.

heading: 1 INTRODUCTION
text: Machine learning (ML) methods are revolutionizing and transforming a wide range of disciplines. This is being boosted by unprecedented integration of new real-time monitoring, sensing, control, and communication devices, which give us a wealth of data to propel the ML techniques. However, ML research is usually done in computer science to solve problems in computer vision (CV), audio recognition, among others. The same techniques used in these domains are adapted to many other different fields and areas. For example, many of these techniques are used to monitor, control, and operate cyber-physical systems (CPSs) such as tracking object systems, water distribution systems, power systems, among others. Therefore, nowadays, data-driven monitoring and control play an important role for Cyber-Physical Systems (CPSs), e.g., smart grids, autonomous automobile systems, robotic systems, etc. Jazdi (2014). For example, Duchesne et al. (2020) gives an overview of ML methods for reliability and management in energy systems. Most of such applied methods fall into classification and regression problems using supervised and unsupervised learning. For example, Marot et al. (2018) automatically segments large-scale power grids into coherent zones for the management of the grid for control operators with ML techniques. However, little research has been done in maximizing the adaptation of ML techniques across domains in a unified framework.
In this work, we claim that while adopting ML techniques solves specific problems, the way to adapt these techniques also open the door on CPSs vulnerability with dire consequences Alguliyev et al. (2018). For example, one can intentionally distort inputs according to how one applies ML method to cause an incorrect control action due to an estimation mistake. Known as adversarial attacks Kurakin et al. (2018), many studies show that outputs from classifiers can be easily changed with imperceptible changes to the input in the computer science domain Carlini & Wagner (2017); Szegedy et al. (2013). In the CPS context, this is known as a False Data Injection Attack (FDIA), where an attacker intercepts and maliciously changes the system measurements to cause harm in the real world. For instance, a cyber-attack in a power system could cause a system operator to take wrong control actions causing a blackout. Similarly, a cyber-attack could cause fatal autopilot crashes in autonomous automobile systems Banks et al. (2018); Dikmen & Burns (2017).
Adapting the ML methods from the originally proposed area to a new one (CV to CPSs, for example) is a non-trivial task that has not been thoroughly analyzed in the literature. The general consensus in the ML community says that there are no standard recipes to apply a given method across multiple domains. The dataset examples are different across domains; for example, whereas the CV dataset samples are images that are represented as three-dimensional tensors, in CPSs the dataset samples are one-dimensional tensor measurements from multiple devices with a timestamp. Then, in the literature, there are two main approaches to adapt methods to a specific domain. (1) Modify the ML learning architecture to fit the dataset samples format. For example, if the original domain is CV in which a convolutional neural network (CNN) in employed, the adapted ML technique will use as a base a fully connected neural network to handle the one-dimensional data in the targeted domain. (2) Transform the new dataset samples into the same format as the original domain to use the same ML architecture. For example, if in the original domain a CNN with a three-dimensional as input, and the targeted domain is a CPS with one-dimensional tensors, then the one-dimensional is transformed into a grid-like format to use a CNN as in the original domain. This begs the question, What are the benefits and downsides of using the first or second approach to adapt ML techniques across different domains?
To better adapt ML to different domains such as CPS systems, it is important to understand the benefits and possible downsides for different methods using particular architectures and dataset types. In this paper, we explore the benefits and downsides of using different approaches to adapt ML methods across disciplines. There are multiple ML methods. To simplify our analysis, we choose to study classifiers. This will allow us to quantify the benefits of the classifier’s accuracy. To quantify or evaluate the downsides, we analyze how different datasets and ML architecture are impacted by adversarial attacks. We will show the trade-offs of using different dataset formats and model architectures. In specific, this paper demonstrates that while CNN improves model accuracy, they are more sensitive to adversarial examples than fully connected neural networks (FCNN).

heading: 2 RELATED WORK
text: Computer vision (CV) is an area that studies how computers can gain high-level understanding from digital images or videos, which utilizes 2- and 3-dimensional data with multi-channels. Convolutional neural networks (CNN) are the most important architecture for CV and can be 2- and 3- dimensional (CNN with higher dimensions are possible but rarely used). These CNN are designed to learn filters from data where there is translation invariant. CNN can pull out this translational invariance that images inherently have. For example, if you want to recognize a cat in an image, it should not matter the cat’s location in the picture. Computer vision also deals with 3-dimensional images, for example, CT and MRI scans. To exploit these 3-dimensional data, 3-d CNN are employed.
Adversarial attacks. Deep Convolutional Neural Networks (CNNs) models are highly susceptible to adversarial examples Goodfellow et al. (2014); Moosavi-Dezfooli et al. (2017); Szegedy et al. (2013). Research on this topic has been done mainly for image classifiers and face recognition systems. These adversarial attacks or adversarial samples have drawn the community’s attention; in 2018, there was an Adversarial Attacks and Defences Competition Kurakin et al. (2018).
A lot of work to create adversarial samples has been done, including techniques such as the L-BFGS attack Szegedy et al. (2013), FGSM Goodfellow et al. (2014), and the CW attack Carlini & Wagner (2016). In specific for classifiers in the white-box context, in addition to the model knowledge, the attacker knows the set of classes Y and the set of valid inputs X to the classifier. These methods do not explicitly make use of the latent features to create adversarial examples. For example, the conventional way to create these adversarial examples is by maximizing the log-likelihood of the target class yt over a -radius ball around the original input (which is usually represented as a vector of d pixels in the range [0, 1]) Athalye et al. (2017):
arg max x′
logP ( yt|x′ ) subject to ∥∥x′ − x∥∥ p <
x′ ∈ [0, 1]d.
(1)
False data injection in power systems. Past work in the power systems area shows the vulnerability of the state estimation functionality. It is based on the observability of the system state.
Some works show different attacking approaches Mohammadpourfard et al. (2017). Based on such analysis, the impact of an attack on the system is also evaluated Mohammadpourfard et al. (2019). Mathematically, the FDIA can be expressed as:
min ma ∥∥ma − h (v̂st)∥∥22 subject to ma ≤m + a,
(2)
where m and ma are the set of original and corrupted measurements, respectively; v̂st is a vector of the state variables, h is the state estimator non-linear model, and a is the attack vector to alter the original set of measurements.

heading: 3 BACKGROUND
text: This section presents the Optimal Transport (OT) problem that will allow us to quantify how different similar two distributions or datasets are from each other.

heading: 3.1 BACKGROUND ON OPTIMAL TRANSPORT
text: Optimal transport (OT) is compares probability distributions (Villani, 2003; 2008). OT leverages the geometry of the underlying space, making them ideal for comparing distributions, shapes and point clouds (Peyré & Cuturi, 2019).
The OT problem considers a complete and separable metric space X , along with probability measures
α ∈ P(X ), β ∈ P(X ). (3)
The Kantorovich formulation Kantorovitch (1942) of the transportation problem can be writeen as:
OT(α, β) , min π∈Π(α,β) ∫ X×X c(x, y) dπ(x, y), (4)
where c(·, ·) : X ×X → R+ is a cost function (the “ground” cost), and the set of couplings Π(α, β) consists of joint probability distributions over the product space X ×X with marginals α and β, that is,
Π(α, β) , {π ∈ P(X×X ) | P1#π = α, P2#π = β}. (5)
Whenever X is equipped with a metric dX , it is natural to use it as ground cost, e.g.,
c(x, y) = dX (x, y) p (6)
for some p ≥ 1. In such case,
Wp(α, β) , OT(α, β)1/p (7)
is called the p-Wasserstein distance. The case p = 1 is also known as the Earth Mover’s Distance (Rubner et al., 2000).
The measures α and β are rarely known in practice. Instead, one has access to finite samples
{x(i)} ∈ X , {y(j)} ∈ X . (8)
In that case, one can construct discrete measures
α = n∑ i=1 aiδx(i) (9)
and
β = m∑ i=1 biδy(j) , (10)
where a, b are vectors in the probability simplex, and the pairwise costs can be compactly represented as an n×m matrix C, i.e.,
Cij = c(x (i),y(j)). (11)
In this case, Equation equation 4 becomes a linear program. Adding an entropy regularization, namely
OT (α, β) , min π∈Π(α,β) ∫ X 2 c(x, y) dπ(x, y) + H(π |α⊗ β), (12)
where
H(π | α⊗ β) = ∫ log(dπ/dα dβ) dπ (13)
is the relative entropy that results into a problem that can be solved more efficiently (Cuturi, 2013; Altschuler et al., 2017) and with better sample complexity (Genevay et al., 2019) than the original one.
We will use OT to measure the distance between datasets as was done by Alvarez-Melis & Fusi (2020), see Fig. 1. In specific, we’ll measure the distance between regular and adversarial datasets.

heading: 4 TRADE-OFFS OF DATASET TRANSFORMATIONS
text: Machine learning (ML) approaches have been extensively applied on different domains to solve unsupervised and supervised tasks Olowononi et al. (2020); Duchesne et al. (2020). Unsupervised tasks mainly encompassed clustering, and supervised tasks primarily included classification problems. In the last section, the different types of data and the most NN architectures were described. Choosing a specific NN design will give you a prior or impose a bias on the model. For example, if you use a vanilla MLP NN, each input will interact with each output. This means either you assume that all the input data is correlated or have no idea of a structure that you might exploit. If you use a CNN, you are assuming that your grid-like data are locally correlated but not further data Xu et al. (2018b); Jeong et al. (2017); Xu et al. (2017). In this way, you are imposing a bias or a prior over the model, which in this case, will be reflected in the learned kernels by the CNN. On the other hand, if your data comes in sequence, you can choose to use a RNN that will exploit this prior knowledge of the problem.
ML techniques have been used across different domains. In CPSs on power systems to do classification or regression, for example. In classification for stability, the work Alimi et al. (2020) gives an overview of the work in this area. The takeaway is that the applied methods are in supervised ML techniques to classify an event. For example, the work Wang et al. (2016) assesses the power system stability after the occurrence of a disturbance. They use a feedforward NN. For regression, ML methods have been applied to forecast. For example, the work Xu et al. (2018a) tries to predict power load with a deep-belief network.
Chen et al. (2018) used time-series data such as wind and solar generation with 5- and 60-minute resolution. They separate the data by day and then stack the time series into a matrix. So, for hourly data, they form a 24 × 24 matrix. Then a convolutional neural network is employed. Gupta et al. (2018) monitored three time-series with each PMU at every generator unit: voltage magnitude, voltage angle, and rate of change of voltage angle. The time series of all generators are stacked to form a 2-D matrix for each of the three variables. This results in three 2-D matrices. Then, these three images are combined to form a three-channel, 2-D image, which they call a heatmap. Finally, a convolutional neural network is trained on the heat maps to classify the power system state. Cai & Hill (2022) take voltage (V), active (P), and reactive (Q) power measurements at each bus. With each V, P, and Q, a matrix is formed. Then the three matrices V, P, and Q are combined to create an “RGB” image. Shi et al. (2020) converted the data as follows. Given N buses PMU phasor voltage time series with length T , two real matrices N × T are formed to get a two-channel image. Then, this is the input to the convolutional neural network. Ren et al. (2020) converted the frequency timeseries to a matrix with wavelet decomposition and polar coordinate Gramian Angular Field. Then, this grid-like data can be used as input to the convolutional neural network. All these applications of ML methods use a given model architecture and dataset format without analyzing the implications of different alternatives.
In summary, we can list the main ideas or principles by which data is transformed:
1. Single time-series
(i) The most straightforward and least principled way to convert one-dimensional data into a grid-like (or image-like) format is to resize your time-series with length T to a matrix with size n×m = T , as Wang et al. (2017) did.
(ii) A more principled way to convert one-dimensional time-series data into a grid-like format is to resize your time-series, exploiting the features of your signal. The time series can be segmented in chunks of one day, and then these chunks can be stacked into a matrix. Chen et al. (2018) followed this approach. For hourly data, for example, they formed a 24 × 24 matrix.
(iii) Given a one-dimensional time-series, a transform can be applied to every sample to obtain grid-like data from the time series. In speech recognition, they use the Gabor transform to obtain a spectrogram, for example. In power systems, Ren et al. (2020) converted the frequency time-series to a matrix with wavelet decomposition and polar coordinate Gramian Angular Field.
2. Multiple time-series
(i) When multiple time-series are available, the data conversion is carried out in a different way. A set of time series of the same variable from many sources can be stacked into a matrix. This process can be applied to every variable resulting in the same number of matrices (c). Then, these formed matrices can be stacked in a tensor to form an “image” with c channels. Examples of these approach are Gupta et al. (2018); Cai & Hill (2022); Shi et al. (2020).

heading: 4.1 DIFFERENT DATASET FORMATS
text: Given all the dataset transformations previously described, in this part we introduce a classification for different dataset types. The use of each dataset type will dictate which neural network (NN) architecture to use. Starting with the most simple dataset format, let DT1 be a dataset type one defined as
DT1 = {( x (n) m,b, y (n) ) : m = [[M ]], b = [[B]] }N n=1 , (14)
where x(n)m,b ∈ RT is the n time-series sample of variable m at location b with length T , M is the number of different measured variables, [[M ]] = {1, . . . ,M}, B is the number of different locations, [[B]] = {1, . . . , B}, N is the number of samples in the dataset, and y(n) is the label associated with the n sample.
As shown before, in the literature, they convert a vector into a matrix. To do so, we can stack the B samples x(n)m,b measuring the same variable from different locations as
V (n)m =  x (n) m,1 ...
x (n) m,B  ∈ RB×T . (15) Then, we can define a dataset type two DT2 as
DT2 = {( V (n)m , y (n) ) : m = [[M ]] }N n=1 . (16)
Similarly, in the literature they go further and create datasets with tensors. To achieve this from a dataset type two, we stack the Vm matrices in a tensor obtaining I(n) = RM×B×T . This results in a dataset type three
DT3 = {( I(n), y(n) )}N
n=1
. (17)
From the last part, some important points and questions are: (1) What are the advantages or disadvantages of using DT1 , DT2 , or DT3? (if any). (2) Do different dataset types have the same vulnerabilities? (3) Can we quantify the difference between dataset types by using Optimal Transport?

heading: 5 EXPERIMENTAL RESULTS
text: To explore the advantages and disadvantages of using different dataset types, we carry trained different classifiers with diverse dataset types and models. In specific, we use the MNIST and FashionMNIST datasets. Although these datasets are not from a different domain (e.g., CPSs), they are helpful to assess what are the implications of using them with other formats (i.e., DT1 or DT2 ) To assess the advantages, we measure the accuracy of the test regular data. To evaluate the disadvantages, we also measure the accuracy but in perturbed test data. The perturbations are done with the fast gradient sign method (Goodfellow et al., 2014). All datasets consists in examples x ∈ R28×28. According to our previous classifications, they are dataset type 2, DMNISTT2 and D Fashion T2
, respectively.
These datasets can be transformed into datasets type 1 by stacking the columns into a vector resulting in examples x ∈ R784. This would result in datasets type 1, DMNISTT1 and D Fashion T1
, respectively. We create classifiers for both dataset types DT1 and DT2 based on fully connected neural networks (Dense) and convolutional neural networks (Conv), respectively. The fully connected neural network (Dense) has two dense layers of size 784 and 128, resulting in 101, 770 parameters. We use a CNN consisting of two convolutional layers with 32 and 62 filters, respectively, each followed by 2×2 max-pooling, and a fully connected layer of size 256 resulting in 84, 553 parameters (Conv1). Also, to see that in the CNN model, the number of parameters does not affect the classifier output, we use a second CNN (Conv2) with two fully connected layers of size 256 and 128 resulting in 116, 168 parameters.
We train these classifiers in two ways: (1) we train them on regular data, and (2) we also adversary train the networks (AT or Adv) with regular and adversarial samples, with the method proposed by Madry et al. (2017).

heading: 5.1 ACCURACY AND DISTANCE BETWEEN ORIGINAL AND ADVERSARIAL EXAMPLES
text: The results for the MNIST-Fashion dataset are shown in Table 1 (with = 0.05). We perturbed both datasets DT1 and DT2 with the fast gradient descent method (FGDM) (Goodfellow et al., 2014). As result, we obtain adversarial or perturbed datsets that DPT1 and D P T2
. Then, we measure the distance between the regular and perturbed datasets with the optimal transport framework. The results are
OT (DT1 ,DPT1) = 1.05, OT (DT2 ,D P T2) = 0.84. (18)
In this case OT (DT1 ,DPT1) > OT (DT2 ,D P T2). (19) The effect of this is that adversarial examples look more similar forDT2 . We hyphotesize that this is correlated with the classifier accuracy on the perturbed dataset. If the distance between the regular and perturbed dataset is larger, then the accuracy on the adversarial trained model will be better as shown in Table 1. We repeat the same experiment for different ranging from 0 to 0.3 with increments of 0.05. The results are shown in Fig. 2. In this Figure, we make two observations. (1) The accuracy for the adversary trained model with dataset type 1 for the perturbed dataset is higher than its counterpart with dataset type 2. (2) The OT distance between the regular and perturbed datasets type 1 is bigger than the distance for dataset type 2.
The results for the MNIST dataset (with = 0.05) are shown in Table 2. The OT distances between regular and perturbed datasets are
OT (DT1 ,DPT1) = 16.24, OT (DT2 ,D P T2) = 9.64. (20)
We can see a similar effect, the distance for the dataset type 1 is larger than the distance for the dataset type 1. This results in bigger increase of accuracy in the adversary train classifier as shown in Table 2. We repeat the same experiment for different ranging from 0 to 0.3 with increments of 0.05. The results are shown in Fig. 3. In this Figure, we make two observations. (1) The accuracy for the adversary trained model with dataset type 1 for the perturbed dataset is higher than its counterpart with dataset type 2. (2) The OT distance between the regular and perturbed datasets type 1 is bigger than the distance for dataset type 2.
This shows that we must be aware when selecting the dataset transformation or ML model architecture across different domains. While using datasets type 2 improves the accuracy, it also makes models more sensible to adversarial examples.

heading: 5.2 COMPARISON OF DISTANCES BETWEEN SAMPLES FOR DIFFERENT MODELS
text: To further investigate the effect of adversarial examples for different models, we compute the distances among the original and pertubed datasets for both the FCN and the CNN. Suppose we have a
dataset with N samples D = {xn}Nn=1. Then, we can compute the distance of an example with all the remaining dataset with the OT framework. Mathematically, it is expressed as follows
emdi = OTi ( xi, {xn ∈ D : xn 6= xi} ) , for i = 1, . . . , N. (21)
With the computed distances for every sample, we define the set as EMD = {emdn}Nn=1. We can interpret this as an empirical PDF of distances. The results for the MNIST dataset for the FCN and the CNN are shown in Fig. 4. We can see in Fig. 4b that the distances between the original and perturbed examples barely move. Fig. 4a, on the other hand, shows that interestingly the perturbed distances are smaller than the original ones. This could be the reason why the CNN is more vulnerable to adversarial attacks than the FCN. In other words, the FCN is more resilient to adversarial attacks because the distribution of distances between the original and perturbed examples is not entirely overlapping as in the case of the CNN case. We can see similar results for the MNISTFasion dataset in in Fig. 5.

heading: 6 CONCLUSION
text: This paper showed that using different dataset types and neural network architectures has benefits and downsides. When adapting ML techniques across domains, practitioners must be aware of the trade-offs on selecting a given dataset type format or model architecture. While using a convolutional neural network improves the model’s accuracy, making it more vulnerable to adversarial examples. We showed that it is possible to infer how vulnerable is a given model with respect to another one by computing their corresponding OT distances between the regular and perturbed datasets. It was proven that a lower OT would result in a more vulnerable model against adversarial examples.

Review summaries:
This paper attempts to study the effect of different transformations of input datasets, and the performance of the resulting models thereof. In particular, authors consider time-series data which is then represented in (1) time-series format, (2) vectorized form, (3) tensorized form. For each of the representation, a different neural network is trained and the performance is compared. Experiments are shown on MNIST dataset.

- Draft attempts to understand the effect of converting dataset format while applying an ML technique. It argues that converting the target data into a format on which the specific ML technique was shown to be successful is suboptimal. Via simple experiments it shows that the data sample format conversion need not be advantageous.

This paper aims to study the effects of different transformation methods when processing a dataset for supervised deep learning tasks. The authors conducted one experiment to show that different data arranging methods leads to different vulnerability for adversarial attacks. In the experimental procedure, an optimal transport based method is utilized to measure the distance between a dataset and its perturbed version.