Dataset transformations trade-offs to adapt machine learning methods across domains

Machine learning-based methods have been proved to be quite successful in different domains . However , applying the same methods across domains is not a trivial task . In the literature , the most common approach is to convert a dataset into the same format as the original domain to employ the same architecture that was successful in the original domain . Although this approach is fast and convenient , we argue it is suboptimal due to the lack of tailoring to the specific problem at hand . To prove our point , ( 1 ) we exhaustively examine the dataset transformations that are used in the literature to adapt machine learning-based methods across domains ; ( 2 ) we show that these dataset transformations are not always beneficial ; and ( 3 ) we show the drawbacks of converting the dataset to adapt a Machine learning-based method to a different domain . To quantify how different the original dataset is with respect to the transformed one , we compute the dataset distances via Optimal Transport . Also , we present simulations with the original and transformed data to show that the data conversion is not always needed or beneficial . 1 INTRODUCTION . Machine learning ( ML ) methods are revolutionizing and transforming a wide range of disciplines . This is being boosted by unprecedented integration of new real-time monitoring , sensing , control , and communication devices , which give us a wealth of data to propel the ML techniques . However , ML research is usually done in computer science to solve problems in computer vision ( CV ) , audio recognition , among others . The same techniques used in these domains are adapted to many other different fields and areas . For example , many of these techniques are used to monitor , control , and operate cyber-physical systems ( CPSs ) such as tracking object systems , water distribution systems , power systems , among others . Therefore , nowadays , data-driven monitoring and control play an important role for Cyber-Physical Systems ( CPSs ) , e.g. , smart grids , autonomous automobile systems , robotic systems , etc . Jazdi ( 2014 ) . For example , Duchesne et al . ( 2020 ) gives an overview of ML methods for reliability and management in energy systems . Most of such applied methods fall into classification and regression problems using supervised and unsupervised learning . For example , Marot et al . ( 2018 ) automatically segments large-scale power grids into coherent zones for the management of the grid for control operators with ML techniques . However , little research has been done in maximizing the adaptation of ML techniques across domains in a unified framework . In this work , we claim that while adopting ML techniques solves specific problems , the way to adapt these techniques also open the door on CPSs vulnerability with dire consequences Alguliyev et al . ( 2018 ) . For example , one can intentionally distort inputs according to how one applies ML method to cause an incorrect control action due to an estimation mistake . Known as adversarial attacks Kurakin et al . ( 2018 ) , many studies show that outputs from classifiers can be easily changed with imperceptible changes to the input in the computer science domain Carlini & Wagner ( 2017 ) ; Szegedy et al . ( 2013 ) . In the CPS context , this is known as a False Data Injection Attack ( FDIA ) , where an attacker intercepts and maliciously changes the system measurements to cause harm in the real world . For instance , a cyber-attack in a power system could cause a system operator to take wrong control actions causing a blackout . Similarly , a cyber-attack could cause fatal autopilot crashes in autonomous automobile systems Banks et al . ( 2018 ) ; Dikmen & Burns ( 2017 ) . Adapting the ML methods from the originally proposed area to a new one ( CV to CPSs , for example ) is a non-trivial task that has not been thoroughly analyzed in the literature . The general consensus in the ML community says that there are no standard recipes to apply a given method across multiple domains . The dataset examples are different across domains ; for example , whereas the CV dataset samples are images that are represented as three-dimensional tensors , in CPSs the dataset samples are one-dimensional tensor measurements from multiple devices with a timestamp . Then , in the literature , there are two main approaches to adapt methods to a specific domain . ( 1 ) Modify the ML learning architecture to fit the dataset samples format . For example , if the original domain is CV in which a convolutional neural network ( CNN ) in employed , the adapted ML technique will use as a base a fully connected neural network to handle the one-dimensional data in the targeted domain . ( 2 ) Transform the new dataset samples into the same format as the original domain to use the same ML architecture . For example , if in the original domain a CNN with a three-dimensional as input , and the targeted domain is a CPS with one-dimensional tensors , then the one-dimensional is transformed into a grid-like format to use a CNN as in the original domain . This begs the question , What are the benefits and downsides of using the first or second approach to adapt ML techniques across different domains ? To better adapt ML to different domains such as CPS systems , it is important to understand the benefits and possible downsides for different methods using particular architectures and dataset types . In this paper , we explore the benefits and downsides of using different approaches to adapt ML methods across disciplines . There are multiple ML methods . To simplify our analysis , we choose to study classifiers . This will allow us to quantify the benefits of the classifier ’ s accuracy . To quantify or evaluate the downsides , we analyze how different datasets and ML architecture are impacted by adversarial attacks . We will show the trade-offs of using different dataset formats and model architectures . In specific , this paper demonstrates that while CNN improves model accuracy , they are more sensitive to adversarial examples than fully connected neural networks ( FCNN ) . 2 RELATED WORK . Computer vision ( CV ) is an area that studies how computers can gain high-level understanding from digital images or videos , which utilizes 2- and 3-dimensional data with multi-channels . Convolutional neural networks ( CNN ) are the most important architecture for CV and can be 2- and 3- dimensional ( CNN with higher dimensions are possible but rarely used ) . These CNN are designed to learn filters from data where there is translation invariant . CNN can pull out this translational invariance that images inherently have . For example , if you want to recognize a cat in an image , it should not matter the cat ’ s location in the picture . Computer vision also deals with 3-dimensional images , for example , CT and MRI scans . To exploit these 3-dimensional data , 3-d CNN are employed . Adversarial attacks . Deep Convolutional Neural Networks ( CNNs ) models are highly susceptible to adversarial examples Goodfellow et al . ( 2014 ) ; Moosavi-Dezfooli et al . ( 2017 ) ; Szegedy et al . ( 2013 ) . Research on this topic has been done mainly for image classifiers and face recognition systems . These adversarial attacks or adversarial samples have drawn the community ’ s attention ; in 2018 , there was an Adversarial Attacks and Defences Competition Kurakin et al . ( 2018 ) . A lot of work to create adversarial samples has been done , including techniques such as the L-BFGS attack Szegedy et al . ( 2013 ) , FGSM Goodfellow et al . ( 2014 ) , and the CW attack Carlini & Wagner ( 2016 ) . In specific for classifiers in the white-box context , in addition to the model knowledge , the attacker knows the set of classes Y and the set of valid inputs X to the classifier . These methods do not explicitly make use of the latent features to create adversarial examples . For example , the conventional way to create these adversarial examples is by maximizing the log-likelihood of the target class yt over a -radius ball around the original input ( which is usually represented as a vector of d pixels in the range [ 0 , 1 ] ) Athalye et al . ( 2017 ) : arg max x′ logP ( yt|x′ ) subject to ∥∥x′ − x∥∥ p < x′ ∈ [ 0 , 1 ] d. ( 1 ) False data injection in power systems . Past work in the power systems area shows the vulnerability of the state estimation functionality . It is based on the observability of the system state . Some works show different attacking approaches Mohammadpourfard et al . ( 2017 ) . Based on such analysis , the impact of an attack on the system is also evaluated Mohammadpourfard et al . ( 2019 ) . Mathematically , the FDIA can be expressed as : min ma ∥∥ma − h ( v̂st ) ∥∥22 subject to ma ≤m + a , ( 2 ) where m and ma are the set of original and corrupted measurements , respectively ; v̂st is a vector of the state variables , h is the state estimator non-linear model , and a is the attack vector to alter the original set of measurements . 3 BACKGROUND . This section presents the Optimal Transport ( OT ) problem that will allow us to quantify how different similar two distributions or datasets are from each other . 3.1 BACKGROUND ON OPTIMAL TRANSPORT . Optimal transport ( OT ) is compares probability distributions ( Villani , 2003 ; 2008 ) . OT leverages the geometry of the underlying space , making them ideal for comparing distributions , shapes and point clouds ( Peyré & Cuturi , 2019 ) . The OT problem considers a complete and separable metric space X , along with probability measures α ∈ P ( X ) , β ∈ P ( X ) . ( 3 ) The Kantorovich formulation Kantorovitch ( 1942 ) of the transportation problem can be writeen as : OT ( α , β ) , min π∈Π ( α , β ) ∫ X×X c ( x , y ) dπ ( x , y ) , ( 4 ) where c ( · , · ) : X ×X → R+ is a cost function ( the “ ground ” cost ) , and the set of couplings Π ( α , β ) consists of joint probability distributions over the product space X ×X with marginals α and β , that is , Π ( α , β ) , { π ∈ P ( X×X ) | P1 # π = α , P2 # π = β } . ( 5 ) Whenever X is equipped with a metric dX , it is natural to use it as ground cost , e.g. , c ( x , y ) = dX ( x , y ) p ( 6 ) for some p ≥ 1 . In such case , Wp ( α , β ) , OT ( α , β ) 1/p ( 7 ) is called the p-Wasserstein distance . The case p = 1 is also known as the Earth Mover ’ s Distance ( Rubner et al. , 2000 ) . The measures α and β are rarely known in practice . Instead , one has access to finite samples { x ( i ) } ∈ X , { y ( j ) } ∈ X . ( 8 ) In that case , one can construct discrete measures α = n∑ i=1 aiδx ( i ) ( 9 ) and β = m∑ i=1 biδy ( j ) , ( 10 ) where a , b are vectors in the probability simplex , and the pairwise costs can be compactly represented as an n×m matrix C , i.e. , Cij = c ( x ( i ) , y ( j ) ) . ( 11 ) In this case , Equation equation 4 becomes a linear program . Adding an entropy regularization , namely OT ( α , β ) , min π∈Π ( α , β ) ∫ X 2 c ( x , y ) dπ ( x , y ) + H ( π |α⊗ β ) , ( 12 ) where H ( π | α⊗ β ) = ∫ log ( dπ/dα dβ ) dπ ( 13 ) is the relative entropy that results into a problem that can be solved more efficiently ( Cuturi , 2013 ; Altschuler et al. , 2017 ) and with better sample complexity ( Genevay et al. , 2019 ) than the original one . We will use OT to measure the distance between datasets as was done by Alvarez-Melis & Fusi ( 2020 ) , see Fig . 1 . In specific , we ’ ll measure the distance between regular and adversarial datasets .

Summary: This paper aims to study the effects of different transformation methods when processing a dataset for supervised deep learning tasks. The authors conducted one experiment to show that different data arranging methods leads to different vulnerability for adversarial attacks. In the experimental procedure, an optimal transport based method is utilized to measure the distance between a dataset and its perturbed version.