Dataset transformations trade-offs to adapt machine learning methods across domains

Machine learning-based methods have been proved to be quite successful in different domains . However , applying the same methods across domains is not a trivial task . In the literature , the most common approach is to convert a dataset into the same format as the original domain to employ the same architecture that was successful in the original domain . Although this approach is fast and convenient , we argue it is suboptimal due to the lack of tailoring to the specific problem at hand . To prove our point , ( 1 ) we exhaustively examine the dataset transformations that are used in the literature to adapt machine learning-based methods across domains ; ( 2 ) we show that these dataset transformations are not always beneficial ; and ( 3 ) we show the drawbacks of converting the dataset to adapt a Machine learning-based method to a different domain . To quantify how different the original dataset is with respect to the transformed one , we compute the dataset distances via Optimal Transport . Also , we present simulations with the original and transformed data to show that the data conversion is not always needed or beneficial . 1 INTRODUCTION . Machine learning ( ML ) methods are revolutionizing and transforming a wide range of disciplines . This is being boosted by unprecedented integration of new real-time monitoring , sensing , control , and communication devices , which give us a wealth of data to propel the ML techniques . However , ML research is usually done in computer science to solve problems in computer vision ( CV ) , audio recognition , among others . The same techniques used in these domains are adapted to many other different fields and areas . For example , many of these techniques are used to monitor , control , and operate cyber-physical systems ( CPSs ) such as tracking object systems , water distribution systems , power systems , among others . Therefore , nowadays , data-driven monitoring and control play an important role for Cyber-Physical Systems ( CPSs ) , e.g. , smart grids , autonomous automobile systems , robotic systems , etc . Jazdi ( 2014 ) . For example , Duchesne et al . ( 2020 ) gives an overview of ML methods for reliability and management in energy systems . Most of such applied methods fall into classification and regression problems using supervised and unsupervised learning . For example , Marot et al . ( 2018 ) automatically segments large-scale power grids into coherent zones for the management of the grid for control operators with ML techniques . However , little research has been done in maximizing the adaptation of ML techniques across domains in a unified framework . In this work , we claim that while adopting ML techniques solves specific problems , the way to adapt these techniques also open the door on CPSs vulnerability with dire consequences Alguliyev et al . ( 2018 ) . For example , one can intentionally distort inputs according to how one applies ML method to cause an incorrect control action due to an estimation mistake . Known as adversarial attacks Kurakin et al . ( 2018 ) , many studies show that outputs from classifiers can be easily changed with imperceptible changes to the input in the computer science domain Carlini & Wagner ( 2017 ) ; Szegedy et al . ( 2013 ) . In the CPS context , this is known as a False Data Injection Attack ( FDIA ) , where an attacker intercepts and maliciously changes the system measurements to cause harm in the real world . For instance , a cyber-attack in a power system could cause a system operator to take wrong control actions causing a blackout . Similarly , a cyber-attack could cause fatal autopilot crashes in autonomous automobile systems Banks et al . ( 2018 ) ; Dikmen & Burns ( 2017 ) . Adapting the ML methods from the originally proposed area to a new one ( CV to CPSs , for example ) is a non-trivial task that has not been thoroughly analyzed in the literature . The general consensus in the ML community says that there are no standard recipes to apply a given method across multiple domains . The dataset examples are different across domains ; for example , whereas the CV dataset samples are images that are represented as three-dimensional tensors , in CPSs the dataset samples are one-dimensional tensor measurements from multiple devices with a timestamp . Then , in the literature , there are two main approaches to adapt methods to a specific domain . ( 1 ) Modify the ML learning architecture to fit the dataset samples format . For example , if the original domain is CV in which a convolutional neural network ( CNN ) in employed , the adapted ML technique will use as a base a fully connected neural network to handle the one-dimensional data in the targeted domain . ( 2 ) Transform the new dataset samples into the same format as the original domain to use the same ML architecture . For example , if in the original domain a CNN with a three-dimensional as input , and the targeted domain is a CPS with one-dimensional tensors , then the one-dimensional is transformed into a grid-like format to use a CNN as in the original domain . This begs the question , What are the benefits and downsides of using the first or second approach to adapt ML techniques across different domains ? To better adapt ML to different domains such as CPS systems , it is important to understand the benefits and possible downsides for different methods using particular architectures and dataset types . In this paper , we explore the benefits and downsides of using different approaches to adapt ML methods across disciplines . There are multiple ML methods . To simplify our analysis , we choose to study classifiers . This will allow us to quantify the benefits of the classifier ’ s accuracy . To quantify or evaluate the downsides , we analyze how different datasets and ML architecture are impacted by adversarial attacks . We will show the trade-offs of using different dataset formats and model architectures . In specific , this paper demonstrates that while CNN improves model accuracy , they are more sensitive to adversarial examples than fully connected neural networks ( FCNN ) . 2 RELATED WORK . Computer vision ( CV ) is an area that studies how computers can gain high-level understanding from digital images or videos , which utilizes 2- and 3-dimensional data with multi-channels . Convolutional neural networks ( CNN ) are the most important architecture for CV and can be 2- and 3- dimensional ( CNN with higher dimensions are possible but rarely used ) . These CNN are designed to learn filters from data where there is translation invariant . CNN can pull out this translational invariance that images inherently have . For example , if you want to recognize a cat in an image , it should not matter the cat ’ s location in the picture . Computer vision also deals with 3-dimensional images , for example , CT and MRI scans . To exploit these 3-dimensional data , 3-d CNN are employed . Adversarial attacks . Deep Convolutional Neural Networks ( CNNs ) models are highly susceptible to adversarial examples Goodfellow et al . ( 2014 ) ; Moosavi-Dezfooli et al . ( 2017 ) ; Szegedy et al . ( 2013 ) . Research on this topic has been done mainly for image classifiers and face recognition systems . These adversarial attacks or adversarial samples have drawn the community ’ s attention ; in 2018 , there was an Adversarial Attacks and Defences Competition Kurakin et al . ( 2018 ) . A lot of work to create adversarial samples has been done , including techniques such as the L-BFGS attack Szegedy et al . ( 2013 ) , FGSM Goodfellow et al . ( 2014 ) , and the CW attack Carlini & Wagner ( 2016 ) . In specific for classifiers in the white-box context , in addition to the model knowledge , the attacker knows the set of classes Y and the set of valid inputs X to the classifier . These methods do not explicitly make use of the latent features to create adversarial examples . For example , the conventional way to create these adversarial examples is by maximizing the log-likelihood of the target class yt over a -radius ball around the original input ( which is usually represented as a vector of d pixels in the range [ 0 , 1 ] ) Athalye et al . ( 2017 ) : arg max x′ logP ( yt|x′ ) subject to ∥∥x′ − x∥∥ p < x′ ∈ [ 0 , 1 ] d. ( 1 ) False data injection in power systems . Past work in the power systems area shows the vulnerability of the state estimation functionality . It is based on the observability of the system state . Some works show different attacking approaches Mohammadpourfard et al . ( 2017 ) . Based on such analysis , the impact of an attack on the system is also evaluated Mohammadpourfard et al . ( 2019 ) . Mathematically , the FDIA can be expressed as : min ma ∥∥ma − h ( v̂st ) ∥∥22 subject to ma ≤m + a , ( 2 ) where m and ma are the set of original and corrupted measurements , respectively ; v̂st is a vector of the state variables , h is the state estimator non-linear model , and a is the attack vector to alter the original set of measurements . 3 BACKGROUND . This section presents the Optimal Transport ( OT ) problem that will allow us to quantify how different similar two distributions or datasets are from each other . 3.1 BACKGROUND ON OPTIMAL TRANSPORT . Optimal transport ( OT ) is compares probability distributions ( Villani , 2003 ; 2008 ) . OT leverages the geometry of the underlying space , making them ideal for comparing distributions , shapes and point clouds ( Peyré & Cuturi , 2019 ) . The OT problem considers a complete and separable metric space X , along with probability measures α ∈ P ( X ) , β ∈ P ( X ) . ( 3 ) The Kantorovich formulation Kantorovitch ( 1942 ) of the transportation problem can be writeen as : OT ( α , β ) , min π∈Π ( α , β ) ∫ X×X c ( x , y ) dπ ( x , y ) , ( 4 ) where c ( · , · ) : X ×X → R+ is a cost function ( the “ ground ” cost ) , and the set of couplings Π ( α , β ) consists of joint probability distributions over the product space X ×X with marginals α and β , that is , Π ( α , β ) , { π ∈ P ( X×X ) | P1 # π = α , P2 # π = β } . ( 5 ) Whenever X is equipped with a metric dX , it is natural to use it as ground cost , e.g. , c ( x , y ) = dX ( x , y ) p ( 6 ) for some p ≥ 1 . In such case , Wp ( α , β ) , OT ( α , β ) 1/p ( 7 ) is called the p-Wasserstein distance . The case p = 1 is also known as the Earth Mover ’ s Distance ( Rubner et al. , 2000 ) . The measures α and β are rarely known in practice . Instead , one has access to finite samples { x ( i ) } ∈ X , { y ( j ) } ∈ X . ( 8 ) In that case , one can construct discrete measures α = n∑ i=1 aiδx ( i ) ( 9 ) and β = m∑ i=1 biδy ( j ) , ( 10 ) where a , b are vectors in the probability simplex , and the pairwise costs can be compactly represented as an n×m matrix C , i.e. , Cij = c ( x ( i ) , y ( j ) ) . ( 11 ) In this case , Equation equation 4 becomes a linear program . Adding an entropy regularization , namely OT ( α , β ) , min π∈Π ( α , β ) ∫ X 2 c ( x , y ) dπ ( x , y ) + H ( π |α⊗ β ) , ( 12 ) where H ( π | α⊗ β ) = ∫ log ( dπ/dα dβ ) dπ ( 13 ) is the relative entropy that results into a problem that can be solved more efficiently ( Cuturi , 2013 ; Altschuler et al. , 2017 ) and with better sample complexity ( Genevay et al. , 2019 ) than the original one . We will use OT to measure the distance between datasets as was done by Alvarez-Melis & Fusi ( 2020 ) , see Fig . 1 . In specific , we ’ ll measure the distance between regular and adversarial datasets .

Summary: - Draft attempts to understand the effect of converting dataset format while applying an ML technique. It argues that converting the target data into a format on which the specific ML technique was shown to be successful is suboptimal. Via simple experiments it shows that the data sample format conversion need not be advantageous.