Year: 2021

Abstract:
We reveal an intriguing connection between adversarial attacks and cycle monotone maps, also known as optimal transport maps. Based on this finding, we developed a novel method named source fiction for semi-supervised optimal transportbased domain adaptation. In our algorithm, instead of mapping from target to the source domain, optimal transport maps target samples to the set of adversarial examples. The trick is that these adversarial examples are labeled target samples perturbed to look like source samples for the source domain classifier. Due to the cycle monotonicity of adversarial attacks, optimal transport can naturally approximate this transformation. We conduct experiments on various datasets and show that our method can notably improve the performance of optimal transport methods in semi-supervised domain adaptation.

text: We reveal an intriguing connection between adversarial attacks and cycle monotone maps, also known as optimal transport maps. Based on this finding, we developed a novel method named source fiction for semi-supervised optimal transportbased domain adaptation. In our algorithm, instead of mapping from target to the source domain, optimal transport maps target samples to the set of adversarial examples. The trick is that these adversarial examples are labeled target samples perturbed to look like source samples for the source domain classifier. Due to the cycle monotonicity of adversarial attacks, optimal transport can naturally approximate this transformation. We conduct experiments on various datasets and show that our method can notably improve the performance of optimal transport methods in semi-supervised domain adaptation.

heading: 1 INTRODUCTION
text: Optimal Transport (OT) is a powerful framework to solve mass moving problems for probability distributions. Over the past decades, it has been successfully applied in mathematics (Ferradans et al., 2014), economics (Reich, 2013) and machine learning (Arjovsky et al., 2017; Mroueh, 2019; Solomon et al., 2015; Colombo et al., 2021). The key property of OT maps is a cycle monotonicity (McCann, 1995), such maps cannot be improved; it is impossible to perturb it and get something more economical (Villani, 2008, M5).
In machine learning, OT finds a rich application in domain adaptation (DA) problem (Courty et al., 2015; Perrot et al., 2016; Rakotomamonjy et al., 2020). In DA, the task is to learn a model fθ, e.g., a classifier, from a source domain Ωs to perform well on a different (related) target domain Ωt (Ben-David et al., 2010a). The conventional OT approach for domain adaptation is to use OT map Ωt → Ωs to transform target domain Ωt to source Ωs and then apply the model fθ to mapped samples.
With few labeled samples in target domain Ωsemi we can apply semi-supervised domain adaptation (Wang & Deng, 2018). For semi-supervised OT, labels are used to prevent the transportation of the target sample into the source sample from another class (Courty et al., 2016; Yan et al., 2018). But labels do not provide supervision for the map between two domains because supervision involves input-output pairs, and labels do not give such pairwise matching between source and target samples. Moreover, supervision for OT has to involve a cycle monotone matching between source and target samples and save the class-wise structure.
In our paper, we propose an algorithm we call source fiction that provides cycle monotone supervision to OT and improves the performance of OT in semi-supervised DA. The core of this algorithm is our finding that adversarial attacks (Szegedy et al., 2014), more precisely Iterative Fast Sign Gradient Descent (Goodfellow et al., 2014, FSGD) are cycle monotone transformations over the dataset with quadratic cost.
We propose to replace source domain Ωs with domain Ωf that is constructed by the adversarial attack on labeled samples in target domain Ωsemi to be accurately classified by source domain classifier fθ. Since domain Ωf is a cycle monotone transformation of the target domain Ωsemi, it becomes natural to use Ωsemi → Ωf as supervision for OT mapping and then use the resulted map to transport from target domain without labels to Ωf .
The main contributions of this paper are: (i) We prove that an adversarial attack produces cycle monotone transformation of the dataset (Section 3); (ii) We propose a novel source fiction algorithm for OT-based semi-supervised domain adaptation. The algorithm incorporates the adversarial attack to the conventional pipeline of OT for DA (Section 4).

heading: 2 BACKGROUND AND RELATED WORK

heading: 2.1 ADVERSARIAL ATTACKS
text: Adversarial examples are samples that are similar to the true samples D(x, x′) ≤ , but “fool” a selected classifier and tend it to make incorrect predictions argmax p(y | x′) 6= ytrue (Szegedy et al., 2014). The phenomenon of the vulnerability of machine learning models to adversarial examples breeds a great deal of concern in many learning scenarios (Papernot et al., 2017; Yuan et al., 2019; Schott et al., 2019; Xie et al., 2017).
A large body of work on adversarial attacks exists; in our paper, we consider Fast Sign Gradient Descent (FSGD) by Goodfellow et al. (2014); iterative version of FSGD can be presented as
x′0 = x, x ′ i+1 = clipx, {x′i − α sign (∇xL (θ, x′i, y))} (1)
With sample x, the target label y, and classifier f with parameters θ, we can obtain adversarial examples using gradient descent, by solving the min-max problem, maximizing the loss L and minimizing perturbation for a sample x in some class y with respect to some perturbation size .
Newly, various properties of adversarial examples were studied (Petrov & Hospedales, 2019; Papernot et al., 2016; Ilyas et al., 2019), and applications of adversarial examples for models accuracy improvements were proposed (Xie et al., 2019; Yang et al., 2020).
Previously connection of OT and adversarial examples were studied in the context of robustness problems (Pydi & Jog, 2020; Bouniot et al., 2021). Wong et al. (2019) proposed the idea of Wasserstein adversarial attack with Sinkhorn iterations; this algorithm allows to find adversarial perturbations with respect to Wasserstein ball.

heading: 2.2 OPTIMAL TRANSPORT
text: OT aims at finding a solution to transfer mass from one distribution to another with the least effort. Monge’s problem was the first example of the OT problem and can be formally expressed as follows:
inf T#µ=ν ∫ Ωµ c(x, T (x))µ(x)dx (2)
The Monge’s formulation of OT aims at finding a mapping T : Ωµ → Ων of the two probability measures µ and ν and a cost function c : Ωµ × Ων → R+, where T#µs = νt represents the mass preserving push forward operator. In Monge’s formulation, T cannot split the mass from a single point. The problem is that with such constraints, the mapping T may not even exist.
To avoid this problem, Kantorovitch proposed a relaxation (Villani, 2008). Instead of obtaining a mapping, the goal is to seek a joint distribution over the source and the target that determines how the mass is allocated. For a given cost function c : Ωµ × Ων → R+, the primal Kantorovitch formulation can be expressed as the following problem:
min γ∈Π(µ,ν) {∫ Ωµ×Ων c(x,y)dγ(x,y) = E(x,y)∼γ [c(x,y)] } (3)
In primal Kantorovitch formulation, we look for a joint distribution γ with µ and µ as marginals that minimize the expected transportation cost. If the independent distribution γ(x,y) = µ(x)ν(y) respects the constraints, linear program is convex and always has a solution for a semi-continuous c:
Π (µ, ν) = { γ ∈ P (Ωµ,Ων) : ∫ γ(x,y)dy = µ(x), ∫ γ(x,y)dx = ν(y) } (4)
The primal Kantorovitch formulation can be also presented in dual form as stated by the RockafellarFenchel theorem (Villani, 2008):
max φ∈C(Ωµ),ψ∈C(Ων)
{∫ φdµ+ ∫ ψdν | φ(x) + ψ(y) ≤ c(x,y) } (5)
After finding a solution to the transport problem, OT provides a measure of dissimilarity between the two distributions. This similarity is also called the Wasserstein distance (Villani, 2008):
Wp (µs, νt) = min γ∈Π(µ,ν) {∫ Ωµ×Ων c(x,y)dγ(x,y) } 1 p
(6)
where c(x,y) = ‖x− y‖p and p > 1. The Wasserstein distance encodes the geometry of the space through the optimization problem and can be used on any distribution of mass.
The main geometric property of the OT is cycle monotonicity. Informally, a c-cycle monotone plan is a plan that cannot be improved, for all points x0 . . . xi, y0 . . . yi holds:
N∑ i=1 c (xi,yi) ≤ N∑ i=1 c (xi,yi+1) (7)
It is impossible to perturb it and get something more economical (Villani, 2008, M5).

heading: 2.3 OPTIMAL TRANSPORT FOR DOMAIN ADAPTATION
text: Domain Adaptation: DA techniques try to make the target domain Ωt samples Xt = (xti) Nt i=1 be closer to the source domain Ωs samples Xs = (xsi ) Ns i=1 to make the source classifier fθ perform accurately on the target domain (Ben-David et al., 2010a;b; Germain et al., 2013). In semi-supervised settings, target domain Ωsemi consists of unlabeled samples Xt = (xti) Nt i=1 and small number of samples Xl = (xli) Nl i=1 associated with labels Yl = (y l i) Nl i=1. These labels can be used to train DA algorithm match target samples with source samples that have the same labels.
Different discrete OT algorithms can be applied for domain adaptation (Courty et al., 2015). For example, Earth Mover’s Distance (EMD) (Courty et al., 2015; Flamary et al., 2021) solves the most typical primal Kantorovich OT problem (equation 4). This method seeks an optimal coupling γ, which minimizes the displacement cost between two domains with respect to some distance.
Sinkhorn algorithm (Cuturi, 2013) and its variations with a group lasso regularization (L1L2) and Laplacian regularization (L1LP) (Courty et al., 2015) are OT algorithms that apply lightspeed computation of OT in DA. Perrot et al. (2016) proposed Linear OT mapping estimators (MapT) that jointly learn the coupling γ equation 4 and transport map T linked to the original Monge problem equation 2.
In semi-supervised DA with OT, labels are used to penalty the transport Ωsemi(γ) = 〈γ,M〉 by a ns × nt cost matrix M〉 where M(i, j) = 0 when ysi = ytj and +∞ otherwise (Courty et al., 2016; Yan et al., 2018). In domain adaptation with label and target shift, OT was used to align probability distributions even for a few domains (Redko et al., 2019; Rakotomamonjy et al., 2020). Most of these algorithms are presented in POT library (Flamary et al., 2021), which provides state-of-the-art OT algorithms to solve the domain adaptation.
Discrete OT: Different discrete OT algorithms can be applied for domain adaptation (Courty et al., 2015). For example, Earth Mover’s Distance (EMD) (Courty et al., 2015; Flamary et al., 2021) solves the most typical primal Kantorovich OT problem (equation 4). This method seeks an optimal coupling γ, which minimizes the displacement cost between two domains with respect to some distance.
Neural OT: The connection between OT and deep networks was proposed for unsupervised domain adaptation (Damodaran et al., 2018) and representation transfer between teacher and student in transfer learning (Li et al., 2020). It was also shown that mini-batch learning that is correctly applied in deep learning can be employed to OT for efficient measure between data distributions (Fatras et al., 2021).
Recently, there has been a solid push to incorporate Input Convex Neural Networks (ICNNs) (Amos et al., 2017) in OT problems. According to Rockafellar’s Theorem (Rockafellar, 1966), every cyclical monotone mapping g is contained in a sub-gradient of some convex function f : X → R. Furthermore, according to Brenier’s Theorem (Theorem 1.22 of Santambrogio (2015)), these gradients uniquely solve the Monge problem equation 2. Following these theorems, a range of approaches explored ICNNs as parameterized convex potentials in dual Kantorovich problem equation 5 (Taghvaei & Jalali, 2019; Makkuva et al., 2020).
Further development of this approach enabled the construction of the non-minimax Wasserstein-2 generative framework (Korotin et al., 2019) that can solve domain adaptation and Wasserstein-2 Barycenters estimation (Fan et al., 2020; Korotin et al., 2021b). Compared to discrete OT, neural methods provide generalized OT methods that can ensure out-of-sample estimates.
In our paper, in comparison to this method, we do not propose a new domain adaptation algorithm, new OT algorithm, or adversarial attack and defense based on OT. We presented a connection between OT and adversarial attacks and showed how this connection can be used for semi-supervised OT.

heading: 3 CYCLE MONOTONICITY OF ADVERSARIAL ATTACKS
text: This section demonstrates the cyclical monotonicity of adversarial attacks. We prove that with mild assumptions on the attack, it is cycle monotone w.r.t. the quadratic cost c(x, y) = 12‖x− y‖ 2.
Lemma 1 (Cycle monotonicity of small perturbations of a dataset.). Let x1, . . . , xN ∈ RD be a dataset of N distinct samples. Let x′1, . . . , x ′ N be its ≤ -perturbation, i.e. ‖xn − x′n‖ ≤ for all n = 1, 2, . . . , N . Assume that ≤ 12 minn1,n2 ‖xn1 − xn2‖, i.e. the perturbation does not exceed 12 of the minimal pairwise distance between samples. Then for all K and 1, N it holds:
K∑ k=1 1 2 ‖xnk − x′nk‖ 2 ≤ K∑ k=1 1 2 ‖xnk − x′nk+1‖ 2 (8)
i.e. set (x1, x′1), . . . , (xN , x ′ N ) or, equivalently, the map xk 7→ xk′ is cycle monotone.
Proof. Due to triangle inequality for ‖ · ‖, we have
‖xnk − x′nk+1‖ ≥ ‖xnk − xnk+1‖︸ ︷︷ ︸ ≥2 −‖xnk+1 − x′nk+1‖︸ ︷︷ ︸ ≤ = . (9)
Taking the square of both sides and summing equation 9 for k = 1, 2, . . . ,K yields
K∑ k=1 ‖xnk − x′nk+1‖ 2 ≥ K∑ k=1 2 = K 2. (10)
Due to the assumptions of the lemma, the following inequality holds true:
K∑ k=1 ‖xnk − x′nk‖ 2 ≤ K∑ k=1 2 ≤ K 2. (11)
We combine equation 10 and equation 11 to obtain
K∑ k=1 ‖xnk − x′nk‖ 2 ≤ K∑ k=1 ‖xnk − x′nk+1‖ 2,
which is equivalent to K∑ k=1 c(xnk , x ′ nk ) ≤ K∑ k=1 c(xnk , x ′ nk+1 ), (12)
and yield cycle monotonicity w.r.t. quadratic cost c(x, y) = 12‖x− y‖ 2.
Adversarial attacks are small perturbations of the dataset, therefore, we immediately obtain
Corollary 1. Let x1, . . . , xN ∈ RD be a dataset ofN distinct samples. Then any adversarial attack xn 7→ x′n on the dataset with ≤ 12 minn1,n2 ‖xn1 − xn2‖ is cycle monotone.
Corollary 1 suggests using the optimal map to transform target domain Ωt to domain Ωf formed by the cycle-monotone adversarial attack.

heading: 4 SOURCE FICTION ALGORITHM
text: Algorithm 1: Source fiction for optimal domain adaptation Input: Classifier fθ, optimal transport algorithm OT, source Ωs, labeled Xl and unlabeled Xt samples in target domain Ωsemi, empty set Xf , perturbations size Initialize: ≤ 12 minn1,n2
‖xn1 − xn2‖ for xn in Xl Pretrain classifier fθ on source Ωs domain Xs Ys pairs. for x, y in (Xl, Yl) do
for steps n, . . . , N do x′0 = x, x ′ n+1 = clipx, {x′n − α sign (∇xL (θ, x′n, y))} end for Append x′ examples to Xf
end for Return: Domain Ωf Find a map Ωsemi → Ωf using OT. Apply the classifier fθ to samples Xf mapped from Xt.
In the previous section, we considered that adversarial attack applies cycle-monotone transformation over the dataset. In this section, we propose an application of this property for DA. To manage this property, we propose to replace the source domain with the domain that is cycle monotone to the target domain, and at the same time, accurately classifying by source domain classifier. Due to the vulnerability of machine learning models to perturbation, we can turn any sample into accurately classified in practice, in the same way we create adversarial examples.
In our method, we propose to use a targeted FSGD adversarial attack (equation 1) to source domain classifier fθ, with target label y equal to the true class of the given sample. Such attack adds to the image features of the class to which the image really belongs (Ilyas et al., 2019). Applying attack using labeled target samples Xl = (xli) Nl i=1 with y value in equation 1 equal to the true class, we obtain a new domain Ωf with samples Xf = (x f i ) Nl i=1 that are examples of cycle monotone transformation of the target domain samples, and at the same time accurately classifying by source
domain classifier fθ. Following corollary 1 to obtain monotonicity, the size of perturbation is set ≤ 12 minn1,n2 ‖xn1 − xn2‖ for all xn in Xl.
As a result, for each sample in the labeled target domain, we obtain a corresponding sample in Ωf . While Ωf is cycle monotone to the target domain, we have a low quadratic cost between each target sample and its corresponding sample in Ωf and a higher quadratic cost between this sample and all other samples in Ωf . To apply adaptation of classifier fθ on the target domain samples without labels Ωt, we use OT to find a map between Ωsemi to Ωf , and then apply transport on samples without labels Ωt.
Figure 1 demonstrates the difference between standard DA by OT and DA using source fiction. The pipeline of learning OT for DA using source fiction Ωf is presented in Algorithm 1.

heading: 5 EXPERIMENTS
text: In this section, we present details and the results of the experiments. The goal of our experiments is to demonstrate that our algorithm can improve the performance of fundamental discrete OT algorithms and novel neural networks based OT methods.

heading: 5.1 DATASETS
text: Digits datasets: We evaluated our method on Digits datasets MNIST (LeCun & Cortes, 2010), USPS (Hull, 1994), SVNH (Netzer et al., 2011), and MNIST-M (Ganin & Lempitsky, 2015). The experiments are conducted using two settings on each dataset: (1) optimizing OT using available labels to find a map between target and source domains and (2) optimizing OT to find a map between target and source fiction domains. Each dataset consists of 10 classes of digits images with different numbers of train and test samples.
Modern Office-31 dataset: Besides the Digits dataset that consists of only ten classes in each domain, we tested the performance of OT with the source fiction method on the Modern Office-31 dataset (Ringwald & Stiefelhagen, 2021). The Modern Office-31 dataset is one of the most extensive and diverse datasets for domain adaptation, with 31 classes in three domains: Amazon (A), Synthetic (S), and Webcam (W). Compared to the Digits and original Office-31 dataset (Saenko et al., 2010), this dataset includes synthetic to real transfer tasks, which is problematic. In our experiments, we used DLSR (D) domain from the original Office-31 to properly estimate the proposed algorithm.

heading: 5.2 SETTINGS
text: Source classifier: For the source domain classifier, we trained ResNet50 (He et al., 2016) to achieve 90+ accuracy on the test set of each domain in Digits and Modern Office-31 dataset. The classifier was trained using Adam (Ruder, 2016) optimizer with 1e-3 learning rate, the size of latent space before the output layer was set equal to 2048. After training, we applied domain adaptation by moving mass in the latent space of the source classifier.
Discrete OT: We tested several OT algorithms that apply semi-supervised adaptation: EMD, Sinkhorn, SinkhornL1L2, SinkhornLPL2, and MapT. For experiments, we used the POT library with quadratic cost distance in each method. Regularization size was equal to 4 for Sinkhorn, SinkhornL1L2, SinkhornLPL2, and MapT; all other hyperparameters were similar to the default, presented in POT. For source fiction, we used 50 steps FSGD with equal to 0.45, we find that such value allows to achieve strong perturbations and at the same time mostly satisfies ≤ 12 minn1,n2
‖xn1 − xn2‖ requirement for all domains. The results are presented in Table 1. All results are averaged over ten runs with different sets of labeled samples in the target domain. The top part of the table represents standard settings the bottom part presents results using source fiction. It is important to note that for mapping target to source, available target domain labels were used in OT solvers to apply regularization (Courty et al., 2015, M5).
Neural OT: Following recent benchmarking results of neural OT (Korotin et al., 2021a), we choose two methods to apply domain adaptation: W2GN (Korotin et al., 2019) and MM:R (Nhan Dam et al., 2019; Korotin et al., 2021a). As potentials φ and ψ in neural OT methods, we used Dense
ICNN (Korotin et al., 2019) with three hidden layers [64, 64, 32] for W2GN and MM:R methods. Potentials were pertained to apply invariant transformation using Adam optimizer with lr equal to 1e-4. For Wasserstein’s objective, potentials were trained 300 epochs with Adam optimizer and lr equal to 1e-3. The results are presented in Table 2, the top part of the table represents standard settings with 10 and 100 known labels per class, and the bottom part presents results using source fiction. In setting with the target to source mapping, labels were used to build class-wise pairs.

heading: 5.3 RESULTS
text: Our results show that adversarially based source fiction improves the performance of OT. For Digits datasets, our approach achieves improvements on all domains, even on MNIST-SVHN domain adaptation, which is a complicated transformation task. Generally, the simplest EMD method achieved less accuracy and MapT method accuracy slightly higher for all domains; group lasso and Laplacian regularizations for Sinkhorn did not provide notable improvements. The mean and std of our results are less than 10% for all experiments. On average, our method achieves mean improvement equal to 32.7 for all adaptation tasks, with min improvement value equal to 4.1 and max value 70.4. The results with only 3 known labels in class presented in Appendix 6
In our opinion, our method advances the performance while OT applications for mass moving assume closeness of source Ωs and target Ωt distributions (Lee et al., 2019). This assumption does not hold on many empirical datasets, especially in some Digits and Modern Office-31 domains.
While OT maps are cycle monotone, i.e., exhibit a specific structure of the map, thus, transportation Ωt → Ωs via OT maps might not apply to some problems, see (Courty et al., 2015, Figure 3) for counter-examples. Also, discrete OT techniques are susceptible to regularization terms (Courty et al., 2015; Dessein et al., 2018; Paty & Cuturi, 2020) and require special scaling (Meng et al., 2021).
As shown in our results, the performance of Neural OT methods is less dependent on the monotonicity of the target domain. It is known that neural networks are data-intensive; using the source fiction domain as a target, we have less training data, which makes optimization harder. However, in a case with a higher number of labeled samples in the target domain, source fiction outperforms the standard settings for Neural OT.
While adversarial examples via FSGD are noisy, there appear problems with learning Neural OT on this data. We hypothesize that adversarial training (Goodfellow et al., 2014) for a robust source classifier can be a solution for this problem. Adversarial attack on robust models allows interpolation between classes and the generation of new data points that are close to human perception (Tsipras et al., 2019). It has been shown that this property can be used for image generation, image-toimage translation, in-painting, and super resolution (Santurkar et al.). With robust source classifier source fiction domain will contain less noisy samples, which can improve the accuracy of Neural OT methods.

heading: 5.4 ABLATION STUDY
text: In this section, we show the adaptation results with different values in the FSGD algorithm. We evaluated few transportation tasks: MNIST to USPS, USPS to MNIST, MNIST to M-MNIST, and MNIST to SVHN datasets with different values of . The value of 12 minn1,n2
‖xn1 − xn2‖ is various for different datasets, for SVHN this value is 0.74, for MNIST is 0.29, for M-MNIST is 10.9 and for USPS is equal to 0.85. These values were computed in the latent space of the ResNet50 classifier trained on the corresponding dataset. For each domain, we applied source fiction with different values of and then fit OT to find a map between target and source fiction domain. Our results
show a trade-off between the size of perturbations and the cyclical monotonicity of the resulted domain. See the adaptation accuracy with different in Figure 2. It can be noticed that when the value becomes larger than 12 min distance between samples, the accuracy of adaptation decreases since the set becomes less monotone. However, with a small value of , the adaptation achieves the highest accuracy.

heading: 6 CONCLUSION
text: We propose an algorithm that modifies domain adaptation toward making target data closer to the domain formed by an adversarial attack. We conducted a range of experiments on different datasets and showed that the OT method works more efficiently in conjugate with our method. In some domains for discrete OT methods, adaptation with source fiction achieved dramatic improvement. The main limitation of our approach is that necessary to have access to the labels in the target domain.
Multi-domain adaptation: Usually, in DA, it is necessary to modify classifier architecture and train separately on each target domain (Ganin & Lempitsky, 2015; Long et al., 2018; 2015; 2017; Gretton et al., 2012). OT solves DA by moving target domains samples closer to the source data, and it is unnecessary to change or fine-tune the source classifier. In this paper, we presented an algorithm that improves OT performance, which means that a single source domain classifier can be used to make predictions on a range of target domains more effectively.
Data privacy: Nowadays, data are shared on separate devices and usually contain personal information, which is inefficient for data transmission and may violate data privacy. Liang et al. (2020) address a challenging DA setting without access to the source data for higher privacy. In our method, we adapt the source classifier to the new domain without access to the source data, which can also help avoid privacy issues.
Future work: Our method admits many straightforward applications:
• Neural OT for adversarial examples generation; fast generation of shareable adversarial examples in a framework similar to Attack-Inspired GAN (Bai et al., 2021) with additional supervising using gradient-based adversarial examples. See experiments using discrete OT for this task in Appendix A.1.
• In our settings, discrete OT methods showed impressive results in domain adaptation with a low number of labeled samples; this ability can be used for low data image recognition, where OT learns to map test dataset to the accurately classifying samples for model learned on train dataset. Such experiments on digits and CIFAR-10 datasets are presented in Appendix A.2
• While OT can solve domain adaptation with target shift and unbalanced classes (Redko et al., 2019; Rakotomamonjy et al., 2020), promising directions is using of source fiction for such problems too.
We expect our research to contribute the development of less complicated domain adaptation techniques and open doors for the future application of adversarial attacks cycle monotonicity.

heading: 7 ETHICS
text: There is currently no regulatory framework for the use of adversarial attacks in a machine learning environment. Adversarial attacks can be used in real-world scenarios and can harm infrastructures with machine learning models in the pipeline. On the contrary, we examined the adversarial attack property and showed how this technique can be used to improve the performance of machine learning models. We do not provide any new methods for adversarial attacks or protection against them.

heading: 8 REPRODUCIBILITY
text: To reproduce our experiment we provide source code in supplementary materials, run ./run_experiments.sh to start the training process. Details on used hyperparameters are presented in Section 5.2.

heading: A APPENDIX
text: A.1 ADVERSARIAL ATTACKS APPROXIMATION
In this section, we propose the usage of OT for Adversarial attack approximation. We used MNIST, MNIST-M, and SVHN datasets and trained three Shirivatsava-like classifiers (Shrivastava et al., 2016) on each dataset individually. To create adversarial examples, we used a simple FSGD method. We transformed 10k images from each training dataset via FSGD into adversarial examples. Having the original and adversarial examples, we used OT to find a map between them. The 10k test samples without associated adversarial examples and labels were input to the OT algorithms too.
As shown in Table 4, the resulted examples significantly reduce the accuracy of the classifiers. Also, the inference of adversarial examples using OT is much faster than iterative gradient descent methods. Our results confirm that OT can naturally learn the distribution of adversarial examples.
A.2 IMAGE CLASSIFICATION
In comparison to the previous section here, we turn to the opposite side. Our idea is to use source fiction for image classification using OT inside the single domain, in other words between test and train data. While discrete OT methods showed impressive results in DA with a low number of labeled samples, we propose to use mapping from the test set to the source fiction dataset of classifier f(OT (x)). Source fiction in such a scenario is obtained by the adversarial attack to the model trained on low number training samples. Shirivatsava-like classifiers (Shrivastava et al., 2016) were used for training. We evaluated the Sinkhorn algorithm on MNIST and USPS datasets with 10 known labels and CIFAR-10 with 100 known labels per class. Results are presented in Table 5.
In Figure 3 are presented examples of cycle monotone transformation for source fiction via adversarial attack in the dataset space.
A.3 ADDITIONAL EXPERIMENTS ON DA
In this section, we provide additional experiments using the source fiction method:
• Results with 3 (Table 6) and 100 (Table 7) known labels per class in target domain using ResNet50 classifier.
• Digits dataset using Resnet18, with 10 (Table 8) and 100 (Table 9) known labels. • ResNet18 trained one epoch on source domain Modern Office-31 datasets 10. • Experiments on the Office-Caltech dataset, unfortunately, full dataset is not available pub-
licly now, so we used an abbreviated version, results are presented in Table 11. • Resnet18 adaptation on CIFAR10-STL10 adaptation task (Table 12).
A.4 ENERGY-BASED MODELS FOR EPSILON FREE SOURCE FICTION
As shown in section 5, our algorithm depends on the size of perturbations, to avoid it and make our method free, we propose to use an input convex energy-based classifier. Energy-based learning provides a unified framework for many probabilistic and non-probabilistic approaches, particularly for non-probabilistic training of graphical models, including discriminative and generative
.
approaches and conditional random fields, graph-transformer networks, maximum margin Markov networks, and several manifold learning methods (LeCun et al., 2006). In energy-based settings for some given fixed x and possibly some fixed elements of y we can perform inference by:
Energy-based learning approaches can be considered as an alternative to probabilistic estimation for prediction, classification, or decision-making tasks. The energy-based representation must capture both the discriminative interactions between x and y and allow for efficient combinatorial optimization over y.
A.4.1 CONVEX ENERGY-BASED INFERENCE
Here is our approach in more detail. First of all, we build a model fθ that is Partly Input Convex Neural Network (PICNNs) (Amos et al., 2017) over x instead of y (Amos et al., 2017), it is made possible to apply convex inference over x. Secondary we train our model fθ on the source domain (Xs, Ys) in setup equal to Structured Prediction Energy Networks (SPEN) (Belanger & McCallum, 2016), but instead of multi-label classification, we simply perform multi-class classification.
In the next step, we apply the core idea of our approach, we use a model trained on (Xs, Ys), to create a source fiction domain which is a cyclical monotone to the target domain (Xt, Yt) and contain features from the source domain samples. We simply input xti samples to the model fθ and apply inference over it with the fixed yti :
arg min x f(x, y; θ) (13)
As stated before, according to Rockfellar (Rockafellar, 1966) and Breirer’s theorems (Theorem 1.22 (Santambrogio, 2015)), we know that the gradient of a convex function is cyclical monotone and solves a Monge problem (Villani, 2008).
We used the gradient descent method to inference and while our fθ is input convex over the x, applying this procedure over the target samples we can collect a new source fiction domain (Xf , Yf ) where Xf samples are class-wise cycle monotone to Xt. And according to the theorems presented before, there exists optimal transport that can solve transportation from Xt to Xe inside each class. Connecting resulted domain with penalty Ωsemi(γ) = 〈γ,M〉 by a ns × nt cost matrix M〉 where M(i, j) = 0 when ysi = y t j and +∞ otherwise (Courty et al., 2016; Yan et al., 2018) we achieve improvement in accuracy without value to restrict the seize of perturbations.
A.4.2 EXPERIMENTS
We tested our model on MNIST (LeCun & Cortes, 2010) and USPS datasets (Hull, 1994). In both experiments, we train PICNNs with a one hidden layer size of 100, in the SPEN settings using SGD (Ruder, 2016) optimizer with a learning rate equal to 1e-3 and momentum equal to 0.9.
.
Based on the POT library (Flamary et al., 2021) we tested the different variations of discrete optimal transport in this task. First of all, we tested and basic EMD and Sinkhorn (Sinkh) (Cuturi, 2013) algorithms. Then we tested regularized versions of the Sinkhorn algorithm with a group lasso regularization (L1L2) and Laplacian regularization (L1LP) (Courty et al., 2015). Finally, our method is benchmarked on MapT (Perrot et al., 2016).
The results presented in Table 13. In the table, we can see that mapping to the source fiction domain improves the accuracy of the optimal transport algorithms, but due to inference problems with energy-based models, accuracy is less than in settings with standard networks.

Review summaries:
The paper proposes an algorithmic trick (related to adversarial examples) to enhance existing semi-supervised domain adaptation techniques. The enhanced method does require at least a few labeled samples in the target domain. The paper first shows that if you perturb target samples by a small enough epsilon, you can maintain cycle monotonicity (and thus it is strongly related to the optimal mapping between two empirical distributions by construction). The paper proposes to create a "source fiction" dataset where each labeled target point is perturbed by a bounded epsilon amount (as in adversarial examples) but *unlike* adversarial examples, you are trying to move the point to be **correctly** classified. Then, all the target data is mapped to this "source fiction" dataset and the classifier is applied to this mapped target data. The paper shows that this idea can possibly boost the performance of existing OT-based domain adaptation methods. 

This paper proposes a technique for domain adaptation in the semi-supervised setting (we have access to some labels of the target domain). The idea developed in the paper is to train a classifier (source classifier) on the source domain in order to use the latent representations of this classifier for two things:  1. “anti” Adversarial examples are computed on the labeled target data in order to have them classified correctly by the source classifier. (I call them anti adversarial examples since the goal is to change the labeled target example to improve the accuracy of the source classifier). Such a modified set is called $\Omega_f$ 2. An Optimal transport algorithm (OT) is used to transport the latent representations of $\Omega_t$ to the latent representations of the target domain $\Omega_f$ (with a consistency constraint between labeled examples of $\Omega_t$ and $\Omega_f$)  The authors eventually try their algorithm experimentally.  

This paper proposes using the method of adversarial attacks for the task of semi-supervised domain adaptation. The adversarial perturbations are constrained to satisfy cyclical monotonicity [1].   [1] Cédric Villani, Topics in Optimal Transportation.

This paper studies the use of adversarial attacks and optimal transport (OT) in the context of semi-supervised domain adaptation. The authors show that adversarial attacks satisfy the cycle monotonicity property. They then propose an algorithm that generates adversarial examples for the labeled target samples using the source classifier, and maps the vanilla target samples to this new domain via OT.