Graph Harmony: Denoising and Nuclear-Norm Wasserstein Adaptation for Enhanced Domain Transfer in Graph-Structured Data
Authors: Mengxi Wu
Mohammad Rostami
Venue: Accepted by TMLR ()

Abstract: Graph-structured data is prevalent in numerous fields, but the scarcity of labeled instances often limits the effective application of deep learning techniques. Traditional unsupervised domain adaptation (UDA) strategies for graphs typically rely on adversarial learning and pseudo-labeling. However, these methods often fail to leverage the discriminative features of graphs, resulting in class mismatches and unreliable label quality. To overcome these challenges, we developed the Denoising and Nuclear-Norm Wasserstein Adaptation Network (DNAN). DNAN utilizes the Nuclear-Norm Wasserstein Discrepancy (NWD), which simultaneously achieves domain alignment and class distinction. The NWD is integrated with a denoising mechanism using a variational graph autoencoder, with a theoretical analysis provided for the denoising process. This denoising mechanism aims to address domain shifts in structural patterns between the source and target domains. Our comprehensive experiments demonstrate that DNAN outperforms state-of-the-art methods on standard UDA benchmarks for graph classification, highlighting its effectiveness and robustness.

Published in Transactions on Machine Learning Research (08/2024)
Graph Harmony: stein Adaptation for Enhanced Domain Transfer in Graph- Structured Data
Denoising and Nuclear-Norm Wasser-
Mengxi Wu
mengziuu edu
Department of Computer Science University Southern California
Mohammad Rostami
rostamim Pusc. edu
Deportment of Computer Science Universily Southern California
Reviewed on OpenReview: https: owtp///eeerrvvee...om net/ forum? id= 7Gg
Abstract
Graph-structured data is prevalent in numerous fields, but the scarcity of labeled instances often limits the effective application of deep learning techniques. Traditional unsupervised domain adaptation (UDA) strategies for graphs typically rely on adversarial learning and pseudo-labeling. However, these methods often fail to leverage the discriminative features of graphs, resulting in class mismatches and unreliable label quality. To overcome these challenges, we developed the Denoising and Nuclear-Norm Wasserstein Adaptation Net- work (DNAN). DNAN utilizes the Nuclear-Norm Wasserstein Discrepancy (NWD), which simultaneously achieves domain alignment and class distinction. The NWD is integrated with denoising mechanism using a variational graph autoencoder, with theoretical anal- ysis provided for the denoising process. This denoising mechanism aims to address domain shifts in structural patterns between the source and target domains. Our comprehensive experiments demonstrate that DNAN outperforms state-of-the the-art methods on standard UDA benchmarks for graph classification, highlighting its effectiveness and robustness. Our implementation available at: http:///ggauub.ooo//////////////
1 Introduction
While deep learning has made substantial progress in handling graph-structured data, it shares a drawback with other methods in the same category heavy reliance labcled data. Zero-shot learning (ZSL)
methods (Pourpanah et al. 2022 Rostami et al. 2022 Mancini et al. 2022 Guo et. al. 2023 Xie et al. 2020) propose to relax the need for data through relying on secondary domains of knowledge. However, these methods require having strong relationships between the main and the secondary domains and are more effective when the input data has a planner structure. The reason is that information between the two domains needs to propagate through the graphs, considering both local and global structures, which is more challenging compared to planner data. This requires complex algorithms that can handle the propagation dynamics and the dependencies between nodes. As result, these methods are not applicable in many practical settings. The persistent need for annotating graph-structure data presents significant obstacle in real-world applications, where the gathering and annotating of graph-structured data come with a steep price tag, both in terms of time and resources. Obtaining detailed labels for graph-structured data, such as chemical molecules, is a considerable challenge because chemical molecules are incredibly complex, comprising a large number of atoms connected in various ways through different kinds of bonds. Collecting annotated graph-structured data like social networks is also challenging due the need protect personal and sensitive information and the continual changes in network relationships In contrast, planner data such images is

---

Published in Transactions on Machine Learning Research (08/2024)
significantly easier to annotate. The scarcity of labels makes it difficult to derive meaningful insights and hinders the development of strategies and solutions based on deep learning. Therefore, it is highly desirable to relax the need for extensive graph-structured data annotation to replicate the success of deep learning in applications.
To navigate the challenge of label scarcity, Unsupervised Domain Adaptation (UDA) (Ganin & Lempitsky 2015 Gabourie et al., 2019 Bousmalis et al. 2017 Liu et al. 2022] has emerged as a promising frontier, aiming to leverage labeled data from related source domain to inform an unlabeled target domain. The challenge of label scarcity necessitates Unsupervised Domain Adaptation (UDA) due to the absence of labels in the target domain. The target domain is our primary area of interest. To overcome this challenge, we utilize a source dataset rich in labels to train our model. However, the inherent differences between the source and target datasets require the application of domain adaptation strategies. These strategies enable the effective application of models trained on the well-labeled source data to achieve high performance in the target domain, despite its lack of labels. The principle of UDA is to align the data distributions between the two domains within common embedding space, allowing a classifier trained on the source domain to perform competently in the target domain.
While UDA has been extensively applied to array-structured data (onng et al. 2016 Kang et al. 2019
Rostami & Galstyan 2023 Ganin & Lempitsky 2015 Bousmalis et al. 2017 Liu et al. 2022: Stan & Ros-
tami 2024b), its translation to graph-structured data remains underexplored. Graph samples exhibit a wide range of structural variations, including differences in connectivity patterns, node degrees, and subgraph structures. The primary challenge in applying UDA graph classification is the domain shift in structural patterns, or simply, the structural variations between the source and target domain graphs. These struc- tural variations make it challenging for models to identify and leverage invariant features across domains. Pioneering methods, such as DANE (Zhang et al. 2019a), integrate generative adversarial networks (GANs) with graph convolutional networks (GCNs) to align the domains. Others, like the approach by Wu et al. 20200, introduce attention mechanisms to reconcile global and local consistencics, again employing GANs for cross-domain node embedding extraction. However, these GAN-based methods have the drawback of class mismatching, lacking clear separability between features from different classes, as they align target and source domain features irrespoctive of their classes. In addition, those methods are designed for node classification. The UDA strategy for graph classification has not been well explored.
In this paper, we focus on the UDA setting for graph classification. We propose the Denoising and Nuclear- Norm Wasserstein Adaptation Network (DNAN) to address the primary challenges in graph UDA tasks and problems in previous GAN-based methods. Our DNAN benefits from the denoising mechanism with a variational graph autoencoder (VGAF) and the Nuclear-Norm Wasserstein Discrepancy. By leveraging the Nuclear-Norm Wasserstein Discrepancy, it tackles the class mismatch issue in existing graph-based UDA methods. Unlike previous GAN-based methods, DNAN performs refined, class-specific alignment of source and target domain distributions within a shared embedding space, preserving the distinct separability of features across classes. The inclusion of the denoising mechanism is motivated by the structural variations between source and target domain graphs for the graph UDA setting. The denoising mechanism of VGAE reconstructs clean adjacency matrices from corrupted versions. This process forces the model to learn robust features that are more invariant to structural variations and helps the model focus on the underlying structure and features relevant the classification task. Thus, we believe the denoising mechanism could help handle the domain shift in UDA tasks for graph classification. By using these two components, DNAN performs competitively and achieves state-of-thh-aa performance on major UDA benchmarks for graph classification.
Our contributions mainly lie in applying existing techniques to new problem and introducing an effective combination of existing approaches. Our first contribution is translating the denoising criteria of variational autoencoders to variational graph autoencoders, providing a theoretical analysis. We utilize the denoising technique to address domain shifts in structural patterns within the graph UDA problem. This translation and utilization of the denoising mechanism are not trivial. We believe we are the first to discuss the denoising criteria of VGAEs and apply this mechanism to address the domain shift in the structural patterns of graphs. Our second contribution is integrating Nuclear-Norm Wasserstein Discrepancy (NWD) with VGAE. Typically, VGAEs are used together with a domain classifier in previous domain adaptation methods. We
2

---

Published in Transactions on Machine Learning Research (08/2024)
use NWD to remove the domain classifier and incorporate the domain adaptation directly into our class classifier. This integration hasn't been proposed before.
2 Related Work
Unsupervised Domain Adaptation A foundational approach within UDA is to reduce the discrepancy
between the source and larget domain distributions using adversarial learning (Tzeng et al. 2017 Long
et al. 2018: Pei et al. 2018 Li et al., 2019 Chen et al.] 2020 Rostami 2024a A representative method
in this space, the Domain Adversarial Neural Network (DANN) (Ganin Lempitsky 2015), employs an
adversarial training framework to align domain representations by confusing domain classifier in a shared embedding space. This strategy is adapted from generative adversarial networks (GANs) (Goodfellww et al. 2020), tailored for domain adaptation purposes. Expanding on this adversarial methodology, the FGDA technique Gaa et al. 2021) uses a discriminator to discern the gradient distribution of features, thereby achieving better performance in mitigating domain discrepancy. Furthermore, DADA (Tang & Jia, 2020) proposed an innovative strategy by integrating the domain-specific classifier with the domain discriminator to align the joint distributions of two domains more effectively.
Adversarial approaches complemented by statistical discrepancy measures like Maximum Mean Discrep- ancy (MMD), utilized in the Joint Distribution Optimal Transport (JDOT) model (Courty et al.) 201 Wasserstein Distance (WD) has been leveraged for distribution aligument in UDA methods (Courty et al. 2017a; Damodaran et al.) (008)), with Redko et al. (2017) providing theoretical foundations for model gen- eralization in the target domain when employing WD. However, the practical application of WD is compu- tationally intensive due to the absence of a closod-form solution. The Sliced Wasscrstein Distance (SWD) [Rabin et al.] 2011 Bonneel et al. 2015 Stan & Rostami 2024a) offers a computationally feasible alter- native. Reconstruction-
objectives constitute another research direction, enforcing feature invariance across domains by reconstructing source domain data from target domain features, as demonstrated in the work by Ghifary et al. (2016). Additionally, the application of variational autoencoders (VAEs) (Kingma & &Welling 2013) to UDA, such as in the variational fair autoencoder (VFAF) (Louizos et al.] 2015). show- cases the capabilities of probabilistic generative models in domain-invariant feature learning. Our proposed method draws inspiration from the variational autoencoder's framework. Other notable approaches like ToAlign Weei et al.) 2021). , SDAT (Rangwani et al. 2022), and BIWAA (Westfechtel et al., 2023) mark re- cent advancements in UDA, surpassing previous models in performance. These three approaches are detailed in the experiment sections our references for current slate-of-the-arl methods. However, extending these existing methods to graph-structured data is often non-trivial.
Graph Representation Learning Graph reprosentation learning (GRL) has emerged as an important approach in machine learning, lasked with distilling complex graph-structun data inlo a tractable, low- dimensional vector space to enable the use of architectures developed for array-structured data. Previously, spectral methods laid the foundation, leveraging graph Laplacian matrices to capture the topological struc- tures of graphs despite limitations in scalability for larger graphs (Belkin & Niyogi 2003 Chung 1997).
The field then evolved with algorithms such as DeepWalk (Perozzi et al. 2014) and Node2 Vec [Grover & Leskoved 2016), which utilized random walks to encode local neighborhood structures into node embeddings, balancing the preservation of local and global graph characteristics. The introduction of Graph Neural Net- works (GNNs) marked a significant advancement in GRL. GNNs, specifically Graph Convolutional Networks (GCNs), offer a way to generalize neural network approaches to graph data, integrating neighborhood in- formation into node embeddings (Kipf & Welling 2016a). This was further refined by GraphSAGE, which scaled GNNs by learning a function to sample and aggregate local neighborhood features (Hamilton et al., 2017b a). Moreover, graph attention networks (GATs) introduced an attention mechanism, enabling the model to adaptively prioritize information from different parts of a nodc's neighborhood, thus enhancing the expressiveness of the embeddings (Velickovic et al. 2017). These advances, along with the development of graph autoencoders like VGAES that focus on graph reconstruction from embeddings, have broadened the applications of GRL and continue to shape its trajectory (Kipf & Welling 2016b). For a fair comparison, when we compare methods originally not proposed for graph domain adaptation, we replace their feature extraction backbones with GAT, as GAT is the feature extraction backbone we use in our approach.
3

---

Published in Transactions on Machine Learning Research (08/2024)
Denoising Diffusion Models Generative models (Ruthotto & Haber 2021 have increasingly incorporated denoising mechanisms to enhance the clarity and fidelity of synthesized data, bridging the gap between artificial and real-world data samples. Within this framework, diffusion models stand out by iteratively applying denoising steps, transforming random noise into dctailed and coherent outputs, and exemplifying the synergy between generative capabilities and denoising techniques. Denoising Diffusion Probabilistic Models (DDPMs) have revolutionized the field of generative modeling by leveraging principles from nonequilibrium thermodynamics, connecting diffusion probabilistic models with denoising score matching and Langevin dynamics to achieve high-quality image synthesis (Ho et al., 2020). Nichol & Dhariwal (2021) improved DDPMs not only to produce exceptional samples but also to refine them to attain competitive log- log-likelshoods while maintaining sample quality. Innovations such as the SinDDM framework demonstrate the capability of DDMs to Irain on single images by exploiting internal statisties through a multi-scale diffusion process [ulikov et al. 2023). However, the computational intensity of training and evaluating these models presents a challenge. This obstacle addressod by the introduction of pyramidal diffusion model that streamlines the gemeration of high-resolution images from coarser ones (Ryu & Ye 2022). Additionally, the Star-Shaped DDPM (SS-DDPM) introduces star-shaped diffusion process that simplifies the computation of transition probabilities and posteriors, enhancing the model's efficiency and applicability (Okhotin et al. 2024). Despite their success in image generation, applying diffusion models to image denoising has been less straightforward due to difficulties in controlling noise placement, leading to the development of a novel diffusion process with linear interpolation (Yang et al. 2023 , A multitude of diffusion model techniques for image-related tasks have been adapted for graph-related tasks (Chamberlain et al.) 2021 Barbe et al., 2021: Kong et al., 2023 Li et al.] 2023). , Although the denoising mechanism of graph generative models closely related to diffusion models, our work does not utilize diffusion models. In this paper, we employ a variational graph autoencoder as the generative model and extend the denoising mechanism of the variational autoencoder, as described by [Im Im et al. 2017), to the domain of variational graph autoencoders.
3 Problem Description
We operate under the assumption that there is a source domain with labeled data and a target domain with unlabeled data. In both domains, each input instance graph-structured data sample. Our primary objective is to develop predictive model for the target domain by transferring knowledge from the source domain.
Graph Classification We focus on a graph classification task, where a graph sample can be represented as G = A). X E RR=P, where represents the number of nodes in G and F represents the dimension of the features for each node. Note that the number of nodes may vary across different graphs. X2 E X corresponds to the feature associated with a node uy- A E Raax is the adjacency matrix. The adjacency matrix A encapsulates the topological structure of G. Each graph is associated with a class, and we use y to denote the ground truth label of the graph sample G. The goal is to train a model capable of classifying graphs effectively and accurately.
Source Domain Dataset We consider a fully labeled source domain dataset as (D,, Y.) = ((G+). (v.1),
where G is the 4th batch of graph samples in D. and 3² are the ground truth labels of G.
Target. Domain Dataset We consider that only an unlabeled target domain dataset D, = (Gt} acces- sible, where G is the kM batch of graph samples in the target datasct D..
UDA for Graph Classification The comprehensive pipeline of a UDA model for graph classification begins by leveraging neural network to extract relevant features from the input graph samples. Then, it aligns the domains by utilizing either distance metric to minimize the discrepancy between the source and target feature distributions or employing adversarial techniques to achieve domain-invariant feature representations. Subsequently, the classification task is performed using the aligned features. Usually, the classifier for the classification task is trained with source labels, but different UDA models may have diffcrent strategies for training the classifier. The performance of the UDA model is evaluated based on how well it performs the classification task on the test portion of the target domain dataset.
4

---

Published in Transactions on Machine Learning Research (08/2024)
4 Proposed Method
Figure visualizes a high-level description of our proposed pipeline. Our algorithm benefits from a denoising mechanism via a variational graph autoencoder (VGAE) and the Nuclear-norm Wasserstein discrepancy for distribution alignment. In a mutshell, our method embeds the graph-structured data from both domains into a shared feature space via a VGAE with a denoising mechanism. Then, we align the distributions of both domains in this shared feature space by Nuclear-norn Wasserstein discrepancy.
Alignment Loss Lawd
Total datallow
Source dasaflow
Target dataflow
Backward Propagation for Cond
Backward Propugation for &en
Data Augmentation
for ExtaO Backward Propagation for
Backward
A Mcrop C A + Maddd
Cross Entropy Loss
LCE
||
||+
I|
A Morop + Ag Madd
DVGAE Loss
GRL
Ldvgae (Ng; As: A prod
Label Classifier
Ldvgue (Na: Ag A pred
G = (X,, AD)
Graph Decoder
Graph Encoder
Zi Ni
Maximum Entropy Loss L
Zu ~Nall 0
G. = (Xa. As)
The block-diagram visualization of DNAN: Inputs are the source batch G, and target batch G.- We first add noise to the graph samples of the source and target batches by applying data augmentation to their adjacency matrices A, and At, using masks Morro, Mdrop, Madd, and Mpdd, Then, the graph encoder of the VGAE produces the latent variables Z, and Z, from node features Xs, X, and augmented adjacency matrices Ay. We train a class classifier using cross-entropy loss Lee between the outputs of the label classifier and the ground-truth labels. To align the latent variables of both domains, we compute Nuclear-noris Wasserstein discrepancy (NWD) using Ze, Zi, and the label classifier. The graph decoder of VGAE reconstructs the original adjacency matrices A., As from Z., Zz. Then, the Lavgae loss is computed based on the outputs of the graph encoder and the original and reconstructed adjacency matrices. Lastly, the model applies maximum entropy regularization L. to the latent variables Z,, Z..
Figure 1:
4.1 Latent Variables Construction with Denoising Mechanism
We start this section by introducing variational inference (lei et al, 2017). In a system composed of the random variables X and Z, X denotes the obscrved variable, while Z is the latent variable. The conditional probability density P(X|Z) () known as the likelihood. From Bayes' theorem, we can compute the posterior probability density P)Z as
P(Z[X)- P(X P(X)
(1)
The marginal probability density P(X) can be computed as
P(X) / ((
(2)
The marginal probability density P(X) known as the evidence, and P(Z) is termed the prior probability density, as it encapsulates the prior information regarding Z.
5

---

Published in Transactions on Machine Learning Research (08/2024)
Variational inference is estimation technique aimed at approximating the complex, often intractable posterior distribution X)Z with more computationally manageable parameterized distribution gs (Z). A variational autoencoder (VAE) is a type of generative model that uses variational inference within a probabilistic framework to encode input data into a latent space and reconstruct outputs from this space. This process allows for the generalion of new data points that are similar to the original inputs. In a VAE, the model defines an approximate distribution, (o()))), which conditioned on an input observation X. This distribution typically takes the form of neural network, where X serves as the input and the latent variable Z as the output. The parameters 6, representing the neural network's weights, are shared across all imput observations X. This neural network, known as the inference network, essentially learns to encode the input data into latent representation. Conversely, the model also defines a parameterized distribution (x( that models the probability of observing X given the latent variable Z. (x (Z) is also lypically chosen as a neural network with Z as the input and a distribution over possible values of X as the output. The weights of this neural network are denoted by 0, and this network referred to as the generative network. The generalive network decodes latent representations back into data points, thereby enabling the generation of new data points by sampling from the latent space. The prior distribution for the latent variable Z, denoted as P(Z), is typically chosen to be an isotropic Gaussian distribution, expressed as NN(0, d²1). To train a VAE, we maximize the variational lower bound, known as the evidence lower bound (ELBO), with respect to the parameters 0 and Φ:
log Po(X, Z) 20 (Z
LELBO = Bea(2(x)
(21x) (log Pa(X]Z)) - KI. (26(())
(3)
where KL represents the Kullback-Leibler divergence.
The variational graph autoencoders (Kipf & Welling 2016b) are basod on variational autocncoder concepts and are designed specifically for graph-structured data. Given graph sample G G-= AX with nodes, the graph encoder (inference network) in VGAF generates a corresponding latent variable Z. (2|4,, X) is used to denote the graph encoder, characterized by the parameter Φ. (2(2), X) aims to approximate the real posterior distribution P(Z]A). The graph decoder (gencrative network) of a standard VGAE is represented as (a(A defined by parameters 0. The prior distribution is denoted by P(Z); assumedd to be a normal distribution, specifically P(Z)= ~ I). The standard variational lower bound or ELBO for VGAE is given
as:
CELBO (2,, (log Po(A))) - (L(g((Z/4.)))I((2))
(4)
The standard VGAE is trained through maximizing the CELBO. Instead of using plain VGAE, we propose to use VGAE with a denoising mechanism. Particularly, we adopt the denoising critcrion of Denoising Variational Autoencoders (DVAE) (Im Im et al. 2017) and translate it to variational graph autoencoders. Like DVAE, the VGAE with denoising mechanism reconstructs clean graph data from inputs perturbed with noise. Similar to the training process of VGAE, however, we have some variations. Given a graph sample G = (X, A), we train the VGAE on both G = -(x, A) and G = AX, where A is the adjacency matrix with noise. We implement data augmentation to add noise to the adjacency matrices. Specifically, we benefit from a random manipulation- based approach (Cai et al. 2021 To this end, edges are dropped and added randomly by modifying the values in the adjacency matrix of the original graph. A is constructed as follows:
A Mddrop @ A + Madd
mi
Bernoulliy p edge).
drop ij
Bernoulli(p drop)
(5)
where padd, podze, and p pdoop denote the edge addition rate, the sparsity of the adjacency matrix A, and the edge dropping rate. @ represents the wles multiplication between two matrices. Mdrop and Madd represent mask matrices with the same dimensions A. For each element mard Madd or m mdrop drrrop we sample its value from Bernoulli distribution.
We're using similar logic from the DVAE paper to explain how the denoising mechanism can be applied to the variational graph autoencoder. In the following explanation, we use 'coruuption' to refer to the noise addition procodure. We start with how the denoising mechanism will affect the inference process of the variational graph autoencoder. We first translate Proposition 1 from the DVAE paper to the VGAE setting follows.
6

---

Published in Transactions on Machine Learning Research (08/2024)
Proposition 1. Let x) x, 0s (A) be Gaussian distribution, where pa (A", X) and (A(A", X) are non-linear functions of (A², x)). Let P(A0)A A) be a corruplion distribulion around A and A be the corrupt adjaceny matria. Then,
EP(AM1A) / 9((2144)1))P4411))4
(6)
is a minture of Gaussian.
If the distribution over a discrete variable, the integral in Equation can be replaced by summation. It's instructive to examine the distribution in the discrete domain to understand that Equation G takes on the form of Gaussian mixture. Essentially, för each instance A drawn from P(A7IA,, substituting into (2(2 X) results in Gaussian distribution. In our scenario, given that each element within the adjacency matrix can only be or 1, AA) is a discrete distribution. Thus, we can formulate our case as follows.
Example 1. Let E {0, 1 be the adjacency matriz of graph G that has n nodes, and consider a corruption distribution Pa (A) Bernoulli(T) around and A is corrupted adjacency matria. Then,
2
Ep (r2(aa1)loo(2(dd).))
E X0
has the form of finite mirture of Gaussian and the number of misture components is 2nxn.
From the DVAE paper, the corruption process at the input can be viewed as adding a stochastic layer at the bottom of the inference network (ga). Specifically, Pa (A) can be seen as a neural network, with v as the following, we first
its weights. P= AA) takes A as input and outputs the corruption distribution. present Lemma 0, result we will use in the later proof. Following Lemma 0, we present the variational lower bound when an extra stochastic layer is included, in Lemma
Lemma 0. For all nonnegative measurable functions f,g R - (0, that satisfy S f(X) S (X) dx=1,
[ (X)) & / f(x) (())
Proof. Let X be a random variable with s(X) be its probability density function. Consider the random s(x) with Ex(x [log F(X)]
variable log
f(x) [(x)
Exx [log a(X]
[(x) By Jensen's inequality,
E Bs(x) log g(X) f(X)
g(X) f(X)
llog (f g(X) dx =
I log E f(x)
Therefore, 5x) (logg(X)) S 5()) [log f(X)-
Lemma 1. Consider un approzimale posterior distribution of the following form:
q4 (Z)A / (662))49.)444444/444
(8)
Here, qu (A')A) represents the stochastic layer for the corruption process. We use & = (0, Then, given (A Z) = P(()) we obtain the following inequality:
log (A) 2 1.oo(ZAA.) log Po(A.
] 2 EBy((2AAx)
log Po(A, Z)
(9)
(Z)0
( (Z|A,

---

Published in Transactions on Machine Learning Research (08/2024)
Proof. By Jensen's incquality, we have
log Po(A, 2) qu(Z|Ar,,
Po(A,
(z
(AoiA)) BBo
log
go
(2
Po 2)
90 (1, x) 11
(A 2)
E (Ar-AA)
Byolz A..X)
log 9o (Z| X)
S log
Bgo(AnIA)
(2
Po(A, 7)
log
(
90 90(ZI dz)
20 (A0, X)
log
Po(A, dZ
(Ssa (A(A)RO(APPA)))
log
Po(A) I, (a(A))A) dA
log
= log (A(A
Therefore, the left inequality of Equation 9 holds, and now, for the right inequality,
Po(A, Z) (o(Z)Ap,
(ZIA,)) log
= (Z)A)))) (o( Po(A, 2)] Eg (
Applying Lemma 0 to the second term, we have
/ log g0 (1)62/244))) dZ < log gu (22 dz
(z 40 [X]
Hence,
Eg (2)4,) log go (2 I Eg (2]A {2) qo (2|A,
Then, we have
Z)
E (2)
log
(o)
(x [log Po -2) (log
(2)
(Z log (o Z)]
E (21, (2IA, X))
P)(A,2)
(20
log go (Z|AA,
Therefore, we obtain
log Po(A, Z) (,(
log Po(A, Tu (2|,,
log (o() 2 E (Z
E (2
Recalling Example 1, our approximate distribution can be defined as follows.
2
ás (Z)A,))
Σ (0(2149.)))(4914)
By treating adding noise via including one stochastic layer parameterized by 1b, we have:
2nx
(Z( - Σ ((12144)))P((4944)
We now can apply Lemma to oblain our denoising variational lower bound for VGAE. The standard variational lower bound Lavgae for the approximation distribution (, is:
log (o(A)= 2 E log Po(A, Z) (2)44 X)
2 Eqo(ZIA.X)
log jo (Z) (Z A,
Po Z)
def
Luvgae
(10)
8

---

Published in Transactions on Machine Learning Research (08/2024)
Now we define the denoising variational lower bound Ldvgae as:
log Po(A) Eg,(ZIAN) log Po Z) (6( X) Lavgae Lavgae
def
(11)
From Equation Lavgan is a lighter lower boud of log (A(A) than standard variational lower bound Lavgae Thus, we need to check whether Lavgae is valid lower bound. To check whether Lavgse is valid, we need to cramine what is achieved by maximizing Ldvgae- In fact, maximizing Lavgae minimize the expectation of the KL divergence between the true posterior distribution (P(Z]A)) and the approximate posterior distribution for each noised input (Z,(Z),, A X)). This is an effective objective as the inference network tries to map the noise-perturbed training data points to the true posterior distribution. In Theorem 1, we prove that maximizing Lavgae achieves this goal.
Theorem 1. Mazimizing ddugas equivalent to minimizing the following objective
(PA(A))A) [KL (90 (2066//44,.x)))1((22A1))))
(12)
In other words,
log (A) Ldvgae + Ero(A-IA) [LL (1)2(A())
Proof. Let us consider 0 being fixed just for the sake of simpler analysis.
log (A) - Lavgae = log Po(A) B(o(ZIA,X) log (A Z) 96 Ao
log P(Z)A))a(A))
= Eg
lox log Po(A)] (xx) 1
(2(
log (0(2|.4,X)
= -E A.X)
A)
log 96 (2
= =Er((A1A)
B ((o(Z( = EEr(AA)A))
P(ZIA)
P(( A))]
Now we prove that Lavgae is valid variational lower bound. To train the VGAE with Lavgse, in the DVAE paper, the authors adopt Monte Carlo sampling. In Monte Carlo sampling, we randomly seleet points from the domain of a function and compute the function's values at these points. We then calculate the average of these values to estimate the expected value the function. Essentially, it involves taking the mean of the function's values at the sampled points. The authors in the DVAE paper apply Monte Carlo sampling twice: once to the inner expectation (A.)A))) and once the outer expectation E, Applying their approximation method, we have the following equations:
Lavgae ([s(((AA,)) log Pa(A.2) (o,
Po(AZ 9 (Z)AA, X)
(20(2(10,)) Bro(aola)
log
Po(A,
M Σ
Po(A, Z(1m)
Lavgae (2 Epp(ariA) log
log
(13)
(2 X)
JM
(Z(m)))) A X)
where Ao, ~ (A))),, Z(lm) qos (2 X), and J, . are Monte Carlo sample sizes. Our approximation method also uses Monte Carlo sampling, but only for the inner expectation (4-(4))) since we adopt
the procedure used train the regular VGAE: (i) sample a corrupted input variable from (2(2) A", A), (iii) reconstruct the original adjacency matrix A. As the authors of DVAE state in their section 3.2 (Training Procedure), our procedure can be viewed special case of Equation 13 Our
(fi) sample a latent

---

Published in Transactions on Machine Learning Research (08/2024)
estimation is shown as follows.
Po(A, Z)
Lavgne (9
(POCAPIA)
log
40 (2)0,, X)
M
Po(A.
B (2
log
M
2o (A X)
m=1
(o(A(Z))(Z) 9 (Z)A X)
(20
log
M
P(Z)
Σ
[log (a(A)) + E Eg. (2, Be (21A2
log
M
960 A2 X)
m=l
M
go (2142,.))) log (o(A - (2)
M
m=1
Lavgae
Bao(21Ag, [lo (a(A)) (I X)] X)](
(14)
M
With this approximation and sctting the Monte Carlo sample size to 1, the training procodure becomes similar to how the regular VGAE is trained, except that the input corrupted by noise distribution. For clean inputs, train VGAE with Equation 4 and for corrupted inputs, we train VGAE with 14
For the structure of VGAE, we utilize graph attention networks (GAT) the graph encoder dø of the VGAE. The encoding equations when the imput is A are given as follows:
μ GA
log GAT (19, 11,X0 X)
2, = 14:4, oi, E; ~ (O
(15)
(2,11,,, X) = N diag(o?))
n (2 = II A3,
The element 2 corresponds to the 7th row of Z. This same row-wise correspondence applies to μr and log o as well. Using the reparamctorization trick, we transform the generated μ4; and o, into the latent variable a, To achieve a cleaner construction of the latent variable, we apply an element-wise maximun entropy loss, Le, regularization term. This maximum entropy loss helps remove irrelevant information from the latent variable, enhancing its clarity and effectiveness. The specifics of the maximum entropy loss described below:
L.
Σ
Σ ME(Z)
Gh
Ntrain G.E GEG*
(16)
my
ME(Z) ni Dr
(zy log logo
where ny is the number of nodes in the graph sample G and Dr denotes the dimension of G's latent variable Z. Atrain represents the number of batches of training samples G in the training dataset.
After obtaining the latent variable Z, an inner product decoder P^(AlZ) is applied to to reconstruct the adjacency matrix before data augmentation. This decoder translates each pair of node representations into binary value, indicating whether an edge exists between these two nodes. Specifically, we first use an MLP (multilayer perceptron) described by parameters (Wo, Wi) to improve the expressive capacity of the latent.
10

---

Published in Transactions on Machine Learning Research (08/2024)
variable Z. Then, compute the dot product for each node representation pair
H ReLU((Z Wo) Wi)
p(A:j = 1144 = (h ht)) " n
(17)
p(A
IIII (A,,)h,,h)) i=1j=1
where h2 represents the eth row of H and As is an element of A, the original adjacency matrix before data augmentation. The parameter 0 describes the graph decoder includes (Wo, Wi).
4.2 Distribution Alignment
By using Lavgue: the graph encoder of VGAE is better equipped to grasp the essential features. However, we still face crucial challenge: addressing the performance degradation that occurs when a model trained on data from source domain is applied to target domain with different data distribution. As mentioned in the previous sections, traditional approaches in unsupervised domain adaptation often use domain discriminator that engages in a min-max game with a feature extractor to produce invaiinnvvaaiant features. However, these hods primarily focus on confusing features at the domain level, which might, negatively impact class-level information and lead to the mode collapse problem (Kurmi & Namboodiri 2019 Tang &
Jia 2020). To address these challenges, our approach integrates the Nuclear-norm Wasserstein discrepancy (NWD) (Chen al.) 2022) effectively align the source and target domains' feature representations while maintaining class-level discrimination by considering it as loss function. We begin explaining NWD by discussing intra-class and inter-class correlations.
From Intra-class and Inter-class Correlations to Domain Discrepancy Consider a prediction ma- trix P gx predicted by classifier C, where b represents the number of samples and e represents the number of classes. P has the following properties:
Σ Pa= = Ps 2020 vi (1,
(18)
j=1
The self-correlation matrix R E RRexe can then be computed by R= p. The intra-class correlation Ia is defined the sum of the main diagonal elements in R, and the inter-class correlation Le is defined as the sum of the off-diagonal elements in R:
L Σ Raj. Le
Raj
(19)
4$1
The Ia and Ie are very different for source and target domains. For the source domain, the Ia is large while the Ia is relatively small, as we train with labels available so that most samples are correctly classified. For the target domain, the Ia is small while the Le is relatively large due to the lack of supervised training. Based on linear algebra, we can represent la = || the squared Frobenius norm of P, and
la - I. 21
(20)
For the source domain, Ia Te will be large; for the target domain, Ia will be small. Therefore, I, Ta can represent the discrepancy between two domains. Since the prediction matrix generated by the classifier C, can rewrite P. C(Z,), P. = C(Z,), where Z, is the feature representation of batch of samples from the source domain and Z, is the feature representation of batch of samples from the target domain. With inspiration from WGAN (Arjovsky et al. 2017 and 1-Wasserstein distance, the domain discrepancy can be formally formulated as
Wr Wr D.)=
sup [
2, (IC(())(( - (,,,5,))) [IC((Z))]=
(21
We call WF(D,, D.) the Frobenius norm- norm-based 1- Wasserstein distance, where D. denotes the source feature domain, D. denotes the target feature domain, || || denotes the Lipschitz semi-norm (Villani al. 2009).
and Kr, denotes the Lipschitz constant.
11

---

Published in Transactions on Machine Learning Research (08/2024)
From Frobenius Norm to Nuclear Norm From the domain discrepancy formulated above, see that the classifier C works like a discriminator in GAN. Therefore, we can perform adversarial training to train the feature encoder via WF(D,, (o(D,,Di) Dz). However, adversarial training with Wr(D,, D,) limits the diversity of predictions. This is because it tends to push the samples in a class with fewer samples near the decision boundary closer to neighboring class with a significantly larger number of samples far from the decision boundary (Cui et al., (202]). To address this limitation, the author of NWD proposes using the nuclear norm instead of the Frobenius norm. The nuclear norm has been shown to be bound by the Frobenius norm (Chen et al. 2022). In addition, maximizing the muclear norm maximizes the rank of the prediction matrix P when || | is near vb (Cui et al., 2020 2021p. Consequently, the diversity of predictions will be enhanced. Thus, the domain discrepancy can be improved to be:
WN(D., D.)=
sup E, Z. [II(())] E Z. 11,
[1C(
(22)
III
WN(D,, D.) is called the Nuclear-norm 1-Wasserstein discrepancy (NWD). To integrate NWD) into imple- mentation, we can approximate the WN by maximizing Luwd that is defined below:
Naraio
Ntrain
Cawd
Σ
|0(Z))1-
1
Σ 11O(2)),, WN(D,, max Lawd
(23)
Ntraiu
Nirain
where 2 represents the latent variables for the k-th batch of source graph samples G and Z represents the latent variables for the k-th batch of target graph samples G || I1=1 denotes the muclear norm. To avoid complex alternating updates, employ a Gradient Reverse Layer (GRL) (Ganin et al. 2016), which allows for updating in a single backpropagation step, The distribution alignment is achieved through min-max game, optimized as:
min max Lawd 0.
(24)
The NWD addresses the class mismatch issue by incorporating class information into the domain adaptation process. The class classifier not only performs class classification but also serves as a domain discriminator. The class classificr is capable of identifying correlations both within and among different classes. These correlations vary between the source domain data and the larget domain data. To achieve domain aligument, NWD aligns the correlations within and between classes in the target domain with those in the source domain. This alignment ensures consistency of classes across different domains, the class mismatch problem is mitigated.
Algorithm 1 DNAN Method
Input: (D,, Y,), Dr
Parameters: VGAE parameters & (Graph Encoder), 0 (Graph Decoder)), Classifier parameter (0c}
Output: Trained Parameters é, 0,0, 0g
1: Randomly sample a batch of {GG v{)
2: Randomly sample a batch of {G!)
3: Forward Propagation
Update é, A, fg based on Equation (26)
Add noise to {G+), (G+) based on Equation (5)
6: Forward Propagation
7: Update 6, 0, O, based on Equation (26)
8: return 6, 0, 0
4.3 Algorithm Summary
In addition distribution alignment, ensure accurate classification, we optimize the graph encoder in VGAE and the classifier C' using a supervised classification loss Ceta for the source domain:
Nisaiia
Lets =
Σ (cz(C(Z),
(25)
Ntrain
12

---

Published in Transactions on Machine Learning Research (08/2024)
Then, by combining all the losses described in the previous sections, our total optimization object is formu- lated as follows:
min Lels - CELBO(or Lavgae) ) + AcCo} + min max Lawd, 6,0.0.
(26)
0.
where o is the parameter of the graph encoder of the VGAE, 0 is the parameter the graph decoder of the VGAE, 0g is the parameter of the classifier, and de is a hyperparameter that weighs the maximum entropy loss Le- It is worth noting that balance the supervised classification loss and the NWD loss equally. In this case, our model effectively learns transferable and distinct features, leading to accurate and diverse prodictions in the target domain. The complete procodures of our UDA approach for graph-structured data are summarized in Algorithm
5 Experimental Validation
We validate our algorithm using two graph classification benchmarks. Our code provided as a supplement For hyperparameter values and details, please refer to
at: http:///ithnoooocoo////6/////aa our implementation.
5.1 Experimental Setup
Datasets
We use the IMDB&REDDIT Dataset (Yanardag & Vishwanathan 2015 and the Ego-network
Dataset QQu et al.
2018) in our experiments. Following previous works, we include Coreness (Batagelj & Za-
versnik
2003), Pagerank
Page et al. 1999), Eigenvector Centrality (Bonacich) 1987), Clustering Coefficient
Watts & Strogatz
1998). and Degree/ Rarity (Adamic & Adar 2003 as node features for both datasets.
IMDB&REDDIT Dataset IMDB&REDDIT consists of the IMDB-BINARY and REDDIT BINARY datasets, each denoting single domain. The Reddit-Binary dataset comprises an average of approximately 430 nodes per graph, with a total of 859,254 nodes across 2,000 samples. The prises an average of approximately 20 nodes per graph, with a total of 19,773 nodes across 1,000 samples. is the source domain and BEEDIIT-BIAAE is the target
dataset com-
We use 'I to R" to denote that domain. Conversely, 'R to I' indicates that REDDIT-BINARY is the source domain and IMDB-BINARY is the larget domain.
IMDB-BINARY Each graph in this dataset represents an ego network for an actor/ actrress. Nodes correspond to actors/ actors/actressos. actresssssss There will be edge between two actors/ace actors/actresses /actrrssessssss who appear in the same movie. A graph is generated from either romance action movies. The task is to classify the graph into romance or action genres.
MEDDIT-BINARE Each graph represents an online discussion thread. Nodes correspond to users. Tf one user has responded to another's comments, then an edge exists between them. The discussion threads are drawn from four communities: AskREDDIT and LAmA are question/answer- /answer- based communities, and Atheism and TrolIXChromosome are discussion- communities. The classification task is to classify a graph into discussion- or prestion/answer
communities.
Ego-network Dataset The L'go-network dataset comprises data from four social network platforms: Digg, OAG, Twitter, and Weibo, each representing a domain. We pair two domains to form twelve tasks, and we test our model on these twelve tasks. Each network in these four domains is modeled as a graph. Each graph consists of 50 nodes, with nodes representing users. Every graph includes an ego user. An edge is drawn between two nodes if a social connection occurs between two users. The definitions of social connections for these four social network platforms are different. We extract the descriptions of social connections and social actions for each social network according to Qiu et al. (2018):
Digg: Users can vote for web content such as stories and news (up or down). The social connection
represents users' friendship, and the social action is voting for content.
OAG: Generated from AMiner and Microsoft Academic Graph, the social connection is represented as co-authorship among users, and the social action is citation behavior.
13

---

Published in Transactions on Machine Learning Research (08/2024)
Twitter: Currently known as X, the social connection Twitter represents users' friendships, and the social action is posting tweets related to the Higgs boson, a particle discovered in 2012.
Weibo: Similar Twitter, the Weibo dataset includes posting logs between September 28th, 2012, and October 29th, 2012, among 1,776,950 users. The social connection is defined as users' friendships and the social action is reposting messages Weibo.
All the graphs the four domains are labeled as active or inactive, indicating the ego user's action status. If the user takes the social action, then the user is active. The task to classify whether the ego are active or inactive. In addition to the previously referenced node features, the Ego-network dataset also contains Deep Walk embeddings for each node (Perozzi et al. 2014), the number/ratie of active neighbors [Backstrom et al.) 2006), the density of the subnetwork induced by active neighbors (Ugander et al. 2012 and the number of connected components formed by active neighbors (Ugander et al. 2012
Baselines for Comparison There a limited number of UDA algorithums specifically designed for graph classification tasks. As a result, we conduct a comparative analysis between our proposed method and updated versions of several representative methods (DANN, MDD, DIVA) as well as current seate-of-the-at UDA methods (SDAT, BIWAA, ToAlign) for array-structured data. To facilitate the adaptation of these algorithms to graph-structured data, and consistent with the structure of our VGAE's graph encoder, we replace the feature extraction backbones, originally designed for array-structured data, with GATs, Below, these methods are explained:
GAT/VGAE: Plain GATs/VGAEs trained without domain adaptation techniques.
DANN:
Domain Adversarial Neural Network (DANN) (Ganin al. 2016) adopts an adversarial learning strategy. It contains domain classifier. The domain classifier tries to distinguish the samples from which domain and the feature extractor aims to confuse the domain classifier.
Margin Disparity Discrepancy (MDD) ZZang et al. 2019b is first proposed for computer vision tasks. measures the distribution discrepancy and is tailored the minimax optimization for training.
MDD:
DIVA:
Domain Invariant Variational Autoencoders (DIVA) [Ise et al. 2020 disentangles the inputs into three latent variables, domain latent variables, semantic latent variables, and residual variations latent variables. It is proposed to solve problems fields such as medical imaging.
Smooth Domain Adversarial Training (SDAT) (Rangwani et al. 2022 focuses on achieving smooth minima with respect to classification loss, which stabilizes adversarial training and improves the performance on the target domain.
SDAT:
BIWAA: Backprop Induced Feature Weighting for Adversarial Domain Adaptation with Ttera- tive Label Distribution Alignment (BIWAA) (Westfechtel et al., 2023) employs a classifier-based backprop-induced weighting of the feature space, allowing the domain classifier to concentrate on features that are important for classification and coupling the classification and adversarial branch more closcly.
ToAlign: Task-oriented Alignment for Unsupervised Domain Adaptation (ToAlign) Wei et al, 2021) decomposes features in the source domain into classification task-related and classification task-inrelevant parts under the guidance of classification knowllen ensuring that the domain adaptation is beneficial for the performance on the classification task.
Evaluation Metrics Following the literature Cai et al. 2021 the F1-Score is employed as a metric for the quantitative asscssment all methods. This score is the harmonic mean of precision and recall. The formula for the F1-Score as follows:
Precision Recall
F = Precision + Recall
(27)
14

---

Published in Transactions on Machine Learning Research (08/2024)
Training Scheme In our evaluation, rigorously train five models for each baseline method by employing five distinct random seeds for parameter initialization and either dropping or adding edges during the data augmentation phase. We report both the average performance and standard deviation of the obtained Fl scores. To ensure a fair comparison across all methods, we maintain the same seed for data shuffling. The optimization process utilizes the Adam (Kingma Ba 2014) optimizer.
5.2 Performance Results and Comparison
Tables 1 and 2 present our performance results. The bold font denotes the highest performance in each column.
Table 1: Performance results on Ego-network dataset
Method
O-T
O-W
O-D
T-0
T-W
T-D
W-0
W-T
WDD
D-0 D-T
D-W
Avg
GAT
40.0
40.420.3
43.812.4 40.210.0
48.0
41.3
40.210.0
46.6.0.9
41.3 40.240.0
40.0
39.8 41.8
VGAE
40.0
40.420.5
42.2+1.9 40.240.0
46.1
41.3
40.240.0
44.9a1.3
41.3
40.240.0
40.0
39.9 41.4
MMD
40.2
41.2+1.1
48.0+a.2
40.24a.0
45.5
41.34a.0
40.2+0.0
46.2+2.9
41.9+1.3
40.24..0
40.1
40.0 42.1
DIVA
42.120.5 42.4214 48.920.7 40.320.2 42.020.4 50.3
40.4
41.220.3
48.720.9
40.6g0.4413100
42.5+0.5 43.4
SDAT
40.220.1
40.120.5
42.221.6 40.220.0
41.621.3
42.943.2 40.3
41.120.7
43.1
40.220.0 40.1 39.940.1 41.0
BIWAA
40.1
41.520.2
45.1 40.310.2 48.012.9
43.343.0
40.210.0
50.42..5
45.7133.
40.310.1
41.020,9
43.110.343.2
ToAlign
36.5:19.443.0.op
49.1
40.810348.1
50.210.5
40.7.0.4
42.8
48.413.2
40.5.0.2
43.4
42.611.8 43.4
DNAN
42.9a1.6 43.4=1.2
50.7
40.8
40.2
53.9
40.8
48.66..
48.6
58.442.9
44.1
11.5
42.8442840. 45.9
DNAN*
43.2+1.4 44.9
40.7 49.8 50.3
40.6.
50.8
3...4...48.5441.
48.5
40.4
43.1
+0.6
+0.6
43.0 45.5
Table
Perofmrance results on IMDB&REDDIT dataset
Task
GAT
VGAE
DANN
MMD
DIVA
SDAT
BIWAA ToAlign DNAN
DNAN
I-R 63.42..2 63.720. 63.940.8 63.740.4
63.6_n.s
63.640.6
64.040.8
63.340.2
64.2-0.6
65.140.4
R-I 72.3 71.8 72.041.7
73.611.7
71.1 74.1
71.441.0
78.4 74.9 73.711.4
Avg
67.8
67.8
68.0
67.3
68.0
68.8
67.7
68.3
69.6
69.4
Ego-network Results Results for this dataset are presented in Table 1 In this benchmark, twelve UDA tasks can be defined by pairing the four domains. Compared to other basclines and SOTA methods, DNAN performs the best on average and achieves state-of-the-art performance on nine tasks: O to T, O to W, O to D, T to O, T to D, W to O, W to D, D to O, and D to T. DNAN has good performances on T to and W to T, and achieves SOTA performance on to and T to tasks, showing that DNAN can successfully handle similar domains, as Digg, Twitter, and Weibo are similar content-sharing platforms. Notably, it exceeds the sectnd-best methods by about 4% on T to D and about 5% W to D. Additionally, DNAN can also achieve SOTA performance when there is large distribution gap between domains, such as tasks between OAG and Twitter or OAG and Weibo. Tt is important to underscore that no single method can achieve the best performance on all tasks, likely due to the diverse range of domain gaps.
IMDB&REDDIT Results Results for this dataset are presented in Table 2 We note that the experi- ments demonstrate that all UDA methods perform better on the REDDIT to IMDB task (R to T) than on the IMDB to REDDIT task (I to R), indicating that the two tasks are not equally challenging, We hypoth- compared to REDDIT-BINARY may result in performance
esize that the smaller size of degradation when testing on the larger REDDIT-BINARY dataset, as less knowledge can be transferred to the target domain. Our experimental results show that DNAN outperforms all other methods on both the R to I and to R tasks, leading lo state-of-the-arl results on average. On the to R task, we observe that all methods perform similarly. Although DNAN does not outperform other methods by a large margin on the I to R task, the results still indicate that DNAN has competitive performance compared to other methods. It is worth noting that the performance of a UDA algorithm may vary to some extent based on hyperparameter tuning. Therefore, when comparing two UDA algorithms with similar performance, they
15

---

Published in Transactions on Machine Learning Research (08/2024)
Source Features
Target Features Before Adaptation Target Features After Adaptation
10
-10
15
Figure 2: UMAP visualizations show the test data representations before softmax activation for the Oag to Weibo task. Blue and red points denote the different classes. The middle plot displays the target domain data representations obtained from model trained on the source dataset before adaptation. The left and right plots show the source and larget domain data representations after adaptation using DNAN.
should be considered equally competitive. Based on this consideration, we can conclude that our proposed method performs competitively on all the UDA tasks and outperforms other UDA methods on average. These findings suggest that DNAN can serve as a robust UDA algorithm.
Our thcoretical framework demonstrates that any augmentation method applied to the adjacency matrix can be modeled adding an extra stochastic layer, and Theorem remains valid. In addition to the augmentation method in Equation 5 performed experiments using "node dropping" data augmentation. We tested our method with the "node dropping" data augmentation technique, which involves removing a node and its connections in a graph with a given probability. We refer to this variation of DNAN as DNAN". The results of DNAN* show that while not as effective as the lel-aaddigg//rooppll augmentation method, it achieves the second-best average result.
5.3 Analytic and Ablative and Experiments
We first conduct analytical experiments to provide deeper insights into our approach, followed by ablation experiments to show the importance of each component for optimal performance.
The effect of DNAN on data representations in the output space the classifter To evaluate the effectiveness of our proposed approach, we analyze how DNAN influences the target domain's distribution in the classificr's output space the Oag to Weibo task (O to W). We chose this task to demonstrate the effect of our model because it is challenging; Weibo and Oag very dissimilar platforms, co-authorship and the other by friendship. We utilize the UMAP (MeInnes et al. 2018) visualization tool and compare the representations of the source domain's test data, the target domain's 's test data before using DNAN, and the target domain's test data after applying DNAN. In Figure 2 each point represents a single data point in the putput space the classifier before the softmax activation. Blue and red colors denote the two classes. In Figure 2 the middle plot shows that the classifier does not work well with the target domain data before adaptation. It is hard to distinguish the class boundary red dots are mixed with blue ones. However, after applying DNAN, the class boundary becomes clearer, and the data representation distribution of the target domain matches well with the source domain. This is evident in the left and right plots of Figure 2 where the patterns of dots are consistent. These visualization results demonstrate that DNAN successfully mitigates the performance degradation caused by the domain shift.
connected by
A Ablative study The ablation experiments were conducted to demonstrate the effectiveness of the two main ideas we used develop DNAN, as well as the auxiliary maximum entropy regularization. We denote the ablated versions of DNAN as: (i) DNAN-D: We exclude the denoising mechanism and apply only the
16

---

Published in Transactions on Machine Learning Research (08/2024)
Table 3: Ablation Study Results on Ego-network Dataset
Method
O-+T
O-W
O-D
T-0
T-W T-D
W-0
W-+1
WDD
D-0
D-1
D-W
Avg
DNAN-D
42.8+1.3 42.5+1.7 52.3+2.6 40.540.2
46.942..50.0-..5
40.4 50.1 40.15.226...2..
40.6=0.2 42.1+1.3
42.5 45.4
DNAN-N
44.442.43. 52.6+10
41.04..3
45.0+2.4 52.7=2.7
40.7+0.3
46.9 53.3+2.5
40.5=0.2 43.0+1.1
43.6
45.6
DNAN-L
44.1:1. 43.121.2 53.6214
40.840.3
46.622.7 52.9=34
40.8
53.3g3,3
40.5 43.0
44.321.345.9
D
42.4+1.4 42.421.4 53.611.2
40.940.2
42.610.7
58.7=13
40.8:0.347.4422
53.6g3.1
20.72..42..11
43.7 45.3
Denoise
43.611.. 42.8 53.212.5 40.740.3
47.2:1.9 53.8_188 40.610.3
44.317.6
53.911.4006.0..
42.2 43.4 45.5
NWD
40.9
41.920.3 52.8+1.4 40.440.2
50.421.7
42.4_1.2 40.3 49.8
47.921.5
40.32o.1
46.220.3
43.842.1 44.8
E.
40.8 41.820.8 49.2 42.020.24..723.
49.9_0.3
40.8go.247.142.3
48.9 40.420.2 42.520.3
41.4g2.2 44.2
DNAN
42.9 23.4 a.. 45.3 53.9 40.8 48.6 53.4+2.9 40.6
44.1+1.5
42. 45.9
Table 4: Ablation Study Results on IMDB&REDDIT Datasct
Task
DNAN-D DNAN-N DNAN-L DNAN+
Denoise
NWD
Le
DNAN
to R
63.8
64.040.5
64.2.u.4
64.5
65.2
63.8
64.8
64.2+u.6
R to 72.3
74.2
73.9
71.421.2
74.0
72.8
74.1
74.9 +2.0
Avg
68.0
69.0
69.1
68.0
69.6
68.3
69.4
69.6
NWD loss and the maximum entropy loss. (ii) DNAN-N: We exclude the NWD loss and apply only the denoising mechanism and the maximum entropy loss. (iii) DNAN-L: We exchude the maximum entropy loss and apply only the NWD loss and the denoising mechanism. (iv) Denoise: We only apply the denoising mechanism. (v) NWD: We only apply the NWD loss. (vi) Le: We only apply the maximum entropy term. (vii) DNAN*: We replace NWD with the widely used Maximum Mean Discrepancy.
For the Ego-network dataset, the results reveal that the integration of all components yields the highest average performance at 45.9%. The DNAN-D configuration, which lacks the denoising mechanism, shows competitive performance with an average of 45.4%. However, the DNAN-N configuration, which excludes the NWD loss, displays an even smaller decrease in performance, with an average of 45.6%. Additionally, the results from DNAN-L indicate that the entropy term does not play significant a role, with an average performance similar to DNAN but falling short in specific tasks. Applying each component individually, the denoising mechanism (Denoise) provides the highest performance on average (45.5%), highlighting the crucial role of the denoising mechanism in enhancing feature extraction and robustness. The Nuclear-Norm Wasserstein Discrepancy (NWD) and maximum entropy regularization (Ce) also improve the performance but to a lesser extent.
For the IMDB&REDDIT dataset, the full DNAN model again demonstrates superior performance with average score of 69.6%. Interestingly, the DNAN-N variant outperforms the DNAN-D with averages of 69.0% and 68.0%, respectively, indicating the greater importance of the denoising mechanism in this context. The DNAN-L variant shows a slight reduction in performance, with 0.5% decrease in the average score compared to DNAN. When analyzing the contribution of individual components, the denoising mechanism (Denoise) again provides the highest performance on average (69.6%). The maximum entropy regularization (L.) improves the performance more (69.4%), while the Nuclear-Norm Wasserstein Discrepancy (NWD) has the lowest performance (68.3%) on this dataset. This suggests that the IMDE&REDDIT dataset contains more variations and noise in graph structure, which hinders distribution alignment more significantly com- pared to the Ego-network dataset. Overall, these results demonstrate the importance of each component in our proposed method. The denoising mechanism plays the most critical rolc, underscoring the effectiveness of using denoising techniques to tackle graph domain adaplation.
The comparison of the DNAN and DNAN+ methods further demonstrates the effectiveness of NWD in our domain adaptation framework. Specifically, across seven out of twelve tasks, DNAN achieves higher F1 scores than D Furthermore, the average F1-score of DNAN is 45.9%, surpassing the 45.3%
17

---

Published in Transactions on Machine Learning Research (08/2024)
Denoise Alone
NWD Alone
Maximum Entropy (Le) Alone
15
10
10-
10
0-
-1-
10 -10
-10
10 -10
-10
-10
Figure UMAP visualizations show the test data representations in the target domain before softmax activation for the Oag to Twitter task. Blue and red points denote different classes. The first. plot displays the data representations obtained from model trained using only the denoising mechanism. The sccond plot shows the data represenlations after adaplation using only NWD. The third plot shows the dała repre- sentations after adaptation using only maximum entropy loss Le.
of DNAN+ In addition, DNAN-N, DNAN-D, and DNAN-L all outperform DNAN+ Comparing the performance these two methods on I to and R to tasks further demonstrates the superiority of NWD. DNAN achieves average scores of 64.2% and 74.9%, resulting an overall average of 69.6%, compared DNAN*'s 's 64.5% and 71.4%, with an overall average of 68%. This clear margin supports that replacing NWD with commonly used MMD would lead to drop in performance, thus affirming the strategic advantage of employing NWD in approach. It also suggests that using secondary mechanisms that can
increase interclass margin can potentially improve the results (Rostami 2024b
In addition to the quantitative evaluation, we present qualitative ablation studies using UMAP visualizations to show the target domain test data reprosentations before the softmax activation for the Oag to Twitter task, as seen Figure 3 The first plot (Denoise Alone) exhibits the simplest and clearest. decision boundary, with blue dots densely clustered in the center and red dots forming distinct boundary around them with minimal overlap, resulting in well-defined scparation between the two classes. The red points uniformly ristted around the blue cluster, enhancing the clarity of the boundary. The second ploi (NWD Alone) shows moderately clear boundary, where blue dots are still centrally clustered but with slightly less density, and the red dots are more intricately intertwined with the blue dots, leading to less sharp but still noticeable boundary. The red points here are less uniformly distributed, causing more intermixing with the blue points. The third plot (Maximum entropy L. Alone) has the most intricate and unclear decision boundary, with significant intermixing of blue and red dots. The red points are densely scattered throughout, resulting in indistinct clusters and a blurred boundary. The qualitative visualization results are consistent with the quantitative evaluations, and the effect of each component is more clearly reflected.
5.4 Hyperparameters Sensitivity Analysis
An important concern for most algorithms is tuning the hyperparameters and measuring the performance sensitivity with respect to them. We evaluate the sensitivity of DNAN with respect to various hyperparam- eters on two tasks: Twitter to Digg (T to D) and Digg to Twitter (D to T). We varied the dimension of the latent embedding space, the output dimension of the graph decoder (before taking the dot products), the weight of the maximum entropy loss (Lz), and the batch size. We present the F1-scores of the DNAN model as a linear function of these hyperparameters Figure 4 with the blue lines representing the to D task and the yellow lines representing the T task. Through inspecting this figure, we deduce:
Dimension of the Latent Embedding Space: We test the performance of DNAN on five different dimension sizes for the embedding space: 32, 64, 128, 256, and 300. The performance of DNAN
18

---

Published in Transactions on Machine Learning Research (08/2024)
Variable
Output Dimension the Graph
Weight Maximum Entropy
Batch
50
48
48
68
06
04
44
120
32
300 0.1
256
Figure 4: The performance of DNAN with different hyperparameter settings on Twitter to Digg (Blue lines) and Digg to Twitter (Ycllow lines) tasks.
peaks at latent variable size of 256 for the T to D task and shows a less pronounced peak on the D to T lask, indicating that a moderately large value for the dimension of the latent. variable is beneficial for capturing the salient features of the data. Performance declines when the dimension too small to capture the complexity or too large, potentially introducing noise or overfitting, However, the results indicate that the performance remains relatively decent across wide range of embedding sizes.
Output Dimension of the Graph Decoder: Similar to the experiments the dimension of the latent variable, we test the performance of DNAN on five dimension sizes: 32, 64, 128, 256, and 300. The output dimension of the graph decoder shows a performance peak at 64 for both the T to D task and the D to T task. This observation suggests that moderately small representation capacity in the graph decoder is more beneficial. Compared with performances on the to T task, the T to task less sensitive to this hyperparameter.
Weight of the L.: We test DNAN with six weights: 0.1, 0.5, 1.0, 2.0, 3.0, 5.0. The weight of the maximum entropy loss presents a clear peak 1.0 for both the T to D and D to T tasks, suggesting that balanced contribution of the entropy loss is helpful for performance.
Batch Size: We test five batch sizes: 64, 128, 256, 512, 1024. For batch size, there is trend of increasing performance the size grows, with notable peak at batch size of 1024 for both the T to D and D to T tasks. This implies that the performance of DNAN benefits from larger batch sizes, possibly due to more stable gradient estimates. Compared with the T to task, the to T task less affected by batch size variations.
The sensitivity analysis of hyperparameters for the DNAN model the T to D and D to T tasks demonstrates the stability of DNAN models when using different hyperparameter values, there is moderate fluctuation around +3%. However, fine-tuning hyperparameters to the specific characteristics of the task and dataset is beneficial. Although optimal performance is achieved with a latent variable dimension of 256, a decoder output dimension of 64, an entropy loss weight of 1.0, and a batch size of 1024, tuning the hyperparamoters is not essential to achieve performance the competitive range.
Table 5: Training time for IMDB&REDDIT dataset
Task DANN MMD
DIVA
SDAT BIWAA ToAlign DNAN
I-R
260
14
12
899
10
10
R-I
2 5
3
5
19

---

Published in Transactions on Machine Learning Research (08/2024)
Table 6: Model complexity on IMDE&REDDII- -INAYY dataset. F represents the imput feature dimension, Dr represents the hidden dimension, and D represents the output dimension of the decoder.
DANN MMD
DIVA
SDAT
BIWAA
ToAlign
DNAN
+3+3D,+3)D, (F+4D,+4))) (E+1I,, (F (F) (F (F+3D, (F
5.5
Time Complexity and Model Complexity Analysis
In this section, we present an analysis of the training time (in minutes) and model complexity for our methods compared to others. The time complexity is detailed in Table 5 and the model complexity is outlined in Table 6
Training Time: The training times reported in Table illustrate the efficiency of the DNAN model relative to its counterparts. For the 1 to R task, DNAN required 10 minutes, positioning it as the fastest in terms training time, alongside DANN and ToAlign, compared to other methods. the R to I task, DNAN again demonstrated moderate efficiency with minutes, with MMD being the fastest 2 minutes and BIWAA the slowest at 34 minutes. These results suggest that DNAN provides balanced trade-off between model performance and training efficiency without adding significant computational overload.
Model Complexity: The model complexity, as shown in Table is assessed based on the number of parameters in the models, which is a function of the imput feature dimension (F), hidden dimension (Dx), and the output dimension of the decoder before taking the dot products (D). Compared
to other methods like DANN and SDAT, which have similar forms, DNAN introduces additional complexity due to the parameters in the graph decoder. However, it remains less complex than DIVA, which includes an extra +2D;+2D++)DD term.
We conclude that the DNAN model shows competitive training times that are significantly lower than the most time-consuming method (BIWAA) while maintaining better performance. Model complexity analysis reveals that DNAN, while not the simplest, avoids the higher complexity seen in more complex methods such as DIVA. DNAN balances the computational cost with the capacity to learn and transfer knowledge effectively for better UDA performance. This observation is important because, in certain applications, it is crucial to perform UDA quickly due to the constant changes in the input distribution and the limited time available update the model.
6 Conclusions
We developed an unsupervised domain adaptation (UDA) method specifically designed for graph-structured data, which often poses unique challenges due to its complex interconnections. Our proposed method in- cludes a denoising mechanism and using the NWD for domain alignment a shared embedding space. The experiments demonstrate that our approach is a promising method. By innovatively combining domain alignment through NWD with a denoising mechanism via a variational graph autoencoder, DNAN has out- performed state-of-theeart methods across two major benchmarks without adding significant computational overload. The ability of our method to handle both subtle and significant domain differences showcases its versatility and robustness. From ablative studies, the two idcas that DNAN benefits from are proven to be crucial for optimal performance. Future works can explore extending our approach to partial domain adaptation scenarios (Cao et al. 2018 Li et al. 2020 Liang et al. 2020). where only a subset of classes are shared between the source and the target domain, or situations where the source domain data is not directly accessible (Mao et al. 2024 to be updated.
Nananukul al. 2024 Zhang et al. 2024 and the loss functions for UDA need
20

---

Published in Transactions on Machine Learning Research (08/2024)
References
Lada A Adamic and Eytan Adar. Friends and neighbors the web. Social networks, 25(3):211 230, 2003.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wassorstein GAN. arXiv preprint Xrxi: 701107875. 2017.
Lars Backstrom, Dan Huttenlocher, Jon Kleinberg, and Xiangyang Lan. Group formation large social networks: membership, growth, and evolution. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 44 54, 2006.
Amélie Barbe, Paulo Gonçalves, Mare Sebban, Pierre Borgnat, Rémi Gribonval, and Titouan Vayer. Opti- mization of the diffusion time in graph fefffssd-waasesseetn distances: Application to domain adaptation. 2021 IEFE 33rd International Conference on Tools with Artificial Intelligence (ICTAT), pp. 786-790. LEEE, 2021.
Vladimir Batagelj and Matjaz Zaversnik. An o (m) algorithm for cores docomposition of networks. arXiv preprint cs/0310049, 2003.
Mikhail Belkin and Partha Niyogi. Laplacian eigennaps for dimensionality reduction and data representation. Newral computation, 15(6):1373-144 -196,,,1,,,.- 2003.
David M Blei, Alp Kucukelbir, and Jon D McAuliffe. Variational inference: A review for statisticians.
Journal of the Americon statistical Association, 112(518):859 877, 2017.
Phillip Bonacich. Power and centrality: A family of measures. Americon journal of sociology, 92(5):1170 1182, 1987.
Nicolas Bonneel, Julien Rabin, Gabriel Peyré, and Hanspeter Pfister. Sliced and Radon Wasserstein baryeen- ters of measures. Journal of Mathematical Imaging and Vision, 51(1):22-4) -5,, 2015.
Konstantinos Bousmalis, Nathan Silberman, David Dohan, Dumitru Erhan, and Dilip Krishnan. Unsuper- vised pixcl-level domain adaptation with generative adversarial networks. In Proceedings of the IEEE conference on compuler vision and pallern recognition, pp. 3722 7722-3731, 3731, 2017.
Ruichu Cai, Fengzhu Wu, Zijian Li, Pengfei Wei, Lingling Yi, and Kun Zhang. Graph domain adaptation: A generative view. ar XrXie preprint ar arXiv:2106 116.02066,7482, 2021.
Zhangjie Cao, Lijia Ma, Mingsheng Long, and Jianmin Wang. Partial adversarial domain adaptation. Proceedings of the European conference on computer vision (ECCV), pp. 135 150, 2018
Ben Chamberlain, James Rowbottom, Maria I Gorinova, Michael Bronstein, Stefan Webb, and Emanuele Rossi. Grand: Graph neural diffusion. In International Conference on Machine Learning, pp. 1407-1418.
PMLR, 2021.
Lin Chen, Huaian Chen, Zhixiang Wei, Xin Jin, Xiao Tan, Yi Jin, and Enhong Chen. Reusing the task- specific classifier as a discriminator: Discriminator-free adversarial domain adaptation. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pallern Recogrilion, pp. 7181-7190, 2022.
Minghao Chen, Shuai Zhao, Haifeng Liu, and Deng Cai. Adversarial-lerrned loss for domain adaptation. Proceedings of the AAAT conference artificial intelligence, volume 34, pp. 3521-3522 2020.
Fan RK Chung. Spectral graph theory, volume 92. American Mathematical Soc., 1997.
N. Courty, R. Flamary, D. Tuia, and A. Rakotomamonjy. Optimal transport for domain adaptation. JEEE TPAMI, 39(9))1853 1865, 2017a.
Nicolas Courty, Rémi Flamary, Amaury Habrard, and Alain Rakotomamonjy. Joint distribution optimal transportation for domain adaptation. Advances in neural information processing systems, 30, 201 7b.
21

---

Published in Transactions on Machine Learning Research (08/2024)
Shuhao Cui, Shuhui Wang. Junbao Zhuo, Liang Li, Qingming Huang, and Qi Tian. Towards discriminability and diversity: Batch nuclear-norm maximization under label insufficient situations. In Proceedings of the LEEE/CVE conference on computer vision and pattern recognition, pp. 3941 3950, 2020.
Shuhao Cui, Shuhui Wang, Junbao Zhuo, Liang Li, Qingming Huang, and Qi Tian. Fast batch nuclear-norm maximization and minimization for robust domain adaptation. arXiv preprint ar 2021.
B. Damodaran, B. Kellenberger, R. Flamary, D. Tuia, and N. Courty. Deepjdot: Deep joint distribution optimal transport for unsupervised domain adaptation. arviv preprint
2018.
Alexander J Gabouric, Mohammad Rostami, Philip E Pope, Soheil Kolouri, and Kuyngnam Kim. Learning a domain-invariant embedding for unsupervised domain adaptation using class-conditioned distribution alignment. In 2019 57th Annual Allerton Conference on Communication, Control, and Computing (Aller- ton), pp. 352-359. IEEE, 2019.
Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In Interna- tional conference on machine learning, pp. 1180-1189. PMLR, 2015.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. The journal of machine learning research, 17(1):2096 2030, 2016.
Zhiqiang Gao, Shufci Zhang, Kaizhu Huang, Qiufeng Wang, and Chaoliang Zhong. Gradient distribution aligument certificates better adversarial domain adaplation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8937-8946, 2021.
Muhammad Ghifary, W Bastiaan Kleijn, Mengjie Zhang, David Balduzzi, and Wen Li. Deep reconstruction- classification networks for unsupervised domain adaptation. In European Conference on Computer Vision, pp. 597-59777 613. Springer, 2016.
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communiccations of the ACM, 63(1)): 139-144, 2020.
Aditya Grover and Jure Leskovec. node2vec: Scalable feature learning for networks. In Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 855 864, 2016.
Jingeai Guo, Song Guo, Qihua Zhou, Ziming Liu, Xiaocheng Lu, and Fushuo Huo. Graph knows unknowns: Reformulate zero-shot learning as sample-level graph recognition. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 7775-7783, 2023.
Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. Advances in neural information processing systems, 30, 2017a.
William L Hamilton, Rex Ying, and Jure Leskovec. Representation learning on graphs: Methods and applications. arXiv preprint orXiv:1709.058: 2017b.
Jonathan Ho, Ajay Jain, and Pieter Abbecl. Denoising diffusion probabilistic models. Advances in neural information processing systems, 13:6840-6851, 2020.
Maximilian Ilse, Jakub M Tomezak, Christos Louizos, and Max Welling. Diva: Domain invariant variational autoencoders. In Medical Imaging with Deep Learning, pp. 322 348. PMLR, 2020.
Daniel Im Im, Sungjin Ahn, Roland Memisevic, and Yoshua Bengio. Denoising criterion for variational auto-encoding framework. In Proceedings of the AAAI conference on artificial intelligence, volume 31, 2017.
Guoliang Kang, Lu Jiang, Yi Yang, and Alexander G Hauptmann. Contrastive adaptation network for unsupervised domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pp. 4893 49002, 2019.
22

---

Published in Transactions on Machine Learning Research (08/2024)
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization.
arXiv preprint
4rxiv: 12. 22.0900 2014.
Diederik P Kingma and Max Welling. Aulo-encoding variational Bayes. Xiv preprinl ar1.::1312.6114 2013.
Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint Xr0i::1609. 02907, 2016a.
Thomas N Kipf and Max Welling. Variational graph auto-encoders. arXiv preprint arXiv:1611.07300 2016b.
Lingkai Kong, Jiaming Cui, Haotian Sun, Yuchen Zhuang, B Aditya Prakash, and Chao Zhang. Autoregres- sive diffusion model for graph generation. In International conference on machine learning, pp. 17391 17408. PMLR, 2023.
Vladimir Kulikov, Shahar Yadin, Matan Kleiner, and Tomer Michaeli. Sinddm: A single image denoising diffusion model. In International Conference on Machine Learning, pp. 17920 17930. PMLR, 2023.
Vinod Kumar Kurmi and Vinay P Namboodiri. Looking back at labels: A class based domain adaptation technique. In 2019 international joint conference on neural networks (IJCNN), pp. 1-8. IEEE, 2019.
Shuang Li, Chi Harold Liu, Binhui Xie, Limin Su, Zhengming Ding, and Gao Huang. Joint adversarial domain adaptation. In Proceedings of the 27th ACM International Conference Multimedia, pp. 729-737, 2019.
Shuang Li, Chi Harold Liu, Qiuxia Lin, Qi Wen, Limin Su, Gao Huang, and Zhengming Ding. Deep residual correction network for partial domain adaptation. IEEE transactions on pollern analysis and machine intelligence, 43(7) 23399 2344, 2020.
Xujia Li, Yuan Li, Xueying Mo, Hebing Xiao, Yanyan Shen, and Lei Chen. Diga: Guided diffusion model for graph recovery in anti-money laundering. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 4404-4413, 2023.
Jian Liang, Yunbo Wang, Dapeng Hu, Ran He, and Jiashi Feng. A balanced and uncertainty-aware approach for partial domain adaptation. In European conference on computer vision, pp. 123 140. Springer, 2020.
Xiaofeng Liu, Chaehwa Yoo, Fangxu Xing, Hyejin Oh, Georges El Fakhri, Je- Won Kang, Jonghye Woo, et al. Deep unsupervised domain adaptation: A review of recent advances and perspectives. APSIPA Transactions on Signal and Information Processing, 11(1), 2022.
Mingsheng Long, Han Zhu, Jianmin Wang, and Michael I Jordan. Unsupervised domain adaptation with residual transfer networks. Advances in neurul information processing systems, 29, 2016.
Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael I Jordan. Conditional adversarial domain adaptation. Advances in neural information processing systems, 31, 2018.
Christos Louizos, Kevin Swersky, Yujia Li, Max Welling, and Richard Zemel. The variational fair autoen- coder. arXiv preprint arXiv:1511.00833 2015.
Massimiliano Mancini, Muhammad Ferjad Naeem, Yongqin Xian, and Zeynep Akata. Learning graph em- beddings for open world compositional zero-shot learning. IEEE Transactions on pattern analysis and machine intelligence, 46(3):1545-1560 2022.
Haitao Mao, Lun Du, Yujia Zheng, Qiang Fu, Zelin Li, Xu Chen, Shi Han, and Dongmei Zhang. Source free graph unsupervised domain adaptation. In Proceedings of the 17th ACM International Conference Web Search and Data Mining, pp. 528, 2024.
Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint ar Xrvi:: 03426, 2018
Navapat Nananukul, Hamid Soltanian- Zadch, and Mohammad Rostami. Multi-source data integration for segmentation of unannotated mri images. IEFE Journal of Biomedical and Health Informatics, 2024.
23

---

Published in Transactions on Machine Learning Research (08/2024)
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International conference on machine learning, pp. 1162-8171. PMTRR 2021.
Andrey Okhotin, Dmitry Molchanov, Arkhipkin Vladimir, Grigory Bartosh, Viktor Ohanesian, Aibek Alanov, and Dmitry P Vetrov. Star-shaped denoising diffusion probabilistic models. Advances Neural Information Processing Systems, 36, 2024.
Lawrence Page, Sergey Brin, Rajecv Motwani, and Terry Winograd. The pagerank citation ranking: Bringing order lo the web. Technical report, Stanford InfoLab, 1999.
Zhongyi Pei, Zhangjie Cao, Mingsheng Long, and Jianmin Wang. Ahutti-ddversarill domain adaptation. Proceedings of the AAAT conference artificial intelligence, volume 32, 2018.
Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. Deepwalk: Online learning of social representations. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 701 710, 2014.
Farhad Pourpanah, Moloud Abdar, Yuxuan Luo, Xinlei Zhou, Ran Wang, Chee Peng Lim, Xi-Zhao Wang, and QM Jonathan Wu. A review of generalized vero-shot learning methods. IEEE transactions pallern analysis and machine intelligence, 45(4):4051-400 2022.
Jiezhong Qiu, Jian Tang, Hao Ma, Yuxiao Dong, Kuansan Wang, and Jie Tang. Deepinf: Social influence prediction with deep learning. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & dala mining, pp. 2110 2119, 2018.
Julien Rabin, Gabriel Peyré, Julie Delon, and Marc Bernot. Wasserstein barycenter and its application to texture mixing. In International Conference on Scale Space and Variational Methods in Computer Vision, pp. 435 446. Springer, 2011.
Harsh Rangwani, Sumukh K Aithal, Mayank Mishra, Arihant Jain, and R. Venkatesh Babu. A closer look at smoothness in domain adversarial training, In Proceedings of the 39th International Conference Machine Learning, 2022.
levgen Redko, Amaury Habrard, and Marc Sebban. Theoretical analysis domain adaptation with optimal transport. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2017, Skopje, Macedonia, September 18-22, 2017, Proceedings, Part II 10, pp. 737-753. Springer, 2017.
Mohammad Rostami. Continuons unsupervised domain adaptation using stabilized representations and experience replay. Neurocomputing, pp. 128017, 2024a.
Mohammad Rostami. Improving unsupervised domain adaptation through class-conditional compact repre- sentations. Neural Computing and Applications, pp. 1 18, 2024b.
Mohammad Rostami and Aram Galstyan. Coguitively inspired learning of incremental drifting concepts. In International Joint Conference on Arlificial Intelligence, pp. 3058 3066, 2023.
Mohammad Rostami, Soheil Kolouri, Zak Murez, Yuri Owechko, Eric Eaton, and Kuynguam Kim. Zero-shot image classification using coupled dictionary embedding. Machine Learning with Applications, 8:100278, 2022.
Lars Ruthotto and Eldad Haber, An introduction to deep generative modeling. GAMM-Mittedhngen, 44(2): e202100008, 2021.
Dohoon Ryu and Jong Chul Ye. Pyramidal denoising diffusion probabilistic models. arXiv preprint
arXiv:2208.0 0e.222208..866, 01864, 2022.
Serban Stan and Mohammad Rostami. Secure domain adaptation with multiple sources. Transactions on Machine Learning Research.
24

---

Published in Transactions on Machine Learning Research (08/2024)
Serban Stan and Mohammad Rostami. Source-free domain adaptation for semantic image segmentation using internal representations. Prontiers in Big Dala, 7:1359317, 2024a.
Serban Stan and Mohammad Rostami. Unsupervised model adaptation for source-free segmentation of medical images. Medical Image Analysis, 95:103179, 2024b.
Hui Tang and Kui Jia. Discriminative adversarial domain adaptation. In Proceedings of the AAAT conference on artificial intelligence, volume 34, pp. 5940 5947, 2020.
E. Tzeng, J. Hoffman, K. Saenko, and T. Darrell. Adversarial discriminative domain adaptation. In Computer Vision and Pattern Recognition (CVPR), volume 1, pp. 4, 2017.
Johan Ugander, Lars Backstrom, Cameron Marlow, and Jon Kleinberg. Structural diversity in social conta- gion. Proceedings of the national academyy of sciences, 109(16):5962-5966 2012.
Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, Yoshua Bengio, et al. Graph allention networks. stat, 1050(20):10 485500,-8500,0. 2017.
Cédric Villani et al. Optimal transport: old and new, volume 338. Springer, 2009.
Duncan J Watts and Steven H. Strogatz. Collective dynamics of networks... nature, 393(6684): 440-442, 1998.
Guoqiang Wei, Cuiling Lan, Wenjun Zeng, Zhizheng Zhang, and Zhibo Chen. Toalign: task-oriented align- ment. for unsupervised domain adaptation. Advances Newral Information Processing Systems, 34:13834- 13846, 2021.
Thomas Westfechtel, Wei Yeh, Qier Meng, Yusuke Mukuta, and Tatsuya Harada. Backprop induced feature weighting for adversarial domain adaptation with itcrative label distribution alignment. Winter Conference on Applications of Computer Vision (WACV), 2023.
Man Wu, Shirui Pan, Chuan Zhou, Xiaojun Chang, and Xingquan Zhu. Unsupervised domain adaptive graph convolutional networks. In Proceedings of The Web Conference 2020, pp. 1457-1467, 2020.
Guo-Sen Xie, Li Liu, Fan Zhu, Fang Zhao, Zheng Zhang, Yazhou Yao, Jie Qin, and Ling Shao. Region graph embedding network for zero-shot learning. In Computer EssionEECV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part IV 16, pp. 562-580. Springer, 2020.
Pinar Yanardag and SVN Vishwanathan. Deep graph kernels. In Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1365 1374, 2015.
Cheng Yang, Lijing Liang, and Zhixun Su. Real-world denoising via diffusion model. arXiv preprint arXiv:2305.00 04457, 2023.
Yizhou Zhang, Guojie Song, Lun Du, Shuwen Yang, and Yilun Jin. Dane: Domain adaptive network embedding. arXiv preprint arXie:1906. 1ete1990600684, 2019a.
Yuchen Zhang, Tianle Liu, Mingsheng Long, and Michael Jordan. Bridging theory and algorithm for domain adaptation. In International Conference on Machine Learning, pp- 7404-7413. PMLR, 2019b.
Zhen Zhang, Meihan Liu, Anhui Wang, Hongyang Chen, Zhao Li, Jiajun Bu, and Bingsheng He. Collaborate adapt: Source-free graph domain adaptation via bi-directional adaptation. In Proceedings of the ACM on Web Conference 2024, pp. 664 675, 2024.
25


Reviews:
[{"review_id": "OW01bjJvLz", "reviewer": "Reviewer_hNLo", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The writing is easy to follow. \n2. Theoretical analysis is provided. \n3. Multiple benchmarks are included to evaluate the proposed method.\n\nWeaknesses:\n1. Overall I feel the technical novelty of DNAN is compromised. DNAN can be seen as applying denoising and NWD which are both existing techniques to a plain VGAE framework. \n2. The denoising mechanism of VGAE is introduced to force the model to learn from corrupted data and reconstruct clean adjacency matrices, which is expected to improve the robustness of the model to domain shift. However, the imposed corruption may not effectively represent the real distribution shift of the target domain. \n3. As shown in Table 3, it seems the maximum entropy loss cannot bring any benefits to the model training. Moreover, the combinations of multiple losses appear to be trivial, comparing the result of \"Denoise\"-only (45.5) to the full version of DNAN (45.9).\n4. How is the performance of plain VGAE without denoising mechanism? \n5. It is expected to compare DNAN with the counterpart which replaces NWD with MMD.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces Denoising and Nuclear-Norm Wasserstein Adaptation Network (DNAN) to tackle unsupervised domain adaptation (UDA) for graph-structured data. A denoising mechanism via a Variational Graph Autoencoder (VGAE) is proposed to capture essential features of both source and target domains, improving the robustness of the domain adaptation process. DNAN employs the Nuclear-norm Wasserstein discrepancy, which allows for effective domain alignment and class distinction within a shared embedding space. This method addresses the class mismatch issues present in existing graph-based UDA methods. Results on multiple benchmarks valid the effectiveness of DNAN.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please refer to the \"Strengths And Weaknesses\" section.", "broader_impact_concerns": "N.A.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "QVBWDKZMvH", "reviewer": "Reviewer_hTBx", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n1. Novel Integration of Techniques:\n\nThe paper successfully integrates a denoising mechanism with VGAE and NWD, a novel approach that enhances domain adaptation by addressing class mismatching and structural variations in graph data. This integration allows for better feature extraction and domain alignment, improving performance on UDA tasks.\n\n2. State-of-the-Art Performance:\n\nDNAN outperforms existing methods on standard UDA benchmarks for graph classification. The combination of denoising and NWD leads to more robust and discriminative feature representations, enabling better classification accuracy.\n\n3. Comprehensive Experimental Validation:\n\nThe paper provides extensive experiments on multiple datasets, demonstrating the effectiveness of DNAN across various graph classification tasks. This thorough validation enhances the credibility and generalizability of the proposed method.\n\n4. Detailed Theoretical Analysis:\n\nThe paper includes a detailed theoretical analysis of the denoising mechanism and NWD, providing a solid foundation for the proposed approach. This theoretical underpinning helps in understanding the effectiveness and limitations of the DNAN framework.\n\n**Weaknesses**\n\n1. Complexity of the Method:\n\nThe proposed DNAN framework involves multiple components and complex integration, which may be challenging to implement and understand. The complexity of the method might limit its accessibility and adoption by practitioners who are not familiar with advanced techniques in graph UDA. Although some ablation studies are presented in the experiment to help understand the effect of each component, all of them are quantitative evaluation, and the difference between these variants is quite subtle. It would more interesting if the authors could show some qualitative evaluation or case studies to help understand these components.\n\n2. Scalability Issues:\n\nThe computational intensity of training and evaluating DNAN, especially with large-scale graphs, is a potential drawback. High computational requirements can hinder the application of DNAN to real-world problems involving massive graph datasets. The datasets used in the experiment are quite small, where each graph consists of only 50 nodes. Would it be possible to do some experiments on some larger datasets?\n\n3. Dependency on Quality of Source Domain Data:\n\nThe effectiveness of DNAN relies heavily on the quality and richness of the labeled source domain data. If the source domain data is not representative or well-labeled, the performance of DNAN on the target domain may degrade. Do the authors have any ideas or insights on how to solve this potential problem?\n\n4. Generalization to Other Types of Graph Data:\n\nThe paper focuses primarily on social network and molecular graph datasets. The generalization of DNAN to other types of graph data is not thoroughly explored. Different graph domains may have unique characteristics that require additional adaptations or modifications to the DNAN framework.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes the Denoising and Nuclear-Norm Wasserstein Adaptation Network (DNAN) for Unsupervised Domain Adaptation (UDA) in graph-structured data. DNAN addresses the challenge of domain shifts and structural variations in graph data by integrating a denoising mechanism via a Variational Graph Auto-encoder (VGAE) and the Nuclear-norm Wasserstein Discrepancy (NWD). The denoising mechanism captures essential features from both source and target domains, while NWD ensures class-specific alignment of domain distributions. The proposed method demonstrates state-of-the-art performance on standard UDA benchmarks for graph classification.\n\nThe main contribution of the paper is the development of the DNAN framework, which combines a denoising mechanism through VGAE with the NWD metric to enhance domain transfer in graph-structured data. This approach effectively addresses class mismatching and structural variations, leading to improved performance in graph classification tasks under the UDA setting.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see the comments of weaknesses.", "broader_impact_concerns": "No", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Jw7dDXYovG", "reviewer": "Reviewer_Bmha", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper's presentation is very clear. It also gives detailed explanations of the related topics of variational inference and domain discrepancy alignments.\n\n2. The proposed method mostly involves modifying the loss. The training procedure is clean and easy to implement, and it does not incur significant extra computational costs.\n\n3. The paper shows improved performance on two graph classification tasks.\n\n4. The problem the paper focuses on is novel, i.e., the UDA on graph classification tasks.\n\nWeaknesses:\n1. The proposed method is mostly a combination of existing methods for graph classification UDA.\n\n2. Based on the ablation study and the empirical performance, it seems that the proposed three components do not bring significant improvements, especially for the ego-network dataset. Particularly, what makes the proposed work special for the new problem is the combination of graph augmentation/denoising with the discrepancy alignments. However, on the ego-network dataset, excluding the denoising part does not cause a terrible decrease in the performance. This makes me wonder if designing more effective augmentation or denoising mechanisms is possible specifically for the graph UDA task. Otherwise, it is just the application of the current UDA method to different datasets.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies the problem of unsupervised domain adaptation for graph classification tasks. The paper proposes a method consisting of two major components: 1) use of the denoising VAE on augmented graph structures when learning the graph representations and 2) use of a nuclear-norm-based Wasserstein discrepancy loss when aligning the source and target domain classifiers. The paper also proposes to add an entropy term to the overall loss. The paper shows that the proposed loss improves performance on two graph classification UDA tasks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. More detailed ablation study, including applying one component at a time.\n\n2. Is it possible to try other graph augmentation methods for the denoising part?\n\nMinor:\n1. Page 4 bottom: \"p(Z)\"\n2. Algorithm: \"G^t_k\"", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This submission has been reviewed by three experts, all of whom have carefully read the manuscript and provided valuable, constructive suggestions in the rebuttal.

All reviewers unanimously point out that the presentation is clear, experimental results on two graph datasets are good, and theoretical analysis is provided. I agree with Reviewer hNLo and Reviewer Bmha that the novelty of submission is limited, as the proposed method combines existing techniques. However, I believe that applying existing methods to a new problem with good integration has good technical contribution as acknowledged by Reviewer hTBx. This integration makes the submission interesting in the graph area.

Furthermore, the rebuttal was constructive. The authors successfully addressed the initial concerns of all three reviewers, and the suggested ablation studies have enhanced the quality of the submission. Following these revisions, all reviewers agreed to accept the paper.

Based on the above, I recommend acceptance. Please ensure that all the suggested experiments and responses are included in the camera-ready version.
Decision: Accept (poster)