Graph Domain Adaptation via Theory-Grounded Spectral Regularization
Authors: Yuning You
Tianlong Chen
Zhangyang Wang
Yang Shen
Venue: ICLR 2023 poster (2023)

Abstract: Transfer learning on graphs drawn from varied distributions (domains) is in great demand across many applications. Emerging methods attempt to learn domain-invariant representations using graph neural networks (GNNs), yet the empirical performances vary and the theoretical foundation is limited. This paper aims at designing theory-grounded algorithms for graph domain adaptation (GDA). (i) As the first attempt, we derive a model-based GDA bound closely related to two GNN spectral properties: spectral smoothness (SS) and maximum frequency response (MFR). This is achieved by cross-pollinating between the OT-based (optimal transport) DA and graph filter theories. (ii) Inspired by the theoretical results, we propose algorithms regularizing spectral properties of SS and MFR to improve GNN transferability. We further extend the GDA theory into the more challenging scenario of conditional shift, where spectral regularization still applies. (iii) More importantly, our analyses of the theory reveal which regularization would improve performance of what transfer learning scenario, (iv) with numerical agreement with extensive real-world experiments: SS and MFR regularizations bring more benefits to the scenarios of node transfer and link transfer, respectively. In a nutshell, our study paves the way toward explicitly constructing and training GNNs that can capture more transferable representations across graph domains. Codes are released at https://github.com/Shen-Lab/GDA-SpecReg.

Published as a conference paper at ICLR 2023
GRAPH DOMAIN ADAPTATION VIA THEORY-GROUNDED SPECTRAL REGULARIZATION
Yuning You', Tianlong Chen², Zhangyang Wang?, Yang Shen
'Department of Electrical and Computer Engineering, Texas A&M University DDpartment of Electrical and Computer Engineering, University of Texas at Austin {yuning. you, yshen 8 edu, ( {tianlong. chen, at eut edu
ABSTRACT
Transfer learning on graphs drawn from varied distributions (domains) is in great demand across many applications. Emerging methods attempt to learn domain- invariant representations using graph neural networks (GNNs), yet the empirical performances vary and the theoretical foundation is limited. This paper aims at designing theory-grounder algorithms for graph domain adaptation (GDA). (i) As the first attempt, we derive a model-based GDA bound closely related to two GNN spectral properties: spectral smoothness (SS) and maximum frequency re- sponse (MFR). This is achieved by cross gross-pollinating between the OT- (opti- mal transport) DA and graph filter theories. (ii) Inspired by the theoretical results, we propose algorithms regularizing spectral properties of SS and MFR to improve GNN transferability. We further extend the GDA theory into the more challenging scenario of conditional shift, where spectral regularization still applies. (iii) More importantly, our analyses of the theory reveal which regularization would improve performance of what transfer learning scenario, (iv) with numerical agreement with extensive real-world experiments: SS and MFR regularizations bring more benefits to the scenarios of node transfer and link transfer, respectively. In a nut- shell, our study paves the way toward explicitly constructing and training GNNs that can capture more transferable representations across graph domains. Codes are released at htttss::// com/ wwwp///eenaaa/////eeaoaaaaaaaaee Sabe-----/DDDDAA-eee-
INTRODUCTION
Many applications call for "transferring" graph representations learned from one distribution (domain) to another, which we refer to as graph domain adaptation (GDA). Examples include temporally-evolvedd social networks (Wang et al. 202I), molecules of different scaffolds (Hu et al.) 2019), and protein-protein interaction networks in various species (Cho et al.) (216)). In general, this setting of transfer learning is challenging due to the data-distribution shift between the training (source) and test (target) domains (i.e. Ps(G, Y) * Pr(G, Y)). In particular, such challenge escalates for graph-structured data that are abstractions of diverse nature You et al. 2021 2022).
Despite the tremendous needs arising from real-world applications, current methods for GDA (as viewed in Section 2 mostly fall short in delivering competitive target performance with theoretical guarantee. Inevitably those approaches assuming distribution invariance (or adopting heuristic prin- ciples) are restricted in theory (Garg et al. [2200] Verma & Zhang 2019). The emerging approaches (Zhang et al. 2019 /u al.) (020) straightforwardly apply adversarial training between source and target representations, intentionally founded on the DA theory to bound the target risk (Redko) al. 2020). However, the generic DA bound in theory is agnostic to graph data and models, which could be more precisely tailored for graphs. We therefore set out to explore the following question: How to design algorithms to boost transfer performance across different graph domains, with the grounded theoretical foundation? Our step-by-step answers are follows.
(i) Derivation of mosel-based GDA bound. Building upon the rigorous assurance established in the DA theory (Section 3. we start by directly rewriting the OT-based (optimal transport) DA bound (Redko et al. 2017: Shen et al.) 2018) for graphs (Corollary 1), which is closely coupled with the Lipschitz constant of graph encoders. The nontrivial challenge here is how formulate GNN Lipschitz w.r.t the distance metric of Non-Euclidean data. Leveraging the graph filter theory Arghal et al. 2021). first state that GNNs can be constructed stably w.r.t. the
(Gama et al. 2020

---

Published as a conference paper at ICLR 2023
misalignment of edges and that of node features, multiplied by two spectral properties respectively: spectral smoothness (SS) and maximum frequency response (MFR) (Lemma 1). Subsequently, we utilize SS and MFR to formulate GNN Lipschitz w.r.t graph distances into general form, and instantiate it as (informally) max (SS)), (MFR)) w.r.t. the commonly-used matching distance Gama et al. 2020; Arghal al.
2021) (Lemma 2). This leads to the first model- GDA bound.
(ii) Theory-grounded spectral regularization. One potential way to tighten the DA bound is to modulate the Lipschitz constant (Section B) Guided by the theoretical results above, we are well- motivated to propose spectral regularization (i.e. SSReg and MFRReg) to restrict the target risk bound (Section 4.2p. We also extend the GDA theory into the more challenging conditional-shi shift scenario (Li et al. 2021a Zhao et al. 2019) (Lemma 3), where spectral regularization still applies.
(iii) Interpretation on how theory drives practice. Our further analyses on theory reveal which regularization would improve performance of what graph transfer scenario: specifically, SSReg and MFRReg are respectively beneficial to the scenarios of node transfer and link transfer (Section (.2). (iv) with extensive numerical evidences from un'semi-supervise (cross-species protein-protee protein interaction) link prediction and (lemporally-shifted paper topic) node classification (Section 5
RELATED WORKS
Self-supervision on graphs. Graph self-supervised learning, surging recently, learns empirically
more generalizable representations through exploiting vast unlabelled graph data (please refer to (Xie al.] (021) for comprehensive review). The success of self-supervision largely hinges on big data and, more importantly, heuristically disiggee pretext tasks. The tasks can be predictive
<Velickovic et al., |019| Hu al.) 2019 Jin et al.] [020]] You Shen 2022; You et al. 2020b; Chien
ct al.
2021, Talukder et al., 2022) or contrastive You et al./ 2020al Zhu et al. 2020b Qiu et al. 2020 Wei ct al. 2022). which does not provide theoretical guarantee of the target performance and,
as result, occasionally leads to "negative transfer" in practice (Hu et al. 2019 You et al. 2020a).
Transferring GNNs with explicit covariate shifts. To promote target performance, one line of work is to utilize more data and make specific assumptions. One such example is to assume the ac- cess to source labels and the explicit covariate shift that Ps PY G)) = Pr G) and (G) + F
in specific way, which enables theoretical tools for certain guarantees. (Ruiz et al., 2020: Yehudai let al. 2012 to develop size-invariant representations. (Bevilacqua et al. [021] works on transfer learning
2021) study the specific setting of size generalization and use the graphon theory (Lváázz]
in shifting d-patterns of subgraphs and adopts the theory of GNN expressiveness (Xu et al.) 2018
Morris et al.
GNNs despite their univer- sal approximation capability. Accordingly, the study proposes d d-pattern classification pre-training to help escape from begative-transferring GNNs. These methods are restricted to the designated transfer learning scenarios. Besides, some other works (Fan et al. 2021 Sui et al. 2021 [.i et al.) 202Tb, Kenlay et al., 202T] Chen et al.) [022] Li al.] 2022; Zhang et al. 2022] Jin et al., 2022) adopt the implicit covariate shift assumption with source labels while lacking assurance in theory, e.g. (Wu et al.) 2022a b) assumes that the shift could be implicitly modeled with an environment
2019) to demonstrate the existence of regative-transferrin
learner (please refer to (Gui et al. 2022) for comprehensive review).
Graph domain adaptation. To deliver a generally applicable guarantee, several methods (Dai et al,, 2019 Cai et al. [202]] /Zhan et al.) [2019] Wu et al., 20200; Xu et al. (022) additionally utilize target graphs to learn domain-invarian
representations. According to the DA theory (Ben-David et al.,
2007 2010 Redko al. 2020; /Zhang al., 2020 Yan et al. 2017). the target risk is guaranteed to be bounded (please refer to (Redko et al., 2020) for a comprehensive review). The generic DA
bound is not designated for graph data or encoders where further improvement could be achieved.
3
PRELIMINARIES
Problem setup. We are given i.i.d. samples (Verma & Zhang 2019: Zhu et al. 2021 Cong et al.
2021 and their labels {G,,, Y,) N from the source distribution Ps Ps Y) of graphs G G and labels Y E y, where = {V. E) is associated with the set of nodes V and edges E, together
with the node feature X E RVVxD and adjacency matrices E R We also have access from the marginalized target distribution f /P Y)dY. With the covariate shift assumption that (G) # (r) Ps|YG)) Pr(Y G) (Ben-Davie et al. 2007 2010), we are expected to train a graph neural network (GNN) h: g + y with the accessible data and then evaluate on target samples from Pr Y).
to unlabeled samples ] 1.rr
2

---

Published as a conference paper at ICLR 2023
Domain adaptation with optimal transport. Studies transfer learning across distinctly dis- tributed data have proliferated in the past few years, known as domain adaptation (DA) (Redko et al, 2020). Based on the aforementioned problem setup, we decompose the trained GNN h g go/ into the feature extractor f + RD' (Z = f(G)) and discriminator g RDD' + +3 (Y (Z)). With- out the loss of generalizability, we consider binary classification task where y = [0, 1] and Y E y the probability belong to class 1. We denote the labeling function given representations as Σ (g( ((7)))
9 RD' - y. and the (empirical) source and target risks as (s 9) Na
and (g. .) = (2) (l)(2) (())), respectively. Applying DA with optimal transport (OT), if the covariate shift holds on representations that (Y( = Pr I bounded as in the following theorem.
the target risk er (g. 9) is
Theorem 1 Redko et al. 2017; Shen et al. 2018: Li et al. 2021a) Suppose that the learned
discriminator g is Cg "Liischchitz where the Lipschitz norm [gp = maxz,, 10(2,) 2()l C2
(l22.72)
holds for some distance function p (Euclidean distance here). Let H = {g : + }) be the set of bounded real-valued functions with the pseudo- dimension PHim(H) = d that g E H, with probability at least 1 - the following inequality holds:
4d eNs d
1 1og( W,, (Ps(Z), ( Ns
log( eT(g. 9) s وو + V Ns
+ +w, (1)
where w = min واااوو sc, { {es(g. 9) + ee(gg
denotes the model discriminative ability (to capture
source and target data), and the first Wasserstein distance is defined as (Villani 2009):
(1 Q = sup
(Erg(z)a(Z) - -)r(())(()))
(2)
The thorough rightness justification of the OT-based DA bound can be found in (Redko et al. 2020), Section 5.3-5.5). The theorem indicates that the generalization gap depends on both the
domain-divergence 2C, (IPs (I))))))rr()) and the model discriminability w.
Adversarial training, Motivated from Theorem 1 (or its variants differing mainly in distribution divergence metrics [Ben-Davic al., 2007; 2010 Mansour et al., 2009)), a well-developed practice is to learn Nomain-invarinne representations via jointly optimizing the source risk and the distribution divergence term (conceptually, (TPss(),, (T())) as in (Redko et al. 2017 Shen et al. 2018):
Nos
Σ Eggo S(G.), Y.) + W (Ps(((()). n=1
min Lip
No
where & is the loss function used for training, Wi is the empirically calculated first Wasserstein distance with implementation details following (Shen et al.] (0188) presented in Appendix E and
y is the trade-off factor. Besides co-optimizing Wi, implementing domain classifiers is another effective way to alleviate the domain discrepancy (Zhang et al. 2019: Wu et al. 2020
4
METHODS
4.1 MODEL-BASED DA BOUND FOR TRRAP-STTRCCTUR DATA
Noticing that the generic DA theory (Theorem 1) is agnostic to data structures and encoders, our first step directly rewrite it for graph staph-structured data (G) accompanied with graph feature extractors
(f) as follows. The covariate shift assumption is now reframed as Ps (Y = PT(YGG).
Corollary 1. Let's assume that the learned discriminator is Cy-Lipschitz continuous as described in Theorem 1, and the graph feature extractor f (also referred to as GNN) is Cy-Lipschitz that
[|ll = max ||/z)l))(GGlll = C2 for some graph distance measure 77. Let H == (h
m(G],G2)
g + } be the set of bounded vealevalued functions with the pseudo-diment Pdim = d that
= =o fe E H. with probability at least 8 the following inequality holds:
4d eNs N
eT er(h, hi) I hs) +
log(
Ns log( + +CC,,,WW (Ps(G)),,)P) + w. (4)
d
where the (empirical) source and target risks are és(h, h) = 1 DM' [h(GG) A(G.)) and (r h) = Ep - i(G))), respectively, where h : -g for graphs and w = min SC. SC, {es(h, h) + er (b(,
No
+ y is the labeling function
3

---

Published as a conference paper at ICLR 2023
The property in Corollary 1 designated for GNNs, as focus on, is the Lipschitz constant Cj ma (IG)))/(G=)IL
g[ G2) Success in instantiating this conceptual model-related property for graphs provides important insights into how to rationally construct or train GNNs for better transferability. Other data-relevant properties (e.g. W1 (PPPs Pr())) are left to future works.
Instantiating the GNN Lipschitz constant is however nontrivial. The distance metric r G2) is formulated w.r.t non-Euclidean structures and its computation hinges on solving the graph matching problem whose time complexity is exponential in the number of nodes (Riesen & Bunke 2009
Riesen et al. 2010).
Rather than working on the denominator of the Lipschitz norm, we take an alternative perspective of the numerator, i.e. (I] f(G2)ll2, which is related to the GNN stability (potentially mul- tiplied with the distance term to eliminate the denominator). This essentially motivates us to draw connection between transferability and stability. Thanks to the permutation invariance property of GNNs, graph matching does not need to be solved here. Building upon the graph filter theory
Gama et al. 2020 Arghal et al. 202T. we first state that GNNs can be constructed stably:
Lemma 1. Suppose that G is the set for graphs of the size the size Ng after padding with isolated nodes, similar to (Zhu al. (021). Given YG1, G2 E G and A₁ = UT A = U2A2UZ the eigen decomposition for adjacency matrices A1 and Ag that A1 = diag [11,1. N..Ne)), A2 = ( ..., Nz)) (eigen values are sorted in the descending order). A GNN is con- structed by composing a graph filter and nonlinear mapping that f(G,) = (8()) ((a(U))((A))) x, W)) where S is the polynomial function that S(Ai) = En=o St.4f W R RDxD' is the learnable weight matrix, r is the mean/sum/max readout function to pool node repre- sentations, and the pointwise nonlinearity holds as (b(b) a[a) N |b lbb-al al. Va, E R. Assuming
|| I 1 and ||llo I 1 (Il-Ilop stands for operator norm), the following inequality holds:
(lf(G:) - f(G2) S Cx(1 + + - Pp'AP""TIE IF
+ (11 A1 - PM P"A2p*T (22) + max ( P'X2llr
(5)
where +- = (1|U1 - U211 F + 1)2 - stands for the eigenvector misalignment which can be bounded,
P = argminren || PX2l1 + ||A, | I1 is the set of permutation matrices, OIIA, - P* Ao P=1 Il²) is the remainder term with bounded multipliers defined in (Gama et al. 2020). and C2 is the spectral Lipschitz constant that Vλ Ay. (S(a,) S(A,)) I CxLA, - Agy-
Proof. See Appendix A
The lemma integrates the stability properties w.r.t node feature and edge perturbations in (Gama et al. 2020) and (Arghal et al. 22221), respectively, where C3 tied to edge perturbations characterizes spectral smoothness (SS) of the underlining graph filter, and max max((S(()))) ((S(A)) tangled with node feature perturbations defines maximum frequency response (MFR). We note that in practice for Lemma 1, no padding is actually needed if the summation pooling is adopted as in our experiments (Xu et al, 2018), since the padded nodes do not contribute the graph embedding (Zhu al. 2021). Analysis on more sophisticated architectures is left to future works.
We next instantiate the GNN Lipschitz constant for the commonly-use matching distance (Gama eet al. 2020: Arghal et al. 2021 in the following lemma, achieved by eliminating the distance term in the numerator and denominator of the GNN Lipschitz constant.
O Lemma 2. Suppose that G is the set for graphs of the size NG after padding with isolated nodes, similar (Zhu et al. 202T). We define the matching distance between G₁, G as (GG, G2) = minpen {
PX + ||A, PA, Suppose that the edge perturbation is bounded that 1G G2 E G, ||A, pa A2 || S E with the optimal permutation p and there exists an eigenvalue X E R to achieve the maximum [S(X*) < xo. We can then calculate the Lipschitz constant of GNN as:
C max {CAKI + eK2, 2 [8(x"))
(6)
where K1, K2 is the supremes of (1 + 7 N/N) and the remainder multiplier in Lemma 1, respec- tively, following similar philosophies in ( (Gama et al. 2020), Theorem 1).
Proof. See Appendix
4

---

Published as a conference paper at ICLR 2023
To recap, Cλ and [S())) characterize SS and MFR, re- spectively. Thereafter, the direct incorporation of Corol- lary and Lemma results in the first model-based DA bound for graph data. It builds the foundation to under- stand and further develop GDA algorithms. Although in our theory GNN is assumed to be composed of a graph filter and nonlinear activation layer, in practice such an architecture is ubiquitous, well-known for its simplicity
GNN
a(A)
Spectral
#(A)
signals
A Aa
and effectiveness (Wu al., 2019; Li al., (219). We also provide theory for two-layer GNNs in Appendix which naturally extendable to multi-layer architectures via induction.
SSRte: Regularize
MF MFRReg: Regularize
(E(Aa) =(A,)
a(A2)
E(AL)
(ra(2:) A,
a(Ayg
4.2 THEORY-GROUNDH SPECTRAL REGULARIZATION
An overview of spectral regular- ization. In implementation we regularize on the subtraction between numerators and nominators multiplied by certain threshold, considering the numerical instability issue in
Figure 1:
Recall that, in the DA inequality (see also Section 3), the gap between the target and source risks is bounded with (i) the distribution divergence between source and division. target (W1 term) multiplied by Lipschitz constants, and (ii) the discriminative capability of NNs to capture invariant knowledge (the w term) restricted by Lipschitz constants. This indicates that varying the Lipschitz constant results in the intrinsic trade-off between domain-divergence and dis- criminability to vary the DA bound.
Therefore, in search of a sweet spot in the trade-off, one way to tighten the bound is to regularize the Lipschitz constant of NNs. Equipped with the derived graph model-based DA bound in Section 4.1 we are now ready to propose regularizing the GNN spectral properties of SS or MFR for balancing between cammii-ddivergnne and discriminability. We implement regularization follows.
Implementations. Driven by the analysis. we incorporate spectral regularization (SpecReg E (SSReg, MFRReg) SSReg for C) and MFRReg for [S(X*)| | domain-invariant representation learning framework (3) as follows:
equation (6) into the traditional
Nis
Σ
lggo f(Gn), + wh, (CCI(G)))PP(((C)))
mün SCC,
Ns
+ 7 ''SpecReg (f. ( )
Nas
NT
where 7 is the trade-off factor, tuned over {1. le-1, le-2, 1e-3) through validation. Denote = (G) and the spectral signals = UTZ, X = U SS and MFR regularization (SSReg, MFRReg)
are specifically implemented by regularizing on the spectral outputs w.r.t. spectral inputs as:
No
SSReg Ne (
sum [eLLU(abs(2n)) No.
DE(S,T)
==1
[m(1 :No:: - 1,:)) v vabs(A A, X [2: No.: [ A, 1: No - 1,:1)). (8)
Np
MFRReg (i No. C (Cm))
Σ (E,
Σ sum (ReLU(abss(2)) abs vabss))) aabs() 12ml
ND
where X|m,1 n] denotes matrix indexing, sum is the matrix summation, ReLU and abs are pointwise rectifier and absolute functions, respectively, and is the threshold value tuned over {0.1, 1, 10, 100, 1000} through validation. An overview of the implementation is depicted in Figure | Analysis on complexity of spectral regularization (with eigendecomposition) is presented in Appendix H
How could theory further drive practice? We also notice that in special scenarios, either of the two regularizations (SS and MFR) can be favored than the other. (i) Node transfer: When the node features preserve (mostly) the invariant label-related information, while edges are less label- relevant (noisy) and distribute highly distinctly across source and target domains, would have
5

---

Published as a conference paper at ICLR 2023
large K E in Lemma 2 that C2K₁ + =K2 > [S(X*). Thus regularizing SS leads to the decreased domain-divergene term (C, decreases) and the less increased discriminability term w.r.t node features (GNN is sensitive enough to label-preserving node features as shown in Lemma 1). (ii) Link transfer: Oppositely, when edges are invariantly label-preserving across source and target domains whereas node features are relatively noisy, regularizing MFR is more desired. This provides with the guidance to select the regularization in practical applications with domain knowledge.
4.3 EXTENSION TO THE SEMI-SUPERVISER SETTING
Results far do not assume the access target labels, which is in the unsupervised setting. In this section, we show that our analysis also applies in the semi-supervised setting, where the more challenging conditional shift considered- Py
() + G in addition the access
small amount of target labelled data (Li et al. 2021a Zhao et al. 2019). We provide a sanite-sample ceme-supeee OT-based GDA bound in the following lemma.
C Lemma 3. Under the assumption of Corollary 1, we further assume that there exists small
amount of i.i.d. samples with labels {G.,, Yn)) from the target distribution PT Y) (N₃- < Nş) and bring in the conditional shift assumption that domains have different labeling function hy + fuT and ma (hn(G)))hhGGa))) = Cn S CjC₁ (D E {S, T}) for some constant C₁ and G2)
distance measure 17. Let H (h 9 }} be the set of bounded real-valued functions with the pseudo-dimensio Pdlim(H) = d, with probability at least 1 - the following inequality holds: N²
Ns + N₁ Es(h, hs)) + Ns N (( log(
Ns
Ns
&d
eN + log
2 NY
hT) S Ns + NT
(r( +
d
8d
eNs d
2
log( 1))
w): (10)
log(
Ns
Ns
where w min (l, hi) - cs (h, br)),, (c h2) - (1, hr)))
Proof. See Appendix C
Since the main form of inequality (T0) is consistent with inequality 4 similar discussion can be made following the thoughts in Section 4.2, to demonstrate that we can properly regularize the spec- tral properties of GNNs (SS and MFR) to tighten the target risk and seek a sweet spot between the domain-divergence and discriminability. Following optimization (7). the semi-supervised training procedure is implemented as:
N No Σ
1
Σ Egg o ((G)),Y,) +
lgg f(Gn), + wis Wi (Ps(T(G)),PP(((C))
min SC
Ns
+ y SSeecReg (,, (N Ns (G,,
Nr )
(11)
EXPERIMENTS
betwoen asi matrices of St and Cap
25
Co-expression Physical
We evaluate our proposed algorithms, SSReg and MFRReg, in two real-world applications of graph transfer learning (i) link prediction of protein-protein interactions (PPIs) across different species (Szklar- czyk et al., 2021), and (ii) node classification of paper topics across
Homaphily ralio difference
20
15
different time periods (Wu al. 2020 Hu 2020).
10-
5.1 PREDICTING
INTERACTIONS IN VARIOUS SPECIES
Datasets. PPI networks have proven important to understand func-
Human Mouse Fruit fly Yeast
tional genomics and analyze biological pathways (Sharan et al. 2007 Navlakha & Kingstord, 2010). But in most species the coverage of experimental PPI data remains low (Sledzieski et al., 2021). We uti- lize protein sequences together with freely accessible computational PPIs via whole-genome comparisons (Szklarczyk et al. 2021) to pre- dict experimental PPIs, i.e. the graph is built with nodes represented as protein sequences and edges as computational PPIs. We collect PPIs of species from the STRING database (Szklarezyk et al. (022)) where PPIs in the neighhorhood, fusion and co-occurrence channels defined computational, and in the
Figure 2: Homophily tio difference between ad- jacency matrices built with sequence identity (SI) and computational PPI (CPPI). Higher values indicate more label-preserving protein se- quences.

---

Published as a conference paper at ICLR 2023
Table 1: Unsupervised transfer of cross-species protein- co-expression interaction prediction. Numbers red are top-2 AUROC (AUPRC) (td%). DA-C, DA-W denotes learning with domain classifiers (Ganin et al. 2016) and guided by Wasserstein distance as in optimization (Redko ct al. 2017 Shen ct al. 2018), respectively.
representation
Co-expression: Link transfer = MFRReg
Methods
Mouse
Zehrafish
Fruit fly
Yeast
Meant
Rank
48 38.99.12.34
51.63:1.81
50.28:12.20
46.31=00.3
43,90
Mashup
9.0
(5.49=0.32)
(5.37=00333)
(5.51=0.17)
(4.9610000)
(5.33)
54.48:+3.27
61.18:11005
66.63++1.41
58.88=0.80
60.29
D-SCRIPT
7.5
(6.01 (6.01+0.63)
(8.12+11.7))
(9.78+0.25)
(7.53+0..11
(7.86)
GraphCL
13.09=:1.56
74.19=0.55
66.80:1..55
62.41=1.12
69.12
5.2
(14.98+2.18)
(18.76+2..1)
(12.12++2.00)
(11322+2.6)
(14.29)
69.55=0.44
69.63:10.4
1..7
63.01=11.45
64.89
Transformer
5.8
018.06:1..13)
1.111)
(10.13+1.02)
(11.25::1.58)
(16.72)
Transformer
76.35:10.38
79.2912.77
66.54:1.111
63.91
71.52
4.0
+GIN
(21991=1.60)
(28.07:44.71)
(13.48=0.71)
(1115=1111
(18.65)
Transformer
78.56:11.55
79,46:12,97
64.78:11.23
60.65:1..85
70.86
4.8
GIN+DA-C
(22.76:+4.42)
(27.1013.100 10)
(1161122.08)
(10.72::2.44)
(18.04)
Transformer
77.38:22.54
79.22++0.99
67.78++0.40
62.43+2.62
71.70
3.6
+GIN+DA-W
(23.03::2.98)
(26.90+2.03)
(13.78=:094))
(11,59:1.98)
(18.82)
Transformer+GiIN
77.57:1:.11
19.44:1:.21
65.27-11.49
62.28
71.14
3.6
+SA-W
(23.13:10.64)
(28.974:2.22)
(1188800 89)
(13.2412,.9))
(19,30)
Transforme++GIN
77.63:11.00
80.81
68.56:10.88
63.7410,27
72.68
1.2
+DA-W+MFRReg
(23.83:.2.75)
(29.0410.62)
(13.94::0:47)
(16.80:12.34)
(20.90)
Table 2: Unsupervised transfer cross-species protein-protedd physical interaction prediction.
Physical: Node transfer
SSReg
Methods
Mouse
Zebrafish
Fruit fly
Yeast
Meant
Rank.]
51.54+3.82
37.82+3.43
46.88=+6.87
$7.99++2.28
48.55
Mashup
9.0
(5.58:10.35)
(3.98:10112
(7.19+3.93)
(6.78:10.22)
(5.88)
58.22-16.97
49.5811112
62.97:10.78
62.431:0.59
58,30
D-SCRIPT
8.0
(7,03+11009)
(5.02110.6)
(9.61:0.21)
(8.56110.15
(7.55)
GraphCL
76.88=0042
79.11+:1.14
81.02-10.98
71.03:10.30
77.01
6.0
(31.16:1
(41.80=3:20)
(38.6342..3))
(14.58=1.16)
(31.54)
77.65=0.88 84
186
76.90:11.64
67.86:10.61
74.50
Transformer
5.6
(35.05=00922)
(45.13=3.15)
(32.72422334)
(12.46=1.08
(31.34)
Transformer
79.77=10.92
80.8542.41
82.38:11133
71.5410.36
78.63
4.3
+GIN
(31.23:1 1.94)
(24.29112.42)
(42.40112.04)
(15.73=0.79) 7.79
(30.91)
Transformer
80.141 1,86
1..58
11.49:1.27
71.3011..61
79.12
3.3
+GIN+DA-C
(3..29:4412)
(44.01 +.000)
(38.944-2.36)
(33,51)
Transformer
80,18:11.88
80.88113.08
80,31
72.66:10.36
78,80
3.6
+GIN+DA-W
(34.14+0.85)
(41.88-2.15)
(42.02+00669)
(16.18=2.67)
(33.55)
Transfomer+GIN
81.20:00.25
11.69
81.79:10.74
73.07++0.30
79.43
1.3
+DA-W+SSReg
1.5)9911511)
(45.15=2.07)
(43.4411.11)
(17.3911001)
(35.49)
Transfomer++GGN
1.19
81.95
80. 80.15:1:.07
72.22-10.67
78.81
3.6
+DA-W-MERReg
(34.63:3.71)
(43.09=4.19)
(35.43411.60)
(16.40=1.12)
(32.38)
co-expression and experiments (we refer to it as physical to prevent confusion) channels (to be pre- dicted) experimental which deals with expensive functional genomies experiments (Parkinson) let al., 2009) or direct lab assays (Brückner et al. 2009). More details of PPI data are shown in Appendix D and ablations in Appendix G
Configurations. We train models on PPIs of Homo Sapiens (human) which contain abundant PPI evidences (Ewing et al. 2007) and evaluate the models in four other species: Mus Muscu- lus (mouse), Danio Rerio (zebrafish). Drosophila Melanogaster (fruit fly) and Saccharomyces Cere- visiae (yeast), which are repeated for three times for statistical significance. In evaluation, we follow 2016)) to perform negative sampling for experimental PPIs, to ensure that the known inter- actions compose only 5%. We adopt Transformer (Tay et al.) (2020) (v.s. HRNN (Karimi et al., 20199 2020ah) to embed protein sequences and then GIN (Xu et al.) (018)) (v.s. GAT (Veličković et al, 2017 to perform message passing, with comparisons in Appendix E The compared state-of-the-an (SOTA) approaches
(Cho et al., 2016)
Mashup (Cho et al. (016)) and D-SCRIPT (Sledzieski et al. 2021) designed for PPI prediction as well as GraphCL (You et al. 2020a) for general graph (self-supervised) repre- sentation learning. See Appendix for details.
Hypotheses. We hypothesize that co-expression interactions are more associated with links (than physical interactions do, link transfer), based upon the existing findings that two genes poten-
7

---

Published as a conference paper at ICLR 2023
tially similar in expression profiles if they are with similar promoter regions (might indicate locating in the neighborhood) (Park et al. 2002), involved in fusion events (Fernebro et al. 2006), or sharing co-occurrence patterns (Larmuseau et al. 2019). For physical interactions, we hypothesize that they rely more on node features (sequences, node transfer) considering the recent breakthrough that protein structure information (and ultimately physical interactions and functions) can be recovered from sequences with high accuracy (Jumper et al.) 2021). We calculate the homophily ratios (Zhu al. 2020al Pei et al. 2020) of out of 4 species Figure numerical evidence to support the hypotheses (see Appendix |F] for the computing procedure, where zebrafish is excluded due to the overly sparse (<200) physical interactions as shown in appendix[D).
Results. The results of unsupervised
[a) expression
(b) Physical
28-
transfer are shown in Table and and 2 We
put semi-supervised results in Appendix
Mouse Auroc
24
34
G with the similar findings. Through
32
comparing between: (i) methods w/o
20-
30
and w/ DA techniques, (ii) methods w/o
28
and w/ spectral regularization, and (iii)
our proposed methods and SOTA com- petitors, we have the following findings.
Zebratih sUPRC (S)
30
(i) Vanilla DA in general provides benefits though occasionally degrades performance. With the assistance of domain-invariant representation learn- ing, GNNs generally lead to a better performance with exceptions. i.e., with Wasserstein distance guided DA (DA-
40-
26
35
S5Reg
MirReeg
Best result
24
30
Frui my AUPRE 18)
14-
40
W, as in optimization (3) (Redko et al.
30-
2017 Shen et al.) 2018)) 11 out of
10-
16 metrics turn higher than w/o DA in the unsupervised setting, and so do 10 out of 16 in semi-supervised. The oc- casional performance degrade fits our analysis in Section 4.2 that the opti- mized source risk and distribution diver- gence of representations could lead to
25
18
Weast AUPRIC 191
14
14
either improved or deteriorated perfor- mances without guarantee.
10 11
2
Threshold value logudo)
Threshold value log.ot))
Similar phenomenon happens DA with domain classifiers (DA-C, (Ganin al. 2016)). Comparison between DA- W and DA-C shows that, DA-W performs better than DA-C when transferring to graphs with the larger domain gap (i.e. from human to fruit fly and yeast, justified by their biological taxonomic ranks and phylogenetic distances (Alberts et al. 2002). see Appendix for details).
Figure 3: Spectral regularization performance of different threshold values e in co-expression and physical interaction prediction (with trade-off factors y' selected via validation).
(ii) In a principled way, spectral regularization further boosts GNN performance with DA-W and consistently alleviates performance degradation. Under the regularization on GNN spectral properties, domain-invariant representation learning provides further benefits in a principled way. Specifically, MFRReg improves 13 out of 16 co-expression interaction prediction versus DA-W, since it assists GNN to mine from the computational PPIs that co-expression interactions are more correlated with, as hypothesized in Section 5.1 We show in Figure 3 that it is consequence of better sweet spot when regularizing MFR, which echos our analysis in Section 4.1 that MFRReg benefits more in the link transfer setting. Moreover, SSReg improves 13 out of 16 physical interac- tion prediction which reinforces GNN to dig more from protein sequence embeddings that physical interactions are more relevant to. This echos our analysis that SSReg benefits more in the node transfer setting, conforming to our sncory-grounded regularization design.
Besides, we observe that the improvements are more significant in the unsupervised setting than semi-supervised. A possible reason is that, under the guidance of certain target labelled data during transferring, DA-W are less prone to capture superfluous information even without regularization.
8

---

Published as a conference paper at ICLR 2023
(iii) Integrating protein sequences together with computational PPIs leads to better trans- fer performance. Comparing with SOTA competitors, we find that utilizing computational PPIs alone (such as Mashup) or utilizing protein sequences alone (such as D-SCRIPT which heavily relies on the pre-trained sequence encoder on a tremendous and diverse population of protein se- quences or even structures (Bepler & Berger 2019) leads to less competitive results than integrat- ing them together when transferring across species. For self-supervised pre-training methods (such as GraphCL), the existence of domain gap prompts "negative transfer" in the unsupervised setting, which is alleviated under the guidance of certain target labels in the semi-supervised setting.
5.2 CLASSIPYING PAPER TOPICS OF DIFFERENT TIME PERIODS
Datasets and Configurationss We also con- duct experiments on the benchmark datasets provided in ArnetMiner (Tang et al.) 2008), to classify paper topics
Table 3: Unsupervised transfer of paper topic classi-
fication in temporally evolved citation networks. Num- bers in red are the best accuracies (mean=ssd%) which without standard deviation are from (Wu et 2020
topics in the temporally shifting citation networks. Papers published between 2000 and 2010 are collected from ACM and those after 2010 are from DBLP, with statistics shown in Appendix[D] The Citation database is not examined here without processed data pub- lic. The networks are built with nodes for pa- pers in six categories (to be predicted) and links representing citations. Following (Wu et al.] 2020), we train methods on ACM/DBLP and evaluate them on DBLP/ACM, which are re- peated for ten times. The compared SOTAs include graph representation learning w/o and w/ DA techniques. Our implementation built upon UDAGCN (Wu et al. 2020). See Appendix for more details. We also assay our methods on the large-scale OGB benchmark (Hu et al. 2020).
Methods
ACM ACM-DBLFF
Link transfor
MFRRcg DBL B-LP-AACM
DeepWalk LINEE
36.49
19.72
41.17
GraphSAGE
72.28
69.61
DNN
42.79
59.04
GCN
64.86
69:45
DGRL Ganin et al. 2016
43,03
$9.47
AdaGCN
71.42
70.45
FERM Wirer pizza
64.95++1.88
633665+0.31
UDAGCN (wo 2020
00.08::0.8873344
75,55=0,31((76 17)
GNIFEDAW
89.08.10.10
75.82:10.05
GNN+DA
+SSRey
90.90+0.08
76.15:10.06
GNN+DA-W W +MFRReg
91.68+00066
76.26+00.05
Hypothesis. Paper topics are verified to have strong homophily with citations mostly (Zhu et al., 2020a Pei et al. 2020) (link transfer), which is adopted as the hypothesis in our study.
Results. The results of unsupervised transfer are shown in Table 3 demon- strating the upplicability of the proposed spectral regularization in different ap- plications. Comparing graph represen- tation learning w/o and w/ DA tech- niques, domain-invariant representations generally improve performances. Further built upon the SOTA UDAGCN, applying DA-W minimize the domain divergence on represen-
Table 4: Paper topic classification on ogbn-arxiv under dif- ferent label rates. Reported numbers are accuracy (%).
Link transfer MFRReg
Label Rate
1%
10%
100%
GNN GNN+DA-N
19
69.00=0.37
71,85:0.19
63,96+0061
68.88
01.93+0.11
GNN+DA W+SSReg
63.53++0.89
69.18=0.31
0.882
GNN-DA- GNN-DA-W-MFREE W+RRg
64.22:10.51
69.21-0.38
72.03-1.233
tations benefits the transfer performance from ACM to DBLP while hurting it from DBLP to ACM,
which is consistent with the observation for the semi-supervised setting) on the large-scale ogbn-arxiv dataset are shown in Table
(i). Similar results (also referring to Appendix G Table
Via spectral regularization, specifically MFRReg to exploit the invariant topology information across domains that is more related to node labels as hypothesized, our methods achieve better performance than all competitors, which is consistent with the observation (ii) in Section 5.1
6 CONCLUSIONS
To fulfill the practical demands of transfer learning on graph data, we develop grouuddd spectral regularization for GNNs to learn transferable graph representations. We first leverage do- main adaptation with optimal transport theory to dive into the guaranteed bound for the transfer performance. This bound reveals that varying the Lipschitz constant of the GNNs could lead to a tighter bound by balancing domain divergence and GNN power. Building the graph filter theory, we next show that one can regularize GNN spectral properties to regularize the Lipsehitz constant, which motivates to propose spectral regularizations. Numerical results conform to our theoretical analysis that regularizing SS and MFR brings benefits to the scenarios of node transfer and link transfer, respectively, in both the unsupervised and supervised settings.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENT
This project was in part supported by the National Institute of General Medical Sciences (R35GM124952 to Y.S.), the National Science Foundation (CCF-1943008 to Y.S.), and the US Army Research Office Young Investigator Award (W91 INF2010240 to Z.W.). Portions of this research were conducted with the advanced computing resources provided by Texas A&M High Performance Research Computing.
REFERENCES
Bruce Alberts, Alexander Johnson, Julian Lewis, Martin Raff, Keith Roberts, and Peter Walter. Molecular biology of the cell. Garland Science, New York, 2002.
Raghu Arghal, Eric Lei, and Shirin Saeedi Bidokhti. Robust graph neural networks via probabilistic lipschitz constraints. arXiv preprint 1rXiv22112.07575, 2021.
Shai Ben-David, John Blitzer, Koby Crammer, Fernando Pereira, al. Analysis of representations for domain adaptation. Advances in neural information processing systems, 19:137, 2007.
Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wort- man Vaughan. A theory of learning from different domains. Machine learning, 79(1)1151-175, 2010.
Tristan Bepler and Bonnie Berger. Learning protein sequence embeddings using information from structure. arXiv preprint arXiv:1902.0866) 2019.
Beatrice Bevilacqua, Yangze Zhou, and Bruno Ribeiro. Size-invariant graph representations for graph classification extrapolations. arXiv preprint arXiv:2103.050453 2021.
Anna Brückner, Cécile Polge. Nicolas Lentze. Daniel Auerbach, and Uwe Schlattner. Yeast two- hybrid, powerful tool for systems biology. International journal of molecular sciences, 10(6): 2763-2788, 2009.
Ruichu Cai, Fengzhu Wu, Zijian Li, Pengfei Wei, Lingling Yi, and Kun Zhang. Graph domain adaptation: A generative view. arXiv preprint arXiv:2106.07482 2021.
Yongqiang Chen, Yonggang Zhang, Han Yang, Kaili Ma, Binghui Xie, Tongliang Liu, Bo Han, and James Cheng. Invariance principle meets -itto--distribution generalization on graphs. arXiv preprint arXiv:2202.054411 2022.
Eli Chien, Wei-Cheng Chang, Cho-Jui Hsieh, Hsiang-Fu Yu, Jiong Zhang, Olgica Milenkovie, and Inderjit S Dhillon. Node feature extraction by self-supervised multi-scale neighborhood predic- tion. arXiv preprint arXiv:2111.00066 2021.
Hyunghoon Cho, Bonnie Berger, and Jian Peng. Compact integration of multi-network topology for functional analysis of genes. Cell systems, 3(6):540-555 548, 2016.
Weilin Cong, Morteza Ramezani, and Mehrdad Mahdavi. On provable benefits of depth in training graph convolutional networks. Advances in Neural Information Processing Systems, 34:9936- 2021.
Quanyu Dai, Xiao Shen, Xiao-Ming Wu, and Dan Wang. Network transfer learning via adversarial domain adaptation with graph convolution. arXiv preprint arXiv: 2019.
Rob M Ewing, Peter Chu. Fred Elisma, Hongyan Li, Paul Taylor, Shane Climie, Linda McBroom- Cerajewski, Mark Robinson, Liam O' O'Connor, Michael Li, et al. Large-scale mapping of human protein-protein interactions by mass spectrometry. Molecular systems biology, 3(1):89, 2007.
Shaohua Fan, Xiao Wang, Chuan Shi, Peng Cui, and Bai Wang. Generalizing graph neural networks out-of-distribution graphs. arXiv preprint arXiv:2111.10657 2021.
10

---

Published as a conference paper at ICLR 2023
Josefin Fernebro, Princy Francis, Patrik Edén, Åke Borg, Ioannis Panagopoulos, Fredrik Mertens, Johan sollon-Christerssonn Mäns Akerman, Anders Rydholm, Henrik CF Bauer, et al. Gene expression profiles relate ss18/ssx. fusion type in synovial sarcoma. International journal of cancer, 118(5): 1 165-117. 2006.
Fernando Gama, Joan Bruna, and Alejandro Ribeiro. Stability properties of graph neural networks.
IEEE Transactions on Signal Processing, 68:5680-5695, 2020.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural net- works. The journal of machine learning research, 17(1):296-2030, 2016.
Vikas Garg, Stefanie Jegelka, and Tommi Jaakkola. Generalization and representational limits of graph neural networks. In International Conference on Machine Learming, pp. 3419-3430. PMLR, 2020.
Shurui Gui, Xiner Li, Limei Wang, and Shuiwang Ji. Good: A graph out-of-distribttion benchmark. arXiv preprint arXiv:2206.084333 2022.
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. Im- proved training of wasserstein gans. arXiv preprint arXiv:1704.00022 2017.
Willem H Haemers. Interlacing cigenvalues and graphs. Linear Algebra and its applications, 226: 593-616, 1995.
William L Hamilton, Rex Ying, and Jure Leskovec. Inductive representation learning on large graphs. In Proceedings of the 31st International Conference Neural Information Processing Systems, pp. 1025-1035, 2017.
Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay Pande, and Jure Leskovec. Strategies for pre-training graph neural networks. arXiv preprint urXiv:/90S. 12265, 2019.
Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. Advances in neural information processing systems, 11-22188-22133 2020.
Wei Jin, Tyler Derr, Haochen Liu, Yiqi Wang, Suhang Wang, Zitao Liu, and Jiliang Tang- Self-supervised learning on graphs: Deep insights and new direction. arXiv preprint arXiv:2006. 10141, 2020.
Wei Jin, Tong Zhao, Jiayuan Ding, Yozen Liu, Jiliang Tang, and Neil Shah. Empowering graph representation learning with test-time graph transformation. arXiv preprint arXiv:2210.03561, 2022.
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Zidek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583-5888 2021.
Mostafa Karimi, Di Wu, Zhangyang Wang, and Yang Shen. Deepaffinity: interpretable deep learn- ing of propound-protein affinity through unified recurrent and convolutional neural networks. Bioinformatics, 35( 35(18)33329-3338 2019.
Mostafa Karimi, Di Wu, Zhangyang Wang, and Yang Shen. Explainable deep relational networks for predicting paapoondd-rotern affinitics and contacts. Journal of Chemical Information and Modeling, 61(1):46 66, 2020a.
Mostafa Karimi, Shaowen Zhu, Yue Cao. and Yang Shen. De novo protein design for novel folds using guided conditional wasserstein generative adversarial networks. Journal of Chemical Infor- mation and Modeling, (0(12):667-56811 2020b.
Henry Kenlay, Dorina Thanou, and Xiaowen Dong. Interpretable stability bounds for spectral graph filters. In International conference machine learning, pp. 5388-5397. PMLR, 2021.
11

---

Published as a conference paper at ICLR 2023
Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional net- works. arXiv preprint arXiv: 1rX.v:1609.02907, 2016.
Maarten Larmuseau, Lieven PC Verbeke, and Kathleen Marchal. Associating expression and ge- nomic data using co-occurrence measures. Biology direct, 14(1):10, 2019.
Bo Li. Yezhen Wang, Shanghang Zhang, Dongsheng Li, Kurt Keutzer, Trevor Darrell, and Han Zhao. Learning invariant representations and risks for supi-vupe domain adaptation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp- 2021a.
Haoyang Li, Xin Wang, Ziwei Zhang, and Wenwu Zhu. Ood-gnn: Out-of-distribttion generalized graph neural network. arXiv preprint arXiv:2112.03800 2021b.
Haoyang Li, Xin Wang, Ziwei Zhang, and Wenwu Zhu. Out-of-distribution generalization graphs: A survey. arXiv preprint arXiv:2202.07987, 2022.
Qimai Li, Xiao-Ming Wu, Han Liu, Xiaotong Zhang, and Zhichao Guan. Label efficient semi- supervised learning via graph filtering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 9582-9591, 2019.
László Lovász. Large networks and graph limity, volume 60. American Mathematical Soc., 2012.
Yishay Mansour, Mehryar Mohri, and Afshin Rostamizadeh. Domain adaptation: Learning bounds and algorithms. arXiv preprint arXiv:0902.3444 33330, 2009.
Haitao Mao, Lun Du, Yujia Zheng, Qiang Fu, Zelin Li, Xu Chen, Han Shi, and Dongmei Zhang. Source free unsupervised graph domain adaptation. arXiv preprint arXiv:21 12.00955, 2021.
Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of machine learning- MIT' press, 2.18.
Christopher Morris, Martin Ritzert, Matthias Fey, William L. Hamilton, Jan Eric Lenssen, Gaurav Rattan, and Martin Grohe. Weisfeiler and leman go neural: Higher-order graph neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 4602-4609, 2019.
Saket Navlakha and Carl Kingsford. The power of protein interaction networks for associating genes with diseases. Bioinformatics, 2658):1057-1063, 2010.
Antonio Ortega, Pascal Frossard, Jelena Kovačević, José MF Moura, and Pierre Vandergheynst. Graph signal processing: Overview, challenges, and applications. Proceedings of the IEEE, 106 (5):808-828. 2018
Peter Park, Atul J Butte, and Isaac S Kohane. Comparing expression profiles of genes with similar promoter regions. Bioinformatics, 18(12):1576-1584 2002.
Helen Parkinson, Misha Kapushesky, Nikolay Kolesnikov, Gabriella Rustici, Mohammad Shojata- lab, Niran Abeygunawardena, Hugo Berube, Miroslaw Dylag, Ibrahim Emam, Anna Farne, et al. Arrayexpress update from an archive of functional genomies experiments the atlas of gene expression. Nucleic acids research, 37(supp_.1): DD82--2222 2009.
William R Pearson. An introduction to sequence similarity ("homology") searching. Current pro- tocols in bioinformatics, 42(1):3-1, 2013.
Hongbin Pei, Bingzhe Wei, Kevin Chen-Chuan Chang. Yu Lei, and Bo Yang. Geom-gen: Geometric graph convolutional networks. arXiv preprint arXiv:2002.05287, 2020.
Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. Deepwalk: Online learning of social repre- sentations. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 701-710, 2014.
12

---

Published as a conference paper at ICLR 2023
Jiezhong Qiu, Qibin Chen, Yuxiao Dong, Jing Zhang, Hongxia Yang, Ming Ding, Kuansan Wang, and Jie Tang. Gee: Graph contrastive coding for graph neural network pre-training. In Pro- ceedings of the 26th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, pp. 1150-1 160, 2020.
V Srinivasa Rao, K Srinivas, GN Sujini, and GN Kumar. Protein-protein interaction detection: methods and analysis. International journal of proteomics, 2014, 2014.
levgen Redko, Amaury Habrard, and Mare Sebban. Theoretical analysis of domain adaptation with optimal transport. In Joint European Conference on Machine Learning and Knowledge Discovery in Databuses, pp. 737-753. Springer, 2017.
levgen Redko, Emilie Morvant, Amaury Habrard, Marc Sebban, and Younès Bennani. A sur- vey on domain adaptation theory: learning bounds and theoretical guarantees. arXiv preprint arXiv:2004. 1829, 2020.
Kaspar Riesen and Horst Bunke. Approximate graph edit distance computation by means of bipartite graph matching. Image and Vision computing, 27(7):950-959, 2009.
Kaspar Riesen, Xiaoyi Jiang, and Horst Bunke. Exact and inexact graph matching: Methodology and applications. In Managing and Mining Graph Data, pp. 217-247. Springer, 2010.
Luana Ruiz, Luiz Chamon, and Alejandro Ribeiro. Graphon neural networks and the transferability of graph neural networks. Advances in Neural Information Processing Systems, 33, 2020.
Roded Sharan, Igor Ulitsky, and Ron Shamir. Network- based prediction of protein function. Molec- ular systems biology, 3(1):88, 2007.
Jian Shen, Yanru Qu, Weinan Zhang, and Yong Yu. Wasserstein distance guided representation learning for domain adaptation. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Samuel Sledzieski, Rohit Singh, Lenore Cowen, and Bonnie Berger. D-script translates genome to phenome with pruuecce-basede structure- aware, genome-scale predictions of protein-protein interactions. Cell Systems, 12(10);999-982, 2021.
Yongduo Sui, Xiang Wang, Jiancan Wu, Xiangnan He, and Tat-Seng Chua. Deconfounded training for graph neural networks. arXiv preprint
2021.
Ke Sun, Zhanxing Zhu, and Zhouchen Lin. Adagen: Adaboosting graph convolutional networks into deep models. arXiv preprint 1x0vv:/908.003 2019.
Damian Szklarczyk, Annika L Gable, Katerina C Nastou, David Lyon, Rebecca Kirsch, Sampo Pyysalo, Nadezhda T Doncheva, Mare Legeay, Tao Fang, Peer Bork, et al. The string database in protein-protein
2021: customizable protein- protein networks, and functional characterization of user-uploaded gene/measurement sets. Nucleic acids research, 49(D1): 2021.
Arghamitra Talukder, Rujie Yin, Yuanfei Sun, Yang Shen, and Yuning You. Does inter-protein contact prediction benefit from multi-modal data and auxiliary tasks? bioRxiv, pp. 2022-11 11, 2022.
Jian Tang, Meng Qu, Mingzhe Wang, Ming Zhang, Jun Yan, and Qiaozhu Mei. Line: Large-scale information network embedding. In Proceedings of the 24th international conference on world wide web, pp. 1067-1077, 2015.
Jie Tang, Jing Zhang, Limin Yao, Juanzi Li, Li Zhang, and Zhong Su. Ametminer: extraction and mining of academic social networks. In Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining. pp. 990-998, 2008
Yi Tay, Dara Bahri, Liu Yang, Donald Metzler, and Da- Da-Cheng Juan. Sparse sinkhorn attention. In International Conference on Machine Learning, pp. 9438-9447. PMLR, 2020.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arXiv: 1710. 10903, 2017.
13

---

Published as a conference paper at ICLR 2023
Petar Velickovic, William Fedus, William L Hamilton, Pietro Liò, Yoshua Bengio, and R Devon Hjelm. Deep graph infomax. ICLR (Poster), 2(3))4, 2019.
Saurabh Verma and Zhi-Li Zhang. Stability and generalization of graph convolutional neural net- works. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Dis- covery & Data Mining, pp. 1539-1548, 2019.
Cédric Villani. Optimal transport: old and new, volume 338. Springer, 2009.
Yanbang Wang. Yen-Yo Yu Chang, Yunyu Liu, Jure Leskovec, and Pan Li. Inductive representation learning in temporal networks via causal anonymous walks. arXiv preprint arXiv:2101.05974 2021.
Tianxin Wei, Yuning You, Tianlong Chen, Yang Shen, Jingrui He, and Zhangyang Wang. Aug- mentations in hypergraph contrastive learning: Fabricated and generative. arXiv preprint arXiv:2210.03801, 2022.
Felix Wu, Amauri Souza, Tianyi Zhang, Christopher Fifty, Tao Yu, and Kilian Weinberger. Sim- plifying graph convolutional networks. In International conference on machine learning, pp- 6861-687 6871. PMLR, 2019.
Man Wu, Shirui Pan, Chuan Zhou, Xiaojun Chang, and Xingquan Zhu. Unsupervised domain adaptive graph convolutional networks. In Proceedings of The Web Conference 2020, pp. 1457- 1467, 2020.
Qitian Wu, Hengrui Zhang, Junchi Yan, and David Wipf. Handling distribution shifts graphs: An invariance perspective. arXiv preprint 2xXi::200.002666, 2022a.
Ying-Xin Wu, Xiang Wang, An Zhang, Xiangnan He, and Tat-Seng Chua. Discovering invariant rationales for graph neural networks. arXiv preprint arXiv:2201. /2872, 2022b.
Yanchen Xie, Zhao Xu, Jingtun Zhang, Zhengyang Wang, and Shuiwang Ji. Self-supervised learning of graph neural networks: A unified review. arXiv preprint arXiv:2102.10757, 2021.
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanic Jegelka. How powerful are graph neural 2018.
networks? arXiv preprint
Zihao Xu, Guang-Ho He Lee, Yuyang Wang, Hao Wang. al. Graph-relational domain adaptation. arXiv preprint arXiv:2202.03628, 2022.
Hongliang Yan, Yukang Ding, Peihua Li, Qilong Wang, Yong Xu, and Wangmeng Zuo. Mind the class weight bias: Weighted maximum mean discrepancy for unsupervised domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2272-2281, 2017.
Gilad Yehudai, Ethan Fetaya, Eli Meirom, Gal Chechik, and Haggai Maron. From local structures to size generalization in graph neural networks. In International Conference on Machine Learning, pp. 11975-1 1986. PMLR, 2021.
Yuning You and Yang Shen. Cross-modality and self-supervised protein embedding for compound- protein affinity and contact prediction. Bioinformatics, Betaeaaaiiiss.aSaaaaae SusSuppplmeeet2)ii6688 -i74, 2022.
Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. Advances in Neural Information Processing Systems, 33:5812-5823, 2020a.
Yuning You, Tianlong Chen, Zhangyang Wang, and Yang Shen. When does self-supervision help graph convolutional networks? In international conference on machine learning, pp. 10871- 10880, PMLR, 2020b.
Yuning You, Tianlong Chen, Zhangyang Wang, and Yang Shen. L2-gen: Layer-wise and learned efficient training of graph convolutional networks. Proceedings of the IÉEE/CVF Conference Computer Vision and Pattern Recognition, pp. 2127-2135, 2020c.
14

---

Published as a conference paper at ICLR 2023
Yuning You, Tianlong Chen, Yang Shen, and Zhangyang Wang. Graph contrastive learning tomated. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Con- ference Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp- 12121-12132. PMLR, 18-24 Jul 2021.
Yuning You, Tianlong Chen, Zhangyang Wang, and Yang Shen. Bringing your own view: Graph contrastive learning without prefabricated data augmentations. arXiv preprint arXiv:220..01702,
2022.
Hanqing Zeng, Hongkuan Zhou, Ajitesh Srivastava, Raigopal Kannan, and Viktor Prasanna. Graph- saint: Graph sampling based inductive learning method. arXiv preprint urXiv:1907.049)) 2019.
Jiying Zhang, Xi Xiao, Long-Kal Kai Huang, Yu Rong, and Yatao Bian. Fine-tuning graph neural networks via graph topology induced optimal transport. arXiv preprint w:X1::2203.10453, 2022.
Yabin Zhang. Bin Deng, Hui Tang, Lei Zhang, and Kui Jia. Unsupervised multi-class domain adap- tation: Theory, algorithms, and practice. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020.
Yizhou Zhang, Guojie Song. Lun Du, Shuwen Yang, and Yilun Jin. Dane: Domain adaptive network embedding. arXiv preprint arXiv: 1906.00644 2019.
Han Zhao, Remi Tachet Des Combes, Kun Zhang, and Geoffrey Gordon. On learning invariant representations for domain adaptation. In International Conference on Machine Learning, pp- 7523-7532. PMLR, 2019.
Jiong Zhu, Yujum Yan, Lingxiao Zhao, Mark Heimann, Leman Akoglu, and Danai Koutra. Beyond homophily in graph neural networks: Current limitations and effective designs. arXiv preprint arXiv:2006. / 1468, 2020a.
Qi Zhu, Carl Yang, Yidan Xu, Haonan Wang, Chao Zhang, and Jiawei Han. Transfer learning of graph neural networks with ego- -ro-graph information maximization. Advances in Neural Informa- tion Processing Systems, 34, 2021.
Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. Deep graph contrastive representation learning. arXiv preprint 2006:0000004133 2020b.
15

---

Published as a conference paper at ICLR 2023
APPENDIX
A PROOF FOR LEMMA 1
D Lemma 1. Suppose that G is the set for graphs of the size NG after padding with isolated nodes, similar to (Zhu et al. 2021). Given VG1, G2 E G and A = U J, A2 = U2AUU2 the eigen decomposition for adjacency matrices A and A2 that A₁ diag [A,, NI, A2 diag((A2, .... A2.Nol) (eigen values are sorted in the descending order). A GNN is con- structed by composing a graph filter and nonlinear mapping that f(G,) ((o((((A)) W)) (A)
where S is the polynomial function that S(A1) = Ek=o Sx W E DDD'' is the learnable weight matrix, r is the mean/sum/max readout function to pool node repre- sentations, and the pointwise nonlinearity holds as (a(b) (a)) S |b al, Va, b R. Assuming
|| S and |Wl I 1, the following inequality holds:
- f(G2)ll S 5
/V - P A2P IF+ + - P'A2P*T(())
+ +max {IS(A>) ||x - P'X2ll..
|
where + = (||U, U2 F + 1)2 stands for the eigenvector misalignment which can be bounded, p = argmin X1 PX2lFF + || - PAyP" [lF]. II is the set of permutation matrices, O(IIA - P
PMA,, A2 p-1 [|p) is the remainder term with bounded multipliers defined in (Gama et al.
2020). and C2 is the spectral Lipschitz constant that Vd,, Ay, (S().) S(X,) S Cx|A: Ag1-
Proof. Denote the optimal permutation matrix for G1, G2 as P we compute the difference of the
GNN outputs:
f(Gz)l
- (((
S(P || (1 (IS(A)) - -(P
Ag S(P"
+ S SPP -(P""A" X211F)
|| |X (1llop1(S(A) - S(P" A.P 12PTT)ll= + || (p (1)p - P*X2ll
-
max [((AA) S(P" A₂ P.) |F max (1S ) P* X2|1F
2 (C(1+T/NN)l4) PP P2 O1| - P2 A + mxx((S PP where (a) is due the permutation invariance property of graph filters; (b) is achieved with the triangle inequality and the assumption lo(6) - [ <
a), Va, E R; (c) and (d) use the fact Bllo)) and (c) further applies that for any two matrices A, B3 ||AB]] S min[[]A|]op1111 |b1||/ the triangle inequality; (e) adopts the assumption || < 1, llop < which in practice can be guaranteed with normalization, and easily extended to the case with || S K, || N K, AK 0, and because S(P* S(P*AgpPTT
can be diagonalized, its operator norms equal the spectral radius; (f) is the direct outcome borrowed from (Gama et al. (020) Theorem We complete the proof.
(P*U, (A2)
B PROOF FOR LEMMA 2
O Lemma 2. Suppose that G is the set for graphs of the size NG after padding with isolated nodes,
similar to (Zhu et al. 2021). ). Define the matching distance between G1, G2 E G as 17G G2) = minpen {
PX2 + || PA,PT]IF) [I=). Suppose that the edge perturbation is bounded that VG1, G2 E 9, [A - PT S with the optimal permutation p*, and there exists an eigenvalue X E R to achieve the maximum [S()*) < co. We can then calculate the Lipschitz
constant of GNN as:
C = max {C.Ki + eK,, te,,, (S(2))
16

---

Published as a conference paper at ICLR 2023
where K1, K2 is the supremes of (1 TV/G) and the remainder multiplier in Lemma 1, respec- tively, following similar philosophies in (/ama et al. 2020), Theorem 1).
Proof. To calculate the Lipschitz constant Cy w.I.t the matching distance, based upon Lemma 1, assure the following inequality:
(lf()) f(G2)ll2 & Cx(1 TV Nc) A1 - P" A2 |F O(IIA, -P" 1²
+ +| - P P2X2lFF,
CCn G2).
the latter inequality of which can be rewritten as:
(CalL+T/NG)ll P" 1, O(IIA1 P" 1 - PP -
- P2 S S0,
which is necessary for:
Cx(1 + +T P A2P O(1IA) P2A2TT C,IIA, - A2 50
-y)lxi- P2 S0,
which is equivalent to:
C 2 C2Ki + C₁ 2 |S(A")--
The bounding of K1, K2 follows (Gama et al. 2020) Theorem 1 and the first minimum solution can be calculated from the quadratic function w.r.t. the edge matching distance || - P P² ||r. Let Cj takes the larger value between them, we complete the proof.
C PROOF FOR LEMMA
D Lemma 3. Under the assumption of Corollary 1, we further assume that there exists small
Nr. amount of i.i.d. samples with labels {Gn, Yn)) from the target distribution (P Y) (N₃ < Ng) and bring in the conditional shift assumption that domains have different labeling function hig + fuT and maxG,,G2 [ -ann))nnnnaa))
Cn S CfC₁ (D E (S, T}) for some constant C₁ and distance measure 17. Let == (h: 9 + y be the set of bounded real-valued functions with the pseudo sseudo-dimension Pdim(H)) d, with probability at least 1 - 8 the following inequality holds:
y(L,,G2)
N₂
Ns
(T(h, -
(h, PuT) + Ns + N₁
ēs Ras)
Ns N2
Ns
+ Ns + N/.
WI PT +
Sd
eN
2
Sd
eNs d
2
log 1
+
log(
log (
+ N
log(
+
No.
Nr
Ns
where w = min (h, hy) - es (h, hr)), (er(h, he) - eT (h, hr)))-
Before showing the designated lemma, we first introduce the following inequality to be used that:
Proof.
(cs(h, hss) cr hr))
= (e, his)) es(h, + es(h, hr) er Kor)l
(es his) es(h, hr)) (es( et (h( hr))
(e, his) cs +0, Wi (P PP(())
where
(a) results from (Shen et al. 2018 (nn(G))-nn(Gn)) ) CyC D {S, T). Similarly, obtain: G()..Gu)
Lemma 1 with the assumption
max([IA|lLap. maxg,
(es(h, (s) (t hr)) (er hss) er hr) 20,0,W, (P(C)))()))
17

---

Published as a conference paper at ICLR 2023
We therefore combine them into:
les (h, his) er hT) & 2CyCyWi (Ps ( + min (les(M, hss) - es(h, lero) cr(h, hvr)):
i.e. the following holds bound the target risk eT (h, hr): eh hit) (s) hss) + W (Ps()),(
min (les(h,, his) es(h, (er),, hs) eT eT (h,, PoT))) We next link the bound with the empirical risk and labeled sample size by showing, with probability at least 1 - that:
Es hr)) ler hs) - eT (h huT) Es(h, hss + 2 PT())) + min (Es hs)
(h,, hrT))
Es hs) + Wi Ps(()),P((C) + min (cs((h, hS) cs(h,, ler cr cr(h,, (v))
2d
eNs
+
Ns
log
d
+
V 2Ns
log( 8
and:
2d
EN d
1 0
(T hT)) S (T(H, +
log( NY
log( 2N
which results from Mohri al. 2018) ties, with probability at least ò that:
Theorem 11.8. Lastly, we combine the above inequali-
eT(h, fTT)
(a)
N₂
2d N1
eNV.
(ér(h, +
log(
log( 5))
Ns + N/
d
2N
Ns
2d
e.s
(és(h, + V
log(
log( $)
Ns + NoT
Ns
2Ns
Ns
(2C Pr)))))))) + min (aan(lls(( (h, his) cs, herr.llrrhh.............
(er hs) - er er(h, (rr)))))))
Ns + NT
NG
( (cr(h, (((),, +V
4d
log(
eN4
+
log( 1 }
Ns + N2.
N/
d
N7.
+ NS + N%.
Ns
(Es(),hs) + V
4d
log(
e.s
1
log( )
Ns
d
Ns
Ns
WC PT()) + min (h,, his) es (er - er(h, er(h,hr)l)) hrr)))
(er(h, hs)
NS + No.
(0)
N
Ns (h hss)
ét (T (h)
Ns + N
Ns + No4
Ns
(PPP)))
+ min(les hs) es(h, hT)),, leT (h, hs) et er hrr))l)))
NS + NZ
8d
eN
2
1
8d
eNs
2 1 11
N2
log(
d
No
log(
8
Ns
log
+
Ns log(
Nº where (a) is the outcome of applying the union bound with coefficient Ns+NG Ng- respectively;
No
(b) and (c) result from the Cauchy-Schwarde inequality and (c) additionally adopt the assumption
N < Ns. following the sleight-of-hang in (Li et al. 202 Theorem 3.2.
D DATASET STATISTICS
For PPI data, we construct the graph with nodes as protein sequences (using protein language model- ing (Tay et al. 2020 Karimi al. 2019 for node features), and edges computational interactions
18

---

Published as a conference paper at ICLR 2023
of neighborhood. fusion and co-occurrence channels (Szklarezyk al. 2021 (i.e. the edge feature dimension is 3).
Dataset statisties for PPI and citation networks shown in Figure and respectively. For the species involved PPI networks, we depict their relationship biological taxonomic ranks in Figure 4 showing the domain gap between species in concept.
Table 5: Dataset statistics of PPI networks of different species. denotes "number of".
Species
Node
Edge
# Co-expression
# Physical
Human
8.369
201,164
15,344
68,812
Mouse
55,55
55,668
4,490
11,264
Zebrafish
4,625
60,077
16,044
150
Fruit fly
5,503
108,858
25,404
14,694
Yeast
4,286
207,641
39,040
60,716
Kingdom
Animalia
Fungi
Phylum
Chordata
Arthropoda
Ascomycota
Class
Mamnnalia
Actinopterygii
Inseetaa
[lemiascomyceter
Order
Primates
Rodentia
Cypriniformes
Diptera
Saccharomycetalles
Family: Hominidae
Family Muridac
Family: Cypriniine Danso Speciec Danio rerio
Family: Prosophilidne
Family: secchroomy.cacceee
Gesus: Homo
Gemus: Mus Species: Mus
Genus: Drosophila
Genus: Saccharomyces
Species: Home
Species: Drosuphila
Species: Snocharomyces
sapiens
musculus
melanogasier
Zebrufish
Human
Mouse
Fruit fly
Ycast
Figure 4: relationship among species biological taxonomic ranks.
Table 6: Dataset statistics of citation networks from different sources (in diffcrent time periods). # denotes "number of".
Sources (periods)
# Node
Edge
Feature
Class
DBLP (2000-2010)
5,578
7,341 11,135
7,537 7.537
6
ACM (2010-)
7,410
6
E DETAILED CONFIGURATIONS
The assayed data are released under the MIT license, and to our best knowledge, contain privacy- infringing contents. Experiments are distributed on computer clusters with Tesla K80 GPU (11 GB memory) and NVIDIA A100 GPU (40 GB memory).
E. WASSERSTEIN-1 DISTANCE ESTIMATION
We follow the most routine procedure (Shen et al, 2018) to estimate Wasserstein-1 distance of graph representations between distributions in adversarial training (3). Speeciiically, given the encoder
19

---

Published as a conference paper at ICLR 2023
and source and target data distributions Ps(G),Pr(G), we estimate the distance as:
Wi (Ps(G),Pr(G))
= max (pa(c)((((G) (c l1fils
where the critic function f is instantiated with a multilayer perceptron and satisfies [[ flfil)) < 1, which is achieved by enforcing the gradient penalty (IV=f(<=)l2 - 1)². sup 1z-
The final estimation is thus implemented as:
W1 (Ps(C).Pr(G)) -max (GC) (fC) -(VV -
where we follow (Shen et al.
2018
Gulrajani et al. 2017) to set 9 10 by default.
E.2 PREDICTING PROTEIN PROTEEIN INTERACTIONS ACROSS VARIOUS SPECIES
During collecting PPI data from the STRING database (Szklarczyk al. 202Tp, we abandon the interactions in channels of beigaberrtoourrunseeeedddd coexpression_Jrangy Jransferre,, and experi- trans-rra to prevent information leakage acToss species (especially from supervisions of co-expression and physical interaction prediction). For co-expression and physical interactions, we use the high-quality threshold of 0.7 (Szklarezyk et al.
2021) to convert them into binary labels.
We use Sinkhorn Transformer TTy et al. 2020) (a variant of Transformer with sparse attention) with depth attention head 4 and bucket size 32 embed protein sequences, and further adopt GIN (Xu et al. 2018) with depth 3 and MLP depth 2 to perform message passing across proteins. We also try HRNN (Kaimii al., 2019; (020a) with k-mer
to embed protein sequence and GAT (Veličkovid et al. 2017) with depth 3 and attention head 8 to perform message passing, with comparison on human to yeast transfer shown in Table 7 which states in our case that, (Sinkhorn) Transformer and GIN outperform HRNN and GAT, respectively. In training, we hold out 20% of human PPIs for validation. For the semi-supervised setting. 0.1 of experimental PPIs are in addition available during training. We train with convergence assured for 500 epochs with learning rate 0.0001, hidden dimension 256 and batch size 128 which is sampled by random walk, optimized by Adam optimizer. For domain-invariant representation learning guided by Wasserstein distance (as described in opti- mization (3)) please refer to (Shen et al. 2018) for implementation, and see (Ganin et al. 2016) for one with domain classifiers. We selected the trade-off factor y in optimization (3) from lc-1, 1c-2, le-3, lc-4, lc-5, 1c-6} through validation. In evaluation, we follow (Cho et al. 2016) to perform negative sampling for experimental PPIs, to ensure that the known interactions compose only 5%, with the assumption the number of unobserved interactions is only small fraction.
Table 7: Comparisons between different protein sequence encoders and GNNs human to yeast transfer setting.
Co-expression
Physical
Methods
AUORC (%)
AUPRC (%)
AUORC (%)
AUPRC (%)
HRNN 1.55 6.16
8.91
65.24=1.17
9.71
Transformer
9.11
66.65=2.28
11.75=#1.46
Transformer+GAT
60.85+11988
9.00+1.55
70.5
14.53:2.22
Transformer+GIN 61.18
10.90+11.40
71.15=1.444
15.02+t1.79
On baseline implementation, for Mashup (Cho et al.) 2016), we apply the official MATLAB code to extract protein representation w.r.t. computational PPIs in the neighborhood. fusion and co- occurrence channels, on top of which we apply a two-layer MIP to train with labels; for D-SCRIPT (Sledzieski et al.) 2021). we apply the official PyTorch APIs to perform the sequence embedding projection, inter-protein residue contact and interaction modeling, where we use the same Sinkhorn Transformer as in our methods to generate sequence embeddings rather than the large model pre- trained on additional data of protein sequences with structures, for fair comparison and also com- plexity consideration; for GraphCL (You et al., 2020a), we follow the main idea and implement with the node masking augmentation, where we randomly replace 20% of animo acids in each protein sequences with mask tokens.
20

---

Published as a conference paper at ICLR 2023
E.3 CLASSIFYING PAPER TOPICS OF DIFFERENT TIME PERIODS
We follow the same experiment setting as in UDAGCN (Wu et al., 2020). which we implement DA-W and spectral regularization on. The compared SOTAs without DA techniques are DeepWalk (Perozzi et al.) 2014), LINE (Tang et al. 2013), GraphSAGE (Hamilton et al. 2017). DNN (MLP on node features alone), and GCN (Kipf & Welling 2016), and ones with DA techniques are DGRL Ganin et al. 2016). AdaGCN (Dai et al. 2019). and UDAGCN (Wu et al. 2020). We notice that reported results (Wu et al. 2020) are not directly comparable with (Mao et al. 2021) due to different experimental configurations (e.g. validation partition). By applying DA-W, we replace the domain classifier with the Wasserstein distance critic (Shen et al. 2018) to learn domain-invariant representations, and further implement our proposed spectral regularization.
F HOMOPHILY RATIO COMPUTATION PROCEDURE
Given graph G = {V. }} with the set of nodes V and edges E as described in Section 2. Denote the labeled edge set as E'. to quantify the association between and E', we borrow the idea from (Zhu et al. 2020a Pei et al. 2020) to calculate the homophily ratio as:
kle, w) w (v) and (v, w) E'H
Hom
(W(b)
BEV
where Nv) is the set of neighbor nodes of v determined by E. In cross-species PPI prediction, E' is the experimental (co-expression physical) PPIs. To measure the association between compu- tational and experimental PPIs, we define as the set of edges if there any computational PPI (neighborhood, fusion or co-occurrence) value greater than the medium threshold of 0.4 (Szklarczyk) et al.] (202T) To measure the association between protein sequences and experimental PPIs, we first calculate the sequence identity (Karimi et al.) (0200) between all protein pairs, and define E the set of edges if the sequence identity is greater than 0.3 (Pearson, 2013).
G MORE EXPERIMENTAL RESULTS
Results of cupersuue cross spossespecies protein- pootein-protein interaction prediction are shown in Tables 8, [ semi-supervised paper topic classification in Table 10] ablation studies of model architectures and adversarial training strategies in Tables
12
Table 8: Semi-supervised transfer of cross-species protein-proteio co-expression interaction prediction. Num- bers in red are the top-2 AUROC (AUPRC) (mdan- DA-C. DA-W denotes domain-invarian representa- tion learning with domain classifiers (Ganin et al.) 2016) and guided by Wasserstein distance as in optimization
(3) RRedko al. 2017;
Shen al. 2018), respectively.
Co-expression: Link transfer
MFRReg Yeast
Methods
Mouse
Zebrafish
Fruit fly
Meant
Rank.
6..06
$1.80+5.64
53.86++2.04
48.79++5.36
50.87
Mashup
(5.66=0.73)
(5.64:0.61)
(5.86+0.27)
(5.18:00.4)
(5.58)
9.0
$2.85+10.45
61.76++..45
61.01+1.14
56.52+00.11
58.03
D-SCRIPT
7.7
06.07+0.10)
(11.12+1.94)
(8.59+0.20)
(7.64+0.115
(8.35)
74.62:+2.75
75.26::0.64
1.75
70.15:10.97
71.14
GraphCL
4.8
(17.54++4.70)
(22.93++2.42)
(11.26++0.98)
(20.75+3.88)
(18.12)
71.57+11.1
69.87+0.52
61.76+2..1
67.01+0.700
67.55
Transformer
6.2
(20.39+0.900)
(26.09+1.58)
(12.14+11600
(14.11+1.24)
(18.18)
Transformer
17.86=3..13
301.9913.79
66.92+:1.65
67.07-0.71
73.06
3.5
+GIN
(22.17+:7.86)
(30.15::5.266
(8.04=00.53)
(14.43=+1.92)
(18.69)
Transformer
75.13-+33.2
79.01 +0.96
66.02+11.19
68.12+00.9
72.07
4.8
+GIN+DA-C
(17.71=1.39)
(28.54:1 (.441.88))
(8.34
(17.65=11.6)
(18.06)
Transformer
77.06=t0.74
1..4
71.11+2.11
67.60+2.77
73.81
+IN+-A--
(20.63++1.81)
(28.96++2.75)
(16.04+00.44)
(15.14+2.00)
(20.19)
Transformer+GIN
76.65.11.44
80.19:10.74
69.36:12.25
67.5 67.58-12.56
73.44
3.2
-DA-W+SSReg
(20.66:14.17)
(27.23:10.75)
(13,56 544)
(18.46:12.22)
(19.97)
Transformer+GIN
77.26:10.91
19.19:1.24
66.58+11.24
68.90-11.13
72.98
2.5
+DA- D--RRgg
(24.16+0.955
(30.02+-0.29)
(12.33+00.3)
(20.10+3.28))
(21.65)
21

---

Published as a conference paper at ICLR 2023
Table 9; Semi-supervised transfer of cross-species -prteinn physical interaction prediction.
Physical: Node trunsfer
SSReg Yeast
Methods
Mouse
Zehrafish
Fruit fly
Meant
Rank]
57.90:+:300
63.88477.77
46.84:16.11
51.68+-5.48
55.07
Mashup
8.7
(9.23+4.18)
(7.63+1.36)
(5.93+1.332)
(5.89+1.222)
(7.17)
49.34:10.67
68.41
69.79:+1.79
59.66++1002
61.85
D-SCRIPT
8.2
(5.87=0.09)
(19.36+t4.99)
(20.25:#0.78)
(7.46+0.115)
(13.23)
79.08++0.63
82.61+1.411
81.04+00.62
70.06 10
78.19
GraphCL
4.6
(34.96:11.31)
(42.81)12.911
(38.65:1 1.40)
(13.33.13.69)
(34.18)
77.44::0.84
7..50+1.20
78.14:0.229
69.15+1.00
75.55
Transformer
6.0
(34.63+0.77)
(46.49:2.333)
135.56::1554)
(14.64#1.88)
(32.58)
Transformer
79.76+22.26
81,09+1.46
82.41+00.96
12.59++1.08
78.9
+GIN
(34,58:11900)
(41,46:14.0) 02)
(40.14
(16.76::144)
(33.23)
Transformer
80.47:10.60
19.88+1001
8..0
71.42+0.67
78.21
4.5
+GIN+DA-C
(35.62++0.43)
(42.28++2.22)
(37.37++.59))
(15.624-2.01)
(32.72)
Transformer
19.72=#1.13
83,45:3.44
1.236::1.21
73.33-11.84
79.71
+GIN+D-WW
(35.47:42.79)
(45.05+22.37)
(44.07=3.422)
(18.78+5.12)
(35.84)
Transformer+GIN
80.69:10.40
84.6942.29
81.9240811
73.01+00.88
80.07
1.8
DDA-WeSSReg
13.440++.334
(45.9543.66)
(41.73+2.55)
(19.08:4
(35.78)
Transformer+Gin
79.73-10.86
$1,6911088
79.54:11.0
72.6410.85
78.40
4.2
+DA DD--WRMFRReg
(31.98:1.554
(42.36::0.38)
(38.97+1.57)
(19.42=1.29)
(33.18)
Table 10: Paper topic semi-supervised classification on ogbn-arxiv under different label rates, Reported num- bers are accuracy (%).
transfer
MFRReg
Label Rate
1%
10%
100%
GNN GNN+DA-
66.64=0.64
69.35=#0.52
71. 71.96=0.22
67.53-0.58
68.8240.43
71.98+0.21
GNN+DA W+SSReg
67.71=0.43
69.92:20.28
72.04=0.17
GNN-DA-W-MFRR. W+RRRgg
67.73=0.42
69.9920.37
72.06=0.29
Unsupervised transfer cross-species (human yeast) protein-pon prott-prottnn co-expression interaction prediction. Numbers in red are the best performance among the sub-row.
Table 11:
Co-expression: Link transfer MFRReg
Physical: Node transfer SSReg
Methods
AUROC (%)
AUPPRC (%)
AUROC (%)
AUPPCC (%)
Transformer+GIK
63.91 1,55
11.15+-1.11
71.54-0.36
15.73-40.79
Transforeer+
61.63.10.41
13.62.1.79
71.8911.78
17.80.1..58
63.01=3.05
15.32=1.84
71.8321.17
17.01#2..09
Transfom==+GIN--DAACCC
60.65+3.85
10.7242.44
71.30:0.61
16.80=0.65
T CDSSRe
61.28:12.79
12.3911.33
73.07:10.85
18.1312.39
66.03:20.85
14.51:2.90
71.61
16.01:11422
Table 12: Unsupervised transfer of cross-species (human to fruit fly) protein-protein co-expression interaction prediction..
Co-expression: Link transler = MFRReg
Physical: Node transfer =b SSReg
Methods
AUROC (%)
AUPRC (%)
AUROC (%)
AUPRC (%)
Transfomer+GGN
66.54#1.11
13.48=0.71
1..38+1.13
42.40+2.00
Thonsformers.NN-SSS.....
63. 12+0077
8.91=144
82.50==0.54
40:90++2.92
Transfommrr-GINN-MMRRRRRR
Transommer-GGN---A...
64.78.1 1.23
11.61.12.08
81.49.1
38 38.94.12.36
Translomeur-GiI--AA-CCCC
6..30.11.551
82.32-10.70
38.87:11.82
Tausfomms=<GINNDDD--CMMMMMB
66.92:t1.26
11.45:22.55
82.28-20.68
11.29::1.76
H COMPLEXITY ANALYSIS OF SPECTRAL REGULARIZATION
Time. Let's consider a graph of nodes. The spectral regularization is composed of two steps: extracting spectral signals (with eigenvalue decomposition (EVD) involved, complexity O(N3)
22

---

Published as a conference paper at ICLR 2023
and neural network propagation (complexity O(N) You al. 2020c)). Thus, in theory, the former step dominates the time complexity.
In practice, when training large networks, mini-batch training is commonly adopted (Hamilton ct al., 2017| Zeng et al. 2019) to avoid computational burden and empirically for better generaliz- ability (Hamilton al. 2017), where the node number in each batch is restricted to be much smaller than the whole network. In our experiments, batch size is set as 128, with I 3 seconds cost brought for each epoch (see Table 13 which is five times less than the cost brought by adversarial training (20 seconds, see Table 13). We also provide the time consumption for the full training in Table 15 including the best- rest-performed SOTA GraphCL. where the time consumption for GraphCL was around 2 times higher in pretraining and 2 times lower in finetuning compared with training from scratch. Nevertheless, training from scratch did not take longer than 10 hours. We further provide Table 14 to demonstrate the numerical running time of EVD is acceptable as we see in Table 14 with batch size < 2048.
Nevertheless, if it is believed that there are indispensable benefits from the full-batch training, we can always perform the full-batch EVD for only once before training, for the repetitive usage later.
Moreover, we would like to clarify that the spectral properties of graph filters in theory do not have to depend on inputs (sampled graphs or not, an analog is in signal processing, a low-pass system, here "low-pass" is the spectral property, would suppress high frequencies for any input signal).
In details, suppose that the graph filter function S(.) with certain domain Dj (which is a polyno- mial function resided in GNN, see Lemma 1) is constructed (or regularized) to be Cx-Lipschitz and bounded by B (i.e. Vz E | [S(2]] S B), then for an arbitrary graph with the adjacency matrix A UAUT with all eigenvalues range in D2 (i.e. AA E X, E D2), the following applies. If D2 C D1, the graph filter S = (SA Ortega et al. 2018) preserves the properties of (i) spectral smoothness (SS) with C) that VY,, >3, response (MFR) B that max max|S(A)] < B. We can see from the above statement that, spectral prop- erties of SS and MFR are formulated eigenvalues but not eigenvectors.
(3AA))-()A))
Cx. and (ii) maximum frequency
[A,-A,1
In practice, the spectral regularization is performed in the spectra of sampled networks (with eigen- values in Ay range in Dn) and the readers might concern whether the regularized properties in DR still hold in the spectra of ego-graphs of nodes (with eigenvalues in AÆ range in DR) since whether DE S D3 is unclear. According to the eigenvalue interlacing theorem (Haemers) 1995), since adjacency matrices of ego-graphs are principal submatrices of sampled networks, we have min (An( S min(Ag) S max(A S max(Ag) that essentially leads to DE S DB. Therefore, regularized spectral properties on spectra DE (in practice) should be preserved on spectra DB (in theory).
Memory. Beyond time complexity, memory consumption results from data processing (while all compared methods use the same amount of data) and model propagation. Compared to the best-performed SOTA GraphCI., memory consumption was similar between GraphCI. and (ia.ffomerrGiNNtD-----sesseeeee
since GraphCI. also uses the same Transformer+GiN back- bone architecture to extract protein representations (Transformer to embed protein sequences as vertex features, and then GIN to conduct message passing along topology, please refer to Appendix E for details). GPU memory taken by Transformer+GIN was around 24GB which is within the capacity of the conventional NVIDIA 1100 GPU (40 GB memory).
The improving AUPRC of PPI prediction with regard to such affordable computational resources is significant. The computational resource is usually not the bottleneck for in-silico methods (in case, NVIDIA A100 GPU less than 10 hours), while detecting PPI (usually highly imbal- anced Rao et al. (2014) via wet laboratories is very costly with expensive reagents for weeks or months. For instance, the routine yeast two-hybrid system requires steps including building vec- tors, transforming plasmids, cell culture, luciferase assay, etc Brückner et al. (2009), where accurate predictions could play a critical role in accelerating the process.
I LEMMA 4: LIPSCHITZ CONSTANT OF TWO-LAYER GNN
Lemma 4. Suppose that 9 is the set for graphs of the size the size Ng after padding with isolated nodes, similar to (Zhu al. 202T). Following the setting in Lemma I and 2, GNN layer is
23

---

Published as a conference paper at ICLR 2023
Tabbe 13: Running time for epoch training in unsupervised cross-species (human yeast) protein-proteie interaction prediction.
Transformer +GIN
Transformer +GIN+DA-W
Transformer+CINN +DA- W+SSReg
Transformer-GiNN
+DA- M
Time
45s
65s
68s
66s
Table 14: Running time for eigenvalue decomposition under different batch sizes in unsupervised cross-species (human to yeast) protein protein-protein interaction prediction.
Batch size
128
512
2048
Time
0.005s
0.01s
0.08s
3.63s
Table 15: Running time for full training in unsupervised cross-species (human to yeast) protein- protein inter- action prediction.
GraphLL Pretraining
GraphCL Finetuning
GraphCL Pretriniing-Finetuiig
Transformer +GIN
Transformer -INN-D--
Transfrmer+GiN +DA W-SSRe
Transfomer+Git
-DA-W-MF W-MFRReg
Time
12.22h
3.75h
15 97hh
6,25h
9.02h
9.44
9.16h
constructed (G(G (A) and a two-layer GNN as S(G) ((2) (G), where r is the mean/sum/mat readout function to pool node representations. Denote the SS and
(t) MFR terms of (() (!) c Ki + æK2, MFR (At)) that c = max ( (1)
MER),
IsS
(1,2) per Lemma 2, we can then calculate the Lipschitz constant of GNN as:
C = max
SS
(1) (2)
? MER / f.
MER
Proof. The key step is to recognize that the input of the 2nd GNN layer actually a graph with the same adjacency matrix A, but different node feature matrix ( (G)) Denote the optimal permutation matrix for G1, G2 as P we thus compute the difference of the GNN outputs:
- -f(G))-f(22ll = =I(ro (2)) ( (ro f(2)) (0)
(1 No)] P" P2 || + +((1 Pa PM Ag 1TT) + (ax()s(( pa x
-1 + / A3Pt 0 [A1 P" Ayp**1(F)
+ +max 22) (A2))) (1 (1+T/ P" A.P*T |F + +(l1A11 P P'AyPTT
- (m (A2))) (1+
+ m
||4; PP P
[12) + max(( (A2)1) max((s P Palll (1+ ( 0(11 P" A²
where (a), (b) are due to the reuse the inequalities (a)-(f) in Lemma 1. Next, following the same spirit in Lemma 2, to calculate the Lipschitz constant of f, we assure the inequality:
(oP (1 (LL+VVN)++ No) + (P))
(1 (1+++ +T)N))) ||I| P P2P2
+ [ + (27)) P" pT (12) + (S(2) (St2s (X-)]I - I Cyn(G1,Ga)
that is:
2) (1 + + (10))))) (1 - C;)11A1 PD P2
(1 (x))) 0(114 P" P" A + (188) -x))) P"Xx $0,
24

---

Published as a conference paper at ICLR 2023
which is necessary for:
(1 No) -o) ||Ar - P A2P"TIIF ||F
+11-18 0(1A, P2 [i] 10
((,)))))))) Gi) ||x1 P=X2|1F 50,
which is equivalent to
2 2 (c Ki + +xK2)+ + (A+)I( K2 +eK2),
C 2 (8 (x))())))))
25


Reviews:
[{"review_id": "u0RzEZV5-q", "reviewer": "Reviewer_no2m", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The main idea of the article is to adapt the results about bounds in domain adaptation (from [Redko et al., 2017] and following wowrk) to situations involving graphs instead of Euclidean space. After a theoretical analysis, very close to the existing results in non-graph settings, the authors focus only on one term which drives the Lipschitz property of GNN (acting as a feature extractor) on graphs. Their methodological work then amounts to add a regularization over the graph spectrum based on spectral smoothness and/or maximum frequency response of the graph filter inside the GNN. Some numerical experiments on two types of datasets (Protein-Protein Interactions and Topic classification of articles) to study the possible performance of the method.", "summary_of_the_review": "As a summary, given the aforementioned weaknesses, I currently vote against the acceptation of the article, which appears to me to be incremental both on the theoretical aspect and on the methodological+experimental side.\n\nUpdate: scores raised after the answers.", "strength_and_weaknesses": "Strength:\n\n1- the idea of adapting  [Redko et al., 2017]  to graphs is relevant ;\n\n2- the obtained theoretical result (Corollary 1 and the two Lemmas of 4.1) appear to be sound.\n\nWeaknesses:\n\n1- the theoretical  results are obtained in a straightfoward re-purposing of the aforementioned works ; they don't look wrong, but not that original in lights of existing works.\n\n2- the reach of this results is limited because, as soon as it is obtained, the authors limit themselves to the $C_f$ Lipschitz constant and the proposed method appear then to be only a slight improvement on existing architectures ;\n\n3- the proposed regularizations, SSReg or MFRReg have an impact which is almost minor in Tables 1 and 2 ; and again in Tables 3 and 4 when comparing to GNN with DA-W ;\n\n4- most of the really novel content is of experimental nature (in Section 5) and the observations reported in 5.1 and 5.2 appear to be quite fragile: i) I am not convinced by the paragraph \"Hypotheses\" in 5.1 on the two tasks on PPI and I think that more experimental validations should be sought for before concluding that ; and ii) the discussion under \"Results\" in 5.1 is not fully convincing about the impact of DA. Ifw we follow the metrics in term of ranks proposed by the authors, methods with DA-W appear to perform better in Tables 1 and 2 than without. Looking at specific instances is toio much cherry-picking of results that seem to saturate (the improvement by the proposed method is often on the third digit in percentage)\n\n5- the article leverages many \"big\" models (Transformes, GNN with many layers) yet there is no major study about the computation power / time / expense incurred when doing this study (the only reported times are in Tables 13 and 14 in SI); The obtained marginal improvements should be discussed and put in balance with the increased computational costs. For instance, is the less than 5% increase in performance w.r.t. GraphCL  useful ? What's the difference in price (or resource consumption) between what is proposed and this latter method, for instance.\n     ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "llIRzty05BU", "reviewer": "Reviewer_xYtM", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies learning with graphs under the covariate shift assumption. The predictor is decomposed into two functions; one is the feature extractor of the graph, f, and the other is the discriminator, g. The generalization performance of the predictor h(G)=g(f(G)) for a graph G is governed by the Lipschitz constant of f and g. Through analysis of the theoretical properties of the feature extractor defined on the set of graphs, the authors proposed two regularization methods; spectral smoothness(SS) for the node transfer and maximum frequency response(MFR) for the link transfer. The theoretical findings well explain numerical results for link prediction and node classification. ", "summary_of_the_review": "The paper is clearly written. The regularization methods for node transfer and link transfer are an interesting contribution of this paper. ", "strength_and_weaknesses": "Strength\n- In the paper, the transfer learning on the graph is roughly categorized into node transfer and link transfer. For each setting of the transfer learning, an appropriate regularization for the feature extractor is presented. An important insight provided by this paper is to recognize two types of information transformations on the graph. For each type, a reasonable regularization is proposed. \n\nWeaknesses\n- Only a single-layer architecture is used as the feature extractor f(G). Extension to multi-layer networks may be rather straightforward, though computational cost becomes high. A brief comment on such an extension can be beneficial for readers.\n- The categorization of node transfer and link transfer is interesting and important. Usually, we may not have sufficient prior knowledge of the scenario. Is it possible to determine which scenario occurs in practice? Which kind of information is beneficial to identify which scenario occurs?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "b5KlPvlnPlR", "reviewer": "Reviewer_bQEk", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes theory-driven graph domain adaptation regularization terms, including spectral smoothing (SS) and maximum frequency response (MFR). The authors try to extend the theorem in generic domain adaptation into the graph field and focus on the definition of Lipschitz on GNN. Via analyzing the graph Lipschitz property, the authors develop two reguarlizations that directly minimize the upper-bound of the target risk (the right hand of the inequality). The experiments also verify the effectiveness of the idea. ", "summary_of_the_review": "I'd like to increase my score during the discussion period.", "strength_and_weaknesses": "### Strength \n- The paper is well-written and the idea is well-motivated. \n- The proposed theory-driven regularization terms are reliable and interesting. To be honest, I enjoy reading this (kind of) paper. \n- The paper is technically sound. \n\n### Weakness\n- The theoretical analysis is limited to the single-layer GNN and it seems non-trivial to extend it into the multi-layer cases. \n- The core theorem is mainly from the existing work of generic domain adaptation. The real theoretical contributions of this paper seem to be the two lemmas about the discussions of Lipschitz on graphs. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "aSZGBw0SRF", "reviewer": "Reviewer_1Hvj", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper provides a framework for domain adaptation in graph neural networks. Algorithmically, the framework relies on addition of two regularizer terms to the loss function. The construction of the regularizer terms is motivated from theoretical results derived using the theory of optimal transport-based domain adaptation and stability of graph filters from the existing literature. Experiments on real world datasets illustrate the utility of the proposed framework. ", "summary_of_the_review": "I think this is a solid contribution in domain adaptation on graphs. The algorithms are well-motivated from theory and the experiments are illuminating and convincing. ", "strength_and_weaknesses": "Strengths:\nDomain adaptation is a very practically relevant problem. The proposed framework is theoretically motivated and provides a solid contribution in this regard. Experiments are well-designed and provide a convincing evidence for the approach. \n\nWeaknesses:\nI don't identify any major weaknesses. However, I couldn't verify the 'padding with isolated nodes' procedure. For $A_1$ in Lemma 1 (and other results where this procedure is mentioned), is this similar to extending the matrix diagonally with dummy nodes to make it of size $N_G \\times N_G$? If so, this does not appear to be an optimal procedure for domain adaptation. \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster