Title: Iteratively Refined Early Interaction Alignment for Subgraph Matching based Graph Retrieval
Venue: NeurIPS 2024
Accepted: yes
Keywords: Graph Neural Networks

Graph Retrieval
Abstract: Graph retrieval based on subgraph isomorphism has several real-world applications such as scene graph retrieval, molecular fingerprint detection and circuit design. Roy et al. [35] proposed IsoNet, a late interaction model for subgraph matching, which first computes the node and edge embeddings of each graph independently of paired graph  and then computes a trainable alignment map. Here, we present $\texttt{IsoNet++}$, an early interaction graph neural network (GNN), based on several technical innovations. First, we compute embeddings of all nodes by passing messages within and across the two input graphs, guided by an *injective alignment* between their nodes. Second, we update this alignment in a lazy fashion over multiple *rounds*. Within each round, we run a layerwise GNN from scratch, based on the current state of the alignment. After the completion of one round of GNN, we use the last-layer embeddings to update the alignments, and proceed to the next round. Third, $\texttt{IsoNet++}$ incorporates a novel notion of node-pair partner interaction. Traditional early interaction computes attention between a node and its potential partners in the other graph, the attention then controlling messages passed across graphs. We consider *node pairs* (not single nodes) as potential partners. Existence of an edge between the nodes in one graph and non-existence in the other provide vital signals for refining the alignment. Our experiments on several datasets show that the alignments get progressively refined with successive rounds,
resulting in significantly better retrieval performance than existing methods. We demonstrate that all three innovations contribute to the enhanced accuracy. Our code and datasets are publicly available at https://github.com/structlearning/isonetpp.

Iteratively Refined Early Interaction Alignment for Subgraph Matching based Graph Retrieval

Abstract

Graph retrieval based on subgraph isomorphism has several real-world applications such as scene graph retrieval, molecular fingerprint detection and circuit design. Roy et al. [35] proposed IsoNet, a late interaction model for subgraph matching, which first computes the node and edge embeddings of each graph independently of paired graph and then computes a trainable alignment map. Here, we present IsoNet++, an early interaction graph neural network (GNN), based on several technical innovations. First, we compute embeddings of all nodes by passing messages within and across the two input graphs, guided by an injective alignment between their nodes. Second, we update this alignment in a lazy fashion over multiple rounds . Within each round, we run a layerwise GNN from scratch, based on the current state of the alignment. After the completion of one round of GNN, we use the last-layer embeddings to update the alignments, and proceed to the next round. Third, IsoNet++ incorporates a novel notion of node-pair partner interaction. Traditional early interaction computes attention between a node and its potential partners in the other graph, the attention then controlling messages passed across graphs. In contrast, we consider node pairs (not single nodes) as potential partners. Existence of an edge between the nodes in one graph and non-existence in the other provide vital signals for refining the alignment. Our experiments on several datasets show that the alignments get progressively refined with successive rounds, resulting in significantly better retrieval performance than existing methods. We demonstrate that all three innovations contribute to the enhanced accuracy. Our code and datasets are publicly available at https://github.com/structlearning/isonetpp.

1 Introduction

In graph retrieval based on subgraph isomorphism, the goal is to identify a subset of graphs from a corpus, denoted { G c } , wherein each retrieved graph contains a subgraph isomorphic to a given query graph G q . Numerous real-life applications, e.g. , molecular fingerprint detection [6], scene graph retrieval [16], circuit design [29] and frequent subgraph mining [43], can be formulated using subgraph isomorphism. Akin to other retrieval systems, the key challenge is to efficiently score corpus graphs against queries.

Recent work on neural graph retrieval [1, 2, 11, 22, 23, 35, 31, 46] has shown significant promise. Among them, Lou et al. [23, Neuromatch] and Roy et al. [35, IsoNet] focus specifically on subgraph isomorphism. They employ graph neural networks (GNNs) to obtain embeddings of query and corpus graphs and compute the relevance score using a form of order embedding [39]. In addition, IsoNet also approximates an injective alignment between the query and corpus graphs. These two models operate in a late interaction paradigm, where the representations of the query and corpus graphs are

computed independent of each other. In contrast, GMN [22] is a powerful early interaction network for graph matching, where GNNs running on G q and G c interact with each other at every layer.

Conventional wisdom suggests that early interaction is more accurate (even if slower) than late interaction, but GMN was outperformed by IsoNet. This is because of the following reasons. (1) GMN does not explicitly infer any alignment between G q and G c . The graphs are encoded by two GNNs that interact with each other at every layer, mediated by attentions from each node in one graph on nodes in the other. These attentions are functions of node embeddings, so they change from layer to layer. While these attentions may be interpreted as approximate alignments, they induce at best non-injective mappings between nodes. (2) In principle, one wishes to propose a consistent alignment across all layers. However, GMN's attention based 'alignment' is updated in every layer. (3) GMN uses a standard GNN that is known to be an over-smoother [36, 40]. Due to this, the attention weights (which depend on the over-smoothed node representations) also suffer from oversmoothing. These limitations raise the possibility of a third approach based on early interaction networks, enabled with explicit alignment structures, that have the potential to outperform both GMN and IsoNet.

1.1 Our contributions

We present IsoNet++, an early interaction network for subgraph matching that maintains a chain of explicit, iteratively refined, injective, approximate alignments between the two graphs.

Early interaction GNNs with alignment refinement We design early interaction networks for scoring graph pairs, that ensure the node embeddings of one graph are influenced by both its paired graph and the alignment map between them. In contrast to existing works, we model alignments as an explicit 'data structure'. An alignment can be defined between either nodes or edges, thus leading to two variants of our model: IsoNet++ (Node) and IsoNet++ (Edge). Within IsoNet++, we maintain a sequence of such alignments and refine them using GNNs acting on the two graphs. These alignments mediate the interaction between the two GNNs. In our work, we realize the alignment as a doubly stochastic approximation to a permutation matrix, which is an injective mapping by design.

Eager or lazy alignment updates In our work, we view the updates to the alignment maps as a form of gradient-based updates in a specific quadratic assignment problem or asymmetric GromovWasserstein (GW) distance minimization [30, 41]. The general form of IsoNet++ allows updates that proceed lockstep with GNN layers ( eager layer-wise updates), but it also allows lazy updates. Specifically, IsoNet++ can perform T rounds of updates to the alignment, each round including K layers of GNN message passing. During each round, the alignment is held fixed across all propagation layers in GNN. At the end of each round, we update the alignment by feeding the node embeddings into a neural Gumbel-Sinkhorn soft permutation generator [10, 26, 37].

Node-pair partner interaction between graphs The existing remedies to counter oversmoothing [8, 33, 40] entail extra computation; but they may be expensive in an early-interaction setting. Existing early interaction models like [22] perform node partner interaction ; interactions are constrained to occur between a node and it's partner , the node in the paired graph aligned with it. Instead, we perform node-pair partner interaction; the interaction is expanded to include the node-pairs (or edges) in the paired graph that correspond to node-pairs containing the node. Consequently, the embedding of a node not only depends on nodes in the paired graph that align with it, but also captures signals from nodes in the paired graph that are aligned with its neighbors.

Experiments The design components of IsoNet++ and their implications are subtle - we report on extensive experiments that tease out their effects. Our experiments on real world datasets show that, IsoNet++ outperforms several state-of-the-art methods for graph retrieval by a substantial margin. Moreover, our results suggest that capturing information directly from node-pair partners can improve representation learning, as compared to taking information only from node partner.

2 Preliminaries

Notation Given graph G = ( V, E ) , we use nbr ( u ) to denote the neighbors of a node u ∈ V . We use u → v to indicate a message flow from node u to node v . Given a set of corpus graphs C = { G c } and a query graph G q , we denote y ( G c | G q ) as the binary relevance label of G c for G q . Motivated by several real life applications like substructure search in molecular graphs [12], object search in scene graphs [16], and text entailment [20], we consider subgraph isomorphism to significantly influence the relevance label, similar to previous works [23, 35]. Specifically, y ( G c | G q ) = 1 when G q is a

subgraph of G c , and 0 otherwise. We define C q + ⊆ C as the set of corpus graphs that are relevant to G q and set C q -= C \ C q + . Mildly overloading notation, we use P to indicate a 'hard' (0/1) permutation matrix or its 'soft' doubly-stochastic relaxation. B n denotes the set of all n × n doubly stochastic matrices, and Π n denotes the set of all n × n permutation matrices.

IsoNet [35] Given a graph G = ( V, E ) , IsoNet uses a GNN, which initializes node representations { h 0 ( u ) : u ∈ V } using node-local features. Then, messages are passed between neighboring nodes in K propagation layers . In the k th layer, a node u receives messages from its neighbors, aggregates them, and then combines the result with its state after the ( k -1) th layer:

h k ( u ) = comb θ ( h k -1 ( u ) , ∑ v ∈ nbr ( u ) { msg θ ( h k -1 ( u ) , h k -1 ( v )) } ) . (1) Here, msg θ ( · ) and comb θ ( · , · ) are suitable networks with parameters collectively called θ . Edges may also be featurized and influence the messages that are aggregated [24]. The node representations at the final propagation layer K can be collected into the matrix H = { h K ( u ) | u ∈ V } . Given a node u ∈ G q and a node u ′ ∈ G c , we denote the embeddings of u and u ′ after the propagation layer k as h ( q ) k ( u ) and h ( c ) k ( u ′ ) respectively. H ( q ) and H ( c ) denote the K th-layer node embeddings of G q and G c , collected into matrices. Note that, here the set of vectors H ( q ) and H ( c ) do not dependent on G c and G q . In the end, IsoNet compares these embeddings to compute the distance ∆( G c | G q ) , which is inversely related to ˆ y ( G c | G q ) .

̸

∆( G c | G q ) = ∑ u,i ReLU [ H ( c ) -PH ( q ) ][ u, i ] (2) Since subgraph isomorphism entails an asymmetric relevance, we have: ∆( G c | G q ) = ∆( G q | G c ) . IsoNet also proposed another design of ∆ , where it replaces the node embeddings with edge embeddings and node alignment matrix with edge alignment matrix in Eq. (2).

In an early interaction network, H ( q ) depends on G c and H ( c ) depends on G q for any given ( G q , G c ) pair. Formally, one should write H ( q | c ) and H ( c | q ) instead of H ( q ) and H ( c ) respectively for an early interaction network, but for simplicity, we will continue using H ( q ) and H ( c ) .

Our goal Given a set of corpus graphs C = { G c | c ∈ [ | C | ] } , our high-level goal is to build a graph retrieval model so that, given a query G q , it can return the corpus graphs { G c } which are relevant to G q . To that end, we seek to develop (1) a GNN-based early interaction model, and (2) an appropriate distance measure ∆( · | · ) , so that ∆( H ( c ) | H ( q ) ) is an accurate predictor of y ( G c | G q ) , at least to the extent that ∆( ·|· ) is effective for ranking candidate corpus graphs in response to a query graph.

3 Proposed early-interaction GNN with multi-round alignment refinement

In this section, we first write down the subgraph isomorphism task as an instance of the quadratic assignment problem (QAP) or the Gromov-Wasserstein (GW) distance optimization task. Then, we design IsoNet++, by building upon this formulation.

3.1 Subgraph isomorphism as Gromov-Wasserstein distance optimization

QAP or GW formulation with asymmetric cost We are given a graph pair G q and G c padded with appropriate number of nodes to ensure | V q | = | V c | = n (say). Let their adjacency matrices be A q , A c ∈ { 0 , 1 } n × n . Consider the family of hard permutation matrices P ∈ Π n where P [ u, u ′ ] = 1 indicates u ∈ V q is 'matched' to u ′ ∈ V c . Then, G q is a subgraph of G c , if for some permutation matrix P , the matrix A q is covered by PA c P ⊤ , i.e. , for each pair ( u, v ) , whenever we have A q [ u, v ] = 1 , we will also have PA c P ⊤ [ u, v ] = 1 . This condition can be written as A q ≤ PA c P ⊤ . We can regard a deficit in coverage as a cost or distance:

Here, [ · ] + = max {· , 0 } is the ReLU function, applied elementwise. The function cost( P ; A q , A c ) can be driven down to zero using a suitable choice of P iff G q is a subgraph of G c . This naturally suggests the relevance distance

Xu et al. [41] demonstrate that this QAP is a realization of the Gromov-Wassterstein distance minimization in a graph setting.

NodeAlignerRefinement

,

EdgeAlignerRefinement

EarlyInteractionGNN

= (comb , msg , Inter

)

(a) Pipeline of EinsMatch

GNN layers of

round

GNN layers of round

Alignment

Refined

alignment

Last round

)

(c) Node pair partner interaction in EinsMatch-Edge

(1) Pipeline of IsoNet++

(2) Node pair partner interaction in IsoNet++ (Node)

Updating P with projected gradient descent As shown in Benamou et al. [3], Peyré et al. [30], Xu et al. [41], one approach is to first relax P into a doubly stochastic matrix, which serves as a continuous approximation of the discrete permutation, and then update it using projected gradient descent (PGD). Here, the soft permutation P t -1 is updated to P t at time-step t by solving the following linear optimal transport (OT) problem, regularized with the entropy of { P [ u, v ] | u, v ∈ [ n ] } with a temperature τ .

Such an OT problem is solved using the iterative Sinkhorn-Knopp algorithm [10, 37, 26]. Similar to other combinatorial optimization problems on graphs, a QAP (4) does not capture the coverage cost in the presence of dense node or edge features, where two nodes or edges may exhibit graded degrees of similarity represented by continuous values. Furthermore, the binary values of the adjacency matrices result in inadequate gradient signals in ∇ P cost( · ) . Additionally, the computational bottleneck of solving a fresh OT problem in each PGD step introduces a significant overhead, especially given the large number of pairwise evaluations required in typical learning-to-rank setups.

3.2 Design of IsoNet++ (Node)

Building upon the insights from the above GW minimization (3) and the successive refinement step (6), we build IsoNet++ (Node), the first variant of our proposed early interaction model.

Node-pair partner interactions between graphs For simpler exposition, we begin by describing a synthetic scenario, where P is a hard node permutation matrix, which induces the alignment map as a bijection π : V q → V c , so that π ( a ) = b if P [ a, b ] = 1 . We first initialize layer k = 0 embeddings as h ( q ) 0 ( u ) = Init θ (feature( u )) using a neural network Init θ . (Throughout, h ( c ) k ( u ) are treated likewise.) Under the given alignment map π , a simple early interaction model would update the node embeddings as follows:

In the above expression, the update layer uses representation of the partner node u ′ ∈ V c during the message passing step, to compute h ( q ) k +1 ( u ) , the embedding of node u ∈ V q . Li et al. [22] use a similar update protocol, by approximating h ( c ) k ( π ( u )) = ∑ u ′ ∈ V c a ( k ) u ′ → u h ( c ) k ( u ′ ) , where a ( k ) u ′ → u is the k th layer attention from u ∈ V q to potential partner u ′ ∈ V c , with ∑ u ′ ∈ V c a ( k ) u ′ → u = 1 . Instead

<latexi sh

1_b

64="

MjUm8kD

pn/

yPREzI

5Cw

>A

B

X

c

V

LSg

F

3

r

Zu

q

o

G

7

Q

JH

O

9

0

2

K

+

Y

v

d

T

W

N

f

<latexi sh

1_b

64="/qdjg2SQNo

L

VM

npJ

DA

>

B+X

c

8

E

3

W

KO

U

0

PR

9

G

T

m7

I

C

z

Y

Z

kH

yur

v

f

F

w

5

<latexi sh

1_b

64="+

gyQC

uYWd/

P

Kf

>A

B

H

c

V

NS8

E

3

X

vw

J

ok

j

m

7

Z

LF

D

2

09

5

rG

O

M

T

UI

R

p

q

z

n

<latexi sh

1_b

64="d3zO

X

Y

Kvn

AI5yGLCWZN

/o

>

c

V

S8

EJ

Ur

q

9

k

j

Q

D

m0

R

T

2

pg

PH

u7

F

f

w

M

<latexi sh

1_b

64="qS2M+g

/3mdB

G

ZHP

u

I

>A

c

VDL

N

EOy

r

fUY9

o

K

F

8Rz

J

p

7

w5

k

0

X

W

j

n

C

v

Q

T

<latexi sh

1_b

64="

MjUm8kD

pn/

yPREzI

5Cw

>A

B

X

c

V

LSg

F

3

r

Zu

q

o

G

7

Q

JH

O

9

0

2

K

+

Y

v

d

T

W

N

f

<latexi sh

1_b

64="/qdjg2SQNo

L

VM

npJ

DA

>

B+X

c

8

E

3

W

KO

U

0

PR

9

G

T

m7

I

C

z

Y

Z

kH

yur

v

f

F

w

5

<latexi sh

1_b

64="+

gyQC

uYWd/

P

Kf

>A

B

H

c

V

NS8

E

3

X

vw

J

ok

j

m

7

Z

LF

D

2

09

5

rG

O

M

T

UI

R

p

q

z

n

<latexi sh

1_b

64="d3zO

X

Y

Kvn

AI5yGLCWZN

/o

>

c

V

S8

EJ

Ur

q

9

k

j

Q

D

m0

R

T

2

pg

PH

u7

F

f

w

M

<latexi sh

1_b

64="qS2M+g

/3mdB

G

ZHP

u

I

>A

c

VDL

N

EOy

r

fUY9

o

K

F

8Rz

J

p

7

w5

k

0

X

W

j

n

C

v

Q

T

<latexi sh

1_b

64="LW

mV

/Bk

0

jRGZDI2

n

v

A

>

c

NS8

E

3Ur

q

ou

J

7

X

Q

T

w9

f+KF

P

z

YH

p

C

Oy

d

g

M

5

<latexi sh

1_b

64="

MjUm8kD

pn/

yPREzI

5Cw

>A

B

X

c

V

LSg

F

3

r

Zu

q

o

G

7

Q

JH

O

9

0

2

K

+

Y

v

d

T

W

N

f

<latexi sh

1_b

64="/qdjg2SQNo

L

VM

npJ

DA

>

B+X

c

8

E

3

W

KO

U

0

PR

9

G

T

m7

I

C

z

Y

Z

kH

yur

v

f

F

w

5

<latexi sh

1_b

64="+

gyQC

uYWd/

P

Kf

>A

B

H

c

V

NS8

E

3

X

vw

J

ok

j

m

7

Z

LF

D

2

09

5

rG

O

M

T

UI

R

p

q

z

n

<latexi sh

1_b

64="d3zO

X

Y

Kvn

AI5yGLCWZN

/o

>

c

V

S8

EJ

Ur

q

9

k

j

Q

D

m0

R

T

2

pg

PH

u7

F

f

w

M

<latexi sh

1_b

64="qS2M+g

/3mdB

G

ZHP

u

I

>A

c

VDL

N

EOy

r

fUY9

o

K

F

8Rz

J

p

7

w5

k

0

X

W

j

n

C

v

Q

T

h

(

q

)

k

h

(

c

)

k

comb

θ

msg

θ

∑

(a) No Interaction

(IsoNet)

h

(

q

)

k

h

(

c

)

k

comb

θ

msg

θ

∑

(b) Node Pair Interaction

(GMN)

h

(

q

)

k

h

(

c

)

k

comb

θ

msg

θ

∑

inter

θ

Interaction (IsoNet++)

of regarding only nodes as potential partners, IsoNet++ will regard node pairs as partners. Given ( u, v ) ∈ E q , the partners ( π ( u ) , π ( v )) ∈ E c should then greatly influence the intensity of assimilation of h ( c ) k ( u ′ ) into h ( c ) k +1 ( u ) . The first key innovation in IsoNet++ is to replace (7) to recognize and implement this insight:

Embeddings h ( c ) k +1 ( u ′ ) for nodes u ′ ∈ V c are updated likewise in a symmetric manner. The network msg θ is provided embeddings from partners π ( u ) , π ( v ) of u, v ∈ V q -this allows h ( · ) k +1 ( u ) to capture information from all nodes in the paired graph, that match with the ( k +1) -hop neighbors of u . We schematically illustrate the interaction between the paired graphs in IsoNet, GMN and IsoNet++ in Figure 2.

Multi-round lazy refinement of node alignment In reality, we are not given any alignment map π . This motivates our second key innovation beyond prior models [1, 22, 23, 35], where we decouple GNN layer propagation from updates to P . To achieve this, IsoNet++ (Node) executes T rounds , each consisting of K layer propagations in both GNNs. At the end of each round t , we refine the earlier alignment P t -1 to the next estimate P t , which will be used in the next round. Henceforth, we will use the double subscript t, k instead of the single subscript k as in traditional GNNs. We denote the node embeddings at layer k and round t by h ( q ) t,k ( u ) , h ( c ) t,k ( u ′ ) ∈ R dim h for u ∈ V q and u ′ ∈ V c , which are (re-)initialized with node features h · t, 0 for each round t . We gather these into matrices

P no longer remains an oracular hard permutation matrix, but becomes a doubly stochastic matrix indexed by rounds, written as P t . At the end of round t , a differentiable aligner module takes H ( q ) t,K and H ( c ) t,K as inputs and outputs a doubly stochastic node alignment (relaxed permutation) matrix P t as follows:

In the above expression, GumbelSinkhorn( · ) performs iterative Sinkhorn normalization on the input matrix added with Gumbel noise [26]; LRL ϕ is a neural module consisting of two linear layers with a ReLU activation after the first layer. As we shall see next, P t is used to gate messages flowing across from one graph to the other during round t +1 , i.e., while computing H ( q ) t +1 , 1: K and H ( c ) t +1 , 1: K . The soft alignment P t is kept frozen for the duration of all layers in round t +1 . P t [ u, u ′ ] may be interpreted as the probability that u is assigned to u ′ , which naturally requires that P t should be

row-equivariant (column equivariant) to the shuffling of the node indices of G q ( G c ). As shown in Appendix D, the above design choice (11) ensures this property.

Updating node representation using early-interaction GNN Here, we describe the early interaction GNN for the query graph G q . The GNN on the corpus graph G c follows the exact same design and is deferred to Appendix E.1. In the initial round ( t = 1 ), since there is no prior alignment estimate P t =0 , we employ the traditional late interaction GNN (1) to compute all layers H ( q ) 1 , 1: K and H ( c ) 1 , 1: K separately. These embeddings are then used to estimate P t =1 using Eq. (11). For subsequent rounds ( t > 1 ), given embeddings H ( q ) t, 1: K , and the alignment estimate matrix P t , we run an early interaction GNN from scratch. We start with a fresh initialization of the node embeddings as before; i.e., h ( q ) t +1 , 0 ( u ) = Init θ (feature( u )) . For each subsequent propagation layer k +1 ( k ∈ [0 , K -1] ), we approximate (8) as follows. We read previous-round, same-layer embeddings h ( c ) t,k ( u ′ ) of nodes u ′ from the other graph G c , incorporate the alignment strength P t [ u, u ′ ] , and aggregate these to get an intermediate representation of u that is sensitive to P t and G c .

Here, inter θ is a neural network that computes interaction between the graph pairs; z ( q ) t +1 ,k ( u ) provides a soft alignment guided representation of [ h ( q ) k ( u ) , h ( c ) k ( π ( u ))] in Eq. (8), which can be relaxed as:

In the above expression, we explicitly feed z ( q ) t +1 ,k ( v ) , v ∈ nbr ( u ) in the msg θ network, capturing embeddings of nodes in the corpus G c aligned with the neighbors of node u ∈ V q in h ( q ) t +1 ,k +1 ( u ) . This allows the model to perform node-pair partner interaction. Instead, if we were to feed only h ( q ) t +1 ,k ( u ) into the msg θ network, then it would only perform node partner interaction. In this case, the computed embedding for u would be based solely on signals from nodes in the paired graph that directly correspond to u , therefore missing additional context from other neighbourhood nodes.

Distant supervision of alignment Finally, at the end of T rounds, we express the relevance distance ∆( G c | G q ) as a soft distance between the set H ( q ) T,K = [ h ( q ) T,K ( u ) | u ∈ V q ] and H ( c ) T,K = [ h ( c ) T,K ( u ′ ) | u ′ ∈ V c ] , measured as

Our focus is on graph retrieval applications. It is unrealistic to assume direct supervision from a gold alignment map P ∗ . Instead, training query instances are associated with pairwise preferences between two corpus graphs, in the form ⟨ G q , G c + , G c -⟩ , meaning that, ideally, we want ∆ θ,ϕ ( G c -| G q ) ≥ γ +∆ θ,ϕ ( G c + | G q ) , where γ > 0 is a margin hyperparameter. This suggests a minimization of the standard hinge loss as follows:

This loss is back-propagated to train model weights θ in comb θ , inter θ , msg θ and weights ϕ in the Gumbel-Sinkhorn network.

Multi-layer eager alignment variant Having set up the general multi-round framework of IsoNet++, we introduce a structurally simpler variant that updates P eagerly after every layer, eliminating the need to re-initialize node embeddings every time we update P . The eager variant retains the benefits of node-pair partner interactions, while ablating IsoNet++ toward GMN. Updating P via Sinkhorn iterations is expensive compared to a single GNN layer. In practice, we see a non-trivial tradeoff between computation cost, end task accuracy, and the quality of our injective alignments, depending on the value of K for eager updates, and the values ( T, K ) for lazy updates. Formally, P k is updated across layers as follows:

We update the GNN embeddings, layerwise, as follows:

Analysis of computational complexity Here, will compare the performance of IsoNet (Node) [35] with multi-layer IsoNet++ (Node) and multi-round IsoNet++ (Node) for graphs with | V | nodes. For multi-layer IsoNet++ (Node) and IsoNet (Node), we assume K propagation steps and for multi-round IsoNet++ (Node), T rounds, each with K propagation steps.

-IsoNet (Node): The total complexity is O ( | V | 2 + K | E | ) , computed as follows: (1) Initialization of layer embeddings at layer k = 0 takes O ( | V | ) time. (2) The node representation computation incurs a complexity of O ( | E | ) for each message passing step since it aggregates node embeddings across all neighbors. (3) The computation of P takes O ( | V | 2 ) time.

-Multi-layer eager IsoNet++ (Node): The total complexity is O ( K | V | 2 + K | E | + K | V | 2 ) = O ( K | V | 2 ) , computed as follows: (1) Initialization (layer k = 0 ) takes O ( | V | ) time. (2) The computation of intermediate embeddings z ( · ) (Eq. 18) involves the evaluation of the expression ∑ u ′ ∈ V c h ( · ) k ( u ′ ) P k [ u, u ′ ] and hence admits a complexity of O ( | V | ) for each node per layer. The total complexity for K steps and | V | nodes is thus O ( K | V | 2 ) . (3) Next, for each node in every layer, we compute h ( · ) k +1 (Eq. 19) which gathers messages z ( · ) from all its neighbors, contributing a total complexity of O ( K | E | ) . (4) Finally, we update P k for each layer which has a complexity of O ( K | V | 2 ) .

-Multi-round IsoNet++ (Node): Here, the key difference from the multi-layer version above is that the doubly stochastic matrix P t from round t is used to compute z and the K -step-GNN runs in each of the T rounds. This multiplies the complexity of steps 2 and 3 with T , raising it to O ( KT | V | 2 + KT | E | ) . Matrix P t is updated a total of T times, which changes the complexity of step 4 to O ( T | V | 2 ) . Hence, the total complexity is O ( KT | V | 2 + T | V | 2 + KT | E | ) = O ( KT | V | 2 ) .

Hence, the complexity of IsoNet is O ( | V | 2 + K | E | ) , multi-layer IsoNet++ is O ( K | V | 2 ) and multiround IsoNet++ is O ( KT | V | 2 ) . This increased complexity of the latter comes with the benefit of a significant performance boost, as our experiments suggest.

3.3 Extension of IsoNet++ (Node) to IsoNet++ (Edge)

We now extend IsoNet++ (Node) to IsoNet++ (Edge) which uses explicit edge alignment for interaction across GNN and relevance distance surrogate.

Multi-round refinement of edge alignment In IsoNet++ (Edge), we maintain a soft edge permutation matrix S which is frozen at S = S t -1 within each round t ∈ [ T ] and gets refined after every round t as S t -1 → S t . Similar to IsoNet++ (Node), within each round t , GNN runs from scratch: it propagates messages across layers k ∈ [ K ] and S t -1 assists it to capture cross-graph signals. Here, in addition to node embeddings h ( · ) t,k , we also use edge embeddings m ( q ) t,k ( e ) , m ( c ) t,k ( e ′ ) ∈ R dim m at each layer k and each round t , which capture the information about the subgraph k ≤ K hop away from the edges e and e ′ . Similar to Eq. (9), we define M ( q ) t,k = [ m ( q ) t,k ( e )] e ∈ E q , and M ( c ) t,k = [ m ( c ) t,k ( e ′ )] e ′ ∈ E c .

M ( · ) t, 0 are initialized using the features of the nodes connected by the edges, and possibly local edge features. Given the embeddings M ( q ) t,K and M ( c ) t,K computed at the end of round t , an edge aligner module ( EdgeAlignerRefinement ϕ ( · ) ) takes these embedding matrices as input and outputs a soft edge permutation matrix S t , similar to the update of P t in Eq. (11).

Here, M ( · ) t,K are appropriately padded to ensure that they have the same number of rows.

Edge alignment-induced early interaction GNN For t = 1 , we start with a late interaction model using vanilla GNN (1) and obtain S t =1 using Eq. (21). Having computed the edge embeddings m ( · ) t, 1: K ( · ) and node embeddings h ( · ) t, 1: K ( · ) upto round t , we compute S t and use it to build a fresh early interaction GNN for round t +1 . To this end, we adapt the GNN guided by P t in Eqs. (12)-

(13),to the GNN guided by S t . We overload the notations for neural modules and different embedding vectors from IsoNet++ (Node), whenever their roles are similar.

Starting with the same initialization as in IsoNet++ (Node), we perform the cross-graph interaction guided by the soft edge permutation matrix S t , similar to Eq. (12). Specifically, we use the embeddings of edges { e ′ = ( u ′ , v ′ ) } ∈ E c , computed at layer k at round t , which share soft alignments with an edge e = ( u, v ) ∈ E q , to compute z ( q ) t +1 ,k ( e ) and z ( q ) t +1 ,k ( e ′ ) as follows:

Finally, we update the node embeddings h ( · ) t +1 ,k +1 for propagation layer k +1 as

In this case, we perform the cross-graph interaction at the edge level rather than the node level. Hence, msg θ acquires cross-edge signals separately as z ( · ) t +1 ,k . Finally, we use h ( · ) t +1 ,k +1 and z ( · ) t +1 ,k +1 to update m ( · ) t +1 ,k +1 as follows:

Likewise, we develop m ( c ) t +1 ,k +1 for corpus graph G c . Note that m ( q ) t +1 ,k +1 (( u, v )) captures signals not only from the matched pair ( u ′ , v ′ ) , but also signals from the nodes in G c which share correspondences with the neighbor nodes of u and v . Finally, we pad zero vectors to [ m ( q ) T,K ( e )] e ∈ E q and [ m ( c ) T,K ( e ′ )] e ′ ∈ E c to build the matrices M ( q ) T,K and M ( c ) T,K with same number of rows, which are finally used to compute the relevance distance

4 Experiments

We report on a comprehensive evaluation of IsoNet++ on six real datasets and analyze the efficacy of the key novel design choices. In Appendix G, we provide results of additional experiments.

4.1 Experimental setup

Datasets We use six real world datasets in our experiments, viz. , AIDS, Mutag, PTC-FM (FM), PTC-FR (FR), PTC-MM (MM) and PTC-MR (MR), which were also used in [27, 35]. Appendix F provides the details about dataset generation and their statistics.

State-of-the-art baselines We compare our method against eleven state-of-the-art methods, viz. , (1) GraphSim [2] (2) GOTSim [11], (3) SimGNN [1], (4) EGSC [31], (5) H2MN [45], (6) Neuromatch [23], (7) GREED [32], (8) GEN [22], (9) GMN [22] (10) IsoNet (Node) [35], and (11) IsoNet (Edge) [35]. Among them, Neuromatch, GREED, IsoNet (Node) and IsoNet (Edge) apply asymmetric hinge distances between query and corpus embeddings for ∆( G c | G q ) , specifically catered towards subgraph matching, similar to our method in Eqs. (14) and (25). GMN and GEN use symmetric Euclidean distance between their (whole-) graph embeddings g ( q ) (for query) and g ( c ) (for corpus) as || g ( q ) -g ( c ) || in their paper [22], which is not suitable for subgraph matching and therefore, results in poor performance. Hence, we change it to ∆( G c | G q ) = [ g ( q ) -g ( c ) ] + . The other methods first compute the graph embeddings, then fuse them using a neural network and finally apply a nonlinear function on the fused embeddings to obtain the relevance score.

Training and evaluation protocol Given a fixed corpus set C , we split the query set Q into 60% training, 15% validation and 25% test set. We train all the models on the training set by minimizing a ranking loss (15). During the training of each model, we use five random seeds. Given a test query q ′ , we rank the corpus graphs C in the decreasing order of ∆ θ,ϕ ( G c | G q ′ ) computed using the trained model. We evaluate the quality of the ranking by measuring Average Precision (AP) and HITS@20, described in Appendix F. Finally, we report mean average precision (MAP) and mean HITS@20, across all the test queries. By default, we set the number of rounds T = 3 , the number of propagation layers in GNN K = 5 . In Appendix F, we discuss the baselines, hyperparameter setup and the evaluation metrics in more detail.

Metrics

→

Mean Average Precision (MAP)

HITS @ 20

AIDS

Mutag

FM

FR

MM

MR

AIDS

Mutag

FM

FR

MM

MR

GraphSim [2]

0.356

0.472

0.477

0.423

0.415

0.453

0.145

0.257

0.261

0.227

0.212

0.23

GOTSim [11]

0.324

0.272

0.355

0.373

0.323

0.317

0.112

0.088

0.147

0.166

0.119

0.116

SimGNN [1]

0.341

0.283

0.473

0.341

0.298

0.379

0.138

0.087

0.235

0.155

0.111

0.160

EGSC [31]

0.505

0.476

0.609

0.607

0.586

0.58

0.267

0.243

0.364

0.382

0.348

0.325

H2MN [45]

0.267

0.276

0.436

0.412

0.312

0.243

0.076

0.084

0.200

0.189

0.119

0.069

Neuromatch [23]

0.489

0.576

0.615

0.559

0.519

0.606

0.262

0.376

0.389

0.350

0.282

0.385

0.472

0.567

0.558

0.512

0.546

0.528

0.245

0.371

0.316

0.287

0.311

0.277

GEN [22]

0.557

0.605

0.661

0.575

0.539

0.631

0.321

0.429

0.448

0.368

0.292

0.391

GMN[22]

0.622

0.710

0.730

0.662

0.655

0.708

0.397

0.544

0.537

0.45

0.423

0.49

IsoNet (Node) [35]

0.659

0.697

0.729

0.68

0.708

0.738

0.438

0.509

0.525

0.475

0.493

0.532

IsoNet (Edge) [35]

0.690

0.706

0.783

0.722

0.753

0.774

0.479

0.529

0.613

0.538

0.571

0.601

IsoNet++ (Node)

0.825

0.851

0.888

0.855

0.838

0.874

0.672

0.732

0.797

0.737

0.702

0.755

IsoNet++ (Edge)

0.847

0.858

0.902

0.875

0.902

0.902

0.705

0.749

0.813

0.769

0.809

0.803

AIDS

Mutag

FM

FR

MM

MR

Node

{

Eager

0.756

0.81

0.859

0.802

0.827

0.841

Lazy

0.825

0.851

0.888

0.855

0.838

0.874

Edge

{

Eager

0.795

0.805

0.883

0.812

0.862

0.886

Lazy

0.847

0.858

0.902

0.875

0.902

0.902

Table 4: Lazy multi-round vs. eager multi-layer. First (Last) two rows report MAP for IsoNet++ (Node) (IsoNet++ (Edge)). Green shows the best method.

4.2 Results

Comparison with baselines First, we compare IsoNet++ (Node) and IsoNet++ (Edge) against all the baselines, across all datasets. In Table 3, we report the results. The key observations are as follows: (1) IsoNet++ (Node) and IsoNet++ (Edge) outperform all the baselines by significant margins across all datasets. IsoNet++ (Edge) consistently outperforms IsoNet++ (Node). This is because edge alignment allows us to compare the graph pairs more effectively than node alignment. A similar effect was seen for IsoNet (Edge) vs. IsoNet (Node) [35]. (2) Among all state-of-the-art competitors, IsoNet (Edge) performs the best followed by IsoNet (Node). Similar to us, they also use edge and node alignments respectively. However, IsoNet does not perform any interaction between the graph pairs and the alignment is computed once only during the computation of ∆( G c | G q ) . This results in modest performance compared to IsoNet++. (3) GMN uses 'attention' to estimate the alignment between graph pairs, which induces a non-injective mapping. Therefore, despite being an early interaction model, it is mostly outperformed by IsoNet, which uses injective alignments.

Lazy vs. eager updates In lazy multi-round updates, the alignment matrices remain unchanged across all propagation layers and are updated only after the GNN completes its K -layer message propagations. To evaluate its effectiveness, we compare it against the eager multilayer update (described at the end of Section 3.2), where the GNN executes its K -layer message propagations only once; the alignment map is updated across K layers; and, the alignment at k th layer is used to compute the embeddings at ( k +1) th layer. In Table 4, we compare the performance in terms MAP, which shows that lazy multi-round updates significantly outperform multi-layer updates.

Node partner vs. node-pair partner interaction To understand the benefits of node-pair partner interaction, we contrast IsoNet++ (Node) against another variant of our method, which performs node partner interaction rather than node pair partner interaction, similar to Eq. (7). For lazy multi-round updates, we compute the embeddings as follows:

AIDS

Mutag

FM

FR

MM

MR

Lazy

{

Node partner

0.776

0.829

0.851

0.819

0.844

0.84

IsoNet++ (Node)

0.825

0.851

0.888

0.855

0.838

0.874

Eager

{

Node partner

0.668

0.783

0.821

0.752

0.753

0.794

IsoNet++ (Node)

0.756

0.81

0.859

0.802

0.827

0.841

0

5

10

Tr

(

P

/latticetop

t

P

∗

)

0

.

04

0

.

08

0

.

12

0

.

16

0

.

20

p

(

Tr

(

P

/latticetop

t

P

∗

))

→

AIDS

Multi-Round

t

= 1

t

= 2

t

= 3

0

5

10

Tr

(

P

/latticetop

k P

∗

)

0

.

04

0

.

08

0

.

12

0

.

16

0

.

20

p

(

Tr

(

P

/latticetop

k P

∗

))

→

AIDS

Multi-Layer

k

= 1

k

= 2

k

= 3

k

= 4

k

= 5

0

5

10

Tr

(

S

/latticetop

t S

∗

)

0

.

04

0

.

08

0

.

12

p

(

Tr

(

S

/latticetop

t S

∗

))

→

AIDS

Multi-Round

t

= 1

t

= 2

t

= 3

0

5

10

Tr

(

S

/latticetop

k S

∗

)

0

.

04

0

.

08

0

.

12

p

(

Tr

(

S

/latticetop

k S

∗

))

→

AIDS

Multi-Layer

k

= 1

k

= 2

k

= 3

k

= 4

k

= 5

→

→

→

→

Table 5 summarizes the results, which shows that IsoNet++ (Node) (node partner pair) performs significantly better than Node partner for both multi-round lazy updates (top-two rows) and multi-layer eager updates (bottom tow rows).

Quality of injective alignments Next we compare between multi-round and multi-layer update strategies in terms of their ability to refine the alignment matrices, as the number of updates of these matrices increases. For multi-round (layer) updates, we instrument the alignments P t and S t ( P k and S k ) for different rounds t ∈ [ T ] (layers k ∈ [ K ] ). Specifically, we look into the distribution of the similarity between the learned alignments P t , S t and the correct alignments P ∗ , S ∗ (using combinatorial routine), measured using the inner products Tr ( P ⊤ t P ∗ ) and Tr ( S ⊤ t S ∗ ) for different t . Similarly, we compute Tr ( P ⊤ k P ∗ ) and Tr ( S ⊤ k S ∗ ) for different k ∈ [ K ] . Figure 6 summarizes the results, which shows that (1) as t or k increases, the learned alignments become closer to the gold alignments; (2) multi-round updates refine the alignments approximately twice as faster than the multi-layer variant. The distribution of Tr ( P ⊤ t P ∗ ) at t = 1 in multi-round strategy is almost always close to Tr ( P ⊤ k P ∗ ) for k = 2 . Note that, our aligner networks learn to refine the P t and S t through end-to-end training, without using any form of supervision from true alignments or the gradient computed in Eq. (6).

Accuracy-inference time trade-off Here, we analyze the accuracy and inference time trade-off. We vary T and K for IsoNet++'s lazy multi-round variant, and vary K for IsoNet++'s eager multi-layer variant and for GMN. Figure 7 summarizes the results. Notably, the eager multi-layer variant achieves the highest accuracy for K = 8 on the AIDS dataset, despite the known issue of oversmoothing in GNNs for large K . This unexpected result may be due to our message passing components, which involve terms

50

80

110

140

Inference time (in ms)

0

.

6

0

.

8

1

.

0

MAP

Lazy

Eager

GMN

(a)

50

80

110

140

Inference time (in ms)

0

.

6

0

.

8

1

.

0

MAP

Lazy

Eager

GMN

(b) Edge, AIDS

like ∑ u ′ P [ u, u ′ ] h ( u ′ ) , effectively acting as a convolution between alignment scores and embedding vectors. This likely enables P to function as a filter, countering the oversmoothing effect.

5 Conclusion

We introduce IsoNet++ as an early-interaction network for estimating subgraph isomorphism. IsoNet++ learns to identify explicit alignments between query and corpus graphs despite having access to only pairwise preferences and not explicit alignments during training. We design a graph neural network (GNN) that uses an alignment estimate to propagate messages, then uses the GNN's output representations to refine the alignment. Experiments across several datasets confirm that alignment refinement is achieved over several rounds. Design choices such as using node-pair partner interaction (instead of node partner) and lazy updates (over eager) boost the performance of our architecture, making it the state-of-the-art in subgraph isomorphism based subgraph retrieval. We also demonstrate the accuracy v/s inference time trade offs for IsoNet++, which show how different knobs can be tuned to utilize our models under regimes with varied time constraints.

This study can be extended to graph retrieval problems which use different graph similarity measures, such as maximum common subgraph or graph edit distance. Extracting information from node-pairs is effective and can be widely used to improve GNNs working on multiple graphs at once.

References

Y. Bai, H. Ding, S. Bian, T. Chen, Y. Sun, and W. Wang. Simgnn: A neural network approach to fast graph similarity computation. In Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining , pages 384-392, 2019.

Y. Bai, H. Ding, K. Gu, Y. Sun, and W. Wang. Learning-based efficient graph similarity computation via multi-scale convolutional set matching. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pages 3219-3226, 2020.

J.-D. Benamou, G. Carlier, M. Cuturi, L. Nenna, and G. Peyré. Iterative bregman projections for regularized transportation problems. SIAM Journal on Scientific Computing , 37(2):A1111A1138, 2015.

Q. Berthet, M. Blondel, O. Teboul, M. Cuturi, J.-P. Vert, and F. Bach. Learning with differentiable pertubed optimizers. Advances in neural information processing systems , 33:9508-9519, 2020.

L. Biewald. Experiment tracking with weights and biases, 2020. URL https://www.wandb. com/ . Software available from wandb.com.

A. Cereto-Massagué, M. J. Ojeda, C. Valls, M. Mulero, S. Garcia-Vallvé, and G. Pujadas. Molecular fingerprint similarity search in virtual screening. Methods , 71:58-63, 2015.

D. Chen, L. O'Bray, and K. Borgwardt. Structure-aware transformer for graph representation learning. ICML , 2022.

E. Cohen-Karlik, A. B. David, and A. Globerson. Regularizing towards permutation invariance in recurrent models. In NeurIPS . Curran Associates Inc., 2020. ISBN 9781713829546. URL https://arxiv.org/abs/2010.13055 .

L. P. Cordella, P. Foggia, C. Sansone, and M. Vento. A (sub) graph isomorphism algorithm for matching large graphs. IEEE transactions on pattern analysis and machine intelligence , 26(10): 1367-1372, 2004.

M. Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems , 26:2292-2300, 2013.

K. D. Doan, S. Manchanda, S. Mahapatra, and C. K. Reddy. Interpretable graph similarity computation via differentiable optimal alignment of node embeddings. pages 665-674, 2021.

H.-C. Ehrlich and M. Rarey. Systematic benchmark of substructure search in molecular graphsfrom ullmann to vf2. Journal of Cheminformatics , 4:1-17, 2012.

X. Gao, B. Xiao, D. Tao, and X. Li. A survey of graph edit distance. Pattern Analysis and applications , 13(1):113-129, 2010.

J. Gilmer, S. S. Schoenholz, P. F. Riley, O. Vinyals, and G. E. Dahl. Neural message passing for quantum chemistry. In International conference on machine learning , pages 1263-1272. PMLR, 2017.

A. Hagberg, P. Swart, and D. S Chult. Exploring network structure, dynamics, and function using networkx. Technical report, Los Alamos National Lab.(LANL), Los Alamos, NM (United States), 2008.

J. Johnson, R. Krishna, M. Stark, L.-J. Li, D. Shamma, M. Bernstein, and L. Fei-Fei. Image retrieval using scene graphs. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3668-3678, 2015.

N. Karalias and A. Loukas. Erdos goes neural: an unsupervised learning framework for combinatorial optimization on graphs. Advances in Neural Information Processing Systems , 33: 6659-6672, 2020.

T. N. Kipf and M. Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 , 2016.

J. Kotary, F. Fioretto, P. Van Hentenryck, and B. Wilder. End-to-end constrained optimization learning: A survey. arXiv preprint arXiv:2103.16378 , 2021.

A. Lai and J. Hockenmaier. Learning to predict denotational probabilities for modeling entailment. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers , pages 721-730, 2017. URL https://www.aclweb.org/anthology/E17-1068.pdf .

Y. Li, D. Tarlow, M. Brockschmidt, and R. Zemel. Gated graph sequence neural networks. arXiv preprint arXiv:1511.05493 , 2015.

Y. Li, C. Gu, T. Dullien, O. Vinyals, and P. Kohli. Graph matching networks for learning the similarity of graph structured objects. In International conference on machine learning , pages 3835-3845. PMLR, 2019. URL https://arxiv.org/abs/1904.12787 .

Z. Lou, J. You, C. Wen, A. Canedo, J. Leskovec, et al. Neural subgraph matching. arXiv preprint arXiv:2007.03092 , 2020.

D. Marcheggiani and I. Titov. Encoding sentences with graph convolutional networks for semantic role labeling. In M. Palmer, R. Hwa, and S. Riedel, editors, Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing , pages 1506-1515, Copenhagen, Denmark, Sept. 2017. Association for Computational Linguistics. doi: 10.18653/ v1/D17-1159. URL https://aclanthology.org/D17-1159 .

B. McFee and G. R. G. Lanckriet. Partial order embedding with multiple kernels. In International Conference on Machine Learning , 2009. URL https://api.semanticscholar.org/ CorpusID:699292 .

G. Mena, D. Belanger, S. Linderman, and J. Snoek. Learning latent permutations with gumbelsinkhorn networks. arXiv preprint arXiv:1802.08665 , 2018. URL https://arxiv.org/pdf/ 1802.08665.pdf .

C. Morris, N. M. Kriege, F. Bause, K. Kersting, P. Mutzel, and M. Neumann. Tudataset: A collection of benchmark datasets for learning with graphs. In ICML 2020 Workshop on Graph Representation Learning and Beyond (GRL+ 2020) , 2020. URL www.graphlearning.io .

R. Myers, R. Wison, and E. R. Hancock. Bayesian graph edit distance. IEEE Transactions on Pattern Analysis and Machine Intelligence , 22(6):628-635, 2000.

M. Ohlrich, C. Ebeling, E. Ginting, and L. Sather. Subgemini: Identifying subcircuits using a fast subgraph isomorphism algorithm. In Proceedings of the 30th International Design Automation Conference , pages 31-37, 1993.

G. Peyré, M. Cuturi, and J. Solomon. Gromov-wasserstein averaging of kernel and distance matrices. In International conference on machine learning , pages 2664-2672. PMLR, 2016.

C. Qin, H. Zhao, L. Wang, H. Wang, Y. Zhang, and Y. Fu. Slow learning and fast inference: Efficient graph similarity computation via knowledge distillation. In Thirty-Fifth Conference on Neural Information Processing Systems , 2021.

R. Ranjan, S. Grover, S. Medya, V. Chakaravarthy, Y. Sabharwal, and S. Ranu. Greed: A neural framework for learning graph distance functions. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, November 29-Decemer 1, 2022 , 2022.

I. Roy, A. De, and S. Chakrabarti. Adversarial permutation guided node representations for link prediction. In AAAI Conference , 2021. URL https://arxiv.org/abs/2012.08974 .

I. Roy, S. Chakrabarti, and A. De. Maximum common subgraph guided graph retrieval: Late and early interaction networks. In NeurIPS , 2022. URL https://openreview.net/forum? id=COAcbu3_k4U .

I. Roy, V. S. Velugoti, S. Chakrabarti, and A. De. Interpretable neural subgraph matching for graph retrieval. In AAAI Conference , 2022. URL https://indradyumna.github.io/pdfs/ IsoNet_main.pdf .

T. K. Rusch, M. M. Bronstein, and S. Mishra. A survey on oversmoothing in graph neural networks. Preprint, 2023. URL https://arxiv.org/abs/2303.10993 .

R. Sinkhorn and P. Knopp. Concerning nonnegative matrices and doubly stochastic matrices. Pacific Journal of Mathematics , 21(2):343-348, 1967.

P. Veliˇ ckovi´ c, G. Cucurull, A. Casanova, A. Romero, P. Lio, and Y. Bengio. Graph attention networks. arXiv preprint arXiv:1710.10903 , 2017.

I. Vendrov, R. Kiros, S. Fidler, and R. Urtasun. Order-embeddings of images and language. arXiv preprint arXiv:1511.06361 , 2015. URL https://arxiv.org/pdf/1511.06361 .

F. Wenkel, Y. Min, M. Hirn, M. Perlmutter, and G. Wolf. Overcoming oversmoothness in graph convolutional networks via hybrid scattering networks, 2022. URL https://arxiv.org/ abs/2201.08932 .

H. Xu, D. Luo, H. Zha, and L. C. Duke. Gromov-wasserstein learning for graph matching and node embedding. In International conference on machine learning , pages 6932-6941. PMLR, 2019.

K. Xu, W. Hu, J. Leskovec, and S. Jegelka. How powerful are graph neural networks? arXiv preprint arXiv:1810.00826 , 2018.

X. Yan, P. S. Yu, and J. Han. Graph indexing: A frequent structure-based approach. In Proceedings of the 2004 ACM SIGMOD international conference on Management of data , pages 335-346, 2004.

Z. Zeng, A. K. Tung, J. Wang, J. Feng, and L. Zhou. Comparing stars: On approximating graph edit distance. Proceedings of the VLDB Endowment , 2(1):25-36, 2009.

Z. Zhang, J. Bu, M. Ester, Z. Li, C. Yao, Z. Yu, and C. Wang. H2mn: Graph similarity learning with hierarchical hypergraph matching networks. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining , pages 2274-2284, 2021.

W. Zhuo and G. Tan. Efficient graph similarity computation with alignment regularization. Advances in Neural Information Processing Systems , 35:30181-30193, 2022.

Iteratively Refined Early Interaction Alignment for Subgraph Matching based Graph Retrieval (Appendix)

A Limitations

We find two limitations of our method each of which could form the basis of detailed future studies.

Retrieval systems greatly benefit from the similarity function being hashable. This can improve the inference time multi-fold while losing very little, if at all any, performance, making the approach ready for production environments working under tight time constraints. The design of a hash function for an early interaction network like ours is unknown and seemingly difficult. In fact, such a hashing procedure is not known even for predecessors like IsoNet (Edge) or GMN, and this is an exciting future direction.

Our approach does not explicitly differentiate between nodes or edges that may belong to different classes. This can be counterproductive when there exist constraints that prevent the alignment of two nodes or edges with different labels. While the network is designed to process node and edge features, it might not be enough to rule out alignments that violate the said constraint. Such constraints could also exist for node-pairs, such as in knowledge graphs with hierarchical relationships between entity types, and are not taken into account by our model. Extending our work to handle such restrictions is an interesting problem to consider.

B Related work

In this section, we discuss different streams of work that are related to and have influenced the study.

B.1 Graph Representation Learning

Graph neural networks (GNN) [14, 22, 21, 18, 42, 38] have emerged as a widely applicable approach for graph representation learning. A graph neural network computes the embedding of a node by aggregating the representations of its neighbors across K steps of message passing, effectively combining information from K -hop neighbors. GNNs were first used for graph similarity computation by Li et al. [22], who enriched the architecture with attention to predict isomorphism between two graphs. Attention acts as a mechanism to transfer information from the representation of one graph to that of the other, thus boosting the performance of the approach. Chen et al. [7] enriched the representation of graphs by capturing the subgraph around a node effectively through a structure aware transformer architecture.

B.2 Differentiable combinatorial solvers

We utilize a differentiable gadget to compute an injective alignment, which is a doubly stochastic matrix. The differentiability is crucial to the training procedure as it enables us to backpropagate through the alignments. The GumbelSinkhorn operator, which performs alternating normalizations across rows and columns, was first proposed by Sinkhorn and Knopp [37] and later used for the Optimal Transport problem by Cuturi [10]. Other methods to achieve differentiability include adding random noise to the inputs to discrete solvers [4] and designing probabilistic loss functions [17]. A compilation of such approaches towards constrained optimization on graphs through neural techniques is presented in [19].

B.3 Graph Similarity Computation and Retrieval

Several different underlying measures have been proposed for graph similarity computation, including full graph isomorphism [22], subgraph isomorphism [23, 35], graph edit distance (GED) [2, 11, 13, 28, 44] and maximum common subgraph (MCS) [2, 11, 34]. Bai et al. [2] proposed GraphSim towards the GED and MCS problems, using convolutional neural network based scoring on top of graph similarity matrices. GOTSim [11] explicitly computes the alignment between the two graphs by studying the optimal transformation cost. GraphSim [2] utilizes both graph-level and node-level signals to compute a graph similarity score. NeuroMatch [23] evaluates, for each node pair across the two graphs, if the neighborhood of one node is contained in the neighborhood of another using order embeddings [25]. GREED [32] proposed a Siamese graph isomorphism network, a late interaction

model to tackle the GED problem and provided supporting theoretical guarantees. Zhang et al. [45] propose an early interaction model, using hypergraphs to learn higher order node similarity. Each hypergraph convolution contains a subgraph matching module to learn cross graph similarity. Qin et al. [31] trained a slower attention-based network on multi-level features from a GNN and distilled its knowledge into a faster student model. Roy et al. [35] used the GumbelSinkhorn operator as a differentiable gadget to compute alignments in a backpropagation-friendly fashion and also demonstrated the utility of computing alignments for edges instead of nodes.

C Broader Impact

This work can be directly applied to numerous practical applications, such as drug discovery and circuit design, which are enormously beneficial for the society and continue to garner interest from researchers and practitioners worldwide. The ideas introduced in this paper have benefitted from and can benefit the information retrieval community as well, beyond the domain of graphs. However, malicious parties could use this technology for deceitful purposes, such as identifying and targeting specific social circles on online social networks (which can be represented as graphs). Such pros and cons are characteristic of every scientific study and the authors consider the positives to far outweigh the negatives.

D Network architecture of different components of IsoNet++

IsoNet++ models consist of three components - an encoder, a message-passing network and a node/edge aligner. We provide details about each of these components below. For convenience, we represent a linear layer with input dimension a and output dimension b as Linear( a, b ) and a linear-ReLU-linear network with Linear( a, b ) , Linear( b, c ) layers with ReLU activation in the middle as LRL( a, b, c ) .

D.1 Encoder

The encoder transforms input node/edge features before they are fed into the message-passing network. For models centred around node alignment like IsoNet++ (Node), the encoder refers to Init θ and is implemented as a Linear(1 , 10) layer. The edge vectors are not encoded and passed as-is down to the message-passing network. For edge-based models like IsoNet++ (Edge), the encoder refers to both Init θ, node and Init θ, edge, which are implemented as Linear(1 , 10) and Linear(1 , 20) layers respectively.

D.2 GNN

Within the message-passing framework, we use node embeddings of size dim h = 10 and edge embeddings of size dim m = 20 . We specify each component of the GNN below.

inter θ combines the representation of the current node/edge ( h · ) with that from the other graph, which are together fed to the network by concatenation. For node-based and edgebased models, it is implemented as LRL(20 , 20 , 10) and LRL(40 , 40 , 20) networks respectively. In particular, we ensure that the input dimension is twice the size of the output dimension, which in turn equals the intermediate embedding dimension dim( z ) .

msg θ is used to compute messages by combining intermediate embeddings z · of nodes across an edge with the representation of that edge. For node-based models, the edge vector is a fixed vector of size 1 while the intermediate node embeddings z · are vectors of dimension 10 , resulting in the network being a Linear(21 , 20) layer. For edge-based models, the edge embedding is the m vector of size 20 which requires msg θ to be a Linear(40 , 20) layer. Note that the message-passing network is applied twice, once to the ordered pair ( u, v ) and then to ( v, u ) and the outputs thus obtained are added up. This is to ensure node order invariance for undirected edges by design.

comb θ combines the representation of a node z · with aggregated messages received by it from all its neighbors. It is modelled as a GRU where the node representation (of size 10 ) is the initial hidden state and the aggregated message vector (of size 20 ) is the only element of an input sequence which updates the hidden state to give us the final node embedding h · .

D.3 Node aligner

The node aligner takes as input two sets of node vectors H ( q ) ∈ R n × 10 and H ( c ) ∈ R n × 10 representing G q and G c respectively. n refers to the number of nodes in the corpus graph (the query graph is padded to meet this node count). We use LRL ϕ as a LRL(10 , 16 , 16) network (refer Eq. 11).

D.4 Edge aligner

The design of the edge aligner is similar to the node aligner described above in Section D.3, except that its inputs are sets of edge vectors M ( q ) ∈ R e × 20 and M ( c ) ∈ R e × 20 . e refers to the number of edges in the corpus graph (the query graph is padded to meet this edge count). We use LRL ϕ as a LRL(20 , 16 , 16) network (refer Eq. 21).

D.5 GumbelSinkhorn operator

The GumbelSinkhorn operator consists of the following operations -

The matrix D out obtained after this set of operations will be a doubly-stochastic matrix. The input D in in our case is the matrix containing the dot product of the node/edge embeddings of the query and corpus graphs respectively. τ represents the temperature and is fixed to 0 . 1 in all our experiments.

Theorem Equation 11 results in a permutation matrix that is row-equivariant (column-) to the shuffling of nodes in G q ( G c ).

Proof To prove the equivariance of Eq. 11, we need to show that given a shuffling (permutation) of query nodes Z ∈ Π n which modifies the node embedding matrix to Z ˙ H ( q ) t,K , the resulting output of said equation would change to Z P t . Below, we consider any matrices with Z in the suffix as being an intermediate expression in the computation of NodeAlignerRefinement ϕ ( Z H ( q ) t,K , H ( c ) t,K ) .

It is easy to observe that the operators LRL ϕ (a linear-ReLU-linear network applied to a matrix), RowNorm , ColumnNorm and element-wise exponentiation ( exp ), division are all permutationequivariant since a shuffling of the vectors fed into these will trivially result in the output vectors getting shuffled in the same order. Thus, we get the following sequence of operations

D 0 ,Z equals exp( D in, Z /τ ) , which according to above equation would lead to D 0 ,Z = Z D 0 . We can then inductively show using Eq. 27 and the equivariance of row/column normalization, assuming the following holds till t , that

The above equivariance would also hold in the limit, resulting in the doubly stochastic matrix D out ,Z = Z D out, which concludes the proof. ■

A similar proof can be followed to show column equivariance for a shuffling in the corpus nodes.

E Variants of our models and GMN, used in the experiments

E.1 Multi-round refinement of IsoNet++ (Node) for the corpus graph

Initialize:

Update the GNN embeddings as follows:

E.2 Eager update for IsoNet++ (Edge)

Initialize:

The edge alignment is updated across layers. S 0 is set to a matrix of zeros. For k > 0 , the following equation is used:

We update the GNN node and edge embeddings as follows:

E.3 Node partner (with additional MLP) variant of IsoNet++ (Node)

Here, we update node embeddings as follows:

Here, z ( q ) t +1 ,k ( u ) is computed as Eq. (12), where inter θ is an MLP network. In contrast to Eq. (13), here, z ( q ) t +1 ,k ( u ) , z ( q ) t +1 ,k ( v ) are not fed into the message passing layer. Hence, in the message passing layer, we do not capture the signals from the partners of u and v in G c . Only signals from partners of u are captured through z ( q ) t +1 ,k ( u ) in the first argument.

E.4 Node pair partner ( msg only) variant of IsoNet++ (Node)

We change the GNN update equation as follows:

Node pair partner interaction takes place because, we feed z from Eq. (12) into the message passing layer. However, we use h in the first argument, instead of z .

F Additional details about experimental setup

F.1 Datasets

We use six datasets from the TUDatasets collection [27] for benchmarking our methods with respect to existing baselines. Lou et al. [23] devised a method to sample query and corpus graphs from the graphs present in these datasets to create their training data. We adopt it for the task of subgraph matching. In particular, we choose a node u ∈ G as the center of a Breadth First Search (BFS) and run the algorithm till | V | nodes are traversed, where the range of | V | is listed in Table 8 (refer to the Min and Max columns for | V q | and | V c ). This process is independently performed for the query and corpus splits (with different ranges for graph size) to obtain 300 query graphs and 800 corpus graphs. The set of query graphs is split into train, validation and test splits of 180 (60%) , 45 (15%) and 75 (25%) graphs respectively. Ground truth labels are computed for each query-corpus graph pair using the VF2 algorithm [9, 15, 23] implemented in the Networkx library. Various statistics about the datasets are listed in Table 8. pairs( y ) denotes the number of pairs in the dataset with gold label y , where y ∈ { 0 , 1 } .

F.2 Baselines

GraphSim, GOTSim, SimGNN, Neuromatch, GEN, GMN, IsoNet (Node), IsoNet (Edge) : We utilized the code from official implementation of [35] 1 . Some for loops were vectorized to improve the running time of GMN.

EGSC : The official implementation 2 is refactored and integrated into our code.

H2MN : We use the official code from 3 .

GREED : We use the official code from 4 . The model is adapted from the graph edit distance (GED) task to the subgraph isomorphism task, using a hinge scoring layer.

The number of parameters involved in all models (our methods and baselines) are reported in Table 9.

F.3 Calculation of Metrics: Mean Average Precision (MAP), HITS@K, Precision@K and Mean Reciprocal Rank (MRR)

Given a ranked list of corpus graphs C = { G c } for a test query G q , sorted in the decreasing order of ∆ θ,ϕ ( G c | G q ) , let us assume that the c th + relevant graph is placed at position pos ( c + ) ∈ { 1 , ..., | C |} in the ranked list. Then Average Precision (AP) is computed as:

Mean average precision is defined as ∑ q ∈ Q AP ( q ) / | Q | .

Precision@ K ( q ) = 1 K # relevant graphs corresponding to G q till rank K . Finally we report the mean of Precision@ K ( q ) across queries.

Reciprocal rank or RR ( q ) is the inverse of the rank of the topmost relevant corpus graph corresponding to G q in the ranked list. Mean reciprocal rank (MRR) is average of RR ( q ) across queries.

HITS@ K for a query G q is defined as the fraction of positively labeled corpus graphs that appear before the K th negatively labeled corpus graph. Finally, we report the average of HITS@ K across queries.

Note that HITS@K is a significantly aggressive metric compared to Precision@K and MRR, as can be seen in Tables 12 and 13.

F.4 Details about hyperparameters

All models were trained using early stopping with MAP score on the validation split as a stopping criterion. For early stopping, we used a patience of 50 with a tolerance of 10 -4 . We used the Adam optimizer with the learning rate as 10 -3 and the weight decay parameter as 5 · 10 -4 . We set batch size to 128 and maximum number of epochs to 1000 .

Seed Selection and Reproducibility Five integer seeds were chosen uniformly at random from the range [0 , 10 4 ] resulting in the set { 1704 , 4929 , 7366 , 7474 , 7762 } . IsoNet++ (Node), GMN and IsoNet (Edge) were trained on each of these 5 seeds for all 6 datasets. Note that these seeds do not control the training-dev-test splits but only control the initialization. Since the overall problem is non-convex, in principle, one should choose the best initial conditions leading to local minima. Hence, for all models, we choose the best seed, based on validation MAP score, is shown in Table 10.

IsoNet++ (Edge) and all ablations on top of IsoNet++ (Node) were trained using the best seeds for IsoNet++ (Node) (as in Tables 4, 5 and 16). Ablations of GMN were trained with the best GMN seeds.

For baselines excluding IsoNet (Edge), models were trained on all 5 seeds for 10 epochs and the MAP scores on the validation split were considered. Full training with early stopping was resumed only for

the best seed per dataset. This approach was adopted to reduce the computational requirements for benchmarking.

Margin Selection For GraphSim, GOTSim, SimGNN, Neuromatch, GEN, GMN and IsoNet (Edge), we use the margins determined by Roy et al. [35] for each dataset. For IsoNet (Node), the margins prescribed for IsoNet (Edge) were used for standardization. For IsoNet++ (Node), IsoNet++ (Edge) and ablations, a fixed margin of 0 . 5 is used.

Procedure for baselines EGSC, GREED, H2MN : They are trained on five seeds with a margin of 0.5 for 10 epochs and the best seed is chosen using the validation MAP score at this point. This seed is also used to train a model with a margin of 0.1 for 10 epochs. The better of these models, again using MAP score on the validation split, is identified and retrained till completion using early stopping.

F.5 Software and Hardware

All experiments were run with Python 3 . 10 . 13 and PyTorch 2 . 1 . 2 . IsoNet++ (Node), IsoNet++ (Edge), GMN, IsoNet (Edge) and ablations on top of these were trained on Nvidia RTX A6000 (48 GB) GPUs while other baselines like GraphSim, GOTSim etc. were trained on Nvidia A100 (80 GB) GPUs.

As an estimate of training time, we typically spawn 3 training runs of IsoNet++ (Node) or IsoNet++ (Edge) on one Nvidia RTX A6000 GPU, each of which takes 300 epochs to conclude on average, with an average of 6-12 minutes per epoch. This amounts to 2 days of training. Overloading the GPUs by spawning 6 training runs per GPU increases the training time marginally to 2 . 5 days.

Additionally, we use wandb [5] to manage and monitor the experiments.

F.6 License

GEN, GMN, GOTSim, GREED and EGSC are available under the MIT license, while SimGNN is public under the GNU license. The licenses for GraphSim, H2MN, IsoNet (Node), IsoNet (Edge), Neuromatch could not be identified. The authors were unable to identify the license of the TUDatasets repository [27], which was used to compile the 6 datasets used in this paper.

G Additional experiments

G.1 Comparison against baselines

In Tables 12 and 13, we report the Mean Average Precision (MAP), HITS@20, MRR and Precision@20 scores for several baselines as well as the four approaches discussed in our paper - multi-layer and multi-round variants of IsoNet++ (Node) and IsoNet++ (Edge). Multi-round IsoNet++ (Edge) outperforms all other models with respect to all metrics, closely followed by multi-round IsoNet++ (Node) and multi-layer IsoNet++ (Edge) respectively. Among the baselines, IsoNet (Edge) is the best-performing model, closely followed by IsoNet (Node) and GMN.

G.2 HITS@20, MRR and Precision@20 for multi-round IsoNet++ and multi-layer IsoNet++

Table 14 compares multi-round and multi-layer IsoNet++ with respect to different metrics. We observe that multi-round IsoNet++ outperforms multi-layer IsoNet++ by a significant margin when it comes to all metrics, both when the models are node-based or edge-based. This reinforces the observations from MAP scores noted earlier in Table 4. Note that a minor exception occurs for MRR but the scores are already so close to 1 that this particular metric can be discounted and our key observation above still stands.

G.3 Refinement of alignment matrix across rounds and layers in multi-round IsoNet++ and multi-layer IsoNet++

The node (edge) alignment calculated after round t is denoted as P t ( S t ). We accumulate such alignments across multiple rounds. This also includes P T ( S T ) which is used to compute the relevance distance in Eq. 14 (Eq. 25). We wish to compare the predicted alignments with ground

truth alignments. We expect our final alignment matrix P t ( S t ) to be one of them. We determine the closest ground truth matrices P ∗ and S ∗ by computing max P Tr( P ⊤ T P ) and max S Tr( S ⊤ T S ) for IsoNet++ (Node) and IsoNet++ (Edge) respectively. We now use the closest ground-truth alignment P ∗ , to compute Tr( P ⊤ t P ∗ ) for t ∈ [ T ] . For each t , we plot a histogram with bin width 0.1 that denotes the density estimate p(Tr( P ⊤ t P ∗ )) . The same procedure is adopted for edges, with S ∗ used instead of P ∗ . The histograms are depicted in Figure 15. We observe that the plots shift rightward with increasing t . The frequency of graph pairs with misaligned P t ( S t ) decreases with rounds t while that with well-aligned P t ( S t ) increases.

Here, we also study alignments obtained through multi-layer refinement. We adopt the same procedure as in Section G.3. One key difference is that the node/edge alignments are computed after every layer k and are accumulated across layers k ∈ [ K ] . In Figure 15, we observe that the plots, in general, shift rightward with increasing k . The frequency of graph pairs with misaligned P t ( S t ) decreases with rounds k while that with well-aligned P k ( S k ) increases.

G.4 Comparison across alternatives of multi-layer IsoNet++ (Node) and multi-round IsoNet++ (Node)

In Table 16, we compare different alternatives to the multi-round and multi-layer variants of IsoNet++ (Node). In particular, we consider four alternatives - Node partner (equation shown in Section 4), Node partner (with additional MLP) [Appendix E.3], Node pair partner ( msg only) [Appendix E.4] and IsoNet++ (Node). We observe that for all metrics, IsoNet++ (Node) and Node pair partner ( msg only) dominate the other alternatives in most cases. This highlights the importance of node pair partner interaction for determining the subgraph isomorphism relationship between two graphs. For the multi-round variant, IsoNet++ (Node) outperforms Node pair partner ( msg only) in four of the datasets and is comparable / slightly worse in the other two. Once again, comparisons based on MRR break down because it does not cause a strong differentiation between the approaches.

0

5

10

Tr

(

P

/latticetop

t

P

∗

)

→

0

.

04

0

.

08

0

.

12

0

.

16

0

.

20

p

(

Tr

(

P

/latticetop

t

P

∗

))

→

AIDS

Multi-Round

t

= 1

t

= 2

t

= 3

0

5

10

Tr

(

P

/latticetop

k P

∗

)

→

0

.

04

0

.

08

0

.

12

0

.

16

0

.

20

p

(

Tr

(

P

/latticetop

k P

∗

))

→

AIDS

Multi-Layer

k

= 1

k

= 2

k

= 3

k

= 4

k

= 5

0

5

10

Tr

(

S

/latticetop

t S

∗

)

→

0

.

04

0

.

08

0

.

12

p

(

Tr

(

S

/latticetop

t S

∗

))

→

AIDS

Multi-Round

t

= 1

t

= 2

t

= 3

0

5

10

Tr

(

S

/latticetop

k S

∗

)

→

0

.

04

0

.

08

0

.

12

p

(

Tr

(

S

/latticetop

k S

∗

))

→

AIDS

Multi-Layer

k

= 1

k

= 2

k

= 3

k

= 4

k

= 5

0

5

10

Tr

(

P

/latticetop

t

P

∗

)

→

0

.

04

0

.

08

0

.

12

0

.

16

p

(

Tr

(

P

/latticetop

t

P

∗

))

→

MUTAG

Multi-Round

t

= 1

t

= 2

t

= 3

0

5

10

Tr

(

P

/latticetop

k P

∗

)

→

0

.

04

0

.

08

0

.

12

0

.

16

p

(

Tr

(

P

/latticetop

k P

∗

))

→

MUTAG

Multi-Layer

k

= 1

k

= 2

k

= 3

k

= 4

k

= 5

0

5

10

Tr

(

S

/latticetop

t S

∗

)

→

0

.

04

0

.

08

0

.

12

0

.

16

p

(

Tr

(

S

/latticetop

t S

∗

))

→

MUTAG

Multi-Round

t

= 1

t

= 2

t

= 3

0

5

10

Tr

(

S

/latticetop

k S

∗

)

→

0

.

04

0

.

08

0

.

12

0

.

16

p

(

Tr

(

S

/latticetop

k S

∗

))

→

MUTAG

Multi-Layer

k

= 1

k

= 2

k

= 3

k

= 4

k

= 5

0

5

10

Tr

(

P

/latticetop

t

P

∗

)

→

0

.

04

0

.

08

0

.

12

0

.

16

0

.

20

p

(

Tr

(

P

/latticetop

t

P

∗

))

→

PTC-FM

Multi-Round

t

= 1

t

= 2

t

= 3

0

5

10

Tr

(

P

/latticetop

k P

∗

)

→

0

.

04

0

.

08

0

.

12

0

.

16

0

.

20

p

(

Tr

(

P

/latticetop

k P

∗

))

→

PTC-FM

Multi-Layer

k

= 1

k

= 2

k

= 3

k

= 4

k

= 5

0

5

10

Tr

(

S

/latticetop

t S

∗

)

→

0

.

04

0

.

08

0

.

12

0

.

16

0

.

20

0

.

24

0

.

28

0

.

32

0

.

36

0

.

40

p

(

Tr

(

S

/latticetop

t S

∗

))

→

PTC-FM

Multi-Round

t

= 1

t

= 2

t

= 3

0

5

10

Tr

(

S

/latticetop

k S

∗

)

→

0

.

04

0

.

08

0

.

12

0

.

16

0

.

20

0

.

24

0

.

28

0

.

32

0

.

36

0

.

40

p

(

Tr

(

S

/latticetop

k S

∗

))

→

PTC-FM

Multi-Layer

k

= 1

k

= 2

k

= 3

k

= 4

k

= 5

0

5

10

Tr

(

P

/latticetop

t

P

∗

)

→

0

.

04

0

.

08

0

.

12

0

.

16

p

(

Tr

(

P

/latticetop

t

P

∗

))

→

PTC-FR

Multi-Round

t

= 1

t

= 2

t

= 3

0

5

10

Tr

(

P

/latticetop

k P

∗

)

→

0

.

04

0

.

08

0

.

12

0

.

16

p

(

Tr

(

P

/latticetop

k P

∗

))

→

PTC-FR

Multi-Layer

k

= 1

k

= 2

k

= 3

k

= 4

k

= 5

0

5

10

Tr

(

S

/latticetop

t S

∗

)

→

0

.

04

0

.

08

0

.

12

0

.

16

0

.

20

0

.

24

0

.

28

p

(

Tr

(

S

/latticetop

t S

∗

))

→

PTC-FR

Multi-Round

t

= 1

t

= 2

t

= 3

0

5

10

Tr

(

S

/latticetop

k S

∗

)

→

0

.

04

0

.

08

0

.

12

0

.

16

0

.

20

0

.

24

0

.

28

p

(

Tr

(

S

/latticetop

k S

∗

))

→

PTC-FR

Multi-Layer

k

= 1

k

= 2

k

= 3

k

= 4

k

= 5

0

5

10

Tr

(

P

/latticetop

t

P

∗

)

→

0

.

04

0

.

08

0

.

12

0

.

16

p

(

Tr

(

P

/latticetop

t

P

∗

))

→

PTC-MM

Multi-Round

t

= 1

t

= 2

t

= 3

0

5

10

Tr

(

P

/latticetop

k P

∗

)

→

0

.

04

0

.

08

0

.

12

0

.

16

p

(

Tr

(

P

/latticetop

k P

∗

))

→

PTC-MM

Multi-Layer

k

= 1

k

= 2

k

= 3

k

= 4

k

= 5

0

5

10

Tr

(

S

/latticetop

t S

∗

)

→

0

.

04

0

.

08

0

.

12

0

.

16

0

.

20

0

.

24

p

(

Tr

(

S

/latticetop

t S

∗

))

→

PTC-MM

Multi-Round

t

= 1

t

= 2

t

= 3

0

5

10

Tr

(

S

/latticetop

k S

∗

)

→

0

.

04

0

.

08

0

.

12

0

.

16

0

.

20

0

.

24

p

(

Tr

(

S

/latticetop

k S

∗

))

→

PTC-MM

Multi-Layer

k

= 1

k

= 2

k

= 3

k

= 4

k

= 5

0

5

10

Tr

(

P

/latticetop

t

P

∗

)

0

.

04

0

.

08

0

.

12

0

.

16

0

.

20

p

(

Tr

(

P

/latticetop

t

P

∗

))

→

PTC-MR

Multi-Round

t

= 1

t

= 2

t

= 3

0

5

10

Tr

(

P

/latticetop

k P

∗

)

0

.

04

0

.

08

0

.

12

0

.

16

0

.

20

p

(

Tr

(

P

/latticetop

k P

∗

))

→

PTC-MR

Multi-Layer

k

= 1

k

= 2

k

= 3

k

= 4

k

= 5

0

5

10

Tr

(

S

/latticetop

t S

∗

)

0

.

04

0

.

08

0

.

12

p

(

Tr

(

S

/latticetop

t S

∗

))

→

PTC-MR

Multi-Round

t

= 1

t

= 2

t

= 3

0

5

10

Tr

(

S

/latticetop

k S

∗

)

0

.

04

0

.

08

0

.

12

p

(

Tr

(

S

/latticetop

k S

∗

))

→

PTC-MR

Multi-Layer

k

= 1

k

= 2

k

= 3

k

= 4

k

= 5

→

→

→

→

G.5 Comparison of GMN with IsoNet++ alternative for multi-layer and multi-round

In Table 17, we modify the GMN architecture to include node pair partner interaction in the messagepassing layer. Based on the reported metrics, we observe that there is no substantial improvement upon including information from node pairs in GMN, which is driven by a non-injective mapping (attention). This indicates that injectivity of the doubly stochastic matrix in our formulation is crucial towards the boost in performance obtained from node pair partner interaction as well.

Mean Average Precision (MAP)

AIDS

Mutag

FM

FR

MM

MR

Multi-Layer

{

GMN

0.622

0.71

0.73

0.662

0.655

0.708

Node pair partner

0.579

0.732

0.74

0.677

0.641

0.713

Multi-Round

{

GMN

0.629

0.699

0.757

0.697

0.653

0.714

Node pair partner

0.579

0.693

0.729

0.69

0.665

0.705

HITS@20

AIDS

Mutag

FM

FR

MM

MR

Multi-Layer

{

GMN

0.397

0.544

0.537

0.45

0.423

0.49

Node pair partner

0.346

0.567

0.551

0.476

0.411

0.5

Multi-Round

{

GMN

0.403

0.533

0.562

0.494

0.431

0.502

Node pair partner

0.344

0.528

0.54

0.502

0.462

0.506

Mean Reciprocal Rank (MRR)

AIDS

Mutag

FM

FR

MM

MR

Multi-Layer

{

GMN

0.877

0.923

0.949

0.947

0.928

0.922

Node pair partner

0.827

0.897

0.958

0.877

0.918

0.92

Multi-Round

{

GMN

0.905

0.862

0.958

0.956

0.906

0.921

Node pair partner

0.811

0.901

0.907

0.908

0.964

0.92

Precision@20

AIDS

Mutag

FM

FR

MM

MR

Multi-Layer

{

GMN

0.751

0.82

0.852

0.809

0.783

0.832

Node pair partner

0.7

0.833

0.861

0.797

0.792

0.846

Multi-Round

{

GMN

0.753

0.795

0.885

0.829

0.792

0.842

Node pair partner

0.694

0.794

0.847

0.835

0.802

0.825

G.6 Variation of IsoNet++ (Node) and IsoNet++ (Edge) with different T and K

In this section, we analyze the accuracy and inference time trade-off of multi-round lazy and multilayer eager variants of IsoNet++ (Node) and IsoNet++ (Edge). In the following tables, we show the MAP and inference time. Additionally, we also analyze the trade-off of GMN and IsoNet (Edge). The T, K parameters for different models are so chosen that they can be compared against each other while fixing the inference time to be roughly similar. For instance, multi-round lazy IsoNet++ (Node) with T = 5 , K = 5 maps to multi-layer eager IsoNet++ (Node) with K = 8 , allowing for a direct comparison of performance without caring much about different compute. Note that in below tables, models are listed in order of increasing inference time (i.e. increasing K or T ).

In tables 19 and 20, we show variations for multi-round lazy IsoNet++ (Node) for fixed T and fixed K respectively. We observe that with fixed T , increasing K from 5 to 10 doesn't improve the model significantly. For fixed K , performance (in terms of MAP) improves notably when increasing T from 3 to 5.

In table 21, we show variations for multi-layer eager IsoNet++ (Node) for varying K . We observe that except for a drop at K = 7 , the performance of the model improves as we increase K . In fact, at K = 8 , the performance is surprisingly good, even outperforming the similarly timed T = 5 , K = 5 variant of lazy multi-round IsoNet++ (Node) on both AIDS and Mutag.

In tables 22 and 23, we compare variants of multi-round lazy IsoNet++ (Edge) with fixed T and fixed K respectively. We observe that when T is fixed and K is increased, the gain is marginal. We observe a significant gain When K is fixed and T is increased from 3 to 4.

In table 24, we study the trade-off for multi-layer eager IsoNet++ (Edge) for varying K . We observe that with increasing K , the performance continues to improve and peaks at K = 8 . Note that even at this K , the performance of multi-layer eager IsoNet++ (Edge) is worse than a similarly timed variant ( T = 5 , K = 5 ) of multi-round IsoNet++ (Edge).

In table 25, we show variations for GMN for varying K . We observe marginal gains while increasing K . From K = 10 to K = 12 , the performance drops.

In table 26, we show how performance varies for IsoNet (Edge) for varying K . We observe that the model does not improve with increasing K .

50

80

110

140

Inference time (in ms)

0

.

6

0

.

8

1

.

0

MAP

Lazy

Eager

GMN

(a) Node, AIDS

50

80

110

140

Inference time (in ms)

0

.

6

0

.

8

1

.

0

MAP

Lazy

Eager

GMN

(b) Edge, AIDS

50

80

110

140

Inference time (in ms)

0

.

6

0

.

8

1

.

0

MAP

Lazy

Eager

GMN

(c) Node, Mutag

50

80

110

140

Inference time (in ms)

0

.

6

0

.

8

1

.

0

MAP

Lazy

Eager

GMN

(d) Edge, Mutag

G.7 Contribution of refining alignment matrix in inference time

In GMN, computing the embeddings of nodes after the message passing step at each layer dominates the inference time. However, in the case of IsoNet++ models, we observe the refinement of the alignment matrix at each layer or round to also be time-intensive. In table 27, we show the contribution of embedding computation and matrix updates to the total inference time. The updates to P constitute the largest share of inference time for multi-layer variants. This can be attributed to the refinement of P after every message passing step, equaling the frequency of embedding computation. In the case of multi-round variants, both embedding computation and updates to P contribute almost equally since P is refined only at the end of each round, after several layers of message passing alongwith embedding computation.

G.8 Transfer ability of learned models

In this section, we evaluate the transfer ability of each trained model across datasets. In table 28, we report the Mean Average Precision (MAP) scores for models trained using the AIDS and Mutag datasets respectively evaluated on all six datasets. We observe that despite a zero-shot transfer from one of the datasets to all others, variants of IsoNet++ show the best accuracy.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Section 1.1 discusses the paper's contributions.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]

Justification: Appendix A discusses the limitations of our work. Details about computational efficiency are included in the main paper as well as in Appendix G.6, expressed explicitly as the running time of each approach.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The paper includes one theorem, which is noted and proved in Appendix D.5. Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The paper introduces new architectures and the designs of each of these are discussed in the main paper under Sections 3.2, 3.3 and Appendices D, E. Hyperparameters, training procedure, hardware and random seeds for all experiments are noted in Appendix F.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Uploaded in github.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not

including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Important parameters that are required to understand and appreciate the results are noted as and when required. Hyperparameters, training procedure, hardware and random seeds for all experiments are noted in Appendix F.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Appendix G.1 includes standard error for the metrics reported in the paper, which includes Mean Average Precision (MAP) and HITS@20, computed across each query graph in the test split. The section also reports the method of calculation of the standard error.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Types of GPUs and inference time are reported in Appendices F.5 and G.6 respectively. Other details about the training setting are mentioned point-wise in Appendix F.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The authors have studied the ethics guidelines and find the work to conform well to them.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Broader societal impacts (both positive and negative) are discussed in Appendix C.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The datasets used in the paper are derived from public graph datasets widely used by researchers and the authors are not aware of any risks for misuse posed by them.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The related work and datasets applicable to this work have been cited in Appendices B and F.1 respectively. Best effort was made by the authors to determine the licenses of the datasets and existing architectures, and are included in Appendix F.6.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: Details of the models and datasets are discussed in Appendices D and F.1 respectively. Datasets used in the paper are publicly accessible. The code is submitted with the paper as a supplementary item and is anonymized & well-documented.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The experiments performed in this paper do not involve human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The experiments performed in this paper do not involve human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
The paper introduces EINSMATCH, an early interaction graph neural network for subgraph matching and retrieval that maintains and refines explicit alignments between query and corpus graphs. EINSMATCH has several novelties: computing embeddings guided by injective alignments between graphs, updating alignments lazily over multiple rounds, and using node-pair partner interactions. The model learns to identify alignments between graphs despite only having access to pairwise preferences during training, without explicit alignments. Experiments on several datasets show that EINSMATCH significantly outperforms existing methods.

STRENGTHS:
1. The analysis of GMN’s worse performance compared to IsoNet (in Section 1) is intriguing.
2. Extensive experimental results are presented with ample baselines and std errors included as well for most numbers reported.
3. Source code with data splits is released for ease of reproducing the results.
4. A large amount of ablation and parameter sensitivity studies are performed.

WEAKNESSES:
1. The datasets used in the experiments are relatively small, with at most ~20 nodes. 
2. It would be interesting to see the transfer ability of the learned model across datasets, e.g. training on PTC and testing on AIDS, to see how the proposed model would handle such challenging but realistic scenarios.

QUESTIONS:
N/A

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a GNN architecture for graph matching problem. The innovations in the work mainly lie in the early interaction between the query graph and the corpus graph, and the node-pair matching setting, as well as the lazy update technique in each round. Empirical results show that this work significantly outperforms the baselines on all the datasets.\n\nSTRENGTHS:\n- The writing of the paper is clear and good. \n- The methodology is solid, and the illustration is informative. \n- The results are very promising, both in predictive performance and runtime. And the ablation experiments are well-designed.\n\nWEAKNESSES:\n- Some background part is not well explained, for example the graph matching problem, and the optimal transport problem. Readers may need external resources to understand them. Of course it is hard to explain everything in detail in the main paper.\n\nQUESTIONS:\n- Any related work indicating early interaction is good? As mentioned in line 36.\n- In line 124 the notation $A_q \\leq PA_c P^T$ is a bit confusing. \n- In equation 5, what is the sign of $\\tau$? If it's positive then you're encouraging the entropy to be higher, which means the entries of P are more uniform? That is a bit counter-intuitive. \n- What is the complexity of Sinkhorn Knopp algorithm and Gumbel Sinkhorn?\n- Is the K-layer GNN in each round t shared?\n- Any intuition why lazy updates works better? Is it easier to optimize or other aspects?\n- I don't quite understand why the work is not differentiable on nodes with different classes, as mentioned in appendix A. \n- Is there other measure between the $P$ and $P^*$? I feel $Tr(P^TP^*)$ does not capture all the information of the matrices.\n\nLIMITATIONS:\nAs mentioned by the authors, the quality of the matching relies on the distance function, which is a general problem for graph matching works. Also I am a little dubious about the time complexity of the P matrix update, in the experiments it is more expensive than the baseline GMN.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an early interaction network for subgraph matching. The proposed method enables (1) early interaction GNNs with alignment refinement at both node and edge levels, where the alignments are refined by GNNs; (2) eager or lazy alignment updates,  and (3) node-pair partner interaction, instead of node partner interaction, which improves the model efficiency and embedding quality. The paper includes extensive experiments on real-world datasets, demonstrating the superiority and effectiveness of the proposed approach.\n\nSTRENGTHS:\n1. The paper is well-written and easy to understand. The notations are clearly clarified and explained.\n2. The idea of iteratively early interaction between graphs for more accurate subgraph retrieval is novel and reasonable to me, which is also verified by the experimental results.\n3. The performance superiority of the proposed method against other baselines is significant and promising.\n\nWEAKNESSES:\n1. Considering the high complexity of the proposed method, an ablation study (or a hyperparameter sensitivity study) would be beneficial to provide a clearer understanding of the success of the proposed method. In addition, how did the authors tune hyperparameters (apart from T and K) for their approach and the competitors?\n2. Experiments are only conducted on small graph datasets (with an average of 11 nodes). It would be beneficial to discuss or evaluate the applicability of the proposed approach to larger graph datasets.\n\nQUESTIONS:\n- What is the motivation and intuition behind lazy updates? In my understanding, \"lazy\" means updating the alignment map after every K GNN layers, and \"eager\" means updating the alignment map and letting the map affect the embedding at each layer. Is there any explanation for the consistently better performance of \"lazy\" updates?\n- What do the multiple points with the same color in Figure 6 represent? Do they indicate variants with different hyperparameters?\n\nLIMITATIONS:\nYes. This paper discussed the limitations in terms of efficiency and", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes EinsMatch, a neural approach for graph retrieval, i.e. the problem of finding  the best candidates of a corpus of graphs that contain a subgraph isomorphic to a given query graph. EinsMatch is based on a Graph Neural Network architecture and introduces technical improvements over existing methods: (i) message-passing iterations across query and candidate graphs which are guided by an injective alignment; (ii) the iterations are lazy in the sense that multiple of them are computed with a fixed alignment before refining it; (iii) node pairs are considered as potential partners instead of single nodes in the message-passing steps. In addition, the authors propose a variant which works at the level of edges and natively leverages edge matchings. This variant seems to be the most competitive one in the experiments the authors run. In general, EinsMatch significantly outperforms a variety of other methods. The authors also report insightful ablation studies on the number of alignment rounds and message passing layers.\n\nSTRENGTHS:\n- The proposed architecture nicely interpolates between late and early aggregation approaches, whilst ensuring injectiveness of the candidate mapping. In this sense, it appears as a sensible generalisation of other approaches.\n\n- The proposed model is highly performant if compared with other methods, and, in particular offers a competitive accuracy-inference time tradeoff if compared with other approaches as GMNs.\n\n- The reported ablation studies are insightful in that they shed light on the impact of important hyperparameters, namely the number of layers and rounds.\n\nWEAKNESSES:\n- I found the paper somewhat hard to follow. The contributions are rather technical and only one single figure in the manuscript illustrate the approach. I think it would be particularly helpful if the authors improved on this aspect.\n\n- Still on the presentation side, I found it hard to clearly understand the specific contributions w.r.t. previous works. I believe the paper would increase in quality if the authors would state these more clearly in the main corpus.\n\n- The authors only provide a high-level intuition behind some specific technical contributions, e.g. the choice of working with node-pair partners. Would it be possible to refer to more formal arguments around those?\n\nQUESTIONS:\nPlease see Weaknesses. Also:\n\n- The readers refer oversmoothing problems in some of the previous methods. Can they expand on how their approach side-steps this?\n\n- Can the authors better explain lines 333 — 336? I found them hard to understand.\n\n- How does the computational complexity compare with that of IsoNet?\n\nLIMITATIONS:\nThe authors do not seem to explicitly discuss them in the main corpus, but mention two aspects in Appendix A.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-21"}