EurNet: Efficient Multi-Range Relational Modeling of Spatial Multi-Relational Data
Authors: Minghao Xu
Yuanfan Guo
Yi Xu
Jian Tang
Xinlei Chen
Yuandong Tian
Venue: Submitted to ICLR 2023 (2023)
Keywords: Multi-Relational Modeling
Image Modeling
Protein Structure Modeling

Abstract: Modeling spatial relationship in the data remains critical across many different tasks, such as image classification, semantic segmentation and protein structure understanding. Previous works often use a unified solution like relative positional encoding. However, there exists different kinds of spatial relations, including short-range, medium-range and long-range relations, and modeling them separately can better capture the focus of different tasks on the multi-range relations (e.g., short-range relations can be important in instance segmentation, while long-range relations should be upweighted for semantic segmentation). In this work, we introduce the EurNet for Efficient multi-range relational modeling. EurNet constructs the multi-relational graph, where each type of edge corresponds to short-, medium- or long-range spatial interactions. In the constructed graph, EurNet adopts a novel modeling layer, called gated relational message passing (GRMP), to propagate multi-relational information across the data. GRMP captures multiple relations within the data with little extra computational cost. We study EurNets in two important domains for image and protein structure modeling. Extensive experiments on ImageNet classification, COCO object detection and ADE20K semantic segmentation verify the gains of EurNet over the previous SoTA FocalNet. On the EC and GO protein function prediction benchmarks, EurNet consistently surpasses the previous SoTA GearNet. Our results demonstrate the strength of EurNets on modeling spatial multi-relational data from various domains. 

Under review as a conference paper at ICLR 2023
EURNET: EFFICIENT MULTI-RANGE RELATIONAL MODELING OF SPATIAL MULTI-RELATIONAL DATA
Anonymous authors Paper under double-blind review
ABSTRACT
Modeling spatial relationship in the data remains critical across many different tasks, such as image classification, semantic segmentation and protein structure understanding. Previous works often use a unified solution like relative positional encoding. However, there exists different kinds of spatial relations, including short-range, medium-range and long-range relations, and modeling them sepa- rately can better capture the focus of different tasks on the multi-range relations (e.g., short-range relations can be important in instance segmentation, while long- range relations should be upweighted for semantic segmentation). In this work, we introduce the EurNet for Efficient multi-range relational modeling. EurNet con- structs the relatiorllatonal graph, where each type of edge corresponds to short- medium- or long-range spatial interactions. In the constructed graph. EurNet adopts novel modeling layer, called gated relational message passing (GRMP), to propagate multi-relational information across the data. GRMP captures multi- ple relations within the data with little extra computational cost. We study EurNets in two important domains for image and protein structure modeling. Extensive experiments on ImageNet classification, COCO object detection and ADE20K se- mantic segmentation verify the gains of EurNet over the previous SoTA FocalNet. On the EC and GO protein function prediction benchmarks, EurNet consistently surpasses the previous SoTA GearNet. Our results demonstrate the strength of EurNets on modeling spatial multi-relational data from various domains.
INTRODUCTION
This work studies the data that lie in the 2D/3D space and incorporate interacting relations on dif- ferent spatial ranges. representative example is the image data, where an object in the image can interact with other adjacent objects via the direct touch, and it can also interact with those distantly relevant ones via gazing, waving hands or pointing. In protein science, the protein 3D structure an- other typical example, in which different amino acids can interact in short range by peptide/hydrogen bonds, and they can also interact in medium and long ranges by hydrophobic interaction. We sum- marize such kind of data as spatial multi-relational data.
In various domains, a lot of previous efforts have been made to model the spatial multi-relational data. For image modeling, multi-head self-attention mechanisms (Dosovitskiy et al. [0200] Liu et al. 2021b), convolutional operations with large receptive fields (Ding et al. [022]] Yang el al. (022) and MLPs for mixing full spatial information (Tolstikhin al. 2021 Touvron al. 202 lab explored to capture multi-range spatial interactions within an image. For protein structure modeling, Zhang al. (2022) builds multiple groups of edges for different short-range interactions and employs re- lational graph convolution (Schlichtkrull et al. (018)) for multi-relational modeling. These works either implicitly treat different kinds of spatial relations (i.e., short-range, medium-range and long- range relations) Tolstikhin et al. [021]] Yang et al. 2022) or handle them by unified scheme like relative positional encoding (Dosovitskiy et al. 2020 Liu et al. 202 However, considering the relative importance of these spatial relations could vary across different tasks (e.g., the great impor- tance of short-range relations in instance segmentation, and the upgraded importance of long-range -ragge relations in semantic segmentation). separately modeling each spatial relation is better solution to capture different tasks' focus. Such a separate modeling approach remains to be explored, and, especially, the approach is expected have efficient adaptation to large data and model scales.

---

Under review as a conference paper at ICLR 2023
To attain this goal, we propose the EurNet for Efficient multi-range relational modeling. In general, EurNets are a series of relational graph neural networks equipped with graph construction layers, where relational edges are constructed by the layers for capturing multi-range spatial interactions. When instantiated with different domain knowledge (e.g. computer vision or protein science), Eur- Nets can be specialized to tackle important problems like image classification, image segmentation and protein function prediction. To be specific, upon the raw data, EurNet first uses the graph con- struction layers build different types of edges that respectively capture the short-, medium- and long-range spatial interactions within the data. For efficient structed graph, we next introduce the gated relational message passing (GRMP) layer as the basic modeling module of EurNet. GRMP separately performs (1) relational message aggregation on each individual feature channel and (2) node-wise aggregation of different feature channels. Compared to the classical relational graph convolution (RGConv) (Schlichtkrull et al. 2018), GRMP enjoys lower computational cost when more relations are to be modcled, and thus can handle more types of spatial interactions given the same computational budget. EurNet also supports dynamic graph construction and multi-stage modeling that are used in domains like image modeling.
modeling over the con-
We demonstrate EurNets in image and protein structure modeling. To model image patches with different granularity, we build EurNets with hierarchical graph construction layers and multiple modeling stages and derive model series with increasing capacity, i.e,, EurNet-T, EurNet-S and EurNet-B. These models enjoy comparable or better top-1 accuracy (82.3% v.s. 82.3%; 83.6% v.s. 83.5%: 84.1% v.s. 83.9%) against the previous SoTA FocalNet,LRF) series (Yang et al.. 2022) on ImageNet-1K IK classification (resolution: 224 x 224). Similar performance gains are preserved on COCO object detection and ADE20K semantic segmentation. To model protein alpha carbons, we build EurNet with a single-stage model architecture as GearNet et al. (2022). Under this fix- architecture comparison, EurNet consistently outperforms the SOTA GearNet on standard protein function prediction benchmarks in terms of protein-centric maximum F-score (EC: 0.768 v.s. 0.730; GO-BP: 0.437 v.s. 0.356; GO-ME: 0.563 v.s. 0.503; GO-CC: 0.421 v.s. 0.414). These performance improvements remain when edge-level message passing is involved. Our results demonstrate that EurNet could be a strong candidate for modeling spatial multi nulti-relational data in various domains.
RELATED WORK
Multi-relational data modeling. Multi-relational data are ubiquitous in the real world, e.g., knowl- edge graphs (Toutanova & Chen 2015) and customer-produee networks (Li et al. 2014). To effec- tively model multiple types of relations/interactions, existing works have explored embedding- methods (Bordes et al. 2013 Sun et al. 2019), multi-headed attention (Vaswani et al. 2017) and different relational graph neural networks (GNNs) (Schlichtkrull et al., 2018 Vashishth et al. 2019 Busbridge et al. 2019 Zhu et al. 2021). Previous relational GNNs mainly focus model expres- sivity and parameter efficiency, and few works (Li et al. 202T) study the computational efficiency for relational modeling at scale. In addition, they can hardly model the spatial multi-relational data whose relational linking structures at different spatial ranges are not originally given (e.g., image patches). EurNet is designed model such kind of data in computationally efficient way.
Image modeling. After the dominance of convolutional vision backbones (He et al., 2016 Jan & Le 2019) in 2010s, researchers rethink the architectures for more effective image modeling in
2020s. Vision Transformers (Dosovitskiy et al. 2020 Liu et al. 2021b Wang et al. 2222] replace
convolutions with the self-attention mechanism (Vaswani et al. interactions and gain So'TA performance, Following such successes, modern convolutional architec-
2017) to better capture non-local
tures (Liu al. 2022; Yang et al. 2022). all-MLP architectures (Tolstikhin et al., 202T Touvron
et al. 2021a) and vision GNNs (Han et al. (202)) are designed to aggregate long-range spatial con-
text. Some earlier works (Chen et al. [201 IZhang et al. 2019 2020) realize non-local modeling by graph convolution on fully fully-connected or dynamic graphs. By comparison, EurNet captures multi- range spatial interactions from a novel graph learning perspective, i.e., multi-rclational modeling.
Protein structure modeling. A variety of protein structure encoders have been developed to ac- quire informative protein representations on different structural granularity, including bssidue-eevel structures (Gligorijevié et al., [002 Zhang et al. (202).. , atom-level structures (Jing et al. 2021] Her-
mosilla et al.) 2021) and protein surfaces (Gainza et al. 2020 Sverrisson et al. 2021). This work focuses the residue-level protein structure modeling- GearNet (Zhang et al.
2022 is closely
2

---

Under review as a conference paper at ICLR 2023
related work which explores multi-relational modeling of residue-level structures with sanrt-range linking and relational graph convolution (RGConv). By comparison, our EurNet models a broader range of interactions including short, medium and long ranges, and it studies the gated relational message passing (GRMP) as a more efficient and equally effective alternative of RGConv.
EURNET FOR EFFICIENT MULTI-RANGE RELATIONAL MODELING
3.1 PROBLEM DEFINITION
This work studies the data V (= with N data units (e.g. patches in an image, alpha carbons in protein, etc,) with the following structure: (1) spatial interaction on multiple ranges: data units can interact with each other across diverse spatial ranges; (2) multi-relational interaction: multiple interaction types (i.e., relations) exist between different units; (3) canonical linking structure: the linking structures of multi-range interactions are not specified in the raw data.
To effectively model such spatial multi-relational data, the model is expected to own following ca- pabilities: (1) dynamic multi-range linking: the model can link relevant data units across different spatial ranges, and the linking structure can change along the whole model if desired; (2) multi- relational linking: the model divides all links into multiple groups based on their interaction types; (3) efficient multi-relational modeling: the model can propagate information among interacting units by taking their interaction types into consideration, and it will not introduce too much ex- tra computation when involving more relations. Keeping all these requirements in mind, we next introduce the high-level designs of EurNet, and we present its detailed instantiations in Sec.
3.2 MULTI-RANCE RELATIONAL GRAPH CONSTRUCTION
We regard each data unit E V as node in the graph. For the lack of canonical linking structure among the nodes, we therefore seek to build edges among them, especially with considering their interactions multiple spatial ranges and dynamically adjusting the graph structure if desired.
Multi-range relational edge construction. Given the concepts of spatial and semantic adjacency in specific domain (e.g., computer vision protein science), we construct three groups of edges Exhort = {(Mu, v, |r E Rehort) Emediom = {((u, v, Hr E Rmeedium] } and Etong = [[(u, r))lr e Riong] to represent short-, medium- and long-range spatial interactions, where (u, r) denotes an edge from node u to node u with relation r, and R RRonn is the set of relations for shortt/mediun--lon--rangg
interactions. To capture the interactions on different spatial ranges, all these edges are gathered into the edge set E = Eshort U Emedium U Esong {((u, v. Hr E RJ with the integrated relation set Rehort U Rmedeium Riong. Now, the raw data V is structured as a multi-relational graph G = (V,E, E, R.) that is aware of diverse types of interactions within the data.
Dynamic edge construction. model can focus on different levels of semantics at different model- ing stages. For example, for the image modeling problem we consider, typical hierarchical image encoder (He et al., [016]] Liu et al., 2021b) is split into multiple stages, and it tends to encode low- level features in shallower stages and encode high-level semantics in deeper stages. To accommodate such a hierarchical modeling manner, our graph construction scheme will be dynamically performed before each modeling stage based on the input features (e.g., node coordinates representations) of the stage, so that each modeling stage can explore its specific neighborhood structures of data units.
3.3 GATED RELATIONAL MESSAGE PASSING
To perform multi-relational modeling over the constructed graph G, the typical method Relational Graph Convolution (RGConv) (Schlichtkrull et al. 2018) employs a unique convolutional kernel matrix W, to aggregate the messages of relation r, leading to [R] different kernel matrices in total for the message aggregation from neighborhoods. Taking node as an example, the RGConv layer updates its representation from 2n to z as below:
Σ
Σ
W Wuzus
2
Wrelf
2p + 2::
aggr
N.(wi]
PER (v)
where sgerr is the aggregated message for node v, N,)() = {ul (u. e, r) E E} are v's neighbors with relation r, and Weell is the weight matrix for self-update (we omit all bias terms for brevity).
3

---

Under review as a conference paper at ICLR 2023
We assume that, when introducing new relation, the in-degree of each node will increase by d on average. By taking the efficient implementation of RGConv with sparse matrix multiplication, can be shown that the floating point operations (FLOPs) of RGConv with C-dimensional input and output node features has the following form (see Appendix f proof):
FLOPs(RGCons) = |R| (2d + 2|V|C + 2 +IVIC.
(2)
Therefore, the computational cost will scale with the relation number [R] by the factor of 2d + 2 2 Considering both the node number [V] and the feature dimension C could be large in many applications, the 2|V|C C term will be the main obstacle of exploring more relations with moderate extra computation, which hurts the model capacity under a strict constraint on computational cost.
For more efficient multi-relationa modeling, aim at an approach that (1) effectively model the interactions among relational messages and among feature channels, and (2) owns a gentle scal- ing behavior when modeling increasing number of relations within the data. To attain this goal, we propose the Gated Relational Message Passing (GRMP). Inspired by light-weight separable graph convolution methods (Balcilar al.) 2020 Li al. 202T) that aggregate neighborhood features in channel-wise way, GRMP decomposes the channell entangled aggregation of RGConv into (i) the aggregation intra- and inter-relation messages on each individual channel and (ii) the aggregation of different feature channels. Specifically, it consecutively performs following steps: ! a pre-layer node-wise channel aggregation with the weight matrix Win, ② an intra-relation message aggregation through channel-wise graph convolution, 3 inter-relation message aggregation by node-adaptive weighted summation, 4 post-layer nose-wise channel aggregation with the weight matrix Wont, and (5 the final node representation update by regarding the aggregated neighborhood information as gate. Formally, GRMP updates the representation of node v from 2n to ey as below:
stop !
step
A8gTT 2v
= Wout
Car(v) Σ
Wr O
(Wing, zn
2
Waelf
agge
(3)
[N-(v) ER (oN.(o)
step !
stop
where (vv) W" 2n E R are the attentive weights assigned all relations on node v (Waa is the weight matrix for node-adaptive relation weighting), wr is the channel convolutional kernel vector for relation (with the same shape as the node feature vector after step D). and @ denotes the Hadamard product. The definitions of 2 NN-(v) and Wsellf follow Eq. ①. and all biases are omitted. We analyze the components of GRMP in Appendix H.T and its expressivity in Appendix We also provide a graphical illustration of the GRMP layer Appendix C
Under the efficient implementation with sparse matrix multiplication, GRMP consumes the FL.OPs as below when taking C-dimensional input and output node features (see Appendix A for proof):
FLOPs (RMP)
FLOPs(GRMP) = [ (2d + 7) + 6 V
(4)
Therefore, the relation number [R] scales the computational cost of GRMP with the scaling factor (2d + C). Com- pared to the scaling factor Vd + 2// of RGConv, this factor gets rid of the quadratic reliance on feature dimension and thus leads to gentler scaling behavior when increasing the number of considered relations. In [ we compare the FLOPs of RGConv and GRMP when they respectively serve as the building block of EurNet-T for image modeling (im- age resolution: 224 x 224; "T" denotes the tiny-scale model). In this illustrative comparison, we simply connect each node (i.e., image patch) with its K-nearest neighbors in terms of representation similarity, and the connection with the k-th nearest neighbor is regarded as the k-th relation, leading to K relations in total. We can observe that, when increasing the number of neighbors and thus the number of relations, the computational cost of GRMP- model increases much more gently than the RGConv based one. This merit enhances the efficiency and effectiveness of GRMP- ERsed models in real-world problems like image and protein structure modeling, studied in the second paragraph of Sec. 5.3 and in the Appendix
RCCoo
GRMP
4.3
1Rcdadon
Figure 1: FLOPS trend of RGConv and GRMP under different relation numbers, evaluated on EurNet-T for image modeling.
4

---

Under review as a conference paper at ICLR 2023
4 INSTANTIATIONS OF EURNET
In the main paper, we focus on two application domains, i.e., computer vision and protein science, where modeling spatial multi-relational data (i.e., images and protein structures) can solve important problems. In Appendix G we further study the effectiveness of EurNet on modeling an important kind of multi-relational data without spatial information, i.e., knowledge graphs.
4.1 EURNET FOR IMAGE MODELING
4.1.1 RELATIONAT. EDGES FOR SHORT-, MEDIUM- AND LONG-RANGE INTERACTIONS
Following previous practices (Dosovitskiy et al. 2020 local patches and regard these patches the node set of our multi-relations graph. Upon these patches, we construct following relational edges to capture different ranges of spatial interactions within an image (see Fig. 2 for graphical illustration):
Liu et al. 2021b), we split an image into
Edges for short-range interactions ([Rshort 4). We connect each patch with its up. down, left and right patches and regard each direction of adjacency relation. These edges capture the one-hop spatial neighbors and thus shortest-range spatial interactions of each image patch.
short-range edges
medium- odgo
long range edge
long range edge (link whole image)
(link glohal context)
Edges for medium-range interactions ([meediuml 1). In the medium range, patch can interact with other patches shar- ing similar semantics (e.g., different body parts of deer in Fig. 2). We thus connect each patch with its K -nearest neighbors in terms of representation similarity measured by negative Euclidean distance (we analyze the sensitivity of K in Appendix D. and these edges are with the same relation. All edges con- necting two patches within the same 2x2 window are removed to avoid short-range linking.
Figure 2: Multi-range relational edges for image.
Edges for long-range interactions (Roon = 2). To model long-range interactions, we intro- duce two kinds of virtual nodes and the associated edges. (1) A virtual node for whole-image representation is derived by global average pooling over all patch representations, and this vir- tual node is linked to all patches. (2) Per-patch virtual nodes for surrounding global context got by a stack of depth-wise 2D convolutions (Yang et al.] 20022) that aggregate each patch's 's con- textual information with large receptive field and low computation; an edge links each of these virtual nodes to its corresponding patch with a different long-range interacting relation against that in (I), due the different global context levels represented by two kinds of virtual nodes.
By gathering all these edges representing different relations, we have the edge set E, the relation (i.e., edge type) set R and the full graph G = (V, E, R) for multi-relational image modeling.
4.1.2 MODEL. ARCHITECTURE
General architecture. In general, we follow the hierarchical image modeling architecture proposed by Swin Transformer (Liu et al. 20216), which is verified to be superior architecture and is applied to many vision backbones (Liu et al. 2022) Yang et al. 2021 2022). Specifically, the whole model is divided into four stages that (1) reduce the number of patches (i.e., nodes in our graph) to a quarter across consecutive stages, and (2) use increasing number of feature channels [C, 2C, 4C, 8C) for all stages. Each stage contains multiple modeling blocks, where we construct each block with a GRMP layer (Sec. B.3) for relational message passing and a feed-forward network (FFN) (Vaswani ct al. 2017) ) for feature transformation. We adjust the number of feature channels and the number of blocks in each stage to get model series with increasing capacity, i.e., EurNet-T, EurNet-S and EurNet-B. The detailed architectures of these models are displayed in Appendix D
Graph construction layers. To adapt the multi-stage modeling manner, we put graph construction layer before each modeling stage of EurNet-T/SA T/S/B. In this way, based on the locations and represen- tations of the patches fed into each stage, the multi-relational graph & will be reconstructed adapt
5

---

Under review as a conference paper at ICLR 2023
these stage imputs. In particular, along the modeling stages, the edges for medium- range interactions are expected to capture the semantic neighbors on different semantic levels (e.g. from the relevance of low-level features to the relevance of high-level semanties), studied in Sec. 5.4
4.2 EURNET FOR PROTHIN STRUCTURE MODELING
4.2.1 RELATIONAL EDGES FOR SHORT-, MEDIUM- AND LONG-RANGE INTERACTIONS
In this work, we consider the alpha carbon (i.e., Ca) graph as the representation of protein structure, which is informative and light- summary of the overall protein 3D structure and is widely used in the literature (Gligonjeevć et al. 202T Baldassarre et al. 202T: Zhang el al. 2022) (see Appendix E for preliminary introduction to protein structure). In specific, we extract all Cas as the node set V of our graph, which, at this time, is actually a set of separate points in the 3D space, since there is no chemical bond among Cas. describe the multi-range spatial interactions within a protein, we build following relational edges (see Fig. for graphical illustration):
Edges for short-range interactions ([Rshort = 6). We adopt two kinds of short-range edges proposed by Zhang et al. (2022). (1) Sequential edges con- nect the Ca nodes that are within the distance of 2 on the protein sequence, where each of the sequen- tial distances (-2,-1.0,1,2) is regarded as a single re- lation (i.e, 5 relations in total). (2) Radius edges con- nect the Ca nodes within the Euclidean distance of 10 angstroms, and all radius edges have the same relation.
apta caebon (exparatia
eirtsal protei (soquential tiet.
shorecang wlgs
mediann-cnat olaz (5 ccures range)
(acidon dei ()
medimm-asge sdge
lorgeango edgo
megg))
whole peoica))
Edges for medium-range interactions ( Rmeddi) 2). To capture medium-range interactions exclusively, for each Co node, we first filter out all its neighbors within the sequential distance of or within the Eu- clidean distance of 10 angstroms. We then connect with the remaining nodes that are 5 nearest and 5~100 nearest to (measured by Euclidean distance), and the connections with these two sets of neighbors are re- garded as two different relations.
Edges for long-range interactions (RRong = 1). To capture the interactions beyond short- and medium- range interactions above, we introduce virtual node representing the whole protein by taking global aver- age pooling over all Ca representations, and this virtual node is linked to all Ca nodes with single relation. These edges make each Ca aware of the status of all other Cas, and thus the long-range interactions beyond short and medium ranges can be captured.
Figure 3: Multi-range relational edges for protein Cas. Abbr., dist.: distance.
We gather all these edges with 9 different relations into the edge set È and the relation set R, which, together with V, derive the full graph g = (V, &, R) for multi-rclational protein structure modeling.
4.2.2 MODEL ARCHITECTURE
This work focuses on comparing the graph construction and message passing schemes of EurNet against the SoTA GearNet (Zhang et al., 2022), and we thus follow its single-stage model archi- tecture for fair comparison. Specifically, EurNet performs graph construction once before this only modeling stage, and the input node feature is the one-hot encoding of each Ca's corresponding amino acid. Upon these inputs, six GRMP layers (Sec. 3.3) are stacked for relational modeling- After each layer, the sum pooling over all Ca representations is deemed as the whole-protein rep- resentation, and these per-layer protein representations are concatenated to produce the final output. Upon this output, EurNet performs downstream task by appending task-specific prediction head. We leave the design of the protein structure encoder with multiple modeling stages as a future work.
Note that in EurNet, all graph construction and message passing operations rely only the quanti- ties (e.g., sequential and Euclidean distance) that are invariant to translation, rotation and reflection. Therefore, EurNet satisfies -i3)-invae (Mumford al. 1994).

---

Under review as a conference paper at ICLR 2023
5
EXPERIMENTS
5.1
PERFORMANCE COMPARISON ON IMAGE MODELING
5.1.1 BASELINE METHOSS
We do point-by-point comparisons between our EurNet series, the SoTA ConvNeXt Liu al. and FocalNet Yang al. 2022) series, and other standard series including Swin Transformer (Liu et al. 2021b), FocalAtt (Yang et al. 2021) and VIG (Han et al., (022). For completeness, we also report the results of EffNet (Tan & Le (019), Eff Tan & Le 202T), ViT (Dosovitskiy al. 2020), DeiT (Touvron et al. 2021b). PVT (Wang et al. 2021), Mixer (Tolstikhin et al. 2021 gMLP (Liu et al.j and ResMLP (Touvron et al. 202 a) in applicable cases.
2022)
5.1.2 IMAGE CLASSIFICATION ON IMAGENET-I 1K
Setups. this set of experiments, we benchmark the classification performance of different back- bones on ImageNet- IK (Deng al., 2009) (with 1.28M training and 50K validation images from 1,000 classes)
Table I: ImageNet- IK classification results. We measure throughput V100 GPU. t denotes the model pre-trained on ImageNet-22K. 224
and 384 denote the image size. "1384" means
terms of top-1 accuracy. We consider both (1) training ImageNet-1K -K from scratch and (2) pretraining on ImageNett222K fol- lowed by ImageNet-1K fine-tuning. For fair com- parison, we follow the standard training configu- rations of Swin Transformer (Liu et al. 2021b) with minor changes. Detailed model and training configurations are stated in Appendix F.T
fine-tuning on 384x384 images for 30 epochs.
Model
Parrmms.
(M)
FLOPs
(G)
Throughpat
(ingshs)
Top-1 (e)
ImageNNtt
ERNes-B7
ERNeev2-LL
120
53.0
84
85.7
VII-S/16
4.5
939
79.9
VIF
87
17.5
330
$1.8
DeiT:S/16
22
4.6
979
79.8
DeiT-SHi S/16
87
17.6
302
8L.8
PVT-Smal
704
Results. In Tab. M for training on ImageNet-IK from scratch, EurNets outperform or align pre- vious SoTA baselines on 224 image size, i.e.
PVT-large
Mixer B/16
12.7
455
76.4
gMLF-S
785
79.6
sMLP-B
75
15.8
301
81.6
EurNet-T v.s. FocalNettLRry- 82.3% v.s. 82.3%;
P-S34
P-S34
30
871
79
ResMLP B24
129
23.0
$1.0
EurNet-S v.s. F 83.6% v.s. 83.5%;
Swin-T
EurNet-B vss FocalNet(LRF--BB 84.1 v.s. 83.9%. Following Swin Transformer, we lift the resolu- tion to 384 for 30 epochs fine-tuning after train- ing EurNet-B for 300 epochs the 224 resolu- tion, this model gains 85.4% top-1 accuracy, out- performing ConvNeXt-B. These results demon- strate the effectiveness of EurNets on modeling images with diffferent resolutions.
FocalAul-
319
FocalNeca
696
82.3
Com-NeXi-T
29
4,5
775
82.
EmnNee- I
29
4.6
530
82.3
Swin-S
435
83.1
Pyramid VIG-M
FocalAn-S
192
F
Com-NeXi-
447
EarNer-S
50
314
83.6
Swin-B
88
15.4
291
K3.5
Pyramad VIG-B
93
16.8
83.7
For ImageNet-22K pre-training, EurNet-B is on par with the previous SoTA ConvNeXt-B and clearly outperforms Swin-B. The pre-training on large scale widely regarded as the strength of the models with few inductive biases like Swin- B; while our results show that the well-designed EurNet-B with more inductive biases could also be effective, aligning with ConvNeXt's finding.
FocalNet,an-
FocalAnt B
16.4
269
138
83.8
ConcNoXt-B
FarNar-B
Swin-B 384
89
45.0
96
85.
BurNac-B B 354
90
66.6
69
85.4
Pre-train Immg & Fiue-tune Imngs-Nt-1K
Swin-81 (2267)
FocalNetsoer
Com:NeXt-BB Bt (224")
15.4
292
85.8
EurNer-B Bi (22IP)
89
15.6
224
65.7
Swin-B: (386+)
47.0
85
85.4
Throughput analysis. The throughput of EurNet higher than AocalAtt while lower than FocalNet and ConvNeXt. We point out that 2D convolu- tions (i.e., the core FocalNet and ConvNeXt) are well supported by CUDA kernels, while such supports are still ongoing for graph operations (Chen et al. 2020 Min et al. 2021). EurNet's further speedup is expected under maturer CUDA supports.
FoealNet, (3884)
Com-NeXt-Rt (384")
86.5
EaiNee--B BF 13844")
46.6
69
87.0
5.1.3 OBJECT DETECTION ON COCO
Setups. This experiment benchmarks the object detection and instance segmentation performance on COCO 2017 (Lin et al. 2014). All models trained on 1 118K training images and evaluated on

---

Under review as a conference paper at ICLR 2023
Table 2: COCO object detection and instance segmentation results with Mask R-CNN (He
2017
Model
#Params, (M)
ILOPs
AP"
Al'a0
Mask
A.
R-CNN Ix
A."
AlP
AI Tm
AIP
AlP b0
Mask R-CNN 3x
APPs
AP"
A.P
PVT-Small
44.1
245
40,4
62.9
43.8
57.8
61.1
40.3
43.0
65.3
46.9
39,9
62.5
42.8
Swin-T
66.6
68.1
FocalAn-T
48.8
291
44.8
67.7
49.2
41.0
64.7
44.2
47.2
69.4
51,9
42.7
66.5
45.9
FocalNeta
48.9
268
46.1
68.2
50.6
41.5
65.1
44.5
48.0
69.7
53.0
42,9
66.5
46.1
PurNet-T
68.7
50.5
44.6
47.8
69.5
52.3
42.9
PVT-Medium
42.1
44.2
48.2
Swin-S
69.1
354
46.5
68.7
51.3
42.1
65.8
45.2
48.5
70.2
53.5
43.3
67.3
46.6
FocalAn-S
71.2
401
47.4
69..8
51.9
42.8
66.6
46.1
48.8
71.5
53.6
43.8
67.7
47.2
FocalNettary-
70.5
19.3
BurNet-S
72.8
48.4
70.5
53.2
43.22
67.4
46.2
49.4
70.7
54.5
44.0
67.6
PVT-Large
81.0
364
42.9
65.0
46.6
39.5
61.9
42.5
44.5
66.0
48.3
40.7
63.4
43.7
Swin-B
107.1
497
46.9
69.2
51.6
42.3
66.0
45.5
48.5
69.8
53.2
43.4
66.8
FocalAn-B
110.0
70.2
52.5
49.0
70.1
53.6
FocalNeti Ko-
1114
507
49.0
70.9
53.9
43.5
67.9
46.7
49.8
70.9
$4.6
44.1
68.2
47.2
PuiNet-R
112.1
506
49.1
71.8
54.0
43.9
68.2
47.2
50.1
71.5
55.1
44.5
68.7
47.8
5K validation images. Two standard training schedules, i.e., the 1x schedule with 12 epochs and the 3x schedule with 36 epochs, are used for benchmarking. Detailed setups are stated in Appendix [F2
Results. In Tab. EurNet performs comparably to FocalNetQRRF on the tiny and small model scales. We can observe the superiority of EurNet-B over F on the base model scale (better performance all 12 metrics). The base-scale EurNet-B owns [2, 2, 18, 2] modeling blocks (more than EurNet-T) and [128, 256, 512, 1024] feature channels (more than EurNet-S) for four modeling stages. Therefore, larger message passing hops (achieved by more modeling blocks) coupled with larger model width favor EurNet's performance on high-resolution dense prediction tasks.
5.1.4 SEMANTIC SEGMENTATION ON ADE20K
Table 3: ADE20K semantic segmentation results with UperNet (Xiao al., 2018).
Setups. In this experiment, we benchmark the seman- tic segmentation performance of different backbones on ADE20K (Zhou et al.) (017) which contains 20K train- ing, 2K validation and 3K test images. The mloU met- rics under both single- and multi-scale (MS) evaluation are reported. We provide more details in Appendix [+3
Model
aParams. (M)
FL.0Ps (G)
mlol
-MS
Swin T
9:1
44.5
45.8
FacalAm-T Com-NeXi-T
998
45.8
47.0
60
939
46.7
FocalNetaass-T
61
949
46.8
47.8
BurNet-T
9:8
47.2
48.4
40.5
FocalAn
ConNNeXiS
1027
49.6
Results. Tab. 3 reports all results. It can be observed that EurNet-T, EurNet-S and EurNet-B achieve the best performance on their corresponding model scales under both evaluation metrics. Such consistent performance gains verify the effectiveness of EurNet on the dense prediction tasks that require to model fine-grained semantics and long-range interactions.
FocalNetamsy FarNer-S
1044
49.1
50.1
85
1042
49.8
50.8
Swin B
121
1188
48.1
49.7
FacalAm-a Com-NeXi-B
126
1354
49.0
50.5
122
1170
49.9
FocalNetaasy.B
BurNet
5.2
PERFORMANCE COMPARISON ON PROTEIN STRUCTURE MODELING
5.2.1 BASELINE METHODS
Table 4: F Fmax results on EC and GO pro-
tein function prediction benchmarks.
We compare with the SoTA GearNet (Zhang et al. 2017) under two settings, i.e., with and without edge message passing ("-Edge" in Tab. 4.. We also include other baselines, Le.. 3DCNN MQA (Deeevyanko et al.
Model
EC
GO-BP
GO-MF
GO-CC
3DCNN MQA
0.077
0.147
GCN
0.320
0.252
0.195
0.329
GAT
2018), GCN (Kipf & Welling 2016), GAT' (Veličković
GVP
0.489
0.326
0.426
0.420
GraphQA
et al. 2017), GVP (Jing et al. (202T), GraphQA (Bal- dassarre et al.) 2021) and New IECony (Hermosilla & Ropinski 2022), for complete comparisons.
New IFConv
0.735
0.374
0.544
0.414
GearNet
0.503
EurNer
0.768
0.437
0.563
0.421
GearNet--dge Edge
0.810
0.403
0.580
0.450
EuiNet-Edge
0.829
0.456
0.592
0.453
5.2.2 PROTEIN FUNCTION PREDICTION
Setups. This set of experiments compare different protein structure encoders on the EC (Gligorijević
et al. 2021) and GO (Gligorijevć al. 2021) protein function prediction benchmarks. We follow Gear to report the protein-cente centric maximum F-score F Fmax- a commonly metric in CAFA challenges (Radivojac et al. 2013). More dataset, model and training details are in Appendix F.4
8

---

Under review as a conference paper at ICLR 2023
modeling saage
modeling stage
modeling stage
modeling stage
Figure 4: Modium-range edges built by EurNet-T (we use different colors different selected target nodes).
Results. In Tab. we can observe that EurNet consistently outperforms GearNet on all four tasks, and the performance gains preserve after involving edge message passing (details of edge message passing are stated in Appendix F.4). Since EurNet follows the single-stage model architecture of GearNet, we can conclude the effectiveness of medium- and range interaction modeling and GRMP-based multi-relational modeling, which are novel modeling mechanisms in EurNet.
5.3 ABLATION STUDY
Table Ablation study of multi-range edges on ImageNet- K with EurNet-T.
Effect of multi-range relational edges. In Tab. we evalu- ate EurNet-T ImageNet- 1K with different ranges of edges. When using single range, the model with long-range edges achieves the highest accuracy 81.7%, which verifies the im- portance of capturing long-range interactions in image classi- fication. By further adding short- or medium range edges, the performance is promoted to 82.0%, where more fine- ine-ggained local interactions are captured. By using all three ranges of edges, the full model of EurNet-T obtains the 82.3% accuracy, which proves the complementarity of short-, medium- and long-range edges. Ablation study for protein structure is in Appendix H.2
short
medium
long
Top-1 Acc (%)
80.7
79.3
81.7
81.5
82.0
82.0
82.3
Effect of GRMP layer. In Tab. 6 we compare hetween RGConv and GRMP under the comparable parameter number, FLOPs and throughput. (1) GRMP's di- mensions are first as (96, 192, 384, 768] in four stages. To reach comparable cost, RGConv can only have the dimensions of
Table 6: Ablation study of multi-relational modeling layer ImageNet- with EurNet-T.
Layer
Hidden Dimensions
#Params.
FLOPS
Throghput (ingsis
Top-1
(MD
(G)
Acc (%)
884
GRMP
96. 192 384, 768
28.8
4.6
538.3
82.3
RGCoBv
966 192, 384, 709
37.3
5.9
451.2
82.2
1108, 216, 402, 864
[84, 168, 336, 672] and achieves a lower accuracy 81.5% than GRMP's 82.3%. (2) After increasing RGConv's dimensions [96, 192, 384, 768). it aligns GRMP's performance while introduces more cost (1.3G more FLOPs). Under comparable cost, GRMP can have [108, 216, 432, 864) dimensions, leading to a higher accuracy 82.7%. These results demonstrate the better efficiency-perfommanc trade-off gained by GRMP. Ablation study for protein structure modeling is in Appendix H.3
5.4 VISUALIZATION
Fig. Adisplays some medium-range edges built by the EurNet-T trained on ImageNet- IK. The edges for the 2nd stage connect the patches with similar low-level features (e,g., the patches of red dog ears in Fig. aa). while the edges for the 4th stage connect semantically relevant patches (e.g.. different body parts of two dogs in Fig. (a)), which shows EurNet-T's hierarchical image modeling ability.
CONCLUSIONS AND FUTURE WORK
This work proposes the EurNet to model spatial multi-relationa data like image patches and protein alpha carbons. It builds relational edges on multiple spatial ranges describe the interactions in the data. It uses the gated relational message passing layer to model the built multi-relational graph, which can efficiently adapt to large data and model scales. The instantiations of EurNet have gained superior performance various image and protein structure modeling tasks.
In future works, will adapt EurNet more tasks of other domains like 3D point cloud model- ing for object and scene understanding, and we will explore a general hierarchical modeling method for the data from various domains.

---

Under review as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
For the sake of reproducibility, we use Tab. to provide detailed architectures of EurNet-T, EurNet-S and EurNet-B for image modeling, state the detailed single-stage architecture EurNet for protein structure modeling in Sec. 4.2.2 and describe the model configurations for each specific task in Sec. We state the detailed training configurations of all considered tasks in Sec. F The derivation of the FLOPs of RGConv (Eq. (2) and GRMP (Eq. 4 are provided in Sec. A In the supplementary material, we submit all source code for reproducing the results ImageNet classification experi- ments, and the source code for COCO object detection, ADE20K semantic segmentation and EC and GO protein function prediction experiments will released public upon acceptance.
REFERENCES
László Babai and Ludik Kucera. Canonical labelling of graphs in linear average time. In Annual Symposium Foundations of Computer Science, 1979.
Minkyung Baek, Frank DiMaio, Ivan Anishchenko, Justas Dauparas, Sergey Ovchinnikov, Gyu Rie Lee. Jue Wang. Qian Cong. Lisa N Kinch, R Dustin Schaeffer, et al. Accurate prediction of protein structures and interactions using three-track neural network. Science, 373(6557):871- 876, 2021.
Muhammet Balcilar, Guillaume Renton, Pierre Héroux, Benoit Gaüzère, Sébastien Adam, and Paul Honeine. Spectral-designed depthwise separable graph neural networks. ICML Workshop on Graph Representation Learning and Beyond (GRL+ 2020), 2020.
Federico Baldassarre, David Menéndez Hurtado, Arne Elofsson, and Hossein Azizpour. Graphqa: protein model quality assessment using graph convolutional networks. Bioinformatics, 37(3): 360-366, 2021.
Helen Berman, John Westbrook, Zukang Feng, Gary Gilliland, Talapady Bhat, Helge Weissig, Ilya N Shindyalov, and Philip E Bourne. The protein data bank. Nucleic Acids Reseurch, 28(1):
235-242, 2000.
Antoine Bordes, Nicolas Usumier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. Translating embeddings for modeling multi-relational data. Advances in Neural Information Pro- cessing Systems, 2013.
Dan Busbridge. Dane Sherburn, Pietro Cavallo, and Nils Y Hammerla. Relational graph attention networks. arXiv preprint arXiv:1904.000 1904/994411111 2019.
Jin-Yi Yi Cai, Martin Fürer, and Neil Immerman. An optimal lower bound on the number of variables for graph identification. Combinatorica, 12(4):389 410, 1992.
Kai Chen, Jiaqi Wang. Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, et al. Mmdetection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv: met.e090600111 07..0, 2019a.
Yunpeng Chen, Mareus Rohrbach, Zhicheng Yan, Yan Shuicheng, Jiashi Feng, and Yannis Kalan- tidis. Graph-based global reasoning networks. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019b.
Zhaodong Chen, Mingyu Yan, Maohua Zhu, Lei Deng, Guoqi Li, Shuangchen Li, and Yuan Xie. fusegnn: accelerating graph convolutional neural network training gpgpu. In IEEF/ACM In- ternational Conference On Computer Aided Design, 2020.
MMSegmentation Contributors. Mmsegmentation:: Openmmlab semantic segmentation toolbox and benchmark. Availahe online: http:://github. comdoppenmmmncchhmmeeeennnuuuai (accessed on 18 May 2022), 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier- archical image database. In JEEE/CVF Conference on Computer Vision and Pattern Recognition, 2009.
10

---

Under review as a conference paper at ICLR 2023
Georgy Derevyanko, Sergei Grudinin, Yoshua Bengio, and Guillaume Lamoureux. Deep convolu- tional networks for quality assessment of protein folds. Bioinformatics, 34(23):4046 40066-4033,0000 2018.
Tim Dettmers, Pasquale Minervini, Pontus Stenetorp. and Sebastian Riedel. Convolutional 2d knowledge graph embeddings. In AAAI Conference Artificial Intelligence, 2018.
Xiaohan Ding, Xiangyu Zhang, Jungong Han, and Guiguang Ding. Scaling up your kernels to 31x31: Revisiting large kernel design enns. In /EEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. / 1929, 2020.
Pablo Gainza, Freyr Sverrisson, Frederico Monti, Emanuele Rodola, D Boscaini, MM Bronstein, and BE Correia. Deciphering interaction fingerprints from protein molecular surfaces using geo- metric deep learning. Nature Methods, 17(2):184-19) 192.844-9922000 2020.
Vladimir Gligorijević, P Douglas Renfrew, Tomasz Kosciolek, Julia Koehler Leman, Daniel Beren- berg, Tommi Vatanen, Chris Chandler, Bryn C Taylor, Ian M Fisk, Hera Vlamakis, et al. Structure- based protein function prediction using graph convolutional networks. Nature Communications, 12(1):1-14, 2021.
Kai Han, Yunhe Wang, Jianyuan Guo, Yehui Tang, and Enhua Wu. Vision gnn: An image is worth graph of nodes. arXiv preprint 2-Xx:2226600027., 2022.
Frank Harary and Robert Z Norman. Some properties of line digraphs. Rendiconti del circolo matematico di palermo, 161-116,,, 1960.
Michael Harms and Joseph W Thornton. Analyzing protein structure and function using ancestral gene reconstruction. Current Opinion in Structural Biology, 2003):360-366, 2010.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. IEEF/CVF Conference on Computer Vision and Pattern Recognition, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In IEEE/CVF Inter- national Conference on Computer Vision, 2017.
Pedro Hermosilla and Timo Ropinski. Contrastive representation learning for 3d protein structures. arXiv preprint arXiv:2205. /5675, 2022.
Pedro Hermosilla, Marco Schäfer, Matěj Lang, Gloria Fackelmann, Pere Pau Vázquez, Barbora Ko- zliková, Michael Krone, Tobias Ritschel, and Timo Ropinski. Intrinsic-extrinsic convolution and pooling for learning on 3d protein structures. International Conference on Learning Representa- fions, 2021.
Bowen Jing, Stephan Eismann, Pratham N. Soni, and Ron Dror. Learning from protein structure with geometric vector perceptrons. In International Conference Learning Representations, 2021.
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Zidek, Anna Potapenko, al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583-588) 2021.
Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional net-
works. arXiv preprint arXiv: 1609.0297, 2016.
Jing Li, Lingling Zhang, Fan Meng, and Fenhua Li. Recommendation algorithm based link prediction and domain knowledge in retail transactions. Procedia Computer Science, 31:875- 881, 2014.
Qimai Li, Xiaotong Zhang, Han Liu, Quanyu Dai, and Xiao-Ming Wu. Dimensionwise separable 2- graph convolution for unsupervised and semi-supervised learning on graphs. In ACM SIGKDD Conference Knowledge Discovery Data Mining, 2021.

---

Under review as a conference paper at ICLR 2023
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference Computer Vision, 2014.
Tsung-Yi Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In IEEE/CVE Conference on Computer Vision and Pattem Recognition, 2017.
Hanxiao Liu, Zihang Dai, David So, and Quoe V Le. Pay attention to mlps. Advances in Neural Information Processing Systems, 2021a.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In JEEF/CVF Interna- tional Conference on Computer Vision, 2021b.
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05100 2017.
Seung Won Min, Kun Wu, Sitao Huang, Mert Hidayetoğlu, Jinjun Xiong, Eiman Ebrahimi, Deming Chen, and Wen-mei Hwu. Large graph convolutional network training with gpu-oriented data communication architecture. arXiv preprint arXiv:2103.033: 2021.
Christopher Morris, Martin Ritzert, Matthias Fey, William L Hamilton, Jan Eric Lenssen, Gaurav Rattan, and Martin Grohe Weisfeiler and leman go neural: Higher-order graph neural networks. In AAAI Conference on Artificial Intelligence, 2019.
David Mumford, John Fogarty, and Frances Kirwan. Geometric invariant theory, volume 34. Springer Science & Business Media, 1994.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic diffcrentiation in pytorch. In NeurlPS Workshop, 2017.
Predrag Radivojac, Wyatt T Clark, Tal Ronnen Oron, Alexandra M Schnoes, Tobias Wittkop, Artem Sokolov, Kiley Graim, Christopher Funk, Karin Verspoor, Asa Ben-Hur, et al. A large-scale evaluation of computational protein function prediction. Nature methods, 10(3):221-227, 2013.
Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne van den Berg, Ivan Titov, and Max Welling. Modeling relational data with graph convolutional networks. European Semantic Web Conference, 2018
Zhiqing Sun, Zhi-Hong Deng, Jian-Yun Nie, and Jian Tang. Rotate: Knowledge graph embedding by relational rotation complex space, arXiv preprint arXiv: 1902:19922.0197,, 2019.
Freyr Sverrisson, Jean Feydy, Bruno E Correia, and Michael M Bronstein. Fast end-to-end learning on protein surfaces. In IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp- 15272-15281 2021.
Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural net- works. In International Conference on Machine Learning, 2019.
Mingxing Tan and Quoc Le. Efficientnetv2: Smaller models and faster training. In International
Conference on Machine Learning, 2021.
Ilya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Un- terthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al. Mip-mixer: An all-mip architecture for vision. Advances in Neural Information Processing Systems, 2021.
Kristina Toutanova and Danqi Chen. Observed versus latent features for knowledge base and text inference. In Proceedings of the 3rd workshop on continuous vector space models and their compositionality, pp. 57-66, 2015.
12

---

Under review as a conference paper at ICLR 2023
Hugo Touvron, Piotr Bojanowski, Mathilde Caron, Matthieu Cord, Alaaeldin El-Nouby, Edouard Grave, Gautier Izacard, Armand Joulin, Gabriel Synnaeve, Jakob Verbeek, et al. Resmlp: Feedforward networks for image classification with data-efficient training. arXiv preprint arXiv:2105.03404, 2021a.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning, 2021b.
Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier, and Guillaume Bouchard. Com- plex embeddings for simple link prediction. In International Conference on Machine Learning, 2016.
Shikhar Vashishth, Soumya Sanyal, Vikram Nitin, and Partha Talukdar. Composition based multi- relational graph convolutional networks. arXiv preprint 19100099 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Informa- fion Processing Systems, 2017.
Petar Veličkovié, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arXiv: 10903, 2017.
Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In IEEE/CVF International Conference on Computer Vision, 2021.
Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In European Conference on Computer Vision, 2018.
Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Deng. Embedding entities and relations for learning and inference in knowledge bases. arXiv preprint arXiv: 2014.
Jianwei Yang, Chunyuan Li, Pengchuan Zhang, Xiyang Dai, Bin Xiao, Lu Yuan, and Jianfeng Gao. Focal attention for long-ran range interactions in vision transformers. Advances in Neural Information Processing Systems, 2021.
Jianwei Yang, Chunyuan Li, and Jianfeng Gao. 29X. 2022.
Focal modulation networks. arXiv preprint
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In IEEF/CVF International Conference on Computer Vision, 2019.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint 17X1v:171 2017.
Li Zhang, Xiangtai Li, Anurag Amab, Kuiyuan Yang, Yunhai Tong, and Philip HS Torr. Dual graph convolutional network for semantic segmentation. arXiv preprint arXiv: 190.::/9900062/// 2019.
Li Zhang. Dan Xu, Anurag Arnab, and Philip HS Torr. Dynamic graph message passing networks. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
Zuobai Zhang, Minghao Xu, Arian Jamasb, Vijil Chenthamarakshan, Aurelie Lozano, Payel Das, and Jian Tang. Protein representation learning by geometric structure pretraining. arXiv preprint 2r3iv:2203.06/25, 2022.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. IEEF/CVE Conference on Computer Vision and Pattern Recognition, 2017.
Zhaocheng Zhu, Zuobai Zhang, Lasisl Xhonneux, and Jian Tang. Neural bellman-for ford net- works: general graph neural network framework for link prediction. Advances in Neural Infor- mation Processing Systems, 2021.
13

---

Under review as a conference paper at ICLR 2023
Zhaocheng Zhu, Chence Shi, Zuobai Zhang, Shengchao Liu, Minghao Xu, Xinyu Yuan, Yangtian Zhang, Junkun Chen, Huiyu Cai, Jiarui Lu, et al. Torchdrug: powerful and flexible machine learning platform for drug discovery. arXiv preprint arXiv:2202.08.22) 2022.
14

---

Under review as a conference paper at ICLR 2023
A FLOPS OF RGCONV AND GRMP
For FLOPs computation, we consider the multi-relational graph 9 = & R) with node set V. edge set & and relation (iee,, edge type) set R, and both input and output node features with C feature channels. In addition, we assume that, when introducing new relation, the in-degree of each node will increase by d average.
Proposition 1. To process the assumed multi-relational graph, the Relational Graph Convolution (RGConv) consumes the FLOPs as below under the efficient implementation with sparse matrix multiplication:
FRGPO)) = (R] (2d + + IVCC
Proof. We divide the computation RGConv into three steps and compute the FLOPs of each step:
① In the first step, the adjacency of all node pairs on [ different relations are summarized in the adjacency matrix A IR [ (RID)) where the element A, (i) indicates the weight of the edge from the i-th node to the j-th node with the k-th relation:
N (155 there is an edge from i-th node to j-th node with k-thh relation,
ALG--1)RR(A
otherwise,
(5)
where Nr, (V²) {ul rk) E E) is the neighborhood set of node vy with relation Tk- Using this adjacency matrix, each node will have [R] different slots to receive the relational messages passed to All relational message passing operations can be realized by sparse matrix multiplication:
A
(6)
where Z E R denotes input node features, and Ž R R denotes the relational slots of all nodes after message passing. By utilizing the sparsity of the adjacency matrix, this step consumes following FLOPs:
FLOPs(RGCOIV- = 2| R|C
In the second step, first integrate the relational slots of each node to get the reshaped Ż R At this time, each node is represented by -dimeennis vector, i.e., the aggregated messages all relations. Next, we concatenate the convolutional kernel matrices all relations to produce Wroow E R R|CxC and this matrix is applied upon Ž to combine the messages in the same relational slot and aggregate messages across different relations:
ggg
ZWwcony conv
where Z E X denotes the aggregated neighborhood information for each node. This step has the FLOPs as below:
FLOPs(RGCOnv--
2/R/IVIC2.
3 the final step. a self-update with matrix W/eeff E CCC is first performed on the input feature of each node, and the self updated node feature is further added with the aggregated neighborhood information:
z'= Wele + aggrr
(10)
where Z' E IVIX denotes output node features. This step has the FLOPs as below:
FLOPs(RGCony - ) = 2 + IV|C.
(11)
Therefore, by summing up the computational cost of three steps, the RGConv consumes the follow- ing FLOPs in total:
FLOPs(RGConv) = |R| .2d + 22D((9 + + + |IC.
15

---

Under review as a conference paper at ICLR 2023
Proposition 2. To process the assumed mulii-relationn graph. the Gated Relational Message Pass- ing (GRMP) consumes the FLOPs as below under the efficient implementation with sparse matrix multiplication:
FLOP)(GRMP)) = (R] (2d+7)/VIc + 6/VIC
Proof. Following the steps of GRMP stated in Eq. we compute the FLOPs of each step:
① In the first step, we conduct a pre-layer node-wise channel aggregation with the weight matrix Win E RCxC;
z = ZW
(12)
where Z IR MI denotes the input node features, and n E R denotes the channel- aggregated node features. This step has the FLOPs consumption as below:
FLOPs(GRMP 1 2/VIC²
(13)
In the second step, we first gather the messages within the same relation for each node, which is realized by the sparse matrix multiplication between Z and the adjacency matrix R [ (A is identically defined as in the step ① of Proposition
2'n
2
(14)
where 7 E R [R]IVIXC represents the relational slots of all nodes after message passing. The relational slots of each node are then integrated to get the reshaped Zin E R V|x|R|C By concatenating the convolutional kernel vectors of all relations, we have Wennv E R|R[Cx1, and this vector is broadcast to all nodes to perform channel-wise message aggregation via Hadamard product:
7 (1conv ) Zin
(15)
where 1cony E R | is the all-one vector for broadcasting, and Ž E B denotes the relational slots of all nodes after intra-relation message aggregation.
To conduct the operations in Eqs. 14 and (5). this step consumes the following FLOPs:
FLOPs(GRMP 2 2 C // + 2
(16)
3 the third step, we first compute the attentive weights assigned to all relations on each node: M ZW
(17)
where Wa RCx|R| is the weight matrix for node-adaptive relation weighting, and M ||] denotes the relation weights on nodes. After that, weighted summation is per- formed aggregate the messages of different relations in Z (in this operation, we use the reshaped 2 ///X||R|XC and the reshaped M
R [|X/R|]X)
[R|
BARR Σ
(M.. 1
AARR :3.
(18)
where 1, E R Cx1 the all-one vector for broadcasting relation weights to all feature chan- nels, and Z E BMV denotes the per-node neighborhood representations after inter- relation message aggregation.
To perform Eqs. I7 and (18), this step has the following FLOPs consumption:
FLOPs(GRMP
2/ [R] 2|V (|R) - 1)] 5/R|MCC- .VIC (19)
The fourth step conducts a post-layer node-wise channel aggregation with the weight matrix Wrout R CxC.
Z aaggr_
Z prout
(20)
where Zaggr R | denotes the channel-aggregated neighborhood representations. This
step consumes the FLOPs as below:
FLOOPs(GRMP 4 2
(21)
16

---

Under review as a conference paper at ICLR 2023
5 In the final step, the input feature of each node first performs self-update with the weight matrix Woelr E RC&C, and the self-updated node feature is further updated by its neighborhood representation via gating mechanism:
Z' = sell @
(22)
where Z' E M denotes output node features. This step has the FLOPs as below:
FLOPs(GRMP = 2|v|C + IVI.
(23)
Therefore, by summing up the computational cost of five steps, the GRMP has the following FL.0Ps consumption in total:
FLOPs(GRMP) = [R] +
+ 6/vIC
(24)
B ANALYSIS OF MODEL EXPRESSIVITY
In this section, we study the expressivity of the proposed GRMP layer (Sec. 3.3
Specifically,
introduce the variant of the Weisfeiler-Leman (WL) algorithm (Morris al. 2019) on multi- relational graphs and show that there exists parameterization of GRMP that is as expressive as the multi-relational WL algorithm.
Following the philosophy of the I-dimensional Weisfeiler-Leman (1-WL) algorithm (Morris et al, 2019) we define the multi-relational 1-WL (I-RWL) algorithm. This algorithm studies labeled multi-relational graph g = E1 E I), where V is the node set, & denotes the edge set associated with the i-th relation, and / the label function that assigns initial node features. The (())) V N for each iteration / > O, and the initial coloring ((0) is consistent with the label function I (i.e., one unique color for the nodes with specific label). For iteration t > 0, 1-RWL updates the color of each node u E V based on the colors of itself and its neighbors with different relations in the last iteration:
1-RWI. computes node coloring
ct = HASH ((C(t))(w),
(w), E u Ni(o)B)). Ve E V,
(25)
where [| {L, ... [ denotes the indices of all relations, Ni(v) is the neighborhood set of node e' with the i-th relation, H...} denotes a multiset. To test the isomorphism of two multi- relational graphs G and g', the I-RWL algorithm run in parallel on both graphs. If the number of nodes assigned with specific color is different across two graphs at an iteration, it is concluded that and G' are non-isomorphic. The algorithm terminates when the color assignments do not change across two iterations, which is reached after at most max [V], [VD iterations (V and V are the node sets of two graphs). Just as the 1-WL test (Cai et al. 1992). the same color assignments along the whole process of 1-RWL cannot guarantee the isomorphism of two graphs, while it is still powerful heuristic for (1) distinguishing the nodes with different structural roles in a multi-relational graph and (2) distinguishing non-isomorphic multi-relational graphs (Babai & Kucera 1979).
We next compare the expressivity between the GRMP layer and the 1-RWL algorithm. For multi- relational graph G, we denote F E x as the node feature matrix and A, E R| as the adjaceney matrix for the i-th relation (i [[]]). The node feature update rule of GRMP can be written as below:
F' (FW self b o ((Ene (Eetre a;A, + bjn)) W out + bout
(26)
where W sell. Win and W oot are the parameter matrices (i.e., weights), baelf, bjn and bout are the
parameter vectors (i.e., biases), and a; [[R|]] are the per-relation scaling factors. Note that, in this analysis, we simplify GRMP's intra- and inter-relation message aggregation (step ② and step 3 in Eq. (3) as per-relation scaling, where the expressivity of this simplified GRMP upper bounded by the original one. Following Morris et al. (2019), consider the model with stack of GRMP
layers and denote the sequence of GRMP parameters GRMP up the t-th layer as below:
((t))
(1t))
(w(r) selfi w(!'), w(O)) outs selfs b bout> ax) (EE
(27)
GRMP
17

---

Under review as a conference paper at ICLR 2023
On such basis, next illustrate there exists parameters W GRMP such that the corresponding model as expressive as the coloring CCt in terms of distinguishing nodes in multi-relational graphs. Theorem 1. Let 9 EE,...,,ERRI be a labeled multi-relational graph. For all > 0, there exists initial node features and sequence GRMP of GRMP parameters such that the following holds:
(v) (w) + F = w
Yu, w E V.
In other words, the node feature matrix F(0) is equivalent to the coloring function (tt) of I-RWL. at all iterations.
Proof. (1) Prerequisites. Following Morris et al. (2019), a matrix is denoted row-independent modulo equality if the set of all different rows in the matrix are linearly independent. For two coloring functions C₁ and C2 of G. we denote their equivalence C₁ C2 if they define the same partition over the node set V. We next prove the result by induction.
(2) Base case. For t = O, we define the initial node features F(0) to be row- row-independent modulo equality and consistent with the label function I (e.g., the one-hot encoding of node labels). Since the initial coloring function satisfies (20) L, we can conclude the equivalence of (1)) and F(") i.e., (()) (v) C( (w) F F
Vu, w E V.
(3) Induction step. For > O, we assume that (Ct) and F(t) are equivalent, and F(t) is row- independent modulo equality. The coloring (L++1) of 1-RWL at iteration 1+1 is derived by applying 1-RWL step to update the coloring ()t);
cll+1 (v) = HASH (c (v), (w). [R]], u E (.(()))). Vu E.
(28)
Let be the number of different colors defined by (+t) and let Qi...., subsets partitioned by C The updated coloring C(++1) can be equivalently represented by the matrix D R [(M(x((((++1) with following entries:
......,Q, Qy be the different node
1 [W((t) n Q₁1
l = iq + for i [[]] j E lg].
Dok
if k E [g] and E Qu.
(29)
otherwise,
in which the corresponding row of node v is the concatenation of an one-hot encoding of v's color and vector encoding for the multisct of the colors in (1)... for each [[]]] Based on the row encoding of nodes in D, we can partition the node set V into subsets (the nodes in the same subset share the same row encoding) and assign unique color to each subset, which defines the coloring function CD. The equivalence CD ((++1) holds.
Since C and ((t) are assumed to be equivalent, there should be distinct rows in F(t), and each of them corresponds to one of q different colors defined by C10)) Let ((() Raxd be the matrix composed of these distinct rows with the order corresponding to Q1...., Qg- By assumption, the rows of (*t) are linearly independent, and thus there is a matrix M e Rdxy such that F(O e Roxg an identity matrix. By extension, the matrix F E R [ has entries:
(F(UM) = {
u Q₁+ otherwise.
(30)
Note that, the matrix D defined in Eq. 22 can be viewed as a block matrix D [Bg B B
where Bg F E NIVixg and B, A, E N Vixy for each [| Since each element of is upper bounded by | - 1, we follow polynomial coding to assign B, (i E [[R]] with the polynomial term [V]* and assign Bo with the term [ defining the following matrix E E Vix which is equivalent to D in terms of partitioning nodes based on the rows matrix:
E R|+1 F(UM
1) O
M (E
A,F M+1).
(31)
where 1 E N is an all-one matrix with fitted shape. The matrix E defines a coloring function CE in the same way the matrix D, and the equivalence CE CD holds.
18

---

Under review as a conference paper at ICLR 2023
By aligning the update rule of GRMP (Eq. 26 with the definition of matrix E (Eq. 31 adopt the parameterization W | Win M. W out l (the identity matrix),
baelf = 1, bm Oi1 (the [V] x q zero matrix), bout 1, and a; = V/.. In this way, the node
feature matrix F(++1) at iteration is updated from F(O) as below:
((E,) iE| Ri]
F(++) F
V ( ((1))) + = E, (32)
Therefore, the coloring function Cp(+1 defined by the updated node feature matrix F satisfies:
CF(+)) CE CD ((++2)
(33)
In particular, we have: (v++ = (w) + F((+1) ((++1) Vu, w E V.
(4) Conclusion. Since both the base and the induction step have been proved to be true, conclude that:
(v)= c(0
F(O)
F(!),
VE 3 0, WU.
(34)
This result proves the equivalent expressivity of the 1-RWL algorithm and the model constructed by simplified GRMP layers (Eq. (26)). Therefore, when constructing the model with standard GRMP layers (Eq. (3) the model is at least expressive as the 1-RWL algorithm.
C GRAPHICAL ILLUSTRATION OF GRMP
Rel.1
step 5
Gated update
Mult. Wsely
step 1
a1
Mult. win
step 2
step Mult. Wout
Intra-rel. aggr.
Rel. [R|
step
a Inter-rel. aggr.
Input node features Z
Transformed inputs Zin
Output node features Z'
Figure 5: Graphical illustration for node representation update in the GRMP layer. We specifically show the neighborhood aggregation and representation update procedure of the node denoted in red. Abbr, Multi.: multiply with; Rel.: relation; aggr.: aggregation.
In Fig. graphically illustrate the mechanism of node representation update in the GRMP layer. In specific, GRMP updates the node representation matrix from Z' with the following steps:
A linear layer transforms the input node representations E R |VXC to Zin E x||x which aggregates the feature channels of each node the beginning of the layer.
For each node, its neighbors assigned different groups according to their relations with the node, and the neighbors in each group are aggregated a channel-wist way.
The aggregated messages different relational groups are then scaled by per-relation sealar weights {a, ) and aggregated to the neighborhood representations Z E R V|xC.
RRR then transformed by a linear layer to aggregate the feature channels of each node's neighbors, deriving the transformed neighborhood representations z V|xc*
Finally, z serves the gate to update all node representations, deriving the output node representations Z' B
19

---

Under review as a conference paper at ICLR 2023
Table 7: Detailed architectures of EurNet-T/S/B for ImageNet-IK IK classification (#parameters and FLOPs are computed under the resolution 224 224). H x W: input image resolution; C: number of feature channels; y: FFN's hidden dimension ratio; K: number of K-nearest neighbors for medium- range edges; J:: label set for classification. "T" denotes the tiny model; "S" denotes the small model;
"B" denotes the base model.
Module
MPatches
TurNet-T
EarNet--
EurNet-B
Stem
4 x
conv, stride=4
cony, stridew4
cony, stride
Graph Construction
# 4
Ishort-range odges.
short-rangg edges.
sbort-range edges.
long-range edges
long-range edges
long-range odges
GRMP (C (C=96),
GRMP (C'=66),
GRMP (C -22)). FFN (C=112,, 7= 4)
Stuge 1
4 T
FEN (C (C=9K, 4=4)
FFN (0=96, 4)
x2
Downssample
PatchMerging
PatchMergingg
PatchMerging
short-range odges,
short-range odges,
sbort-range edges.
Graph Construction
#/4 x
medium-range edges (K = 12),
medium-range edges (K = 12).
mediam-range edges (K = 12).
long-range edges
long-range edges
long-range odges
Stage 2
#/4 x
FUN 19)) x2
GRMP (C 1922)
GRMP (C= 192).
x2
GRMP (C (C=256),
x2
(C2=12,
FFN (C=256, 7=
Downsample
1/ x/
PatchMerging
PutchMerging
PatchMerging
Graph Construction
21 36 18
medium-range edges (K = 12).
short-range odges.
medium-range edges (K = 12).
short-raage edges,
mediumm-rang edges (K = 12).
sbort-range edges.
long-range odges
long-range edges
long-range odges
Stage 3
# w 30 15
FFN (C 7-4)
GRMP 384),
XG
FTN (C'= 384,
GRMP 384)
x18
FFN (0=5)
GRMP 512).
x18
Downsample
3 x 85
PatchMerging
PatchMorging
PatchMerging
short-range edges,
short-range edges,
sbort-range edges.
Graph Construction
! w
meddium-range edges (K = 12),
medium-range edges (K = 12).
medium--range edges (K 12).
long-rango edges
long-range edges
long-range edges
Stage 4
#
W.
FFN (C' (C'-768, 4)
GRMP 768),
x2
FFN (C' - 768, 4)
GRMP (68768).
x2
FFN (C) - 1024, 7-4)
GRMP 10240244,
x2
Head
1x1
Pooling & Lincar (()) - 1000)
Pooling & Lincar ((>) - 1000)
Pooling & Linear (D) = 1000)
#Parameters (M)
28.8
50.2
88.7
FLOPS (G)
4.6
D DETAILED MODEL ARCHITECTURE FOR IMAGE MODELING
For image modeling, we basically follow the hicrarchical architecture proposed by Swin Trans- former (Liu et al. 2021b), as summarized in Tab. 7 The architecture begins with a patch embedding module implemented by non-overlapping 2D convolution. After that, the model is split into 4 mod- eling stages: (1) the number of patches (i.e., nodes in our graph) is reduced to quarter across con- secutive stages by the "PatchMerging" operation (Liu et al. 2021b); (2) increasing feature channels [C, 4C, 8C] are used for all stages. We place a graph construction layer before each modeling stage to update the multi-relational graph structure. For the first stage, we only use short- and long- range edges reduce the computational cost (computing medium-range edges by representation similarity comparison is expensive in the first stage with many patches), and the relational edges of all three ranges are adopted in the last three stages. Each stage is composed of multiple modeling blocks, where each block contains GRMP layer (Sec. 3.3) for relational message passing and a feed-forward network (FFN) (Vaswani et al. 2017) for feature transformation. In the end, a global average pooling layer produces the whole-image representation, and linear head outputs the fi- nal prediction. We adjust the number of feature channels and the number of blocks each stage to derive EurNet-T, S and EurNet-B with standard number of parameters and FLOPs. We implement the models based on the Py (Paszke et al. 2017) deep learning library.
E INTRODUCTION TO PROTEIN STRUCTURE
Proteins are macromolecules that perform critical biological functions in living organisms. A protein owns multiple levels of structures, as described below:
Primary structure (Fig. (()). At the chemical level, a protein is composed of one or multiple chains of amino acid residues, forming the protein sequence which is the primary protein struc-
20

---

Under review as a conference paper at ICLR 2023
Tertary
Alpho catbons
Figure 6: The primary structure, secondary structure, tertiary structure and all alpha carbons of the single-chain insulin protein (ID in PDB (Berman et al. 2000 2LWZ).
ture. In the protein sequence ($1, $2," SL), each element Si denotes type of amino acid (there are 20 common amino acids and two rare ones, i.e., Selenocysteine and Pyrrolysine). The primary structure tells the sequential order of amino acids in protein, but otherwise it does not reveal any information about the 3D folded structure of the protein. This fact limits its use- fulness in the analysis/p siaddiii/preiittionn protein functions, due the principle that "protein folded structures largely determine their functions" (Harms & Thornton 2010).
Secondary structure (Fig. b().. The secondary structures of proteins are some repeatedly- occurred local structures like the a- a-helices shown in Fig. [[).. ((b. These structures are stabilized by hydrogen bonds, and, together with the tight turns and flexible loops in between, they constitute the complete protein folded structure.
Tertiary structure (Fig (.). The spatial arrangement of different secondary structure compo- nents leads to the formation of the tertiary structure (i.c., the folded structure of protein). The tertiary structure is jointly held by short-range interactions like hydrogen bonding and long-range interactions like hydrophobic interactions. Thanks to the recent advances of highly accurate pro- tein folded structure predictors based on deep learning (Jumper et al.] 2021] Back et al.) [202I), efficiently acquire numerous previously unknown protein tertiary structures with reasonable confidence. These advances are expected to promote the understanding of protein functions based on tertiary structures.
In this work, we focus on protein function prediction tasks based on tertiary structures. Specifically, we adopt an informative and liigh representation format, i.e., all alpha carbons (Cos) in the tertiary structure (Fig. d)), which is widely used in the literature (Gligorijevi et al. 2021 Baldassarre al. 2 Zhang al., 2022). A Ca can be seen as the center of its corresponding amino acid, and thus the overall tertiary structure of separate points in the 3D space, since there is no chemical bond among them. To hetter describe the interactions within a protein, we seek to construct edges among Cas and lead more informative representation format, i.e., the Ca graph.
protein can well captured by the collection
of all Cas. At this time, the Cas are actually
F MORE EXPERIMENTAL SETUPS
F.1 MORE EXPERIMENTAL. SETUPS ON IMAGENET- 1K CLASSIFICATION
In the following, we state the detailed model and training configurations of (1) training on ImageNet- 1K from scratch and (2) pretraining ImageNet-22K followed by ImageNet-1K -K fine-tuning. For training configurations, we mainly follow the standards set up by Swin Transformer (Liu et al.) 2021bb for fair comparison.
F.1.1 FROM-SCRATCII TRAINING ON IMAGENET-IK
Model configurations. The whole model architectures of EurNet-T, EurNet-S and EurNet-B are presented in Tab. 7 For medium-range edges, 12 nearest semantic neighbors each patch are linked to it to capture medium-range interactions. For long-range edges, compute the represen- tations of per-patch global contelt virtual nodes by a stack of depth-wise 2D convolutions with the accumulative receptive field 7, and these virtual nodes are linked to their corresponding patches.
21

---

Under review as a conference paper at ICLR 2023
Training configuration. An AdamW (Loshchiloy & Hutter 2017) optimizer (betas: [0.9, 0.999], weight decay: 0.05) is employed to train each EurNet model for 300 epochs. We set the batch size as 2048, the base learning rate 0.002 and the gradient clipping norm 5.0. A cosine learning rate scheduler is adopted to adjust the learning rate from 2.0 x 10-6 to 0.002 in the first 20 warm-up epochs, and the learning rate is decayed to 2.0 10-5 in the rest epochs with cosine rate. The stochastic depth drop rates are set to 0.15, 0.3 and 0.5 respectively for EurNet-T, EurNet-S and EurNett-.. We follow the augmentation functions and mixup strategies used in Swin Transformer. All experiments are conducted on 16 Tesla- V V100-32GB GPUs.
F.1.2 PRE-TRAINING ON IMAGENET-222 22K AND FINE-TUNING ON IMAGENET-IK
Model configurations. The EurNet-B with the standard model architecture as in Tab. 7 used, ex- cept that the last linear classification head outputs 21,841 1184-44mmensionn logits to perform ImageNet- 22K classification.
Training configuration. For ImageNet-22K pre-training we train EurNet-B with an AdamW opti- mizer (betas: [0.9, 0.999], weight decay: 0.05) for 90 epochs with the batch size 4096 and the image resolution 224 x 224. cosine learning rate scheduler employed to linearly increase the learning rate from 0 to 4.0 x 10-3 in the first 5 warm-up epochs, and it decays the learning rate to 1.0 x 10-6 in the rest epochs with a cosine rate. The stochastic depth drop rate set as 0.1. All augmenta- tion functions and mixup strategies follow Swin Transformer. The pre-training is performed on 64 Tesla-V100-323 V100-32GB GPUs.
For ImageNet-IKK IK fine-tuning, the pre-trained model is fine-tuned for 30 epochs by AdamW optimizer (betas: [0.9, 0.999), weight decay: 1.0 x 10 $). The cosine learning rate scheduler adjusts the learning rate from 8.0 x 10- to 8.0 10-5 in the first 5 warm-up epochs, and the learning rate is decayed to 8.0 x 10-? ? in the rest epochs with a cosine rate. The stochastic depth drop rate is set as 0.2. Both Mixup (Zhang et al. (017) and CutMix (Yun et. (019) are muted during fine-tuning, following NocalNet Yang et al. (2022). The fine-tuning is performed on 16 Tesla- V100-32GB GPUs.
F.1.3 THROUGHPUT COMPUTATION
We follow Swin Transformer to measure the inference throughput on a Tesla- V100-32GB GPU with batch size 128. We adopt graph checkpoints enhance the speed of inferring an image that has been seen. During inference, we add short-range edges to the list of medium-range edges and merge their corresponding relations to further promote the efficiency.
F.2 MORE EXPERIMENTAL SETUPS ON COCO OBJECT DETECTION
Model configurations. We use the EurNet-T, EurNet-S and EurNet-B pre- rre-trained on ImageNet- 1K as the backbone of Mask R- R-CNN (He et al.] 2017). In specific, we take the patch representations output from all four modeling stages as the inputs of the Feature Pyramid Network (FPN) LipN)Lin et al. (2017). For medium-range edge construction on the high-resolution images of COCO, we select the semantic neighbors of each patch from a 112 x 112 dilated window (dilation ratio: 2) reduce the computational cost. For long-range edge construction, the representations of per-patch global- context virtual nodes are computed by a stack of depth-wise 2D convolutions with the accumulative receptive field as 31, and these virtual nodes are linked to their corresponding patches.
Training configurations. We follow Swin Transformer (Liu al.) 202Ib) to adopt multi-scale training strategy where the shorter side of an image is resized to [4S0, 800]. and the longer side is with length 1,333. An Adam (Loshchilov decay: 0.05) with initial learning rate 1.0 x 10- is employed for model training. In the 1x schedule with 12 total epochs, the learning rate is decayed at the 9th and I 11th epoch with the decay rate 0. In the 3x schedule with 36 total epochs, the learning rate is decayed at the 27th and 33rd epoch with the decay rate 0.1. The stochastic depth drop rate is set as 0.1, 0.2, 0.3 in Ix schedule and 0.25, 0.5, 0.5 in 3x schedule for EurNet-T/Si
Hutter (017) optimizer (betas: [0.9, 0.999], weight
respectively. All models are trained with batch size 8 on 8 Tesla- V V100 100B GPUs (i.e., one image per GPU). Our implementations are based on the midetectioon (Chen et al. 2019a) framework.
22

---

Under review as a conference paper at ICLR 2023
F.3 MORE EXPERIMENTAL SETUPS ON ADE20K SEMANTIC SEGMENTATION
Model configurations. The EurNet-T, EurNet-S and EurNet-B pre-trained ImageNet-1 serve as the backbone of UperNet (Xiao et al., 2018) to perform semantic segmentation. The patch rep- resentations output by all four modeling stages serve as the inputs of the Feature Pyramid Network (FPN) Lin et al. (2077). For medium-range edge construction, each patch is connected with its semantic neighbors from 144 144 dilated window (dilation ratio: 2). For long-range edge con- struction, we use a stack of depth-wise 2D convolutions with accumulative receptive field 31 to compute the representations of per-patch global-context virtual nodes, and we connect these virtual nodes with their corresponding patches.
Training configurations. All input images are resized the resolution 512 512. We adopt an AdamW W (Loshchilov & Hutter (2017) optimizer (betas: [0.9. 0.9991, weight decay: 0.01) to train the model for 160K iterations with the base learning rate 6.0 x 10-5. All models are trained with batch size 16 on 8 Tesla- V 100-32GG GPUs (i.e., two images per GPU). Our implementations are based on the mmsegmentation (Contributors 2020) framework.
F.4 MORE EXPERIMENTAL SETUPS ON PROTEIN FUNCTION PREDICTION
Edge message passing. Zhang et al. 2022) proposes to enhance the GearNet by edge-level message passing, which well captures the interactions between edges. To compare with the GearNet-Edge model enhanced in this way, we adapt the same edge message passing scheme to our EurNet.
Specifically, based on the constructed multi-relational graph g = (V. E, RR,, we further construct a line graph (Harary & Norman 1960) Gime = (Viiee Etime: Rline). In this graph, each node E Vine corresponds to an edge in the original graph G. There will an edge (u, 2, r) between nodes u, v E Vine if the corresponding edges of and v are adjacent in the original graph, and the edge type r E {0, 1,... 7 is determined by the angle L(u.o) 's allocation in 8 equally-divided bins of [0, a (L(m.)) denotes the angle between the corresponding edges of and in the original graph). Based on this multi- relational line graph, we employ the GRMP layer (Sec. (33) to propagate information between the nodes in Gtine and thus between the edges in the original graph G. Readers referred to ZZhag et al.] (2022) for more details. We name the EurNet equipped with such an edge message passing scheme as Eurget-Edge.
Dataset details. Two standard protein function prediction benchmarks are used in our experiments:
Enzyme Commission (EC) number prediction Gligorijević et al. (2021) requires the model to predict the EC numbers of a protein based on its tertiary structure, where the EC numbers describe the protein's 's catalysis of biochemical reactions. This task involves the binary prediction of 538 different EC numbers, forming 538 binary classification problems. This dataset contains 15,550 training, 1,729 validation and 1,919 test proteins.
Gene Ontology (GO) term prediction (Gligorijevi et al.. 2021 seeks to predict the GO terms owning by a protein based on its tertiary structure. This benchmark further split into three branches based on three types of ontologies: biological process (BP), molecular function (MF) and cellular component (CC). Each branch is formed by multiple binary classification problems. The GO benchmark dataset contains 29,896 training, 3,322 validation and 3,415 test proteins.
Model configurations. The backbone architecture of EurNet described in Sec. 4.2.2 Based on this backbone, we append a three-layer MI.P with the architecture Linear( InnerrCCott: Cout.) ReI.U - Linear (Cout: Cout)) > ReLU > Linear(Cout. Nrask) to predict the binary classification logits of all tasks simultancously (Cout: : the dimension of output protein representation; Niskk the number of binary classification tasks). We employ the binary cross entropy loss for model optimization.
Training configurations. An AdamW (Loshchilov & Hutter 2017) optimizer (betas: [0.9, 0.999], weight decay: 0) is utilized to train the model for 200 epochs. We adopt a cosine learning rate scheduler to linearly increase the learning rate from 1.0 10 7 to 1.0 x 10 and the learning rate is decayed to 1.0 x 10 6 in the rest epochs with a cosine rate. All models are trained with batch size 16 on 4 Tesla V 1100-32GB GPUs (i.e. four proteins per GPU).
23

---

Under review as a conference paper at ICLR 2023
Table 8: Performance comparison on knowledge graph completion benchmarks. "I" denotes the metric is the lower the better; "1" denotes the metric is the higher the better.
FB15k-237
WN18RR H@1,
Class
Model
MR.
MRR;
H@1,
H@3
H 10;
MR;
MRR;
H@3,
H@ 1010-
TransE
357
0.294
0.465
3384
0.226
0.501
DistMult
254
0.241
0.155
0.263
0.419
5110
0.43
0.39
0.44
0.49
Embedding
ComplEx
339
0.247
0.158
0.275
0.428
5261
0.44
0.41
0.46
0.51
RotaFE
177
0.338
0.241
0.375
0.553
3340
0.476
0.428
0.492
0.571
RGCN
221
0.273
0.182
0,303
0,456
2719
0,402
0,345
0.437
0.494
GNN
CompGCN
197
0.355
0.264
0,390
0.535
3533
0.479
0,443
0.494
0,546
EurNet
126
0.374
0.276
0.415
0.571
680
0.527
0.472
0.547
0.636
G EURNET FOR KNOWLEDGE GRAPH COMPLETION
Datasets. We conduct experiments on two standard knowledge graphs, FB15k-237 (Toutanova & Chen 2015) and WNI8RR (Dettmers et al., 2018). FB15k-237 contains 14,541 entities, 237 relation, 272,115 training triplets, 17,535 validation triplets and 20,466 test triplets. WN18RR has 40,943 entities, 11 relations, 86,835 training triplets. 3,034 validation triplets and 3, 134 test triplets. We follow the TorchDrug library (Zhu et al., 2022) to process knowledge graphs. For each triplet <h, rrr. 1>, its flipped counterpart <1, st,,-1,, h> is included for data augmentation. All triplets from the validation and test sets are removed to form the fact graph for training.
Model architecture. In this experiment, we instantiate the EurNet with 6 GRMP layers, each with 32 feature channels. Upon the EurNet, we adopt a two-layer MLP activated by ReLU to score each candidate triplet.
Training and evaluation. For model training, we follow the default setting in the TorchDrug li- brary (Zhu et al. 2022) sample 32 negative triplets for each positive triplet and perform binary classification with the binary cross entropy loss. On both knowledge graphs, the EurNet is trained for 20 epochs by Adam optimizer with learning rate 5.0 10-3 and batch size 16. Model train- ing is performed on 4 Tesla- V100-32GB GPUs. For evaluation, we follow previous works] Vashishth al. 2019); ZZuu et al. (2021) report mean rank (MR), mean reciprocal rank (MRR) and HITS N (H @ N) for knowledge graph completion.
Baselines. We compare the proposed EurNet with four classical knowledge graph embedding meth- ods, i.e., TransE (Bordes et al. 2013). DistMult (Yang et al.] 2014). ComplEx (Trouillon et al. 2016 and RotatE (Sun et al. [2019), and two typical relational GNNs, i.e., RGCN (Schlichtkrull ct al.
2018) and CompGCN Vashishth et al. 2019).
Results. We present the performance of EurNet and baselines in Tab. It can be observed that EurNet clearly outperforms the embedding- based and GNN baselines on all metrics of two datasets. Although knowledge graphs contain no spatial information, they are representative multi-relational graphs and are good test fields for evaluating the capacity of relational GNNs. The superior per- formance of EurNet on these benchmarks demonstrates the effectiveness of the GRMP layer on modeling the complex relational patterns in knowledge graphs.
H MORE ABLATION STUDY
Table 9: Ablation study of the key components of GRMP on ImageNet- 1K with EurNet-T.
H. 1 EFFECT OF GRMP COMPONENTS
Throughput (impess
Top-1 Acc (%)
Setting
In Tab. 9 we analyze the key components of GRMP by substituting removing the original component. This part abla- tion studies are conducted on ImageNet- IK classification with EurNet-T.
(M)
(G)
82.3
GRMP (gating addition) (e) [R] -3)
28.8
4.6
530.3
81,6(107)
81.9/104)
GRMP (wo Wyle)
26.7
4,3
561.9
81.7q10)
W
81,5
Effect of gating mechanism. In the first of the second block, study the importance of the gating mechanism in GRMP by substituting the Hadamard product in the step 2 Eq. 3 with the addition. After such change, the top-1 accuracy decays by 0.7%. This performance
24

---

Under review as a conference paper at ICLR 2023
decay demonstrates that, by using the separable graph convolution scheme in GRMP, the gating operation is more suitable than addition for node representation update (in contrast to the additive node representation update of RGConv in Eq. (T), which shares similar insights with the modulation mechanism in FocalNet (Yang et al. 2022).
Effect of node-adaptive relation weighting. In the second row of the second block, replace GRMP's node mode-adaptive relation weighting operation with simply taking the mean over all relations. This change leads to a 0.4% drop of accuracy. This relation weighting operation helps the GRMP layer to adaptively aggregate the messages of different relations based on each node's status, which benefits the model performance.
Effeet of pre-layer and post-layer node-wise channel aggregation. In the third and fourth rows of the second block, we respectively evaluate the model variants without W and Wout, Under these two settings, the model accuracy decays by 0.6% and 0.8%, respectively. Therefore, it is important to perform both pre-layer and post-layer node-wise channel aggregation in the GRMP layer.
H.2 EFFECT OF MULTI-RANGE EDGES FOR PROTEIN STRUCTURE MODELING
Table 10: Ablation study of multi- range edges on EC with EurNet.
Tab. T0 shows the performance of EurNet on the EC func- tion prediction benchmark by using different ranges of edges. When single range of edges are employed, the model with short-range edges obtains the highest F max score 0.750. This result illustrates the importance of capturing short-range in- teractions for protein structure modeling, which coincides with the fact that many short-range interactions (e.g., pep- tide and hydrogen bonds) contribute to the formation of pro- tein structures. By adding long-range edges, the model per- formance is improved to 0.760, where the extra modeling of long-range interactions (e.g., hydrophobic interactions) con- tributes to this improvement. By using all three ranges of edges, the full model of EurNet achieves the best F Fmax score 0.768, which demonstrates the necessity of capturing short-, medium- and long- range interactions for protein structure modeling.
short medium
long
Fmx
0.750
0.708
0.647
0.755
/
0.760
0.720
0.768
H.3 EFFECT OF GRMP FOR PROTEIN STRUCTURE MODELING
In Tab. we compare between RGConv and GRMP
Table II: Ablation study of nulti-relational modeling layer EC with EurNet.
under the comparable throughput (i.e., the number of
proteins that the model can process in one second). All experiments are performed on EC with EurNet. (1) We first set the hidden dimension of GRMP as 512. Under the comparable throughput, RGConv can only have the dimension of 422, and its F max score 0. 752 is lower than GRMP's 0.768. (2) We then increase RGConv's 's hidden dimension to 512. At this time, RGConv achieves the F Finax score 0.767 which is comparable to
Hidden Dimension
Throughput (proteins/s)
Layer
maa
RGConv
422
34.4
0.752
GRMP
512
34.6
0.768
RGConv
512
31.2
0.767
GRMP
592
31.5
0.780
GRMP's performance under the same dimension, while its throughput is decreased by 3.2. Under the comparable throughput, GRMP can have the hidden dimension of 592, which leads higher F Fiaxx score 0.780. These results demonstrate that GRMP owns a better sirciincy-pefformance trade-off than RGConv on protein structure modeling.
SENSITIVITY ANALYSIS
Table 12: Sensitivity analysis of semantic neighbor size on ImageNet-Ie IK with EurNet-T.
Image modeling sensitivity to semantic neighbor size. In Tab. 12 we report the performance of EurNet-T on ImageNet-1K 1K classification under different semantic neighbor sizes for medium-range edge construction. Though some marginal improvements are observed by using larger neighborhood size (i.e., more than or equal to 18 neighbors), the image modeling performance on this task is in general insensitive to the semantic neighbor size. By default, EurNet-T uses 12 semantic neighbors (denoted by the gray cell in Tab. [2). which achieves comparable performance with the configurations using more semantic neighbors.
#Neighbers
12
15
18
21
24
Top-1 (56
82.16 $2.20
N2.34
82.28
N2.34
25


Reviews:
[{"review_id": "xhq6broc6c", "reviewer": "Reviewer_WbJw", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces the EurNet for Efficient multi-range relational modeling. It constructs the multi-relational graph to encode the short-medium and long-range relation embedding. It also introduces a gated relational message passing layer to achieve that. Extensive experiments on ImageNet classification, COCO object detection and ADE20K semantic segmentation show the effectiveness of proposed approaches. ", "summary_of_the_review": "I appreciate the author present detailed rebuttal. However, the related works I present share the similar ideas with the author work. The authors also admit they ignore these related works. Please do cite these paper for new draft.\n\nAfter discussion with AC and other reviewers, I keep rating unchanged.", "strength_and_weaknesses": "Strength \n1, The motivation is clear, and the writing is easy to follow. \n\n2, The experiments results are extensive. The proposed approach outperforms previous work FocalNet in many settings. \n\n3, The experiment results for protein function prediction are interesting. \n\nWeakness.\n1, The idea or motivation of using different range graph structure is not new before the vision transformer.  The authors do not cite or compare the closely related works: 1, Graph-Based Global Reasoning Networks CVPR-2019. 2, Dynamic Graph Message Passing Network CVPR-2020. 3, Dual Graph Convolutional Network for Semantic Segmentation, BMVC-2019.\n\n2, The core contribution in Equ.3 is not new or novel. It contains the different window sizes of MHSA-like operator and along with a channel attention. \n\t\n3, The core contribution in Equ.3 should be simplified for easier understanding.  For example, why not give a figure to better illustrate the core operation?\n\n4, What the advantages of combining different range for classification tasks?\nWhat are the advantages over CNN+ transformer-like models?\n\n5, The performance on ImageNet and COCO are not very competitive. In particular, compared with Convnext, the GFlops and Throughput increases while the performance improvements are within 0.2 over different baselines. \n\nMoreover, the proposed framework does not show any performance gains over con-current methods.\nCMT: Convolutional Neural Networks Meet Vision Transformers. CVPR-2022\nCMT: Convolutional Neural Networks Meet Vision Transformers. Arxiv-2022.02\n\n\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "NscG_xh8RM", "reviewer": "Reviewer_aeee", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "1, Introduce a task called multi-range relational modeling.\n2, Their EurNet constructs the multi-relational graph, where each type of edge corresponds to short-, medium- or long-range spatial interactions.\n3, The proposed GRMP separately performs (1) relational message aggregation on each individual feature channel and (2) node-wise aggregation of different feature channels\n", "summary_of_the_review": "The paper introduces EurNet to do multi-range relational modeling. The proposed GRMP separately models multi-relational graph.", "strength_and_weaknesses": "Strength:\n1, Propose a novel modeling layer, called gated relational message passing (GRMP), to propagate multi-relational information across the data.\n2, Demonstrate EurNets in image and protein structure modeling cases.\nWeaknesses:\n1, Why your model is more \"efficient\" than other models?\n2, Why do we need multi-range relational modeling in Image classification?\n3, Do you have an inference time comparison between previous works?\n4, The method part is somehow hard to follow. Do you have any figures to demonstrate your method?\n5, Why relation-channel entangled aggregation is better?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "K5HtuHTD2k2", "reviewer": "Reviewer_nyUd", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors introduce a novel layer, called Gated Relational Message Passing (GRMP), for efficient multi-relational modelling over graphs. Importantly, GRMP scales better than existing multi-relational models when increasing the number of considered relations.\n\nThe authors use GRMP as building block for Efficient multi-range relational graph neural networks (EurNet) that they apply to multi-scale modelling problems. Across image classification, object detection, image segmentation and protein function prediction tasks, the authors show that EurNet performs comparably or better than state-of-the-art methods.", "summary_of_the_review": "This is an important contribution to multi-relational modelling and graph neural networks. The paper is clearly written, experiments are extensive and detailed supplementary info is provided.", "strength_and_weaknesses": "Strengths:\n* Contributions are clear and important\n* Claims are well supported; experiments are extensive (compared with multiple methods, using different model capacities, and considered different datasets / tasks)\n* The paper is well written. Figures are clarifying.\n\nWeakness:\n* This is a minor point: Maybe the authors could more explicitly showcase the empirical advantages of GRMP speedup in an applied setting when comparing with graph-based multi-relational models. (this is somehow already discussed in the “Throughput analysis.” paragraph, but the authors could make it more evident)", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "3mpiGajMfm0", "reviewer": "Reviewer_S9ug", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a graph convolution module that learns representations of a node by aggregating the representations from its neighbour nodes in a graph. Compared with the standard one (RGConv from Schlichtkrull et al., 2018), the proposed module introduces factorised version of the kernel matrix and other adaptations to reduce the computational complexity. The paper then applied the proposed module to several tasks of computer vision as well as protein structure modeling.", "summary_of_the_review": "Very comprehensive experiments and novelty is a bit weak.\n\nI have read the response of the authors. Additional experiments are appreciated.", "strength_and_weaknesses": "Strength:\n\n1. The paper provides comprehensive experiments on various tasks of computer vision and protein structure modeling. In each of the tasks, the paper compares with strong baselines on large-scale datasets. Comprehensive ablation study is also provided to show the efficiency of the proposed module. The empirical study of the paper is strong and supports the claims of the paper well.\n\n2. The proposed module shows promising results when applied with different backbone architectures in various tasks, which demonstrates the usability of the method.\n\n3. The paper is well written and structured. Experimental details are elaborated.\n\nWeaknesses:\n\n1. Despite the comprehensive empirical study, I feel that the novelty of the proposed method might be a bit weak. The proposed method is an adaptation from a well studied method (RGConv) from the angle of efficiency. The technical depth might not be very strong. But I would not complain much as similarity might be a factor that the method can be applied in various tasks.\n\n2.  The construction of the graphs may require additional tuning such as the number of KNN. It is also a bit heuristic as it is constructed by some predefined rules (although it can be dynamic with the learning of the model). Moreover, although the paper claims \"spatial\", I feel that it does not apply specific configurations to deal with \"spatial\" and \"spatial\" is just one of the predefined relations.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject