Look in The Mirror: Molecular Graph Contrastive Learning with Line Graph
Authors: Junran Wu
Xueyuan Chen
Bowen Shi
Jiaheng Liu
Shangzhe Li
Ke Xu
Venue: Submitted to ICLR 2023 (2023)

Abstract: Trapped by the label scarcity in molecular property prediction and drug design, graph contrastive learning came forward. A general contrastive model consists of a view generator, view encoder, and contrastive loss, in which the view mainly controls the encoded information underlying input graphs. Leading contrastive learning works show two kinds of view generators, that is, random or learnable data corruption and domain knowledge incorporation. While effective, the two ways also lead to molecular semantics altering and limited generalization capability, respectively. Thus, a decent view that can fully retain molecular semantics and is free from profound domain knowledge is supposed to come forward. To this end, we relate molecular graph contrastive learning with the line graph and propose a novel method termed LGCL. Specifically, by contrasting the given graph with the corresponding line graph, the graph encoder can freely encode the molecular semantics without omission. While considering the information inconsistency and over-smoothing derived from the learning process because of the mismatched pace of message passing in two kinds of graphs, we present a new patch with edge attribute fusion and two local contrastive losses for performance fixing. Compared with state-of-the-art (SOTA) methods for view generation, superior performance on molecular property prediction suggests the effectiveness of line graphs severing as the contrasting views.

Under review as a conference paper at ICLR 2023
LOOK IN THE MIRROR: MOLECULAR GRAPH CON- TRASTIVE LEARNING WITH LINE GRAPH
Anonymous authors
Paper under double-blind review
ABSTRACT
Trapped by the label scarcity in molecular property prediction and drug design, graph contrastive learning came forward. A general contrastive model consists of a view generator, view encoder, and contrastive loss, in which the view mainly controls the encoded information underlying input graphs. Leading contrastive learning works show two kinds of view generators, that is, random learnable data corruption and domain knowledge incorporation. While effective, the two ways also lead to molecular semantics altering and limited generalization capa- bility, respectively. Thus, a decent view that can fully retain molecular semantics and is free from profound domain knowledge is supposed to come forward. To this end, we relate molecular graph contrastive learning with the line graph and pro- pose novel method termed LGCL. Specifically, by contrasting the given graph with the corresponding line graph. the graph encoder can freely encode the molec- ular semantics without omiission. While considering the information inconsistency and over-smoothing derived from the learning process because of the mismatched pace of message passing in two kinds of graphs, we present a new patch with edge attribute fusion and two local contrastive losses for performance fixing. Compared with state-of-the-art (SOTA) methods for view generation, superior performance molecular property prediction suggests the effectiveness of line graphs severing as the contrasting views.
INTRODUCTION
deep understanding of molecular properties plays a vital role in the chemical and pharmaceutical domains. In order to computationally discover novel materials and drugs, the molecules will be abstractly regarded as graphs, in which atoms are vertices and bonds are edges Gilmer et al. (2017);
Goh et al. (2017); CCen et al. (2018)). Thus, the marriage between molecular property prediction and graph learning captured a bunch of researchers and showed their happiness in several fields Yang let al. (2019); (Sogg et al. (2020); Chen al.) (2021): /Wu et al. (2022a). However, this relationship faces the challenges of label scarcity, as deep learning methods are known to consume massive amounts of labeled data, and annotated data often of limited size and hard to acquire when considering many specific domains. In addition, given the immense differentiation among chemical molecules, existing supervised models could be barely reused in unseen cases Hu al. (2020); Rong et al. (2020). Therefore, there are increasing demands for molecular representation learning in an unsupervised or self-supervised manner.
Plenty of works have attempted to learn molecule representations discarding the supervision of la- bels, like graph context prediction Liu al. (2019), graph-level motif prediction Rong et al. (2020) and masked attribute prediction Hu et al. (2020). In light of the contrastive learning from com- puter vision, researchers go one step further to model molecules in a contrastive manner with data augmentations You et al. (2020); Suresh al. (2021). Considering the inherent characteristics of chemical molecules, graph contrastive learning incorporating well-designes domain knowledge has also shown excellent capacity in molecular properties prediction Sun al. (2021) |Fng al. (022).
Analogously, everything comes with a price. Inspecting the generated views in previous molecular graph contrastive learning unveils two intrinsic limitations. First, data nugmentation-based methods adopting random or learnable corruption (c.g., node/edge dropping and graph generation) would lead to inevitable variance in the crucial semantics and further misguide the contrastive learning

---

Under review as a conference paper at ICLR 2023
Line Graph
Dual-Helix Graph Encoder
Maximize Information
Transformation
Original Graph
Frisaldings
Projectios Head
GNN
GINN
GNN
Edge
Attribute
Fusion
Intro Local Contrastive Loes
Inter-Local
+
Contrasive Loss
Contrastive
Graph
GNN
GNN
GNN
Projection Head
Edge Embedkings
Figure 1: Framework overview of LGCL. Contrasted views consist of the original graph and the corresponding line graph. Input graphs are encoded by a dual-helix graph encoder with edge attribute fusion for information consistency. The whole model is jointly optimized via minimizing the NT-Xent loss and the two local contrastive losses.
You et al. 2020 Sun et al. (2021). Second, based predefined sub-structure substitution rules Sun et al. 2022) or contrasted with 3-dimension geometric views Liu et al. (2022); Stärk et al. (2022), domain knowledge-baid
methods intend alleviate the problem of semantic alteration. While effective, they are stinted to the profound domain knowledge that is unfriendly to researchers without such knowledge, thus limiting their generalization capability in other domains.
In this context, we are seeking for a decent view that will not be bothered by prefabricated domain knowledge and can maintain the molecular semantic information integrally. Fortunately, we met the line graph, also known as congruent graph in graph theory Whitney (1932); Harary & Norman (1960); Jung 1966). In a line graph, the nodes correspond to the edges of the original graph, and the edges refer to the common nodes of the pair edges in the original graph. In particular, the iso- morphism of two line graphs is judged to be consistent with the corresponding two original graphs Whitny 1932 Jung 1966), which ensures the congruent semantic structure after line graph trans- formation. light of the line graph, we propose a method termed LGCL to tackle our expectations.
The framework of LGCL is shown in Figure Specifically, to fill the framework demanding two views, all input molecular graphs are transformed into the corresponding line graph. On such a basis, I.GLL equips with a dual-helix graph encoder learn the hidden representation of two views. Note that, due to the different pace of message passing in the original graph and the corresponding line graph, issues derive from the learning process, that is, information inconsistency and over- smoothing. For information consistency, we further update the graph encoder with edge attribute fusion bridge the edge attributes between the two kinds of graphs. Over- smoothing addressed by novel lntra-local contrastive loss based on the idea of NT-Xent loss; put differently, the intra- local contrastive loss aims maximize the consense between the edge pairs in the two corresponding views and minimize the consense between different edge pairs within the same views. Moreover, we further give an inter-local contrastive loss to enhance the representation learning.
The effectiveness of LGCL is verified under the ubiquitous setting of transfer learning for molecular property prediction Hu al. (2020). Through pre-training on two million molecular graphs from ZINC15, LGCL shows superior performance prediction and acquires the highest position on both average ROC-AUC and average ranking. Addi- tionally, delve deeper into the proposed components via analytical experiments to further assess their benefits. The contributions are elaborated below:
six out of eight benchmarks for molecular property
To the best of our knowledge, we are the first to figure out way to freely and fully excavate molecular semantics within graph contrastive learning.
Inspired by the line graph, present approach, termed LGCL, tackle expecta- tions, in which edge attribute fusion and two local contrastive losses are united to address the concomitant issues and enhance molecular representation learning.
2

---

Under review as a conference paper at ICLR 2023
Leveraging eight benchmarks for molecular property prediction under the setting of transfer learning, LGCL exhibits its superiority against the SOTA methods for view generation.
RELATED WORKS
This research focuses molecular graph contrastive view generation, especially considering the case that free from the intricate domain knowledge. We will elaborate on these topics below.
Molecular graph contrastive learning. Along with the development of graph contrastive learning, plenty of research efforts have been devoted to designing contrastive learning models for molecular
graphs Sun et al. (2021); Xu et al. (2021); Fang al. (2022); Stärk et al. (2022); Liu et al. (2022); Li
et al. (2022a). Besides random or learnable corruption, several works presented various contrastive learning models to embed the molecular geometry information by means of contrasting the generic 2D graph with its 3D conformers Liu et al. (202);; Stärk et al. (2022); Li et al. (2022a). They in- deed get rid of the semantic altering issue caused by random corruption on molecular graphs, while introducing another semantic altering issue caused by 3D conformers, because a single 2D molec- ular graph generally has multiple conformers with different chemical properties Stärk et al. (2022). To enhance the performance in molecular property prediction, the domain knowledge-drive
con-
trastive learning frameworks were proposed to preserve the semantics of graphs in the augmentation process Sun et al. (2021); | Fang et al. (2022). However, their learning capability heavily relies on the dissolved domain knowledge, that is the well-designed substitution rules in MoCL Sun et al. (202I) and the prefabricated associations among chemical elements in KCL |CLFaan et al. (2022). Further- more, the domain knowledge varies across domains, which limits the application of these methods.
Recently, besides the contrasting view exploration. GraphLoG Xu et al. (2021) and OEPG Yang & Hong (2022) are built upon the generic graph contrastive learning methods to discover the global semantic structure underlying the whole dataset and present excellent performance. In this work, we are devoted to the domain of contrasting view generation, which is orthogonal to the works for dataset semantic structure exploration; put differently, extensive works for contrasting view genera- tion can work with the framework of GraphLoG and OEPG to produce more superior performance.
Line graph. The line graph is a classic concept and has long history in graph theory Whitney (1922); |arrary & Norman (1960); Jung (1966). In a line graph, the nodes correspond to the edges of the original graph, and the edges refer the common nodes of the pair edges in the original graph. Thus, the graph neural networks (GNNs) built on line graphs are capable of encoding edge features and enhancing feature learning on graphs. Recently, based on the line graph structures, several line graph neural networks have shown promising performance
various graph-related tasks (Seen et al. (2018)); Diang al. (2019); (Bandyopadhyay et al.) (2020). In the chemistry domain, the structure of compound can be treated as graph, where the edges derived from chemical bonds link the corresponding atom nodes. Thus, the edges in such graphs have different properties and various functions. In generic GNNs, however, cessage-passinngg operations among nodes do not pay enough attention to the edge properties. Fortunately, the line graph structure enables generic GNNs to take advantage of the edges as equals nodes Jiang al. (2019); Chen al. (2018b).
To this far, there is still no graph contrastive learning model to encode the molecular semantics integrally without well-designed domain knowledge. In this paper, we revisit the line graph from the angle of graph contrastive learning. Based on it, we design a novel contrastive model, termed LGCL, freely and fully excavate molecular semanties.
3 PRELIMINARIES
Here, we first present some preliminary concepts and notations. In this work, let G (G1, GG,,G,,... GNn be graph dataset with size N, and molecular graph can be formulated as G (V, E, Xv, X E)), where is the node set, E is the edge set, Xv E V denotes the node features, and X R|EXX stands for the edge attributes.
3

---

Under review as a conference paper at ICLR 2023
3.1 GRAPH REPRESENTATION LEARNING
In generic GNNs, the message- scheme is adopted for information transmission among nodes Xu et al. ) Wu et al. (2022b). Through stacking L layers, a GNN will produce hidden representation by E IR with hophop neighbor information for each node and a feature vector lig E IR global readout function for the entire graph G. Each node initialized with node feature X and sent to the GNN input. Formally, the (-th layer of a GNN Xu et al. (2019) can be written as
h0 AGGREGATE( (1NN- 1/1 N(v))),
(1)) COMBINE (h(), nG-1),
(2)
where 1! represents the feature vector of node v at the 1-th iteration, (e) covers the 1-hop neigh- bors of v, AGGREGATE denotes the crucial pesssage-passingg scheme in GNNs, and COMBINE is used to update the hidden feature of via merging information from its neighbors and itself. Finally, GNN can produce the feature vector lGg of the entire graph with prefabricated readout function:
hG READOUT((h,,P V}),
where READOUT aggregates the final set of node representations.
3.2 GRAPH CONTRASTIVE LEARNING
In a generic graph contrastive learning model, two correlated views from the same graph G; are required for contrasting and generally produced by two augmentation operations. Here, we denote the augmented views as G and G?. Then, graph encoder and a projection head stacked behind the two augmentation operators to map the correlated views into an embedding space and yield corresponding feature vectors h and h?. The released hidden representations contain the essential features the original graph G so that they can recognize themselves from the others. Thus, the objective of graph contrastive learning is maximize the consensus between the two positive views via the widespread NT-Xent loss Chen et al. 2020):
supposed to
C,
log
N Ejm1 Lillij sim(k)
where N is the batch size, T refers to the temperature parameter, and sim(A ,4²) denotes a cosine n similarity function І The numerator part is the similarity of the correlated views as positive pair. The rest pairs that consist of views from different graphs are regarded as negative pairs and act as the denominator part. Note that the negative pairs can come from two directions, put differently, h can pair with all h3. and h? can pair with all hj.
4 METHODOLOGY
In this section, we bring about the proposed graph contrastive learning framework, termed LGCL, by revisiting the line graph of corresponding molecules. Given the issue of label scarcity in real- world graph data, LGCL is designed to encode the molecular semantics integrally and free from well-designed domain knowledge. Specifically, of molecular semantics, we first need to transform the given molecule to the corresponding line graph. On such basis, LGCL equips with a dual-helix graph encoder to learn the hidden represen- tation of two views with edge attribute fusion. In particular, besides the ubiquitous contrastive loss for the readout graph representations, we further propose two local contrastive losses enhance representation learning and alleviate the over smoothing issue in deep GNNs. Next, we elaborate on the LGGLI framework below.
produce contrastive views without any loss
4.1 LINE GRAPH TRANSFORMATION
Here, first present an illustration of line graph transformation from simple graph. As shown in Figure? 2 let G = (V. E) be a simple undirected graph, the output line graph L(G) after trans- formation is such graph that reveals the adjacencies of edges in G. Specifically, each edge in G
4

---

Under review as a conference paper at ICLR 2023
2
3,4
3
4.5
3
5
(b) Vertices of L(G) born from edges of G
(a) Graph G
(c) Add edges in L(G)
(d) Line graph L(G)
Figure 2: An illustration of line graph transformation. (a) shows a simple undirected graph G; (b) reveals the derivation of vertices in line graph, every vertex of line graph is marked with green and labeled with the pair nodes of the corresponding edge in G: (c) establishes the associations in L(G) based on the common nodes of two edges; (d) delivers the output line graph L(G) of the original graph G.
is mapped into a node of L(G) and each edge in L(G') indicates that the corresponding two ver- tices have common node in G. Formally, the line graph can be written as L(G) = (Vz, EL), where Vz = {(0,, (1,, 15) E E)} and Ez = {((e,, 02), (uj, vk)) vj), (1), ux)) E). At this point, we have settled the topology transformation of line graphs. Besides the relationships among nodes and edges, the node and edge attributes underlying the molecular graph should also be delivered to the corresponding line graph. In this paper, based on the one-to-one correspondence between the edges of G' and the nodes of L(G), the node attributes of the line graph can be directly obtained from the edge attributes of original graphs, that is, Xv, = Xg. As for the edge attributes of L(G), because several edges in the line graph would correspond to the same node in the original graph, a mapping function with such relationships is required to endow the line graph edge with the original node attribute. In the light of Fz after the line graph transformation, the edge attribute mapping function can be formulated as (eL) = (v,, vy) n (ej, vk), thus the edge attributes of the line graph can be obtained via EL = MXv. Finally, the line graph of a molecular graph given by L(G) (Vz. EL, Xvz: XEL ). According to Roussopoulos's algorithm Roussopoulos 1973), the time complexity of line graph transformation is ((max(VV).(E))
As stated in the Whitney graph isomorphism theorem Whitney 1932). the isomorphism of two line graphs is judged to be consistent with the corresponding two original graphs, which convinces us that the semantic structure information of G is encoded in the line graph L(G). In particular, as described in the line graph transformation, there one-to-one correspondence between the edges in the graph G and the vertices in the line graph L(G). Therefore, a vertex with edges in G will produce e x (e - 1)/2 edges in L(G). Meanwhile, the message-passing frequency around this node will drift from O(e) in G O(e²) in L(G), put differently, this node feature in G is only passed to e neighbors, while the corresponding line graph will pass such information to e x (e 1)/2 nodes. Actually, in light of the line graph capacity in Table A. the line graph encoder only requires about 50% more computation than the original graph encoder. While this nature of the line graph could cause two inevitable issues in the contrastive learning framework with stacked graph convolutional layers, that is information inconsistency and over-smoothing. In this paper, we propose two approaches, edge attribute fusion and two local contrastive losses, to alleviate the two issues and strengthen molecular representation learning. Next, we give a detailed description.
4.2 EDGE ATTRIBUTE FUSION
In the chemistry domain, the structure of a compound can be treated as graph, where the edges derived from the chemical bonds link the corresponding atom nodes. Thus, the edges in such graphs have different properties and various functions. Besides the topology information weaved by atoms, a well-designed graph convolution with edge attributes plays a crucial role in molecular property and protein function prediction.
5

---

Under review as a conference paper at ICLR 2023
Given a molecular graph, its imput node features and edge features are both represented dimensional categorical vector (see Appendix A for details), denoted as Xv E | and X E E
2-
R [] respectively. In previous works regarding molecular property prediction the raw node categorical vectors are embedded in the input layer by
Hu et al. 2020
( EMBEDDING(( EMBEDDING(Z.)
where z. and x. are the atomic number and chirality tag of node v, respectively, EMBEDDING(.) denotes an embedding function that transfers a single integer into d-dimensional vector space. Meanwhile, the raw edge categorical vectors are embedded in each layer by
(
EMBEDDING(2") + EMBEDDING(2))
(6)
where xa and z represent the bond type and bond direction, respectively, and l denotes the index of GNN layers. At the 1-th layer, the node representation can be updated by
h o (MLP (h)
+
Σ
11-1)
Σ
h D)
wEN(v)
where (-) is an activation function, and vv) represents the self-loop edge.
Under this GNN architecture, the output molecular representations will be decorated with edge at- tributes. However, as discussed above, there a significant difference in message-passing frequency between the original graph and the corresponding line graph, which could lead to information in- consistency between the outputs. Here, we present novel edge attribute fusion approach tackle this issue. Specifically, we bridge the edge information between the molecular graph and line graph to help the original graph encoder keep pace with the line graph encoder. The edge and node em- beddings are still employed as the initial edge attributes in the first layer (i.e., I = 0). As for / > 1, the edge attributes of the original graph are obtained from the node hidden features in the line graph, which is formally given by
G.(,,,,) = h (1-1) ((..)))
(1)
where (mi,") E E updated by the node hidden features in the original graph, which formally formulated as:
and U5) E Vv. Correspondingly, the edge attributes of the line graph can be
(1-1) L (Lvee)))) G1G
where (u;, E, (Uz, Ux) E E and (11, (1),, (ey, v&)) E Er. Based on the dual-helix graph encoder with edge attribute fusion, the hidden features from the line graph will be dissolved into the original graph representations, allowing information consistency between the two contrastive views and enhancing molecular representation learning.
4.3 INTRA-LOCAL CONTRASTIVE Loss
In this part, we look forward tackling the over-smoothing issue introduced by the line graph. Motivated by NT-Xent loss for contrastive learning, an intra-local contrastive loss is proposed. The design concept of NT-Xent loss aims maximize the representation similarities of positive pairs consisting of hidden features of the same molecules and enforce dissimilarity of negative pairs com- prising hidden features of different molecules simultaneously. Similarly, the proposed intra-local contrastive loss seeks to optimize the consensus between the same nodes opposed to different nodes in a single graph. Considering the one-to-one correspondence between the edges in G and the vertices in L(G), the contrastive samples of this loss composed of the edge hidden features in G and the node hidden features in L(G). Thus, given a graph G, the intra-local contrastive loss of one edge pair is formally defined as:
Ce: Cnnrrat = - -loog
(10)
Σ simetha b:
where e; (Um, Un), e; E' and (cm, Un) Vz. In particular, the edge representations from G are formed by such hidden features of the two endpoints of each edge, that is, ha- MLP((h.... l. In light of this contrastive loss designed inside the graph, we look forward to reducing the similarity between different nodes and further alleviating the over-smoothing.

---

Under review as a conference paper at ICLR 2023
4.4 INTER-LOCAL CONTRASTIVE LOss
Here, we give another contrastive loss to enhance molecular graph contrastive learning. As we capable of enforeing dissimilarity between different edge representations, we move forward to generalizing the edge dissimilarity to all contrasted samples. Our critical insight is that the wide- spread NT-Xent loss only provides graph-level contrast, while contrastive angle based on node representations would also be meaningful in crucial structure identification. In light of the intra- local contrastive loss, the inter-local contrastive loss can be formally formulated as:
o
C InterC log
(11)
E simtha " L
GGG
where E, E EG, Cj E ViG and represents training batch. Note that the negative pairs of the inter-local contrastive loss also come from two directions.
Currently, we have presented the main components of the proposed LGCL that aims to help molec- ular graph contrastive learning free from well-designed domain knowledge and maintain the seman- tics. For unsupervised molecular graph representation learning, the final objective function of LGCL for pre-training is given by
min f LG + aLInterC + [LintraC,
(12)
where LG denotes the NT-Xent loss, and 3 hyper-parameters for loss weight controlling.
EXPERIMENT
In this section, we are devoted evaluating LGCL with extensive experiments | Following the procedure of pre-training and fine-tuning, we validate the effectiveness of our approach against SOTA competitors for view generation. Furthermore, we carry out analytical studies assess each proposed component. Unsupervised and semi-supervised learning are shown in appendix.
5.1 EXPERIMENTAL SETUP
To be in line with the previous graph contrastive learning methods without prefabricated domain knowledge and make the comparisons fair, we follow the experimental setup under the guidance of Hu et al. (2020).
Pre-training dataset. ZINC15 Sterling Irwin 2015) dataset is adopted for LGCL pre-training In particular, subset with two million unlabeled molecular graphs are sampled from the ZINC15.
Pre-training details. In the graph encoder setting in Hu etal. (2020), Graph Isomorphism Network (GIN) Xu et al.] (2019) with five convolutional layers is adopted for message passing. In particular, the hidden dimension fixed to 300 across all layers and a pooling readout function that averages graph nodes is hired for NT-Xent loss calculation with the scale parameter
0.1. The hidden representations at the last layer are injected into the average pooling function. An Adam optimizer minimize the integrated losses produced by the 5-layer GIN encoder. The batch size is set as 256, and all training processes will run 100 epochs. The two hyper-parameters (i.e., and 3) for loss weight controlling both
Kingma & Ba (015)) is employed
1.
Fine-tuning dataset. We employ the eight ubiquitous benchmarks from the MoleculeNet dataset Wu et al. (2018) to validate LGCL as downstream experiments. These benchmarks include variety of molecular tasks like physical chemistry, quantum mechanics, physiology, and biophysics. For dataset split, the scaffold split scheme Chen et al. (2012) is adopted for train/validation/test set generation. Table A.I summarizes the basic characteristics of the datasets, such as the size, tasks and molecule statistics. Detailed descriptions can be found in Appendix A.
Fine-tuning details. For downstream tasks, linear layer is stacked after the pre-trained graph encoders for final property prediction. The downstream model still employs the Adam optimizer for 100 epochs fine-tuning. All experiments on each dataset are performed for ten runs with different
The code of LGCL will he public after acceptance.

---

Under review as a conference paper at ICLR 2023
seeds, and the results are the averaged ROC-AUC scores (%) = standard deviations. The hyper- parameters tuned for each dataset are: (a) the learning rate E {0.01, 0.001, 0.0001}: (b) the batch size {32, ]: (c) the dropout ratio {0, 0.5}. The node representations for graph pooling are adopted from the last layer or the concatenation of all layers. These hyper-parameters are selected
by the grid search on the validation sets.
Baselines. In this paper, we choose the SOTA competitors that follows the experimental setup in Hu The first category is self-supervised graph learning algorithms, including EdgePred. Attr ContexPred Hu et al. (2020), Infomax Velickovic et al. (2019). and GraphMAE Hou
et al. (2020).
al. 2022)
The second category are graph contrastive learning methods for view generation, such
as GraphCL
You et al. (2020). JOAO(v2) You et al. (2021). LP-Info You et al. (2022), AutoGCL Yin
al. 2022).
GraphMVP|Lin et al. (2022), RGCL at (2022b) and D-SL.A Kim al. 2022).
5.2
RESULTS
The results of LGCL along with SOTA competitors for molecular property prediction on eight benchmarks are shown Table [ To summarize, the proposed graph contrastive learning frame- work with the line graph, LGCL. obtains superior performance compared with the previous works. Specifically, in light of the last column for average rank, our method seizes the highest ranking po- sition from SOTA contrastive learning methods well self-supervised learning methods, and significant ranking improvement can be witnessed as opposed to the second place (D-SLA gives the A.R. 5.0). In particular, LGCL achieves the best performance on six of eight benchmarks, and the best comprehensive performance is also with us (see the penultimate column). Thus, we can con- clude that LGCL captures the molecular semantic information well in the absence of vell-designed domain knowledge, and the line graph provides an excellent contrastive view without altering the molecular semantics.
Table I: Average test ROC-AUC (%) + Std. over different 10 runs of LGCL along with all baselines on eight downstream molecular property prediction benchmarks. The results of baselines are derived from the published works. Bold indicates the best performance among all baselines. Avg. shows the average ROC-AUC over all datasets. A.R. denotes the average rank.
Dataset
88BP
Tox21
ToxCast
SIDER
Clin'ox
MUV
HIV
BACE
Avg.
A.R.
No Pre-Tram
65.8143
74.01088
0.64-0.6
S1.311.6
58.0144
7L8225
15.3:119
70.135.4
66.96
13.3
Inforax
6888+88
75.3+066
62.7-0.4
584+008
699++3.
75.3+22.
76.0+0.77
75.9+16
70.29
11.2
EdgaPeed
67.34124
76.01066
64.1=0.6
60.420.7
74.122.1
76.3:110
AttrMasking
64.31188
76.710.
64.2-0.5
61.010.7
71814.11
74.72:1.4
77.2-1.11
79.341.6
69.90
8.9
Contextreed
68.42200
75.7+0.7
63.9-0.6
60.900.6
65.943.8
75,8+1.7
77.3+10
79.6+11.2
70.89
8.1
GraphMAB
75.5206.
64.1=0.3
76.3=24
77,221.0
73.85
5.8
GrsphCL
69.6810067
133710086
62.40_0.37
60531033
75
09.80122000 86
78.47.1.22
75.38 1.44
10.77
10.9
10A0(v2)
71.39+0922
74.27+062
63.16-0.45
60.49:0.74
00:97+1.64
73.67+1.00
77.51+1.17
75.49+ 1.27
72.12
9.0
LP-Info
51
44.54:0045
63.04=0.30
59.70:20443
72.99:2.28
76.96:21.10
80.21
71.71
AD-GCL
20.01111007
76.5410.82
63.07-0.72
63.28.10.79
79.7813.52
72.30:1161
78.28:10.97
78.51
72.72
7.1
AntaGCL
73.36:10.77
75.69:0.29
63.47-0.38
62.51=0063
80:991:3.38
15.83=11.30
78.35:10.64
83.26::
74.18
4.1
GVP
68.5+0.22
74.5+04
62.7=0.1
62.3+116
79.0+22.5
75.0=:144
74.8:1144
ROCL
7144210.66
25:20:10344
63.33-0.17
61.38:10.11
83.3810.91
76.66:20.99
77,9011.80
76.03:10.7
73.16
5.9
D-SLA
72.60110.79
76.5110.52
64.24=0.50
60.22:11.13
10.17:11.50
76.64220.91
78.59110:44
83.8111.01
74.14
3.8
LGCL
70.99+1.05
76.95-0.43
0.72
63.37+0.56
77.59+1.54
77.70+3.00
78.69-1.10
84.68+0.73
74.33
2.6
5.3 ABLATION STUDY
Here, we delve deeper into the performance influence of each proposed component. First, we an- alyze the performance boosting from the introduction of the line graph and edge attribute fusion without ZINC15 pre-training. As for the two local contrastive losses, we present the test results of various combinations from these parts following the transfer learning settings. The detailed discus- sions are as follows.
The effect of line graph. In Figure 3 we analyze the effect of the line graph. In comparison with the red bar (i.c., 'No Pre-Traini )) that denotes the results from random initialization, introducing the line graph (i.e., 'No Pre-Train w/LG') shows overall superior performance, which empirically suggests that the semantics underlying the edges can be better captured by the line graph.
The effect of edge attribute fusion. Based the performance boosting of the line graph, further present the results with edge attribute fusion in Figure (i.e., 'No Pre- Tre-Train w/G w/AF). Besides the general promotion compared with the results of random initialization, the edge attribute
8

---

Under review as a conference paper at ICLR 2023
80
No Pro-Train
No Pre-Train WLG
75
No Pre-Train w/AF
ROC-AUCIS
70
65
60
55
50
ToxCast
SIDER
ClinTox
BBBP
Tox21
MUV
HIV
BACE
Figure 3: Average test ROC-AUC (%) gain within 'No Pre-Train' from the line graph (w/LG) and edge attribute fusion (w/AF) across all datasets.
Table 2: Average test ROC-AUC (%) of LGCL with different components. Avg. shows the average ROC-AUC over all datasets. A.R. denotes the average rank.
Cinteat Eisuee
HIV
Avg.
15.40-0330
02.00:1:35
59.321
15.70:2.715
15.2230:4
81:0020013
71.16
3,2
1111-121
75.85-0.99
61.3-0.74
50.35-0.51
75.00-235
06.37-1.81
21.30-0.055
80.70-1
77.40
3.6
89.77=122
74.920.52
64:0--068
60.49:1.32
76.44:4.32
75.52-1445
77.40:1.0 49
84.00:0:66
72.64
4.0
13.74-063
63:00:065
00:47:1088
70.884432
10.12:195
71.09.0.79
84.23-0087
73.12
2.9
1
20.1
15.520035
63.991000
60.15:0.49
1456-3:17
15.92:180
41.44
83.8721
72.08
70.99 105
6471-0.72
63.37-0.36
11.9--1.4
77.00-300
78.09-1
10.73
fusion also brings five out of eight better results in contrast to the solo line graph. Furthermore, following the setting of transfer learning. the performance differences between the first and fifth rows as well as the fourth and sixth rows in Table (aloo also validate the effectiveness of edge attribute fusion. Thus, we may conclude that edge attribute fusion can alleviate information inconsistency and enhance molecular graph representation learning.
The effect of intra-local contrastive loss. The test results under the supervision of the proposed losses are shown in Table 2 To achieve comprehensive comparison, we first give baseline only pre-trained with the NT-Xent loss (see the first row). The effectiveness of the proposed intra-local contrastive loss is confirmed by the performance differences between the second and first rows as well as the fourth and third rows, in which the only experimental setup difference the LintraC- Specifically, least six out of eight better results are obtained via deploying this contrastive loss, which informs us of its effectiveness over-smoothing addressing.
The effect of inter-local contrastive loss. Analogously, when comparing the results of the first and third rows as well as the second and fourth rows in Table 2] we can observe that six and five datasets achieve performance exaltation, respectively. This metrie promotion indicates the effec- tiveness of the inter-local contrastive loss in crucial structure identification. Finally, despite several failures within these ablation studies, the last that simultaneously adopts all proposed compo- nents performs best; thus, the proposed parts of LGCL are complementary to each other in molecular semantic exploration regardless of the intricate domain knowledge.
CONCLUSIONS
In this work, we try to figure out decent view for molecular graph contrastive learning that can maintain the integrity of molecular semantic information and is friendly to researchers without pro- found domain knowledge. Inspired by the line graph, we propose method, called LGCL, to tackle our expectations. Due to the different pace of message passing in the original graph and the corre- sponding line graph, further present three crucial components address the concomitant issues and enhance molecular graph representation learning. Under the setting of transfer learning, we empirically present the superior performance of LGCL over SOTA works.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Aids antiviral screen data, Accessed: terrentttaa
Tox21 data challenge. challenge 2014.
Accessed: 2017-09-27, attp::/txipo..nih..ov////so
Bijaya Adhikari, Yao Zhang, Naren Ramakrishnan, and B Aditya Prakash. Sub2vec: Feature learn- ing for subgraphs. In Pacific-Asia Conference Knowledge Discovery and Data Mining, pp-
170-182. Springer, 2018.
Rajasckar Venkatesan Lihui Chen Yang Liu Annamalai Narayanan, Mahinthan Chandramohan and Shantanu Jaiswal. graph2vec: Learning distributed representations graphs. In Proceedings of the 13th International Workshop Mining and Learning with Graphs (MLG), 2017.
Sambaran Bandyopadhyay, Kishalay Das, and M Narasimha Murty. Hypergraph attention isomor- phism network by learning line graph expansion. In 2020 IEEE International Conference Big Data (Big Data). pp. 669-678 678. IEEE, 2020.
Guy W Bemis and Mark A Murcko. The properties of known drugs. 1. molecular frameworks. Journal of Medicinal Chemistry, 39(15):2887-2893, 1996.
Bin Chen, Robert P Sheridan, Viktor Hornak, and Johannes H Voigt. Comparison of random forest and pipeline pilot naive bayes prospective qsar predictions. Journal of Chemical Information and Modeling, 52(3):792-803, 2012.
Hongming Chen, Ola Engkvist, Yinhai Wang, Marcus Olivecrona, Thomas Blaschke. The rise of deep learning in drug discovery. Drug Discovery Today, 2316:1241-1250, 2018a
Jianwen Chen, Shuangjia Zheng. Ying Song, Jiahua Rao, and Yuedong Yang. Learning attributed graph representations with communicative message passing transformer. In IJCAI, volume 2021, pp. 2242-2248, 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference Machine Learning, pp. 1597-1607. PMLR, 2020.
Zhengdao Chen, Lisha Li, and Joan Bruna. Supervised community detection with line graph neural networks. In International Conference on Learning Representations, 2018b.
Thomas M Cover. Elements of information theory. John Wiley & Sons, 1999.
Yin Fang, Qiang Zhang, Haihong Yang, Xiang Zhuang, Shumin Deng, Wen Zhang, Ming Qin, Zhuo Chen, Xiaohui Fan, and Huajun Chen. Molecular contrastive learning with chemical element knowledge graph. Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 3968-9966,, 2022.
Fleanor Gardiner, John D Holliday, Caroline O'Dowd, and Peter Willett. Effectiveness of 2d fingerprints for scaffold hopping. Future Medicinal Chemistry, 344):405-414 2011.
Kaitlyn M Gayvert, Neel Madhukar, and Olivier Elemento. A data-driven approach to predicting successes and failures of clinical trials. Cell Chemical Biology, 23910):1294-1301 2016.
Justin Gilmer, Samuel Schoenholz, Patrick Riley, Oriol Vinyals, and George Dahl. Neural message passing for quantum chemistry. In International Conference on Machine Learning, pp. 1263-1272. PMLR, 2017.
Garrett Goh, Nathan O Hodas, and Abhinav Vishmu. Deep learning for computational chemistry. Journal of Computational Chemistry, 38(16):1291-1307 2017.
Aditya Grover and Jure Leskovec, node2vec: Scalable feature learning for networks. SIGKDD, pp- 865-864, 2016.
10

---

Under review as a conference paper at ICLR 2023
Frank Harary and Robert Z Norman. Some properties of line digraphs. Rendiconti del circolo matematico di palermo, 122:161-168, 1960.
Zhenyu Hou, Xiao Liu, Yukuo Cen, Yuxiao Dong. Hongxia Yang, Chunjie Wang, and Jie Tang. Graphmae: Self-supervised masked graph autoencoders. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, 2022.
W Hu, B Liu, Gomes, M Zitnik, P Liang, V Pande, and J Leskovec. Strategies for pre-training graph neural networks. International Conference on Learning Representations (ICLR), 2020.
Xiaodong Jiang, Pengsheng Ji, and Sheng Li. Censnet: Convolution with edge-node switching in graph neural networks. In IJCAI, pp. 2656-2662, 2019.
HA Jung. Zu einem isomorphiesatz von h. whitney für graphen. Mathematische Annalen, 163: 270-271, 1966.
Dongki Kim, Jinheon Baek, and Sung Ju Hwang. Graph self-supervised learning with accurate discrepancy learning. Advances in Neural Information Processing Systems, 2022.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR (Poster), 2015.
Thomas N Kipf and Max Welling. Variational graph auto-encoders. NIPS Workshop on Bayesian Deep Learning, 2016.
Michael Kuhn, Ivica Letunic, Lars Juhl Jensen, and Peer Bork. The sider database of drugs and side effects. Nucleic Acids Research, 44(D1):1105-D1107, 2016.
Greg Landrum. Rdkit documentation. Release, 1(1-79), 2013.
Shuangli Li, Jingbo Zhou, Tong Xu, Dejing Dou, and Hui Xiong. Geomgel: Geometric graph contrastive learning for molecular property prediction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 4549,, 2022a.
Sihang Li, Xiang Wang, An Zhang, Yingxin Wu, Xiangnan He, and Tat-Seng Chua. Let invariant rationale discovery inspire graph contrastive learning. In International Conference on Machine Learning, pp. 13052-13000 13065. PMLR, 2022b.
Shengchao Liu, Mehmet F Demirel, and Yingyu Liang. N-gram graph: Simple unsupervised repre- sentation for graphs, with applications to molecules. Advances in Neural Information Processing Systems, 32, 2019.
Shengchao Liu, Hanchen Wang, Weiyang Liu, Joan Lasenby, Hongyu Guo, and Jian Tang. Pre- training molecular graph representation with 3d geometry. International Conference on Learn- ing Representations, 2022.
Ines Filipa Martins, Ana I. Teixeira, Luis Pinheiro, and Andre O Falcao. A bayesian approach to in silico blood-brain barrier penetration modeling. Journal of Chemical Information and Modeling, 6266):168611697, 2012.
Christopher Morris, Nils M. Kriege, Franka Bause, Kristian Kersting, Petra Mutzel, and Marion Neumann. Tudataset: A collection of benchmark datasets for learning with graphs. ICML 2020 Workshop on Graph Representation Learning and Beyond, 2020.
Paul A Novick, Oscar F Ortiz, Jared Poelman, Amir Y Abdulhay, and Vijay S Pande. Sweetlead: an in silico database of approved drugs, regulated chemicals, and herbal isolates for computer-aided drug discovery. PloS One, 8(11))79568, 2013.
Bharath Ramsundar, Peter Eastman, Patrick Walters, and Vijay Pande. Deep learning for the life sciences: applying deep learning to genomics, microscopy, drug discovery, and more. O' O'Reilly Media, 2019.
11

---

Under review as a conference paper at ICLR 2023
Ann M Richard, Richard S Judson, Keith A Houck, Christopher M Grulke, Patra Volarath, Inthirany Thillanadarajah, Chihae Yang. James Rathman, Matthew T Martin, John F Wambaugh, et al. Toxcast chemical landscape: paving the road to 21st century toxicology. Chemical Research in Toxicology, 1255)::225-2555: 2016.
Yu Rong, Yatao Bian, Tingyang Xu, Weiyang Xie, Ying Wei, Wenbing Huang, and Junzhou Huang. Self-supervised graph transformer on large-scale molecular data. Advances Neural Information Processing Systems, 33:12559-12571, 2020.
Nicholas D Roussopoulos. A max {m, n} algorithm for determining the graph h from its line graph g. Information Processing Letters, 2 108-112, 1973.
Robert P Sheridan. Time-split cross-validation as a method for estimating the goodness of prospec- tive prediction. Journal of Chemical Information and Modeling, 2013.
Nino Shervashidze, SVN Vishwanathan, Tobias Petri, Kurt Mehlhorn, and Karsten Borgwardt. Ef- ficient graphlet kernels for large graph comparison. In Artificial Intelligence and Statistics, pp- 488-495. PMLR, 2009.
Nino Shervashidze, Pascal Schweitzer, Erik Jan Van Leeuwen, Kurt Mehlhorn, and Karsten M Borg- wardt. Weisfeiler-lemman graph kernels. Journal of Machine Learning Research, 12(9), 2011.
Ying Song, Shuangjia Zheng, Zhangming Niu, Zhang-Hua Fu, Yutong Lu, and Yuedong Yang. Communicative representation learning on attributed molecular graphs. In IJCAI, volume 2020, pp. 2831-28338 2020.
Hannes Stärk, Dominique Beaini, Gabriele Corso, Prudencio Tossou, Christian Dallago, Stephan Günnemann, and Pietro Lio. 3d infomax improves gnns for molecular property prediction. In International Conference on Machine Learning. PMLR, 2022.
Teague Sterling and John Irwin. Zine li-lid discovery for everyone. Journal of Chemical Information and Modeling, 55(11):2324-2333 2015.
Govindan Subramanian, Bharath Ramsundar, Vijay Pande, and Rajiah Aldrin Denny. Computational modeling of 3-secretase I (bace-1 1) inhibitors using ligand based approaches. Journal of Chemical Information and Modeling, 56 36(10):11 1936-199939999 2016.
Fan-Yur Yun Sun, Jordon Hoffman, Vikas Verma, and Jian Tang. Infograph: Unsupervised and semi- supervised graph-level representation learning via mutual information maximization. ICLR, 2020.
Mengying Sun, Jing Xing, Huijun Wang, Bin Chen, and Jiayu Zhou Moel: data-driven molecular fingerprint via knowledge-aware contrastive learning from molecular graph. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp. 3585-3594, 2021.
Susheel Suresh, Pan Li, Cong Hao, and Jennifer Neville. Adversarial graph augmentation to improve graph contrastive learning. Advances in Neural Information Processing Systems, 34, 2021.
Petar Velickovic, William Fedus, William L Hamilton, Pietro Lið, Yoshua Bengio, and R Devon Hjelm. Deep graph infomax. ICLR (Poster), 2(3):4, 2019.
Hassler Whitney. Congruent graphs and the connectivity of graphs. American Journal of Mathe- matics, 5401)1150-168, 1932.
Junran Wu, Xueyuan Chen, Ke Xu. and Shangzhe Li. Structural entropy guided graph hierarchical pooling. In International Conference on Machine Learning, pp. 24017-24030. PMLR, 2022a.
Junran Wu, Shangzhe Li, Jianhao Li, Yicheng Pan, and Ke Xu. A simple yet effective method Proceedings of the Thirty-First International Joint Conference on
for graph classification. Artificial Intelligence, IJCAI 2022, Vienna, Austria, July 23-29, 2022. ijcai. 2022b.
Zhenqin Wu, Bharath Ramsundar, Evan N Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh Pappu, Karl Leswing, and Vijay Pande. Moleculenet: a benchmark for molecular machine learn- ing. Chemical Science, 92):513-530, 2018.
12

---

Under review as a conference paper at ICLR 2023
Keyulu Xu, Weihua Hu, Jure Leskovee, and Stefanie Jegelka. How powerful graph neural networks? ICLR, 2019.
Minghao Xu, Hang Wang, Bingbing Ni, Hongyu Guo, and Jian Tang. Self-supervised graph-level representation learning with local and global structure. In International Conference on Machine Learning, pp. 1 1848-11558, PMLR, 2021.
Pinar Yanardag and SVN Vishwanathan. Deep graph kernels. SIGKDD, pp. 1365-1374, 2015.
Kevin Yang, Kyle Swanson, Wengong Jin, Connor Coley, Philipp Eiden, Hua Gao, Angel Guzman- Perez, Timothy Hopper, Brian Kelley, Miriam Mathea, et al. Analyzing learned molecular repre- sentations for property prediction. Journal of Chemical Information and Modeling, 59(8):337- 3388, 2019.
Ling Yang and Shenda Hong. Omni-granulat ego-semantic propagation for self-supervised graph representation learning. In International Conference on Machine Learning, pp. 25022-25037. PMLR, 2022.
Yihang Yin, Qingzhong Wang, Siyu Huang, Haoyi Xiong, and Xiang Zhang. Autogel: Automated graph contrastive learning via learnable view generators. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 8892-8900, 2022.
Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. Advances in Neural Information Processing Systems, 33:5812-5823, 2020.
Yuning You, Tianlong Chen, Yang Shen, and Zhangyang Wang. Graph contrastive learning auto- mated. In International Conference on Machine Learning, pp. 12121-12132. PMLR, 2021.
Yuning You, Tianlong Chen, Zhangyang Wang, and Yang Shen. Bringing your own view: Graph contrastive learning without prefabricated data augmentations. WSDM '22, pp. 1300-1309, New York, NY, USA, 2022. Association for Computing Machinery.
Zaixi Zhang, Qi Liu, Hao Wang. Chengqiang Lu, and Chee-Kong Lee. Motif-based graph self- supervised learning for molecular property prediction. Advances in Neural Information Process- ing Systems, 34:15870-11 2021.
13

---

Under review as a conference paper at ICLR 2023
Supplementary Materials for Look in The Mirror: Molecular Graph Contrastive Learning with Line Graph
A DETAILS OF MOLECULAR DATASETS
As discussed in Section 4.1 vertex with e edges in G will produce x (e 1)/2 edges in L(G), which could lead to severe runtime complexity when the original graphs are dense. Therefore, our method only suits sparse graphs. As for molecules adopted in this work, we can see from Table A.I that the computation encoder according to the average degrees in the transformed line graphs. However, as for the realistic time required for model pre-training, detailed comparisons are shown in Appendix
the line graph encoder is 1.5 times that of the original graph
Input graph representation. For simplicity, we use minimal set of node and bond features that unambiguously describe the two-dimensional structure of molecules. We use RDKit Landrum 2013) to obtain these features.
Node features:
Atom number: [1, 118]
Chirality tag: (unspecified, tetrahedral ow, tetrahedral cow, other)
Edge features:
Bond type: {single, double, triple, aromatic)
Bond direction: {-, endupright, enddownright)
Table A.1: Datasets statistics summary.
Dataset
#lasks
#Grraphs
Avg.Node
Avg.Degree
LG Degree
ZINC15
2,000,000
26.63
57.72
80.98
BBBP
I
2,039
24.06
51.90
75.10
Tox21
12
7,831
18.57
38.58
53.03
ToxCast
617
8,576
18.78
38.52
52.95
SIDER
27
1,427
33.64
70.71
99.97
ClinTox
2
1,477
26.15
55.76
79.76
MUV
17
93,087
24.23
52.55
73.06
HIV
1
41,127
25.51
54.93
78.42
BACE
1
1,513
34.08
73.71
105.78
Downstream task datasets. 8 binary graph classification datasets from MoleculeNet Wu et al. 2018) are used to evaluate model performance.
BBBP Martins et al. (2012). Blood-brain barrier penetration (membrane permeability), involves records whether a compound carries the permeability property penetrating the blood-brain barrier.
Tox21 Tox 2014). Toxicity data on 12 biological targets, which has been used in the 2014 Tox21 Data Challenge and includes nuclear receptors and stress response pathways.
ToxCast Richard et al. (2016). Toxicology measurements based on over 600 vitro high-
throughput screenings.
SIDER Kuhn et al. (2016). Database of marketed drugs and adverse drug reactions (ADR),
grouped into 27 system organ classes and also known as the Side Effect Resource.
ClinTox Novick al. (2013); Gayvert al. (0016). Qualitative data classifying drugs approved by the FDA and those that have failed clinical trials for toxicity reasons.
MUV Gardiner et al. (2OIT). Subset PubChem BioAssay by applying a refined nearest neighbor analysis, designed for validation of virtual screening techniques.
14

---

Under review as a conference paper at ICLR 2023
HIV HIV Experimentally measured abilities to inhibit HIV replication.
BACE Subramanian et al. (2016). Qualitative binding results for set of inhibitors of
human B-secretase
Dataset splitting. For molecular prediction tasks, following Ramsundar al. (211)), we cluster molecules by scaffold (molecular graph substructure) Bemis & Murcko 1996). and recombine the clusters by placing the most common scaffolds in the training set, producing validation and test sets that contain structurally different molecules. Prior work has shown that this scaffold split provides a more realistic estimate of model performance prospective evaluation compared to random split Chen et al. ) (hhridan (213)). The split for train/validdtion/testt sets is 80%: 10%: 10%.
B MOLECULAR PROPERTY PREDICTION VIA UNSUPERVISED LEARNING AND SEMI-SUPERVISED LEARNING
As discussed above, LGCL only suits sparse graphs to avoid too bad runtime complexity. As shown in Table A.2. the capacities of all social network datasets and dense bioinformatics dataset (i.e., DD) heavily increase after the line graph transformation, which leads to unaffordable computation consumption in graph representation learning. Therefore, only employ sparse bioinformatics datasets for unsupervised and semi-supervised learning. Experiment details are elaborated below.
Table A.2: Summary statistics of ubiquitous benchmarks from TUDataset.
Dataset
#Graphs
#Classes
Avg.Nodes
Avgreegree
LG Avg.Degree
Social Networks
COLLAB
5,000
74.49
4914.43
786967.36
REDDIT-BINARY
2,000
429.63
995.50
184826.67
REDDIT-MMULTI-SK
4,999
508.52
1189.74
81066.29
IMDB-BINARY
1,000
19.77
193.06
2782.11
IMDB-MULTI
1,500
13.00
131.87
2037.64
GITHUB
12,725
113.79
469.27
19574.33
Bioinformatics
MUTAG
188
17.93
39.58
57.74
NCII
4,110
2
29.87
64.60
93.21
PROTEINS
1,113
39.06
145.63
448.99
DD
1,178
284.32
1431.31
6581.59
B.1 UNSUPERVISED LEARNING
Datasets. Three sparse bioinformatics datasets adopted from TUDataset Morris al. (2020)
for unsupervised learning, including NCII and MUTAG, and PROTEINS. Table A.2 summarizes the characteristics of the three employed datasets.
NCII is dataset made publicly available by the National Cancer Institute (NCI) and is a subset of balanced datasets containing chemical compounds screened for their ability to suppress or inhibit the growth of a panel of human tumor cell lines; this dataset possesses 37 discrete labels.
MUTAG has seven kinds of graphs that are derived from 188 mutagenic aromatic and heteroaromatic nitro compounds.
PROTEINS is dataset where the nodes are secondary structure elements (SSEs), and there is an edge between two nodes if they are neighbors in the given amino acid sequence or in 3D space. The dataset has 3 discrete labels, representing helixes, sheets or turns.
To keep in line with GraphCL You et al. (2020), the same GNN architectures employed with their original GIN Xu et al. (2019) with 3 layers is set up in unsupervised representation learning. The encoder
Configuration.
under individual experiment settings. Specifically,
15

---

Under review as a conference paper at ICLR 2023
hidden dimensions are fixed for all layers to keep in line with GraphCL under individual experiment setting. Models are trained 20 epochs and tested every 10 epochs. Hidden dimension 32, and batch size is {32, 128). An Adam optimizer Kingma & Ba 2015) is employed to minimize the contrastive lose and learning rate is E (0.01, 0.001, 0.0001].
Learning protocol. Following the learning setting in SOTA works, the corresponding learning protocols are adopted for a fair comparison. unsupervised representation learning [Snn et al. (2020), all data is used for model pre-training and the learned graph embeddings are then fed into non-linear SVM classifier to perform classification. Experiments are performed for times each of which corresponds to 10-fold evaluation as Sun al. (2020), with mean and standard deviation of accuracies (%) reported.
Compared methods. We adopt nine baselines that are composed of three categories. The pub- lished hyper-parameters of these methods are adopted. The first set is three SOTA kernel-based methods that include GL Shervashidze et al. (2009). WL Shervasssee et al. (2011) and DGK Ya- nardag & Vishwanathan (2015). The second set is four heuristic self-supervised methods. cluding node2vec Grover & Leskovec (2016), sub2vec Adhikari et al. (2018), graph2vec Anna- malai Narayanan Jaiswal (2017), and InfoGraph [Sun al. (2020). The final compared methods GraphCL You et al. (2020). (0) You et al. (2021). AD-GCL Suresh et al. (2021). Auto- GCL Yin et al. (2022) and RGCL Li et al. (2022b).
The results of LGCL along with SOTA competitors on three benchmarks are shown in Table A.3 To summarize, the proposed graph contrastive learning framework with the line graph, LGCL, obtains superior performance compared with the previous works. In particular, except for NCI1, LGCL achieves the best performance on two out of three benchmarks. Thus, we can conclude that LGCL captures the molecular semantic information well in the setting of unsupervised learning.
Results.
Table A.3: Average accuracies (%) Std. of compared methods via unsupervised learning. Bold indicates the best performance over all methods. Italic marks the second best performance.
NCTI
PROTEINS
MUTAG
GL
62.33
71.75
81.6612.11
WL
80.01+0.50
72.92
80.7243.00
DGK
80.31+0.46
73.30+0.82
87.44+2.72
node2vec
54.89+1.61
57.49+3.557
72.63+10.2
sub2vec
52.84+1.477
53.03=15.55
61,05+15.8
graph2vec
73.22++.81
73.30+2.005
83.15+99.25
InfoGraph
7.020+1.06
7..44
89.01+1.13
GraphCL.
77.87+0.41
74.39+0045
86.80+1.34
JOAO
78.07+0.47
74.55=00.41
87.35+1.02
JOAOv2
78.36:10.53
74.07-11.10
87.67+0.779
AD-GCL
75.86-10.62
75.04
88.62+1.27
AutoGCL
82.00
75.80-10.36
18.64
RGCL
18.14
75.03+0043
8..66+1.011
LGCL.
79.76-10.46
76.47+0.39
90.32+1.14
B.2 SEMI-SUPERVISED LEARNING
Datasets. Two sparse bioinformatics datasets are adopted from TUDataset Morris al. (2020) for semi-supervised learning, including NCI1 and PROTEINS.
Configuration. ResGCN with 128 hidden units and 5 layers is set up in semi-supervised learning. For all datasets we perform experiments with 10% label rate for 5 times, each of which corresponds to 10-fold evaluation as You et al. (2020), with mean and standard deviation of accuracies (%) reported. For pre-training, learning rate is tuned in {0.01, 0.001, 0.0001} and epoch number in (20, 40, 60, 80, 100) where grid search is performed. For fine-tuning, following the default
16

---

Under review as a conference paper at ICLR 2023
setting in You al. (2000) that is, learning rate is 0.001, hidden dimension is 128, bath size is 128, and the pre-trained models are trained 100 epochs.
Learning protocols. Following the learning setting in SOTA works, the corresponding learning protocols are adopted for fair comparison. In semi-supervised learning You et al. (2020), there exist two learning settings. For datasets with public training/validation//ii performed only on training dataset, finetuning is conducted with 10% of the training data, and final evaluation results are from the validation/test sets. For datasets without such splits, all samples are employed for pre-training while finetuning and evaluation performed over 10 folds.
split, pre-training is
Compared methods. Under the setting of semi semi-supereised learning, eight baselines are adopted, including (1) the naive GCN without pre-training You et al. (2020), which is directly trained with 10% labeled data from random initialization; (2) GAE Kipf & Welling (2016), predictive method by edge-based reconstruction in the pre-training phase; (3) Infomax Velickovic et al. (2019), node embedding method with global-local representation consistency; (4) ContextPred Hu ct al. (2020), a method via sub-structure information preserving; (5) GraphCL You et al. (2020). the first graph contrastive learning method with data augmentations. (6) JOAO(v2) You al. (2021).
opti- mization framework to automatically select data augmentations; (7) AD- Suresh et al. (202I), framework aim exclude redundant information during the training by optimizing adversarial graph augmentation strategies; (8) AutoGCL Yin et al. (2022), a model with learnable graph view generators orchestrated by an auto augmentation strategy.
Results. The results of LGCL along with SOTA competitors on the two benchmarks are shown in Table A..4 in which LGCL suppresses the SOTA view generation works on the two employed datasets. Thus, can conclude that LGCL captures the molecular semantic information well in the setting of semi-supervised learning.
Table A.4: Average accuracies (%) + Std. of compared methods via semi-supervised representation learning with 10% labels. Bold indicates the best performance over all methods. Italic marks the second best performance.
NCIT
PROTEINS
No Pre-Train
73272=0.24
70.40=1.551
GAE
74.36=0.24
70.51
Infomax
74.86=0.26
72.27=0.40
ContextPred
73.00=0.30
7..23+0063
GraphCI.
74.63=0.25
74.17+0.34
JOAO
74.48=0.27
72.1340.92
JOAOy2
74.86-0.39
73.31
AD-GCL
75.18=0.31
73.96=00.47
AutoGCL
73.33=2.86
74.57+3.29
LGCL
75.82=0.28
74.87
C FURTHER ABLATION STUDY
C.I HYPER-PARAMETER SENSITIVITY
In the design of LGCL, besides the general hyper-parameters (i.e., learning rate, batch size, dropout ratio, etc.), we introduce two hyper-parametes a and 3, for loss balance in pre-training stage. To clearly show the essential effectiveness of the two losses rather than the two hyper-parameters fix a and 3 to in the main text. The other pyrer--arameters in pre-training phase are also fixed and consistent with Hu al. 2020)
sensitivity of LGCL, we tuned the candidates of and B in the range of [0.01, 0.1, 1, 10, 100], respectively. In the tunning of a, fix the 3 to I and vice versa. In the fine-tumin monituning stage, the learning is fixed to 0.001; the batch size is fixed to 32; the dropout ratio is fixed 0.5. The node representations for graph pooling are adopted from the last layer. All
To further inspect the
17

---

Under review as a conference paper at ICLR 2023
S 8.8.. 8 88888
ROC-AUGT
612
a
622
01.
10"
12"
10' BACE (A)
SIDER (0)
ClinTex (o)
88
00.0
00.3
ROC-AUG
638
92.
72-
SIDER (8)
Cin (B)
BACE (8)
Figure A.I: Sensitivity w.r. hyper-parameter a and 3.
experiments on each dataset are performed for ten runs with different seeds, and results are averaged ROC-AUC scores (%) + standard deviations.
The average ROC-AUC scores of downstream tasks are shown in Figure A. As can be seen, different downstream tasks prefer different loss controls. Specifically, Clin and BACE prefer small but large B, while SIDER would like large a and insensitive to 3, which suggests that ClinTox and BACE suffer more from the over-smoothing. It is worth noting that more superior results can be obtained from tuning these two hyper-paramen parameters, such as the 85.88% Clin'ox with 1 and / 10 compared to 77.59% in the main text with and B 1, which implies the huge potential of LGCL with the tuning of the two hyper-parameters.
C.2 EFFICIENCY
In this paper, to address the issues of molecular semantics alteration and generalization capability in molecular contrastive learning, we introduce the line graph and further propose edge attribute fusion, intra-local contrastive loss, and inter-local contrastive loss to enhance contrastive learning. In Section we have shown the superior performance of L.GCL. Here, we further present the efficiency of LGCL. by comparing the pre-training time on 2 million molecular graphs of ZINC15 with the baselines. In addition, the actual runtime of LGCL through introducing each item is also given, including LGCL with only line graph (i.e., LGCL w/o AF LIntraC LinterC); LGCL with line graph and edge attribute fusion (i.e., LGCL w/o Lntrae LinterC); LGCL with line graph, edge attribute fusion and intra-local contrastive loss (i.e., LGCL wlo LinterC). The setting of pre-training time comparison consistent with the main text Hu et al. (2020). All baselines and LGCL are rerun in our platform (Tesla V100 GPU and Intel(R) Xeon(R) Silver 4214 CPU). Note that, set the num workess dataloader to default for fair comparison.
GraphCL
LGCL-wfo AF Eietnc futee
JOADv2
LGCL-woo Éetuuc Crart
AD-GCL
AutoGCL
LGOL-wfo
RGCL
LGCL
LGCL
Pre-train Time (Hours)
Pro-train (Houm)
(a) Comparison between baselines and LGCL.
(b) Comparison among parts of LGCL.
Figure A.2: Pre-training time comparison. The time required by LGCL is much less than the time
needed by baselines except AutoGCL.
Figure A.2 shows the pre-training time required for 2 million molecular graphs from ZINC15 pre- training with 100 epochs. As shown in Figure A.2al because the contrastive views of LGCL static, the time required by LGCL is much less than the time needed by baselines except AutoGCL, which reveals that LGCL has not only superior performance but also excellent efficiency. In partic-
18

---

Under review as a conference paper at ICLR 2023
ular, LP-Info requires almost 16 hours for epoch pre- pre-training, that is about 1,600 hours for 100 epochs, thus we do not report its pre-training time in Figure A.2a
Furthermore, shown in Figure A.2b comparing to the model only with line graph (i.e., LGCL w/o AF LintraC Literr),, the additional time from the introduction of edge attribute fusion is nearly negligible. The biggest time consumption gap comes from the proposal of two local contrastive losses, but they still only occupy 16.26% of the total time, which is a small price in contrast to the accompanying performance boosting.
C.3 VISUALIZATION OF OVER-SMOOTHING ALLEVIATING
Count
Donstyy
Denett
Test 0.000
KS
Test P-00lue-0.00
(a) Node similarity distribution of GraphCL and (b) Node similarity distribution of GraphCL and LGCL..
LGCL w/o Lintrac & LinterC.
Figure A.3: Visualization of intra-local contrastive loss in over-smoothing alleviating. We ex- amine the two groups of distributions by a Caimogorov-Smmrnne test (KS test), where the KS test p-values show that the two groups of distributions are distinct (i.e., the p-values are less than 0.01).
In the ablation study, we empirically inspect the effect of intra-local contrastive loss in over- smoothing alleviating. Here, a visualization case study is presented to further directly show how the intra-local contrastive works. Specifically, we calculate the similarities among nodes within the same graph from the pre-training dataset via the model after 100-epoch pre-training. As shown in Figure A.3 we give the node similarity distribution of GraphCL., CraphCL,LLGCL, L.GCL, and LGCL without the two local contrastive losses. First, based on the distribution shown in Figure we can see that the in- troduction of line graph leads to right shift of node similarity distribution, that is, over-smoothing Then, the left shift of node similarity distribution in Figure A.3b suggests that the proposed loss could alleviate the over-smoothing.
C.4 VISUALIZATION OF INFORMATION INCONSISTENCY ALLEVIATING
LGCL wis Al faosc Poac LOCL eis Curn: from
LOCL wi Af fooac foxe LOCL wis feome fooe
Loss Barch
Doniky
020
608
231
P.value=0.00
Epoch
120
(a) Positive view similarity distributions of LGCL (h) Pre-traning loss curves of L.GLL with and without with and without edge attribute fusion.
edge attribute fusion.
Figure A.4: Visualization of edge attribute fusion in information inconsistency alleviating. We examine the two view similarity distributions by a Kolmogorov-Sminng Smmirov test (KS test), where the KS test p-values show that the two groups of distributions distinct (i.e., P-value < 0.01).
19

---

Under review as a conference paper at ICLR 2023
In the ablation study, empirically validate the effectiveness of edge attribute fusion in information inconsistency alleviating. Here, visualization case studies are further presented to show how edge attribute fusion works. First, we calculate the view similarity of positive pairs from the pre-training dataset via the model after 100-epoch pre-training. As shown in Figure A.4a we present the view similarity distribution of LGCLL with and without the edge attribute fusion We can that the introduction of edge attribute fusion significantly increases the view similarity between positive pairs, Second, we inspect the convergence NT-Xent loss of LGCL with and without edge attribute fusion. As shown in Figure A.44) LGCL equipped with edge attribute fusion converges faster and has lower loss, which suggests the higher alignment of the view embeddings. summary, edge attribute fusion can address the information inconsistency accompanying the line graph.
D OTHER BASELINES
Here, we further compare LGCL with other suler-sue molecular representation learning:
(2020) predicts the contextual properties based on atom embeddings to encode contextual information into node embeddings.
GROVER Rong et al.
MGSSL Zhang et al.
2021 aims to capture the rich information in graph motifs.
33D-Infomax Stärk et al. (2022) aims to reason about the geometry of molecules given only their 2D molecular graphs by pre-training model with existing 3D molecular datasets.
The results are shown in Table A.5 As can be seen, LGCL suppresses GROVER on all datasets, MGSSL on six out of eight datasets, and 3D-Infomax on six out of seven datasets. Moreover, LGCL also achieves the highest average results among the these baselines.
Table A.5: Average test ROC-AUC (%) + Std. over different 10 runs of LGCL along with all baselines on eight downstream molecular property prediction benchmarks. The results of baselines are derived from the published works. Bold indicates the best performance among all baselines. Avg. shows the average ROC-AUC over all datasets. A.R. denotes the average rank. indicates the data missing in the such works.
Dataset
BRAP
Tox21
ToxCast
SIDER
Clinfox
MUV
HTV
BACE
Avg.
GROVER
6..011.3
76.310.6
6..4006
60.710.5
76.9199
7581117
77881.44
79.510.88
72.3
MGSSL(DFS)
70.5:11.1
76.4200
63.8=033
60.5:0.7
79.7+2.2
78.1=188
79.5=1111
79.7=20.8
73.5
(GSSL(DDS))
69.71.09
76.5.10.3
64.1.0.7
61.810.88
80.712.11
78.711.
79.1.10.9
3D-Infomax
69.11
74.46:20.74
64.41-088
53.3743.34
59.43:3.21
76.08:1.33
79.42=1.94
3D-Infomax+
19.64=2.19
73.73+20.69
63.95-0.38
58.43-+1288
83.59:3.64
75.38=0095
79.28-13.61
L.GLL.
70.99-1.05
76.95+0..3
64.71-0.72 0.72
63.37-0.56
1.,59+1.54
77:30+3.00
78.69+1.10
84.68-00.33
74.33
E THEORETICAL UNDERSTANDING OF LGCL
Besides the superior performance of LGCL shown in the main text for molecular property prediction, here, we further present theoretical understanding of how LGCL obtains better performance.
Definition E.1. (Graph Quotient Space). Define the equivalence > between two graphs G1 = G2 if G1. G2 cannot be distinguished by the 1-WL test. Define the quotient space G = G/ =.
So every element in the quotient space, i.c., G E G, a representative graph from a family of graphs that cannot be distinguished by the 1-WL test. Note that our definition also allows attributed graphs.
Theorem E.2. Suppose G is a countable space and thus G! is a countable space. Because and G' are countable, PG and Poy are defined over countable sets and thus discrete distribution. Later we may call a function 2(.) can distinguish two graphs G1, G2 if 2(G1) + 2(G2). Moreover, for notational simplicity, we consider the following definition. Suppose the encoder f implemented by a GNN. The optimal encoder I is the best model which GNN can find. Because f is as powerful as the 1-WL lest. Then, for any two graphs G1, G2 E G, G z G2, (G = f (G2).. We may define a mapping over G', also denoted by f which simply satisfies (G) : (G),, where G G',
and G E G and G' E G'. Suppose t(() is the data augmentation function and L(-) is the line graph transformation function. We have
The two models are both not equipped with the two local contrastive losses
20

---

Under review as a conference paper at ICLR 2023
1. [[L(G):G) G(");
2. I(L(G):Y) (((());))
The statement in Theorem E.2 indicates that the retained information after line graph transforma- tion from given graphs is more than contrastive views with data augmentation.
The statement 2 in Theorem suggests that the essential information underlying the line graph for larget prediction is more than contrastive views with data augmentation.
Proof. Given G, G = L(G) is an injective deterministic mapping. Therefore, for any random variable Q,
(13)
Of course, we may set Q G. So,
([G]):))I((GG))
(14)
Then, we have
T(((G);;")
(15)
where (a) is because the data processing inequality Cover 1999). Moreover, because f could be powerful the 1-WL test and is injective in g Meanwhile, as stated in the Whitney graph isomorphism theorem Whitney][1932 the isomorphism of two line graphs is judged to be consistent with the corresponding two original graphs, thus have
I(L(G)); (L(G)) (L
G(L(G);)).
(16)
Here, the second equality is because the transformation of line graph will not change the isomorphic relationship between two graphs G" and G. meanwhile (²' ('" statement I:
(G). Therefore, achieve the
I[L(G); G) 2 I(t(G)):G')
(17)
Again, because by definition f argmar (G); G), f must be injective. Given G", G* f" (G") is injective deterministic mapping. Of course, may set Q Y. So,
(G) = I(G Y).
(18)
Because = L(G) an injective deterministic mapping.
((f = =I(L()):Y)
(19)
Further because of the data processing inequality Cover (9999.
I(f (G): = I(G;Y)
2 T(I(G);))
=I(M((G)))))
(20)
Combining above equations, we have the statement
I(L(G);Y) (LG))
(G)Y)
2 ((())
=[(
(21)
which concludes the proof of the essential information of line graph.
21


Reviews:
[{"review_id": "k8MSBBJHWb", "reviewer": "Reviewer_KhWt", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a pre-training strategy for graph neural networks based on line graphs. The method is then benchmarks on several molecular property prediction tasks.", "summary_of_the_review": "The proposed pre-training strategy is reasonably novel, altough not well embedded into related work. The results and conclusions are hardly relevant since the predictive quality is even below classical molecular descriptors. There are several technical errors in the experimental part. ", "strength_and_weaknesses": "Strengths:\n- The work approaches a relevant problem, how to pre-train molecular encoders\n- Clearly written and structured paper\n\nWeaknesses:\n- The lacks embedding into related works outside of this small community of graph pre-training\n- The claimed improvement is hardly relevant since even the most simplistic molecular descriptors (Morgan, ECFP) produce better representations\n- Only a single experiment has been performed\n- There are technical errors, such as the lack of error bars and confidence intervals, and unclear model and hyperparameter selection \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "jZxzBZq2FE", "reviewer": "Reviewer_3j8n", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new framework for molecular property prediction. The proposed framework consists of (1) a new edge attribute fusion-based graph neural network architecture and (2) three pretraining objectives. The newly proposed GNN architecture additionally uses line graph-based view of a molecule as an additional source of information. Furthermore, the contrastive learning objectives consist of (1) contrastive learning between graph-based and line graph-based molecular representations, (2) intra-contrastive learning between edge representations, and (3) inter-contrastive learning between edge representations. After combining all the components, the proposed method outperforms the existing pretraining frameworks for molecular property prediction.", "summary_of_the_review": "This paper proposes an interesting idea for pretraining of molecular representations. However, I am concerned by the empirical evaluation which is not thorough enough to validate the significance of the proposed work.", "strength_and_weaknesses": "Strength 1: The idea of using line graph as an additional view for a molecule is novel for pretraining. \n\nStrength 2: The proposed method shows strong empirical performance.\n\nWeakness 1: Most importantly, the proposed method is combination of many ideas, but the current experiments are insufficient to demonstrate significance of such ideas. \n\nWeakness 1.1: In the ablation studies, simply combining line graph and contrastive learning achieves average performance of 71.16, which is worse than six of the considered baselines. \n\nWeakness 1.2: After combining edge attribute fusion, the proposed method generates property prediction using different neural network architecture compared to the baselines. If this is the case, the authors should consider the same architecture for the baselines too.\n\nWeakness 1.3: The proposed method is based on combination of three loss functions, while most of the baselines rely on one loss function. To measure the significance of these loss functions, the experiments should be constructed more carefully. For example, one could fix the budget for tuning the hyperparameters and try to combine the previously proposed pretraining objectives.\n\nWeakness 2: Table 2 does not have any “overall” statistics like ranking and average, so it is hard to interpret the table and make conclusion.  \n\nWeakness 3: I suggest comparing with 3D-infomax [1] and MGSSL [2] as a pretraining baseline. I also suggest comparing with works that simultaneously propose graph neural network (or Transformer) architectures with pretraining objective (GROVER [3], MolR [4]). Comparing with other architecture is important since the work also proposes a new architecture for property prediction. \n\nWeakness 4: I appreciate how the authors showed the candidate hyperparameters for the finetuning experiments. However, there is no detail on which metric is used to choose the hyperparameters. Furthermore, it is not clear whether if the budget used for tuning the pretraining hyperaparameter is the comparable across the baselines. \n\n[1] 3D Infomax improves GNNs for Molecular Property Prediction, ICML 2022\n[2] Motif-based Graph Self-Supervised Learning for Molecular Property Prediction, ICML 2022\n[3] Self-Supervised Graph Transformer on Large-Scale Molecular Data, NeurIPS 2020\n[4] Chemical-Reaction-Aware Molecule Representation Learning, ICLR 2022", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "XY4Nz1QIr-", "reviewer": "Reviewer_xTh3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "**(Motivation:)** This paper points out the limitations of existing graph contrastive learning, which not only alters the semantic of original graphs during perturbation, but also leverages the domain-specific knowledge that might not be generalizable across different graph datasets. \n\n**(Method:)** To tackle those two problems, the authors propose to use the line graph of the original graph, wherein the edge of the original graph is transformed to the node of the line graph. In particular, the authors maximize the representation similarity of original and its line graph, while minimizing other negative original-line graph pairs, under the existing contrastive learning objective. Also, the authors propose to minimize the inconsistency between node/edge features of the original graph and edge/node features of the line graph, by interchanging their information. Furthermore, the authors propose two additional contrastive losses, which capture edge-level similarities within and between graphs. \n\n**(Experiment:)** The authors validate the proposed LGCL on the graph classification tasks, showing the propose model outperforms relevant baselines.", "summary_of_the_review": "While there are some revision points: discussing more work and improving clarify of some sections, I believe they require minor revisions. Thus, given the high quality and novelty of this work with strong empirical results, I recommend the acceptance. ", "strength_and_weaknesses": "### Strengths\n* The idea of using line graph transformation for graph contrastive learning, where the positive pair consists of the original and its line graph, is interesting and novel. \n* The problem of information inconsistency between original and line graphs is well justified, which is nicely solved with the proposed edge attribute fusion method. \n* The proposed method empirically outperforms other competitive baselines.\n* This paper is well-structured, and easy to follow.\n\n### Weaknesses\n* There is a recent work [1] that points out the limitation of contrastive learning: altering the semantics of graphs during perturbation, and this problem-level idea is the same as the authors suggest. Thus, this relevant work [1] should be discussed, and might be compared. \n* The tackled over-smoothing issue incurred by the line graph is unclear. Why the line graph additionally introduces the over-smoothing issue? And how this over-smoothing issue is tackled by the proposed edge-level contrastive losses in Section 4.3 and Section 4.4?\n* The experimental setup of Figure 3 is unclear. Does \"No Pre-Train w/LG\" denote the fine-tuning model that uses representations of both original and line graphs? Then, for the proposed method, is the READOUT function in equation (3) changed to include the line graph representation?\n\n---\n\n[1] Graph Self-supervised Learning with Accurate Discrepancy Learning, NeurIPS 2022.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "OM-tBqPSfd9", "reviewer": "Reviewer_awP4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper claims that random or learnable graph augmentation methods may inevitably vary molecular semantics and make them inappropriate applied to molecular structure. The authors propose a novel approach to augment the original graph, i.e., converting it to a line graph. The edge attribute fusion tackles the inconsistent representation issue. Two following contrastive losses enhance representation learning.", "summary_of_the_review": "The paper would benefit from several revisions before publication, including adding some experiments, improving the model design, and checking the necessity of the components carefully.", "strength_and_weaknesses": "Strength:\n1. The paper is well-written and easy to follow. The storyline is clear, and the figures are intuitive.\n\n2. The authors are willing to share the code link after acceptance. The reproducibility of this paper is satisfactory.\n\n3. The experimental setting is very exhaustive. The authors describe almost all the interesting experimental details.\n\nWeakness:\n1. The experimental settings are kind of solid but not enough. The experimental section only includes overall performance comparison and ablation study in two different settings. The hyper-parameter sensitivity experiments are missing. And the authors report that the pretraining experiments of their model require 20 hours to converge. I am curious about the efficiency of their method compared with other methods.\n\n2. Some model designs are too simple to come up with. The idea behind the model is kind of interesting. But some model designs are too simple to come up with. For example, the GNN architecture and contrastive loss are similar to other works.\n\n3. The authors claim that edge attribute fusion is essential to fix the inconsistency problem between the representations from two views. But the experimental results shown in figure 3 demonstrate that the model performs worse on half of all the datasets with this attribute fusion.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "5cSq0Ty63iv", "reviewer": "Reviewer_TRA8", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper fouse on learning good representation for molecuars. The authors introudce line graph to graph contrastive learning and propose two contrastive loss (i.e., inter contrastive loss and intra contrastive loss) to retain molecular semantics. Experiment shows proposed LGCL ahieve SOTA performance on several datasets.", "summary_of_the_review": "The author give a trival solution to the learning the represeatation for molecuars. The novelty of this paper is not enough for ICLR.", "strength_and_weaknesses": "**Strength**\nPaper is well writen and easy to follow the idea\n\n**Weakness**\nOverall, the idea of this paper is not novel and the method propsed by the authors is incremental. \n1. The idea of introducing the line graph of the molecuars to graph representation learning is trivial. It has been explorer in the various GNN papers. Although this paper might be the first one to apply the line graph to the contastive learning, the effort has been made is minor and not significant. It just apply line graph as a contrastive veiws.\n\n2. There is no evidence to show that the proposed intra-local contrastive loss can allivate the oversmooth problem, however, the author claim for that. Need either thertical analysis or empericial result to show this.\n\n3. Some reuslt is much worse than the baseline, can author give some explaination on that.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject