Self-Labeling of Fully Mediating Representations by Graph Alignment
Authors: Martijn Oldenhof
Adam Arany
Yves Moreau
Jaak Simm
Venue:  (2021)
Keywords: domain adaptation
self-labeling
chemical graph recognition

Abstract: To be able to predict a molecular graph structure ($W$) given a 2D image of a chemical compound ($U$)  is a challenging problem in machine learning. We are interested to learn $f: U \rightarrow W$ where we have a fully mediating representation $V$ such that $f$ factors into $U \rightarrow V \rightarrow W$. However, observing V requires detailed and expensive labels. We propose \textbf{graph aligning} approach that generates rich or detailed labels given normal labels $W$. In this paper we investigate the scenario of domain adaptation from the source domain where we have access to the expensive labels $V$ to the target domain where only normal labels W are available. Focusing on the problem of predicting chemical compound graphs from 2D images the fully mediating layer is represented using the planar embedding of the chemical graph structure we are predicting. The use of a fully mediating layer implies some assumptions on the mechanism of the underlying process. However if the assumptions are correct it should allow the machine learning model to be more interpretable, generalize better and be more  data efficient at training time.
The empirical results show that, using only 4000 data points, we obtain up to 4x improvement of performance after domain adaptation to target domain compared to pretrained model only on the source domain. After domain adaptation, the model is even able to detect atom types that were never seen in the original source domain. Finally, on the Maybridge data set the proposed self-labeling approach reached higher performance than the current state of the art.

Under review as a conference paper at ICLR 2021
SELF-LABELING OF FULLY MEDIATING REPRESENTA- TIONS BY GRAPH ALIGNMENT
Anonymous authors Paper under double-blind review
ABSTRACT
To be able to predict a molecular graph structure (W) given a 2D image of a chemical compound (U) is a challenging problem in machine learning. We are interested to learn f U + W where we have a fully mediating representation V such that I factors into - and expensive labels. We propose graph aligning approach that generates rich detailed labels given normal labels W. In this paper we investigate the scenario of domain adaptation from the source domain where have access the ex- pensive labels to the target domain where only normal labels W are available. Focusing the problem of predicting chemical compound graphs from 2D im- ages the fully mediating layer is represented using the planar embedding of the chemical graph structure implies some assumptions on the mechanism of the underlying process. However if the assumptions are correct it should allow the machine learning model be more interpretable, generalize better and be more data efficient at training time. The empirical results show that, using only 4000 data points, we obtain up to 4x improvement of performance after domain adaptation to target domain compared to pretrained model only on the source domain. After domain adaptation, the model is even able to detect atom types that were never seen in the original source domain. Finally, on the Maybridge data set the proposed self labeling approach reached higher performance than the current state of the art.
W. However, observing V requires detailed
predicting. The of fully mediating layer
INTRODUCTION
Chemical compounds are often represented by graph representation of their chemical structure. These graph representations are actually simplification of the chemical compound as it loses some information about the electronic structure of the molecule. However, in the field of drug discovery this graph representation often used as valuable input for machine learning pipelines. Examples of formats describing the graph representation of chemical compounds are SMILES (36] and MOLfile (5). However, especially in patents but also scientific literature the chemical compound only described using image format. Automatically recognizing the chemical structures these images is valuable for machine learning approaches to be able to process these sources of chemical compounds.
Learning to recognize a graph structure from 2D images of chemical compounds seems like a fairly simple task for humans. However, for machine learning models seems that generalization to new domains of images (e.g. different line width, font face) [1 is not happening naturally. When we humans see an image with graph structure that we do not recognize completely, we start reasoning and analyzing the part of the graph we are not sure about. We humans automatically align the graph part we recognized on the image with the complete graph including the unrecognized part of the graph. One way to finish our graph prediction is to guess the unknown nodes edges after which we check for correctness. If the graph prediction was correct we know that this guess was most probably correct and could try to apply this new knowledge to other images.
To be able to do this reasoning on for example images using graph alignment in machine learning need detailed (on pixel level) representation. Therefore we assume fully mediated model 2 where we are interested to learn f U + W having a fully mediating representation V such that factors into U V W. which is visualized in Figure Thus, in order to predict

---

Under review as a conference paper at ICLR 2021
W
GRAPH
FULLY MEDIATING LAYER (planar embedding)
U
We are interested to learn ʃ U + W having a fully mediating representation V such that f factors into U + W. In the case of optical graph recognition of chemical compounds from 2D images, the fully mediating layer is represented using the planar embedding of the chemical graph structure we are predicting.
Figure I:
W from U we first need to pass the fully mediating layer, no side paths are allowed. When fully mediating representation is used some assumptions [3 26 25 are made about the mechanism the underlying process. This mechanistic prior restricts the space of possible models to all the models that follow the mechanistic assumption. We hypothesize that the use of this richer representation (fully mediating representation) enables for better generalization. Additionally, an interesting side effect, we observe that the mechanistic assumption allows for a better interpretability of the underlying model.
In the case of optical graph recognition of chemical compounds from 2D images, the fully mediating layer is represented using the planar embedding of the chemical graph structure order to learn the planar embedding of a chemical graph structure, we start from a model described in Oldenhof al. [21] which has two steps: image segmentation and image classification step. To train this model, pixel-wise labels are needed for every image describing precise locations of nodes and edges in the graph (planar embedding) which will call rich detailed labels in our setup. However, these rich labels are not always available and implies a manual process where intermediate organic chemistry knowledge is required. In the more common cases, data sets only contain 2D images of chemical compounds (U in Figure T and on the other side the final output in SMILES[36 or (5L format (W in Figure [T. These formats deseribe the graph structure of the chemical compound but not the particular planar embedding of this graph structure (V in Figure in the context the image. To solve this problem, we propose a graph aligning approach that generates rich labels V given normal labels W. This method would enable learning of the fully mediating representations given only normal labels W.
predicting, In
In section we empirically evaluate domain adaption method. We observe that compared to the non-adapted model we drastically increase accuracy even on atoms and bond that were not present in source domain.
Key contributions: (1) we propose a novel rich labeling framework by introducing the use of fully mediating representations, (2) in the of graph recognition show that the rich labeling can be performed by graph aligmment, (3) we show it enables data efficient domain adaption and (4) reaches state-of-the-art performance Maybridge compound data set.
2

---

Under review as a conference paper at ICLR 2021
2
RELATED WORK
Structural scene representation and visual reasoning. Our work has similarities with research done structural scene representation and visual reasoning [T] [4:1119 [9]. The disentanglement of the reasoning and the representation described in Yi et al. [41] enables the model to solve com- plex reasoning tasks. In our work the complex reasoning task would be graph alignment which is disentangled from the optical graph recognition.
Slot A Attention. Our method is related with method called slot attention [[7 where the Hungarian algorithm [16] is incorporated in a model for object detection. This Hungarian algorithm is limited to only sets while in our case we need to map more complicated structures composed of different atoms connected with different bond for which we need graph alignment in order to adapt iteratively a model to a new target domain.
Image to Graph methods. In the field of computational chemistry there are several tools avail- able (2) 34 (20 7 2 33] convert an 2D image of chemical compound a SMILES (36] format or similar which in fact represents graph structure of chemical compound. Also for road extraction from satellite images there are several methods available
Graph Matching. In computer vision graph alignment is usually known as graph matching. It
can be useful to (1) locate objects from features [10), (2) to transfer knowledge 42 and (3) to find matches in database [3] Also for comparing social networks graph matching can be very important to allow to uncover identities of communities [14. In chemistry, comparing graphs can be helpful to identify identical chemicals, substructures or maximum common part of chemicals. In the work of Willett et al. 37 an overview is presented about the use of similarity searches in chemical databases.
Domain Adaptation In the work of Kouw & Loog [1] comprehensive overview is given for domain adaptation methods when labels for the target domain are not available. Our method has some similarities with semi-supervised iterative self-labeling 4 27 approaches where predictions on a data set of a new domain of pre-trained model are used as pseudo-labels and used to retrain the model again iteratively until convergence. In the work of Das & Lee (6] even graph matching loss is first used to learn a domain invariant representation for source and target domain after which the use of pseudo-labels show a significant improvement of performance. In our work the graph matching is used for a different purpose as opposed to the the work of Das & Lee [6]. Graph matching is used in our work to generate rich labels given the 'normal' labels we have from the target domain. This is where our method also differs from other semi-supervised methods for domain adaptation when no target label information at all is assumed and no distinction is made between rich and 'normal' labels.
Weak Supervision In our setup we use the term 'rich' or 'detailed' labels to differentiate from the normal labels. We would like to contrast these 'rich' labels with the term 'strong labels' used in the setting of weak supervision. For example, in the machine learning task of image segmentation pixel-wise labels are needed which are expensive and often not readily available. Therefore, weak supervision methods have been developed to address this issue. Weak supervision can be used to help image segmentation by only using image labels (no pixel-wise labels) [55 38 A more general framework was presented Xu et al. 339 to be able to learn semantic segmentation from a variety of types of weak labels (e.g., image tags, bounding boxes and partial labels). Another approach is to augment the strong labeled data set using weakly labeled data [40]. However, the main difference with all of the methods mentioned above is that our method does not work on weak labels because our end goal is different. The main goal of our machine learning approach is to help to predict 'normal' labels by using rich labels.
Front Door Criterion Our framework exploits fully mediating variables. A variable is called a me- diator when it meets several conditions regarding the relationship with other variables as described in Baron & Kenny [2]. Another perspective of the mediating relationship is given by Pearl [33; Pearl et al. (26): Pearl (25) who introduce the front door criterion where the mediator actually enables to estimate unbiased causal effects. A more formal interpretation of these causal effects is presented in Pearl (24). In order to use a mediating model the mediator needs to be identified or assumed first, which is not always straightforward. In our setup (see Figure I.. the assumption means that the relation between input u E U and planar embedding v E V is a map and as well as the relation between e and the final graph E W. Furthermore, we assume no side paths from to w.
3

---

Under review as a conference paper at ICLR 2021
(a) Example of 2 chemical compounds with graph edit (b) Example of 2 chemical compound with graph edit distance of node substitutions.
distance of edge substitution.
Two examples of chemical compounds graphs with their graph edit distance. The nodes of the graphs first aligned before computing the graph edit distance. The node alignments marked with the gray dashed arrows. The differences after graph alignment are highlighted and the substitutions are marked with the red dashed arrows.
Figure 2:
SELF-LABELING OF FULLY MEDIATING REPRESENTATIONS
Our goal is to learn / + W assuming a fully mediating representation V such that / factors into U + V + W. In order to learn the first part of (U + V) we need labels for V which are expensive in the case optical graph recognition of chemical compounds from 2D images where V represented the planar embedding of the chemical graph structure. Our method tries to address this issue by iteratively updating the model using self-labeled labels for V by graph aligning the graph predictions using the model from previous iteration with the given true graphs (labels W).
3.1 GRAPH ALIGNMENT
A possible and often used closeness score to compare graphs is the graph edit distance 129 given graphs. not necessarily of equal size and set of operations, that are O (vertex/ /edge//labe insertion insertion/deletion// deleeeiionsuussttatttiiie ssubsiitutiin) and cost function c O + R2 so we find the cheapest sequence of operations that convert 91 into G2, which translates an optimization problem:
min
c(ex).
(e) w0..eo..... Go (o.
X
i=1
Although there are some efficient algorithms available 30 in order to compute the graph edit distance, it remains computational hard problem.
Closely related with the concept of graph edit distance we introduce for our method the map E(v) which gives the allowed operations parameter which can be tuned for a specific data set or problem domain. Examples of such constraints maximum 2 node substitutions maximum edge substitution shown in Figure
given graph given specific constraint. This constraint
3.2 METHOD
Let us now say we have a trained neural network model for f: f + V. a projection V + W, a pair (u, w) of input u E U and normal label w W and we would like to infer rich label v E V from the given datapoint (u, w). We also assume the map F(()) which gives all allowed graph edits for the graph v. Let é f(u) be the predicted rich label from the model, then we define a term correcting edit as
Definition 3.1 Edit is correcting edit if when e is applied to the prediction è and then projected the W space the resulting graph is the graph (up isomorphism), i.e.
Hle x 6) w.
where is the application of edit the planar embedded graph 6.
4

---

Under review as a conference paper at ICLR 2021
Notice that for given ê and there be multiple edits that correcting edits which create dilemma of choosing the best correcting edit. Therefore, we make the following assumption:
Assumption 3.1 monotonely decreasing with respect the size of edit (i.e., (el).
The probability that a correcting edit e results in the tre underlying rich label
In other words, if we take two correcting edits e1, then we assume the following:
leik< [eal
P(ei x u) x P(e2xx==))
The assumption based on the fact the probability of any individual mistake in a graph by the model is low. This is because if the probability of mistake would be high the model would not be able to produce a graph with a total of 1-2 edit distance. Thus, the graphs with few edits have low mistake probability and for them the Assumption 33.1 valid.
Then we use the following optimisation problem to find the best correcting edit e to convert to rich label for input u:
arg min le] such that d(E 6) = w, e(F
where arg min returns the set of minimal solutions or the empty set if no solutions exist.
There are three possible outcomes of last mentioned optimization problem: (1) no solution is found, (2) a single is found or (3) multiple equal size are found. In the optimal case (2) single is found so we can label a new e for our given datapoint (u, v). the case of (1) when no solution found,
v is labeled. In the last case (3) when multiple equal size solutions are four options we could do. First (3.1), we could discard the solutions and not label u. Second (3.2), we could take that results in the highest likelihood for x é based the model f. Third (3.3), a solution e is picked uniformly randomly in order to generate the rich label label v. Fourth (3.4), pick randomly according to the likelihood of i in the model f-
found there
This process is repeated for every datapoint (u, w) we have available from the target domain. Thus, several new labels are found for different datapoints. Once all datapoints processed these new rich labeled datapoints are added to the training data set after upsampling and our model can be retrained. After this, new iteration begins and all available datapoints (u, v) are again processed to find even more new rich labels v and we can retrain the model again. This iterative process can be repeated until convergence (see Algorithm I
Algorithm 1: Iterative algorithm for Self Self-Labeling of Fully Mediating Representations
Data:
Target domain data L {(u), ) Source domain data S = ((u), of) 3=1 (rich labels)
Result: f V
repeat
// Inferring rich labels for target data
T = []:
for (u, w) in do
è f(u);
E + min such that o(e x 6) = w;
eEF(()
if Ed not empty then
e - choose(E*); é; appendRtchLabels(T. (u, v));
end
end
T - UpSample(T), (T);
f RetrainModel(S, T);
until Converged
5

---

Under review as a conference paper at ICLR 2021
Dataset
Orig. Size
Tot. #samples
Cand. rich labeling #samples
Test #samples
Indigo
50,000
5,000
4,000
1,000
Maybridge
5,740
5,000
4,000
1,000
Table Summary of datasets from the 2 different target domains
4 EXPERIMENTS
For the experiments we focus on the problem of predicting chemical compound graphs from 2D images where the fully mediating layer is represented using the planar embedding of the chemical graph structure we are predicting. In order to measure empirically the performance of our method of self-labeling fully mediating representations we perform three steps. (1) We pre-train (training
details in Appendix A.2) a ChemGrapher [2I] model (summarized in Appendix [A.I) wherefore, corresponding the pipeline described in the work of Oldenhof et al. (2I], we sample around 130K chemical compounds from ChEMBL [9] in SMILES format and artificially generate, using an RDKit fork [1, a rich labeled dataset with 2D images of chemical compounds.
(2) Secondly, we test the baseline performance of this pre-trained model on two different test sets from two different target domains than the source domain of the pre-trained model. (3) Thirdly, we apply our domain adaptation method and measure performance again the two target domains.
For the first target domain we take a data set from the work from Staker et al. [33], which we will call Indigo data set. For the second target domain we take the data set which was published by the developers of MolRec [8] which we will call the Maybridge data set. Both data sets provide 2D images from chemical compound together with corresponding identifier of the chemical compound like SMILES [36] or MOLfile [5]. These identifiers describe the graph structure of the chemical compound however they do not provide the planar embedding of the graph (e.g. no information about the pixel coordinates of every node or edge in the image). Visually we can also observe that the Maybridge dataset contains images where the style is closer related the training images style used for the pre-trained model compared with the images in Indigo dataset where the style of images is quite different. Therefore we expect a significant worse starting performance of the pre-trained model on the Indigo dataset compared with the Maybridge dataset.
From both data sets we randomly sample 5,000 datapoints which are split in 4,000 datapoints used for our method and 1,000 datapoints to measure performance on (summarized in Table D. When processing the 4,000 datapoints our method will be able to generate rich labels for the datapoints where the graph prediction could be graph aligned with the true graph. As the number of rich labeled datapoints this way is maximum 4,000 we will upsample them (x number of copies) before adding them to the training data set. In our experiments we differentiate between two strategies of upsampling. One way is to upsample all the rich labeled data points equally from the target domain to fixed number, for example 20,000, Another way is to take into account, while upsampling, the number of atom types that are rich labeled and make sure that the rare atom types are upsampled to specific threshold.
One important tuning parameter in our method is the number of allowed operations. For our ex- periments we will try two differrnnt values for this parameter. Firstly, we set this parameter zero meaning we do not allow any operation for graph alignment. We will call this exact graph align- ment. Secondly, we allow a maximum of 2 node substitutions maximum of I edge substitution for graph alignment, which we will call correcting graph alignment.
In total we will measure the performance of variations of our method (varying allowed operations and upsampling strategy) on both data sets. The performance we will measure is the accuracy of U -> W as we only have access the normal labels of target domain. However, we assume that if the final graph prediction is correct (W) it is highly likely that also the planar embedding (V) is correct. As our method is an iterative method we will report results for every iteration starting with the initial performance before applying our method. The results of these experiments are summa- rized in Figure S
We observe that all variations of our method are able to improve performance on target domain compared with initial pre-trained model on source domain. On the Indigo data set the best variation even able to obtain 4x improvement. The best variation of our method on the Indigo data set was using correcting graph alignment without upsampling of rare atom types

---

Under review as a conference paper at ICLR 2021
40
corr_a
exex graph
84-
82
percentage
80
78
13
corr. gww_grrophrgrg
corr. graph
74
exact, greah
greottrare
72
Ner Veretionn
Aerations
(a) Results on Indigo data set
(b) Results on Maybridge data set
Figure 3: Comparison performance methods on Indigo and Maybridge data set. Self-labeling by correcting graph alignment is clearly better performing than when exact graph alignment is used. Sometimes upsampling of rare atoms to a specific threshold (note postfix rare) before retraining of model can boost performance. Performance target domain at iteration is the performance of pre-trained (on source domain) ChemGrapher before domain adaptation.
while on the Maybridge data set the best variation was also using correcting graph alignment but with upsampling of atom types. Some of the underperforming variations of method were stopped early in order to save computational resources.
stence
80
R-group
percentage correct
60
porcentage
40
ND2
3
Iterations
tteration
(a) Results on Indigo data set
(b) Results on Maybridge dataset
Figure 4: We take the best performing methods and analyze their performance on different atom and bond types per iteration. We observe that for some atom types the method is able to increase performance even though initial performance was 0%. This is the case in for example R-groups in Indigo data set superatom NO₂ in Maybridge data set.
We choose the best variation of method for every data set and analyze the performance different atom and bond types per iteration. We measure for every atom or bond type the percentage of graphs predicted correctly from the total number of graphs containing that specific atom or bond type per iteration, which is visualized in Figure 4 Most of the performances of the different atom and bond types increase per iteration for both data sets even when initial performance was 0%.

---

Under review as a conference paper at ICLR 2021
CHz
CHa
R₂
N-N
(a) Input image
(b) Initial Segmentation
(c) Final Segmentation
Figure 5: Comparison initial segmentation with final segmentation after applying self-labeling of fully mediating representations for Indigo data set. We observe that the initial model making mistakes on the R R-group atom type and carbon represented with 'C'. In the final model we see that now predictions are all correct.
O.N.
Ci
CI
(a) Input image
(b) Initial Segmentation
(c) Final Segmentation
Figure 6: Comparison initial segmentation with final segmentation after applying self-labeling of fully mediating representations for Maybridge data set. The initial model predicts the superatom NO₂ two separate atoms O and N which is chemically not correct. The final model makes the correct prediction.
The atom types where initial performance was 0% are atom types never seen before in source do- main. For example in the Indigo data set there are compounds with atom labels like R1, R2 and R3 representing R- R-groups which were not present in the original data set from the source domain. For illustration purposes we visualize in Figure the segmentation step which forms part of the graph recognition model used in this study. In the initial segmentation from the pre-trained model we can clearly see that the model confuses the R-group atoms with the oxygen atom type and the hydrogen atom type. After applying our method the model is able make correct predictions. In the same Figure we also observe that the Indigo data set carbon sometimes also represented using a which was never the case in the original data set. The initial segmentation mainly confuses these carbon atom types with the oxygen atom type. After applying our method the model again makes the correct prediction.
Similarly, the superatom NO₂ present in the Maybridge data set was never observed in the source domain. However, again after applying method the model is able detect superatom NO, correctly. We illustrate the segmentation step of the graph recognition model in Figure 6 for an example image taken from the Maybridge data set. We observe that in the initial segmentation the pre-trained model confuses NO, with nitrogen atom and also oxygen atom which chemically is not the correct prediction. In the final segmentation after applying two iterations of our method the newly trained model is able to make the correct prediction.
Additionally Figure and Figure show an interesting side effect when using fully mediating representation. Consider classical model where input is an image and output is SMILES. When the output prediction of the model incorrect was made but in the expert can see where and how the mistakes happened. This makes the model more interpretable.
clear in which part of the image the mistake of having available the planar embedding (mediation representation) the
Finally we compare in Table resulting best performance of the model after applying our method on the Maybridge data set with several other methods available. We observe that our approach enables to reach higher performance than the current state of the art. For the freely available tools OSRA 7 and Molvec (28) we measured the performance using the same randomly 1000 datapoints from the Maybridge dataset. For MolRec [18) this was possible but report for information
8

---

Under review as a conference paper at ICLR 2021
Method
Training Dataset
Maybridge Dataset
Test
Accuracy
Source domain
Target domain
OSRA (v2.1.0)
N/A
N/A
same random 1,000
80.4%
Molvec (v0.9.8) 28
N/A
N/A
same random 1,000
78.4%
ChemGrapher [2]]
130K images
N/A
same random 1,000
72.6%
ChemGrapher
21
130K images
40 manually handpicked and rich-
same random 1,000
81.6%
(using manually rich-labeling)
labeled images
(upsampled)
Proposed domain adaptation
130K images
4,000 non-rich
same random 1,000
86.3%
labeled
MolRec [88
N/A
N/A
Total 5740
83.8% from IR
Table Comparison performance on Maybridge data set. observe that our approach enables to reach higher performance than the current state of the art. Most of the tools available for chemical graph recognítion are rule based approaches for which a training dataset is not relevant.
the performance on the total Maybride dataset as reported in the work of M. Sadawi et al. [18] Finally for ChemGrapher [2] we measured performance using three different training datasets. Firstly, we measure the performance when we only have access to the source domain (generated using RDKit (I). Secondly, we measure performance using the same training dataset from source domain but adding upsampled (100 copies) 20 handpicked manually rich labeled datapoints from target Maybridge domain (as was done in the work of Oldenhof al. (2T).. Finally, instead of manually rich labeling datapoints,
process the 4,000 datapoints from Maybridge target domain
where our method will be able to generate rich labels for the datapoints where the graph prediction could be graph aligned with the true graph, after which these rich labeled datapoints added to the training dataset.
5 CONCLUSION
Machine learning models often are faced with the problem not generalize well to new domain. This is also the case for chemical graph recognition from images. We have shown that fully me- diating layers be exploited in machine learning models to adapt in data efficient way to domains, without the need of rich expensive labels as they can be generated using our method. In the case of chemical graph recognition new domain of chemical compounds, with previously unseen atom or bond types. Our rich-labcling method required only 4,000 normal labeled points in the target domain go from 10% accuracy to 39%, i.c., almost 4x improvement in the difficult Indigo data set. Furthermore, on Maybridge data set, again using only 4,000 images, we reached high accuracy obtaining better performance than the current state of the art.
empirically show that method is able adapt
Effective tools of chemical structure recognition from images enable access to the knowledge in chemical literature which is currently only available through expensive chemistry databases. We believe as an important step towards open pharmaceutical science.
It would be interesting to apply this method to other contexts where the output of machine learning model could be represented with a graph structure. For example, the case structural scene repre- sentation, where scene could be represented using graph where every vertex could represent an object and every edge would represent the relations between the objects (e.g. side-by by-side, on-top- of, under). This structural scene could be in form of 2D images or it could be even generalized to 3D models, where point clouds are available and one is interested to transform them into 3D graphs of connected parts.

---

Under review as a conference paper at ICLR 2021
REFERENCES
[1] Fork of the official sources for the rdkit library. URL htttss://ithuu: com/ accessed on 12.11.2020.
biolearning stllleariig--staduuss//
[2] Reuben M Baron and David Kenny. The moderator-mediator variable distinction in so- cial psychological research: Conceptual, strategic, and statistical considerations. Journal of personality and social psychology, (1))1 1986.
[3] Davide Belli and Thomas Kipf. Image-conditioned graph generation for road network extrac- tion. arXiv preprint arXiv: 14388, 2019.
[4] Lorenzo Bruzzone and Mattia Marconcini. Domain adaptation problems: A dasvm classifica- tion technique and circular validation strategy. IEEE transactions on pattern analysis and machine intelligence, 32(5):770-777, 2009.
[5] Arthur Dalby, James G. Nourse, W. Douglas Hounshell, Ann K. I. Gushurst, David L.. Grier, Burton A. Leland, and John Laufer. Description of several chemical structure file formats used by computer programs developed at molecular design limited. J. Chem Inf. Com- put. Sci., 32(3):244-255, 1992. (ttt:tt:: //pbs acs. org/ dwi//bsi///////// 0taps/10..0 10210 c100037a012
[6] Debasmit Das and CS George Lee. Graph matching and pseudo-label guided deep unsuper- vised domain adaptation. In International conference artificial neural networks, pp. 342- 352. Springer, 2018
[7] Igor V. Filippov and Mare C. Nicklaus. Optical structure recognition software to recover chemical information: Osra, an open source solution. Chem. Inf. Model., 49(3):740-743, 2009.
[8] Lin Gao, Weidong Song, Jiguang Dai, and Yang Chen. Road extraction from high-resolution remote sensing imagery using refined deep residual convolutional neural network. Remote Sensing, 11(5:5552, 2019.
(9] Anna Gaulton, Anne Hersey, Michal Nowotka, A Patricia Bento, Jon Chambers, David Mendez, Prudence Mutowo, Francis Atkinson, Louisa J Bellis, Elena Cibrián- Gibrián-Uhalte, et al. The chembl database in 2017. Nucleic acids research, 45945)44945- 2017.
[10] Steven Gold and Anand Rangarajan. Graduated assignment graph matching. In Proceedings of International Conference on Neural Networks (ICNN 196,, volume 3, pp. 1474-1479, IEEE, 1996.
[11] Chi Han, Jiayuan Mao, Chuang Gan, Josh Tenenbaum, and Jiajun Wu. Visual concept- metaconcept learning. In Advances in Neural Information Processing Systems, pp. pp. 5001-550-511, 2019.
[12] Corentin Henry, Seyed Majid Azimi, and Nina Merkle. Road segmentation in sar satellite images with deep fully convolutional neural networks. IEEE Geoscience and Remote Sensing Letters, 15(12):1867-1871, 2018.
[13] Dakshina Ranjan Kisku, Ajita Rattani, Enrico Grosso, and Massimo Tistarelli. Face identifi- cation by sift-based complete graph topology. In 2007 IEEE workshop on automatic identifi- cation advanced technologies, pp. 63-68. IEEE, 2007.
[14] Xiangnan Kong, Jiawei Zhang, and Philip Yu. Inferring anchor links across multiple het- erogencous social networks. In Proceedings of the 22nd ACM international conference Information & Knowledge Management, pp. 179-188, 2013.
[15] Wouter Marco Kouw and Marco Loog. A review of domain adaptation without target labels. IEEE transactions on pattern analysis and machine intelligence, 2019.
[16] Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly, 2(1-2):83-97, 1955.
10

---

Under review as a conference paper at ICLR 2021
[17] Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. arXiv preprint arXi::2006. 5005, 2020.
[18] Noureddin M. Sadawi, Alan Sexton, and Volker Sorge. Chemical structure recoggitioonn rule based approach. Proc. SPIE, 8297:32-, 01 2012. URL httppss///e....r//11111111
12. 912185
[19] Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua B Tenenbaum, and Jiajun Wu. The neuro- symbolic concept learner: Interpreting scenes, words, and sentences from natural supervision. arXiv preprint arXiv: 1904. 12584, 2019.
[20] Joe R. McDanicl and Jason R. Balmuth. Kekule: Ocr-optical chemical (struccurr)) recognition J. Chem. Inf. Comput. Sci., 32(4):373-378, 1992. URL /ttpp:://@d..orr/11...
c100038a018
[21] Martijn Oldenhof, Adam Arany, Yves Moreau, and Jaak Simm. Chemgrapher: Optical graph recognition of chemical compounds by deep learning. arXiv preprint arXiv:2002.09914, 2020. doi: 10. 10.1002//aaa 1/acs22/acs,,po wew..0c000555
[22] Jungkap Park, Gus Rosania, Kerby A Shedden, Mandee Nguyen, Naesung Lyu, and Kazuhiro Saitou. Automated extraction of chemical structure information from digital raster images. Chem. Cent. J., 3:4, 03 2009,
/tg.///oo.og/...//// /10. 10..000000 1186/ 752 1531-
[23] Judea Pearl. Causal diagrams for empirical research. Biometrika, 82(4):669 688, 1995.
[24] Judea Pearl. Direct and indirect effects. In Proceedings of the Seventeenth conference Uncertainty in artificial intelligence, pp. 4-1-2 2001.
[25] Judea Pearl. Causality, Cambridge university press, 2009.
[26] Judea Pearl et al. Models, reasoning and inference. Cambridge, UK: CambridgeUniiversitty Press, 2000.
[27] Óscar Pérez and Manuel Sánchez-Montañés. A new learning strategy for classification prob- lems with different training and test distributions. In International Work-Conference Artifi- cial Neural Networks, pp. 178-185. Springer, 2007.
[28] Tyler Peryea, Daniel Katzel, Tongan Zhao, Noel Southall, and Dac-Trung Nguyen. Molvec: Open source library for chemical structure recognition. In ABSTRACTS OF PAPERS OF THE AMERICAN CHEMICAL SOCIETY, volume 258. AMER CHEMICAL SOC 1155 16TH ST,
NW, WASHINGTON. DC 20036 USA, 2019.
[29] Alberto Sanfeliu and King-Sun Fu. distance measure between attributed relational graphs for pattern recognition. IEEE transactions on systems, man, and cybernetics, (3):353-362) 1983.
[30] Francesc Serratosa. Fast computation of bipartite graph matching. Pattern Recognition Letters, 2014.
[31] Francesc Serratosa. Computation of graph edit distance: reasoning about optimality and speed- up. Image and Vision Computing, 40:38-49, 48, 2015.
[32] Francesc Serratosa. Speeding up fast bipartite graph matching through new cost matrix. International Journal of Pattern Recognition and Artificial Intelligence. 290020:1500010, 2015.
[33] Joshua Staker, Kyle Marshall, Robert Abel, and Carolyn M. McQuaw. Molecular Structure Extraction from Documents Using Deep Learning. J. Chem. Inf. Model., 1913):0017-1029, March 2019. ISSN 1549-9596. doi: 10. 10.100///s..cim 8b00669. URL WtRLhttts:://// .org/
10. 1021/acaccc.cmm jcim. 8b00669
[34] Aniko T. Valko and A. Peter Johnson. Clide pro: The latest generation of clide, tool for optical chemical structure recognition. J. Chem. Inf. Model,, 49(4):780-787, 2009. URL https: //do:://do...o wtw.//....rr//... c 800449t

---

Under review as a conference paper at ICLR 2021
[35] Alexander Vezhnevets, Vittorio Ferrari, and Joachim M Buhmann. Weakly supervised struc- tured output learning for semantic segmentation. In 2012 IEEE conference on computer vision and pattern recognition, pp. 845-852 IEEE, 2012.
[36] David Weininger. SMILES, chemical language and information system. 1. introduction to methodology and encoding rules. J. Chem. Inf. Comput. Sci., 28(1):31-36, 1988. URL
https:
wwtg//o/////oooooooooo abs/ /0.s/10.. 1..0.00200 10111110000100000000
[37] Peter Willett, John M Barnard, and Geoffrey M Downs. Chemical similarity searching. Chem. Inf. Comput. Sci., 38(6);88-996, 1998.
[38] Jia Xu, Alexander Schwing, and Raquel Urtasun. Tell me what you see and i will show you where it is. In Proceedings of the IEEE conference on computer vision and paltern recognition, pp. 3190-3197, 2014.
[39] Jia Xu, Alexander G Schwing, and Raquel Urtasun. Learning to segment under various forms of weak supervision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3781-3790, 2015.
[40] Zhe Xu, Shaoli Huang. Ya Zhang, and Dacheng Tao. Augmenting strong supervision using web data for fine-grained categorization. In Proceedings of the IEEE international conference on computer vision, pp. 2524-2532, 2015.
[41] Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmeet Kohli, and Josh Tenenbaum. Neural-symbolic vqa: Disentangling reasoning from vision and language understanding. In Advances in neural information processing systems, pp. 1031-1042, 2018
[42] Honghui Zhang. Jianxiong Xiao, and Long Quan. Supervised label transfer for semantic segmentation of street scenes. In European Conference on Computer Vision, pp. 561-574. Springer, 2010.
A APPENDIX
A.I ARCHITECTURE SUMMARY OF GRAPH RECOGNITION TOOL
Every iteration of our method we need to train the graph recognition tool described in Oldenhof al. (1I) This graph recognition tool built using combination of different convolutional neural networks. The first part is semantic segmentation network to pixel-wise predict every atom, bond and charge type. The second part consists three classification networks to classify every segment predicted by the semantic segmentation network. After the first step of the ChemGrapher model 21), the segmentation network, the predicted segments are processed so that for every segment the center of mass is calculated. These centers of mass would be the atom/bond/chaage candidates to be classified by the classification networks.
Table Summary of the layers of the segmentation network
Layer
Kernel
Nonlinearity
Padding
Dilation
conv1
3x3
ReLU
no dilation
conv2
3x3
ReLU
2
conv3
3x3
ReLU
4
conv4
3x3
ReLU
conv5
3x3
ReLU
8
8
conv6
3x3
ReLU
conv7
3x3
ReLU
2
conv8
3x3
ReLU
no dilation
last
1x1
none
no padding
no dilation
12

---

Under review as a conference paper at ICLR 2021
Table 4: Different layers in the classification network
Layer
Kernel
Nonlinearity
Padding
Dilation
depthcony1
3x3
ReLU
no dilation
conv2
3x3
ReLU
2
2
conv3
3x3
ReLU
4
conv4
3x3
ReLU
8
conv5
3x3
ReLU
no dilation dilation
global maxpool
input size
None
no padding
last
Ixl
None
no padding
no dilation
A.2 TRAINING DETAILS FOR GRAPH RECOGNITION TOOL
Training details of the graph recognition tool for every iteration of our method are summarized in Table The imput images used for training of the different networks are mix images from source domain and upsampled rich labeled images from target domain. For pretraining of the ChemGrapher model only images from source domain were used. The training was performed using a compute node with 2 NVIDIA v100 GPUs with 32GB of memory.
Table 5: Training details for different networks
Network
#input images target domain (upsampled)
#epochs
walltime
minibatch size
learning rate
source domain
Segm. network
114K
20K
5
24h
8
0,001
Atom Clas,
12.4K
2.6K
2
8h
16
0.001
Charge Clas.
12.4K
2.6K
2
8h
16
0.001
Bond Clas.
4.4K
2.1K
2
4h
0.001
A.3 COMPUTATIONAL COST PER RICII-LABELING ITERATION
In the following Table 6 the computational cost for rich-labeling iteration is summarized including all steps: (re)training, predicting and graph aligning rich-labeling.
Table 6: Computational costs per rich-labeling iteration
Training
Predict
Graph Aligning
Hardware
2 NVIDIA v100 GPUs
1 NVIDIA v100 GPU
Intel Xeon Gold 6240 2.6Ghz
Dataset
Source+Target domain
Indigo/Maybride
Indigo
Maybridge
#datapoints
see Table
4,000
4,000
4,000
Walltime
44h (details Table
~2h
~00mi
~-3in
A.4 EXAMPLES OF CASES WHERE GRAPH ALIGNMENT FAILS
We would like to showcase examples where the constrained (max 2 node substitutions edge substitution) graph alignment fails. At the same time important to note that our proposed domain adaptation method is iterative method, so if graph alignment fails in previous iteration could succeed a next one when the new model makes a new graph prediction closer to the true graph.
13

---

Under review as a conference paper at ICLR 2021
OH
R.
CH2
(a) Input Image(U)
Single Bond
Tiple Bond
Prediction
0
O0)
(N)
(C)
(C)
(C)
(C)
(S)
(S)
(C)
(N)
(N)
(C)
Planar emhedding prediction (V')
(c) Graph Prediction (W")
(d) True Graph (W)
Figure 7: Example 1: It is clear that to align the graph prediction W with the true graph W more than node substitutions are needed. So no rich labeling is possible for this example this iteration.
14

---

Under review as a conference paper at ICLR 2021
Single
Double Bond
HO
HO
Atom Prediction
(C)
(C)
(C)
(C)
(C)
(C)
N
R1
R1
(C)
(b) Planar emhedding prediction (V")
(a) Input Image(U))
OH
HO
(c) Graph Prediction (W')
(d) True Graph (W)
Figure 8: Example 2: It is clear that alignment of graph prediction W" with the true graph W can not be solved with only substitutions.
Single Bond
(C)
Double Bond
Tiple
Atom Prediction
N
(N)
(C)
CI
(Br)
(C)
(C)
N)
C)
(C)
CC
(b) Planar embedding prediction (V")
(a) Input Image(U))
(d) True Graph (W)
(c) Graph Prediction (W')
Figure 9: Example It is clear that alignment of the graph prediction W' with the true graph W can not be solved with only substitutions.
15


Reviews:
[{"review_id": "seOkrlNmZn-", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\nSummary:\nThis article presents a methodology to generate the complete molecular graph (including connectivity between atoms and functional groups) from a 2D image by using an intermediate representation (called fully mediating representation, V). This problem is of high relevance and is likely to have a considerable impact in the chemistry community. The particular advance of this work is to include such intermediate representation based on a graph alignment approach that generates “strong” labels. \n##########################################################################\nReasons for score: \nOverall, I vote for not accepting. As mentioned before, this problem is of high relevance and is likely to have a considerable impact in the chemistry community, I see myself using it, in particular if it is paired with SMILES. Nevertheless, my major concern is about the clarity of the paper, but I think that beyond clarity, the lack of the full or at least partial code should be available from the beginning for reviewers. This would make the evaluation of the article much easier. \n\n##########################################################################Pros: \n \n1. The paper addresses an interesting problem that, if solved, every chemist would like to make use of it in everyday life. \n \n2. The proposed method uses and intermediate step in the learning process to encode a higher complexity as well as information. This provides flexibility that is reflected in a marginal gain in error prediction compared to other methods but it needs a lower number of data points. \n \n##########################################################################\nCons: \nAbstract: The abstract should be better structured. It should state first what the framework is or the context in which this work is set. The proceed to the specifics and the technical part. \n\nA specific motivation for the use of the mediating representation is missing. \n\nBe specific, “relatively low number of data points” doesn't provide any information. \n\nThe authors repeatedly use “strong labels are expensive”, just for the sake of completeness, would be good to be explicit in this fact instead of assuming that the reader will infer what the meaning is. \n\nThe authors state: “In order to measure empirically the performance of our method of self-labeling fully mediating representations we start with a pre-trained model and perform two steps.” What type of model or trained on what? It should be specific. \n\nThe reference “Slot attention” should be described more in detail and the advantages of the authors’ method over the Hungarian algorithm should be very clear.\n\nI think a clear sentence is missing to describe in detail what the authors mean by strong and weak labels. In this regard, the section “Weak Supervision” is not straightforward to read. It should be rewritten or rearranged to make a better and clear reading. \n\n“We also assume the map E(v) which gives all allowed graph edits for the graph v” it is not clear.\n\nWhat is the origin on the such a different performance on the Indigo and Maybridge datasets.\n\nFig. 5 and 6 nicely summarise the good performance of the model, but in order to understand better the method and its limitations or type of graphs that struggle with, it would be good to present out-layers where the system doesn't work.", "extra_scores": {}, "extra_text": {}}, {"review_id": "oip8fx2MU05", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Clarity\nThe paper is well written. The relevant work, introduction and main text allows the user to understand the problem easily.\n\nOriginality\nPaper uses intermediate layers (called mediating layers) as a concept to solve the problem of mapping  an input (that can represent a graph on a 2D image) to the graph structure. The base system for segmentation and classification is derived from previous work. The paper introduces graph aligning the mediating layer to the ground truth labels and propose an algorithm for the process. The originality is in the proposal of graph alignment as an intermediate step to generate additional training data while allowing different edit corrections.\n\nQuality\nThe experimentation showing the effectiveness of using a mediating layer compared to the base seems adequate for the 2 datasets. However there doesn't seem to be a good comparison to previous work. Moreover the previous work had different sampling (number of training and test samples) and is hard to understand if Table 1 is useful i.e. the comparison of numbers aren't useful. Having a clear fair comparison would have suggested the mediating layer's use more strongly.\n\nSignificance\nThe work claims to improve previous performance by a few percent points but as noted above the comparison may not be proper.\n\nQuestions\n1. Since the values of V are not really known (both during training and testing), why are the labels termed strong labels?\ni.e. is the correctness of the planar graph verified at any stage?\n\n2. There is a statement that the mediating layer can make the model more interpretable. But there is no further discussion in the paper on how this is true.\n\n3. It is not clear how the system is able to correctly classify never observed atoms and bonds in test examples. Is this valid only during training and alignment? Otherwise how does it learn to classify them?\n\n4. Above the key contribution section on page 2, there is mention that after domain adaptation, the performance is checked on the same test data. Why is the use of doing this? And why would we expect better results than not using domain adaptation?\n\n5. How is set of V initialized? Is it an empty set in the beginning? In algo 1, input seems to mention there are already m strong labels before the start, does that need some correction? Also, perhaps one line needs update of S\nS <- appendStrongLabels(T, (u,v));\n\n6. Do iteration 0 always correspond to the Oldenhof model or were they different models, not clear from the text other than mention that it was pre-trained?  \n\n\n\n\nGoal of the paper is to learn a function that maps an input (that can represent a graph on a 2D image) to the graph structure. Paper has experiments showing importance of mediating layer but doesn't make similar comparisions to previous work making it harder to understand if the mediating layer is really useful or not.", "extra_scores": {}, "extra_text": {}}, {"review_id": "XKrLwTGbHxg", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a domain adaption technique for self-labeling for strong/expensive planer graph labels given normal labels. For the application of molecular graphs, a graph alignment method on the planer graph level is proposed to find an isomorphism with a minimal edit distance of the predicted strong labels. The results show the proposed method can gradually correct the strong labels and improve the prediction performance with interpretable explanations. \n\nHowever, there are several concerns about the paper:\n1. The correction requires close edit distances between wrong labels and correct labels, and thus requires a relatively accurate U->V function. A plot regarding the correct percentage related to the accuracy of the initial U->V function will be interesting.\n2. How are strong labels picked? The authors mention the strong labels are \"a selection of these 4,000 datapoints\". But ablation study on the selection, size, and quality(edit distance to other graphs) of those datapoints should be investigated. \n3. The cost for the optimization problem argmin|e| is not discussed. The normal complexity for a single datapoint is N^d where N is the number of distance 1 editing and d is the distance allowed. A faster searching algorithm with domain knowledge is expected.\n4. For the other methods mentioned in table 1, are strong labels required? The percentage of strong labels should be reported for a fair comparison. The performance with different initial correct percentage will be interesting to investigate. \n5. The generalization of the model is not discussed. The adaption only works when the error of the prediction strong labels is smaller than k edit distance. However, the prediction depends on the distribution of the strong labels, the quality of the pre-trained models, the size of the graph, etc.\n\nOther questions:\n1. The structure of the paper can be improved. For example, the background of chemical structure recognition and the settings could be introduced at the beginning of the paper. \n2. In the experiments part, why the percentage correct decrease with iterations? \n3. In the graph alignment part, why to introduce sub-graph isomorphism?\n4. What is the relationship between function U->V, V->W, the segmentation network, the classification network?\n5. The segmentation network uses 134K  images. How is this data related to the training data?\n\nOverall, the idea of gradually correcting strong labels using graph alignment is interesting, but more discussions and results are required to make the paper stronger.", "extra_scores": {}, "extra_text": {}}, {"review_id": "QYCxs3VLiPd", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\n\nSummary:\n\nThe paper describes a method to convert 2D molecular images to molecular graph structures, with applications in extracting raw chemical structures from journal articles and other publications. The model has two components: a semantic segmentation network that first predicts the location of the atoms and bonds in the image, and a series of classification networks that classifies each segment. The paper proposes some domain adaptation techniques to reduce the amount of expensive ‘pixel-wise’ labels required for training the segmentation network\n\n##########################################################################\n\nReasons for score: \n\nOverall, I currently vote for rejection. I have some questions about the current evaluation setup that I hope the authors could clarify  \n\n##########################################################################\n\nStrengths:\n\n*The proposed iterative strong labeling and graph alignment framework seems to improve the performance of the pre-trained model  \n\nWeaknesses:\n\n*The model evaluation seems to show some mixed results. On the Mayfield dataset, there is an improvement over some baseline models, but on the Indigo dataset, the proposed model seems to be perform significantly worse than the model described in the citing reference 30. (~40% vs ~80%) \n\n*I found that some parts of the paper were difficult for me to understand (see below)\n\n##########################################################################\n\nQuestions and other comments:\n\n*Paper clarity:\n**I think there should be more information on how the underlying Chemgrapher model converts 2D molecular images to the molecular graph structures. How does the model actually construct the graph structure? Also, there is a lot of analysis on the image segmentation part of the model (eg figure 5, 6). Does the image classification part of the Chemgrapher model play any significant role? \n**Information about the Indigo and Maybridge dataset could be provided in a more accessible way. Eg the total number of examples in each dataset\n**Some additional information about how the original model was pre-trained would be useful\n\n*I think there needs to be more justification for why this proposed approach [option 1] of mapping the 2D molecule image to the molecular graph structure via an intermediate representation that explicitly identifies all the atoms and bonds in the image is preferred over the alternative approach [option 2] of directly mapping the 2D molecule image to the molecular graph structure (eg using by outputting a smiles text representation that can be converted to the molecular graph). The cost of option 1 is that it requires the very expensive pixel-wise labels that describes the locations of atoms and bonds in the 2D molecule image to train the segmentation model, thus motivating the domain adaptation part of this work. In terms of pure performance, it’s not clear to me that option 1 is superior, for example, [ref 30] which directly predicts the molecular smiles from the 2D molecular image [option 2] can attain ~80% in the indigo dataset, while this proposed approach seems to attain only ~40%\n\n*Figures 2 and 3 show various performance metrics over multiple iterations of the re-training. How did you decide which iteration to stop the re-training? Also, how computationally expensive is it to perform the iterative re-training procedure?\n\n*In Figure 3b, the performance at iteration 0 is ~72%, which I’m understanding to be the vanilla performance of the Chemgrapher model on the Maybridge dataset? But Table 1 shows that the performance of the Chemgrapher model on Maybridge is 83.3%\n\n*What is the reasoning for allowing a max of 2 node substitutions or 1 edge substitution for the ‘correcting graph alignment’ case?\n\n*In the future, it would be interesting to see how this proposed method compares in this recently published benchmark: https://github.com/Kohulan/OCSR_Review. NB: out of scope for this ICLR submission since it was published after the paper submission deadline\n\n\nRef [30]: Joshua Staker, Kyle Marshall, Robert Abel, and Carolyn M. McQuaw. Molecular Structure\nExtraction from Documents Using Deep Learning. J. Chem. Inf. Model., 59(3):1017–1029,\nMarch 2019. ISSN 1549-9596. doi: 10.1021/acs.jcim.8b00669. URL https://doi.org/10.1021/acs.jcim.8b00669.\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a graph aligning approach generating rich and detailed labels given normal labels. Authors cast the problem in a domain adaptation setting, considering a source domain where "expensive" labels are available, and a target domain where only normal labels are available. The application scenario is the prediction of chemical compound graphs from 2D images, where a fully mediating layer is introduced to represent using a planar embedding of the chemical graph structure to be predicted. 

The paper received ratings all below-threshold.
The main issue transversal to all reviewers relate to clarity of the presentation.
Clear motivations for some of the adopted choices of the method and of the experimental procedure were also missing. In particular, missed to provide the clear usefulness of the main paper's contribution, i.e., to neatly show the importance of the mediating layer (ref. R4, R2).

The lack of important details in the method description and experimental results were also deemed a major shortcoming: cost of the optimization, model generalization not discussed, contradictory results on the different datasets considered, comparative analysis, partial ablation, are among the main quoted remarks. 

Authors' rebuttal is carefully provided in general, but several issues are still remaining.

Hence, overall, given the above issues, I consider the paper not yet ready for publication in ICLR 2021.

Decision: Reject