Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: ##########################################################################

Summary:

The paper describes a method to convert 2D molecular images to molecular graph structures, with applications in extracting raw chemical structures from journal articles and other publications. The model has two components: a semantic segmentation network that first predicts the location of the atoms and bonds in the image, and a series of classification networks that classifies each segment. The paper proposes some domain adaptation techniques to reduce the amount of expensive ‘pixel-wise’ labels required for training the segmentation network

##########################################################################

Reasons for score: 

Overall, I currently vote for rejection. I have some questions about the current evaluation setup that I hope the authors could clarify  

##########################################################################

Strengths:

*The proposed iterative strong labeling and graph alignment framework seems to improve the performance of the pre-trained model  

Weaknesses:

*The model evaluation seems to show some mixed results. On the Mayfield dataset, there is an improvement over some baseline models, but on the Indigo dataset, the proposed model seems to be perform significantly worse than the model described in the citing reference 30. (~40% vs ~80%) 

*I found that some parts of the paper were difficult for me to understand (see below)

##########################################################################

Questions and other comments:

*Paper clarity:
**I think there should be more information on how the underlying Chemgrapher model converts 2D molecular images to the molecular graph structures. How does the model actually construct the graph structure? Also, there is a lot of analysis on the image segmentation part of the model (eg figure 5, 6). Does the image classification part of the Chemgrapher model play any significant role? 
**Information about the Indigo and Maybridge dataset could be provided in a more accessible way. Eg the total number of examples in each dataset
**Some additional information about how the original model was pre-trained would be useful

*I think there needs to be more justification for why this proposed approach [option 1] of mapping the 2D molecule image to the molecular graph structure via an intermediate representation that explicitly identifies all the atoms and bonds in the image is preferred over the alternative approach [option 2] of directly mapping the 2D molecule image to the molecular graph structure (eg using by outputting a smiles text representation that can be converted to the molecular graph). The cost of option 1 is that it requires the very expensive pixel-wise labels that describes the locations of atoms and bonds in the 2D molecule image to train the segmentation model, thus motivating the domain adaptation part of this work. In terms of pure performance, it’s not clear to me that option 1 is superior, for example, [ref 30] which directly predicts the molecular smiles from the 2D molecular image [option 2] can attain ~80% in the indigo dataset, while this proposed approach seems to attain only ~40%

*Figures 2 and 3 show various performance metrics over multiple iterations of the re-training. How did you decide which iteration to stop the re-training? Also, how computationally expensive is it to perform the iterative re-training procedure?

*In Figure 3b, the performance at iteration 0 is ~72%, which I’m understanding to be the vanilla performance of the Chemgrapher model on the Maybridge dataset? But Table 1 shows that the performance of the Chemgrapher model on Maybridge is 83.3%

*What is the reasoning for allowing a max of 2 node substitutions or 1 edge substitution for the ‘correcting graph alignment’ case?

*In the future, it would be interesting to see how this proposed method compares in this recently published benchmark: https://github.com/Kohulan/OCSR_Review. NB: out of scope for this ICLR submission since it was published after the paper submission deadline


Ref [30]: Joshua Staker, Kyle Marshall, Robert Abel, and Carolyn M. McQuaw. Molecular Structure
Extraction from Documents Using Deep Learning. J. Chem. Inf. Model., 59(3):1017–1029,
March 2019. ISSN 1549-9596. doi: 10.1021/acs.jcim.8b00669. URL https://doi.org/10.1021/acs.jcim.8b00669.


review_title: Recommendation to reject

final_score: rating: 5: Marginally below acceptance threshold
review_content: The authors propose a domain adaption technique for self-labeling for strong/expensive planer graph labels given normal labels. For the application of molecular graphs, a graph alignment method on the planer graph level is proposed to find an isomorphism with a minimal edit distance of the predicted strong labels. The results show the proposed method can gradually correct the strong labels and improve the prediction performance with interpretable explanations. 

However, there are several concerns about the paper:
1. The correction requires close edit distances between wrong labels and correct labels, and thus requires a relatively accurate U->V function. A plot regarding the correct percentage related to the accuracy of the initial U->V function will be interesting.
2. How are strong labels picked? The authors mention the strong labels are "a selection of these 4,000 datapoints". But ablation study on the selection, size, and quality(edit distance to other graphs) of those datapoints should be investigated. 
3. The cost for the optimization problem argmin|e| is not discussed. The normal complexity for a single datapoint is N^d where N is the number of distance 1 editing and d is the distance allowed. A faster searching algorithm with domain knowledge is expected.
4. For the other methods mentioned in table 1, are strong labels required? The percentage of strong labels should be reported for a fair comparison. The performance with different initial correct percentage will be interesting to investigate. 
5. The generalization of the model is not discussed. The adaption only works when the error of the prediction strong labels is smaller than k edit distance. However, the prediction depends on the distribution of the strong labels, the quality of the pre-trained models, the size of the graph, etc.

Other questions:
1. The structure of the paper can be improved. For example, the background of chemical structure recognition and the settings could be introduced at the beginning of the paper. 
2. In the experiments part, why the percentage correct decrease with iterations? 
3. In the graph alignment part, why to introduce sub-graph isomorphism?
4. What is the relationship between function U->V, V->W, the segmentation network, the classification network?
5. The segmentation network uses 134K  images. How is this data related to the training data?

Overall, the idea of gradually correcting strong labels using graph alignment is interesting, but more discussions and results are required to make the paper stronger.
review_title: Self-Labeling of Fully Mediating Representations by Graph Alignment

final_score: rating: 5: Marginally below acceptance threshold
review_content: ##########################################################################
Summary:
This article presents a methodology to generate the complete molecular graph (including connectivity between atoms and functional groups) from a 2D image by using an intermediate representation (called fully mediating representation, V). This problem is of high relevance and is likely to have a considerable impact in the chemistry community. The particular advance of this work is to include such intermediate representation based on a graph alignment approach that generates “strong” labels. 
##########################################################################
Reasons for score: 
Overall, I vote for not accepting. As mentioned before, this problem is of high relevance and is likely to have a considerable impact in the chemistry community, I see myself using it, in particular if it is paired with SMILES. Nevertheless, my major concern is about the clarity of the paper, but I think that beyond clarity, the lack of the full or at least partial code should be available from the beginning for reviewers. This would make the evaluation of the article much easier. 

##########################################################################Pros: 
 
1. The paper addresses an interesting problem that, if solved, every chemist would like to make use of it in everyday life. 
 
2. The proposed method uses and intermediate step in the learning process to encode a higher complexity as well as information. This provides flexibility that is reflected in a marginal gain in error prediction compared to other methods but it needs a lower number of data points. 
 
##########################################################################
Cons: 
Abstract: The abstract should be better structured. It should state first what the framework is or the context in which this work is set. The proceed to the specifics and the technical part. 

A specific motivation for the use of the mediating representation is missing. 

Be specific, “relatively low number of data points” doesn't provide any information. 

The authors repeatedly use “strong labels are expensive”, just for the sake of completeness, would be good to be explicit in this fact instead of assuming that the reader will infer what the meaning is. 

The authors state: “In order to measure empirically the performance of our method of self-labeling fully mediating representations we start with a pre-trained model and perform two steps.” What type of model or trained on what? It should be specific. 

The reference “Slot attention” should be described more in detail and the advantages of the authors’ method over the Hungarian algorithm should be very clear.

I think a clear sentence is missing to describe in detail what the authors mean by strong and weak labels. In this regard, the section “Weak Supervision” is not straightforward to read. It should be rewritten or rearranged to make a better and clear reading. 

“We also assume the map E(v) which gives all allowed graph edits for the graph v” it is not clear.

What is the origin on the such a different performance on the Indigo and Maybridge datasets.

Fig. 5 and 6 nicely summarise the good performance of the model, but in order to understand better the method and its limitations or type of graphs that struggle with, it would be good to present out-layers where the system doesn't work.
review_title: Report on "Self-Labeling of Fully Mediating Representations by Graph Alignment"

final_score: rating: 4: Ok but not good enough - rejection
review_content: Clarity
The paper is well written. The relevant work, introduction and main text allows the user to understand the problem easily.

Originality
Paper uses intermediate layers (called mediating layers) as a concept to solve the problem of mapping  an input (that can represent a graph on a 2D image) to the graph structure. The base system for segmentation and classification is derived from previous work. The paper introduces graph aligning the mediating layer to the ground truth labels and propose an algorithm for the process. The originality is in the proposal of graph alignment as an intermediate step to generate additional training data while allowing different edit corrections.

Quality
The experimentation showing the effectiveness of using a mediating layer compared to the base seems adequate for the 2 datasets. However there doesn't seem to be a good comparison to previous work. Moreover the previous work had different sampling (number of training and test samples) and is hard to understand if Table 1 is useful i.e. the comparison of numbers aren't useful. Having a clear fair comparison would have suggested the mediating layer's use more strongly.

Significance
The work claims to improve previous performance by a few percent points but as noted above the comparison may not be proper.

Questions
1. Since the values of V are not really known (both during training and testing), why are the labels termed strong labels?
i.e. is the correctness of the planar graph verified at any stage?

2. There is a statement that the mediating layer can make the model more interpretable. But there is no further discussion in the paper on how this is true.

3. It is not clear how the system is able to correctly classify never observed atoms and bonds in test examples. Is this valid only during training and alignment? Otherwise how does it learn to classify them?

4. Above the key contribution section on page 2, there is mention that after domain adaptation, the performance is checked on the same test data. Why is the use of doing this? And why would we expect better results than not using domain adaptation?

5. How is set of V initialized? Is it an empty set in the beginning? In algo 1, input seems to mention there are already m strong labels before the start, does that need some correction? Also, perhaps one line needs update of S
S <- appendStrongLabels(T, (u,v));

6. Do iteration 0 always correspond to the Oldenhof model or were they different models, not clear from the text other than mention that it was pre-trained?  




Goal of the paper is to learn a function that maps an input (that can represent a graph on a 2D image) to the graph structure. Paper has experiments showing importance of mediating layer but doesn't make similar comparisions to previous work making it harder to understand if the mediating layer is really useful or not.
review_title: Goal of the paper is to learn a function that maps an input (that can represent a graph on a 2D image) to the graph structure. Paper has experiments showing importance of mediating layer but doesn't make similar comparisions to previous work making it harder to understand if the mediating layer is really useful or not.

Meta-review: The paper proposes a graph aligning approach generating rich and detailed labels given normal labels. Authors cast the problem in a domain adaptation setting, considering a source domain where "expensive" labels are available, and a target domain where only normal labels are available. The application scenario is the prediction of chemical compound graphs from 2D images, where a fully mediating layer is introduced to represent using a planar embedding of the chemical graph structure to be predicted. 

The paper received ratings all below-threshold.
The main issue transversal to all reviewers relate to clarity of the presentation.
Clear motivations for some of the adopted choices of the method and of the experimental procedure were also missing. In particular, missed to provide the clear usefulness of the main paper's contribution, i.e., to neatly show the importance of the mediating layer (ref. R4, R2).

The lack of important details in the method description and experimental results were also deemed a major shortcoming: cost of the optimization, model generalization not discussed, contradictory results on the different datasets considered, comparative analysis, partial ablation, are among the main quoted remarks. 

Authors' rebuttal is carefully provided in general, but several issues are still remaining.

Hence, overall, given the above issues, I consider the paper not yet ready for publication in ICLR 2021.

Decision: Reject