Kati8/ICLR_2023_2025 huggingface dataset:
authors: Yiliang Yuan
authors: Mustafa Misir
reviews: 
Comments: This paper introduces GNNAS-Dock, an algorithm selection method tailored for molecular docking. GNNAS-Dock first generates molecular representations for both ligand and protein by combining GCN, GAT, and GIN models. These representations are concatenated and then passed through two separate models—an accuracy model and an efficiency model—that predict the performance of each algorithm. Experimental results demonstrate GNNAS-Dock’s performance compared to previous molecular docking approaches.
Strengths: - The paper is easy to follow.
- The use of two criteria, accuracy and efficiency, for algorithm selection is a reasonable approach that adds value.
Weaknesses: - Is the Accuracy Model and Efficiency Model trained using supervised learning with a labeled dataset that includes labels for both accuracy and efficiency?
- When a new molecular docking algorithm is released, does GNNAS-Dock need to be retrained from scratch? Would this require a new labeled dataset for the accuracy and efficiency of the new docking algorithms? Could transfer learning be applied to GNNAS-Dock in this scenario? If transfer learning is not feasible, this would represent a significant weakness.
- Table 2 shows that DiffDock achieves comparable or even superior performance to GNNAS-Dock. Given that GNNAS-Dock requires additional overhead compared to DiffDock, it raises the question: why would practitioners choose GNNAS-Dock over DiffDock? Would DiffDock be a preferable choice for practical applications?
- Is there any baseline in the literature for algorithm selection that could be used as a comparison, rather than individual molecular docking algorithms?
- I suggest moving the explanations of GCN, GAT, and GIN from the Methods section to a Preliminary section. This would allow the Methods section to focus solely on the novel methodological contributions, which would better clarify the paper’s contributions.
- Typo: Line 221 "From"
Recommendation: 3
Reviewer confidence: 3
Soundness/correctness: 2
Clarity: 3
reviews: 
Comments: This paper introduces GNNAS-Dock, a novel Graph Neural Network-based algorithm selection system for molecular docking. The system aims to address two key challenges: (1) selecting the most accurate docking algorithm in terms of RMSD, and (2) maintaining computationally efficiency and acceptable accuracy. The approach was validated on the PDBBind 2020 refined dataset across 9 different docking algorithms, demonstrating improvements accuracy compared to individual algorithms while maintaining efficiency.
Strengths: 1. The authors conducted a comprehensive comparison experiment with the individual method, demonstrating potential improvements in accuracy while maintaining computational efficiency.
2. The authors innovatively address an important problem by applying stacking learning to molecular docking, where ensemble learning and algorithm selection approaches have been previously overlooked.
Weaknesses: 1. Some recently published SOTA works are missing (according to the ICLR reviewer guide, these works do not qualify as concurrent work), e.g. DiffDock-L.
2. The paper lacks comparison with other algorithm selection methods. While the comparison with individual algorithms demonstrates the effectiveness of ensemble learning or algorithm selection (which is an expected outcome), it does not sufficiently validate the merits of the authors' proposed core methodology.
Recommendation: 3
Reviewer confidence: 3
Soundness/correctness: 2
Clarity: 2
reviews: 
Comments: GNNAS-DOCK processes a ligand and protein using graph neural networks to predict which docking tool will provide the best result (algorithm selection).  Both the accuracy and computational time of each algorithm is predicted so they can be simultaneously optimized.
Strengths: This is an interesting, practical idea for exploiting/rectifying the fact that different docking tools have different performance characteristics on different systems.
Weaknesses: The evaluation is done on a random split of PDBBind, which means there is massive information leakage between the training and test sets. Given this methodological choice, it is unlikely any reported results will generalize to realistic scenarios.

Only the redocking task is considered, which is not of much practical interest.

An unbiased assessment of Table 2 is unlikely to reach the same conclusion of the authors that their tool is a better approach than using DiffDock (unsurprising as DiffDock is being evaluated on its training set).

There is no analysis of the model's predictions - it would be very interesting to learn if there is an understandable pattern to what systems different docking tools are predicted to do better on (presumably whether the system is in the method's training set, for ML methods, is highly relevant).

There are concerning methodological choices in the neural network construction.  Raw Cartesian coordinates are used as features, and not as part of an equivariant GNN.  It does not appear data augmentation was used to learn equivariance. This means the trained models are highly brittle and unlikely to generalize.  The protein graph only connects along peptide bonds, so it is a line (so why use a graph representation?).  The graph pooling operator isn't disclosed in the text.
Recommendation: 3
Reviewer confidence: 5
Soundness/correctness: 2
Clarity: 2