Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper designs a directed weight operation to bake the SO(3)-action into the neural network by adding an extra dimension in the weight matrix. Except for the directed weight perceptrons, the paper proposes an equivariant message passing neural network combing the DWP module. The synthetic experiment shows the effectiveness of the DWP, and the experiments on protein 3D structures achieve good performance. Overall, though the experiments are able to show performance enhancement using such a directed weight matrix, the paper has a limited illustration of the DWP module. This paper also contains some typos, need to check more carefully.
strength_and_weaknesses: **Strength:**
1. The DWP is novel to combine the SO(3)-rotation and the linear transformation, unlike the VNN model, the authors also consider extracting orientation information with the weight matrix. Adding an extra dimension on the weight matrix is quite similar to the 3D convolution filters that extract 3D information from the 3D image data.
2. The synthetic experiments are a bonus to show the effectiveness of DWP and beat other related methods.
3. The proposed DWP module is compatible with other GNN models, which is quite interesting to investigate in many applications that require more orientation information.
 
**Weaknesses:**
1. The number of learning parameters will triple the size of the linear transform matrix and the complexity might be too big.
2. The ablation experiments should also consider keeping the model parameter almost the same as your proposed model since the DWP brings more parameters to the model than the other methods.
3. More illustrations or experiments to directly research the directed weight matrix compared with the undirected one.
4. More experiments are needed to verify the quality of the model.

**Question:**
1. The proposed method (even the illustrative figure) is similar to GVP. What is the merit of using DWP compared to GVP?
2. Is there any constraint act on the DWP to preserve some property? If we regard the directed weight matrix as the same as the normal weight matrix, we are not sure whether the orientation information in the data is extracted by the DWP module.


clarity,_quality,_novelty_and_reproducibility: The paper tries to enhance the representation of protein data, which is a vital problem to promote AI research on protein-related tasks. It remains unclear which role the DWP plays since it only expands the dimension of the learning parameter and no further analysis of the DWP or the feature it extracts. Compared with other related methods, we think the paper has limited contribution either to the proposed GNN model or the explainability of the directed weight matrix. The extended experiments show the model a promising work, more research should be taken to refine the mechanism that DWP makes the model aware of the orientation.

summary_of_the_review: The paper proposed a DWP plus SO(3)-equivariant GNNs for protein engineering. The merit of the design compared to the existing GVP and other equivariant message passing is not significant and experiments are not sufficient for illustrating the new model's effectiveness.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors propose a graph neural network architecture that uses vectorized neurons, extensive hidden feature mixing operations, and global/local reference frame transformations to produce protein embeddings that are sensitive to or "aware of" residue-specific orientations within the larger macromolecule. They first introduce Directed Weights, combinations of multidimensional tensors that generalize scalars to vectors and vectors to higher-dimensional tensors. They then describe the Directed Weight Perceptron module, which performs numerous scalar and vector operations such as linear multiplication, dot products, and cross products. Armed with networks that more naturally describe data and relationships in 3D space, the authors next describe their SO(3) equivariant message passing paradigm. They then describe a few variants of their model adapted to mimic the architectures of recent graph models such as GCNs, GINs, and GATs. Finally, the authors demonstrate their models' performance on a number of protein-related tasks.
strength_and_weaknesses: This paper introduces a novel concept, but the presentation, design choices, and testing methodology could all be much clearer. Moving some of the information from the appendix to the text would clarify the role of various mechanisms presented. For example, discussing more directly the scalar and vector feature constructions earlier in the text and not leaving them to the appendix would justify the use of the separate scalar features in the first place, which may otherwise seem extraneous when compared to feature vectors that could be magnitudes larger. Second, it isn't clear why the Directed Interaction model is necessary. Its stated purpose is to establish a connection between the scalar and vector features, which seems redundant given the operations blending scalar and vector features in the Directed Linear module. Clarification on what this module provides that the latter does not would be helpful.

The experimental results generally seem reliable and convincing, but it is not immediately clear what configurations of models are being compared to each other. It even seems in some places that conflicting information is given i.e. in B.2 the DWP model is 1 layer for the Synthetic Task but B.3 "Our model uses 4-layer message passing, where each message passing consists of 3-layer DWP". The latter further seems redundant with "there is 4 message passing...3-layer networks." One can tease these statements/cases apart, but more concise wording and clearer statement of model architecture/hyperparameters would be beneficial. Further, the synthetic task does not seem convincing without more information about how the points on a sphere were sampled, and statistics such as the variances of the positive and negative samples. Finally, it seems rather astounding that an MLP can match the performance of the most sophisticated models with a comparable number of weights on this task. The universal approximation theorem is relevant for arbitrarily deep and/or wide networks, so this is not sufficient justification. Similarly, it appears that GVP fails completely on this synthetic task, but is tested in later tasks whereas VNN performs well and is excluded without explanation.

The visualizations are informative but the table layout (tables 3-5) is confusing. There are also many citations to the arxiv version of papers that have been published in ML conference proceedings or other venues. The references should be carefully checked and corrected.

Some specific questions:
1. Why was GVP chosen over VNN for testing on benchmarks?
2. What applications outside of protein embedding is this work relevant to?
3. Why is the Directed Interaction module necessary when feature blending already occurs in the Directed Linear modules?
4. How are the k-nearest neighbors computed? Is the full N x N distance matrix available?
5. How could MLPs produce similar results as more efficient graph models with the same number of parameters?
6. Is there rigorous rationale for averaging Pearson r, Spearman rho, and Kendall tau? Was there consideration of weighting the factors? Is there any redundancy between the metrics?
7. How exactly were the synthetic task vectors sampled? Were the positive and negative areas of equal area? was the negative area less densely sampled?
8. Will code and trained models be released?
clarity,_quality,_novelty_and_reproducibility: The proposed method is novel and the results look reasonable, but the paper lacks clarity and is missing some important details and justification of some of the modeling decisions.
summary_of_the_review: An interesting method, but the presentation lacks clarity and several design decisions are not well justified.

What would improve my score: add clarifications and additional details.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This is a work on rotationally equivariant feed forward networks for applications in protein folding and related tasks.
A number of equivariant operations which appear useful for these applications are introduced, such as a cross product operator.
Experiments are done on several useful tasks using protein structure databases, and significantly improved results are reported.
strength_and_weaknesses: Strengths: simple ideas, broad experimental testing, good performance improvement.

Weaknesses: very empirical, leaves much for future work.
clarity,_quality,_novelty_and_reproducibility: I found the presentation clear and the results look convincing.   I am not really able to judge its originality.
summary_of_the_review: Valuable improvements to ML for protein folding.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a class of SO(3)-equivariant neural networks called Orientation-Aware Graph Neural Networks (OAGNNs). 
The model is based on specific layers acting on scalar and vector features, designed to ensure better expressive power (compared to previous works) while maintaining SO(3) equivariance. 

Compared to previous related work, the main novelty of the proposed method is to have order-3 tensors of learnable weights acting on the features, effectively expanding classical perceptron weights from scalars to oriented vectors in 3d. 

In the experimental section, the authors focus primarily on tasks related to structural biology. 
This is a natural playground for SO(3) equivariant models, since many important characteristics of proteins can be described in terms of local frames irrespective of the global orientation of the molecules. 

All results show that the proposed method is significantly better than the tested baselines, on almost all tasks and metrics considered. 
strength_and_weaknesses: **Strengths**: 

- The paper is interesting and tackles an important problem. I expect that the paper could be of interest to the GNN, computer vision, and computational biology communities. 
- The results are strong and there are no concerns in terms of significance (although confidence intervals are not reported). The design of the model is validated also through ablation studies. 

**Weaknesses**: 

- Equation (6) implies that $c \vec W \in \mathbb{R}^{C \times 3}$ for $s \in \mathbb{R}^C$ and $\vec W \in \mathbb{R}^{C \times 3}$.  How is the product $c\vec W$ defined here?
- A lot of the geometrical intuition behind the paper is never formally stated. For example: 
	- Why is equivariance to rigid transformations broken in Equations 7,8?
	- What does it mean that the network can "sense the orientational features"?
	- What set of transformations is the proposed method equivariant to, that VNNs weren't before? 
	- What does it mean that the proposed method is "more adaptive to the geometrically meaningful features?"
- In Experiment 5.1, do all models have a comparable number of parameters? DWP uses significantly more parameters than VNN, so this should be factored into the comparison. 
- Why is there no comparison with VNN in Experiment 5.2? Is there some reason that makes it impossible to design a GNN with vector neurons? This is important because GVP is the worst-performing baseline in Experiment 5.1, so the more interesting comparison is between OA and VNN. 
- Typos: 
	- Missing parenthesis in the numerator of Eq. 24, right.
clarity,_quality,_novelty_and_reproducibility: - Clarity: the paper is clear, although it could be more precise in explaining the main contributions. 
- Quality: the quality of the paper is good, and there are no major concerns regarding the design of the method. 
- Novelty: the main idea of the paper is novel, sensibly extending previous work. However, my suggestion is to compare the proposed method with VNNs in a more in-depth way (not just on one synthetic experiment, and also ensuring that the comparison is fair). 
- Reproducibility: it should be possible to implement the proposed method from the description, and experimental details are reported in the appendix. 
summary_of_the_review: There are no major concerns with the paper, but there is margin for improvement. Extending the comparison with previous work is particularly important, since it is not clear whether the proposed method is simply making use of the extra parameters compared to VNN.

I have recommended a weak acceptance, conditional on the authors addressing my concerns above.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper considers the important problem of protein representation learning. Directed Weights Perceptrons are introduced to better represent geometric relation, along with an equivariant message passing algorithm. The resulting framework makes it possible to learn the global backbone structure as well as the local fine-grained relationships between amino acids. The approach is demonstrated on multiple benchmarks.

The  AC is very familiar with the problem and carefully examined the manuscript and discussion. All reviewers and AC believe that the proposed approach is interesting and promising. The author feedback and additional experiments address many of the reviewers concerns However, more work is needed to understand more precisely what the approach is doing and how the introduction of DWs affect performance.  

- In particular, as suggests by reviewer Reviewer XTJW, it would be important to characterize more precisely the directed weight matrix compared with the undirected one. 
- Also it is critical to compare training/inference times of the proposed approach compared to comparison approaches to get a full picture of how the Directed Weights affect the performance.

We strongly encourage the authors to pursue this work and incorporate the above comments, in addition to the points made in their responses to the reviewers. 
justification_for_why_not_higher_score: The paper is promising but it needs to more systematically and more fully investigate what the proposed directed weights are doing and how they affect training/inference time.
Decision: Reject