Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors propose a Synergy-Oriented LeARning (SOLAR) framework that exploits the redundant nature of DoF in robot control.  The method is inspired by the well-known idea of "muscle synergy" which the human central nervous system has and reduces the DOF of the whole-body musculoskeletal system. The synergy is learned in an unsupervised manner.  The experimental results show the effectiveness of the approach.
strengths_and_weaknesses: + Application of the notion of muscle synergy to modular reinforcement learning (in an unsupervised manner) is quite new and interesting.
+ Experimental results show the effectiveness of the method.
+ Emerged synergies are analyzed and exhibit interesting results.
- The relationship between the conventional muscle synergy and the self-attention mechanism is slightly unclear.
questions: To what extent does the proposed approach share the same idea with the conventional kinematic or muscle synergies which are often analyzed by PCA. It may be fruitful to show the mathematical comparison explicitly. I think many ML researchers may not know muscle synergy and researchers who are familiar with muscle synergy (e.g., people studying human body control, musculoskeletal system, CPG, and so on) don't know much about this type of ML studies. This paper potentially becomes an important paper bridging the two separated communities. From that viewpoint, the connection between the proposed method and the conventional notion of muscle synergy is better to be described more clearly in my opinion (regarding the relation to the self-attention mechanism in particular).

Do you even hypothesize that human muscle synergies are organized in this way, or do you have any implications about the human muscle synergy obtained through this study?  Such a discussion may add extra value to the paper.
limitations: It is better to compare not only modular RL methods but also SOTA central RL methods will be beneficial for discussion.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: In this paper, the authors proposed a new modular reinforcement learning algorithm using unsupervised learning. By auto clustering joints that have synergies, i.e. based on their contribution to the tasks through value functions, the proposed method alleviates the control to the synergy space instead of the joint space. Then, a learned mapping function will convert the synergy action to joint action signals. With the proposed framework, the authors show that they can significantly outperform the SOTA baselines SMP and AMORPHEUS in many multiple task benchmarks. 
strengths_and_weaknesses: The strengths of the paper:

The results of the paper show a clear edge from the proposed framework in many benchmarks. 

The authors did a good amount of ablation study. I especially like how the authors visualize the synergy group’s propagation over time in Figure 5. 

The weaknesses of the paper:

The description of affinity propagation can be more clear. Currently affinity propagation is introduced in the background section and the writing is very similar to what has been written on Wikipedia. It is better to incorporate this section to methods, and define the similarity, responsibility and availability matrices in the context of the proposed system. For example, I assume that the exponential of the distance matrix in section 3.1 is used for the off-diagonal elements of the similarity matrices while delta-Q values are used as the diagonal elements. Consider re-writing this part and making it clear.. 

Sometimes, notations are used without proper definition. For example, the Q function in Eq (5) suddenly takes three different types of inputs, with both bold and non-bold symbols. This makes it difficult for the readers to understand. Consider revising. 

No illustration diagrams of the proposed network architecture for section 3.2. It is a bit hard to decipher the data flow just from the text descriptions. 
questions: See the weaknesses section for revising recommendations. 
limitations: N/A
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: In this paper, the authors focus on the problem of modular RL for systems with a large degrees of freedom such as humanoid characters. 

The central idea of this work can be divided into two parts. The first is the use of an unsupervised clustering algorithm to group different actuators on the agent based on their impact on the value function and the agent’s morphology. This is done using an existing algorithm named affinity propagation. The second part is an attention-based architecture that achieves a hierarchical controller where the policy produces low dimensional synergy actions, which are then mapped to per-actuator actions.

In the experiments, the method is evaluated on locomotion tasks for a variety of morphologies like walker, humanoid, unimal, etc. When compared to several state of the art baseline methods, the proposed algorithm shows superior learning performance. An ablation study was also carried out to demonstrate the importance of the proposed idea of learning synergy and the attention-based model.
strengths_and_weaknesses: Strengths:
1) The idea of synergy for lowering the control complexity in high dof systems is reasonable and the proposed method for finding the synergy amongst actuators is solid.
2) The experiments in general support the proposed idea and achieves good performance in highly challenging control problems.
3) The analysis is pretty thorough, including an ablation study to show the impact of the proposed ideas, as well as a visualization of the discovered synergy throughout learning, which is helpful in gaining insights into the algorithm.
4) The method could potentially be applied to other high dof control problems.

Weaknesses:
1) The proposed seems especially effective for the humanoid task, while achieving comparable results for the baselines on the other tasks. It’d be helpful to have some discussion regarding why humanoid shows the biggest difference.
2) Currently the tasks have been focused on one locomotion task (go forward). If more tasks can be demonstrated it can further support the effectiveness of the method.
3) Some details are missing, as mentioned below.

questions: In addition to some of the issues mentioned above, I also have the following questions:
1. When training in a multi-task setting, do all tasks (variant of the agents) share the same critic and actor weights?
2. For synergy of different morphologies, my understanding is that they are independently computed, is that right? I.e. the subfigures in Fig 5 are independent even some of them share the same color?
3. How frequent are the actor-critic update interleaved with the update in the synergy?
4. According to the text, the motivation for the proposed synergy formulation is that the one used in neuroscience requires optimal policies and the clustering could be erroneous. However, the proposed method also suffers from this issue as value functions would also be wrong at the beginning. By interleaving the policy learning and synergy discovery this doesn't really seem to be a problem, as demonstrated in Figure 6. I'm wondering if there are other issues or potential benefits with using a more neuroscience-inspired approach.

limitations: The author didn't discuss much limitation about the proposed work. I would suggest adding some discussions in terms of what kind of problem the current formulation may not be suitable for.
review_title: null

Meta-review: metareview: This paper propose an approach for learning low-rank synergies for morphology-generalizable robot control.

All the reviewers agree that the paper is interesting and a valuable contribution.
Hence, I recommend acceptance.

Additional comments:
- The related work section does not cover virtually any past work from robotics that deal with synergies and dimensionality reduction for large action-spaces. It would be good to include some of this literature to better place your work.
- The dimensionality of the action space in the environments used is not very high. However, the fact that in neither the manuscript nor the appendix the values of K (i.e., the number of actuators) are specified might create ambiguity. It would be good to make these values more visible.
confidence: Certain
Decision: Accept