Low-Rank Modular Reinforcement Learning via Muscle Synergy
Authors: Heng Dong
Tonghan Wang
Jiayuan Liu
Chongjie Zhang
Venue: NeurIPS 2022 Accept (2022)
Keywords: Reinforcement Learning
Low-Rank
Muscle Synergy

Abstract: Modular Reinforcement Learning (RL) decentralizes the control of multi-joint robots by learning policies for each actuator. Previous work on modular RL has proven its ability to control morphologically different agents with a shared actuator policy. However, with the increase in the Degree of Freedom (DoF) of robots, training a morphology-generalizable modular controller becomes exponentially difficult. Motivated by the way the human central nervous system controls numerous muscles, we propose a Synergy-Oriented LeARning (SOLAR) framework that exploits the redundant nature of DoF in robot control. Actuators are grouped into synergies by an unsupervised learning method, and a synergy action is learned to control multiple actuators in synchrony. In this way, we achieve a low-rank control at the synergy level. We extensively evaluate our method on a variety of robot morphologies, and the results show its superior efficiency and generalizability, especially on robots with a large DoF like Humanoids++ and UNIMALs. 

Low-Rank Modular Reinforcement Learning via
Muscle Synergy
Heng Dong" III, Tsinghua University drdhxi @ihxiigmmal.... com
Tonghan Wang" Harvard University twangl0g harvard. edu
Jiayuan Liu IIIS. Tsinghua University georgejlayuan@gmail. com
Chongjie Zhang
IIIS, Tsinghua University chongjieftsinghua edu. en
Abstract
Modular Reinforcement Learning (RL) decentralizes the control multi-joint robots by learning policies for each actuator. Previous work on modular RL has proven its ability to control morphologically different agents with shared actua- tor policy. However, with the increase in the Degree of Freedom (DoF) of robots, modular controller becomes exponentially
training a difficult. Motivated by the way the human central nervous system controls numer- LeARning (SOLAR) framework that
ous muscles, we propose a Synergy-Orienie exploits the redundant nature of DoF in robot control. Actuators grouped into synergies by an unsupervised learning method, and a synergy action is learned to control multiple actuators in synchrony. In this way, achieve low-rank con- trol at the synergy level. We extensively evaluate our method on variety of robot morphologies, and the results show its superior efficiency and generalizability. pecially on robots with a large DoF like Humanoid++ and UNIMALs.
Introduction
Deep reinforcement learning (RL) has contributed significantly to the sensorimotor control of both simulated [Heess et all 2017, Zhu all, [020] and real-world [Levine et all, 2016, Mahmood atall 2018 robots. Monolithic learning is a popular paradigm for learning control policies. In this paradigm, a policy inferring a joint action for all limb actuators based on global sensory state learned. Although monolithic learning has made impressive progress [Chen atl 2020, Kuznetsoy cLall 2120]. it has two major shortcomings. First, the input and output space is large. For robots with more joints, learning controlling policies puts heavy burden on the representational capac- ity of neural networks. Second, the input and output dimensions are fixed, making it inflexible to transfer the learned control policies to robots with different morphologies.
Modular reinforcement learning provides an elegant solution to these problems. this learning paradigm [Wang et al], 2018), the control policy is decentralized [Peng et all, 2021]]. and each limb actuator controlled by a local policy. Recent research efforts show that the local policies can learn high-penn gegh-performanc and transferable control strategies by sharing parameters [Huang et al, 2020]. communicating to each other by message passing [Huang et all 2020), and adaptively paying atten- tion to other actuators via graph neural networks [Kurin el all 2020). By exploiting the flexibility and generalizability provided by modularity, a modular policy can now control robots of up to thou- sands of morphologies [Gupta all [
These authors contributed equally this work.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Despite the significant progress, modular reinforcement learning is still limited in terms of the com- plexity of morphological structures that can be controlled and struggles on robots with many joints like Humanoid [[Kurin et all, 2020]. The large degree of control freedom presents a major chal- lenge for learning control policies. A question why humans can control hundreds of muscles with dexterity while the most advanced RI. policy can only control less than fifteen actuators.
Studies on muscle synergies [i[ et aLl M[XE may provide an answer. A human central ner- vous system decreases the control complexity by producing small number of electrical signals and activating muscles in groups [Ting and McKay, 2007]. Muscle synergy the coordination of muscles that are activated in synchrony. With muscle synergies, the human nervous system achieves low-rank control over its actuators. In this paper, we aim to use the inspiration of muscle synergies to reduce the control complexity and improve the learning performance of modular RL.
The first challenge of incorporating muscle synergies into modular is to discover synergy struc- ture that can promote policy learning. Neuroscience researchers factorize electrical signals [Saito caall 2018, Ealakı et all 2017, Kıeliaa ct all 201J] analyze the synergy structure, but policy signals are sub-optimal or even absent during reinforcement learning. We thus exploit the functional similarity and morphological context of actuators and use a clustering algorithm to identify actuators in the same synergy. The intuition is that muscles in a synergy typically serve the same functional purpose and have similar morphological contexts. We quantify the functional similarity by the influ- ence of an actuator's 's action on the global value function, and the morphological structure is encoded as distance matrix. To use the two types of information simultaneously, we adopt the affinity prop- agation algorithm [Frey and Dueck, 2007]. The synergy structure is updated periodically during learning to promptly reflect changes in value functions.
To exploit the discovered synergy structure, we design a synergy-aware architecture for policy learn- ing. The major novelty here is that the policy learns action selection for each synergy, and the synergy actions are transformed linearly to get actuator actions. Since the number of synergies is typically much smaller than actuators, we actually learn low-rank control policy where the physical actions are a linear mapping from a www-dimesiiona action space. Moreover, for better processing state information, the synergy-awart policy adopts a two-level transformer structure, which first aggregates information within each synergy and then processes information across synergies.
We evaluate our Synergy-Oriene Oriented LeARning (SOlaR) framework on two MuJoCo [[odorov et all 2012 locomotion benchmarks [Huang et all, 2020 Gupta et all 20216) and in multi-task to zero- shot learning, single-task settings. SOLAR significantly outperforms previous state-of-the-art algo- rithms in terms of both sample efficiency and final performance on all tested settings, especially on robots with large DoF like fumanoid++ [Huang all, 200I and UNIMALS [Gupta et all (02I Performance comparison and the visualization of learned synergy structures strongly support the ef- fectiveness of our synergy discovery method and synergy- aware transformer based policy learning approach. Our experimental results reveal the low-rank nature of multi-joint robot control signals.
2 Background
Modular RL. Modular Reinforcement Learning decentralizes the control of multi-joint robots by learning policies for each actuator. Each joint has its controlling policy and they coordinate with each other via various message passing schemes. Modular RL usually needs to deal with agents with
different morphologies. To do so. Wang et al] [20]3]] and Pathak et all [201]]] represent the robot's s morphology as a graph and use GNNs as policy and message passing networks. Huang et [2020] uses both bottom-up and top-down message passing scheme through the links between joints for coordinating. All of these GNN-like works show the benefits of modular policies over a monolithic
policy in tasks tackling different morphologies. However, recently, Kurin et all [[20] validated hypothesis that any benefit GNNs can extract from morphological structures is outweighed by the dif- ficulty of message passing across multiple hops. They further propose transformer-
method, AMORPHEUS, that utilizes self- oeff-attention mechanisms as a message passing approach. AMORPHEUS outperforms prior works and our work is based on AMORPHEUS. Previous works mainly focused on effective message passing schemes, while our work aims at reducing learning complexities when the DoF of the robot large.
Muscle Synergy. How the human central nervous system (CNS) coordinates the activation of a large number of muscles during movement is still an open question. According to numerous studies,
2

---

the CNS activates muscles in groups to decrease the complexity required to control each individual muscle Avelll et all 2003, [in and Mckay, 20071] According to muscle synergy theory, the CNS produces a small number of signals. The combinations of these signals are distributed to the museles [Wojtara et al. 2014]. Muscle synergy is the term for the coordination of muscles that acti- vate at the same time verrante_ettal 20TG]. A synergy can include multiple muscles, and muscle can belong to multiple synergies. Synergies produce complicated activation patterns for a set of muscles during the performance of a task, which is commonly measured using electromyography (EMG) [[reech et all 2002, Singh et all 2018]. EMG signals are typically recorded as a matrix with a column for activation signals for moment and row for activation of muscle [Rabbi etall 0I20]. Factorisation methods on the matrix are used to extract muscle synergies from mus- cle activation patterns. Four most commonly used factorization methods
non-negative matrix factorisation [Steele et all 2015, Schwartz et all, 2016, Lee and Seung. 1999, Rozumalski et all Shuman et all 2016, Laito_etal, 2018), , principal component analysis [[Ting and Macpherson, Ting et all, 2015, Danion and Latash, 2010. Falaki et all 2017], independent component anal-
2017.
20055
ysis
Hyvarien and Oja, 2000, Hart and Giszte 2015 and factor analysis [Kieliba all 2018.
Saioo el all [0T.)].
In the field of robot control, only few works [Palli et al, 2014, Wimböck et all, 2011. Ficucielld
all 2016] have exploited the idea of musele synergy for dimensionality reduction to simplify the control. However, these works usually first use motion dataset from humans to obtain the synergy space and then learn to control in this synergy space. In contrast, our work learns the synergy space simultaneously with the control policy in the synergy space.
Affinity propagation [Frey and Dueck, [X07] a clustering algorithm based on multi-round mes- sage passing between input data points. It does not need to pre-define the number of clusters and proceeds by finding each instance an exemplar. Data points that choose the same exemplar belongs to the same cluster.
Suppose {= is set of data points. Define S Raxa as similarity matrix. When K j. the element $ty at ith row and jth column is the similarity between I; and zj. which can be measured as, for example, the negative squared distance of two data points. When i j. the element si represents how likely the corresponding instance is to become an exemplar. The vector of diagonal elements, (811, 822, S,n), called preference. Non-diagonal elements S constitute the affinity matrix. The algorithm takes S as input and proceeds by updating two matrices: the responsibility matrix R whose values re,j represent whether xy is well-suited to be the exemplar for zı; the availability matrix A whose values di quantify the appropriateness for z; picking z; as its exemplar (Frey and Dueck, 201Z7). These two matrices are initialized to be zeroes and can be regarded as fog-probability tables. The algorithm then alternatives between two message-passing steps. First, the responsibility matrix updated:
T& f S6,
max (ass) + 5t,r j'H)
Then, the availability matrix is updated:
ai + min (0.ris+ + Σ max((O,rr,)) for i # jj; ajs +
max (erx)) rr.).
Messages are passed until the clusters stabilize or the pre-determined number of iterations is reached. Then the exemplar of i is arg max, Fig + di.j-
3 Method
In this section, we present our Cenergy-Orientee LeARning (SOLAR) scheme that incorporates the muscle synergy mechanism into modular reinforcement learning to reduce its learning complexity.
Our method has two major components. The first one is an unsupervised learning module that utilizes the morphological structure and value information to discover the synergy hierarchy. The second is a novel iiveettionbbaeld policy architecture that supports synergy the components are specially designed to enable the control of robots with different morphologies. We first introduce the problem settings and then describe the details of the two components.
learning. Both of
Problem settings. We consider robots, each with a unique morphology. Agent n contains K, limb actuators that are connected together to constitute its overall morphological structure. Exam-
3

---

ples of such robots that are studied in this paper include Humanoid++ and UNIMALS. At each discrete timestep t, actuator & ...2..... K of robot 2,., receives local sen- sory state St as input and outputs individual torque values Or n..e for the corresponding actuator. Then the robot executes the joint action at = (a,"" Jk=1 at time t, after which the environment
K, "Kn
all limbs of the agent n and collective reward for the whole morphology ri ( a.)) We learn a policy
Mk,, returns the next state 81+1 - (s (= corresponding
generate actions based on states. The learning objective the policy is to maximize the expected return on all the tasks:
[(0) Eg Σ (s)r(()))))))
where y is discount factor. We adopt aetor-critic framework for policy learning. The critic is shared among all tasks and estimates the Q-function for each robot n:
(s" a²)) )- E.
(s)
: { s", ab =a"]
(4)
Preference: 60..
Affinity:ex(--D (DD))
Intra Synergy Self Self-Attention
Inter- Synergy Self- Self-Attention
Action Transformation
Per actuator action
(Gaan Structure
Affinity Propsgation
Synergy S.
Robot
State
Hidden Rapr. Ang
Synergy Act. di
Actuator Act. 9x,
Figure 1: Synergy-aware policy learning. The intra-synergy attention module aggregates actuator information within each synergy. The inter-synergy attention module synthesizes information from all synergies to produce synergy actions. Synergy actions are then transformed linearly to obtain actuator actions. Actuator actions are lower rank, reducing the control complexity.
3.1 Discovering synergy structure
In Neuroscience, the muscle synergies are usually discovered by a factorization of electrical muscle signals during performing tasks [[oddory and Ghahramani, 200 Rabbi ot al. 2020) Factorization a method hindsight it statistically analyzes the optimal control policies of animals embodied in the electrical signals. By contrast, in reinforcement learning, do not have the optimal con- trol policies advance. The synergy hierarchy learned from non-optimal policies is more likely incorrect, which will hamper policy learning. Therefore, propose learning the synergy hierarchy by an unsupervised learning method that incorporates morphological information besides learning information. In this section, we describe our synergy hierarchy discovery methods.
Intuitively, actuators the same synergy are activated simultaneously and they together finish motion of end effector. This gives hint that actuators with similar functions supposed to be a synergy. Formally, the function of an actuator (the kth actuator robot n) can be modelled by its influence to the value function:
AQ Eg an [QT (s", (a am==1))= Q= [B", am=+))]
(5)
where [. combines two terms, n. is the actual action of actuator k. gack is default action of actuator & (b 0 in MuJoCo environments) and a - is the actions of actuators robot

---

except for the kth one. In practice, we use a SoftMax function to regularize AQ AQn.k exp(AQn,x)/ E, exp(AQn.j)-
Ideally, AQn,k contains actuator clustering information, but the Q-values may be inaccurate during learning. We propose to treat (AQm.., 2Q..2...... AQn,K,, propagation clustering algorithm (Sec. D). i.e, the diagonal elements of similarity matrix, and in the next paragraph further encode stable morphological information in the affinity matrix of affinity propagation, i.e, the non-diagonal elements of similarity matrix. The advantage of this design is that (1) can simultaneously consider functional similarity and morphological context when using clustering to discover synergies and (2) static morphological information may help stabilize the clustering results.
the preference vector of the affinity
We now discuss how to encode morphological information in the affinity matrix. A robot morphol- ogy is usually organized in a tree structure with its torso the root node [[uang et al, 2020]. We define the adjacency matrix A, E {0, 1] K,JKK,K,, x.,, to represent robot's morphology, where A.j = 1 if actuators j are directly linked. To measure the connectiveness of every actuator pair, we use Floyd- Warshall algorithm eo 2022] to Comp the shortest distances between actuators based Ad, resulting in distance matrix D NK. x xK., The matrix cxp(- will serve as the affinity matrix A of affinity propagation. After defining the preference and affinity for affinity propagation, the clustering algorithms run Nmaa rounds of message passing to determine actuator clusters. We treat each cluster as a synergy.
In practice, we update the synergy structure periodically during the learning process. While D remains unchanged, AQn,k changes as the Q-function is updated. Each time we run the affinity propagation with the new preference vector and then freeze the synergy structure for a while.
3.2 Learning synergy-aware low-rank policies
We now describe how we design a synergy-aware actor-critic learning framework that exploits the synergy hierarchy discovered in Sec. B
The core idea is that the policy generates control signals (actions) for synergies instead of actuators. As the number of synergies is much fewer than actuators, the learning complexity is reduced. The synergy actions will combined linearly to generate physical actions for actuators. this way, we learn low-rank control policy the physical actions are actually a linear combination of actions in a low-dimensional space.
Suppose that we find Ln synergies (m for the nith robot. We use mask M,, E {0. K..KK, to represent the synergy structure. The element at the ith row,jth column of M, is if actuator and j are in the same synergy. The policy He consists of three components: intra-synergy attention module, an inter-synergy attention module, and an action transformation matrix as in Figure II
The first module aggregates information of actuators in synergy. Input to this module is the joint state 8n, and this module outputs a hidden representation hn, E Rd for each synergy S..s. In practice,
transformer with two-head self attentttn layer where the attentions between actuators belonging to different synergies are masked out. The output of the self-attention layer is of the size K, d. We then mean pooling to aggregate information of actuators in the same synergy. In this way, we get hidden state hn E IR Lnd with each row being the aggregated information of synergy.
The second module aggregates information from different synergies and outputs the synergy actions. It has two multi-head self-attention layers. We feed hn into this these layers and get vector-valued synergy actions a", E _L. x1 The synergy actions are then transformed by learnable matrices T e RK.XL, and we obtain actions for actuators by a, Ta One question about the transformation matrix T is that L, may change during the learning process, requiring the dimension of T to be dynamic. In Appendix Al we discuss how to deal with this problem.
The critic uses the same intra- and inter-synergy attention structure as the policy for processing state information. The outputs of the inter-synergy attention module are fed into fully-connected network to obtain the Q-value estimation. In practice we use TD3 [Fujimoto et alj, 2018) for the training of the policy and the critic. The intra- and inter-synergy transformer and the transformation
5

---

matrix are updated in an end-to-end manner. We provide more details of our learning framework in Appendix [
4 Experiments
In this section, we benchmark our method SOLAR on various MuJoCo [[odorov el all 2012] loco- motion tasks. we evaluate the effectiveness of SOLAR by asking the following questions: (1) Can our SOLAR outperforms other modular RL approaches when simultaneously trained on large num- ber of diverse agents with different morphologies? (Sec. H.2) (2) Can the learned policy in multi-task environments generalize to new tasks with unseen morphologies? (Sec. B. (3) How does SOLAR learn synergy clusters and how do synergy clusters facilitate learning? (Sec. -=!! (4) Can SOLAR
scale to single-tasks with numerous joints? (Sec. E.5).. For qualitative results, please refer to the videos on our project website". And our code is available at GitHub
4.1 Experiment setup
We run experiments on two benchmarks, and report several results in this section. For additional results, please refer to Appendix B We test all methods with 4 random seeds and show the mean performance as well as 95% confidence intervals.
Environments. For multi-task and zero-shot evaluation, we adopt the widely-used modular MTRL benchmarks [[uang et all 2020, Kurin all 2020 Hong al, [02I]] which are created based on Gym MuJoCo locomotion tasks by Huang el all [2020]] In this section, we report results on in-domain settings: (1) 6 variants of walker [Walker+t] (2) variants of humanoid [Humanoid+ d+++] (3) 3 variants of hopper [Hopper++]; and cross-domain settings: all 6 variants of walker, all 8 variants of humanoids, and all variants of hopper evaluation, we sample 3 morphologies from UNIMALS [Gupta et all, 2021hal) which are obtained from evolution and have numerous joints, making it suitable to evaluate the performance of modular RL policies in single-tasks. For detailed description of the environments please refer to Gupta atal [2021]] and Appendix [
(opoidepooprtt For single-task
Baselines. In this section, we compare our method SOLAR against state-of-the-art modular RL methods SMP (Huang all, 2020] and AMORPHEBS [Kurmı et all 2020]. SMP passes messages along the limbs using a bottom-up and top-down scheme. SOLAR and AMORPIIEUS use transformer- based actors and critics, whose message passing schemes realized by self-attention. We also compare SOLAR with standard TD3-based non-modular RL: Monolithic. Please refer to Appendix A for more details about baselines.
Ablations. There are two contributions that characterize our method. (1) An unsupervised learning module that utilizes the value information and the morphological contextual structure discover the synergy hierarchy. (2) A novel attention-basel synergy-aware policy architecture that supports synergy-awar learning. Our novelties mainly about the discovery and utilization of synergies, and these two contributions closely rely on each other. If we totally ablate synergies, we will get AMORPHBBS. Therefore, the effectiveness of synergy-based learning can be demonstrated by com- paring SOLAR against PMORPIEUS. We further design the following ablations: (1) AMORPHEUS w/ synergy mask. Incorporate synergy- aware inter-cluster masks into AMORPHEUS test the contri- bution of synergy hierarchy. (2) SOLAR w/o preference. Remove value information (the preference vector) from affinity propagation of SOLAR. In this case, affinity propagation is based solely on morphological information, and we can test the effects of value information.
Implementations. We use TD3 [Fujimoto et all 2013 the underlying reinforcement learning algorithm for training the policy over all baselines, ablations and our method for fairnsss. We implement SOLAR in the AMORPHEUS codebase. And like AMORPHEUS, there is no weight sharing between actor and critic. SOLAR uses a traversal-based embedding for action transformation, which may incorporate structural information. For a fair comparison, we concatenate these embeddings into the original observation vectors of each environment which is applied to all tested algorithms.

---

4.2
Multi-task with different morphologies
SOLAR (Ours)
Amorphous
SMP
Monalithic
w/o preference
Amorpheus synergy mask
Episade Reward
4000
5000
Hummnood+++
Epicode Reward
3000
Hopper++
Episodde Reward
1500
Walken-llummnood Hopper++
2000
2000
2000
2000
1000
1000
8.0
0.5
1.0
8.0
0.5
1.0
8.0
0.50
0.75
1.00
8.0
1.0
1.5
[mil)
T(mil)
[mml)
Figure Multi-task performance our method SOLAR compared to baselines and ablations.
(Ours)
Amorphous
SOLAR w/o preference
Amorpheus w/ mask
6000
humanoid 26 left arm
6000
humeroid 7 lower arms
1500
humanoid 2d right leg
Reword
Epicode Rowardd
Episode Reward
1000
2000
8.0
0.5
1.0
1.5
8.0
0.5
1.0
2.0
8.0
0.5
1.0
1.5
2.0
(ml))
T [mill
(mll)
right
6000
Reward
Epicode Reward
Reward
1.50
1500
4000
2000
1000
2000
8.0
0.5
1.0
1.5
2.0
8.0
0.5
1.0
1.5
2.0
0.5
3.0
1.5
2.0
(mll)
[mill
(mil)
Figure 3: Multi-task performance of our method SOLAR compared to baseline and ablations on Humanoid+t.
We summarize the multi-task results in Figure 2 SOLAR outperforms the previous state-of-the- art algorithms AMORPHEUS and SMP in all settings. Furthermore, the performance gap between SOLAR and AMORPHEUS is notably larger in Humanoidt+ which has the maximum number of joints among the tested settings. We also plot the learning curves of six training variants (Figure R) and two testing variants (Figure HD of Humanoid++. These results consistently demonstrate the effectiveness of SOLAR in training robots with large degree of freedom (DoF). We speculate that the reason of the effectiveness is that SOLAR controls robots in lower-rank action space than the original space and the action spaces of robots with a large DoF are more likely to be reduced via the synergy mechanism. To further validate this speculation, conduct extra evaluations robots with much more joints in Sec. A
As for ablations, AMORPHEUS w/ synergy mask outperforms AMORPHEUS in most tasks in Figure D and Figure E. which indicates the effectiveness the unsupervised learning method that discovers synergy clusters and synergy-aware masks in the policy. However, SOLAR w/o preference performs comparable to or even worse than AMORPHEBS, which reflects the prominent usefulness of value information.
4.3 Zero-shot generalization
In this section, benchmark SOLAR in a zero-shot learning setting where the policy is trained in multiple tasks and then generalized to unseen tasks. We compare SOLAR with AMORPHEUS, SMP, AMORPHEUS w/ synergy mask and SOLAR w/o preference Humanoidt+, which has the largest number of joints among the tested environments. Humanoid++ has eight variants of humanoids, where six of them are used as training tasks (see Figure K and the other two are used as testing tasks (Figure H). To obtain the results of testing tasks at each time step, we periodically load policy network parameters when training and evaluate the policy in these tasks. The trajectories on testing tasks will be discarded and will not be used for learning.

---

SOLAR (Ours)
Amorpheus
SMP
Menolithic
SOLAR wro preference
Amorpheus w/ synergy mask
Humandid+
1500
humanoid 2d 7 left leg
6000
humanoid 2d right am
Rleaard
3000
Epicode Rewarrd
Egicode Reward
2000
4000
2000
1000
2000
8.0
0.3
1.0 ((mll)
1.5
2.0
8.0
8.0
(mil)
Figure 4: Zero-shot performance of our method SOLAR compared against baseline and ablations.
As shown in Figure g. SOLAR outperforms all the baselines by large margin in these unseen tasks. SOLAR learns faster than other baselines and shows higher sample efficiency Figure B showing that the knowledge obtained from training tasks is effectively used in unseen tasks.
4.4 Analysis of synergies
A Synergy Cluster
Cluster Centers
Left Arm
Left Knee
Left Leg
Lower Arms
Right Arm
Right Knee
Right Leg
Full
Figure 5: Synergy clustering results of SOLAR in Humanoid++. Different colors represent different synergy clusters, and joints marked with the same color are in the same cluster. Joints marked with triangles are the centers of their corresponding clusters.
To investigate why SOLAR performs better than baselines, in this section, we visualize the synergy clusters learned by SOLAR and the evolutionary process of synergy clusters in Humanoidtt.
Affinity propagation which we used as the method to discover synergy clusters can naturally outputs clustering results and the cluster centers in each cluster. We first visualize the synergy clusters of SOLAR Humanoid++ at the end of training in Figure E Here we use different colors and two shapes to mark each joint in robots. Different colors represent different synergy clusters, and joints

---

humanoid 2d 8 left knee
3500
Phase
Phase
Phase
3000
Episode Reward
2500
2000
1500
Mean Episode Reward
of Amorpheus
1000
500
8.00
J.25
0.50
0.75
1.00
1.25
1.50
1.75
2.00
(mil)
left upper arm
right upper arm
left lower arm
right lower arm
right thigh
left thigh
right shin
Figure 6: Synergy structure evolution of SOLAR in Humanoid. Phase are divided according to the change of synergy clusters and synergy clusters are masked with colored shapes.
marked with the same color in the same cluster. Joints marked with triangles are the centers of their corresponding clusters. For example, the Full humanoid in the lowermost right corner has 4 synergy clusters, and the joints between the torso and two thighs and two arms are the cluster centers.
To further investigate how these clusters evolve, we visualize the learning process of Left Knee Humanoid in Figure fl According to the change of synergy clusters, the learning process can be divided into three phases. In Phase 1, the episode reward very low and AQ may only provide noisy information, which results in all joints being clustered in same cluster. In Phase SOLAR finds that the right shin is important and divides it into a separate cluster. Then in Phase 3, SOLAR finally obtains suitable clustering solution and the episode reward increases correspondingly.
The visualization results reflect two facts: (1) close joints are more likely to in the same synergy cluster, and (2) joints near the torso may be more influential than those who are far from the torso, and are thus selected as the cluster centers. These clusters can reduce learning and controlling com- plexities but hard to discover through standard learning mechanism. Instead, SOL.RR makes of early, less accurate learning information AQ and structural information via unsupervised learning to discover synergy clusters. The synergy clusters then further facilitate learning.
4.5 Single-task with numerous joints
To consolidate the speculation in Sec. ! that SOLAR suitable for tasks with numerous joints, we test SOLAR in three single-robot tasks sampled from UNIMALS [Gupta et all 20216 The average number joints about twice as many as that in Humanoid++. please refer to Appendix Al We showcase the results in Figure @ SOLAR learns faster than ofate-of- the-art A AMORPHEUS, showing much higher sample efficiency, and again outperforms average episode rewards.
more details about these tasks,

---

SOLAR (Ours)
Amorpheus
4000
Unimal
3000
Unimal
Unimal
Eploade Reward
Episada Roward
4000
0000
2000
Episode
2000
2000
1000
8.0
8.6
1.0
8.0
0.6
(mil)
[mill
(mill
Figure 7: Multi-Task performance of method SOLAR compared to AMORPHEUS
Conclusions
In this paper, use the inspiration of muscle synergies from neuroscience to reduce the control complexity of modular reinforcement learning in tasks with large number of degree of freedom. We present SOLAR that leverages both structural information and policy learning information to learn synergy clusters. Moreover, a trankeddtransoormer architecture is proposed for learning syn- ergy actions which are combined linearly produce low-rank actuator actions. The evaluations of SOLAR on various tasks and settings showcase its effectiveness. The authors do not see obvious negative societal impacts of the proposed method.
Acknowledgments
This work is supported in part by Science and Technology Innovation 2030 New Generation Artifi- cial Intelligence Major Project (No. 2018AAA0100904) and National Natural Science Foundation of China (62176135).
References
Nicolas Heess, Dhruva TB, Srinivasan Sriram, Jay Lemmon, Josh Merel, Greg Wayne, Yuval Tassa, Tom Erez, Ziyu Wang, SM Eslami, et al. Emergence of locomotion behaviours in rich environ- ments. arXiv preprint arXiv: 02286, 2017.
Yuke Zhu, Josiah Wong, Ajay Mandlekar, and Roberto Martin-Martin. robosuite: A modular simu- lation framework and benchmark for robot learning. arXiv preprint arXiv:2009.12299: 2020.
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuo- motor policies. The Journal of Machine Learning Research, 17(1):1334-1373, 2016.
Rupam Mahmood, Dmytro Korenkevych, Gautham Vasan, William Ma, and James Bergstra. Benchmarking reinforcement learning algorithms real-world robots. In Conference on robot learning, pages 561-591. PMLR, 2018.
Xinyue Chen, Che Wang, Zijian Zhou, and Keith Ross. Randomized ensembled double q- learning: Learning fast without model. In International Conference on Learning Represen- tations, 2020.
Arsenii Kuznetsov, Pavel Shvechikov, Alexander Grishin, and Dmitry Vetrov. Controlling overesti- mation bias with truncated mixture of continuous distributional quantile critics. In International Conference on Machine Learning, pages 5556-5566. PMLR, 2020.
Tingwu Wang, Renjie Liao, Jimmy Ba, and Sanja Fidler. Nervenet: Learning structured policy with
graph neural networks. In International conference on learning representations, 2018.
Bei Peng, Tabish Rashid, Christian Schroeder de Witt, Pierre-Alexandre Kamienny, Philip Torr, Wendelin Böhmer, and Shimon Whiteson. Facmac: Factored multi-agent centralised policy gra- dients. Advances in Neural Information Processing Systems, 34, 2021.
Wenlong Huang, Igor Mordatch, and Deepak Pathak. One policy to control them all: Shared modular policies for agent-agnostic control. International Conference on Machine Learning, pages 4455-4464. PMLR, 2020.
10

---

Vitaly Kurin, Maximilian Igl, Tim Rocktäschel, Wendelin Boehmer, and Shimon Whiteson. My body is a cage: the role of morphology in graph-based incompatible control. In International Conference on Learning Representations, 2020.
Agrim Gupta, Linxi Fan, Surya Ganguli, and Li Fei-Fei. Metamorph: Learning universal controllers with transformers. In International Conference on Learning Representations, 2021a.
Andrea d' Avella, Philippe Saltiel, and Emilio Bizzi. Combinations of muscle synergies in the con-
struction of a natural motor behavior. Nature neuroscience, 6(3):300-308, 2003.
Lena H Ting and J Lucas McKay. Neuromechanies of muscle synergies for posture and movement.
Current opinion in neurobiology, 17(6):622-628, 2007.
Akira Saito, Aya Tomita, Ryosuke Ando, Kohei Watanabe, and Hiroshi Akima. Muscle synergies are consistent across level and uphill treadmill running. Scientific reports, 8(1):1-10, 2018.
Ali Falaki, Xuemei Huang, Mechelle M Lewis, and Mark L Latash. Motor equivalence and structure of variance: Musti-muscle postural synergies in parkinsons disease. Experimental brain research, 235(7):2243-222) 2258, 2017.
Paulina Kieliba, Peppino Tropea, Elvira Pirondini, Martina Coscia, Silvestro Micera, and Fiorenzo Artoni. How are muscle synergies affected by electromyography pre-processing? IEEE Transac- tions on Neural Systems and Rehabilitation Engineering, 26(4):882-89 893222999222 2018.
Brendan J Frey and Delbert Dueck. Clustering by passing messages between data points. science, 315(5814):9729977 976, 2007.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference intelligent robots and systems, pages 5026-5033. IEEE, 2012.
Agrim Gupta, Silvio Savarese, Surya Ganguli, and Li Fei-Fei. Embodied intelligence via learning and evolution. Nature communications, 12(1):1-12, 2021b.
Deepak Pathak, Christopher Lu, Trevor Darrell, Phillip Isola, and Alexei Efros. Learning to control self-assembling morphologies: study of generalization via modularity. Advances in Neural Information Processing Systems, 32, 2019.
Tytus Wojtara, Fady Alnajjar, Shingo Shimoda, and Hidenori Kimura. Muscle synergy stability and human balance maintenance. Journal of neuroengineering and rehabilitation, 11(1):1-9, 2014.
Simona Ferrante, Noelia Chia Bejarano, Emilia Ambrosini, Antonio Nardone, Anna M Turcato, Marco Monticone, Giancarlo Ferrigno, and Alessandra Pedrocchi. A personalized multi-channel fes controller based on musele synergies to support gait rehabilitation after stroke. Frontiers in neuroscience, 10:425, 2016.
Matthew C Tressch, Philippe Saltiel, Andrea dAvella, and Emilio Bizzi. Coordination and localiza- tion in spinal motor systems. Brain Research Reviews, 40(1-3))66677) 220666679..22......... 2002.
Rajat Emanuel Singh, Kamran Iqbal, Gannon White, and Tarun Edgar Hutchinson. systematic review muscle synergies: from building blocks of motor behavior to neurorehabilitation tool. Applied bionics and biomechanics, 2018, 2018
Mohammad Fazle Rabbi, Claudio Pizzolato, David G Lloyd, Chris P Carty, Daniel Devaprakash, and Laura E Diamond. Non-negative matrix factorisation is the most appropriate method for extraction of muscle synergies in walking and running. Scientific reports, 10(1):1-11, 2020.
Katherine M Stecle, Adam Rozumalski, and Michael H Schwartz. Muscle synergies and complex- ity of neuromuscular control during gait in cerebral palsy. Developmental Medicine & Child Neurology, 57(12):
2015.
Michael H Schwartz, Adam Rozumalski, and Katherine M Steele. Dynamic motor control is associ- ated with treatment outcomes for children with cerebral palsy. Developmental Medicine & Child Neurology, 58(11):1139-1144 2016.

---

Daniel D Lee and H Sebastian Seung. Learning the parts of objects by non non-negative matrix factor- ization. Nature, 401(67555)788-799)) 1999.
Adam Rozumalski, Katherine M Steele, and Michael H Schwartz. Musele synergies are similar when typically developing children walk treadmill different speeds and slopes. Journal of biomechanics, 64:1 12-119, 2017.
Benjamin Shuman, Marije Goudriaan, Lynn Bar-On, Michael H Schwartz, Kaat Desloovere, and Katherine M Steele. Repeatability of muscle synergies within and between days for typically developing children and children with cerebral palsy. Gail & Posture, 45:127-132, 2016.
Lena H. Ting and Jane M Macpherson. A limited set of muscle synergies for force control during postural task. Journal of neurophysiology. 93(1):609-613, 2005.
Lena H Ting, Hillel J Chiel, Randy D Trumbower, Jessica L. Allen, J Lucas McKay, Madeleine E Hackney, and Trisha M Kesar. Neuromechanical principles underlying movement modularity and their implications for rehabilitation. Neuron, 86(1):38-54, 2015.
Frédéric Danion and Mark Latash. Motor control: theories, experiments, and applications. Oxford
University Press, 2010.
Aapo Hyvärinen and Erkki Oja. Independent component analysis: algorithms and applications. Neural networks, 1344-5):411-44 1-4000,, 2000.
Corey Hart and Simon Giszter. Distinguishing synchronous and time-varying synergies using
point process interval statisties: motor primitives in frog and Frontiers computational neuroscience, 7:52, 2013.
Akira Saito, Kohei Watanabe, and Hiroshi Akima. Coordination among thigh muscles including the vastus intermedius and adductor magnus at different cycling intensities. Human Movement Science, 40:14-23 2015.
Gianluca Palli, Fanny Ficuciello, Umberto Scarcia, Claudio Melchiorri, and Bruno Sieiliano. Ex- perimental evaluation of synergy-based in-hand manipulation. IFAC Proceedings Volumes, 47(3): 299-304, 2014.
Thomas Wimböck, Benjamin Jahn, and Gerd Hirzinger. Synergy level impedance control for mul- tifingered hands. In 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 973-979. IEEE, 2011.
Fanny Ficuciello, Damiano Zaccara, and Bruno Siciliano. Synergy-based policy improvement with path integrals for anthropomorphic hands. In 2016 IEEE/RSJ International Conference Intelli- gent Robots and Systems (IROS), pages 1940-1945. IEEE, 2016.
Emanuel Todorov and Zoubin Ghahramani. Analysis of the synergies underlying complex hand manipulation. In The 26th Annual International Conference of the IEEE Engineering Medicine and Biology Society, volume 2, pages 4637-4640. IEEE, 2004.
Thomas H Cormen, Charles E Leiserson, Ronald L Rivest, and Clifford Stein. Introduction to algorithms. MIT press, 2022.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International Conference on Machine Learning, pages 1587-1596, 2018.
Sunghoon Hong, Deunsol Yoon, and Kee-Eung Kim. Structure-aware transformer policy for inho-
mogencous multi-task reinforcement learning. In International Conference on Learning Repre- sentations, 2021.
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arX1vv1600.01140,, 2016.
Sequence-to-sequencc modelng with trantsnsforeer and torchtext. https://pytorch.. org/
futorralls/oggimee//tttttffooon http://rrremmemmmmmetrtrrraraaaa 2020.
tutorial.htm.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you descrihe the limitations of your work? [Yes] In Appendices.
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as a URL)? [Yes] In Appendices and supplemental materials.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] In Appendices.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] All results have confidence intervals.
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes] In Appendices.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] In Sec. A.
(b) Did you mention the license of the assets? [Yes] In Appendices.
(c) Did you include any assets either in the supplemental material URL? [Yes] Website for qualitative results.
(d) Did you discuss whether and how consent was obtained from people whose data you're re using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifi- able information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "deqURfzJpoM", "reviewer": "Reviewer_a1E4", "summary": "The authors propose a Synergy-Oriented LeARning (SOLAR) framework that exploits the redundant nature of DoF in robot control.  The method is inspired by the well-known idea of \"muscle synergy\" which the human central nervous system has and reduces the DOF of the whole-body musculoskeletal system. The synergy is learned in an unsupervised manner.  The experimental results show the effectiveness of the approach.", "questions": "To what extent does the proposed approach share the same idea with the conventional kinematic or muscle synergies which are often analyzed by PCA. It may be fruitful to show the mathematical comparison explicitly. I think many ML researchers may not know muscle synergy and researchers who are familiar with muscle synergy (e.g., people studying human body control, musculoskeletal system, CPG, and so on) don't know much about this type of ML studies. This paper potentially becomes an important paper bridging the two separated communities. From that viewpoint, the connection between the proposed method and the conventional notion of muscle synergy is better to be described more clearly in my opinion (regarding the relation to the self-attention mechanism in particular).\n\nDo you even hypothesize that human muscle synergies are organized in this way, or do you have any implications about the human muscle synergy obtained through this study?  Such a discussion may add extra value to the paper.", "limitations": "It is better to compare not only modular RL methods but also SOTA central RL methods will be beneficial for discussion. ", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "+ Application of the notion of muscle synergy to modular reinforcement learning (in an unsupervised manner) is quite new and interesting.\n+ Experimental results show the effectiveness of the method.\n+ Emerged synergies are analyzed and exhibit interesting results.\n- The relationship between the conventional muscle synergy and the self-attention mechanism is slightly unclear.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "aYV2oolsu3R", "reviewer": "Reviewer_tr2n", "summary": "In this paper, the authors focus on the problem of modular RL for systems with a large degrees of freedom such as humanoid characters. \n\nThe central idea of this work can be divided into two parts. The first is the use of an unsupervised clustering algorithm to group different actuators on the agent based on their impact on the value function and the agent’s morphology. This is done using an existing algorithm named affinity propagation. The second part is an attention-based architecture that achieves a hierarchical controller where the policy produces low dimensional synergy actions, which are then mapped to per-actuator actions.\n\nIn the experiments, the method is evaluated on locomotion tasks for a variety of morphologies like walker, humanoid, unimal, etc. When compared to several state of the art baseline methods, the proposed algorithm shows superior learning performance. An ablation study was also carried out to demonstrate the importance of the proposed idea of learning synergy and the attention-based model.", "questions": "In addition to some of the issues mentioned above, I also have the following questions:\n1. When training in a multi-task setting, do all tasks (variant of the agents) share the same critic and actor weights?\n2. For synergy of different morphologies, my understanding is that they are independently computed, is that right? I.e. the subfigures in Fig 5 are independent even some of them share the same color?\n3. How frequent are the actor-critic update interleaved with the update in the synergy?\n4. According to the text, the motivation for the proposed synergy formulation is that the one used in neuroscience requires optimal policies and the clustering could be erroneous. However, the proposed method also suffers from this issue as value functions would also be wrong at the beginning. By interleaving the policy learning and synergy discovery this doesn't really seem to be a problem, as demonstrated in Figure 6. I'm wondering if there are other issues or potential benefits with using a more neuroscience-inspired approach.\n", "limitations": "The author didn't discuss much limitation about the proposed work. I would suggest adding some discussions in terms of what kind of problem the current formulation may not be suitable for.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1) The idea of synergy for lowering the control complexity in high dof systems is reasonable and the proposed method for finding the synergy amongst actuators is solid.\n2) The experiments in general support the proposed idea and achieves good performance in highly challenging control problems.\n3) The analysis is pretty thorough, including an ablation study to show the impact of the proposed ideas, as well as a visualization of the discovered synergy throughout learning, which is helpful in gaining insights into the algorithm.\n4) The method could potentially be applied to other high dof control problems.\n\nWeaknesses:\n1) The proposed seems especially effective for the humanoid task, while achieving comparable results for the baselines on the other tasks. It’d be helpful to have some discussion regarding why humanoid shows the biggest difference.\n2) Currently the tasks have been focused on one locomotion task (go forward). If more tasks can be demonstrated it can further support the effectiveness of the method.\n3) Some details are missing, as mentioned below.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "TrdWdFRp8u_", "reviewer": "Reviewer_NEKf", "summary": "In this paper, the authors proposed a new modular reinforcement learning algorithm using unsupervised learning. By auto clustering joints that have synergies, i.e. based on their contribution to the tasks through value functions, the proposed method alleviates the control to the synergy space instead of the joint space. Then, a learned mapping function will convert the synergy action to joint action signals. With the proposed framework, the authors show that they can significantly outperform the SOTA baselines SMP and AMORPHEUS in many multiple task benchmarks. ", "questions": "See the weaknesses section for revising recommendations. ", "limitations": "N/A", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The strengths of the paper:\n\nThe results of the paper show a clear edge from the proposed framework in many benchmarks. \n\nThe authors did a good amount of ablation study. I especially like how the authors visualize the synergy group’s propagation over time in Figure 5. \n\nThe weaknesses of the paper:\n\nThe description of affinity propagation can be more clear. Currently affinity propagation is introduced in the background section and the writing is very similar to what has been written on Wikipedia. It is better to incorporate this section to methods, and define the similarity, responsibility and availability matrices in the context of the proposed system. For example, I assume that the exponential of the distance matrix in section 3.1 is used for the off-diagonal elements of the similarity matrices while delta-Q values are used as the diagonal elements. Consider re-writing this part and making it clear.. \n\nSometimes, notations are used without proper definition. For example, the Q function in Eq (5) suddenly takes three different types of inputs, with both bold and non-bold symbols. This makes it difficult for the readers to understand. Consider revising. \n\nNo illustration diagrams of the proposed network architecture for section 3.2. It is a bit hard to decipher the data flow just from the text descriptions. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept