MyoDex: Generalizable Representations for Dexterous Physiological Manipulation
Authors: Vittorio Caggiano
Sudeep Dasari
Vikash Kumar
Venue: Submitted to ICLR 2023 (2023)
Keywords: Musculoskeletal
Machine Learning
human dexterity
muscle synergies

Abstract: The complexity of human dexterity has attracted attention from multiple fields. Still, much is to be understood about how hand manipulation behaviors emerge. In this work we aim at learning dexterous manipulation behaviors with a physiologically realistic hand model: MyoHand. In contrast to prior works demonstrating isolated postural and force control, here we demonstrate musculoskeletal agents (MyoDex) exhibiting contact-rich dynamic dexterous manipulation behaviors in simulation. Furthermore, to demonstrate generalization, we show that a single MyoDex agent can be trained to solve up-to 14 different contact-rich tasks. Aligned with human development, simultaneous learning of multiple tasks imparts physiological coordinated muscle contractions i.e., muscle synergies, that are not only shared amongst those in-domain tasks but are also effective in out-of-domain tasks. By leveraging these pre-trained manipulation synergies, we show generalization to 14 additional previously unsolved tasks. While physiological behaviors with large muscle groups (such as legged-locomotion, arm-reaching, etc), have been demonstrated before, to the best of our knowledge nimble behaviors of this complexity with smaller muscle groups are being demonstrated for the first time.

Under review as a conference paper at ICLR 2023
MyoDex: GENERALIZABLE REPRESENTATIONS FOR DEXTEROUS PHYSIOLOGICAL MANIPULATION
Anonymous authors Paper under double-blind review
Figure Contact rich manipulation hehaviors acquired by MyoDex with physiological MyoHand
ABSTRACT
The complexity of human dexterity builds on the coordinated actuation of a large number of muscles. Still, much to be understood about how the control of such overactuated system for hand manipulation behaviors emerge and quickly and flexibly adapts to new behaviours. able representations for dexterous manipulation behaviors with a physiologically realistic hand model: MyoHand. In contrast to prior works demonstrating isolated postural and force control, here we demonstrate musculoskeletal agents (MyoDex) exhibiting contact-rich dynamic dexterous manipulation behaviors in simulation. Furthermore, to demonstrate generalization, can be trained to solve up-to 14 different contact-rich tasks. Aligned with hu- man development, simultancous learning of multiple tasks imparts physiological coordinated muscle contractions i.c., muscle synergies, that are not only shared amongst those in-domain tasks but also effective to large series of new out- of-domain tasks. By leveraging these pre-trained manipulation synergies, we show generalization 38 additional previously unsolved tasks. While phys- iological behaviors with large muscle groups (such as begged-locomotion arm- reaching, etc) have been demonstrated before, the best of our knowledge nimble behaviors of this complexity with smaller muscle groups and generalizable repre- sentations for the control of the overactuated human hand are being demonstrated for the first time. Project Webpage: http:///ttt.ggoggeecooom//www///
this work we aim learning generaliz-
show that a single MyoDex agent
Musculoskeletal, muscle synergies, Machine Learning, human dexterity
INTRODUCTION
Human hands are astonishingly complex and require effective coordination of various muscle groups to impart effective manipulation abilities. Manipulation behaviors are incredibly sophisticated as, because of the overactuated musculoskeletal system, they evolve in a high high-dimensional search space populated with intermittent contact dynamics between the hands" degrees of freedom and the object. Indeed, even the field of roboties where joints and actuations are simpler, finding effective manipulation strategies nonetheless remains a challenge Kumar et al. (2016) Rajeswaran ctt al. (2018); Nagabandi et al. (2020).
The human hand consists of 29 bones, 23 joints, and more than 50 muscles Sobinov & Bensmaia multi-joint, pulling-only properties of the musculoskeletal
20221. The complex mulli-articull system Sobinoy & Bensmaia (2021) make physiological dexterous manipulation very different and unique problem as opposed to joint based control typically adopted in robotics. biology, the control of such complex musculoskeletal system is made possible by the fact that muscles

---

Under review as a conference paper at ICLR 2023
not activated in isolation, but rather, that different muscles are activated in proportional way as a unit. This phenomenon is known as muscle synergy Bizzi & Cheung (2013). Synergies allows the biological motor system via the modular organization of the movements in the spinal cord Bizzi & Cheung 2 Caggiano et al. (2016) to simplify the control problem, solving tasks by building on a limited number of shared solutions d' Avella et al. (2003); d' Avella & Bizzi (2005). Those shared synergies are suggested to be the fundamental building blocks for quickly learning new and more complex motor behaviours Yang et al. (2019); Dominici al. (201 Cheung et al. (2220). Manipulation behaviors, the subject of this investigation, are further complicated because they unfold on a sequence of phases: reaching to the object, hand-object contact, and manipulation with object maneuvers. Before the hand-object contact, the human hand is pre-shaped to conform to the object such that it often possible to predict the object that is going to be grasped just by observing the hand pose before hand-object contact pettncc/e (1988); (Santello et al. (2002); (Thakur itt al. al. (2008): Yan et al. (2020). Contact and manipulation of the object are goal-driven so that the way the object is held depends on both the object affordance and the intermediate task goals Jeannerod 1988).
In this work, we seek to further our understanding of physiological dexterity by imparting dexter- ous manipulation ability to an anatomically realistic hand-fore-arm model [Caggiano et al. (2022).
While prior works have not been able scale beyond dexterous grasping McFarland et al. [202
Mirakhorlo et al.] (2018); Saito et al. (202T); Crouch & Huang (2015); Engelhardt et al. (2021) in
a controlled setting with physiologically realistic models of the hand, here we present MyoDer agents capable of dynamic dexterous contact rich manipulation behaviors with multiple objects and a variety of tasks c.g. drinking from a cup, playing with toys, etc. Furthermore, by jointly training multiple tasks, we capture reusable synergies in form of general pre- pre-trained policy that can be fur- ther fine-tuned to manipulate 38 previously unsolved tasks with non-trivial affordances. We provide a detailed analysis of emergent physiological details in our achieved behaviors.
While we do not claim to have solved physiological dexterous manipulation, we emphasize that ma- nipulation abilities demonstrated here significantly advance the state of the art of the bio-mechanics and ncuroscience fields. Along these lines, this investigation is among the first to yield robust control policies exhibiting basic physiological constructs such as synergistic activations of muscle groups during dexterous manipulations. Nevertheless, further work is required to rigorously ground them in experimental validation. More specifically, our main contributions are:
We show for the first time that despite the high numbers of degrees of freedom, the multi- articular-multi-jonn and the third order muscle dynamics of muscle control, it is possible to control a physiologically realistic musculoskeletal model of the hand to perform contact-rich skilled manipulation behaviors up-to 14 different tasks.
We show that joint multi-task learning facilitates the learning of physiological represen- tations that exploit musele coordination in a lower-dimensional space of synergies to solve specific tasks.
Our framework MyoDex leverages joint multi-task learning to recover reusable repre- sentations (synergies) that allows for easier fine-tuning in both in-domain and out-of- domain tasks (including one/few shot learning). Leveraging these synergies the MyoDex solves 38 previously unsolved tasks.
2 RELATED WORKS
Experimental studies of functional hand manipulations have been limited both by challenges in sensing, the discontinuous hand-object interactions and because of the limited ability record many muscles of the hand simultancously. Musculoskcletal models of the hand McFarland et al. (202T);
Lee et al. (2015p: Saul et al. (2015) have been developed to overcome some of the experimental limitations and produce insights on the kinematic information of the muscles and joints. While musculoskelatal models of large muscle groups have been extensively developed and used Delp et al. (2007); Seeh et al. (2018). models of the hand have been more challenging both because of the smaller muscle groups involved and the complexity of the behaviour they can produce. Indeed, simulations of the hand mostly focus on fingertips. pinch force McFarland et al. (2021). kinematic motion McFarland et al. (202I). and passive grasping McFarland et al. (2021). Furthermore. most of those studies are also limited by intensive computational needs and restricted contact forces. Those
2

---

Under review as a conference paper at ICLR 2023
conditions prevented the study of complex hand object interactions and limited by using optimization methods that could not leverage data-driven state of the art.
Recently, a new hand and wrist model MyoHand Caggiano et al. (2022) Wang al. (2022) has been developed. This model overcomes some limitations of alternative hand models and is suitable for computationally intensive data-driven explorations. Indeed, it has been shown that MyoHand can be trained to solve individual in-hand tasks on very simple geometries (ball, pen).
Hand dexterity has been also very active field in robotics. Robotic hands have been shown to be able to perform complex in-hand manipulation of real-world objects OpenAl et al. (2019); Huang eet al. (202T); [Chen et al. (202T) and solve complex manipulation tasks, such as HanaManipulatee and MandMaaipulatteeen Plappert et al.] (2018). Still, both the hardware and the control of robotic hands do not match the level of dexterity of human hands and remain limited in-hand movements.
Data driven approaches have consistently used Reinforcement Learning (RL) on joint-based control to solve simple locomotion tasks Miki et al. (2022). in animation of physics based characters Heess et al. (2017) and to solve complex dexterous manipulation in robotics RRaeswaran et al. (2018) ; Ku- mar et al. (2016); Nagabandi et al. (2019); Chen et al.) (2021). Typically, in order to yield more naturalistic movements, different methods have leveraged motion capture data |eeeel et al. (2017) 2019| Haseenceeree al. 2020). By means of those approaches, it has been possible to learn com- plex movements and athletic skills such as high jumps Yin et al. (2021), boxing and fencing Won ctt al. (202I) or playing basketball Liu & Hodgins (2018). More recently, approaches that prime models with hand pre-shaped for specific task have been shown to be successful simplifying the search of RL solutions on complex robotic manipulations Dasari et al.] (202)). In contrast to joint-based control, in biomechanical models machine learning has been applied on musele actua- tors to control movements and produce more naturalistic behaviors. This is a fundamentally different problem than robotic control as the overactuated control space of biomechanical systems leads to ineffective explorations Schumacher et al. (2022). Wang et al. Wang et al. (2012), Geijtenbeek et al. Geijtenbeek al. (2013). Borno et al. 2020 Al Borno al. (2020), and Ruckert al. Rückert & d' Avellla (2013), have been using optimization methods on biomechanical models to synthesize walking and running, reaching movements, and biped locomotion. More recently, deep reinforce- ment learning has been used to either map the muscle-actuation joint-actuation control to produce movements that are more human-like than those generated by torque-based control at the jointsJiang et al. (2019), in order to directly control shoulder and arm muscles for isometric arm movements Joos et al. (2020) and reaching |Schumacher et al.] (2022); Ikkala et al.] (2022). ) hand muscles for hand dexterous manipulations Caggiano al. (2022), co-learning elbow exoskeleton movements Caggiano et al. )2022); Wang et al. (2022). walking/running [Snn et al.] (2020); Park et al. (0022), or produce movements such as juggling, weight lifting, cart-wheeling and other highly stylistic movements Lee et al. (2018) 2019). Musculoskelctal models have been used also to improve the realism of simulated animal movements; for example, in controlling movements in animal models of dog Stark et al. (2021) and more recently, of an ostrich Barbera et al. (2021); Schumacher et al. (202)).
While musculoskeletal control with large muscles groups have been demonstrated Song et al. (2020) 2021 [Schumacher al. (2022): Ikkala et al. (2022), nimble contact rich musculoskeletal behaviors with smaller muscle groups such as hand- hand-manipulation remains an open challengeCaggilee
et al.
2022). MyoDex, in addition of showing that indeed it is possible, presents evidence that the learned physiological representations share muscle coordination across tasks which, like human synergistic control, facilitate both learning and generalization across tasks.
3 PHYSIOLOGICAL DEXTERITY
Human hand dexterity builds on the fundamental characteristics of the physiological actuation: mus- cle are multi-articular and multi-joints, the dynamics of the muscle is of the third order, muscle have pulling only capabilities, and coordinated synergistic muscle control with intermittent contact with objects. Furthermore, the key aspect of the control of such physiological effectors is that the hu- man central nervous system optimizes movements through coordinated muscle contraction - mus- cle synergies which are meant to simplify the control problem, allowing generalization. Fields of bio micheechanics, rehabilitation, neuro-surgery, etc. have long benefited from physiological un- derstanding of neuro-mechanical control. To further our understanding, here we embed the same
3

---

Under review as a conference paper at ICLR 2023
control challenges e.g. by controlling a physiologically accurate musculoskeletal model of the hand (see Sec. 3.1) in complex manipulation tasks (see Sec. 3.2). This allows a window to peek into the mechanisms behind human dexterity that enables generalization across different tasks.
3.1 PHYSIOLOGICALLY ACCURATE MYOHAND
In order to simulate a physiologically accurate hand model, a complex musculoskeletal hand model comprised of 29 bones, 23 joints, and 39 muscles-tendon units) Wang et al. (2022) -MyoHand model implemented in the MuJoCo physics simulator Todorov et al. (2012) was used (see Figure A.I This hand model has been shown allow dexterous in-hand manipulation of one or multiple objects when trained using reinforcement learning Caggiano et al. (0022)
We extended the MyoHand model to include translations and rotations at the level of the shoulder. We limited the translation on the frontal (range between [-0.07, 0.03) and longitudinal (range between 0.05, 0.05)) axis to favor shoulder and wrist rotation.
3.2 TASK
Dexterous manipulation is often posed problem of achieving the final configuration of the object. In this study we are interested to capture the whole continuous aspects the manipulation behaviour with object maneuver e.g., drinking, playing, cyclic movement like hammering. Those tasks are hard to capture as goal reaching. To effectively capture the temporal behaviors, we instead define dexterous manipulation as a task of realizing a desired object trajectory (X). We use two metrics to measure task performance. The object error metric E(X) calculates the average Euclidean distance between the object's 's center-of-mass position and the desired position from the desired trajectory:
E(X)= Ei I| ==fl2-- In addition, the success metric S(X) reports the fraction of time-steps where object error is below a e lem threshold. It is defined as: 5(X) =
+ + 1||= if lla <
3.3 DEXTERITY OBJECTIVES
While the complexity of human level dexterity can be hard to fully quantify, none the less we outline few objective measures we consider in this work. First, dexterous agent should be capable of exhibiting contact-rich manipulation behaviors. Next, the agent's behavior should seamlessly generalize to multiple tasks/objects without additional assumptions. Finally, these agents should exhibit coordinated muscles movements (synergies) that are shared amongst different behaviors, as well as generalize to new unseen tasks.
4 MyoDex: ACQUIRING DEXTERITY
In this section we discuss our approach to build agents that can learn contact-rich manipulation behaviors and generalize across tasks.
4.1 PROBLEM FORMULATION
manipulation task can formulated as Markov Decisions Process (MDP) Sutton & Barto (2018) and solved via Reinforcement Learning (RL). In RL paradigms, the Markov decision process is de- fined as a tuple M = (S, T, R. p. 7)). where S C R" and A C R" represent the continuous state and action spaces respectively. The unknown transition dynamics is described by s' T(+ a). R : S -> [0, Rmaxx denotes the reward function, 7 E [0, 1) denotes the discount factor, and and p the initial state distribution. In RI,, a policy is mapping from states to a probability distribution over actions, i.e. π S - P(A), which is parameterized by 0. The goal of the agent is to learn policy Te (a(s))) argmaxe [ M)). where J maxg Exg= aaooo (arls [E R(sr (( ac)]
For interpretabilly, omit orientations hecause center-of-mass enrrr and orientation error were highly correlated in practice
4

---

Under review as a conference paper at ICLR 2023
State Space. The state vector 8, = { 6l, the, the, ve) consisted of Φ a 29 dimensional vector of 23 hand and 6 arms joints and velocity Φ, and object pose & and velocity vh. In addition, positional encoding 7 Vaswani et al.) 2017). used to mark the current simulation timestep, was appended to the end of the state vector. This was needed for learning tasks with cyclic motions such as hammering.
Action Space. The action space dt was a 45-dimensional vector which consists of continuous activations for 39 muscles of wrist and fingers (to contract muscles), together with 3D translation (to allow for displacement in space), and 3D rotation of the shoulder (to allow for a wider range of arm movements).
Reward Function. The manipulation tasks we consider involved approaching the object and ma- nipulating in free air after lifting off a horizontal surface. The hand interacts with the object adjusting its positions and orientation X for a fixed time horizon. Similar to Dasari et al. (2022) this is translated into an optimization problem where we are searching for a policy that can match desired object trajectory X = [+0, ...,,"), which is captured using the following reward function:
R(I,, 22) == (axpp(-alaz? - 2121/12 3|<0, - 2/ + 12 As |7, l2
(1)
where 7 is the quaternion angle between the two orientations, x!!) is the desired object position, (o) It is the desired object orientation, L(lifted) encourages object lifting, and me the is overall
muscle effort.
4.2 PREGRASP TO SIMPLIFY SEARCH SPACES
Owing to the third order non linear actuation dynamics and high dimen- sionality of the search space, direct optimisation of M leads to no mean- ingful behaviors. We leverage the state directly preceding the hand ini- tiating contact with an object i.e. pre-grasp - to greatly decrease the complexity of learning dexterous be- haviors Dasari et al. (2022). Pre- grasp implicitly incorporates infor- mation pertaining to the shape of the object and its associated affordance with respect to the desired task Jean- herod (1988); Santello et al. (2002). Additionally, pre-grasps can be used in our experiments without additional assumptions as they can be easily mined from MoCap recordings, annotated by human labelers, or even predicted by learned mod- els Taheri al. (2020a). We choose adopt the technique from Dasari et al. (202)) and break
A
Expert
B
Student
MyoDex
π
π
73
TN
Figure 2: Learning Paradigms. A Single Experts policies were obtained by training policies to solve the individual tasks. Then, by means of an expert-student approach, unified Student policy was distilled. B single policy (MyoDex) was obtained by learning all tasks at once.
the learning into two phases. In the first phase the hand learns to reach the pre- grasp pose using free-space planners (no object conditioning required). Next, RL agents are trained to perform either a single target task or family of tasks. We now describe these approaches in detail.
Single task agents. our first setting, we adopt a standard RI. algorithm (see (11e to learn a goal- conditioned policy (a @e,, Φe, the, Vt, Tr, Xobjects pregrasp) (see notation in Sec. 4.1 that can solve a single task. This approach will define a set of expert agents defined as n, with v E / where is the set of tasks (see Tigure[A).
Multi-task agent. Ideally, an agent would be able to solve multiple tasks using goal con- ditioning variable. Thus, we additionally train a single agent to solve all 14 tasks in paral- lel (see Figure BB. This approach proceeds in a similar fashion as the single-task learner, but trajectory rollouts are sampled from the 14 tasks in parallel. All other details of the agent # To (arlor, de, the, s/,, Te, & Xobject; goregrasp) (e.g. hyperparameters, algorithm, etc.) stay the same.
5

---

Under review as a conference paper at ICLR 2023
We encode manipulation behaviors in term of goal-conditioned policies Te(a [t). A standard im- plementation the PPO Schulman et al. (2017) method from Stable-Base Baselines Rafin et al. (2021) was used. Same hyperparameters were used for all tasks (see Appendix Table A.2)
Imitation learning. In addition to MyoDex # we also train a bascline agent using xt expert- student method al. (20T9); Chen et al. 2022T (see A) Individual task specifie policies (T,) were used as experts. We developed a dataset with 1M samples observation-
tuples for each of those policy. Then, we extended the observation vector include the vector Trask repre- senting the object and trajectory. Finally, a neural network similar to Dasari et al Dasari et al. (2022) was trained via supervised learning to learn the association between observations and actions (see hyperparameters in Appendix A.1 to obtain a single policy x² (ar (er Pcs the, V Tt, Trask) capable of multiple task behaviors (see Figure A)
5
EXPERIMENTAL DESIGN
5.1 TASK DESIGN
In this study, we need a large variability of manipulations, hence was important to include 1) objects with different shapes and weights, 2) complexity both terms of translation and rotation of the object. Also, having different movements on the same objects allows to investigate how the different hand phe-shapes and object trajectory affected the solution.
Figure 3: A subset of abject-pregrasp pair from our task-set. See Table A.1 for a complete description A set of 11 Objects and 14 different behaviors (see Table A.T similar to the TDCM dataset pre- sented by Dasari et al Dasari al. (022) were used. The setup (see Figure consisted of table- top environment, an object from the ContactDB dataset Brrahbbatt et al. 2019) and the MyoHand CCggiano et al. 2022). This dataset was implemented in the MuJoCo physies engine Todorov et al.
(2012) To define our tasks, we adopted Dasari et all Dasari et al. (2022) solution where the desired
object trajectory X [60 ,..., 2²' and the hand-object pre-grasp posture poregrasp [ .... Jn.
obijent
where needed. We extracted these information from the GRAB motion capture Taheri et al. (202bb) dataset which contains high quality human-object interactions.
Following the same approach of Dasari et al Dasari et al. (2022). hand postures were computed by matching the human fingertip of the ContactDB dataset and MyoHand by means Inverse Kinematics. In the context of this work, only the hand pre-shaped to grab the object before the initial contact (see Figure3) was considered. For each task, a phe-shaped hand was used to initialize the posture of the hand and the goal was to follow given trajectory of the object. This allow to avoid any physical or geometric information about the object. Each tasks consisted of a pair of one trajectory of object and its associated pre-shaped posture.
5.2 SYNERGY PROBING
To quantify the level of muscle coordination required for accomplishing given task, we calculated muscle synergies by means of Non-Negative Matrix factorization (NNMF) Tresch et al. (2006). After training. we played policies for 5 roll-outs to solve specific tasks and we stored the muscle activations (value between and 1) required. Then, a matrix of muscle activations over time (di- mension 39 muscle total task duration) was fed into non-negative matrix decomposition (sklearn) method. The NNMF method finds two matrices W and H that are respectively the coefficients and the basis vectors which product approximates A, Musele synergies identified by NNMF capture the

---

Under review as a conference paper at ICLR 2023
spatial regularities the muscle activations whose linear combination minimize muscle reconstruc- tion Bizzi & Cheung (2013). This method reveals the amount of variance explained by each of the components. We calculated the Variance Accounted For (VAF) as:
VAF 100 (1
(A W H)
A2
Similarity of synergies between two different tasks was calculated using cosine similarity (CS) such as: CS wi : Wij where [wy, wy] E W are synergy coefficients respectively for the task i and j. We used then a threshold of 0.8 to indicate that synergies were similar Appendix-A.6
poit
0.25
Erer Metric
0.6
0.20
Cal
0.4
0.30
0.05
0.00
0K
6x
6k
10k
12k
0k
2k
8k
10k
12k
Environment Iteration
Environment Iteration
(a) Success Metric
(b) Error Metric
Figure 4: Single task expert terk-peeformance. Metries: (a) Success and (b) Error rate iteration steps. Average and standard deviation (shaded arcas) seeds shown.
6 EXPERIMENTAL RESULTS
First we present how by leveraging pregrasps in standard RL pipeline, it was possible to control the physiological MyoHand perform various tasks involving contact rich manipulation of objects (..c/6.1). Then, we detail deeper investigation illustrating how the muscle coordination evolves and changes as function of task conditions, learning (Sec. 6..3) and support generalization (Sec. 6.4
Success Rate
Error Rate
6.1 MyoDex's S TASK DEXTERITY
wineglass-rrink
First, we wanted explore if can learn a ries of complex dexterous object manipulations required for performing specific tasks (see Sec.
pourl
Ir trainelayy
5.I. A set of agents were trained to solve each task independently: expert solutions. The same pipeline and parameters were used to solve all tasks without any object or task specific tuning (see Table A.2). Qualitatively, all objects in the sample were properly manipulated while mov- ing them to follow the target trajectory (see Fig- ure [ for a sequence quantified by means of 2 metrics (section 3.2) Success Metric (Figure 44) and Error Metric (Figure (4)). In all cases than 80% success and, overall, an error below 0.01. These analysis indicated that MyoDex is able to effectively drive musculoskeletal model of the hand to learn stable object manip- ulations within very tight margins. To the best of our knowledge, this is the first demonstration of such nimble manipulation (see project website for behavior videos) with physiological musculoskeletal hand.
flyl
3
lamane-passs
hammerusel
mag drinees
drink
passe-pass1
snapshots). This was
stamp-stare stamp1
wneglassstoastt
scissors use
achieved greater
ogert
apgregate
student
MyDDax
0.0
0.5
1.0
0.000
0.055
0.000
Figure 5: Baselines: Success and emror rate metries for Expert w,, student x", and MyoDex ## policics.
6.2 MyoDex's MULTI TASK LEARNING
Next, to investigate MyoDex task generalization, we trained one single policy to handle all tasks simultaneously. We approached this problem in two ways. First, used the above policies trained
7

---

Under review as a conference paper at ICLR 2023
to solve each single task expert policies and, by means of expert (teacher)-studen) learn- ing paradigm, we distilled the experts into one policy that solved all tasks at the same time (see S.2.4.2). Second, by means of joint multi-task learning, we obtained one policy MyoDex that solved multiple tasks (see Sec 4.2). In both cases, the policies were able to solve the tasks but with key differences. The student policy, when compared with the MyoDex multi-task policy, showed reduced success rate (see Figure [ median success rate: expert 0.99, student 0.37, MyoDex 0.98) and overall greater eтror (see Figure [5] median error rate: expert 0.00031, student 0.00195, Myo- Dex 0.00046). In particular, the multi-task policy (see Figure (A.2a) reached an error below 0.01 in 9.1k iterations while the expert policies reached that error cumulatively around 34K iterations L.e. 4x slower. On the other hand, expert policies reached success rate of 80% cumulatively in 70k: vs 123k iterations needed for the MyoDex policy i.e. double the time. This indicates that jointly learning multiple tasks greatly facilitates the initial phase of learning, while slowing the learning of detailed aspeets of each specific tasks. This is likely because tasks like airplane- airplane-fiy and cup-pour require task specific and unique wrist rotations to be accomplished.
While the student policy - obtained with imitation learning produced muscle activations similar (Figure A.3) to that of the respective task expert (Figure A.5 but effectiveness quite low in task metrics.
6.3 DOES MyoDex PRODUCE REUSABLE SYNERGIES?
Biological systems simplify the problem to control the redundant and complex mus- cuolokeletal systems by resorting activat- ing particular muscle groups consort, a phe- nomenon known as muscle synergies. Here, we want to analyse if synergies emerge and facili- tate learning.
92.5
90.0
87.5
VAF [%]]
85.0
6 synergies synergies
80.0
For MyoDex where agent has to simultaneously leam multiple manipulations / tasks, common patterns emerges and fewer synergies i.c. 12 (Figure 6). can explain the more than 80% of the variance the data (see Figure [A.4). Fur- thermore, we observe that tasks start sharing more synergies (on average 6, see Figure (A.)). This is expected as each task needs com- bination of shared (task-aspecific) and task- specific synergies. Common patterns activa- tions seems to be related with learning. Indeed, earlier the training more synergies are needed to explain the same amount variance the data. The peak is reached at 12.5k iterations where more than 90% of the variance is explained by 12 synergies.
77.5
10 synergies
12 synergies
75.0
14 synergies
2.5
7.5
12.5
17.5
22.5
27.5
32.5
37.5
Iterations [k]
Figure 6: Muscle Synergies over learning iterations for the joint multi-task policy. Variance of the muscle activations (see Sec. [5.2] explained function of the number synergies at different steps of the learning process.
As expected, the expert policies shared fewer common muscle activations as indicated by fewer synergies shared between tasks (on aver- age 2, see Figure (A and by the overall greater number of synergies needed to explain most of the variance: to explain more than 80% of the variance it is needed use more than 20 syn- ergies (see Figure A.4). Similar results were obtained with the student policy (on average 1 similar synergies between tasks, see Figure A.6).
Figure 7: Zero-shot generalization. MyoDex success- fully initiated manipulations on new objects and trajec- tories. Hand rendering includes skin (see Fig. A.I
6.4 MyoDex OUT OF DOMAIN GENERALIZATION
The joint multi task learning yields a policy that, by only knowing the hand posture (pre-grasp) and without information about the object, shows generalization unseen objects and trajectories.
8

---

Under review as a conference paper at ICLR 2023
Qualitatively in zero-shot out-of-domain task (Figure the MyoDex policy initiates effective ma- nipulations of new objects. Nevertheless, it's is not able to achieve success rates above 0.1. The lack of complete generalization is very likely due to the missing sensory information e.g. from skin ceptors, needed in order to properly hold objects with complex geometries. Indeed, in humans, when sensory information at the finger tips inhibited, proper manipulation cannot be accomplished.
Given this initial indication of generalization capabilities, we want to explore the possibility of using the shared representation provided by the multi-task policy for 1) improving performance on the single tasks i.e. fine-tuning, 2) learning new out-of-domain tasks which were experts policies were not able to learn. We use earlier learned models i.e. 12.5k iterations, which provide the most general representation of coordinated movements as shown by the greater variance explained by fewer muscles synergies (see Figure 6) First, for most of the tasks, fine-tuning the multi-task representation allows faster learning of the in-domain tasks (see Table A.3)) Indeed, it was possible to achieve 80% success using almost less than half of the iterations (2.8k vs. 5k; fine-tuned vs experts) required for experts trained without the same model initialization. Second, we used the multi-task representation on a series of 44 new tasks (see Figure and Table A..3). ). In most of those tasks, the shared representation allowed to quickly learn them. To be noticed, it was not possible to learn expert policies for most of those tasks without this initialization of the model. This indicates
that the representation obtained by jointly learning multiple tasks helps to initialize solution space that avoids local minima.
7 CONCLUSION
In this manuscript we showed how it is possible to con- trol musculoskeletal model of the human hand learn skilled dexterous manipulation of complex objects. We
Expert Success
MyaDox Success
0.0
0.5
1.0
0.0
0.5
1,0
sirplane-llyl
alant clock sco
were able to learn these tasks by imposing a pre-shaping of the hand that reduced the search space. In addition, by means of the joint multi-task learning we showed that it is possible to extract generalizable representations that leverage synergies- muscles that are activated as a unit -wicc allows both faster fine-tur tuning downstream in- domain and out-of-domain tasks. All in all, this study provides strong bases for how physiologically realistic hand manipulations can be obtained by pure exploration via Reinforcement Learning i.e. without the need of mo- tion capture data to imitate specific behaviour.
dup-dres
hommerune use
scisorr uner
ass
mineqqlas toasti
apple lit
torusmodium lif
eepiartt bil
sphere medium
flas Vight lif
duck
winellose
aotch IIt
eatch Itt
sylinder moduum IP
stama-rrt
8 LIMITATIONS AND FUTURE WORK
tadthpastasqqaeee
diplaniit
watch passi
While we have been able to show that we can produce realistic behavior without the need of fitting human data, one important limitation is understanding and matching the results with physiological data. Indeed, our explo- ration method via RL, produced only one of the very high dimensional combination of possible ways that human hand could hypothetically grab and manipulate an object. For example, there are several valid ways to hold cup e.g. by using the thumb and one or multiple fingers. Al- though our investigation points us in the right direction of physiological fcasibility of the result, these findings have yet to be properly validated, Future works will need to consider the ability synthesize new motor behaviors while simultancously providing muscle validation.
cylindersmal gassi
flachi flachight-ons
zonfordbunity inspecti
clophant Inspectl
indersmall:
flashi gli-oon
spheres pressppos
faother booelod Ince
mug IIt
cubesmall
bowl passi
torussmall passi
aggreagate
Figure 8: Summary of all Tasks. Left col- umn tasks solved by single expert policies. Right columns, task fine tuning based on MyoDex. Aggregate success Expert vs My- oDex 0,51 vs 0.93, See also Table A.3
9

---

Under review as a conference paper at ICLR 2023
REFERENCES
Mazen Al Borno, Saurabh Vyas, Krishna V Shenoy, and Scott L. Delp. High-fidelity musculoskeletal modeling reveals that motor planning variability contributes to the speed-accuracy tradeoff. eLife, 9:e57021, December 2020. ISSN 2050-084X. doi: 10.7554/eLife55022.. URL/https: //doi.
orq/10. /0./ 71..0..555eeeffee 57021
Publisher: eLife Sciences Publications, Ltd.
Vittorio La Barbera, Fabio Pardo, Yuval Tassa, Monica Daley, Christopher Richards, Petar Ko- ruushev, and John Hutchinson. OstrichRL: A Musculoskeletal Ostrich Simulation to Study Bio-mechanical Locomotion. In Deep RL Workshop NeurIPS 2021, 2021. URL https:
7/openzeview.nee net/foruuu 2mdd- 7oP1117772zZSOPOO
Emilio Bizzi and Vincent CK Cheung. The neural origin of muscle synergies. Frontiers in Com- putational Neuroscience. 7, 2013. ISSN 1662-5188 doi: 10. 00338///ncom.2213300000
URL
https: www.ffrontersin..... wwt.://gg/attteeeoeeooo . 3389/ fncom. 2013.0003
Samarth Brahmbhatt, Cusuh Ham, Charles C. Kemp, and James Hays. ContactDBB Analyzing and Predicting Grasp Contact via Thermal Imaging. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8701-8711, 2019.
Vittorio Caggiano, Vincent C. K. Cheung, and Emilio Bizzi. An Optogenetic Demonstration of Motor Modularity in the Mammalian Spinal Cord. Scientific Reports, 6(1):35185, October 2016. ISSN 2045-2322, doi: 10.1038/srep35185. URL https:// /angs://dci..o wtt.:/....o..11..///1111 srep35185
Vittorio Caggiano, Huawei Wang, Guillaume Durandau, Massimo Sartori, and Vikash Kumar. MyoSuite- A contact-rich simulation suite for musculoskeletal motor control, May 2022. URL http::/ wwtp://o/g/bbbbbbbbbbbb 13600 Number: arXiv:2205.13600 #xXi:2205513600 [cs].
Tao Chen, Jie Xu, and Pulkit Agrawal. A System for General In-Hand Object Se-OOrentation. arXiv preprint Stt::211100344: 2021.
Vincent C. K. Cheung, Ben M. F. Cheung, Janet H. Zhang, 7oe S. Chan, Sophia C. W. Ha, Ying Chen, and Roy T. H. Cheung. Plasticity of musele synergies through fractionation and merging during development and training of human runners. Nature Communications, 11 (1):4356, August 2020. ISSN 2041-1723. doi: 10.10038/41467-020- 18210-4. URL https: //doi.org// 10. 1038/ 00035/44467-0000 - -20-1000--02010---------- 4
Chao- Chao-Ying
Dustin L. Crouch and He Huang. Musculoskeletal model predicts multi-joint wrist and hand move- ment from limited EMG control signals. In 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp. 1132-1135, 2015. doi: 10. 10.1109/EMBBC220155333888
Sudeep Dasari, Abhinav Gupta, and Vikash Kumar. Learning Dexterous Manipulation from Ex- emplar Object Trajectories and Pre-Grasps. 2022. htl.tttss//axxxi.orrgg// org/ wttpg/gg/ab////s.///////
11221
Andrea d' Avella and Emilio Bizzi. Shared and specific muscle synergies in natural motor behaviors. Proceedings of the National Academy of Sciences, 102(8):3076-305) February 2005. ISSN 0027-8424, 1091-6490. doi: 10. Ful 1nl1/10.10731 000009990000199110
URL htttps:/ppnas.orgg
Andrea d' Avella, Philippe Saltiel, and Emilio Bizzi. Combinations of muscle synergies in the con- struction of a natural motor behavior. Nature neuroscience, 6(3):300-300 -008 2003. Publisher: Nature Publishing Group.
Scott L. Delp, Frank C. Anderson, Allison S. Arnold, Peter Loan, Ayman Habib, Chand T. John, Eran Guendelman, and Darryl G. Thelen. OpenSim: Open-Source Software to Create and Ana- lyze Dynamic Simulations of Movement. IEÉE Transactions on Biomedical Engineering, 54(11): 1950-1950, 2007. doi: 10.1 19900BMM.22007.000 990102..
10

---

Under review as a conference paper at ICLR 2023
Nadia Dominici, Yuri P. Ivanenko, Germana Cappellini, Andrea d'Avell, Vito Mondi, Marika Cicchese, Adele Fabiano, Tiziana Silei, Ambrogio Di Paolo, Carlo Giannini, Richard E. Poppele, and Francesco Lacquaniti. Locomotor Primitives in Newbomn Babies and Their Development. Science, 334(6058)9997-999 2011. doi: 10.1126/science. 1210617. 11261..11////cceeee 1210617 _eprint:
URL Utt.tttts::///wwe http://s//www.cciceneeeee.eee www.ecciennee... https:/wwww.ssttcccooooooll//lld
Lucas Engelhardt, Maximilian Melzner, Linda Havelkova, Pavel Fiala, Patrik Christen, Se- bastian Dendorfer. and UIrich Simon. A new musculoskeletal AnyBodyTM detailed hand model. Computer Methods in Biomechanics and Biomedical Engineering, 24 (7):777-787. 2021. doi: 10. 10.10800102558422220..555 18516667 URL https://doi,
10t:/10.1080110000 00180/112258422 2020 1851367
Publisher: Taylor & Francis .eprint:
http://bb.orr//10000000055555522
Thomas Geijtenheek, Michiel van de Panne, and A. Frank van der Stappen. Flexible muscle-based locomotion for bipedal creatures. ACM Transactions Graphics, 3216):1-11, November
November 2013. acm org/
ISSN 0730-0301, 1557-7368. doi: 10. 11111140833332225 25088999 URL BtR.https:////
doi doi Roi/10.1145/2500 2508399
Leonard Hasenclever, Fabio Pardo, Raia Hadsell, Nicolas Heess, and Josh Merel. CoMic: Comple- mentary Task Learning Mimicry for Reusable Skills. In Proceedings of the 37th International Conference Machine Learning, ICML' 20. JMLR.org, 2020.
Nicolas Heess, Dhruva TB, Srinivasan Sriram, Jay Lemmon, Josh Merel, Greg Wayne, Yuval Tassa, Tom Erez, Ziyu Wang, S. M. Ali Eslami, Martin Riedmiller, and David Silver. Emergence of Locomotion Behaviours in Rich Environments, July 2017. URL
1707.02288 arXiv:1 wew.1777.02288. .02286 [es].
Wenlong Huang, Igor Mordatch, Pieter Abbeel, and Deepak Pathak. Generalization in Dexterous Manipulation via Geometry-Aware Multi-Task Learning. arXiv preprint
2021.
Aleksi Ikkala, Florian Fischer, Markus Klar, Miroslav Bachinski, Arthur Fleig, Andrew Howes, Perttu Hämäläinen, Jörg Müller, Roderick Murray-Smith, and Antti Oulasvirta. Breathing life into biomechanical user models. In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology, UIST '22, New York, NY, USA, 2022. Association doi: 10.1145/35261 15.355569. URL
for Computing Machinery. ISBN 978 1783430333
htt///doi..o httt:///cbb..coo000.1113/3333333 1115533352113335556699 3545689
Divye Jain, Andrew Li, Shivam Singhal, Aravind Rajeswaran, Vikash Kumar, and Emanuel Todorov. Learning Deep Visuomotor Policies for Dexterous Hand Manipulation. In 2019 International
Conference on Robotics and Automation (ICRA), pp. 3636-364) 2019. doi: 10.1109/ 2019.8794033.
Mare Jeannerod. The neural and behavioural organization of goal- movements. Clarendon Press/Oxford University Press., 1988.
Yifeng Jiang, Tom Van Wouwe, Friedl De Groote, and C. Karen Liu. Synthesis of biologically realistic human motion using joint torque actuation. ACM Transactions on Graphics, 38(4):1-12, August 2019. ISSN 0730-0301, 1557-7368.
doi: 10. 1445/33306346633222666 URL https:
3322966
Emanuel Joos, Fabien Péan, and Oreun Goksel. Reinforcement Learning of Musculoskeletal Control
from Functional Simulations, July 2020, URL lttp:://arxi
Number: arXiv:2007. 2.X6690007.06669 arXiv:2007.0666 06669 [cs, cess].
Vikash Kumar, Emanuel Todorov, and Sergey Levine. Optimal control with learned local models: Application to dexterous manipulation. In 2016 IEEE International Conference on Robotics and Automation (ICRA), pp. 578-383, 2016. doi: N0.1109//CRA.2216.77783 156.
Jong Hwa Lee, Deanna S. Asakawa, Jack T. Dennerlein, and Devin L. Jindrich. Finger Muscle Attachments for OpenSim Upper-Extremity Model. PLOS ONE, 10(4):c012171,, April 2015. ISSN 1932-6203, doi: 10. 1/joraallommal.ooo.. 00121/111......111112.2....1. 0121112.. 0
URL https: wwwps//xxpppoopppp/o/ppo//ooossp orc/10.
1371/journal pone. 0121712 001110011000000001011111111
11

---

Under review as a conference paper at ICLR 2023
Seunghwan Lee, Ri Yu, Jungnam Park. Mridul Aanjaneya, Eftychios Sifakis, and Jehee Lee. Dex- terous manipulation and control with volumetric muscles. ACM Transactions on Graphics, 37 (4):1-13, August 2018. ISSN 0730-0301, 1557-7368. doi: 10.11453119751.33201330 URL 3201330
https://dd acm. org do /
Seunghwan Lee, Moonseok Park, Kyoungmin Lee, and Jehee Lee. Scalable muscle-actuated human simulation and control. ACM Transactions on Graphics, 38(4):1-13, August 2019. ISSN 0730-
0301, 1537-7368. doi: 10.1 1..1444533009463322299. URL https: www.aaaam.oooooo http://dd...am.ooro/o////////
1145/33063 3322972
Libin Liu and Jessica Hodgins. Learning baskethall dribbling skills using trajectory optimization and deep reinforcement learning. ACM Transactions Graphics, 37(4):1-14, August 2018. ISSN
0730-0301, 1557-7368. doi: 10. (0.1145/1197551732213333 /
10.1145/3197511 3201315
Daniel C. McFarland, Benjamin I. Binder-Markey, Jennifer A. Nichols, Sarah J. Wohlman, Mar- ije de Bruin, and Wendy M. Murray. Musculoskeletal Model of the Hand and Wrist Ca- pable of Simulating Functional Tasks. bioRxiv, pp. 2021.12.28.474333. January 2021. doi: 10.1101/0021.122284447733 URL Utt.tttpp//biooo.....c/oooototee
12 28 474357 abstract
Josh Merel, Yuval Tassa, Dhruva TB, Sriram Srinivasan, Jay Lemmon, Ziyu Wang, Greg Wayne, and Nicolas Heess. Learning human behaviors from motion capture by adversarial imitation, July 2017. URL http:://aaai http:://aax.v..o/aabb/////// wttp:o////......22.2222. arXiv:1707.02200 [es].
Number: arXiv:1707.02201
Josh Merel, Leonard Hasenclever, Alexandre Galashov, Arun Ahuja, Vu Pham, Greg Wayne, Yee Whye Teh, and Nicolas Heess. Neural probabilistic motor primitives for humanoid control, January 2019.
1111..11111111111111111111111.1. Number: arXiv:1811.11711
arXiv:1811.11771 [cs].
Takahiro Miki, Joonho Lee, Jemin Hwangbo, Lorenz Wellhausen, Vladlen Koltun, and Marco Hutter. Learning robust perceptive locomotion for quadrupedal robots in
the wild. Science Robotics, 7(62)xeabk2822, 2022. doi: 10/siii/sssssboticss.
abk2822, URL https:: httt::::/wwwwssiennne...o. htt.://ww.....aa...oooo/////../a 110p:///...///c/ccc//acaoaa robotiis.... abk2822 eeprint: https://www..ccene..g/////////// http://wwa.ooooooooooooooooooona 126//cciroootccsaa222222
M. Mirakhorlo, N. Van Beek, M. Wesseling, H. Maas, H. E. J. Veeger, and I. Jonkers. A museu- loskeletal model of the hand and wrist: model definition and evaluation. Computer methods in biomechanics and biomedical engineering, 21(9):548-557, July 2018. ISSN 1476-8259 1025- 5842. doi: 10. 10.1000/102558422222
1490952.
Place: England.
Anusha Nagabandi, Kurt Konoglie, Sergey Levine, and Vikash Kumar. Deep Dynamics Models for Learning Dexterous Manipulation, September 2019. htt.htt:p/axxxv.o////aaaa 11652 arXiv:1909.11652 [cs].
Anusha Nagabandi, Kurt Konolige, Sergey Levine, and Vikash Kumar. Deep dynamics models for learning dexterous manipulation. In Conference on Robot Learning, pp. 1101-1112. PMLR,
2020.
OpenAI, 1lge Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, Jonas Schneider, Nikolas Tezak, Jerry Tworek, Peter Welinder, Lilian Weng, Qiming Yuan, Wojciech Zaremba, and Lei Zhang. Solving Rubik's 's Cube with a Robot Hand. arXiv preprint 190.0719, 2019.
Jungnam Park, Schee Min, Phil Sik Chang. Jaedong Lee, Moonseok Park, and Jehee Lec. Gen- crative GaitNet, Jamuary 2022. URL arXiv:2201. 12044 arXiv:2201.1204 [cs].
Number:
Matthias Plappert, Marcin Andrychowicz, Alex Ray, Bob McGrew, Bowen Baker, Glenn Powell, Jonas Schneider, Josh Tobin, Maciek Chociej. Peter Welinder, Vikash Kumar, and Wojeiech Zaremba. Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request eprint: 1802.09464.
for Research, 2018.
12

---

Under review as a conference paper at ICLR 2023
Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Reliable Reinforcement Learning Implementations. Journal of Machine Learning Research, 22(268):1-8 2021. URL http::/jjm= http://tr...oaaaaaoooeaoooooeeee pttppapprs/v22/
Dormann.
20- 1364. html
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning Complex Dexterous Manipulation with Deep Reinforce- ment Learning and Demonstrations. In Proceedings of Robotics: Science and Systems (RSS), 2018.
Elmar Rückert and Andrea d' Aveela.. Learned parametrized dynamic movement primitives with shared synergies for controlling robotic and musculoskeletal systems. Frontiers in computational neuroscience, 7:138, 2013. ISSN 1662-5188. doi: 015399/fncom.2133.001111..
Tsuyoshi Saito, Naomichi Ogihara, Tomohiko Takei, and Kazuhiko Seki. Musculoskeletal Modeling and Inverse Dynamic Analysis of Precision Grip in the Japanese Macaque. Frontiers in Systems Neuroscience, 15, 2021. ISSN 1662-5137. doi: 1.3388//nsss.2222.7745999 URL
https:
w/ww:ffrontii.. iettei.iiiiiiii.. www.://www....t....eeeeeeeeeeeee
3389/ fnsys/nnsssssm 77459
Marco Santello, Martha Flanders, and John F. Soechting. Patterns of Hand Motion during Grasping and the Influence of Sensory Guidance. The Journal of Neuroscience, 22(4):1426 February 2002. doi: 10. 101123/NEUROSCI22204000000 Content 220p:////22//4..........a. abstract
URL /www
Katherine R. Saul, Xiao Hu, Craig M. Goehler, Meghan E. Vidt, Melissa Daly, Anca Velisar, and Wendy Murray. Benchmarking of dynamic simulation predictions in two software platforms using upper limb musculoskeletal model. Computer Methods in Biomechanics and Biomedical Engineering, 18 18(13):1445-11458 2015. doi: 10.1080/1022554222144.0009
URL https:
//doi. 0w111100011152222 201 2014. 916698 10..
Publisher: Taylor & Francis .eprint:
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov, Proximal Pol- icy Optimization Algorithms, August 2017. /tLL:ttt://aatt [cs].
Number: arxiv: 170v::177...0 .06347 arXiv: NVV:1177..00 .0344
Pierre Schumacher, Daniel Häufle, Dieter Büchler, Syn Schmitt, and Georg Martius. Dep-rl: Em- bodied exploration for reinforcement learning in overactuated and musculoskeletal systems, 2022.
URL /https://aaxiaa. wwwpexivvvvor..eeo..eeee hbspg//bb//22262 00484
Ajay Seth, Jennifer L. Hicks, Thomas K. Uchida, Ayman Habib, Christopher L. Dembia, James J. Dunne, Carmichael F. Ong, Matthew S. DeMers, Apoorva Rajagopal, Matthew Millard, Samuel R. Hamner, Edith M. Arnold, Jennifer R. Yong. Shrinidhi K. Lakshmikanth, Michael A. Sherman, Joy P. Ku, and Scott L. Delp. OpenSim: Simulating musculoskeletal dynamics and neuromuscular control to study human and animal movement. PLOS Computational Biology. 14: 1-20, July 2018 doi: fo.137//oouna...cci... 100.0222..ii.00022.. 1006223. URL /tctps://cci.rree
journal pcbi 1006223
Publisher: Public Library of Science.
Anton R. Sobinov and Sliman J. Bensmaia. The neural mechanisms of manual dexterity. Na- ture Reviews Neuroscience, 22(12):441-57, December 2021. ISSN 1471-0048. doi: 10.1038/ -41583-021-00552-7. URL htttts:/t org./i...rr/o 10. 1038/ 021-00528-7
Seungmoon Song, (Lukasz Kidziński, Xue Bin Peng. Carmichael Ong, Jennifer Hicks, Sergey Levine, Christopher G. Atkeson, and Scott L. Delp. Deep reinforcement learn- ing for modeling human locomotion control neuromechanical simulation. bioRxiv, 2020. doi: 10.1101/2020.08.1..222 1..26600.. Publisher: Cold Spring Harbor Laboratory .eprint: /ttp:///ww....///////////////... w...14661....llll. fulll.dff.
Seungmoon Song, Lukasz Kidziński, Xue Bin Peng, Carmichael Ong, Jennifer Hicks, Sergey Levine, Christopher G. Atkeson, and Scott L. Delp. Deep reinforcement learning for modeling human locomotion control in neuromechanical simulation. Journal of Neuro- Engineering and Rehabilitation, 18(1):126, December 2021. ISSN 1743-0003. doi: 10. 11866/122844-2110009999 URL https: / www.mddenttaaaaa. com/ htwp://wwwwwoooooootooaaoe//o/o.
articless es 2rt
13

---

Under review as a conference paper at ICLR 2023
Heiko Stark, Martin S. Fischer, Alexander Hunt, Fletcher Young. Roger Quinn, and Emanuel Andrada. threc-dimensional musculoskelctal model of the dog. Scientific Reports, 11(1):
11335, May 2021. ISSN 2045-2322. doi: (00008//4159-001--000000 7/doi. org/10.0 0000048811550 11t/:/111188000 021 021-90058-0
URL https:
Richard S. Sutton and Andrew G. Barto, Reinforcement Learning: An Introduction. The MIT Press, second edition, 2018. URL httt:://incompleeeiiiee htt.:/eeeeeeeeeeeeeoooooo/eeeeee
htm
Omid Taheri, Nima Ghorbani, Michael J. Black, and Dimitrios Tzionas. GRAB: A Dataset of
Whole-Body Human Grasping of Objects. volume 12349, pp. 581-600. 2020a. doi: 10.1007)
978-3-030-58548-88 URL http::/aaxii..rrrr/////// wttp:///gg..oooo..oooo 11200 arXiv:2008. [cs].
Omid Taheri, Nima Ghorbani, Michael J. Black, and Dimitrios Tzionas. GRAB: Dataset of Whole-Body Human Grasping of Objects. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (eds.), Computer Vision - ECCV 2020, pp. 581-600, Cham, 2020b. Springer International Publishing. ISBN 278-3-030-58548-4
Pramodsingh H. Thakur, Amy J. Bastian, and Steven S. Hsiao. Multidigit Movement Synergies of the Human Hand an Unconstrained Haptic Exploration Task. The Journal of Neuroscience, 28(6):121,, February 2008. doi: 10. www.55/NNEERRSSA111210011......0
URL http:://ww.
ineurosci www.uoosci..rg//eoooo content wog./onteettttonooeeoeeeeoo.e..e
28 6/1.1/..11..stn.....
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012.
Matthew C. Tresch, Vincent C. Cheung, and Andrea d' Avella. Matrix Factorization Algo- rithms for the Identification of Muscle Synergies: Evaluation Simulated and Experimen- tal Data Sets. Journal of Neurophysiology. 95(4):2199-2212, April 2006. ISSN 0022-3077,
1522-1598. doi: 10.1155//n.000222.2000 1052/ wp..i//...... 0ww.002222222.2
URL https: www http://wwwwwwiooggggg./o org/do1/
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is All you Need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Asso- ciates, Inc., 2017. URL /ttp::::: htsttpss//procceddngss.oo neuripps...eeip.pp.ep..........s http::/ccc....aorrrr....//////// pdf
Huawei* Wang. Vittorio* Caggiano, Guillaume Durandau, Kumar Sartori, Massimo, and Vikash. MyoSim: Fast and physiologically realistic MuJoCo models for musculoskeletal and exoskeletal studies. In 2022 IEEE international conference on robotics and automation (ICRA). IEEE, 2022.
Jack M. Wang. Samuel R. Hamner, Scott L. Delp, and Vladlen Koltun. Optimizing locomotion controllers using biologically- actuators and objectives. ACM Transactions on Graphics, 31(4):1-11, August 2012. ISSN 0730-0301, 1557-7368. doi: 10.1145221855222185551.. URL
https://dl acm. http://///aaaacc.oo/o/oo//////// / 2185521
Jungdam Won, Deepak Gopinath, and Jessica Hodgins. Control strategies for physically simulated
characters performing two-player competitive sports. ACM Transactions on Gruphics, 40(4):1-
11, August 2021. ISSN 0730-0301. 1557-7368. doi: 10.1 (00114/555502663559761 URL https:
//dl. acm. wtt.://..0//o///...111.. 11t....1111113456211101 3459761
Yuke Yan, James M. Goodman, Dalton D. Moore, Sara A. Solla, and Sliman Bensmaia. Un- expected complexity of everyday manual behaviors. Nature Communications, 11(1):3564, July 2020. ISSN 2041-1723 doi: 0173//41467-020-1220 1038/s41. 416444467722 020- - 1740--0)
URL
Qi Yang, David Logan, and Simon F. Giszter. Motor primitives determined in early development and are then robustly conserved into adulthood. Proceedings of the Na- fional Academy of Sciences, 100522):102 2019. doi: 18.1073//pas.1124455116
14

---

Under review as a conference paper at ICLR 2023
URL http://wwww..aa..oo///////////// http://wwww.oooooooooo/o/...//ss
-eprint:
Zhiqi Yin, Zeshi Yang. Michiel Van De Panne, and Kangkang Yin. Discovering diverse athletic
jumping strategies. ACM Transactions on Graphics, 40(4):1-17, August 2021. ISSN 0730-0301,
1557-7368. doi: 10.1 14145334500626.35588177 URL https: acm. /rtm.o//////111155/// 3450626. 3459817
15

---

Under review as a conference paper at ICLR 2023
A APPENDIX
Object
Weight [g]
behaviors
airplane
172
fly
airplane
172
pass
alarmclock
542
see
banana
277
pass
cup
300
drink
cup
300
pour
mug
432
drink
stamp
210
stamp
waterbottle
364
shake
wineglass
178
drink
wineglass
178
toast
train
400
play
hammer
210
scissors
47
use
Table A.I: Collection of Object, weight and tasks performed on.
Figure A.1: Hand models. On left, rendering musculoskeletal structure illustrating bone in gray and muscle - in red. On the right a skin like surfaces for soft contacts is overlaid to the musculoskeletal model.
Success Metric Error Metric
Success Metric Error Metric
0.05
0.2
0.0
30k
1.0
0.50
0.5
0.25
0.0
0.00
Environment Iteration
Environment Iteration
(a) Success Metric
(b) Error Metric
Figure A.2: Training (rollout) of the multi-task policy i.e. jointly training on all tasks. Error metrics (top) and Success metries (bottom) illustrate that multi-task training is very efficient reducing smaller emror.. Never- theless, overall success is achieved more slowly than expert solutions. (a) long term training (b) magnification with average- continuous line and standard deviation shaded area over 5 seeds.
16

---

Under review as a conference paper at ICLR 2023
hammer-usel
alarmclockssee1
waterbottle-ssakee
Expert
Muscles
26
Muscle Activation
10
Stud een
Muscles
26
05
00
MyoDex
Muscles 26
60
Samples
Samples
Samples
Figure A.3: Example of muscle activations. task/MyoDex (bottom).
Expert (top) policies, student policies (middle) and multi-
100
90
80
VAF [%)
70
60
50
MyoDex
student
40
combined expert
30
10 15
20
25
30
35
Number of Muscle Synergies
Figure A.4: Muscle Synergies. Number of muscle synergies as function of the explained variance (see Sec. 5.2) of the data shows that, given the same number of synergies, the multi-tast basti-task learning explain more variance of the data. Color coded experts (gray), student (orange) and MyoDex(purple).
A.1 PARAMETERS OF THE NEURAL NETWORK FOR THE EXPERT-STUDENT
For distilling the single expert agents into neural network of the same size of the single agent was used. We adopted a batch size 256, and Adadelta optimizer with a learning rate of 0.25, Discount Factor (7) 0.995, and 10 epochs.
17

---

Under review as a conference paper at ICLR 2023
Environment Iterations
12k
Discount Factor (7)
0.95
GAE-À
0.95
VF Coefficient (cl)
0.5
Entropy Bonus (c2)
0.001
Clip Parameter (c)
0.2
Batch Size
256
Epochs
5
Network Size
pi 256, 128). of [256, 128)
Table A.2: Parameters adopted for the reinforcement learning models.
muscle activity
Correlation with expert
BCR8
ECU
ECRB
ECRL
KCU
KR
R054
EDSS
H0S1
TDP5
10%
R0P3
FDF1
10P2
DP?
D444
DC4
DC
LOM
B1L
P
APL
08
R12
82
RB2
UC
055
IU R81
urug3
Uru
RB4
Ur U84
expart
R85
Ur:
MyoDex
studen
uruss U85
0.0
01
0.
0.0
0.5
10
Figure A.5: Relationship between musele activations. Left - Average Musele Activation of experts (gray), student (orange) and MyoDex(purple). Right conrelation against the expert policies of student (orage) and MyoDex (purple).
18

---

Under review as a conference paper at ICLR 2023
Expert
0.0
2.5
Tasks
7.5
10.0
2
12.5
Student
Num of similar synergies
0.0
2.5
Tasks
5.0
7.5
10.0
12.5
MyoDex
10
2.5
Tasks
5.0
4
7.5
10.0
-2
12.5
0
5
10
Tasks
Figure A.6: Cosine Similarity between 12 synergies extracted from 14 different tasks 37.5k iterations. Top- expert policies. Middle student policy. Bottom MyoDex policy. average number similar synergies for expert, student, MyoDex (mean +/- std over 10 repetitions with different random seeds) was 1.88 + 0.9, 1.45 + 0.9 and 5.48 + 0.17, respectively.
19

---

Under review as a conference paper at ICLR 2023
Task
Multi-fask Success
Iter. reach Success of 0.8
@ 1k Iter.
@ 2k Iter.
@ 3k Iter.
Multi-Task Task
Expert
stamp-stampl
0,981538
0.997949
1.000000
247
3458
banana-passl
0.910000
0.993333
0,998571
247
4446
cup-drinkl
0.991724
0.998161
0.977471
247
3952
mug-drink3
0.978667
0.999467
1,000000
247
3458
alarmclock-sed
0.984444
0.997778
1 1.000000
494
4940
train-playl
0.822278
0.9291 14
0.987848
741
8398
scissors-usel
0.754699
0,945542
0.986988
1235
5434
wineglass-drink2
0.714943
0.924138
0.985287
1235
4446
hammer-usel
0.781429
0,870000
0.972857
1482
3952
wineglass-toast
0.713846
0.796410
0,902051
2223
4199
cup-pourl
0.743429
0.730857
0.830286
2964
4446
waterbottle-shake
0.574595
0.709189
0.743784
3458
5434
airplane-fly 1
0.564675
0.606753
0.631 169
12350
7657
airplane-pass!
0.436322
0.497011
0,509425
12597
6669
mouse-lift
1.000000
1,000000
1. 1,000000
247
apple-lift
1.000000
1,000000
1 ..000000
247
spheresmall-iif
0.986667
1.000000
00000000
247
(orusmedium-lift
0.980571
1. 1,000000
0000000
247
airplane-lift
0.995122
0,000000
000000
247
elephant-liff
1.000000
1. 1.000000
0000000
247
alarmelock -lit
1.000000
1. 1,000000
0000000
247
spheremedium-liff
0.998947
1. 1.000000
I .000000
494
toothpaste-lift
0.971818
0. 0.952727
0.990000
494
flashlight-lns
0.941714 714
0. 0.942857
0.942857
494
stapler-stapee2
0,991529
1.000000
0.999529
494
duck-lift
0.994737
1.000000
1. 1.000000
494
wineglass-lift
0.933000
0.979500
O. 0,980000
494
watch-lift
0.925333
0.955556
0.955556
741
phone-lift
0.960000
0,967742
0.967742
741
watch-lift
0.825778
0.953778
0.955556
988
stapler-staple
0,893809
0.989524
0.996190
988
5187
cylindermedium-lif
0.841111
0.970000
0.972222
988
torussmall-lift
0.690285
0.931428
0.915428
1235
stamp-lift
0.709756
0.980488
0.992195
1235
3211
toruslarge-lift
0.707273
0.965455
0,977273
1235
cup-pass
0.609048
0,995238
1 1.000000
1235
4446
toothpaste-squeeza
0.598421
0.943157
0.977368
1482
stapler-lift
0.650732
0.868293
0.982439
1482
watch-pass
0.492593
0.887407
0.884444
1729
cylindersmall-pass
0.571200
0.826667
0.901333
1976
4693
flashlight-on2
0.168791
0.695385
0.920000
2470
toruslarge-inspectl
0.251852
0.645926
0.817778
2470
snuffordbunny-insppe
0.289157
0.591325
0.921446
2470
6422
elephant-pass
0.506667
0.621235
0.834568
2964
duck-inspectl
0.621299
0.624935
0.820260
2964
cy
sylindersmall-nnspeel
0.420000
0.713333
0.639444
3705
6422
flashlight-on
0.234483
0.541609
0.626207
4446
10127
spheresmall-pass
0.191905
0.351905
0.674286
4446
5928
apple-pase salle-pass1
0.344198
0.481481
0.583210
5187
toothbrush-brush
0.119375
0.353
0.589063
5434
4199
bowl-drink2
0.075714
0.089524
O. 06188810
7657
4693
atheresmall-inspedd
0.235676
0.332432
0.438919
8151
mug-lift
0.326575
0,335342
0,397808
7904
cubesmall-pass
0.024691
0.024691
0.024691
5928
bowl-pass
0.114430
0.153418
0.184810
7163
teapot-pour2
0.137627
0.150508
0.162712
7657
torussmall 1
0.038987
0.037975
0.037975
5928
pyramidsmall-insee inspeett
0.028571
0.033333
0.035238
5187
Table A.3: Fine-tuning of 58 different tasks for MyoDex and expert agents. Expert solutions could reliably reach 0.80 success for the bottom show success only for expert solutions. We indicated with the lack of success in achieving the success threshold. The first columns report the succ20: rate respectively at 1k, 2k and 3k iterations. The 4th and 5th column, document the iterations at which 0.80 success for MyoDex and experts reached.
14 tasks but in many other cases they were not able to. few exceptions

---

Under review as a conference paper at ICLR 2023
1.0
botthbrush-brustee
deepantt-psst
bowl-passl
boohhaste squeezel
0.8
airplane-lift
yylindemedium-tte
Sucess Metric
wineglass toastl
duck-inspect1
wineglass-lft
mug-lift
0.6
cubesmall passsllllpassl
torusmedium-lif
cup-drink1
stamp-lift
stapler-lif
slarmclock-see1
0.4
phone-lift
stanforbbumn-innpect
airplane-lly1
cylindersmal--nspectt
0.2
torussmall-pass
cup passl
pyramidsmall- insppetcti
gylindermmallpasse
spheresmall-iif
duck-lift
0.0
spheresmall-passe
2k
5k
8k
10k
12k
15k
torussmall-if
apple pass1
spheremedium-iff
stapler-staple]
Environment Iteration
banana pass]
pourl
Figure A.7: Finetuning MyoDex on large set tasks. Success rate over itcrations finetuning MyoDex.
Shaded areas indicate standard deviation over 3 seeds.
21

---

Under review as a conference paper at ICLR 2023
cup-pouri alarmclock-seet
airplane-fly1
10
mug-drink3
banana-passi
cup-pourl
0.8
waterbottle-shakel
hammer-usel
wineglass-toastl
0.6
airplane-fly1
Success
scissors-usel
wineglass-drink2
0.4
stamp-stampl
cup-drink1
0.2
airplane-passl
alarmclock-seel
train-playl
0.0
Figure A.8: Fine-tuning based on expert policies. Success rate fine-tuning experts solutions (columns) on 14 different environments. This matrix shows that the combination of pre-grasps and the initialization on pre-trained task is not enough to generalize to new tasks.
22


Reviews:
[{"review_id": "c_rARzHtmaQ", "reviewer": "Reviewer_G4dn", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents s framework to learn manipulation skills with a musculoskeletal hand. Furthermore, this paper demonstrates that a multi-task policy can provide better generalization, e.g., better measure in term of synergies and better generalization to new tasks.", "summary_of_the_review": "This paper provides a good baseline for learning musculoskeletal hand control. The findings that multitask learning help better generalization is interesting. Some questions to addressed during the rebuttal:\n\n1. The comparison in Table I seems unfair, since MyoDex is initialized with some pretrained policy and the expert is trained from scratch, isn't it expected that MyoDex will win? And it is interesting that in the final two cases of the in distribution task, the expert actually wins. A more fair comparison might be to also taking into account the number of iterations used during MyoDex initial training.\n\n2. It is unclear why the expert fails all the out-of-domain tasks. Is it because these tasks are fundamentally harder? How are the initial set of task to train the expert/MyoDex selected? It will be nice to provide some intuition. Does decreasing/increasing the set of initial tasks affect the performance of the multitask learned policy? i.e., how do we know what tasks to train for initially?\n\n3. Why does the student policies fail to match the performance of the expert policies? Have the authors tried DAGGER? It is unclear to me whether it is due to fundamental challenge to imitate a large number of experts or if it is a purely implementation issue. More evidence is needed to support the claim. e.g., Are the student able to just imitate one expert effectively? What about two? What is the minimum number of experts that will cause the student to fail to learn effectively?", "strength_and_weaknesses": "Pro:\n\n1. A simple system that can learn a wide range of manipulation tasks for a musculoskeletal hand. This can serve as baseline for future research in this direction.\n\n2. Demonstration of the advantage of learning multiple tasks at the same time, using metrics like synergies and task generalization.\n\nCon:\n\n1. There are some problems about the comparison in Table I, see comment below.\n\n2. The failure of the student policy to learn effectively from expert policies need more analysis.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "T_j1EOAlOz", "reviewer": "Reviewer_sdCp", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a benchmark that touches on an interesting and important topic, musculoskeletal dexterous hand manipulation. The authors further demonstrate that pre-training the policy on 14 manipulation tasks allows for easier fine-tuning in both in-domain and out-of-domain tasks.", "summary_of_the_review": "Overall, the paper is of good quality. However, the main contribution of this work is to demonstrate the effectiveness of pretraining the musculoskeletal dexterous agent, which I think is not enough for acceptance. I would consider increasing my rating if the authors could address my concerns.", "strength_and_weaknesses": "\nStrength:\n1.\tThe proposed contact-rich manipulation tasks for the musculoskeletal dexterous agent are novel.\n2.\tMyoDex leverages joint multi-task learning to recover reusable representations (synergies) that allow for easier fine-tuning in both in-domain and out-of-domain tasks (including one/few-shot learning).\n\nWeaknesses\n1.\tAll the tasks in this paper seem based on grasping. The difficulties of these tasks are too homogeneous. Perhaps refer to these projects to design more tasks, such as [1] and [2].\n2.\tThe State Space contains the pose and velocity of the object without any vision-based input, thus resulting in missing geometric features of the object. This may limit the out-of-domain generalization ability. Recently, lots of dexterous hand works contain visual input with a strong generalization ability, such as [3] and [4].\n3.\tThere seem to be no ablation experiments on synergies.\n4.\tThe experiment seems to be done with only one seed, and the authors should consider doing several more seeds in order to better demonstrate the robustness of the method.\n5.\tThe previous work [5], which also presents a dexterous manipulation benchmark with musculoskeletal hands, detracts from the novelty of this work.\n\n[1] https://bi-dexhands.ai/\n[2] https://openreview.net/pdf?id=k2Ml8FGtJZp\n[3] https://openreview.net/pdf?id=k2Ml8FGtJZp\n[4] https://openreview.net/pdf?id=tJE1Yyi8fUX\n[5] https://arxiv.org/pdf/2205.13600.pdf\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "DVnZY_TSSOz", "reviewer": "Reviewer_RfAu", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a reinforcement learning problem on MyoHand aiming at learning generalizable representation for manipulating different objects to finish different tasks. The authors designed an RL problem based on the previous MyoSim platform and trained agents to solve 14 tasks together. Grasp pose detection is applied to simplify the RL training. Experiments show that training together outperforms the baseline of first training experts separately and then distilling them into one policy through behavior cloning. Authors show that MyoDex, the agent trained to solve all tasks, has a certain level of zero-shot generalization ability and can be fine-tuned on single tasks. ", "summary_of_the_review": "The paper presents a  study of training multi-task RL agents on MyoHand. I appreciate the efforts in designing tasks and running RL experiments. Still, the task of grasping and moving objects with a dexterous hand could be more novel, while the authors only use existing RL methods to solve this task without further improvements. Such contributions do not reach the acceptance bar. I suggest authors figure out challenges in dexterous hand manipulation and design or evaluate algorithms, showing the value of studying such problems.", "strength_and_weaknesses": "Strength:\n\n- the paper constructs an environment for studying dexterous physiological manipulation problems.\n- The paper illustrates the potential of reinforcement learning in multi-task learning.\n\nWeakness:\n\n- Though the paper makes an effective empirical study of RL for dexterous manipulation, I wonder if this is a significant contribution given previous progress in learning dexterous manipulation with RL (Chen et al. (2021)). I do not see significant differences between a MyoHand and a ShadowHand for an RL agent.\n- For the baseline evaluation, simply running the supervised learning is not enough; it is better to use imitation learning, at least DAPG, to have some RL components to fine-tune the student agent.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "67QDKzjhQyx", "reviewer": "Reviewer_Sq1z", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents the MyoDex framework, which applies goal conditioned pre-grasping and PPO (introduced in prior work) to grasping and manipulation tasks with a simulated human hand. The paper considers multi-task and transfer-learning scenarios showing the superiority of the method over baselines.", "summary_of_the_review": "The paper explores the application of a method to grasp and manipulate objects with a biologically accurate hand model. The authors provide empirical evidence on the effectiveness of this method and additionally aim to understand the use of the method for multi-task-learning, transfer-learning, and qualitatively probe the resulting behaviors for synergies among muscle groups.\n\nHowever, I currently recommend weak rejection of the manuscript. I am most concerned about the clarity of the manuscript on key methodological and experimental details (*W1-3, 6, 9, 10, 12-15*), the fairness of the multi-task baseline (*W11*), and the lack of attribution to the Dasari et al., 2022 paper on key aspects of the methodological and task (*W4, 7, 15*). Additionally, I am not convinced that the task is non-trivial (*W7*).\n\nI am willing to revisit my evaluation during the rebuttal/discussion period.", "strength_and_weaknesses": "**Strengths**\n\n*S1.* The problem of controlling small, dexterous muscle groups seems understudied and this paper takes a step in understanding how techniques might be applied to address this problem.\n\n*S2.* Experiments are creative and well intentioned. For example, probing for synergies between muscle groups is an interesting idea, with possible implications for transfer-learning on hand models.\n\n*S3.* Extension of the MyoHand simulation to include shoulder movement.\n\n*S4.* Empirical novelty demonstrating the application of Pre-grasp + PPO to the manipulation tasks considered in the evaluated benchmark.\n\n**Weaknesses**\n\n*W1.* The task formulation is not entirely clear (Sec. 3.2). Currently it just presents metrics, which is different than the task itself. Also it is not clear why there are both \"Task Formulation\" and \"Problem Formulation\" (Sec. 4.1) sections.\n\n*W2.* What is $\\phi$ in Sec. 4.1? What is the dimensionality? How does it relate to the hand model? What is meant by a \"joint angle\" in this case?\n\n*W3.* The action space is not clear in the manuscript. What is an example of an action? What does an action represent? Some continuous contractions of muscles? What is an example of a state?\n\n*W4.* The reward function used this this work looks to be the same as that in Dasari et al., 2022. The authors must cite this to make this attribution clear.\n\n*W5.* The method assumes a lot of information about the scene (1) GT state (2) densely annotated ground truth trajectories for their dense reward (3) does not consider complications associated with vision. I consider these to be limitations of the work in its current state, but still feel the exploration is valid.\n\n*W6.* The training dataset details are not clear in the manuscript. For example for each task how many GT training trajectories are sampled? A single trajectory? Multiple trajectories? Given a single task, are the target objects always spawned in the same pose? The manuscript discusses pose variations between different tasks but it is unclear if there is any randomization when training one task.\n\n*W7.* The pre-grasp, introduced in Dasari et al., 2022. seems to greatly simplify the problem and search space. However, I consider pre-grasp as privileged information and wonder if the pre-grasp makes grasping the object trivial. Given the pre-grasp what is the challenge of closing the MyoHand?\n\n*W8.* Only one seed seems to be used in evaluation calling into question the reproducibility and stability of the results.\n\n*W9.* In the explanation of VAF metric (Sec. 5.3) it is not clear on what matrix NNMF is done on. What is the matrix A exactly and what is its dimensions? Is it the output of your network? Perhaps including an example here would add clarity.\n\n*W10.* In Sec. 5.3, the use of cosine similarity to count the number of synergies is not completely convincing to me. My understanding is that H is a matrix of basis vectors and and W of coefficients for the basis vectors for a task. Now consider a permutation of the basis H, which leads to a permutation of W. This may change the cosine similarity when comparing two synergies.\n\n*W11.* It appears that $\\pi^*$ does not have access to the pre-grasp, while MyoDex does. I consider a policy \"$\\pi^*$ + pre-grasp\" to be a more fair point of comparison as it has access to the same amount of privileged information.\n\n*W12.* No network architecture details are presented. It is unclear how big (or small) the networks are. I suggest adding more details about the network to the main paper or appx.\n\n*W13.* It is hard to contextualize what a synergy is in Fig. 6 and what exactly 12 synergies means. Does this map to muscles firing together? Are there interpretable examples of these synergies that can be observed in the resulting behavior? Is it possible to have a human or scripted policy baseline here to contextualize what 12 synergies means?\n\n*W14.* The zero-shot protocol is not clear. MyoDex is conditioned on the goal object. What is the protocol for conditioning on an unseen object? How is the pre-grasp provided? I recommend adding details to make this section more clear.\n\n*W15.* How were multi-task and transfer task splits chosen? Are the tasks taken from the TCDM benchmark introduced in Dasari et al., 2022? If so, I strongly recommend citing the benchmark and making the benchmark setting details more transparent.\n\n*W16.* There are many holes in Tab. 1. Is it the case that these policies did not reach 0.8 success for these tasks? If so, I recommend making this more clear in the caption, so it is clear that these evaluation were actually run. Also I recommend reporting the success rate curves in the appx. so that future work can try to improve on this task subset.\n\n**Minor**\n\n*M1.* Consider not splitting 4.2 and 5.2 into different sections\n\n*M2.* 6.2 is not necessarily about task generalization, but about multi-task experiments. I feel these are different things and the title should be adjusted accordingly.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject