In-Flight Attitude Control of a Quadruped using Deep Reinforcement Learning

Authors: tarek  el-agroudi
finn  gross maurer
jørgen anker olsen
kostas  alexis
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-06 22:42:18

Abstract: We present the development and real world demonstration of an in-flight attitude control law for a small low-cost quadruped with a five-bar-linkage leg design using only its legs as reaction masses. The control law is trained using deep reinforcement learning (DRL) and specifically through Proximal Policy Optimization (PPO) in the NVIDIA Omniverse Isaac Sim simulator with a GPU-accelerated DRL pipeline. To demonstrate the policy, a small quadruped is designed, constructed, and evaluated both on a rotating pole test setup and in free fall. During a free fall of 0.7 seconds, the quadruped follows commanded attitude steps of 45 degrees in all principal axes, and achieves an average base angular velocity of 110 degrees per second during large attitude reference steps.

Main paper: 
 introduction: 
 The exploration of extraterrestrial environments with robots is a daunting endeavor. On one hand, the impossibility of fast human intervention requires unprecedented robustness and autonomy while, on the other, the complexity and uncertainty of extraterrestrial surfaces requires dexterous locomotion. One particularly complex class of environments are martian and lunar lava tubes, the exploration of which might provide valuable geological insights and the potential for early human bases [1] . Wheeled rovers, of which several have been deployed on Mars and on the Moon [2] , are inadequate to explore these terrains due to the need for significant vertical motion for lava tube exploration. Helicopters, such as Ingenuity [3] , have also been demonstrated on Mars but have limited payloadcarrying capacity. With the recent advances in quadrupedal robots (quadrupeds) such as the Boston Dynamics Spot [4] , the ANYbotics ANYmal [5] and the Unitree Go 1 [6] , a proposed solution for the robotic exploration of lava tubes has been to use specially designed quadrupeds that can exploit the lower gravity experienced on Mars or on the Moon by jumping and experiencing long phases of "flight" to seamlessly traverse complex terrain [7] .In order to cancel out any accumulated angular velocity during a prolonged flight-phase and to enable landing on inclined surfaces, an in-flight attitude 1 control law is a prerequisite for such a robotic platform. A proposed solution for the actuation of such capabilities is for the quadruped to use its own legs as reaction masses in maneuvers that resemble the self-righting motions of a falling cat. This has certain advantages compared to other orientation approaches, such as the reaction wheels exploited in satellites [8] . Primarily, the considered approach does not require extra hardware and thus keeps the overall system weight smaller, which has significant mission payload implications.With the recent advances in simulation technologies and parallelization hardware, many complex control tasks have been solved with DRL-based approaches. DRL allows for the exploitation of the equivalent of many hours of real-time experience in just a few minutes [9] , while also being able to generate lightweight control policies implementable on resource-constrained hardware. This motivates the use of DRL for synthesizing an in-flight attitude control law for a jumping quadruped and the implementation of such a policy on a quadruped in real life.Driven by the above, the main contribution of this work is the development of a DRL-based attitudecontrol policy for a quadruped with a five-bar-linkage leg design and lateral leg movement capabilities, and the demonstration of this policy on a custom-designed prototype in all degrees of freedom (DOFs). The proposed test setups include maneuvering on a rotating pole and free fall experiments (a subset of the results is depicted in Figure 1 ) with the system following commanded attitude setpoints. The considered testing setups are inexpensive and easy to replicate. Furthermore, the developed quadruped design features low-cost hardware with of-the-shelf actuators and electronics, demonstrating the potential for robust DRL without exact knowledge of factors like actuator dynamics. Key insights that transfer to related robotic hardware are the treatment of low-cost RC servo motors through a reference model and motor model, and the explicit simulation of the closed kinematic chain of the robot's legs. We further show how the same reward function can be used to learn a policy for both fast and slow dynamics. To the authors' best knowledge, this work is the first real-world demonstration of an in-flight attitude-control feedback policy for a quadruped in 3D. This paper is structured as follows: Related work is outlined in Section 2, while Section 3 formally defines the control problem to be solved using DRL. An overview of the robotic research platform and experimental setups is provided in Section 4, and the proposed approach for training and deploying the DRL-based policy is outlined in Section 5. Section 6 evaluates the experimental results, and Section 7 outlines central conclusions, limitations and future work. 
related work: 
 This work builds upon a range of previous contributions. The work in Rudin et al. [10] presented SpaceBok, a quadruped specifically designed for the exploration of lunar surfaces, and demonstrated a DRL-based policy in 2D on a test bed designed for micro-gravity experiments [11] . Kurtz et al. [12] demonstrates self-righting on the quadruped Mini Cheetah by restricting attitude errors to pitch and utilizing a combination of trajectory optimization and supervised learning techniques. The work on the tripedal robot SpaceHopper [13] has trained and demonstrated a DRL-based policy for attitude control in 3D on a parabolic flight and a specially designed gimball setup. Furthermore, Olsen and Alexis [14] presented OLYMPUS, a quadruped with a five-bar-linkage leg design tailored for extraterrestrial exploration. The modeling and control of closed kinematic chains (CKC), such as the N-bar-linkage legs of OLYMPUS and SpaceBok, has been extensively researched [15, 16] , along with free-floating dynamical systems with manipulator structures [17] . Kamidi et al. [15] explicitly simulates the CKC and SpaceBok establishes an open-loop equivalent. For the synthesis of the policy using DRL, this work builds heavily upon [9] , which demonstrated the learning of highperformance locomotion policies for quadrupeds in mere minutes by exploiting GPU parallelization. The main technique used for sim2real transfer of the learned policy for SpaceBok and SpaceHopper is domain randomization, which has been evaluated against a range of sim2real methods in [18] . 
problem formulation: 
 With reference to Figure 2 , the considered control problem can be formally defined as follows: Given the orientation quaternion2 q w b of the quadruped's body frame B in an inertial world frame W and a target orientation T expressed in W through q w t , control the quadruped towards q w b = q w t . The latter is equivalent to controlling the angle of the axis-angle representation of the quaternion error q t b = (q w t ) * ⊗ q w b to zero, denoted ∠q t b → 0, which motivates the reward function presented in Section 5. The inputs made available to the control law are the the quaternion error q t b , its angular velocity ω t b and the positions and velocities of the 12 actuators, θ m and θm , forming the observation vector o = [q t b , ω t b , θ m , θm ] T ∈ R 31 . Actuation is restricted to the 12 actuators of the quadruped, and the output of the control law is thus a vector of motor position references θ r ∈ R 12 sent to the motors, which in turn apply a torque τ m ∈ R 12 to the actuated joints. We note that the target application of canceling angular momentum from a jump is a velocity tracking problem rather than a regulation problem. However, if position tracking is demonstrated, velocity tracking can be achieved by the introduction of a moving position target.  
system overview: 
 The proposed research platform, called Eurepus, consists of four legs with a five-bar-linkage design, each consisting of a lateral motor and two transversal motors. The base is constructed with interior rails to allow the quadruped to be mounted on a forked rotating pole for single DOF attitude-control tests. The quadruped has a weight of 2.5 kg and the dimensions presented in Figure 3 . Key design considerations include aluminum joints and carbon-foam links to ensure low weight and robustness. 80 gram aluminum paw masses equip the quadruped with the necessary inertia for using the legs as reaction masses, and constitute about an eighth of the system weight. The IB53BHP servo from AGF-RC is used for all three motors on a leg. It is a fast (180 RPM), light-weight (70 g), hightorque (2Nm) and low-cost servo with potentiometer position feedback. The motors are driven by a PWM driver and powered by a 2 cell LiPo battery. Position feedback is converted from analogue potentiometer readings via 12 ADS1115 ADCs. The Khadas Vim 3 (A311D) SBC is the central control unit of the quadruped. To test the system at lower joint velocities and over extended periods of time, a stiff forked aluminum pole is mounted to an open ball bearing, such that the quadruped can be slid onto it in all principal axes, illustrated in Figure 4 . The pole fixes the center of rotation and restricts attitude errors to 2D. We therefore conduct 3D free fall experiments, shown in Figure 5 , by raising the quadruped into a mount designed to lock it into a known initial configuration. A magnetic switch is installed to remotely release the quadruped into a foam pit. The resulting configuration allows for a free fall duration of 0.7 seconds. Attitude feedback is provided through a motion capture (MoCap) system for the free fall tests and through an encoder for the pole setup. Encoder Reference plateWe present in Table 1 the training times for the two developed DRL-based policies. The GPU used is the NVIDIA GeForce RTX 3090, and the GPU memory usage was 4GB out of the available 24 GB RAM. The training time for the low-motor-velocity policy for the pole experiments is slower due to the inherently slower dynamics. Furthermore, training times are generally slower than similar related work using GPU-parallelisation2 . This is mainly attributed to the need for extremely small simulation time steps due to stability issues of the Closed Kinematic Chain(CKC) simulation. Figure 4 : 45-degree step responses where the policy is trained with only the three first rewards in Table 1 in the manuscript.We train and evaluate the performance of a policy trained with a reduced reward function comprising only the first three reward terms,R total = C 1 R 1 + C 2 R 2 + C 3 R 3 .The five terms omitted correspond to the negative penalty terms of the original reward function. The scaling factors are not altered.The results, in simulation, of 45-degree setpoint following in roll, pitch and yaw are provided in Figure 4 . The results show that successful attitude control policies can be trained with the reduced reward function, which demonstrates that the main task objective is sufficiently represented by only three terms. An important observation is the oscillatory response when compared to the responses presented in the work. This is a natural consequence of the removal of the penalty terms which reduce actuator wear and tear (reward terms R 5 , R 6 and R 7 ) and directly punish oscillatory motor action (reward term R 8 ). 
proposed approach: 
 We create a simulated environment of the Eurepus quadruped in NVIDIA Omniverse Isaac Sim and simulate 4096 environments in parallel to collect action-state-reward trajectories. The modified PPO implementation from [9] is leveraged to learn a control policy from the collected trajectories during simulation. Two policies are learned, for slow and fast motor velocities respectively. Implementation wise, the work builds on the OmniIsaacGymEnvs repository [19] for GPU-accelerated DRL.The reward function is defined as a weighted sum of eight reward terms, R total = 8 i=1 C i • R i , where the terms R i and scaling factors C i are given in Table 1 . The terms are split into three mostly positive reward terms and five negative penalty terms. R 1 rewards closeness to the target orientation. To encourage small orientation errors, it is constructed as an exponential kernel with temperature T orient around the absolute value of the error angle ∠q t b , where the latter was defined in Section 3. R 2 provides an extra discrete reward whenever the quadruped is within a threshold ϵ orient radians of the target, where ϵ orient > 0 is tuned according to the required accuracy. Finally R 3 acts as a sparse reward provided only when the quadruped terminates due to a success criteria of having been within ϵ orient degrees of the target for more than half a second consecutively. Its design renders it negative when the orientation error is larger than 90 degrees and positive when it is less. With reference to Section 5.3, R 4 punishes the squared norm of the difference between interpolated motor targets, θ interpol , and the motor targets after being clamped to prevent self-collisions, θ r . Similarly, R 5 punishes the excess torque that is clipped away during saturation at maximum torques τ max , where K p and K d are the parameters of the motor's interior PD controller. Both terms punish the setting of targets that are clipped or clamped in subsequent steps, and are inspired by [10] . The final reward terms R 6 , R 7 and R 8 are regularization penalties and serve the purpose of enforcing smooth policies that minimize actuator effort, to reduce motor wear and tear. R 6 punishes the squared norm of the motor velocity vector and R 7 punishes the squared norm of the difference between consecutive applied torques, while both terms serve the purpose of minimizing wear and tear. Finally, R 8 is a discrete penalty given anytime the motors change direction, to enforce smooth maneuvers, where ϵ noise > 0 is a threshold included to exclude numerical noise.i Reward/Penalty R i C i 1 Orientation Reward exp (-|∠(q t b )|/T orient ) 2 2 Inside Threshold Reward 1 |∠(q t b )| < ϵ orient 0 otherwise. 1 3 Terminal Reward π 2 -|∠(q t b )| 100 4 Self-Collision Clamping Penalty -||θ interpol -θ r || 2 1 × 10 -6 5 Torque Saturation Penalty -||K p (θ r -θ m ) -K d θm -τ max || 2 1 × 10 -2 6 Velocity Penalty -|| θm || 2 2 × 10 -5 7 Torque Regularization Penalty -||τ m (t) -τ m (t -1)|| 2 3.5 × 10 -2 8 Change Direction Penalty -1 θm (t) • θm (t -1) < ϵ noise 0 otherwise. 2 × 10 -3A few important practical considerations were necessary for successful training. Since the loss function of PPO considers only the total reward R total , the reward weights C i are chosen such that the reward terms are comparable in order of magnitude. Furthermore, implementing the orientation rewards as positive rewards instead of penalties was essential to avoid the local optimum of selftermination. The most important reward is the orientation reward R 1 , and it alone suffices to achieve a working policy with PPO, but increased performance is experienced with the addition of the other terms. While self-collisions of a leg are prevented by explicit clamping, as will be outlined in Section 5.3, no explicit collision filter is implemented to avoid collisions between separate legs. Instead, the quadruped is reset upon the occurrence of collisions such that the accumulated reward after the trajectory is smaller (total reward is in general positive), thus implicitly incentivising collisionfree maneuvers. Another approach that was explored was to explicitly punish collisions through negative penalties, but this led to conservative policies. The developed policy can be combined with an explicit collision filter during deployment on hardware. Finally, when training a policy for lower motor velocities, we use the exact same reward function as for the unconstrained velocity problem. We find that reducing the inference frequency and extending the time horizon parameter of PPO with the same ratio as the ratio of maximum motor velocities results in successful learning. This likely stems from the fact that a trajectory will contain similar degrees of exploration.A series of steps were necessary to achieve a stable and realistic simulation in Isaac Sim. Isaac Sim simulates systems of rigid bodies through articulations, a reduced-coordinate representation which is fully determined by a root body and joint angles between all bodies. This mechanism does not support loops, which required separately modeling the ankle joint as a constraint. In Isaac Sim, this is achieved by excluding the joint from articulation. This constraint and the significant mass differences between the links of the quadruped render the dynamics stiff. To ensure stability of the resulting simulation model, the most important consideration was the need for a low simulation time step ∆t = 1/480s, which is eight times smaller than the simulation step in [9] .A prerequisite for the successful sim2real transfer of the policy was an actuator model. The actuators are modeled by treating the internal motor dynamics as a static torque-speed mapping, F ts ( θ), where θ is the motor joint angle and F ts is a linear curve provided by the manufacturer. This simplification is based on the intuition that electrical dynamics are much faster than the mechanical dynamics θ. The parameters K p and K d of the actuator's internal PD controller were unknown and hence identified from step response data. The resulting simplified motor model is given by: EQUATIONEQUATIONFinally, the interpolated joint targets θ interpol are clamped to avoid self collisions of a leg. This is implemented as a condition based on the sum θ sum . = θ i + θ o of the inner and outer transversal motors angles θ i and θ o , and an acceptable range for this sum [θ min sum , θ max sum ], according to the following conditions, where -= and += are the decrement and increment operators:Clamping Operationθ sum > threshold θ i -= (θ max sum -θ sum ) 2 θ o -= (θ max sum -θ sum ) 2Clamping Operationθ sum < threshold θ i += (θ min sum -θ sum ) 2 θ o += (θ min sum -θ sum )2 If a policy is trained or deployed at reduced motor velocities, a smooth commanded target sequence θ c is achieved by sending the clamped and interpolated motor targets θ r through the saturated second order reference model given by (3) . ν c denotes the commanded velocity, and the absolute damping factor ζ and natural frequency ω n are filter tuning parameters. A second order reference model is advantageous as it generates smooth velocities in addition to smooth position signals.EQUATION 
evaluation studies: 
 To evaluate the performance of the learend policy on Eurepus, tests were conducted both per DOF on a rotating pole and during free fall 3D maneuvering. A comparison with other control approaches and specifically with Model Predictive Control is presented in the supplementary materials.We evaluate a 3D policy trained with joint velocities below 300°/s through (3) against a time varying reference on the rotating pole setup.Step responses for roll, pitch and yaw are given in Figure 7 .Figure 8 presents an ablation study of the paw masses. The policy successfully follows the reference for all DOFs, validating the test setup. The spike in yaw indicates that the policy has learned to sometimes take the longest path to a reference, and the degradation in the pitch DOF after 60 seconds originates from a leg collision and the corresponding accumulated angular velocity.A key finding is that the 3D policy trained in a pole-less environment and with no variations of paw masses is robust to the effects of the pole as well as variations of the paw mass. This indicates that the learned feedback policy has not overfitted to the system dynamics and justifies the simple feed-forward network structure as opposed to an adaptive approach through for instance recurrence in the policy. A potential drawback of the policy is excessive control effort and oscillations at the reference. Possible reasons for this are noise in joint measurements, which directly transmits to the actuators through the linear interpolation ( 2), inadequate motor model ( 1) leading to excessive actuation or slower dynamics on the real system, and potential collisions of the legs with the pole. Figure 8 : 3D policy roll response for different paw masses. Policy trained at 120 g.We demonstrate attitude control of the quadruped through a range of free-fall experiments from a height of about 3.5 meters and landing into a foam pit. Figure 9 displays the simulated and real response of the error quaternion decomposed into Euler angles (ZYX convention) for 45 degrees setpoint in roll, pitch and yaw. Each experiment is repeated thrice and the plot indicates the mean and variance of the response to demonstrate repeatability. The target orientation is reached in less than 0.4 seconds for yaw and pitch, and 0.6 seconds for roll. Next, Figure 10a demonstrates nonprincipal axis attitude control by applying a simultaneous 30 degrees reference in roll, pitch and yaw. Finally, we show that the policy is capable of controlling 80 degrees of orientation during the free fall time of 0.7 seconds by applying large reference steps in pitch and yaw, as seen in Figures 10b and 10c . Extrapolating the results yields an estimated acheivable average attitude velocity of 110 degrees per second. Frames from a recording of selected experiments are provided in Figure 1 .Compared with the simulated trajectories, the real responses demonstrate robust sim2real transfer, but are simultaneously characterized by a slower transient and oscillations at the reference orientation. The main contributing factors to this are a) a time lag before a measurement is received from the MoCap system as well as b) a higher time constant in the real motor control loop than in the simulation, experimentally identified to be 50% higher for the real actuators. The former is estimated to be ≈ 0.03 -0.05 seconds, and such time lags can cause oscillations in feedback control systems dealing with fast dynamics as in Eurepus. The higher time constant in the actuators likely stems from the static approximation of the internal motor dynamics employed in the simplified motor model (1). The proposed robotic platform is designed to be modular, and all parts can be 3D printed. CAD and STL files will be open-sourced, such that the robot can be replicated easily.As is illustrated in Figure 6 , the base of Eurepus consists of three main plates. The four legs slide onto the corners of a central base plate through the four lateral motor mounts. The electronics are screwed to the upper and lower electronics plates and fit on top of the lateral motor mounts. The space between the base plate and the electronics plates is occupied by a configuration of rails. These, along with the parallel holes in the centre of each plate, are designed to allow a forked pole to be slid through the quadruped in all three principal axes. To protect the electronics on the upper plate and the ADCs mounted on top of the lateral motor mounts, a base cage and ADC cage are fitted over these. The entire configuration is fastened through three screws in each corner of the plates.Each leg consists of a lateral motor and two transversal motors. The transversal motors fit snugly into a transversal motor mount that is mounted directly to the rotational axis of the lateral motor.The two thighs are mounted to the transversal motors with spacer plates to provide additional room when the legs traverse over the base. The leg itself is held together by parts designed to clamp to a thigh or shank on one end and connect in a ball-bearing joint on the other end. The paw clamp at the bottom of the leg contains additional fastening mechanisms for the paw mass.Finally, we present an annotated picture of the electronics on Eurepus in Figure 5 to highlight how they fit in the mechanical design. v 
conclusions, limitations and future work: 
 We demonstrate fast attitude control on a custom-built low-cost quadruped with a five-bar-linkage leg design on a rotating pole test setup at lower speeds and in free fall. Tuning the horizon and inference frequency of PPO to replicate another problem with faster dynamics proved to be an effective way to relearn a problem for slower dynamics. Limiting assumptions of the work are a disturbance-free environment and the set-point-regulation problem formulation. Future work should consider more realistic extraterrestrial conditions and also address the velocity-tracking problem. Key limitations of the proposed approach are the simplified motor model and the possibility of leg collisions. This should be addressed through better actuator models and an explicit safety collision filter. Before operational system deployment, considerations should be made as to whether the learned attitude-control policy efficiently prepares the quadruped for landing.Finally, tuning of the MPC is no trivial task, due to the cascade structure and number of parameters. While tuning is also non-trivial for the proposed DRL-based method, we find the required tuning efforts comparable, such that ease of tuning is not an advantage of the MPC-based approach. To assess the impact of the motor model on the performance of the policy, we repeat a 45-degree pitch experiment with four different configurations of the K p gain of the motor controller in simulation (Figure 2 ). K p is the nominal proportional controller gain and is the one employed during training. The ablation demonstrates degraded performance from changes to the gain indicating the importance of an accurate motor model. We further note the similarity between the degraded performance with the real experimental results of Figure 9d -9f in the paper, which may highlight the merit of further effort into motor modelling, through for instance an actuator network.3 Angular Velocity Tracking To assess the suitability of the developed policy to velocity tracking problems, we evaluate the velocity tracking performance to ramp references with a rate of 30 degrees per second in all principal axes (Figure 3 ). The velocity evaluated is motivated by the estimates of the angular velocity following a jump derived in related work2 . The oscillation observed around the target is a result of the interpolation step of the policy. However, tracking is demonstrated in yaw and pitch. Inferior tracking is observed in roll, although a velocity of 30 degrees per second in roll is more than what can reasonably be expected from a normal jump according to 2 . This further clarifies the applicability of the learned policy to offset any initial angular velocity from a jump. 
 
 Appendix: We use attitude to denote the 3D orientation of the quadruped.We use the Hamilton convention of the quaternion that defines a transformation from a frame B to frame W according to x w = q w b ⊗ x b ⊗ (q w b ) * , where ⊗ and * denote the quaternion product and conjugate respectively.Y. Song, A. Romero, M. Müller, V. Koltun, and D. Scaramuzza. Reaching the limit in autonomous racing: Optimal control versus reinforcement learning. Science Robotics, 8(82), Sept. 2023. ISSN 2470-9476. doi:10.1126/scirobotics.adg1462. URL http://dx.doi.org/10.1126/scirobotics.adg1462.A. Olsen and K. Alexis. Martian lava tube exploration using jumping legged robots: A concept study. In 74th International Astronautical Congress (IAC), Baku, Azerbaijan, October 2-6 2023. IAF.N. Rudin, D. Hoeller, P. Reist, and M. Hutter. Learning to walk in minutes using massively parallel deep reinforcement learning, 2022. 


Decision: # Strengths:
Multiple reviewers acknowledge the writeup clarity, effort in the experiments, and the detailed insights presented in the paper

# Weakness:
Reviewers are failing to recognize the contribution of the proposed work beyond well-known techniques (domain randomization, sim2real, etc) in legged locomotion. Lack of baseline comparisons has also been raised.

# Recommendations:
It is recommended to improve the positioning of the paper. Authors are encouraged to clarify the contributions made by the paper and improve baselines and ablations. Not necessarily, all papers need a novel method. Often well-known techniques applied in interesting ways move the needle in a tough problem. Leverage improvements in baselines and ablations to highlight such contributions and strengthen the submissions.

# Rebuttals
The authors added new experiments and clarified concerns raised by the reviewer. Two authors increased their scores. Reviewers are split on this paper - two in favor and one not in support. Based on the points raised by all reviewers, clarity of presentation, experimental details, and the contributions the paper will make to the field AC recommends acceptance of the paper. Authors are strongly advised to make a best care effort to address all comments from all reviewers, especially dUvT.

Reviews: review: title: title: A great demonstration of deep reinforcement learning (DRL) on a quadrupled system, but lack baseline comparisons and the need of DRL is unclear.
paper_summary: paper_summary: The paper describes the design of a fast attitude controller for a customized quadrupled robot platform. The attitude controller is learned using PPO via simulated experiences using Isaac Sim. The policy is transferred to real hardware in two tests: free falling and  pole rotating using domain randomization. The learned policy seems to work well on hardware.
main_review: main_review: ### Significance:
- The paper introduces a customized and low-cost quadrupled robot and a nice setup, which would be useful for future research about extraterrestrial exploration if open-sourced


### Clarity: 
- The paper is well-written. However, it seems that no appendix was submitted except for a supplementary video

### Originality:
- The paper lacks originality. The approach isn’t new, and limited insights are provided. More specifically, learning policies in simulation with highly engineered rewards with domain randomization and then performing sim-to-real to hardware is commonly used. 

### Weakness
- Lack of baseline comparisons: I believe an obvious baseline would use classical controllers such as PIDs + model identification. It is unclear why we need to apply DRL here when we can use classical methods, which potentially will work well, too
- As mentioned earlier, limited new insights are provided. The paper takes the usual approach for sim-to-real and adapts it to a new hardware system
- The reward is highly engineered, and the system requires a MoCap system, which limits the potential for real applications
questions: questions: - If possible, please add more comparisons to baselines, especially with classical controllers, to validate the need for using reinforcement learning in this setting
- In Fig. 9 and Fig. 10, I assume the errors are plotted, but it isn’t clear from the description text and the figures’ caption. Also, in Fig.9 and Fig. 10, why is only one trial plotted? Also, the same for Fig.7 and Fig. 8.
- Can the authors clarify why the actuator model is important for successful sim2real transfers? Also, where is the actuator model in Fig. 6?
- Can authors elaborate on the difference between the regulation and velocity tracking problems and the paper was formulated as a regulation problem?
limitations: limitations: 3
review_summary: review_summary: The paper is a nice technical write-up but lacks novelty to become a science paper as limited new insights are provided.
title: title: Review
paper_summary: paper_summary: This paper proposes a method to use reinforcement learning (RL) for in-flight attitude control of a quadrupedal robot. The setup is discussed in detail, and both simulation and real-world experiment results are presented.
main_review: main_review: UPDATE: updated to weak accept after rebuttal.

While the results presented in this paper are a bit incremental, I find that the authors put significant effort in the experiment, and the paper provides a lot of insights besides the results. The paper currently has a few issues in results and presentation, which is why I'm giving a "weak reject" at this moment. However, I feel this paper could merit an acceptance after revision.

**Strengths:**

1. Detailed evaluation in simulation and real-world.

The authors presented detailed results for single-pole results (Section 6.1) and free-fall results (Section 6.2) with ablation studies and sim-to-real comparisons. These results clearly demonstrates the capability of the learned controller, which is greatly appreciated. 

2. Clear and thorough presentation

I enjoyed the author's detail-oriented presentation of the proposed method, including the environment setup (last paragraph of Section 5.1), simulator configuration and sim-to-real (Section 5.2), policy representation (Section 5.3). These details are often critical in ensuring the successful deployment of a reinforcement learning policy. However, they are usually omitted from the paper, and can only be found in the Appendix, open-sourced code (or not found at all). In contrast, I feel the authors presented these important details in a thorough, coherent and easy-to-follow manner.

**Weaknesses:**

1. The result is incremental.

Given the current status of DeepRL for legged locomotion (esp. [10]), the result from this paper feels a bit incremental, and does not demonstrate a lot of novelty compared with existing works. Would the learned attitude controller lead to better pose control of the robot in difficult movements? Would the controller improves the balance of the robot? Extending the paper in some of these directions could improve the impact of the work.

2. Lack of comparison with existing work.

While the results presented are complete and thorough, the authors should compare their results with existing frameworks to better demonstrate their proposed methods (e.g. [10]). Would the method in [10] directly lead to good performance on the author's hardware, or would their be additional designs?
questions: questions: I would like the authors to answer the following questions in the rebuttal:
* In section 5.1 (Line 112), the authors mention that "two policies are learned" for slow and faster motor velocities. What is the motivation behind the two-policy design? Which policy is used for the evaluation results in Section 6? How do you determine which policy to use, and how do you handle the switch between policies?

* Is the result presented in section 6.1 in simulation or in the real world?

* How would the proposed method compared with the results in [10]?
limitations: limitations: 3
review_summary: review_summary: Despite the lack of ground-breaking novelty, I think the paper provides useful insights and could be accepted after revisions.
title: title: Review for submission 354
paper_summary: paper_summary: A modified PPO for quadruped robot in in-flight attitude control
main_review: main_review: This paper proposed a modified PPO approach for Quadruped robot control.

Strength:
The target task is interesting. 
Real-world robot experiment was conducted.

Weakness:
The novelty of the proposed method based on PPO should be further explained.
The impact of the proposed method to the superior experimental results is not detailed.
Comparison with other RL baseline seems not provided.

More information related to the hard ware and RL setting should be added. More figures related to real-world task are welcome.
Why the original PPO in [9] can not be directly used in this task? The impact of the Modified PPO to the target task should be carefully analyzed.
It is suggested to discuss the potential of the proposed method to pure real-world task without relying on simulation and sim2real. Also the generalization ability of the proposed method when the real hardware and simulation are mismatch is worth discussing.
It is also suggested to add comparison with other RL baselines to further demonstrated the advantage of the proposed method.
questions: questions: More information related to the hard ware and RL setting should be added. More figures related to real-world task are welcome.
Why the original PPO in [9] can not be directly used in this task? The impact of the Modified PPO to the target task should be carefully analyzed.
It is suggested to discuss the potential of the proposed method to pure real-world task without relying on simulation and sim2real. Also the generalization ability of the proposed method when the real hardware and simulation are mismatch is worth discussing.
It is also suggested to add comparison with other RL baselines to further demonstrated the advantage of the proposed method.
limitations: limitations: 2
review_summary: review_summary: More detials of real-world experiment and some comparsion with RL baselines could further improve this paper. It is also suggested to explain the proposed method as one unique approach for the target task rather than just a modified PPO.