# Decentralized Multi-Agent Control of a Manipulator in Continuous Task Learning
## 4 Implementation of Proposed Approach
### Reward Shaping  
The reward function helps the agents determine if their actions were desirable or not, thus allowing the agents to decide what they need to do. The dense reward signal guides the agent's exploration process and enables efficient learning; however, shaping a reward function is an exhaustive process. Another approach to overcome the exploration issue is to provide an interactive feedback to the agent where the external teacher may overwrite the actions prescribed by the policy [44]. This approach is termed policy shaping, and allows the agent to learn even in the presence of binary reward signals. In this work, the reward shaping approach has been used to overcome the exploration issue for multi-agent learners in order to compare the results with a centralized single-agent that uses the same reward function. Shaping the multi-agent policies using interactive feedback, either from external human teachers or artificial trainers, is a promising direction for future work. The reward function is divided into two phases, and a separate reward is given for each phase. The reward for the reaching phase is as follows:  
\[r_{t,reach}=w_{dist}r_{dist}+w_{vel}r_{vel}+w_{grip}r_{grip}. \tag{4}\]  
In the above equation three different rewards are represented:  
* \(r_{dist}\): the distance reward which is computed using the relative position of the gripper w.r.t. the object;
* \(r_{vel}\): the velocity reward that takes into account the end-effector velocity vector when approaching the object;
* \(r_{grip}\): the gripper open reward depending upon the action of the gripper. The combination of \(r_{vel}\) and \(r_{grip}\) incentivizes the agents to approach the object with an open gripper and a small velocity. All three contributions (\(r_{dist}\), \(r_{vel}\), \(r_{grip}\)) are weighted with respective weights \(w_{dist}\) = 0.6, \(w_{vel}\) = 0.3, and \(w_{grip}\) = 0.1, and defined as:  
Figure 5: Schematic representation of: (**top**) policy network with 46-dim input (environment states) and 8-dim output (actions); (**bottom**) value network with considered 46-dim input and 1-dim output. \[r_{dist}=1-tanh(\mathbf{p}_{grip}-\mathbf{p}_{cube}), \tag{5}\] \[r_{vel}= \begin{cases}1-tanh(abs(\mathbf{v}_{ee}/N_{dim})&if\ \ r_{ dist}<8\ \text{cm}\\ 0&otherwise,\end{cases} \] \[r_{grip}= \begin{cases}abs(action_{gripper})&if\ \ action_{gripper}<0\\ 0&otherwise,\end{cases} \]  
where \(\mathbf{p}_{grip}\) and \(\mathbf{p}_{cube}\) denote the position vector of the gripper and the cube in the world reference frame; \(\mathbf{v}_{ee}\) represents the velocity of the end-effector, which contains the linear and angular components of the velocity with \(N_{dim}=6\); \(action_{gripper}\) specifies the action of the gripper with negative values indicating that the gripper is opening. The lifting phase of the reward is initiated when the robot's gripper is in close proximity to the object. The distance threshold for switching between the grip and lift rewards is set to 2.5 cm. This is because the nominal cube measures 6 cm, and the robot should be able to grasp the cube when its grip site is within the cube boundaries. The reward for the lifting phase is defined as:  
\[r_{t,lift}=r_{dist}+r_{vel}+w_{grip}r_{grip}+r_{grasp}+r_{success}, \tag{6}\]  
and consists of the following five contributions: \(r_{dist}\) and \(r_{vel}\) are the same as the previous equation; \(r_{gip}\) reward is now reversed so it gives a reward when the gripper closes in order to grasp the cube. This reward is weighted by \(w_{grip}\) = 0.1. Additional rewards are \(r_{grasp}\) and \(r_{success}\). \(r_{grasp}\) is given if both fingers of the gripper make contact with the walls of the cube and \(r_{success}\) is an additional reward if the episode is successful, i.e., the task is completed. A successful episode constitutes if the gripper holds the cube at a certain height above the table. \(r_{grip}\); \(r_{grasp}\); \(r_{success}\) in the lifting case are defined as:  
\[r_{grip}=