This study examines the system integration of a game engine with robotics middleware to drive an 8 degree of freedom (DoF) robotic upper limb to generate human-like motion for telerobotic applications. The developed architecture encompasses a pipeline execution design using Blender Game Engine (BGE) including the acquisition of real human movements via the Microsoft Kinect V2, interfaced with a modeled virtual arm, and replication of similar arm movements on the physical robotic arm. In particular, this study emphasizes the integration of a human “pilot” with ways to drive such a robotic arm through simulation and later, into a ﬁnished system. Additionally, using motion capture technology, a human upper limb action was recorded and applied onto the robot arm using the proposed architecture ﬂow. Also, we showcase the robotic arm’s actions which include reaching, picking, holding, and dropping an object. This paper presents a simple and intuitive kinematic modeling and 3D simulation process, which is validated using 8-DoF articulated robot to demonstrate methods for animation, and simulation using the designed interface. Keywords: Robot Simulation, Robot Arm, Motion Capture, Game Engine, Visual Servoing. 1. INTRODUCTION The development of mechatronics has promoted the rapid improvement of human-robot interaction (HRI) leading many researchers and robot engineers to im- plement upper limb humanoid systems with prosthetic hands [1]. The collaborative robot systems, as opposed to autonomous systems, work within the controlled human environment that requires regular human intervention [2]. While the human arm is highly dexterous and a com- plex musculoskeletal structure, it is capable of perform- ing movements including reaching, grasping and manip- ulation and it is considered an essential component for human interactive robots in a collaborative engagement. The design and development of a biomimetic arm de- pends on complex parameters like the number of degrees of freedom (DoF), working envelope, and working space that the arm covers, kinematics, payload, speed and ac- celeration, accuracy, and repeatability [3]. For a hu- man operator to maximize the potential of an advanced robotic arm, the implementation of various innovative control strategies remains a key challenge in the ﬁeld of humanoid robotics. Modeling and simulation in general offers a testbed for experimenting and analyzing the behavior of the robotic system. The availability of several game engines such as Unity and Blender Game Engine (BGE) enable virtual simulations to estimate the working of the manipulator in a virtual scenario and provide tools to generate human like behavior for robotic systems [4]. Therefore, the im- plemented system integrates the open source computer animation program Blender. In [5], the BGE is used as, both, a visualisation tool and as a virtual lab (simulator), for the purpose of markerless human motion capturing This paper is supported by the E-Health Research Center (EHRC) at IIIT-Bangalore, India.in the ﬁrst place but gradually more and more as a full- ﬂedged component in a robot simulation and program- ming environment. Markerless human motion capture and depth cam- era tracking has become popular due to inexpensive en- tertainment devices such as Microsoft’s Kinect sensor, whose development kit provides skeletal tracking algo- rithms with the camera at a low cost compared to the marker-based motion capture which has the additional overhead of preparation time. Designed to track the front side of the user [6], the Kinect’s algorithm [7, 8] reduces the depth image to a representation of a stick skeleton estimating a total of 25 joints (Figure 3) making real- time skeletal tracking very accessible, non-invasive and it is shown to have a high correlation with the marker- based motion capture systems [9]. However, it is sus- ceptible to noise and suffers the same occlusion disad- vantage as marker-based capture systems to a greater de- gree. [10] examined the accuracy of Kinect depth data for static objects, showing that the average error of the depth measurement ranges from a few mm up to about 4 cm at the maximum range of 5 m and suggested that the data should be acquired within 1–3 m distance to the sensor for mapping applications. [11] proposed a method to re- produce arm movements in an upper-body humanoid us- ing a Kinect-based MoCap system. [12] uses the Kinect sensor to recognize different body gestures and generate visual HRI interface without the calculation of inverse kinematics (IK) to make the robot arm follow the posture of the human arm. The development of a computer puppetry system adopts retargetting [13,14] mechanism, where in the cap- tured movements of a user are mapped to a target char- acter in real-time [15]. The challenges of mapping a mo- tion from the user to the target character becomes more difﬁcult when the target character and user are of differ- ent size and proportion [16–18], since all aspects of the original motion cannot be preserved. Therefore, the arm model was deﬁned to match the anthropometric measures of the human subject and the arm design from [19], avoid- ing the retargeting problem (i.e., compensate for body differences). While there are various commercial robotics simu- lators already available with the most popular being Player/Stage [20] and Gazebo [21], these projects are cur- rently limited in terms of system simulation of robots in a realistic 3D environment. To overcome this, [22] de- veloped a new robotics simulator based on Blender and Python for high reusability under all kinds of robotics re- search. Moreover, Blender allows for modelling, rigging, animation, simulation, rendering and compositing of 3D meshes. While Unity was also capable of making custom objects though the object editor, it is not well optimised for creating complex meshes and has a much narrower range of accepted ﬁle types when importing non-Unity models. BGE together with Python scripting language is useful in many aspects of simulation and visualization of cogni- tive autonomous agents. It can be used as a framework which allows to develop and experiment with building blocks of the cognitive agent memory. Many advantages of using BGE include fast prototyping, realistic physics, and ﬂexibility to model virtual environments. Hence, Blender was chosen as the simulator for this study. The paper presents the implementation of a telerobotic system driven by BGE and allows users to intuitively con- trol a 3D-printed robotic arm to interact with the sur- roundings. Particularly, the study incorporates the In- verse Kinematics solver, which permits the use of IK armatures in the Game Engine, useful for simulating robotic arms and humans. The forward and inverse kine- matic control methods are showcased for testing and experimentation purposes. The integrated system de- sign includes the usage of Microsoft Kinect v2 to en- able easy markerless motion capture to the BGE inter- face, and replicate the same action on the 8-DoF physical robot. The overall integrated system design is signiﬁcant to robotics researchers who are at the intersection of cog- nitive science and human motor control. One of the main interests of creating new simulator is making it reusable by researchers and engineers, the system design reﬂected the need for simplicity and intu- itiveness while keeping the technical complexity easy to use and understand. 2. SYSTEM DESIGN In this paper, we propose an architecture designed to meet the following primary goals: (i) ability to gener- ate real-time control outputs for teleoperation control, (ii) ability to reproduce motion capture simulations. The sys- tem design, as shown in Figure 1, consists of: (1) BGE interface - for Forward and Inverse Kinematics control, (2) motion capture using Microsoft Kinect V2, and (3)the biomimetic robot arm constituting shoulder, elbow, wrist, and gripper as an interactive platform. BGE is used to create the virtual environment that simulates the real robot and to generate a communication channel be- tween the computer and the robot including motion track- ing with the integration of Microsoft Kinect. The graphical design of the kinematic structure of the human arm, particularly the shoulder-elbow-wrist mechanism, and the human skeleton is modeled in BGE through rigging. The virtual arm was designed with iden- tical bone structure mapping the biomimetic robotic arm, and to enable the transfer of the captured joint angles di- rectly to the physical robotic arm for HRI implementa- tion. The graphically modeled arm deforms based on the transformation of the bones during motion and the mag- nitude of deformation is controlled using weight painting method, allowing user to decide the level of deformation for animation purposes. The different bones control the deformations on different sections of the arm by introduc- ing differences, represented by the gradient in Figure 2, between weights of the vertices that form the arm object. BGE’s Python environment is used as an interface for the implementation and execution of forward and inverse kinematic control mechanisms. Also, the interface is de- signed to capture human motion by employing the Kinect sensor for skeleton tracking, and to apply the captured motion onto the 8-DoF biomimetic robot arm. The Kinect V2 enables tracking of 25 joints of human body and this data is relayed to BGE using NI-Mate soft- ware which is a cross-sensor real-time motion capture suite. These joints are rendered in the BGE as points, referred to as empties shown in the Figure 3 (a). The joint coordinates and orientations in the empties format are mapped to a bone structure created inside BGE in- terface to further analyze any captured or recorded mo- tion, as shown in the Figure 3 (b). This data is used for reconstruction of the human skeleton allowing real time feedback via tracking and modeling of human motion as shown in the Figure 3 (c). To preserve signiﬁcant quali- ties (i.e., arm length) of the animated motion such from motion capture to the virtual arm model in BGE interface, the motion was subjected to joint angles and orientations only instead of their coordinates. The NI-Mate tool takes real-time motion capture data from an OpenNI compliant device, such as the Microsoft Kinect V2, and conﬁgures to a industry standard Open Sound Control (OSC) protocol. OSC is a protocol for net- working sound synthesizers, computers, and other mul- timedia devices and it works by specifying an address and its content. For human motion capture, the address and the contents are the skeleton joint’s name, and po- sition and orientation values respectively. The skeletal tracking information as viewed by the Kinect sensor is communicated by NI-Mate and mapped to the BGE inter- face model for animation and control of the robotic arm. A representation of the human arm with the connected bones, represented in BGE is shown in the Figure 4. The quality of results were assessed by comparing the motion Fig. 1. Proposed architecture design for operating 8-DoF robot arm through tele-robotic and human-motion capture inter- face. Fig. 2. Weight painting applied to the 3D model of the arm. Fig. 3. Skeleton tracking using Kinect V2 in BGE. Fig. 4. Visual Representation of human arm in BGE. of the virtual arm to that of the robot, both visually and quantitatively. The biomimetic robotic left arm was 3D-printed based on [19] design principles, using Polycarbonate material weighing nearly 1.4 kilograms with a working envelope of 70 cm-radial hemisphere centered on its shoulder joint.The 8-DoF robotic arm comprises of 8 Dynamixel motors as shown in the Figure 5, with each motor responsible for one DoF and their role and range of motion is tabu- lated in the Table 1. The robotic arm design is more for studying reaching than manipulation [19]. The shoulder- to-elbow complex of the robotic arm is operated by the MX series as they offer more load carrying and current driving capability. The forearm, wrist, and gripper joints which do not require as much power are operated by AX series motors. Therefore, the robotic arm’s weight distri- bution gradually decreases towards the distal end of grip- per. The individual Dynamixel motors allow tuning of an internal Proportional-Integral-Derivative (PID) controller to establish desired parameters in the form of motor speed and maximum torque. The motors provide feedback on the following parameters: goal position set by the in- terface, present position reached by the motor, instanta- neous speed, load applied, voltage levels, temperature of the motor, and current consumed by the motor. All the motors are daisy chained to reduce wiring in the system and are driven by the Arbotix-M micro-controller, which is used as an interpreter of the signals generated for the robot through serial communication. An Arduino micro- controller is conﬁgured through I2C communication with the Arbotix micro-controller, to handle the feedback data from the assembled motors. Table 1. Various Dynamixel motors used to conﬁgure the biomimetic robotic arm. Model ID DoF Range of Motion MX-106 AT 1 Shoulder Pitch 180◦ MX-64 AT 2 Shoulder Roll 90◦ MX-64 AT 3 Arm Yaw 180◦ MX-64 AT 4 Elbow Pitch 100◦ AX-18 A 5 Forearm Yaw 300◦ AX-18 A 6 Wrist Pitch 124.8 AX-18 A 7 Wrist Roll 101.95◦ AX-18 A 8 Gripper 74.707◦ Fig. 5. Biomimetic robot arm in its rest position. 3. EXPERIMENTS AND RESULTS The joints’ data is communicated serially to the 8-DoF robotic arm through the BGE environment and the feed- back data from the motors are used to validate the pro- posed kinematic control methods. Two kinematic control schemes: forward kinematics (FK), and inverse kinemat- ics (IK) [23] were employed to drive the robot arm inde- pendently, and it is investigated for telerobotic applica- tion. An experiment to apply FK to the robotic arm was performed to sequentially orient the elbow, wrist, and gripper closing in a sequence as shown in the Fig- ure 6 (a,b,c,d), suggesting that the action was successfully replicated via inspection. Fig. 6. Forward Kinematics applied to virtual arm and the same is reﬂected in the biomimetic robot arm. An alternate control method involving inverse kine- matics (IK) is implemented using BGE’s Jacobian IK solver to compute the pose of the virtual arm. The IK Tar- get bone is manipulated directly to reach the desired posi- tion and the bending of the arm at the elbow is controlled by positioning the Pole Target bone. The Pole Target and the IK Target bone can be seen in Figure 4 (a). The IK chain consists of 4 bones from arma toarmd as shown in 4 (b). Unlike the efforts taken in FK to achieve a desired end effector orientation, the IK target bone was translated to apply transformations to target’s parent bone and its pre- decessors in series, to follow the end effector. The in-terpolation was performed linearly and in a Bezier fash- ion where inserted frames comply with a Bezier curve. The quantitative comparison between FK and IK using the motor feedback on elbow joint is shown in Figure 7. The labels AandBshown in the Figure 7 (a,b) correspond to the elbow actions shown in Figure 6 (a,b). The range of rotation applied on BGE virtual arm shown is similar to that of the robotic arm’s elbow joint as shown in Fig- ure 7 suggesting that there is one to one correspondence between the input from BGE interface to the biomimetic robot arm. Fig. 7. The orientation of elbow joints when subjected to FK and IK control mechanism. Fig. 8. Image showing four different positions of virtual arm in BGE and corresponding robotic arm, while fol- lowing a circular path, on applying IK control method. An experiment to explore the telerobotic operation abilities using the BGE interface was performed. The vir- tual arm was controlled to drive the robot arm to perform the following steps: (1) reaching to the position of the cup with open pose of the gripper and then grasping it, (2) changing its orientation, (3) carrying it, and (4) dropping it at another location. These applications were validated by comparing the motor feedback with the joint orien- tation data transmitted by BGE. The driven inputs and attained positions were found to be in agreement. Fig- ure 9(a,b,c,d) shows picking, reorienting the pose, mov- ing, and dropping of the object as dictated by Blender. Fig. 9. Telerobotic operation showing picking, holding, carrying, and dropping a cup using IK control method. Fig. 10. Motion Tracking viewed in BGE using Kinect. Fig. 11. Image showing screenshots of pre-recorded mo- tion captured positions in the form of skeletal tracking using Kinect sensor, followed by mapping to the BGE interface model, and applying to the biomimetic robotic arm. The Kinect V2 sensor is integrated to the BGE inter- face for recording and reproducing the arm motion on the robotic arm. The motion was captured at an operational distance of 3 m away from it, with the user facing the Kinect to avoid occlusion during skeletal tracking. Using NI-Mate, the position and orientation of the joints is com- municated to BGE in OSC format where it is decoded and then applied onto a rendition of a human skeleton as seen in Figure 10. The upper-limb motion is recorded by inserting keyframes in BGE following which, the im- plemented script transmits joint orientation data to the micro-controller. Figure 11 shows the stack of recorded motion rendered on a human skeleton, applied motion onto the modeled virtual arm, and corresponding roboticarm action. The three step process starting from record- ing the motion to applying onto the robotic arm was vali- dated successfully in the Figure 11 (a,b,c,d) which show- cases the sequence of reaching, grasping and dropping of a water bottle being replicated by the robotic arm. Blender also enables the user to smoothen the keyframes before transmission using its graph editor features. 4. LIMITATIONS While the systems motion is jerky in nature, it is pri- marily due to the step input response of the system. The servos, while having a resolution of 0.2930◦(AX) and 0.088◦(MX), are actuated for every degree change dur- ing the motion to avoid the servos from shutting down due to buffer overﬂow. This is attributed to the hardware limitations of the Arbotix microcontroller. The jerky motion can be smoothened over by ﬁne tun- ing the PID controller. Moreover, the system can be im- proved by dynamic adjustment of the PID gains in ac- cordance to load conditions which induce time-varying delays between control. The current robotic arm is more suitable for studying reaching than manipulation. The baud rate speciﬁcations for the MX and AX series dynamixels are 8,000 bps-4.5 Mbps and7843 bps-1 Mbps respectively. However, the experiments were conducted at a baud rate of 1 Mbps. Therefore, there is scope for improvement in the perfor- mance of the system. 5. CONCLUSION The integrated BGE system interface can be equipped with control strategies, offered adequate accuracy and can be improved to achieve real-time performances, repro- duced human-like arm movements by the robotic arm. The novelty lies in the designed open-source architecture ﬂow to drive the robot arm either via motion capture us- ing virtual reality technology or through the game engine. While there can be performance improvements, the archi- tecture ﬂow successfully executed telerobotic operation when the virtual arm was applied to FK and IK control strategies independently. The integration of the Kinect sensor to capture and apply recorded motion shows the architectures ﬂexibility with external hardware. The in- terface has shown a capability to track the major motions of the upper limb and is intuitively rendered to the robotic arm with delay from the Kinect’s hardware limitations. This study is considered signiﬁcant in the human robot interactions where the robot can eventually learn and re- spond to the human actions in the future.