Development of a Retrofit Backhoe Teleoperation Systemusing Cat Command
Koshi Shibata1 , Yuki Nishiura1 , Yusuke Tamaishi1,3 ,Kohei Matsumoto2 , Kazuto Nakashima2 , and Ryo Kurazume2
Abstract— Most existing retrofit remote-control systems forbackhoes are large, hard-to-install, and expensive. Therefore,
we propose an easy-to-install and inexpensive teleoperation
system. The proposed system comprised remote-control andsensing systems. The remote-control system retrofits robot
arm-based devices to ”Cat Command”, a compact embedded
teleoperation system with a limited communication range, andcontrols these devices via a 5G commercial network to realize
control from a remote office. Because this system does not
require any additional modifications to the embedded control
unit in the cockpit, the operator can continue working in the
cockpit even if the backhoe is remotely controlled. The systemenables the remote control of various devices from an extremely
long distance by changing the joint parts between the robot arm
and the embedded remote-control device. The sensing systemestimates the posture and position of the backhoe by attachingoriginal sensing devices to the backhoe. In addition, a 360◦
camera was installed in the cockpit to transmit work images
from the construction site to a remote office in real time.
The sensing device was smaller and lighter than conventional
devices. We confirmed that the proposed system can be used tooperate a construction site backhoe from a remote office, andthat the system can be used to excavate soil using an actual
backhoe.

I. INTRODUCTION
Construction sites are hazardous environments, and somedeveloped countries face problems such as fatal accidents
and a declining workforce owing to falling birthrates andaging populations. Unmanned and automated construction
work has gained attention as a solution, and the developmentof teleoperating systems for construction equipment has beenpromoted worldwide as part of this effort.
Teleoperation systems can be broadly classified into two
types: embedded and retrofitted. Embedded systems are developed by construction-equipment manufacturers, and theirinternal specifications are often not disclosed, making it
difficult for users to add functions. In addition, because most
systems are designed to be visually monitored by the operator, it is difficult to remotely control construction equipmentfrom a safe remote office away from the job site. To solve
this problem, construction companies develop retrofitting
systems. Previous studies proposed systems in which a robot
is retrofitted to an operator’s seat. However, most existing
1 K. Shibata, Y. Nishiura and Y. Tamaishi are with the Graduate Schoolof Information Science and Electrical Engineering, Kyushu University,
Fukuoka, Japan shibata@irvs.ait.kyushu-u.ac.jp
2 K. Matsumoto, K. Nakashima and R. Kurazume are with the Facultyof Information Science and Electrical Engineering, Kyushu University,
Fukuoka, Japan kurazume@ait.kyushu-u.ac.jp
3 Y. Tamaishi is with Tamaishi Juki Co., Ltd., Japansystems are large, hard-to-install, and expensive, posing achallenge for the automation of construction work.
This study proposed a retrofit teleoperation system that iscompact, inexpensive, and easy-to-install. The proposed system comprises a remote-control system and a sensing system.
Both systems were developed using a robot operating system(ROS) to reduce development costs and ensure scalability.
The remote-control system was designed to control an
on-site backhoe from a remote office. In this system, robot
arms and electric cylinders were retrofitted to an embedded
remote controller for a backhoe of Cat Command, a compact
embedded remote control system with a maximum communication distance limited to 400 m. The robot arms and electric
cylinders were controlled from a safe office away from a site
via a 5G commercial network. The system is more compact
and cost-effective than existing systems because it retrofits
an equipment to an embedded remote controller.
The sensing system was intended to transmit the posture and position information of a backhoe and the stateof the construction site environment to a remote operator.
To obtain the backhoe’s posture and position information,
we developed a sensing device with built-in accelerometer,gyroscope, and global navigation satellite system (GNSS)
receiver functions. The sensing device was fixed to thecabin, boom, arm, and bucket to measure the joint angles.
The sensing devices were fixed to the metal surface of thebackhoe using a magnetic force, and each device transmitted
the measured values via wireless communication, eliminating
the need for additional backhoe machining or wiring. The
backhoe surface did not require additional work. The sensing
device used an inexpensive microcontroller and sensor, and aresin housing fabricated using a fused deposition modeling(FDM) 3D printer was used to protect the sensor module,
making it inexpensive and easy to manufacture. Considering
outdoor operations, the enclosure was structurally designed
to have few gaps and was entirely covered with an acrylic
coating to ensure waterproofing and dustproofing. A 360◦
camera was installed in the backhoe cockpit to provide realtime images of the work to a remote operator. The operator
could control the backhoe using the remote-control system
while viewing the estimated backhoe posture obtained by
the sensing devices and images transmitted from the 360◦
camera, thereby enabling teleoperation.
This paper reports the results of an experiment conductedusing an actual backhoe to confirm its effectiveness
The main contributions of this study are as follows.
1) Development of an inexpensive, compact and easy-

to-install teleoperation system
The remote-control system used part of the embedded remote-control system to reduce the size of theretrofitted device to a 524.0 mm × 445.0 mm × 760.5
mm frame. The proposed remote-control system did
not require any additional equipment to be fixed tothe control device in the cockpit; thus, the operator
could easily get into the cockpit to work even if thebackhoe was remotely controlled. The sensing device
weighed 71.6 % less than a conventional sensor box
because its housing used resin coated with a waterproof
material. The required size to effectively affix was
reduced by 60.0 %. Furthermore, by incorporating a
GNSS antenna, the backhoe’s posture and position
could be acquired using only a sensing device that
could be easily attached to and removed from thebackhoe’s metal surface.
2) Development of a versatile remote-control systemusing robot arms
The remote-control system used 6-axis generalpurpose robot arms to operate the equipment used tocontrol the backhoe. Therefore, the robot arm could
be remotely controlled from an office by changing theconnection components between the robot arm and theembedded control device for various embedded control
device that can be incorporated into the relay device
described below.
II. RELATED WORKS
To date, several retrofit backhoe teleoperation systems
have been proposed. Most retrofit remote-control systems
have robots installed in cockpits, and remote control isachieved using robots [1], [2]. These systems are large,
hard-to-install and expensive. A system can be operated
by a manned operator even if a remote-control device isinstalled in the cockpit. However, the installation of thedevice deteriorates the operability of some embedded devices[3]. When a backhoe is operated from a remote office, it isnecessary to understand the status of the construction siteand backhoe to proceed with the work safely and smoothly.
Therefore, a removable sensor box was developed to measure
the backhoe behavior during loading operations [4]. Thissensor box is easily detachable by a magnetic force and canacquire the angle of the installed point; however, a GNSS
antenna must be installed separately on the backhoe.
Therefore, this study proposed a remote-control systemthat is compact and easy to use by retrofitting a robot armbased device to an embedded remote-control system without
adding any devices to the control unit in the cockpit. We
proposed a sensing device with a built-in GNSS antenna thatwas smaller and lighter than a conventional sensor box.
III. RETROFIT TELEOPERATION SYSTEM FORBACKHOE
We developed an inexpensive and easy-to-install retrofit
backhoe teleoperation system for autonomous construction.
The proposed system consisted of a remote-control system toRemote-control system
Operator

Joysticks

Relay device

Robot arms

Electric cylinders

Embedded controller

Backhoe
Posture and positionof backhoe
360 °video image
from the cockpit

Fig. 1.

Sensing system
Sensing devices

360°camera

Architecture of teleoperation system.

control a backhoe from a remote office and a sensing system
to transmit the backhoe’s posture and images observed fromthe backhoe’s cockpit to a remote operator (Fig 1). The
backhoe was controlled by an operator using a remotecontrol system based on the information transmitted fromthe sensing system. The following sections describe these
two systems in detail.
A. REMOTE-CONTROL SYSTEM
1) HARDWARE: We developed a remote-control system
to control a backhoe onsite from a remote office. The aim of
this study was to develop a system that is smaller and easier
to install than conventional retrofit remote-control systems
for backhoes. The architecture of the system is shown in
Fig 2. The system consisted of a joystick that allowed aremote operator to input control commands to the backhoe
and a device that reflected the joystick input to the backhoe’s
embedded controller on behalf of the operator (hereinafterreferred to as ”relay device”). The joystick and relay device
were connected via a 5G commercial network, and inputs
to the joystick were transmitted to the relay device in aconstruction site via an ROS topic. The relay device consisted of two 6-axis small general-purpose robot arms (myCobot 280 M5, Elephant Robotics) and two electric cylinders (EACM2WE10ARMK-G, Oriental Motor) retrofitted to
the remote controller (hereinafter referred to as ”embedded
controller”) of the embedded remote-control system (CatCommand, Caterpillar). The robot arm was connected to ananalog stick to control the backhoe manipulator and cabinswivel angle through the original connection component (Fig.

Electric cylinder
Remote officezφCrank mechanismyθOperatorx

Joysticks
ROS topic

Relay device
Construction site

Electric cylinder

Fig. 4.

Embedded controllerof Cat Command
Robot arm

Fig. 2.

Lever to control crawler

Command

Architecture of the remote-control system.

Lego-compatible
connector

Original connection component
Fig. 3. Original connection component to easily connect a robot arm andanalog stick on the embedded controller.

3). The connecting parts were fabricated using an FDM
3D printer and connected to the robot arm using Legocompatible connectors. The robot arm was connected to theembedded controller by covering the analog sticks on theembedded controller using the connecting component. Whenthe relay device was used, the relative positions of the end
effector on the robot arm and the upper surface of the analog
stick on the embedded controller were fixed. Therefore, theabove connection components provided a firm connection
between the robot arm and embedded controller while therelay device was in use; the robot arm and embedded
controller could be easily connected and disconnected when
the device was not in use. The connecting parts had notches
to prevent interference with the embedded controller, and therobot arm could tilt the analog stick in any direction up to
the physical limit of the embedded controller. The electric
cylinder was connected to a lever to control the crawlerof the backhoe. To prevent physical interference during thedevice operation, the slider-crank mechanism shown in Fig. 4
was designed. This mechanism allowed the crawler’s control
lever to be folded up to its physical limit on the embedded
controller.
The robot arms and electric cylinders described above
were controlled using joysticks. Specifically, when the joy-Mechanism to move a lever to control the crawler.

stick was tilted without pulling the joystick trigger, the robot
arm corresponding to the joystick tilted the analog stickon the embedded controller according to the joystick tilt.
However, when the joystick was tilted by pulling the joystick
trigger, the corresponding electric cylinder tilted the lever on
the embedded controller according to the joystick tilt.
Thus, the remote-control system enables remote control
by retrofitting devices based on 6-axis general-purpose robot
arms onto an embedded control device. Therefore, various
embedded control device that can be incorporated into therelay device can be controlled remotely from an office bychanging the connection components between the robot arm
and the embedded control device.
2) SOFTWARE: This section describes the control software used for the robot arm in the relay system. To reduce
motion delay, the system calculates and records the postureof the robot arm in response to inputs, and recalls therecorded posture each time an input is made. First, weconsidered the joystick input pattern. As shown in Fig. 2, therange of θ [deg.] was 0 ≤ θ < 360 and that of φ [deg.] wasconsidered to be 0 ≤ φ ≤ 45. Assuming that the joystick input
was in 1◦ increments, the total number of arbitrary θ and φpairs was 360 × 46. Subsequently, for each arbitrary pair ofθ and φ , the corresponding robot-arm posture was calculated
and recorded. The robot arm posture was calculated using
the open motion-planning library (OMPL) implemented in
MoveIt [5], a planning framework for manipulators compatible with the ROS. To evaluate the improvement in thelatency of the implemented method, we compared the time
between the case in which the target position of the end
effector and the posture calculated and the case in which
the method implemented ran. The time was measured byrandomly generating 100 joystick inputs and measuring thetime required to obtain the target posture value for the robot
arm for each input. The average measured values are listed
in Table I. The table shows that the time was reduced by
precalculating the posture of the robot arm.
B. SENSING SYSTEM
We developed a sensing system that can transmit theposture of a backhoe and the viewpoint image from the
cockpit of the backhoe to a remote operator. The system used

TABLE I
AVERAGE TIME FROM INPUT TO GET TARGET POSTURE OF ROBOT ARM .

Time [ms]Sequential computation
99.8

Precalculation
6.94 × 10−4

MQTT
Construction site

Sensing device

Estimate
postureAccelerometer

360°camera

Gyroscope
Wi-Fi add-on board

Via 5G commercial
network

Microcontroller

Remote office

Fig. 6. Sensing device that has angle estimation and self-position estimation
functions.

Fig. 5.

Architecture of the sensing system.

an inexpensive high-performance microcontroller (Spresense,
SONY) and sensors to reduce costs while maintaining performance comparable to conventional systems. The architectureof the proposed sensing system is shown in Fig. 5. The
system consisted of a sensing device for measuring thebackhoe’s posture and self-position and a 360◦ camera (Theta
Z1, RICOH) for transmitting viewpoint images from thebackhoe cockpit. Sensing devices were attached to the cabin,boom, arm, and bucket of the backhoe, and each joint angle
was measured. Each sensing device sent the measured values
to an external PC, which estimated the posture of the backhoe
from the values of each joint angle. The estimated posture
was transmitted to a remote office via a ROS topic. The work
scene was transmitted from a 360◦ camera installed in thebackhoe cockpit via a 5G commercial network. This allows
the remote operator to receive the necessary information for
the task.
Table II presents the characteristics of the sensing device.
Fig. 6 shows the specifications of the sensing device, which
was a sensor module with a microcontroller, gyroscope(BMI160, Bosch) add-on board, accelerometer (KX126,
Kionix) add-on board, Wi-Fi add-on board (iS110B, IDY),
and mobile battery (PowerCore 10000, Anker). The sensing device had angle-estimation and self-position-estimation
functions. For the angle estimation function, the sensing
device acquired the angle of the main body at 100 Hz by
applying a Madgwick filter [6] to the values measured by
the built-in gyroscope and accelerometer. The sensing device

was attached to the cabin, boom, arm, and bucket of thebackhoe to estimate its posture (Fig. 5). However, because
the direct attachment of the sensing device to the bucket
interferes with the soil, only the sensing device for the bucket
was attached to a portion of the four-joint linkage mechanismthat included the bucket.
For the self-position estimation function, Spresense was
equipped with a GNSS receiver and an onboard chip antenna
and used quasi-zenith satellite system (QZSS), which werecompatible with the submeter-level augmentation service(SLAS). The sensing device implemented a self-positionestimation function using a GNSS receiver function and an
onboard chip antenna. Because each sensing device had aself-position estimation function, the backhoe direction could
be estimated from the position information of each sensing
device.
The angle and location information acquired by the sensing device was sent to an external PC using a messagequeuing telemetry transport (MQTT) protocol. Communication between the sensing device and PC was performed
wirelessly. Each sensing device had a built-in mobile battery; therefore, wiring was not required when installing thesensing device.
The housing of the sensing device was fabricated using
an FDM 3D printer. As the minute gaps created during
3D printing can cause water leakage, an acrylic coating
was applied to the entire enclosure to make it waterproof.
The cover of the enclosure was made of an acrylic plate,
and the gap between the cover and enclosure could besealed to improve airtightness. Waterproof washers wereused in screw-fastened parts to improve airtightness. As
described above, the design considered both waterproofingand dustproofing. Four neodymium magnets were embedded
at the bottom of the sensing device enclosure, to make itTABLE II
DATA FOR THE PROPOSED SENSING DEVICE .
Item
WeightRequired size to affix
Accelerometer tolerance
Accelerometer resolution
Gyroscope tolerance
Gyroscope resolution
Angle sampling frequency
Satellite system in use
GNSS-sensor sampling frequency

Unit
g
mm
g
LSB/g
◦ /s
LSB/◦ /s
Hz
Hz

Value
659.4
140.0 × 80.0
±2
16384
2000
16.4
100
GPS L1 C/A, QZSS L1 C/A, QZSS L1 S
1easily attach to the metal surface of the backhoe. The weightand required size to effectively affix of the sensing device
were 71.6% and 60.0% smaller than those of a conventional
sensor box [4]. This makes the retrofitting of the sensor to
the backhoe more easy.
IV. EXPERIMENTS
A. ACCURACY OF ESTIMATED ANGLES
The accuracy of the angle estimation of the sensing device
was determined as follows: First, Spresense, a gyroscope
add-on board, and an accelerometer add-on board were fixed
to a rod connected to a servomotor. The device was fabricatedto stand upright and tilt by 45◦ every 2 s, and the acceleration
and angular velocity were measured at 100 Hz. The estimatedangles were calibrated using 10 s of upright state data.
The true values were measured using an encoder (UN-1000,
MUTOH).
The measured and estimated angles are shown in Fig. 7.
The root mean square error (RMSE) of the estimated value
was 1.48◦ . It was found that there was a discrepancy between
the converged value of the estimated angle after the servomotor stopped and the measured value. This discrepancy
is thought to be partially due to the low accuracy of theaccelerometer, which estimates the tilt angle relative to thedirection of gravity in static conditions. The accuracy of theangle estimation required for teleoperation and autopilot will
be verified in the future.
B. TELEOPERATION EXPERIMENTIn this experiment, with the cooperation of an experienced
operator, we confirmed that the developed system could be
used to excavate soil. The excavation procedure is as follows:
1) The backhoe was moved to a pile of earth.
2) A pile of earth was excavated.
3) While holding the excavated soil in the bucket, thebackhoe moved to the position where the soil was tobe released.
4) The soil was released.
5) The backhoe was moved to the starting position.
The backhoe used was a backhoe (CAT 320, Caterpillar)
with a sensing device and 360◦ camera, as shown in Fig. 5.
A remote operator teleoperated the backhoe from an officelocated 1.2 km away from the construction site, based on theFig. 7.

Results for measured and estimated angle.

information transmitted by the sensing system. The equipment at the remote office and construction site communicatedvia SINET6 [7] using 5G communication.
The scene of this experiment is shown in Fig. 8. The
remote-control system transmitted inputs from the remote
office to the backhoe. The sensing system estimated thelatitude and longitude of the backhoe and the angles of thecabin, boom, arm, and bucket, and transmitted the estimatedposture of the backhoe and viewpoint image from the cockpit
to a remote operator. As shown in Fig. 9, the proposed system
can be used to excavate soil.
However, there were situations where the backhoe was
moved more than expected, due to delays in the remotecontrol system and transmitted video. The delay from input
to the joystick until the backhoe started moving was 2.0 s.
The delay of the transmitted video was 1.0 s. Because communication between the remote office and the construction
site was conducted over a 5G commercial network, the delay
could vary depending on the communication status of theentire network. A potential remedy is to reduce transmission
volume by lowering the quality of the transmitted video.
We also observed that Wi-Fi communication was interrupted
when the distance between the Wi-Fi router and the devices

backhoe operation.
V. CONCLUSIONS

Fig. 8. Photograph and schematic of the teleoperation experiment using
a backhoe operated by the proposed system. Operator (upper left), relay
device (upper right), actual backhoe (lower left), and estimated posture of
the backhoe (lower right).

1)

2)

We developed a retrofitted backhoe teleoperation systemthat is compact, relatively inexpensive, and easy-to-install using Cat Command. The proposed system comprised remotecontrol and sensing systems. The remote-control systemused a portion of an embedded remote-control system tocontrol remotely at a relatively low cost with less difficulty.
The proposed system is highly versatile because it can beused with a variety of embedded remote-control systems
by changing the joint components between the embedded
remote-control system and the robot arm-based equipment
to be retrofitted. For the sensing system, we developed acompact, inexpensive, and high-performance device withbuilt-in microcontrollers and sensors. The sensing device canbe easily attached to and removed from the metal surfaceof the backhoe, and can estimate the posture and positionof the backhoe in real time. The estimated posture canbe transmitted to a remote operator via a 5G commercial
network together with the viewpoint image from the cockpit
streamed by a 360◦ camera. We confirmed that the proposedsystem can be used to excavate soil using an actual backhoe,
controlled from an office located 1.2 km away.
ACKNOWLEDGMENT

3)

This work was supported by the JST Moonshot R&D
Program] Grant Number [JPMJMS2032].
R EFERENCES

4)

5)

Fig. 9. Excavation procedure. Actual backhoe (left) and a viewpoint image
from the cockpit of the backhoe (right).

was far apart or when obstacles were placed in between.
One possible solution is to construct a mesh Wi-Fi network
to maintain a constant communication path.
In the future, we will verify the accuracy of the latitude
and longitude and verify whether the direction of the backhoe
can be estimated from the location information of multiple
sensing devices. In addition, the system will be able todisplay images from a 360◦ camera on a head-mounted
display to enable maneuvering under conditions similar to
those in a construction site. The proposed system will belinked to the ROS2-TMS-FOR-CONSTRUCTION [8] cyberphysical system for construction sites to realize autonomous[1] K. Yokoi, K. Nakashima, M. Kobayashi, H. Mihune, H. Hasunuma,
Y. Yanagihara, T. Ueno, T. Gokyuu, and K. Endou, “A tele-operated
humanoid robot drives a backhoe in the open air,” in Proceedings of theIEEE/RSJ International Conference on Intelligent Robots and Systems(IROS), vol. 2, 2003, pp. 1117–1122.
[2] T. Sasaki, T. Nagai, and K. Kawashima, “Remote control of backhoe forrescue activities using pneumatic robot system,” in Proceedings of theIEEE International Conference on Robotics and Automation (ICRA),
2006, pp. 3177–3182.
[3] K. Sumi, “Development of Simple Attachment for Remote Control(DokaTouch),” in Proceedings of the International Symposium on
Automation and Robotics in Construction (ISARC), vol. 37. IAARC
Publications, 2020, pp. 1452–1457.
[4] K. Yamada, K. Ohno, N. Miyamoto, T. Suzuki, S. Kojima, R. P. B.
Neto, T. Suzuki, K. Nagatani, Y. Shibata, K. Asano, T. Komatsu, and
S. Tadokoro, “Attachable Sensor Boxes to Visualize Backhoe Motion,”in Proceedings of the IEEE/SICE International Symposium on System
Integration (SII), 2020, pp. 706–711.
[5] I. A. Sucan and S. Chitta, “Moveit!” (Date last accessed 16-July-2023).
[Online]. Available: https://moveit.ros.org/[6] S. O. H. Madgwick, A. J. L. Harrison, and R. Vaidyanathan, “Estimationof IMU and MARG orientation using a gradient descent algorithm,” inProceedings of the IEEE International Conference on Rehabilitation
Robotics (ICORR), 2011, pp. 1–7.
[7] T. Kurimoto, K. Sasayama, O. Akashi, and S. Urushidani, “SINET6:
Nationwide 400GE-Based Academic Backbone Network in Japan,” inProceedings of the Optical Fiber Communications Conference andExhibition (OFC), 2023, pp. 1–3.
[8] IRVS,“ROS2-TMS-FOR-CONSTRUCTION,”
(Date
last
accessed
17-July-2023).
[Online].
Available:
https://github.com/irvs/ros2 tms for construction.git