# On Collaborative Aerial and Surface Robots for Environmental Monitoring of Water Bodies

Eduardo Pinto, Pedro Santana, José Barata

► To cite this version:

Eduardo Pinto, Pedro Santana, José Barata. On Collaborative Aerial and Surface Robots for Environmental Monitoring of Water Bodies. 4th Doctoral Conference on Computing, Electrical and Industrial Systems (DoCEIS), Apr 2013, Costa de Caparica, Portugal. pp.183-191, 10.1007/978-3-642-37291-9_20. [https://doi.org/10.1007/978-3-642-37291-9_20](https://doi.org/10.1007/978-3-642-37291-9_20). [hal-01348752](https://HAL.inistre.fr/hal-01348752)

HAL Id: hal-01348752

https://hal.science/hal-01348752

Submitted on 25 Jul 2016

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# On Collaborative Aerial and Surface Robots for Environmental Monitoring of Water Bodies

Eduardo Pinto<fnref n="1" />, Pedro Santana<fnref n="1" />,<fnref n="2" />, José Barata<fnref n="1" />

**Abstract.** Remote monitoring is an essential task to help maintaining Earth ecosystems. A notorious example is the monitoring of riverine environments. The solution purposed in this paper is to use an electric boat (ASV - Autonomous Surface Vehicle) operating in symbiosis with a quadrotor (UAV - Unmanned Air Vehicle). We present the architecture and solutions adopted and at the same time compare it with other examples of collaborative robotics systems, in what we expected could be used as a survey for other persons doing collaborative robotics systems. The architecture here purported will exploit the symbiotic partnership between both robots by covering the perception, navigation, coordination, and integration aspects.

**Keywords:** Collaborative robotics, ASV, UAV, remote riverine monitoring

## 1 Introduction

The monitoring of water in riverine environments is of enormous importance not only in controlling some of the most delicate ecosystems (some of the highest concentration of life forms are in the estuaries) but also for economic reasons, since almost all the potable water come from rivers or lakes making the control of water quality essential for industry, agricultural and human consume. It is an enormous challenge attending to the actual robotics state of the art to build a single robot with all the capabilities to survive and to be autonomous in this kind of environment. Although sensor networks are becoming popular solutions for this kind of problem, they are seriously limited by their static characteristics. Several projects [1], [2]had dealt with this limitation by introducing a very large number of sensors or combining static sensors with sensors with more or less locomotion capabilities or even using ASV - Autonomous Surface Vehicles. The use of small (ASV) are also becoming a reasonable and attractive alternative[3], [4]. However, providing these boats with full autonomous behaviour is not a trivial task, there are many challenges, namely: The low height the sensors equipping these robots are from the water's surface limits their ability to perceive the far-field and consequently to ensure safety by means of a

<footnote n="1">¹ Universidade Nova de Lisboa, Faculdade de Ciências e Tecnologia, Departamento de Engenharia Electrotécnica, 2829-516 Quinta da Torre, Portugal</footnote>

<footnote n="2">² Instituto Universitário de Lisboa (ISCTE-IUL), Departamento de Ciências e Tecnologias da Informação, Avenida das Forças Armadas, 1649-026 Lisboa, Portugal</footnote>
proper path planning strategy. Another related limitation is their inability to inspect riverbanks, which are key elements in the riverine ecosystem. In RiverWatch we overcome the limitations imposed from observing the environment from a low vantage point, by pairing the ASV with a small unmanned aerial vehicle (UAV). With an aerial perspective, the UAV will extend the visual field of the ASV as well as it will enable the survey of riverbanks. Equipped with solar panels, the ASV will be able to do energy harvesting for itself and for the UAV, in a mechanism known in biology as Trophallaxis. The project will exploit this symbiotic partnership between both robots by covering the perception, navigation, coordination, and integration aspects making the system extremely flexible, reliable and with extended capabilities for adaptation, self-organization and self-development.

We expect the RiverWatch project to impact on scientific (novel perception, navigation, and coordination algorithms), technological (novel autonomous multi-robot integrated platform), and application (novel environmental application scenario) dimensions.

## 1.1 Progress beyond the state-of-the-art

In order to overcome some of the limitations of static sensor networks, [5] propose the addition of a small ASV into the monitoring system. Although this approach improves the resolution of the performed analysis, the overall system continues to be spatially constrained. Several solutions have been proposed for dynamic monitoring of riverine ecosystems. A simple solution is to deploy drifters that flow with the river current gathering data along way [6]. This solution is limited to wide and obstacle free river streams, as otherwise drifters may get stuck. A more efficient solution is to use a set of ASVs [7], [8] to navigate along rivers while collecting environmental data. The use of robotic sensors is motivated by the fact that they can achieve sufficient spatial coverage to perform complete measurements, i.e., they are not limited to a set of discrete sampling points. Although many work exists in aquatic robots, some already capable of performing energy harvesting [9] and multi-vessels coordinated behaviour [10], many challenges are still open. One particularly demanding is the ability to robustly segment the river from land and obstacles up to the far-field. The low vantage point afforded by medium size surface vehicles poses additional difficulties to this problem. Previous work [11] capitalized on the benefits of multi-robot systems to handle some problems arising from the low vantage point, by using an helicopter to provide the human operator with improved situation awareness in a hurricane post-disaster situation. In this project we intend to exploit the same idea but in this case to extend the surface vehicle's situation awareness. Hence, we consider the coordination between both robots for autonomous behaviour, instead of the context of tele-operation. Although previous work considered the docking of a UAV on a ground-
based platform [12],[13], [14] docking on a moving aquatic platform has not been
reported yet. Waves, wind, and displacement are variables that must be taken into
account by the model. We will depart from previous work by considering the problem
not only under the control perspective. A smart mechanism will be used to grasp the
UAV, thus reducing the need to hover close to the landing surface, where chaotic
airflow complicates the process. Furthermore, both ASV and UAV will negotiate a
rendezvous site where it is more likely to occur successfully, such as by searching a
less windy spot. This may require the ASV to actively search for this spot while the
UAV is performing its delegated surveying task.

For short range obstacle detection in the ASV, we will mostly recur to 3-D data
provided by stereo vision, laser scanner, and underwater sonar scanner. For this
purpose we intend to adapt some work on all-terrain obstacle detection [15][16] to the
domain in question. An interesting point of this work is the use of visual attention
parallel mechanisms, which we have been shown that, if operating in an intricate way
with obstacle avoidance, the amount of processing can be considerably reduced [17],
and as a consequence, the computational requirements as well. A key aspect of
obstacle avoidance in aquatic environments, yet to be solved, is to filter out false
positives induced by waves and large ripples in the water. For this purpose we intend
to exploit the fact that false positives in this domain exhibit structured spatio-temporal
patterns (propagating waves). These patterns can be determined in volumetric terms
using the range image produced by the stereo-vision sensor and in appearance terms
as produced by the aerial image provided by the UAV.

To enable path planning in aquatic environments [18], long-range water/land
segmentation of the environment is required. This includes the ability to detect sand
banks. For this purpose we will recur mostly to appearance models learnt under the
self-supervised learning paradigm, similar to work on ground-based vehicles [19]. In
these models, near-field range data provided by stereo/laser is used to label patches of
the robot's input image as belonging to the obstacle or non-obstacle class. In our case
the classes are water and land and the input images are the ones obtained by both
ASV and UAV. Then, an on-line machine learning mechanism can be used to learn a
classifier in the appearance space given the provided water/land labels. This way the
system learns to segment water from land in the appearance-space of the images
obtained by both robots. This learning process requires to register the obstacle
detection results obtained with range data onto the aerial image produced by the
UAV. This in turn will require both robots to localise each other with respect to the
other. We expect to solve this problem using their global localisation mechanism (i.e.,
GPS based) in addition to having the UAV performing visual tracking of the ASV.
The transformations required to match the image obtained from an omnidirectional
camera mounted on the ASV with the aerial image provided by the UAV may be
useful to provide an additional cue on both robots' relative localisation. The proposed
method differs considerably from previous work on river [20] and coast-line [21] detection from aerial images, where human intervention was key to assist the learning process. We also expect to exploit maps provided by the Google Maps API, where rivers are already segmented, to score potential river hypotheses generated by the perceptual system against the map.

With the purpose of reducing ambiguities in images classification process, we consider the possibility of having the classifiers as dynamical entities exploring the image to build contextual information [22]. Context is well recognised as a key component of any robust and parsimonious perceptual system [23]. In addition to 3-D information, the sky/water interface can also be useful to generate training labels. This innovative approach can exploit cues regarding the profile of the detected skyline.

Shoreline orientation and position is also an important cue for ASV safe navigation. We intend to use particle swarms to detect and track shorelines. For this purpose we expect to use an omni-directional camera. Particle swarms have recently exhibited interesting properties, even better than particle filters, for object tracking [24]. Moreover, making these particles sensitive to contextual cues, we expect to improve detection rate while reducing computational cost. These techniques will also be used to determine shoreline from the aerial images obtained by the UAV, besides being equipped with a conventional photographic camera, it will be equipped with a near-infrared camera, as it is recognizably useful for water/land segmentation [25]. Our expectation is that the fusion of all these methods in addition to the long-range water/land segmentation mechanism results in a robust system, which in turn reduces the accuracy requirements for each of the techniques alone.

## 2 Contribution to the development of the Internet of Things

Remote monitoring is an essential task to help maintaining Earth ecosystems and will be without doubt influenced by the Internet of Things. The use of network sensors for monitoring some environmental aspects is already a reality [26]. Several TIWS – Tsunami International Warning Systems (IOTWS, NEAMTWS, PTWC, etc.) based on fixed DART® stations are actually working all over the world, coordinated by the Intergovernmental Oceanographic Commission of UNESCO integrated within the program GOOS – Global Ocean Observing System. Inspired on these systems and in the IoT the RiverWatch architecture will enable the cooperative robots of the system (UAV and the ASV) to interact and communicate among themselves and with the environment by exchanging data and information ‘sensed’ about the environment, while reacting autonomously to the ‘real/physical world’ events with or without direct human intervention depicted as one of the most important aspects in European IoT roadmap. It is our goal that with RiverWatch and by using ROS as the operating
system with its capacity of making transparent the communications between several
nodes and the recent developments in support to IPv6 to allow those requirements.

## 3 RiverWatch Architecture

We use the ROS operating system as the background of our system with all the
advantages that cross-language inter-process communication system will allow.
We've decided to adopt as possible open source solutions as pointed on the
following sub-chapters.

### 3.1 UAV (Multi-rotor, Variable Configuration)

We had evaluated several platforms Comercial: AirRObot, Asctec, Microdones, Draganfly, Cyberquad and several open source: MikroKopter, Mikro's Aeroquad, NG-UAVP, UAVX & UAVP, OpenPilot, Arducopter, Multipilot 32. The price of these commercial solutions ranges from 5000€ to more than 20000€. All of them, uses some patented and closed mechanisms of control what was unacceptable to our project. With the open-source projects, we have access to all the code and in some of them the solutions available are more complex and with more functions than the commercial solutions, and so we decided to build a platform based in open source solutions. For the low level control boards, we've adopted the Multipilot 32 solution that is totally open source, compatible with the project Arducopter (a very large active community) and offers the best processor and IMU capabilities. For the mechanical platform we've selected the open source Mikrokopter solution that offers reconfigurable frames that allow to build any configuration: Quadrotor, Quadrotor coaxial, X6, X6 axial, Octo. Power distribution boards that allow a very easy mount with a minimum of connections and wires, very good motors (MK3638) and I2C high speed motor controllers. The batteries selected were the Maxamps 11Ah 4S. For manual and emergency control we will use the system OSRC.
<figure><img src="image_2.png" /><figcaption>Fig.1. UAV components and architecture</figcaption></figure>

## 3.2 USV

In partnership with the company Jetbuster we've decided to use the commercial hulls from model JETRIDER XXL and the water turbine propulsion system. We adopted a modular configuration with 4 main components (2 hulls, connection platform and electronics boxes) that will allow the easy setup and transport of the ASV. The propulsion unit based on ducted water jets systems have great advantages in shallow waters when compared with tradition propellers, namely the possibility to navigate with few centimetres of water, more resistance to impacts, etc.

The on board sensors are divided in two categories: "Localization and attitude" and "Obstacle detection". In "Localization and attitude" we have a PB200 weather station and a CS4500 water speed sensor from Airmair and a complete GPS RTK from Ashtech (Proflex 800 base and rover). In "Obstacle detection" we have a multibeam DT837B sonar from Imagenex mounted in a tilt system based on servos EX-106 from Dynamixel, sonar that will allow us to create a 3D map of the bottom of the river to detect and predict what will be the best way in the presence of sand banks or other subaquatic obstacles. For collision detection, the main sensor we will use is the LIDAR LD-LRS2100 system from Sick. For segmentation water algorithms and detection of obstacles, we will use a Ladybug3 360 degrees camera from Pointgrey. For low visibility (nigh, fog, etc) and for segment the ASV from the water using its
special thermal signature, we will use thermal cameras. In the ASV the FLIR Quark 640 with wide angle lens, while for the UAV the camera selected was a Quark 336. For special safety precautions we will have on board an active radar deflector, infrared and visible light marks and redundant communications with VHF modem (ADL Vantage radio system), Long range Wifi and GSM modem as we could see in Fig.2.

<figure><img src="image_3.png" /><figcaption>Fig.2. USV components and architecture</figcaption></figure>

**3.3 Operations Centre**

In its first version, the control centre has been developed to run as a web-based application. This has the advantage of allowing the use of a simple web-browser for interacting with the robotic system. However, it is also relevant to enable the integration of the control centre software in custom applications. For instance, the control centre can be part of a larger software package that not only allows human-robot interaction but also integration with simulators and information management systems. Bearing this in mind, the control centre has been wrapped as a QT component that can be integrated in any QT application. An important part of this
software package will be the GSC – Ground Station Control that will allow to have
access to all the attitude information from ASV and from the AGV. Since we adopted
the open-source solution based the project Multipilot32, that follow the MAVLINK
protocol there are several Mission Planners open source available. In Fig.3 we can see
the hardware to run the GSC.

<figure><img src="image_4.png" /><figcaption>Fig.3. Control center hardware</figcaption></figure>

4 Conclusions and Further Work

The use of robots working in a symbiotic relation is a solution that shows very
promising results, although the topic is still in its infancy. Some results are: Increase
of autonomy of the entire system, the increase in flexibility and reliability. There are
however drawbacks and problems that must be yet solved. They constitute challenges
that could lead to promising research directions. As examples: Intelligent power share
mechanisms, intelligent docking, identification of each robot (not only the robot but
also its relative orientation) in the system by the others, the possibility of sensorial
expansion of the main robot of the system or the whole system itself, etc. With
RiverWatch, we hope to contribute with some answers. Many more questions will
prevail and it's our hope that this paper contributes to arouse some curiosity in the
reader and makes him think about them.
**Acknowledgments.** This work was developed within the scope of the RIVERWATCH experiment from the European FP7 Large Scale Integrating Project ECHORD and it was also partially supported by CTS multi-annual funding, through the PIDDAC Program funds.

References

1. Bayen, A. In: Floating Sensor Network. Available at: http://float.berkeley.edu/

2. Beacon Institute for Rivers and Estuaries: River and Estuary Observatory Network (REON). Available at: http://www.bire.org/institute/innovation.php

3. Manley, J.: Unmanned surface vehicles, 15 years of development. In : OCEANS 2008 (2008)

4. M Fallon, G.: Cooperative AUV navigation using a single surface craft. In Springer, ed. : Field and Service Robotics (2010)

5. Sukhatme, G. .: Design and Development of a Wireless Robotic Networked Aquatic Microbial Observing System., 205-215 (2007)

6. Emery, L. .: Drifting buoy for autonomous measurement of river environment. In : MTS/IEEE OCEANS, pp.1-8

7. Bhadauria, D. .: A Robotic Sensor Network for monitoring carp in Minnesota lakes. In : IEEE International Conference on Robotics and Automation (ICRA), pp.3837-3842

8. Tokekar, P. .: robotic system for monitoring carp in Minnesota lakes. Journal of Field Robotics (2010)

9. Low, K. .: Robot Boats as a Mobile Aquatic Sensor Network. In : ESSA Workshop, San Francisco

10. Arrichiello, F. .: Cooperative caging using autonomous aquatic surface vehicles. In : International Conference on Robotics and Automation (ICRA), Anchorage (2010)

11. Murphy, R. .: Cooperative use of unmanned sea surface and micro aerial vehicles at Hurricane Wilma. Journal of Field Robotics 25, 164-180 (2008)

12. Herisse, B. .: The Landing Problem of a VTOL Unmanned Aerial Vehicle on a Moving Platform Using Optical Flow. In : IEEE/RSJ 2010 International Conference on Intelligent Robots and Systems (IROS) (2010)

13. KE Wenzel, A.: Automatic take off, tracking and landing of a miniature UAV on a moving carrier vehicle. Journal of intelligent & robotic systems (2011)
14. M.V. Srinivasan, S. W.: How honeybees make grazing landings on flat surfaces. Biological Cybernetics (March 2000)

15. Santana, P. .: A Saliency-Based Solution for Robust Off-Road Obstacle Detection. In : IEEE International Conference on Robotics and Automation (ICRA), Anchorage (2010)

16. Santana, P. .: Saliency-Based Obstacle Detection and Ground-Plane Estimation for Off-Road Vehicles. In : Conference on Computer Vision Systems (ICVS), Liege (2009)

17. Santana, P.: Swarm Cognition on Off-Road Autonomous Robots. Swarm Intelligence (2010)

18. Singh, A. .: Efficient informative sensing using multiple robots. J. Artificial Intelligence Research 34, 707–755 (2009)

19. Dahlkamp, H.: Self-supervised monocular road detection in desert terrain. In : Robotics: Science and Systems (RSS). (2006)

20. Rathinam, S. .: Autonomous Searching and Tracking of a River using an UAV. In : American Control Conference, New York, pp.359-364

21. Xu, A.: A Vision-Based Boundary Following Framework for Aerial Vehicles. In : IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems (IROS) (2008)

22. Santana, P. .: Saliency-Based Approach to Boost Trail Detection. In : IEEE International Conference on Robotics and Automation (ICRA), Anchorage (2010)

23. Oliva, A.: The role of context in object recognition. Trends in Cognitive Sciences 11(12), 520-527 (2007)

24. Zhang, X. : Sequential particle swarm optimization for visual tracking. In : IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp.1-8 (2008)

25. Rathinam, S. .: Autonomous Searching and Tracking of a River using an UAV. In : American Control Conference, New York, pp.pp. 359-364 (2007)

26. J Yick, B.: Wireless sensor network survey. In : Computer Networks 52. Elsevier (2008) 2292–2330