# Research of Interaction Between Applications of Augmented Reality and Control Methods of UAVs

Maria Makolkina, Ruslan Kirichek, Valeria Teltevskaya, Elisaveta Surodeeva

► To cite this version:

Maria Makolkina, Ruslan Kirichek, Valeria Teltevskaya, Elisaveta Surodeeva. Research of Interac- tion Between Applications of Augmented Reality and Control Methods of UAVs. 15th International Conference on Wired/Wireless Internet Communication (WWIC), Jun 2017, St. Petersburg, Russia. pp.186-193, 10.1007/978-3-319-61382-6_15 . hal-01675422

HAL Id: hal-01675422

https://inria.hal.science/hal-01675422

Submitted on 4 Jan 2018

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
# Research of Interaction between Applications of Augmented Reality and Control Methods of UAVs

M. Makolkina¹,², R. Kirichek¹, V. Teltevskaya¹, E. Surodeeva¹

¹The Bonch-Bruevich State University of Telecommunications, 22 Pr. Bolshevikov, St. Petersburg, Russian Federation, ²Peoples’ Friendship University of Russia (RUDN University) 6 Miklukho-Maklaya St, Moscow, 117198, Russian Federation kirichek@sut.ru, malokina@list.ru, valeria-telt@yandex.ru, elisaveta.surodeeva@gmail.com

**Abstract.** In this paper, the interaction between applications of augmented reality and methods of control the unmanned aerial vehicles (UAVs) is studied. During the experiment, the application of augmented reality was used to control the UAV and the camera. Assessment of quality of service of the network and subjective score of quality of experience are carried out. Hurst exponent is used to determine the correlation between quality of service of the communication network and quality of experience.

**Keywords:** Augmented reality, flying ubiquitous sensor networks, unmanned aerial vehicles, quality of service, quality of experience, Hurst exponent

## 1 INTRODUCTION

The management of unmanned aerial vehicles is one of the most important problems, which should be solved for the widespread implementation of UAV systems. The flying touch networks use the UAVs as devices to collect data from land sensors and transfer it to the server for processing. Such networks can be used, for example, in industrial production or agricultural industry for monitoring the processes or the UAVs can be used as flying routers [1, 2, 3, 4].

There are three types of management of the UAV: manual control, automated management, automatic control. In case of remote manual control, the pilot is guided by the image transferred from the camera. Such control can be exercised by means of technologies of augmented reality and the tactile Internet. Augmented reality (AR) represents addition of the surrounding physical world with virtual data with the help of special devices of display, for example, of glasses [5]. The main task of the tactile Internet is transfer information about impact on a remote subject and the return transfer of response of this subject to influence with the minimal delays [6].

The managing man has an opportunity to influence the UAV through the application of augmented reality turns of the head and a ducking, watching change of the image from the camera in the device of display of augmented reality.
Solving a problem of management of the UAV by means of applications of aug-
mented reality, it is necessary to solve several subtasks:

* to provide transfer data about turn of the head from the device of augmented reality on the quadcopter;

* to provide transmission of a video image from the camera UAV on the AR device;

* to estimate quality of service on the network;

* to estimate quality of experience of the user in augmented reality at management of the UAV.

## 2 FEATURES OF DATA TRANSFER

The pilot can control movements of the quadcopter or the camera mounted on it by means of turns and a ducking. As the suspension bracket of the camera provides rotation only on a vertical axis, for support of the maximum review it is necessary to turn the pilotless device. In the AR device, which is on the head of the pilot, display of a video stream from the camera is provided. Thanks to it, the pilot can understand location of the UAV and transfer information about desirable turn of the camera. For this purpose, high-quality information display from camera and a timely response of the UAV and camera to movements of the managing man should be provided.

<figure><img src="image_2.png" /><figcaption>Fig. 1. Model of network for interaction between the AR device and UAV.</figcaption></figure>

During transmission of a video image, we can face a problem of noises in data trans-
fer channel. Transfer of video information is influenced by such parameters of quality
of service of a network as losses and time delays [6]. It influences on quality of expe-
rience and can lead failures in control of the UAV, because in case of control the pilot
can possess information about location of the device proceeding from the arriving
video information. In this regard, for research was selected the model of a network
(Fig. 1), in which the device of augmented reality exchanges data with the quadcopter
with use technology of wireless data transfer of Wi-Fi of IEEE 802.11n. This technol-
ogy allows to transfer the large volume of information with throughput to 150 Mbps
[7], as it is required for high-quality transmission of video information.

Within this paper, influence of quality of service of network on quality of percep-
tion of the user in the AR system at management of the UAV will be investigated.
## 3 SETTING OF EXPERIMENT

**Purpose:** A research of influence of quality of experience in systems of augmented reality on control of the UAV with use of the laboratory stand.

For this research the laboratory stand in laboratory of the Internet of Things of St. Petersburg State University of Telecommunications was developed [8, 9]. The hardware complex consists of the UAV (quadcopter 3D Robotics IRIS Plus) with the camera onboard. To capture video, an Intel Edison module connected to a Microsoft LifeCam HD-3000 camera is used, which allows to record HD video in 720p. Video-streaming is implemented by means of the FFServer library and using the RTP and RTSP protocols. Via the device of wireless access, the quadcopter is connected to AR glasses through Wi-Fi gateway. Epson Moverio BT-200 augmented reality glasses represent the transparent binocular video augmented reality glasses. They are equipped with such sensors as a gyroscope and the accelerometer, have an opportunity to interact through Wi-Fi access point. The AR glasses are equipped with the Android OS, which allowed us to use the LibVLC library and quickly develop an application for transmitting video from the quadcopter and managing its camera. The structural scheme of the stand is provided on Fig. 2.

<figure><img src="image_3.png" /><figcaption>**Fig. 2.** Structure of the experimental stand</figcaption></figure>

For collection of the experimental data and entering of additional noises into a network, the additional server with installed NetDisturb software was implemented between augmented reality glasses and the quadcopter. NetDisturb allows to make different noises and distortions in a network between the quadcopter and points, such as time delays, losses, capacity reduction [10]. NetDisturb was installed on a PC with two Gigabit Ethernet network cards. Two Wi-Fi access points were connected to this PC. The first AP was on channel 1 and the second AP was on channel 11, acting as router and repeater, correspondingly. Also software based on the pcap.net library for interception and the analysis of a traffic was developed to estimate influence of quality of service of a network on quality of experience of augmented reality in AR glasses in case of control of the quadcopter. The experiment was carried out in a room large enough to allow the UAV fly freely up to 7 meters away.

The software, created for determination of influence of quality of service on quality of experience in systems of augmented reality, intercepts a traffic of data between the quadcopter and AR glasses. On base of received data, exactly time, when the packet was received, and packet size calculation of Hurst exponent is made. Hurst exponent is the measure, used in time series analysis [11, 12, 13]. Parameter will de-
crease, if the time delay between couple of identical changes of values of a time series increases. In this case the coefficient reflects a level of self-similarity of a traffic:

* while $0 < H < 0,5$ – a time series is not self-similar, antipersistent, more probable for changing of the direction of a deviation, high definitions of a deviation follow for low and conversely;

* while $H = 0,5$ – random time series, the following value don't depend on the previous values;

* while $H > 0,5$ – time series is self-similar, persistent.

* The self-similar traffic is a traffic which looks equally at any temporal scales. In other words, the probability that the traffic on the following step will deviate average value in the same direction, as on the previous step, is so high as Hurst exponent is closer to one.

For clear determination of Hurst exponent, interception of packages will be carried out during one minute. For 1 minute time series we have to calculate dispersion – a deviation of a random value from its average value among.

$$M[(R/S)_t] \sim cn^H, \quad \text{when } n \to \infty \quad (1)$$

where $(R/S)_t$ — the normalized assessment of width of a range, $c$ — a constant, $n$ — the size of the researched unit of observations, $H$ — Hurst exponent, i.e. a level of self-similarity of the researched process. For the given finite time series of $X = X1, X2, ..., Xn$ we will calculate average value of $m$:

$$m = \frac{1}{n} \cdot \sum_{i=1}^{n} X_i \quad (2)$$

Then we will calculate a time series of deviations from average $Y$:

$$Y_t = X_t - m, \quad t = 1, 2, \dots, n \quad (3)$$

Calculate a time series of deviations $Z$:

$$Z_t = \sum_{i=1}^{t} Y_i, \quad t = 1, 2, \dots, n \quad (4)$$

Calculate a range series $R$:

$$R_t = \max(Z1, Z2, \dots, Zt) - \min(Z1, Z2, \dots, Zt), \quad t = 1, 2, \dots, n \quad (5)$$

Calculate a standard deviations series $S$:

$$\sqrt{\frac{1}{t} \sum_{i=1}^{t} (X_i - \bar{X})^2}, \quad t = 1, 2, \dots, n \quad (6)$$
where u — average definition of values of a row from X1 to Xn.

Calculate a normalized range series (R/S):

$$ (R/S)_t = \frac{R_t}{S_t}, \quad t = 1, 2, \dots, n \qquad (7) $$

To average a row (R/S)$_t$ on areas [X1, Xn] in case of different values t on an interval from 1 to n, one usually takes not all n points, but some enough set, which is evenly distributed on all interval from 1 to n.

The received averaged values and corresponding to them block lengths of averaging of t, need to be marked in logarithmic axes. The received points need to be approximated the linear function, constructed by means of the least-squares method. The slope (an inclination tangent of angle to the positive direction of an X-axis) the received line according to a formula (1) will be assessment of Hurst's coefficient of an initial time series.

In this research Hurst exponent is used for determination of correlation between quality of service and quality of experience in AR systems in case of control of the UAV.

## 4 DESCRIPTION OF EXPERIMENT

The examinee of augmented reality glasses, where information from the camera of the quadcopter is displayed, turns the head, thereby directing the camera, and subjectively estimates a response of the quadcopter on the done movements and change of the image in glasses. The experiment u be made several times, each time displays influence of different noises, such as

<figure><img src="image_4.png" /></figure>

a time delay, losses and reduction of throughput. For correct subjective evaluation, the experiment without introduction of noises was taken for a standard, i.e. as close as possible to ideal conditions of data transfer.

## 5 ANALYSIS OF RESULTS OF EXPERIMENT

After experiment without introduction of noises, Hurst exponent was calculated. It was equal to 0,61. It means that the traffic was self-similar and these results can be taken as a standard in case of further assessment of quality of experience, i.e. assess-
ment of these results is 5. Further experiments with adding of a time delay were made. Results are provided in table 1.

<table><caption>**Table 1. EXPERIMENT WITH DELAYS**</caption><thead><tr><td>Delay</td><td>Subjective score</td><td>Hurst exponent</td></tr></thead><tbody><tr><td>Without delay</td><td>5</td><td>0,61</td></tr><tr><td>10-30 ms</td><td>4</td><td>0,60</td></tr><tr><td>50-150 ms</td><td>3</td><td>0,59</td></tr><tr><td>150-250 ms</td><td>2</td><td>0,56</td></tr></tbody></table>

From the table is seen, that in case of time delays less 100 ms, quality of experience of video remains at the satisfactory level, but there are a lot of difficulties in case of control with this delay. Because of delay the managing man doesn't know when it is necessary to stop movement, i.e. control is made at random, commands are transferred with delay, the response returns with a time delay. In this case Hurst exponent truly defines correlation between quality of service and quality of experience in the AR systems – the greatest value complies to the highest assessment of the testee. Thus, in case of time delays higher than 150 ms it is impossible to exercise control of the UAV by means of the application of augmented reality.

The next experiments were made with adding of random losses in link between the quadcopter and the AR device. Results of this experiment are provided in table 2.

<table><caption>**Table 2. EXPERIMENT WITH LOSSES**</caption><thead><tr><td>Losses</td><td>Subjective score</td><td>Hurst exponent</td></tr></thead><tbody><tr><td>Without losses</td><td>5</td><td>0,61</td></tr><tr><td>0%-1%</td><td>4</td><td>0,61</td></tr><tr><td>0%-3%</td><td>3</td><td>0,60</td></tr><tr><td>0%-5%</td><td>2</td><td>0,59</td></tr><tr><td>0%-7%</td><td>2</td><td>0,53</td></tr><tr><td>0%-9%</td><td>1</td><td>0,47</td></tr></tbody></table>

Proceeding from the received results, one may say, that losses significantly influence quality of experience. In addition to difficulties with control, there are also noises in recognition of the image on the device of augmented reality. Quality of experience of video promptly falls in case of increase in number of losses, it becomes difficult to sort the image, besides some timepoints "fall out" of video information, that exerts a negative impact on control of the quadcopter.

The last experiment was connected to throughput restriction, resulting in packet losses.

<table><caption>**Table 3. EXPERIMENT WITH THROUGHPUT RESTRICTION**</caption><thead><tr><td>Throughput</td><td>Subjective score</td><td>Hurst exponent</td></tr></thead><tbody><tr><td>not limited</td><td>5</td><td>0,61</td></tr><tr><td>to 1700 kbps</td><td>5</td><td>0,61</td></tr></tbody></table>
<table><tr><td>to 1500 kbps</td><td>3</td><td>0,56</td></tr><tr><td>to 1000 kbps</td><td>2</td><td>0,55</td></tr><tr><td>to 500 kbps</td><td>1</td><td>0,49</td></tr></table>

According to table 3 it is possible to draw a conclusion that throughput restriction critically influences quality of experience of the user in systems of augmented reality. Also in case of capacity reduction, control of the quadcopter significantly becomes more difficult, because the managing man ceases to hold information about any movement of the aircraft. All data are transferred with big time delays, part of them is lost, in case of observation in the device of augmented reality begins the dizziness.

# 6 CONCLUSION

For control of unmanned aerial vehicles, it is possible to use applications of augmented reality. Because control takes place in real time, it is necessary to impose certain requirements to quality of service of a network for support of high-quality and comfortable control of the UAV. In this paper, influence of quality of service of a network on quality of experience of the user in the device of augmented reality by transmission of a video image from the aircraft camera was considered. By the received results we will draw a conclusion that on a network with the minimal time delays to 100 ms it is possible to control the quadcopter by means of the application of augmented reality without loss of quality of experience and control.

Further, it is possible to continue a research in the area of influence of quality of service of a network on a quadcopter response (the tactile Internet) in case of control by means of the DR applications, and also to consider the AR applications as a method of management of other different robotic systems.

# ACKNOWLEDGMENT.

The publication was financially supported by the Ministry of Education and Science of the Russian Federation (the Agreement number 02.a03.21.0008), RFBR according to the research project No. 16-37-00209 mol_a "Development of the principles of integration the Real Sense technology and Internet of Things".

# REFERENCES.

1. Furukawa, T.; Bourgault, F., Durrant-Whyte, H.F., Dissanayake, G., (2004), "Dynamic allocation and control of coordinated UAVs to engage multiple targets in a time-optimal manner", IEEE International Conference on Robotics and Automation ICRA'04, (3):pp. 2353-2358.

2. H.S. Park, K.H. Kim, in Virtual, Augmented and Mixed Reality. Applications of Virtual and Augmented Reality (Springer, 2014), pp. 435-442.

3. W.T. Fu, J. Gasper, S.W. Kim, in Mixed and Augmented Reality (ISMAR), 2013 IEEE International Symposium on (IEEE, 2013), pp. 59-66.

4. Filonenko A., Vavilin A., Kim T., Jo KH. "Augmented Reality Surveillance System for Road Traffic Monitoring," ICIC 2014, LNAI, vol. 8589, pp. 310-317. 2014.

5. Bilton N., "Google begins testing its augmented reality glasses," April 4, 2012.
6. A. Aijaz, M. Dohler, A. H. Aghvami, V. Friderikos, M. Frodigh, "Realizing the tactile internet: Haptic communications over next generation 5G cellular networks" in IEEE Wireless Commun., 2015.

7. IEEE 802.11-2012 IEEE Standard for Information technology--Telecommunications and information exchange between systems Local and metropolitan area networks--Specific requirements Part 11: Wireless LAN Medium Access Control (MAC) and Physical Layer (PHY) Specifications

8. R. Kirichek, A. Koucheryavy "Internet of things laboratory test bed", in ser. Lecture Notes in Electrical Engineering, vol. 34, pp. 485-494, 2016.

9. Kirichek R., Vladyko A., Zakharov M., Koucheryavy A. "Model networks for internet of things and SDN", in Proc. 18th ICACT, pp. 76-79, 2016.

10. A. Koucheryavy, A. Vladyko, R. Kirichek, "State of the art and research challenges for public flying ubiquitous sensor networks", in ser. Lecture notes in computer science, vol. 9247, pp. 299-308, 2015.

11. M. Makolkina, A. Prokopiev, A. Paramonov, A. Koucheryavy "The quality of experience subjective estimations and the Hurst parameters values interdependence", Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), LNCS, vol. 8638, pp. 311-318, 2014.

12. R. Kirichek, A. Paramonov, A. Koucheryavy, "Flying ubiquitous sensor networks as a queueing system", in Proc. 17th ICACT, pp.127-132, 2015.

13. A. Paramonov, A. Koucheryavy "M2M traffic models and flow types in case of mass event detection", in ser. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 8638 LNCS, pp. 294-300, 2014.