title: Ð Ð°Ð·ÑÐ°Ð±Ð¾ÑÐºÐ° Ð°Ð»Ð³Ð¾ÑÐ¸ÑÐ¼Ð° Ð½Ð°Ð²Ð¸Ð³Ð°ÑÐ¸Ð¸ Ð´Ð»Ñ ÐºÐ²Ð°Ð´ÑÐ¾ÐºÐ¾Ð¿ÑÐµÑÐ° Ð½Ð° Ð¾ÑÐ½Ð¾Ð²Ðµ Ð¾Ð±ÑÑÐµÐ½Ð¸Ñ Ñ Ð¿Ð¾Ð´ÐºÑÐµÐ¿Ð»ÐµÐ½Ð¸ÐµÐ¼
title: Ð²ÑÐ¿ÑÑÐºÐ½Ð°Ñ ÐºÐ²Ð°Ð»Ð¸ÑÐ¸ÐºÐ°ÑÐ¸Ð¾Ð½Ð½Ð°Ñ ÑÐ°Ð±Ð¾ÑÐ° Ð¼Ð°Ð³Ð¸ÑÑÑÐ°
titleType: Subtitle
title: Development of a navigation algorithm for a quadrocopter using reinforcement learning
Creators: name: ÐÑÐ±Ð°Ñ, Ð­Ð»Ð¸Ð°Ñ
Publisher: name: Ð¡Ð°Ð½ÐºÑ-ÐÐµÑÐµÑÐ±ÑÑÐ³ÑÐºÐ¸Ð¹ Ð¿Ð¾Ð»Ð¸ÑÐµÑÐ½Ð¸ÑÐµÑÐºÐ¸Ð¹ ÑÐ½Ð¸Ð²ÐµÑÑÐ¸ÑÐµÑ ÐÐµÑÑÐ° ÐÐµÐ»Ð¸ÐºÐ¾Ð³Ð¾
Year: 2020
Subjects: ÐÐµÑÐ°ÑÐµÐ»ÑÐ½ÑÐµ Ð°Ð¿Ð¿Ð°ÑÐ°ÑÑ Ð±ÐµÑÐ¿Ð¸Ð»Ð¾ÑÐ½ÑÐµ
ÐÑÐ¸Ð½ÑÑÐ¸Ðµ ÑÐµÑÐµÐ½Ð¸Ð¹
ÐÐ°Ð²Ð¸Ð³Ð°ÑÐ¸Ñ
Ð¾Ð±ÑÑÐµÐ½Ð¸Ðµ Ñ Ð¿Ð¾Ð´ÐºÑÐµÐ¿Ð»ÐµÐ½Ð¸ÐµÐ¼
Ð²ÑÐµÐ¼ÐµÐ½Ð½ÑÐµ ÑÐ°Ð·Ð»Ð¸ÑÐ¸Ñ
reinforcement learning
temporal difference

description: Ð Ð¿Ð¾ÑÐ»ÐµÐ´Ð½Ð¸Ðµ Ð³Ð¾Ð´Ñ Ð±ÐµÑÐ¿Ð¸Ð»Ð¾ÑÐ½ÑÐµ Ð»ÐµÑÐ°ÑÐµÐ»ÑÐ½ÑÐµ Ð°Ð¿Ð¿Ð°ÑÐ°ÑÑ (ÐÐÐ) Ð¸Ð³ÑÐ°ÑÑ Ð²Ð°Ð¶Ð½ÑÑ ÑÐ¾Ð»Ñ Ð² Ð²ÑÐ¿Ð¾Ð»Ð½ÐµÐ½Ð¸Ð¸ Ð·Ð°Ð´Ð°Ñ, ÐºÐ¾ÑÐ¾ÑÑÐµ Ð¼Ð¾Ð³ÑÑ Ð±ÑÑÑ ÑÐ»Ð¾Ð¶Ð½ÑÐ¼Ð¸, ÑÑÑÐ´Ð¾ÑÐ¼ÐºÐ¸Ð¼Ð¸ Ð¸ ÑÐµÑÑÑÑÐ¾ÑÐ¼ÐºÐ¸Ð¼Ð¸ Ð´Ð»Ñ ÑÐºÑÐ¿ÐµÑÑÐ¾Ð². ÐÐµÐºÐ¾ÑÐ¾ÑÑÐµ Ð¸Ð· ÑÑÐ¸Ñ Ð·Ð°Ð´Ð°Ñ Ð²ÑÐ¿Ð¾Ð»Ð½ÑÑÑÑÑ Ð² Ð½ÐµÐ¸Ð·Ð²ÐµÑÑÐ½ÑÑ ÑÑÐ»Ð¾Ð²Ð¸ÑÑ, Ð¿Ð¾Ð»Ð½ÑÑ Ð¿ÑÐµÐ¿ÑÑÑÑÐ²Ð¸Ð¹, ÐºÐ¾Ð³Ð´Ð° Ð¾Ð¿ÐµÑÐ°ÑÐ¾ÑÑ Ð½ÐµÐ¾Ð±ÑÐ¾Ð´Ð¸Ð¼Ð¾ ÑÐ¿ÑÐ°Ð²Ð»ÑÑÑ ÐÐÐ Ð² ÑÐ¾Ð¾ÑÐ²ÐµÑÑÑÐ²Ð¸Ð¸ Ñ ÑÐµÐ»Ð¾Ð²ÐµÑÐµÑÐºÐ¸Ð¼ Ð¾Ð¿ÑÑÐ¾Ð¼ Ð¿ÑÐ¸Ð½ÑÑÐ¸Ñ ÑÐµÑÐµÐ½Ð¸Ð¹. Ð¢Ð°ÐºÐ¸Ð¼ Ð¾Ð±ÑÐ°Ð·Ð¾Ð¼, Ð°Ð²ÑÐ¾Ð½Ð¾Ð¼Ð½Ð°Ñ Ð½Ð°Ð²Ð¸Ð³Ð°ÑÐ¸Ñ Ð² ÑÐ°ÐºÐ¸Ñ ÑÑÐ»Ð¾Ð²Ð¸ÑÑ ÑÐ²Ð»ÑÐµÑÑÑ Ð¾Ð´Ð½Ð¾Ð¹ Ð¸Ð· ÑÐ»Ð¾Ð¶Ð½ÑÑ Ð·Ð°Ð´Ð°Ñ Ð¿Ð¾ ÑÐ¿ÑÐ°Ð²Ð»ÐµÐ½Ð¸Ñ ÐÐÐ. ÐÐ²ÑÐ¾Ð½Ð¾Ð¼Ð½Ð¾Ðµ Ð¾ÑÑÐ»ÐµÐ¶Ð¸Ð²Ð°Ð½Ð¸Ðµ Ð´Ð²Ð¸Ð¶ÑÑÐ¸ÑÑÑ ÑÐµÐ»ÐµÐ¹ ÑÐ²Ð»ÑÐµÑÑÑ Ð¾Ð´Ð½Ð¾Ð¹ Ð¸Ð· ÑÐ°ÐºÐ¸Ñ ÑÐ»Ð¾Ð¶Ð½ÑÑ Ð·Ð°Ð´Ð°Ñ, ÑÑÐµÐ±ÑÑÑÐ¸Ñ Ð¾Ñ ÑÐ¾Ð±Ð¾ÑÐ° ÑÐ¿Ð¾ÑÐ¾Ð±Ð½Ð¾ÑÑÐ¸ Ð¿Ð»Ð°Ð½Ð¸ÑÐ¾Ð²Ð°ÑÑ Ð¸ Ð¿ÑÐ¸Ð½Ð¸Ð¼Ð°ÑÑ ÑÐµÑÐµÐ½Ð¸Ñ, ÑÑÐ¾Ð±Ñ ÑÐ¿ÑÐ°Ð²Ð¸ÑÑÑÑ Ñ ÑÐ¸ÑÑÐ°ÑÐ¸ÑÐ¼Ð¸, ÐºÐ¾ÑÐ¾ÑÑÐµ Ð¿ÐµÑÐµÑÐ²Ð°ÑÑÐ²Ð°ÑÑ ÐºÐ²Ð°Ð´ÑÐ¾ÐºÐ¾Ð¿ÑÐµÑ Ð½Ð° Ð¿ÑÑÐ¸ Ðº Ð´Ð¾ÑÑÐ¸Ð¶ÐµÐ½Ð¸Ñ Ð¶ÐµÐ»Ð°ÐµÐ¼Ð¾Ð¹ ÑÐµÐ»Ð¸. ÐÑÐ¿Ð¾Ð»ÑÐ·ÑÑ oÐ±ÑÑÐµÐ½Ð¸Ðµ Ñ Ð¿Ð¾Ð´ÐºÑÐµÐ¿Ð»ÐµÐ½Ð¸ÐµÐ¼, Ð´Ð°Ð½Ð½Ð°Ñ ÑÐ°Ð±Ð¾ÑÐ° ÑÐ°ÑÑÐ¼Ð°ÑÑÐ¸Ð²Ð°ÐµÑ Ð°Ð²ÑÐ¾Ð½Ð¾Ð¼Ð½Ð¾Ðµ ÑÐ»ÐµÐ¶ÐµÐ½Ð¸Ðµ Ð·Ð° Ð´Ð²Ð¸Ð¶ÑÑÐ¸Ð¼Ð¸ÑÑ ÑÐµÐ»ÑÐ¼Ð¸ Ñ Ð¿Ð¾Ð¼Ð¾ÑÑÑ ÐºÐ²Ð°Ð´ÑÐ¾ÐºÐ¾Ð¿ÑÐµÑÐ° ÐºÐ°Ðº Ð²ÑÑÐ¾ÐºÐ¾ÑÑÐ¾Ð²Ð½ÐµÐ²ÑÑ Ð·Ð°Ð´Ð°ÑÑ, ÑÐ°Ð·Ð±Ð¸Ð²Ð°Ñ ÐµÑ Ð½Ð° Ð±Ð¾Ð»ÐµÐµ Ð¼ÐµÐ»ÐºÐ¸Ðµ Ð¿Ð¾Ð´Ð·Ð°Ð´Ð°ÑÐ¸, Ð½Ð°ÑÐ¸Ð½Ð°Ñ Ñ Ð´Ð¾ÑÑÐ¸Ð¶ÐµÐ½Ð¸Ñ Ð°Ð²ÑÐ¾Ð½Ð¾Ð¼Ð½Ð¾Ð¹ Ð½Ð°Ð²Ð¸Ð³Ð°ÑÐ¸Ð¸ Ð±ÐµÐ· Ð¼Ð¾Ð´ÐµÐ»Ð¸ Ð¸ Ð·Ð°ÐºÐ°Ð½ÑÐ¸Ð²Ð°Ñ ÑÑÐ°ÑÐ¸ÑÐµÑÐºÐ¾Ð¹ ÑÐ¾ÑÐºÐ¾Ð¹. Ð ÐµÐ·ÑÐ»ÑÑÐ°ÑÑ Ð´ÐµÐ¼Ð¾Ð½ÑÑÑÐ¸ÑÑÑÑ ÑÑÑÐµÐºÑÐ¸Ð²Ð½Ð¾ÑÑÑ Ð¸ÑÐ¿Ð¾Ð»ÑÐ·Ð¾Ð²Ð°Ð½Ð¸Ñ Ð°Ð»Ð³Ð¾ÑÐ¸ÑÐ¼Ð¾Ð² Ð²ÑÐµÐ¼ÐµÐ½Ð½Ð¾Ð¹ ÑÐ°Ð·Ð½Ð¸ÑÑ Ð¾Ð±ÑÑÐµÐ½Ð¸Ñ Ñ Ð¿Ð¾Ð´ÐºÑÐµÐ¿Ð»ÐµÐ½Ð¸ÐµÐ¼: Off-policy Q-Ð¾Ð±ÑÑÐµÐ½Ð¸Ñ Ð¸ On-policy SARSA, Ð° ÑÐ°ÐºÐ¶Ðµ Ð¸Ñ ÑÐ²Ð¾Ð¹ÑÑÐ²Ð°. ÐÐ¾Ð»ÑÑÐµÐ½Ð½ÑÐµ ÑÐµÐ·ÑÐ»ÑÑÐ°ÑÑ Ð´ÐµÐ¼Ð¾Ð½ÑÑÑÐ¸ÑÑÑÑ Ð²Ð°Ð¶Ð½ÐµÐ¹ÑÐ¸Ðµ ÑÑÑÐµÐºÑÑ Ð¿ÑÐ¾ÐµÐºÑÐ¸ÑÐ¾Ð²Ð°Ð½Ð¸Ñ ÑÑÐ½ÐºÑÐ¸Ð¸ Ð¿Ð¾Ð¾ÑÑÐµÐ½Ð¸Ñ Ð¸ Ð²ÑÐ±Ð¾ÑÐ° Ð³Ð¸Ð¿ÐµÑÐ¿Ð°ÑÐ°Ð¼ÐµÑÑÐ¾Ð², Ð° ÑÐ°ÐºÐ¶Ðµ Ð¿ÑÐµÐ´Ð»Ð°Ð³Ð°ÑÑ ÑÐµÑÐµÐ½Ð¸Ñ Ð´Ð»Ñ Ð±ÑÐ´ÑÑÐµÐ¹ ÑÐ°Ð±Ð¾ÑÑ, Ð¾ÑÐ½Ð¾Ð²Ð°Ð½Ð½ÑÐµ Ð½Ð° Ð¾Ð¿ÑÑÐµ Ð¾Ð±ÑÑÐµÐ½Ð¸Ñ Ð¸ Ð°Ð½Ð°Ð»Ð¸Ð·Ðµ.
description: In recent years, unmanned aerial vehicles (UAVs) have played an important role in performing tasks that can be complex, time consuming and resource intensive for experts. Some of these tasks are performed in unknown conditions full of obstacles, when the operator needs to control the UAV in accordance with human decision-making experience. Thus, autonomous navigation in such conditions is one of the most difficult tasks for UAV control. Autonomous tracking of moving targets is one such challenging task, requiring the robot to be able to plan and make decisions in order to cope with situations that intercept the quadcopter on its way to achieving the desired target. Using reinforcement learning, this paper treats autonomous tracking of moving targets with a quadcopter as a high-level task, breaking it down into smaller sub-tasks ranging from achieving autonomous navigation without a model to a static point. The results demonstrate the effectiveness of using the algorithms for the time difference of reinforcement learning: Off-policy Q-learning and On-policy SARSA, as well as their properties. The findings demonstrate the critical design effects of the reward function and hyperparameter selection, and suggest solutions for future work based on learning and analysis experiences.