title: Ð Ð°Ð·ÑÐ°Ð±Ð¾ÑÐºÐ° ÑÐ¸ÑÑÐµÐ¼Ñ ÑÐ°ÑÐ¿ÑÐµÐ´ÐµÐ»ÐµÐ½Ð½Ð¾Ð³Ð¾ Ñ ÑÐ°Ð½ÐµÐ½Ð¸Ñ Ð¸ Ð¾Ð±ÑÐ°Ð±Ð¾ÑÐºÐ¸ Ð±Ð¾Ð»ÑÑÐ¾Ð³Ð¾ Ð¾Ð±ÑÐµÐ¼Ð° LIDAR Ð´Ð°Ð½Ð½ÑÑ
title: Ð²ÑÐ¿ÑÑÐºÐ½Ð°Ñ ÐºÐ²Ð°Ð»Ð¸ÑÐ¸ÐºÐ°ÑÐ¸Ð¾Ð½Ð½Ð°Ñ ÑÐ°Ð±Ð¾ÑÐ° Ð¼Ð°Ð³Ð¸ÑÑÑÐ°
titleType: Subtitle
title: Development of a distributed storage and processing system for a large amount of LIDAR data
Creators: name: Ð¡ÑÐµÐ¿Ð°Ð½Ð¾Ð², ÐÐ°Ð²ÐµÐ»
Publisher: name: Ð¡Ð°Ð½ÐºÑ-ÐÐµÑÐµÑÐ±ÑÑÐ³ÑÐºÐ¸Ð¹ Ð¿Ð¾Ð»Ð¸ÑÐµÑÐ½Ð¸ÑÐµÑÐºÐ¸Ð¹ ÑÐ½Ð¸Ð²ÐµÑÑÐ¸ÑÐµÑ ÐÐµÑÑÐ° ÐÐµÐ»Ð¸ÐºÐ¾Ð³Ð¾
Year: 2020
Subjects: LIDAR
rosbag
hadoop
hdfs
apache spark
ÑÐ°ÑÐ¿ÑÐµÐ´ÐµÐ»ÐµÐ½Ð½ÑÐµ Ð²ÑÑÐ¸ÑÐ»ÐµÐ½Ð¸Ñ
ÑÐ¶Ð°ÑÐ¸Ðµ Ñ Ð¿Ð¾ÑÐµÑÑÐ¼Ð¸
distributed computing
lossy compression

description: Ð¢ÐµÐ¼Ð° Ð²ÑÐ¿ÑÑÐºÐ½Ð¾Ð¹ ÐºÐ²Ð°Ð»Ð¸ÑÐ¸ÐºÐ°ÑÐ¸Ð¾Ð½Ð½Ð¾Ð¹ ÑÐ°Ð±Ð¾ÑÑ: Â«Ð Ð°Ð·ÑÐ°Ð±Ð¾ÑÐºÐ° ÑÐ¸ÑÑÐµÐ¼Ñ ÑÐ°ÑÐ¿ÑÐµÐ´ÐµÐ»ÐµÐ½Ð½Ð¾Ð³Ð¾ ÑÑÐ°Ð½ÐµÐ½Ð¸Ñ Ð¸ Ð¾Ð±ÑÐ°Ð±Ð¾ÑÐºÐ¸ Ð±Ð¾Ð»ÑÑÐ¾Ð³Ð¾ Ð¾Ð±ÑÐµÐ¼Ð° LIDAR Ð´Ð°Ð½Ð½ÑÑÂ».
Ð¢ÐµÑÐ½Ð¾Ð»Ð¾Ð³Ð¸Ñ LIDAR ÑÐ²Ð»ÑÐµÑÑÑ Ð¾Ð´Ð½Ð¾Ð¹ Ð¸Ð· Ð½Ð°Ð¸Ð±Ð¾Ð»ÐµÐµ Ð¿ÐµÑÑÐ¿ÐµÐºÑÐ¸Ð²Ð½ÑÑ ÑÐµÑÐ½Ð¾Ð»Ð¾Ð³Ð¸Ð¹ Ð² Ð¾Ð±Ð»Ð°ÑÑÐ¸ ÐºÐ¾Ð¼Ð¿ÑÑÑÐµÑÐ½Ð¾Ð³Ð¾ Ð·ÑÐµÐ½Ð¸Ñ Ð¸ ÑÐ°ÑÐ¿Ð¾Ð·Ð½Ð°Ð²Ð°Ð½Ð¸Ñ Ð¾Ð±ÑÐµÐºÑÐ¾Ð². ÐÐ°Ð½Ð½Ð°Ñ ÑÐµÑÐ½Ð¾Ð»Ð¾Ð³Ð¸Ñ Ð°ÐºÑÐ¸Ð²Ð½Ð¾ Ð¿ÑÐ¸Ð¼ÐµÐ½ÑÐµÑÑÑ Ð² Ð±ÐµÑÐ¿Ð¸Ð»Ð¾ÑÐ½ÑÑ Ð°Ð²ÑÐ¾Ð¼Ð¾Ð±Ð¸Ð»ÑÑ. ÐÐ¾ Ð¸Ð·-Ð·Ð° Ð±Ð¾Ð»ÑÑÐ¸Ñ Ð¾Ð±ÑÐµÐ¼Ð¾Ð² Ð¸ ÑÐ»Ð¾Ð¶Ð½Ð¾Ð¹ ÑÑÑÑÐºÑÑÑÑ ÑÐ°ÐºÐ¸Ñ Ð´Ð°Ð½Ð½ÑÑ Ð¸Ñ ÑÐ»Ð¾Ð¶Ð½Ð¾ ÑÑÑÐµÐºÑÐ¸Ð²Ð½Ð¾ ÑÑÐ°Ð½Ð¸ÑÑ Ð¸ Ð¾Ð±ÑÐ°Ð±Ð°ÑÑÐ²Ð°ÑÑ. ÐÐ¾ÑÑÐ¾Ð¼Ñ Ð´Ð°Ð½Ð½Ð°Ñ ÑÐ°Ð±Ð¾ÑÐ° Ð¿Ð¾ÑÐ²ÑÑÐµÐ½Ð° Ð¸ÑÑÐ»ÐµÐ´Ð¾Ð²Ð°Ð½Ð¸Ñ Ð¼ÐµÑÐ¾Ð´Ð¾Ð² ÑÐ°ÑÐ¿ÑÐµÐ´ÐµÐ»ÐµÐ½Ð½Ð¾Ð³Ð¾ ÑÑÐ°Ð½ÐµÐ½Ð¸Ñ Ð¸ Ð¾Ð±ÑÐ°Ð±Ð¾ÑÐºÐ¸ Ð±Ð¾Ð»ÑÑÐ¸Ñ Ð¾Ð±ÑÐµÐ¼Ð¾Ð² LIDAR Ð´Ð°Ð½Ð½ÑÑ, ÑÐ¾Ð±ÑÐ°Ð½Ð½ÑÑ Ñ Ð±ÐµÑÐ¿Ð¸Ð»Ð¾ÑÐ½ÑÑ Ð°Ð²ÑÐ¾Ð¼Ð¾Ð±Ð¸Ð»ÐµÐ¹ Ð¿Ð¾Ð´ ÑÐ¿ÑÐ°Ð²Ð»ÐµÐ½Ð¸ÐµÐ¼ ÑÑÐµÐ¹Ð¼Ð²Ð¾ÑÐºÐ° ROS. ÐÐ°Ð´Ð°ÑÐ¸, ÐºÐ¾ÑÐ¾ÑÑÐµ ÑÐµÑÐ°Ð»Ð¸ÑÑ Ð² ÑÐ¾Ð´Ðµ Ð¸ÑÑÐ»ÐµÐ´Ð¾Ð²Ð°Ð½Ð¸Ñ:
1. ÐÑÑÐ»ÐµÐ´Ð¾Ð²Ð°Ð½Ð¸Ðµ ÑÑÑÐµÑÑÐ²ÑÑÑÐ¸Ñ ÑÐµÑÐµÐ½Ð¸Ð¹ Ð² Ð¾Ð±Ð»Ð°ÑÑÐ¸ ÑÑÐ°Ð½ÐµÐ½Ð¸Ñ Ð¸ Ð¾Ð±ÑÐ°Ð±Ð¾ÑÐºÐ¸ LIDAR Ð´Ð°Ð½Ð½ÑÑ;
2. ÐÑÐ±Ð¾Ñ ÑÑÐµÐºÐ° ÑÐµÑÐ½Ð¾Ð»Ð¾Ð³Ð¸Ð¹;
3. ÐÐ¾ÑÑÑÐ¾ÐµÐ½Ð¸Ðµ Ð°ÑÑÐ¸ÑÐµÐºÑÑÑÑ Ð¿ÑÐ¾Ð³ÑÐ°Ð¼Ð¼Ð½Ð¾Ð³Ð¾ ÑÑÐµÐ´ÑÑÐ²Ð°;
4. Ð Ð°Ð·ÑÐ°Ð±Ð¾ÑÐºÐ° Ð¿Ð¾Ð´ÑÐ¾Ð´Ð° Ðº ÑÐ°ÑÐ¿ÑÐµÐ´ÐµÐ»ÐµÐ½Ð½Ð¾Ð¼Ñ ÑÑÐµÐ½Ð¸Ñ bag ÑÐ°Ð¹Ð»Ð¾Ð²;
5. Ð ÐµÐ°Ð»Ð¸Ð·Ð°ÑÐ¸Ñ Ð°Ð»Ð³Ð¾ÑÐ¸ÑÐ¼Ð° ÑÐ¶Ð°ÑÐ¸Ñ LIDAR Ð´Ð°Ð½Ð½ÑÑ;
6. ÐÐ½Ð°Ð»Ð¸Ð· ÑÐµÐ·ÑÐ»ÑÑÐ°ÑÐ¾Ð²;
ÐÐ»Ñ Ð¿Ð¾Ð²ÑÑÐµÐ½Ð¸Ñ Ð¿ÑÐ¾Ð¸Ð·Ð²Ð¾Ð´Ð¸ÑÐµÐ»ÑÐ½Ð¾ÑÑÐ¸ Ð¾Ð±ÑÐ°Ð±Ð¾ÑÐºÐ¸ LIDAR Ð´Ð°Ð½Ð½ÑÑ Ð² ÑÑÐ¾Ð¹ ÑÐ°Ð±Ð¾ÑÐµ Ð±ÑÐ»Ð¸ Ð¿ÑÐ¸Ð¼ÐµÐ½ÐµÐ½Ñ Ð¸Ð½ÑÑÑÑÐ¼ÐµÐ½ÑÑ ÑÐºÐ¾ÑÐ¸ÑÑÐµÐ¼Ñ Apache Hadoop Ð´Ð»Ñ Ð¿Ð¾ÑÑÑÐ¾ÐµÐ½Ð¸Ñ ÑÐ¸ÑÑÐµÐ¼Ñ Ñ ÑÐ°ÑÐ¿ÑÐµÐ´ÐµÐ»ÐµÐ½Ð½ÑÐ¼ ÑÑÐ°Ð½ÐµÐ½Ð¸ÐµÐ¼ Ð¸ Ð¿Ð°ÑÐ°Ð»Ð»ÐµÐ»ÑÐ½Ð¾Ð¹ Ð¾Ð±ÑÐ°Ð±Ð¾ÑÐºÐ¾Ð¹ Ð´Ð°Ð½Ð½ÑÑ. ÐÑÐµÐ´Ð»Ð°Ð³Ð°ÐµÑÑÑ Ð¸ÑÐ¿Ð¾Ð»ÑÐ·Ð¾Ð²Ð°ÑÑ ÑÐ°Ð¹Ð»Ð¾Ð²ÑÑ ÑÐ¸ÑÑÐµÐ¼Ñ HDFS Ð² ÐºÐ°ÑÐµÑÑÐ²Ðµ Ð½Ð°Ð´ÐµÐ¶Ð½Ð¾Ð³Ð¾ ÑÑÐ°Ð½Ð¸Ð»Ð¸ÑÐ°, Ð° Apache Spark Ð² ÐºÐ°ÑÐµÑÑÐ²Ðµ Ð´Ð²Ð¸Ð¶ÐºÐ° Ð´Ð»Ñ Ð¿Ð°ÑÐ°Ð»Ð»ÐµÐ»ÑÐ½Ð¾Ð¹ Ð¾Ð±ÑÐ°Ð±Ð¾ÑÐºÐ¸. Ð ÐºÐ°ÑÐµÑÑÐ²Ðµ Ð¿ÑÐ¸Ð¼ÐµÑÐ° ÑÐ°ÑÐ¿ÑÐµÐ´ÐµÐ»ÐµÐ½Ð½Ð¾Ð¹ Ð¾Ð±ÑÐ°Ð±Ð¾ÑÐºÐ¸ Ð½Ð° Ð´Ð²Ð¸Ð¶ÐºÐµ Apache Spark Ð¿ÑÐµÐ´ÑÑÐ°Ð²Ð»ÐµÐ½ Ð°Ð»Ð³Ð¾ÑÐ¸ÑÐ¼ ÑÐ¶Ð°ÑÐ¸Ñ LIDAR Ð´Ð°Ð½Ð½ÑÑ Ñ Ð¿Ð¾ÑÐµÑÑÐ¼Ð¸.
Ð ÑÐµÐ·ÑÐ»ÑÑÐ°ÑÐµ ÑÐ°Ð±Ð¾ÑÑ Ð±ÑÐ»Ð° ÑÐ°Ð·ÑÐ°Ð±Ð¾ÑÐ°Ð½Ð° ÑÐ¸ÑÑÐµÐ¼Ð° ÑÐ°ÑÐ¿ÑÐµÐ´ÐµÐ»ÐµÐ½Ð½Ð¾Ð³Ð¾ ÑÑÐ°Ð½ÐµÐ½Ð¸Ñ Ð¸ Ð¾Ð±ÑÐ°Ð±Ð¾ÑÐºÐ¸ Ð±Ð¾Ð»ÑÑÐ¾Ð³Ð¾ Ð¾Ð±ÑÐµÐ¼Ð° LIDAR Ð´Ð°Ð½Ð½ÑÑ Ñ Ð¸ÑÐ¿Ð¾Ð»ÑÐ·Ð¾Ð²Ð°Ð½Ð¸ÐµÐ¼ HDFS Ð¸ Ð´Ð²Ð¸Ð¶ÐºÐ° Apache Spark. ÐÑÐµÐ´ÑÑÐ°Ð²Ð»ÐµÐ½Ð½ÑÐ¹ Ð² ÑÐ°Ð¼ÐºÐ°Ñ Ð´Ð°Ð½Ð½Ð¾Ð¹ ÑÐ°Ð±Ð¾ÑÑ Ð¿Ð¾Ð´ÑÐ¾Ð´ Ðº ÑÑÐµÐ½Ð¸Ñ bag ÑÐ°Ð¹Ð»Ð¾Ð² Ð¿Ð¾Ð·Ð²Ð¾Ð»Ð¸Ð» Ð´Ð¾Ð±Ð¸ÑÑÑÑ Ð·Ð½Ð°ÑÐ¸ÑÐµÐ»ÑÐ½Ð¾Ð³Ð¾ Ð¿ÑÐ¸ÑÐ¾ÑÑÐ° Ð² ÑÐºÐ¾ÑÐ¾ÑÑÐ¸ ÑÑÐµÐ½Ð¸Ñ Ð¸ Ð¾Ð±ÑÐ°Ð±Ð¾ÑÐºÐ¸ Ð»Ð¾Ð³-ÑÐ°Ð¹Ð»Ð¾Ð² Ð¾Ð¿ÐµÑÐ°ÑÐ¸Ð¾Ð½Ð½Ð¾Ð¹ ÑÐ¸ÑÑÐµÐ¼Ñ ROS Ð² ÑÑÐ°Ð²Ð½ÐµÐ½Ð¸Ð¸ Ñ Ð²ÑÑÐ¸ÑÐ»ÐµÐ½Ð¸ÑÐ¼Ð¸ Ð½Ð° Ð¾Ð´Ð½Ð¾Ð¹ Ð¼Ð°ÑÐ¸Ð½Ðµ.
description: The subject of the graduate qualification work is âDevelopment of a distributed storage and processing system for a large amount of LIDAR dataâ.
LIDAR technology is one of the most promising technologies in computer vision and object recognition. This technology is already actively used in self-driving cars. However, it is challenging to effectively store and compute LIDAR data because of its large volumes and complex structure. Therefore, this work is devoted to the study of methods for distributed storage and processing of large volumes of LIDAR data collected from unmanned vehicles controlled by the ROS framework. The research set the following goals:
1. Studying existing solutions in the field of storage and processing of LIDAR data;
2. Technology stack selection;
3. Development of the software architecture;
4. Development of the approach for distributed reading of bag files;
5. Implementation of the LIDAR data compression algorithm;
6. Analysis of the results;
In order to increase processing productivity this paper proposes usage of Hadoop ecosystemsâ tools to build a system with distributed storage and parallel data processing. It is offered to use HDFS as a reliable data storage and Apache Spark as a parallel processing engine for LIDAR data. Also, this paper presents a lossy compression algorithm as an example of distributed processing with Apache Spark. The results show that the proposed tools can effectively process a large amount of LIDAR data.
As the result of this work, a system for distributed storage and processing of a large amount of LIDAR data was developed. The approach to reading bag files presented in this work made it possible to achieve a significant increase in the speed of reading and processing log files of the ROS operating system in comparison with calculations on a single machine.