title: Ð Ð°Ð·ÑÐ°Ð±Ð¾ÑÐºÐ° Ð²ÑÑÐ¾ÐºÐ¾Ð¿ÑÐ¾Ð¸Ð·Ð²Ð¾Ð´Ð¸ÑÐµÐ»ÑÐ½Ð¾Ð¹ ÑÐ¸ÑÑÐµÐ¼Ñ Ð´Ð»Ñ ÑÐ±Ð¾ÑÐ° Ð´Ð°Ð½Ð½ÑÑ Ñ ÑÐ´Ð°Ð»ÐµÐ½Ð½Ð¾Ð³Ð¾ Ð¸ÑÑÐ¾ÑÐ½Ð¸ÐºÐ° Ð¿Ð¾ ÑÐµÑÐ¸
title: Ð²ÑÐ¿ÑÑÐºÐ½Ð°Ñ ÐºÐ²Ð°Ð»Ð¸ÑÐ¸ÐºÐ°ÑÐ¸Ð¾Ð½Ð½Ð°Ñ ÑÐ°Ð±Ð¾ÑÐ° Ð¼Ð°Ð³Ð¸ÑÑÑÐ°
titleType: Subtitle
title: Development of a high-performance system for collecting data from a remote source over the network
Creators: name: Ð¥Ð¸ÑÐ¼Ð°ÑÑÐ»Ð»Ð¸Ð½, ÐÐ°Ð¼Ð¸Ð»Ñ
Publisher: name: Ð¡Ð°Ð½ÐºÑ-ÐÐµÑÐµÑÐ±ÑÑÐ³ÑÐºÐ¸Ð¹ Ð¿Ð¾Ð»Ð¸ÑÐµÑÐ½Ð¸ÑÐµÑÐºÐ¸Ð¹ ÑÐ½Ð¸Ð²ÐµÑÑÐ¸ÑÐµÑ ÐÐµÑÑÐ° ÐÐµÐ»Ð¸ÐºÐ¾Ð³Ð¾
Year: 2024
Subjects: ÑÐ°ÑÐ¿ÑÐµÐ´ÐµÐ»ÐµÐ½Ð½Ð°Ñ ÑÐ¸ÑÑÐµÐ¼Ð° ÑÐ±Ð¾ÑÐ° Ð´Ð°Ð½Ð½ÑÑ
Ð²ÐµÐ±-ÑÐºÑÐ°Ð¿Ð¸Ð½Ð³
Ð¿Ð°ÑÑÐ¸Ð½Ð³
Ð³Ð¾ÑÐ¸Ð·Ð¾Ð½ÑÐ°Ð»ÑÐ½Ð¾Ðµ Ð¼Ð°ÑÑÑÐ°Ð±Ð¸ÑÐ¾Ð²Ð°Ð½Ð¸Ðµ
Ð¿Ð»Ð°Ð³Ð¸Ð½Ñ
distributed data collection system
web scraping
parsing
horizontal scaling
plugins

description: ÐÑÐ¿ÑÑÐºÐ½Ð°Ñ ÐºÐ²Ð°Ð»Ð¸ÑÐ¸ÐºÐ°ÑÐ¸Ð¾Ð½Ð½Ð°Ñ ÑÐ°Ð±Ð¾ÑÐ° Ð¿Ð¾ÑÐ²ÑÑÐµÐ½Ð° Ð¸ÑÑÐ»ÐµÐ´Ð¾Ð²Ð°Ð½Ð¸Ñ Ð¿Ð¾Ð´ÑÐ¾Ð´Ð¾Ð² Ðº Ð¾ÑÐ³Ð°Ð½Ð¸Ð·Ð°ÑÐ¸Ð¸ Ð²ÑÑÐ¾ÐºÐ¾Ð¿ÑÐ¾Ð¸Ð·Ð²Ð¾Ð´Ð¸ÑÐµÐ»ÑÐ½Ð¾Ð³Ð¾ ÑÐ±Ð¾ÑÐ° Ð´Ð°Ð½Ð½ÑÑ Ñ Ð²ÐµÐ±-ÑÐµÑÑÑÑÐ¾Ð². ÐÐ¿Ð¸ÑÐ°Ð½Ñ Ð¾ÑÐ½Ð¾Ð²Ð½ÑÐµ Ð¿Ð¾Ð´ÑÐ¾Ð´Ñ Ðº ÑÐµÐ°Ð»Ð¸Ð·Ð°ÑÐ¸Ð¸ Ð²ÐµÐ±-ÑÐºÑÐ°Ð¿Ð¸Ð½Ð³Ð° Ð¸ ÐµÐµ Ð½ÐµÐ¾Ð±ÑÐ¾Ð´Ð¸Ð¼ÑÐµ ÐºÐ¾Ð¼Ð¿Ð¾Ð½ÐµÐ½ÑÑ. ÐÑÐ¾Ð²ÐµÐ´ÐµÐ½ Ð¾Ð±Ð·Ð¾Ñ ÑÑÑÐµÑÑÐ²ÑÑÑÐ¸Ñ ÑÐ°ÑÐ¿ÑÐµÐ´ÐµÐ»ÐµÐ½Ð½ÑÑ ÑÐ¸ÑÑÐµÐ¼ Ð¿Ð¾ ÑÐ±Ð¾ÑÑ Ð´Ð°Ð½Ð½ÑÑ, Ð±ÑÐ»Ð¸ Ð¾Ð¿Ð¸ÑÐ°Ð½Ñ Ð¸Ñ Ð¿ÑÐµÐ¸Ð¼ÑÑÐµÑÑÐ²Ð° Ð¸ Ð½ÐµÐ´Ð¾ÑÑÐ°ÑÐºÐ¸. ÐÑÐ¾Ð²ÐµÐ´ÐµÐ½ Ð¾Ð±Ð·Ð¾Ñ Ð¿Ð¾ Ð¿Ð¾Ð´ÑÐ¾Ð´ÑÑÐ¸Ð¼ Ð´Ð»Ñ ÑÐµÐ°Ð»Ð¸Ð·Ð°ÑÐ¸Ð¸ Ð²ÐµÐ±-ÑÐºÑÐ°Ð¿Ð¸Ð½Ð³ ÑÐ¸ÑÑÐµÐ¼Ñ Ð±Ð¸Ð±Ð»Ð¸Ð¾ÑÐµÐº. Ð ÑÐ°Ð¼ÐºÐ°Ñ Ð¿ÑÐµÐ´ÑÑÐ°Ð²Ð»ÐµÐ½Ð½Ð¾Ð¹ ÑÐ°Ð±Ð¾ÑÑ Ð±ÑÐ»Ð° ÑÐ¿ÑÐ¾ÐµÐºÑÐ¸ÑÐ¾Ð²Ð°Ð½Ð° Ð°ÑÑÐ¸ÑÐµÐºÑÑÑÐ° Ð²ÑÑÐ¾ÐºÐ¾Ð¿ÑÐ¾Ð¸Ð·Ð²Ð¾Ð´Ð¸ÑÐµÐ»ÑÐ½Ð¾Ð¹ ÑÐ°ÑÐ¿ÑÐµÐ´ÐµÐ»ÐµÐ½Ð½Ð¾Ð¹ ÑÐ¸ÑÑÐµÐ¼Ñ ÑÐ±Ð¾ÑÐ° Ð´Ð°Ð½Ð½ÑÑ. ÐÐ»Ñ ÑÐ´Ð¾Ð±ÑÑÐ²Ð° ÑÐ°ÑÑÐ¸ÑÐµÐ½Ð¸Ñ Ð²Ð¾Ð·Ð¼Ð¾Ð¶Ð½Ð¾ÑÑÐµÐ¹ ÑÐ¸ÑÑÐµÐ¼Ñ Ð¸ÑÐ¿Ð¾Ð»ÑÐ·ÑÐµÑÑÑ Ð¿Ð¾Ð´Ð´ÐµÑÐ¶ÐºÐ° Ð¿Ð»Ð°Ð³Ð¸Ð½Ð¾Ð², Ð¾ÑÐ´ÐµÐ»ÑÐ½Ð¾Ð³Ð¾ Ð´Ð»Ñ ÐºÐ°Ð¶Ð´Ð¾Ð³Ð¾ Ð½Ð¾Ð²Ð¾Ð³Ð¾ Ð²ÐµÐ±-ÑÐµÑÑÑÑÐ°. Ð¡Ð¸ÑÑÐµÐ¼Ð° Ð±ÑÐ»Ð° ÑÐµÐ°Ð»Ð¸Ð·Ð¾Ð²Ð°Ð½Ð° Ð² Ð¿ÑÐ¾Ð³ÑÐ°Ð¼Ð¼Ð½Ð¾Ð¼ ÑÑÐµÐ´ÑÑÐ²Ðµ Ð½Ð° Ð¿ÑÐ¸Ð¼ÐµÑÐµ ÑÐ°Ð¹ÑÐ° ÐÐ²Ð¸ÑÐ¾ ÐÐ²ÑÐ¾, Ð±ÑÐ»Ð¸ Ð¿ÑÐ¾Ð°Ð½Ð°Ð»Ð¸Ð·Ð¸ÑÐ¾Ð²Ð°Ð½Ñ Ð¿ÑÐ¾Ð±Ð»ÐµÐ¼Ñ Ð² ÑÐµÐ°Ð»Ð¸Ð·Ð°ÑÐ¸Ð¸ Ð¸ ÑÐ°Ð±Ð¾ÑÐµ ÑÐµÐ°Ð»Ð¸Ð·ÑÐµÐ¼Ð¾Ð¹ ÑÐ¸ÑÑÐµÐ¼Ñ ÑÐ±Ð¾ÑÐ° Ð´Ð°Ð½Ð½ÑÑ Ð¸ Ð±ÑÐ»Ð¸ Ð¿ÑÐµÐ´Ð»Ð¾Ð¶ÐµÐ½Ñ Ð²Ð°ÑÐ¸Ð°Ð½ÑÑ Ð¸Ñ ÑÐµÑÐµÐ½Ð¸Ñ. Ð ÑÐµÐ·ÑÐ»ÑÑÐ°ÑÐµ Ð±ÑÐ» Ð¿ÑÐ¾Ð²ÐµÐ´ÐµÐ½ ÑÐµÑÑ Ð¿ÑÐ¾Ð¸Ð·Ð²Ð¾Ð´Ð¸ÑÐµÐ»ÑÐ½Ð¾ÑÑÐ¸ ÑÐ¸ÑÑÐµÐ¼Ñ, Ð½Ð° ÐºÐ¾ÑÐ¾ÑÐ¾Ð¹ Ð¾Ð½Ð° Ð¿Ð¾ÐºÐ°Ð·Ð°Ð»Ð° Ð´Ð¾ÑÑÐ¾Ð¹Ð½ÑÐ¹ ÑÐµÐ·ÑÐ»ÑÑÐ°Ñ Ñ ÑÑÐµÑÐ¾Ð¼ Ð¾Ð³ÑÐ°Ð½Ð¸ÑÐµÐ½Ð¸Ð¹ ÑÐ¾ ÑÑÐ¾ÑÐ¾Ð½Ñ Ð²ÐµÐ±-ÑÐµÑÑÑÑÐ°. Ð ÑÐ¾Ð´Ðµ Ð´Ð»Ð¸ÑÐµÐ»ÑÐ½Ð¾Ð¹ ÑÐºÑÐ¿Ð»ÑÐ°ÑÐ°ÑÐ¸Ð¸ ÑÐ¸ÑÑÐµÐ¼Ñ Ð´ÐµÐ³ÑÐ°Ð´Ð°ÑÐ¸Ð¸ Ð¿ÑÐ¾Ð¸Ð·Ð²Ð¾Ð´Ð¸ÑÐµÐ»ÑÐ½Ð¾ÑÑÐ¸ Ð½Ðµ Ð±ÑÐ»Ð¾ Ð¾Ð±Ð½Ð°ÑÑÐ¶ÐµÐ½Ð¾, ÑÑÐ¾ Ð³Ð¾Ð²Ð¾ÑÐ¸Ñ Ð¾ ÐµÐµ ÑÑÐ°Ð±Ð¸Ð»ÑÐ½Ð¾ÑÑÐ¸.
description: The masters thesis is devoted to the study of approaches to the organization of high-performance data collection from web resources. The main approaches to the implementation of web scraping and its necessary components are described. An overview of existing distributed data collection systems was conducted, their advantages and disadvantages were described. A review was conducted on libraries suitable for the implementation of web scraping systems. As part of the presented work, the architecture of a high-performance distributed data collection system was designed. For the convenience of expanding the systems capabilities, support for plug-ins is used, separate for each new web resource. The system was implemented in a software tool using the example of the Avito Auto website, problems in the implementation and operation of the implemented data collection system were analyzed and solutions were proposed. As a result, a performance test of the system was conducted, on which it showed a decent result, taking into account the limitations of the web resource. During the long-term operation of the system, no degradation of performance was detected, which indicates its stability.