title: ÐÐµÑÐ¾Ð´ ÑÐ°ÑÐ¿ÑÐµÐ´ÐµÐ»ÐµÐ½Ð½Ð¾Ð³Ð¾ Ð¸ Ð²ÑÑÐ¾ÐºÐ¾Ð¿ÑÐ¾Ð¸Ð·Ð²Ð¾Ð´Ð¸ÑÐµÐ»ÑÐ½Ð¾Ð³Ð¾ Ð¸Ð·Ð²Ð»ÐµÑÐµÐ½Ð¸Ñ Ð¸Ð½ÑÐ¾ÑÐ¼Ð°ÑÐ¸Ð¸ Ñ web-ÑÐµÑÑÑÑÐ¾Ð²
title: Ð²ÑÐ¿ÑÑÐºÐ½Ð°Ñ ÐºÐ²Ð°Ð»Ð¸ÑÐ¸ÐºÐ°ÑÐ¸Ð¾Ð½Ð½Ð°Ñ ÑÐ°Ð±Ð¾ÑÐ° Ð¼Ð°Ð³Ð¸ÑÑÑÐ°
titleType: Subtitle
title: Method of distributed and high-performance retrieval of information from web resources
Creators: name: Ð­Ð¹Ð·ÐµÐ½Ð°Ñ, ÐÐµÐ½Ð¸Ñ
Publisher: name: Ð¡Ð°Ð½ÐºÑ-ÐÐµÑÐµÑÐ±ÑÑÐ³ÑÐºÐ¸Ð¹ Ð¿Ð¾Ð»Ð¸ÑÐµÑÐ½Ð¸ÑÐµÑÐºÐ¸Ð¹ ÑÐ½Ð¸Ð²ÐµÑÑÐ¸ÑÐµÑ ÐÐµÑÑÐ° ÐÐµÐ»Ð¸ÐºÐ¾Ð³Ð¾
Year: 2021
Subjects: ÐÐ½ÑÐµÑÐ½ÐµÑ
ÑÐºÑÐ°Ð¿Ð¸Ð½Ð³
ÑÐ°ÑÐ¿ÑÐµÐ´ÐµÐ»ÐµÐ½Ð½ÑÐ¹ ÑÐ±Ð¾Ñ Ð´Ð°Ð½Ð½ÑÑ
Ð¾ÑÐºÐ°Ð·Ð¾ÑÑÑÐ¾Ð¹ÑÐ¸Ð²Ð°Ñ ÑÐ¸ÑÑÐµÐ¼Ð° Ð¿Ð°ÑÑÐ¸Ð½Ð³Ð°
Ð³Ð¾ÑÐ¸Ð·Ð¾Ð½ÑÐ°Ð»ÑÐ½Ð¾Ðµ Ð¼Ð°ÑÑÑÐ°Ð±Ð¸ÑÐ¾Ð²Ð°Ð½Ð¸Ðµ
web-scraping
distributed data collection
fail-safe parsing system
horizontal scaling

description: ÐÑÐ¿ÑÑÐºÐ½Ð°Ñ ÐºÐ²Ð°Ð»Ð¸ÑÐ¸ÐºÐ°ÑÐ¸Ð¾Ð½Ð½Ð°Ñ ÑÐ°Ð±Ð¾ÑÐ° Ð¼Ð°Ð³Ð¸ÑÑÑÐ° Ð¿Ð¾ÑÐ²ÑÑÐµÐ½Ð° Ð¸ÑÑÐ»ÐµÐ´Ð¾Ð²Ð°Ð½Ð¸Ñ Ð² Ð¾Ð±Ð»Ð°ÑÑÐ¸ ÑÑÐµÐ´ÑÑÐ² Ð°Ð²ÑÐ¾Ð¼Ð°ÑÐ¸ÑÐµÑÐºÐ¾Ð³Ð¾ ÑÐ±Ð¾ÑÐ° Ð´Ð°Ð½Ð½ÑÑ Ñ Ð²ÐµÐ±-ÑÐµÑÑÑÑÐ¾Ð². ÐÐ¿Ð¸ÑÐ°Ð½Ñ Ð¿Ð¾Ð´ÑÐ¾Ð´Ñ Ð°Ð²ÑÐ¾Ð¼Ð°ÑÐ¸Ð·Ð¸ÑÐ¾Ð²Ð°Ð½Ð½Ð¾Ð³Ð¾ Ð¿Ð¾Ð¸ÑÐºÐ° Ð¸ ÑÐ±Ð¾ÑÐ° Ð´Ð°Ð½Ð½ÑÑ Ñ Ð²ÐµÐ±-ÑÐµÑÑÑÑÐ¾Ð². ÐÐ·ÑÑÐµÐ½Ñ ÑÑÑÐµÑÑÐ²ÑÑÑÐ¸Ðµ Ð²ÑÑÐ¾ÐºÐ¾Ð¿ÑÐ¾Ð¸Ð·Ð²Ð¾Ð´Ð¸ÑÐµÐ»ÑÐ½ÑÐµ ÑÐ¸ÑÑÐµÐ¼Ñ Ð°Ð²ÑÐ¾Ð¼Ð°ÑÐ¸Ð·Ð¸ÑÐ¾Ð²Ð°Ð½Ð½Ð¾Ð³Ð¾ ÑÐ±Ð¾ÑÐ° Ð´Ð°Ð½Ð½ÑÑ, Ð¾ÑÐ½Ð¾Ð²Ð°Ð½Ð½ÑÐµ Ð½Ð° ÑÑÐµÐ¹Ð¼Ð²Ð¾ÑÐºÐµ Scrapy. ÐÑÐ¾Ð²ÐµÐ´ÐµÐ½ Ð¾Ð±Ð·Ð¾Ñ ÑÐ°Ð·Ð»Ð¸ÑÐ½ÑÑ Ð±Ð¸Ð±Ð»Ð¸Ð¾ÑÐµÐº Ð¸ ÑÑÐµÐ¹Ð¼Ð²Ð¾ÑÐºÐ¾Ð² Ð´Ð»Ñ ÑÐµÐ°Ð»Ð¸Ð·Ð°ÑÐ¸Ð¸ ÑÐ¸ÑÑÐµÐ¼ Ð²ÐµÐ±-ÑÐºÑÐ°Ð¿Ð¸Ð½Ð³Ð°. Ð ÑÐ°Ð¼ÐºÐ°Ñ Ð´Ð°Ð½Ð½Ð¾Ð¹ ÑÐ°Ð±Ð¾ÑÑ Ð¾Ð¿Ð¸ÑÐ°Ð½Ð° Ð²ÑÑÐ¾ÐºÐ¾Ð¿ÑÐ¾Ð¸Ð·Ð²Ð¾Ð´Ð¸ÑÐµÐ»ÑÐ½Ð°Ñ Ð¸ Ð¾ÑÐºÐ°Ð·Ð¾ÑÑÑÐ¾Ð¹ÑÐ¸Ð²Ð°Ñ ÑÐ°ÑÐ¿ÑÐµÐ´ÐµÐ»ÑÐ½Ð½Ð°Ñ ÑÐ¸ÑÑÐµÐ¼Ñ ÑÐ±Ð¾ÑÐ° Ð´Ð°Ð½Ð½ÑÑ Ñ Ð²ÐµÐ±-ÑÐµÑÑÑÑÐ¾Ð². ÐÑÐµÐ´Ð»Ð¾Ð¶ÐµÐ½Ð½ÑÐ¹ Ð¿Ð¾Ð´ÑÐ¾Ð´ Ðº Ð¾ÑÐ³Ð°Ð½Ð¸Ð·Ð°ÑÐ¸Ð¸ Ð²ÑÑÐ¸ÑÐ»ÐµÐ½Ð¸Ð¹ Ð·Ð°ÐºÐ»ÑÑÐ°ÐµÑÑÑ Ð² Ð¸ÑÐ¿Ð¾Ð»ÑÐ·Ð¾Ð²Ð°Ð½Ð¸Ðµ ÑÐ¸ÑÑÐµÐ¼Ñ Ð¾ÑÐºÐµÑÑÑÐ°ÑÐ¸Ð¸ ÐºÐ¾Ð½ÑÐµÐ¹Ð½ÐµÑÐ¾Ð² Kubernetes, Ð´Ð»Ñ Ð±Ð°Ð»Ð°Ð½ÑÐ¸ÑÐ¾Ð²ÐºÐ¸ Ð½Ð°Ð³ÑÑÐ·ÐºÐ¸ Ð¼ÐµÐ¶Ð´Ñ ÑÐ»ÐµÐ¼ÐµÐ½ÑÐ°Ð¼Ð¸ ÐºÐ»Ð°ÑÑÐµÑÐ° Ð¸ ÑÐµÐ½ÑÑÐ°Ð»Ð¸Ð·Ð°ÑÐ¸Ð¸ ÑÐ¿ÑÐ°Ð²Ð»ÐµÐ½Ð¸Ñ. Ð ÐºÐ°ÑÐµÑÑÐ²Ðµ Ð¾ÑÐ½Ð¾Ð²Ð½Ð¾Ð³Ð¾ ÑÑÐµÐ´ÑÑÐ²Ð° ÑÐµÐ°Ð»Ð¸Ð·Ð°ÑÐ¸Ð¸ Ð²ÐµÐ±-ÑÐºÑÐ°Ð¿ÐµÑÐ¾Ð² Ð¸ÑÐ¿Ð¾Ð»ÑÐ·ÑÐµÑÑÑ ÑÑÐµÐ¹Ð¼Ð²Ð¾ÑÐº Scrapy. Ð ÑÐµÐ·ÑÐ»ÑÑÐ°Ñ Ð¿ÑÐ¸Ð²ÐµÐ´ÐµÐ½Ð¾ ÑÑÐ°Ð²Ð½ÐµÐ½Ð¸Ðµ ÑÐºÐ¾ÑÐ¾ÑÑÐ¸ ÑÐ±Ð¾ÑÐ° Ð´Ð°Ð½Ð½ÑÑ Ñ Ð¸ÑÐ¿Ð¾Ð»ÑÐ·Ð¾Ð²Ð°Ð½Ð¸ÐµÐ¼ Ð¿ÑÐ¾ÑÑÐ¾Ð³Ð¾ Ð¿ÑÐ¸Ð»Ð¾Ð¶ÐµÐ½Ð¸Ñ Ð½Ð° Ð±Ð°Ð·Ðµ Scrapy Ð¸ ÑÐ°Ð·ÑÐ°Ð±Ð¾ÑÐ°Ð½Ð½Ð¾Ð¹ ÑÐ¸ÑÑÐµÐ¼Ð¾Ð¹, Ð° ÑÐ°ÐºÐ¶Ðµ Ð¿Ð¾ÐºÐ°Ð·Ð°Ð½Ñ ÑÐµÐ·ÑÐ»ÑÑÐ°ÑÑ ÑÐ²ÐµÐ»Ð¸ÑÐµÐ½Ð¸Ñ Ð¿ÑÐ¾Ð¸Ð·Ð²Ð¾Ð´Ð¸ÑÐµÐ»ÑÐ½Ð¾ÑÑÐ¸ Ð·Ð° ÑÑÐµÑ Ð¿Ð¾Ð²ÑÑÐµÐ½Ð¸Ñ ÐºÐ¾Ð»Ð¸ÑÐµÑÑÐ²Ð° ÑÐ°Ð±Ð¾ÑÐ¸Ñ ÑÐ·Ð»Ð¾Ð²
description: The master's final qualifying work is devoted to research in the field of means of automatic data collection from web resources. Approaches of automated search and collection of data from web resources are described. Existing high-performance automated data collection systems based on the Scrapy framework have been studied. An overview of various libraries and frameworks for the implementation of web scraping systems is carried out. As part of this work, a high-performance and fault-tolerant distributed system for collecting data from web resources is described. The proposed approach to organizing computations is to use the Kubernetes container orchestration system to balance the load between cluster elements and centralize management. The Scrapy framework is used as the main means of implementing web scrapers. The result is a comparison of the data collection speed using a simple Scrapy-based application and the developed system, and also shows the results of increasing productivity by increasing the number of worker nodes.