title: Ð£Ð²ÐµÐ»Ð¸ÑÐµÐ½Ð¸Ðµ Ð¾Ð±ÑÐµÐ¼Ð° Ñ ÑÐ°Ð½Ð¸Ð¼ÑÑ Ð´Ð°Ð½Ð½ÑÑ Ð¸ Ð¿Ð¾ÑÑÑÐ¾ÐµÐ½Ð¸Ðµ Ð¿Ð¾Ð»Ð½Ð¾ÑÐµÐ½Ð½ÑÑ Ð¸Ð½Ð´ÐµÐºÑÐ¾Ð²
title: Ð²ÑÐ¿ÑÑÐºÐ½Ð°Ñ ÐºÐ²Ð°Ð»Ð¸ÑÐ¸ÐºÐ°ÑÐ¸Ð¾Ð½Ð½Ð°Ñ ÑÐ°Ð±Ð¾ÑÐ° Ð±Ð°ÐºÐ°Ð»Ð°Ð²ÑÐ°
titleType: Subtitle
title: Increasing the amount of stored data and building full-fledged indexes
Creators: name: ÐÑÐ±ÐºÐ¸Ð½, ÐÐ»Ð°Ð´Ð¸ÑÐ»Ð°Ð²
Publisher: name: Ð¡Ð°Ð½ÐºÑ-ÐÐµÑÐµÑÐ±ÑÑÐ³ÑÐºÐ¸Ð¹ Ð¿Ð¾Ð»Ð¸ÑÐµÑÐ½Ð¸ÑÐµÑÐºÐ¸Ð¹ ÑÐ½Ð¸Ð²ÐµÑÑÐ¸ÑÐµÑ ÐÐµÑÑÐ° ÐÐµÐ»Ð¸ÐºÐ¾Ð³Ð¾
Year: 2022
Subjects: Ð¸Ð½Ð´ÐµÐºÑ
Java
Apache Lucene
Apache Cassandra
Ð°Ð½ÑÐ¸ÑÐ¿Ð°Ð¼
index
antispam

description: ÐÐ°Ð½Ð½Ð°Ñ ÑÐ°Ð±Ð¾ÑÐ° Ð½Ð°Ð¿ÑÐ°Ð²Ð»ÐµÐ½Ð° Ð½Ð° Ð¸ÑÑÐ»ÐµÐ´Ð¾Ð²Ð°Ð½Ð¸Ðµ ÑÐ¿Ð¾ÑÐ¾Ð±Ð¾Ð² ÑÐ²ÐµÐ»Ð¸ÑÐµÐ½Ð¸Ñ ÑÑÐ°Ð½Ð¸Ð¼ÑÑ Ð´Ð°Ð½Ð½ÑÑ Ð² ÑÐµÑÐ²Ð¸ÑÐµ ÑÐ¾ÑÑÐ°Ð²Ð»ÐµÐ½Ð¸Ñ ÑÐ½Ð¸Ð²ÐµÑÑÐ°Ð»ÑÐ½ÑÑ Ð¾ÑÑÐµÑÐ¾Ð² Ð´Ð»Ñ Ð°Ð½ÑÐ¸ÑÐ¿Ð°Ð¼Ð° ÐÐ´Ð½Ð¾ÐºÐ»Ð°ÑÑÐ½Ð¸ÐºÐ¾Ð² Ð¸ Ð½Ð° ÑÐ°Ð·ÑÐ°Ð±Ð¾ÑÐºÑ Ð½Ð¾Ð²Ð¾Ð¹ Ð°ÑÑÐ¸ÑÐµÐºÑÑÑÑ ÑÐµÑÐ²Ð¸ÑÐ° Ð´Ð»Ñ ÑÐ²ÐµÐ»Ð¸ÑÐµÐ½Ð¸Ñ Ð¾Ð±ÑÐµÐ¼Ð° ÑÑÐ°Ð½Ð¸Ð¼ÑÑ Ð´Ð°Ð½Ð½ÑÑ Ð¸ Ð¿Ð¾ÑÑÑÐ¾ÐµÐ½Ð¸Ñ Ð¿Ð¾Ð»Ð½Ð¾ÑÐµÐ½Ð½ÑÑ Ð¸Ð½Ð´ÐµÐºÑÐ¾Ð² Ð´Ð»Ñ Ð±ÑÑÑÑÐ¾Ð³Ð¾ Ð´Ð¾ÑÑÑÐ¿Ð° Ðº ÑÑÐ¸Ð¼ Ð´Ð°Ð½Ð½ÑÐ¼. ÐÐ°Ð´Ð°ÑÐ¸, ÐºÐ¾ÑÐ¾ÑÑÐµ ÑÐµÑÐ°Ð»Ð¸ÑÑ Ð² ÑÐ¾Ð´Ðµ Ð¸ÑÑÐ»ÐµÐ´Ð¾Ð²Ð°Ð½Ð¸Ñ Ð¸ ÑÐ°Ð·ÑÐ°Ð±Ð¾ÑÐºÐ¸: ÐÐ·ÑÑÐµÐ½Ð¸Ðµ Ð°ÑÑÐ¸ÑÐµÐºÑÑÑÑ Ð¸ Ð¾ÑÐ¾Ð±ÐµÐ½Ð½Ð¾ÑÑÐµÐ¹ ÑÐµÑÐ²Ð¸ÑÐ° Ð´Ð»Ñ ÑÐ¾ÑÑÐ°Ð²Ð»ÐµÐ½Ð¸Ñ ÑÐ½Ð¸Ð²ÐµÑÑÐ°Ð»ÑÐ½ÑÑ Ð¾ÑÑÐµÑÐ¾Ð² Ð°Ð½ÑÐ¸ÑÐ¿Ð°Ð¼Ð° ÐÐ´Ð½Ð¾ÐºÐ»Ð°ÑÑÐ½Ð¸ÐºÐ¾Ð². ÐÑÑÐ»ÐµÐ´Ð¾Ð²Ð°Ð½Ð¸Ðµ Ð¸Ð½ÑÑÑÑÐ¼ÐµÐ½ÑÐ¾Ð² Ð´Ð»Ñ Ð¿Ð¾ÑÑÑÐ¾ÐµÐ½Ð¸Ñ Ð¸Ð½Ð´ÐµÐºÑÐ¾Ð² Ð¸ Ð²ÑÐ±Ð¾Ñ Ð½Ð°Ð¸Ð±Ð¾Ð»ÐµÐµ Ð¿Ð¾Ð´ÑÐ¾Ð´ÑÑÐµÐ³Ð¾ Ð¸Ð· Ð½Ð¸Ñ. ÐÑÐ±Ð¾Ñ ÑÐ¾ÑÐ¼Ð°ÑÐ° ÑÑÐ°Ð½ÐµÐ½Ð¸Ñ Ð°ÑÑÐ¸Ð²Ð½ÑÑ Ð·Ð°Ð¿Ð¸ÑÐµÐ¹ Ð² Ð±Ð°Ð·Ðµ Ð´Ð°Ð½Ð½ÑÑ. Ð Ð°Ð·ÑÐ°Ð±Ð¾ÑÐºÐ° Ð½Ð¾Ð²Ð¾Ð¹ Ð°ÑÑÐ¸ÑÐµÐºÑÑÑÑ ÑÐµÑÐ²Ð¸ÑÐ°. Ð Ð°Ð·ÑÐ°Ð±Ð¾ÑÐºÐ° Ð¿Ð°ÐºÐµÑÐ° ÐºÐ»Ð°ÑÑÐ¾Ð² Ð´Ð»Ñ Ð¿Ð¾ÑÑÑÐ¾ÐµÐ½Ð¸Ñ Ð¿Ð¾Ð»Ð½Ð¾ÑÐµÐ½Ð½ÑÑ Ð¸Ð½Ð´ÐµÐºÑÐ¾Ð² Ð¸ Ð¿Ð¾Ð¸ÑÐºÐ° Ð¿Ð¾ Ð½Ð¸Ð¼ Ð¸ Ð¿Ð°ÐºÐµÑÐ° ÐºÐ»Ð°ÑÑÐ¾Ð² Ð´Ð»Ñ ÑÐ¾Ð·Ð´Ð°Ð½Ð¸Ñ Ð¸ Ð²Ð·Ð°Ð¸Ð¼Ð¾Ð´ÐµÐ¹ÑÑÐ²Ð¸Ñ Ñ Ð½Ð¾Ð²ÑÐ¼ ÑÐ¾ÑÐ¼Ð°ÑÐ¾Ð¼ ÑÑÐ°Ð½ÐµÐ½Ð¸Ñ Ð´Ð°Ð½Ð½ÑÑ. Ð Ð°Ð±Ð¾ÑÐ° Ð¿ÑÐ¾Ð²Ð¾Ð´Ð¸Ð»Ð°ÑÑ Ð½Ð° Ð±Ð°Ð·Ðµ ÑÐµÑÐ²Ð¸ÑÐ° Ð¿Ð¾ ÑÐ¾ÑÑÐ°Ð²Ð»ÐµÐ½Ð¸Ñ ÑÐ½Ð¸Ð²ÐµÑÑÐ°Ð»ÑÐ½ÑÑ Ð¾ÑÑÐµÑÐ¾Ð² Ð´Ð»Ñ ÑÐ¿Ð°Ð¼-Ð°Ð½Ð°Ð»Ð¸ÑÐ¸ÐºÐ¾Ð² Ð¸ ÑÐ°Ð·ÑÐ°Ð±Ð¾ÑÑÐ¸ÐºÐ¾Ð² ÑÐ¾ÑÐ¸Ð°Ð»ÑÐ½Ð¾Ð¹ ÑÐµÑÐ¸ ÐÐ´Ð½Ð¾ÐºÐ»Ð°ÑÑÐ½Ð¸ÐºÐ¸, Ð³Ð´Ðµ Ð½Ð° Ð¾ÑÐ½Ð¾Ð²Ð°Ð½Ð¸Ð¸ Ð¸ÑÑÐ¾Ð´Ð½Ð¾Ð¹ ÐºÐ¾Ð´Ð¾Ð²Ð¾Ð¹ Ð±Ð°Ð·Ñ Ð±ÑÐ»Ð¸ Ð¸Ð·ÑÑÐµÐ½Ñ ÑÐ¿ÐµÑÐ¸ÑÐ¸ÑÐµÑÐºÐ¸Ðµ Ð¾ÑÐ¾Ð±ÐµÐ½Ð½Ð¾ÑÑÐ¸ Ð¿ÑÐ¾Ð³ÑÐ°Ð¼Ð¼Ð½Ð¾Ð³Ð¾ Ð¿ÑÐ¾Ð´ÑÐºÑÐ°, ÑÐ¾ÑÑÐ°Ð²Ð»ÐµÐ½Ñ ÑÑÐµÐ¼Ñ ÑÑÐ°ÑÐ¾Ð¹ Ð¸ Ð½Ð¾Ð²Ð¾Ð¹ Ð°ÑÑÐ¸ÑÐµÐºÑÑÑÑ ÑÐµÑÐ²Ð¸ÑÐ° Ð¸ ÑÐ¾Ð±ÑÐ°Ð½Ñ ÑÐ¾Ð¾ÑÐ²ÐµÑÑÑÐ²ÑÑÑÐ¸Ðµ ÑÑÐ°ÑÐ¸ÑÑÐ¸ÑÐµÑÐºÐ¸Ðµ Ð´Ð°Ð½Ð½ÑÐµ, ÐºÐ¾ÑÐ¾ÑÑÐµ Ð¿Ð¾Ð·Ð²Ð¾Ð»Ð¸Ð»Ð¸ Ð²ÑÐ±ÑÐ°ÑÑ Ð¸Ð½ÑÑÑÑÐ¼ÐµÐ½Ñ Ð´Ð»Ñ Ð¿Ð¾ÑÑÑÐ¾ÐµÐ½Ð¸Ñ Ð¸Ð½Ð´ÐµÐºÑÐ¾Ð². Ð ÐµÐ·ÑÐ»ÑÑÐ°ÑÐ¾Ð¼ Ð´Ð°Ð½Ð½Ð¾Ð¹ ÑÐ°Ð±Ð¾ÑÑ ÑÐ²Ð»ÑÐµÑÑÑ Ð½Ð¾Ð²Ð°Ñ Ð°ÑÑÐ¸ÑÐµÐºÑÑÑÐ° ÑÐµÑÐ²Ð¸ÑÐ° Ð´Ð»Ñ ÑÐ¾ÑÑÐ°Ð²Ð»ÐµÐ½Ð¸Ñ ÑÐ½Ð¸Ð²ÐµÑÑÐ°Ð»ÑÐ½ÑÑ Ð¾ÑÑÐµÑÐ¾Ð² Ð°Ð½ÑÐ¸ÑÐ¿Ð°Ð¼Ð° ÐÐ´Ð½Ð¾ÐºÐ»Ð°ÑÑÐ½Ð¸ÐºÐ¾Ð² Ð¸ Ð¿Ð°ÐºÐµÑÑ ÐºÐ»Ð°ÑÑÐ¾Ð² Ð¿Ð¾ÑÑÑÐ¾ÐµÐ½Ð¸Ñ Ð¸Ð½Ð´ÐµÐºÑÐ¾Ð² Ð¸ Ð¿Ð¾Ð¸ÑÐºÐ° Ð¿Ð¾ Ð½Ð¸Ð¼, Ð° ÑÐ°ÐºÐ¶Ðµ Ð´Ð»Ñ ÑÐ¾Ð·Ð´Ð°Ð½Ð¸Ñ Ð¸ Ð²Ð·Ð°Ð¸Ð¼Ð¾Ð´ÐµÐ¹ÑÑÐ²Ð¸Ñ Ñ Ð±Ð°Ð·Ð¾Ð¹ Ð´Ð°Ð½Ð½ÑÑ, ÐºÐ¾ÑÐ¾ÑÑÐµ Ð²Ð·Ð°Ð¸Ð¼Ð¾Ð´ÐµÐ¹ÑÑÐ²ÑÑÑ ÑÐ¾ ÑÑÐ°ÑÐ¾Ð¹ Ð°ÑÑÐ¸ÑÐµÐºÑÑÑÐ¾Ð¹, Ð¿ÐµÑÐ¸Ð¾Ð´Ð¸ÑÐµÑÐºÐ¸ Ð°ÑÑÐ¸Ð²Ð¸ÑÑÑ Ð´Ð°Ð½Ð½ÑÐµ Ð¸Ð· Ð¾Ð¿ÐµÑÐ°ÑÐ¸Ð²Ð½Ð¾Ð¹ Ð¿Ð°Ð¼ÑÑÐ¸ Ð½Ð° Ð´Ð¸ÑÐº. Ð¡Ð¾Ð·Ð´Ð°Ð½Ð½Ð°Ñ Ð°ÑÑÐ¸ÑÐµÐºÑÑÑÐ° Ð¸ Ð¿ÑÐ¾Ð³ÑÐ°Ð¼Ð¼Ð½ÑÐµ ÐºÐ¾Ð¼Ð¿Ð¾Ð½ÐµÐ½ÑÑ Ð¾ÑÐ²ÐµÑÐ°ÑÑ ÑÐ¾Ð²ÑÐµÐ¼ÐµÐ½Ð½ÑÐ¼ Ð·Ð°Ð¿ÑÐ¾ÑÐ°Ð¼ Ð¸ ÑÑÐµÐ±Ð¾Ð²Ð°Ð½Ð¸ÑÐ¼ ÑÐ°Ð·ÑÐ°Ð±Ð¾ÑÑÐ¸ÐºÐ¾Ð² Ð¸ ÑÐ¿Ð°Ð¼-Ð°Ð½Ð°Ð»Ð¸ÑÐ¸ÐºÐ¾Ð², Ð¿Ð¾ÑÐºÐ¾Ð»ÑÐºÑ Ð¾Ð½Ð¸ Ð¿Ð¾Ð·Ð²Ð¾Ð»ÑÑÑ ÑÑÐ°Ð½Ð¸ÑÑ Ð±Ð¾Ð»ÑÑÐµÐµ ÐºÐ¾Ð»Ð¸ÑÐµÑÑÐ²Ð¾ ÑÑÐ°ÑÐ¸ÑÑÐ¸ÑÐµÑÐºÐ¸ Ð´Ð°Ð½Ð½ÑÑ, ÑÐ¾ ÐµÑÑÑ Ð¸Ð¼ÐµÑÑ Ð±Ð¾Ð»ÑÑÑÑ Ð¸ÑÑÐ¾ÑÐ¸Ñ Ð¾ Ð´ÐµÐ¹ÑÑÐ²Ð¸Ñ ÑÐ¿Ð°Ð¼ÐµÑÐ¾Ð² Ð¸ Ð¾Ð±ÑÑÐ½ÑÑ Ð¿Ð¾Ð»ÑÐ·Ð¾Ð²Ð°ÑÐµÐ»ÐµÐ¹.
description: The given work is aimed at studying the increase in the volume of stored data in the Odnoklassniki universal antispam reporting service and developing new service architecture to increase the volume of stored data and build full-fledged indexes for quick access to this data. The goals that were solved during research and development: Studying the architecture and features of the Odnoklassniki universal antispam reporting service. Research tools for building indexes and choosing the most appropriate one.Selecting the format for storing archived records in the database. Development of a new service architecture. Development of a class package for building full-fledged indexes and searching through them, and a class package for creating and interacting with a new data storage format. The work was carried out based on the Odnoklassniki universal antispam reporting service for spam analysts and developers, where, based on the original code base, the specific features of the software product were studied, diagrams of the old and new architecture of the service were developed, and relevant statistical data were collected, which made it possible to select the appropriate tool for building indexes. The result of this work is a new service architecture for the Odnoklassniki universal antispam reporting service and class packages for building indexes and searching through them, as well as for creating and interacting with a database that interact with the old architecture, periodically archiving data from RAM on disk. The created architecture and software components meet the modern needs and requirements of developers and spam analysts, since it allows you to store a larger amount of statistical data, that is, to have a greater history of the actions of spammers and ordinary users.