title: ÐÑÑÐ»ÐµÐ´Ð¾Ð²Ð°Ð½Ð¸Ðµ Ð¸ ÑÐ°Ð·ÑÐ°Ð±Ð¾ÑÐºÐ° ÐºÐ¾Ð¼Ð±Ð¸Ð½Ð¸ÑÐ¾Ð²Ð°Ð½Ð½Ð¾Ð¹ Ð¼Ð½Ð¾Ð³Ð¾ÑÑÐºÐ°Ð²Ð½Ð¾Ð¹ ÑÐ²ÑÑÑÐ¾ÑÐ½Ð¾Ð¹ Ð½ÐµÐ¹ÑÐ¾ÑÐµÑÐµÐ²Ð¾Ð¹ Ð¼Ð¾Ð´ÐµÐ»Ð¸ Ð´Ð»Ñ ÑÐµÑÐµÐ½Ð¸Ñ Ð·Ð°Ð´Ð°ÑÐ¸ ÐºÐ»Ð°ÑÑÐ¸ÑÐ¸ÐºÐ°ÑÐ¸Ð¸ Ð¼ÑÐ»ÑÑÐ¸Ð¼Ð¾Ð´Ð°Ð»ÑÐ½Ð¾Ð³Ð¾ ÐºÐ¾Ð½ÑÐµÐ½ÑÐ°
title: Ð²ÑÐ¿ÑÑÐºÐ½Ð°Ñ ÐºÐ²Ð°Ð»Ð¸ÑÐ¸ÐºÐ°ÑÐ¸Ð¾Ð½Ð½Ð°Ñ ÑÐ°Ð±Ð¾ÑÐ° Ð±Ð°ÐºÐ°Ð»Ð°Ð²ÑÐ°
titleType: Subtitle
title: Research and development of a Ñombined multi-arm convolutional neural network model for multimodal content classification problem
Creators: name: Ð¢Ð¾ÑÐ¾Ð¿Ð¾Ð², ÐÐ»ÐµÐºÑÐµÐ¹
Publisher: name: Ð¡Ð°Ð½ÐºÑ-ÐÐµÑÐµÑÐ±ÑÑÐ³ÑÐºÐ¸Ð¹ Ð¿Ð¾Ð»Ð¸ÑÐµÑÐ½Ð¸ÑÐµÑÐºÐ¸Ð¹ ÑÐ½Ð¸Ð²ÐµÑÑÐ¸ÑÐµÑ ÐÐµÑÑÐ° ÐÐµÐ»Ð¸ÐºÐ¾Ð³Ð¾
Year: 2023
Subjects: ÐºÐ¾Ð¼Ð±Ð¸Ð½Ð¸ÑÐ¾Ð²Ð°Ð½Ð½ÑÐ¹ ÐºÐ»Ð°ÑÑÐ¸ÑÐ¸ÐºÐ°ÑÐ¾Ñ
ÑÐ¿ÐµÑÐ¸Ð°Ð»ÑÐ½ÑÐµ Ð´ÐµÑÐºÑÐ¸Ð¿ÑÐ¾ÑÑ Ð¸Ð·Ð¾Ð±ÑÐ°Ð¶ÐµÐ½Ð¸Ð¹
Ð²Ð¸Ð·ÑÐ°Ð»ÑÐ½ÑÐµ Ð¸ ÑÐµÐºÑÑÐ¾Ð²ÑÐµ Ð¿ÑÐ¸Ð·Ð½Ð°ÐºÐ¸
combined classifier
specialized image descriptors
visual and textual features

description: Ð Ð´Ð°Ð½Ð½Ð¾Ð¹ ÑÐ°Ð±Ð¾ÑÐµ ÑÐ°ÑÑÐ¼Ð°ÑÑÐ¸Ð²Ð°ÐµÑÑÑ Ð¿ÑÐ¾Ð±Ð»ÐµÐ¼Ð° ÐºÐ»Ð°ÑÑÐ¸ÑÐ¸ÐºÐ°ÑÐ¸Ð¸ ÑÐ¾ÑÐ¾Ð³ÑÐ°ÑÐ¸Ð¹ ÑÐ°ÑÐ°Ð´Ð¾Ð² ÐºÐ¾Ð¼Ð¼ÐµÑÑÐµÑÐºÐ¸Ñ Ð·Ð´Ð°Ð½Ð¸Ð¹ Ð¿Ð¾ ÑÐ¸Ð¿Ñ Ð¿ÑÐµÐ´Ð¾ÑÑÐ°Ð²Ð»ÑÐµÐ¼ÑÑ ÑÑÐ»ÑÐ³. ÐÐ±ÑÐµÐºÑÐ¾Ð¼ Ð¸ÑÑÐ»ÐµÐ´Ð¾Ð²Ð°Ð½Ð¸Ñ ÑÐ²Ð»ÑÐµÑÑÑ ÐºÐ¾Ð¼Ð±Ð¸Ð½Ð¸ÑÐ¾Ð²Ð°Ð½Ð½Ð°Ñ Ð¼Ð½Ð¾Ð³Ð¾ÑÑÐºÐ°Ð²Ð½Ð°Ñ Ð½ÐµÐ¹ÑÐ¾ÑÐµÑÐµÐ²Ð°Ñ Ð°ÑÑÐ¸ÑÐµÐºÑÑÑÐ° ÑÐ¾ ÑÐ¿ÐµÑÐ¸Ð°Ð»ÑÐ½ÑÐ¼Ð¸ Ð´ÐµÑÐºÑÐ¸Ð¿ÑÐ¾ÑÐ°Ð¼Ð¸ Ð¾Ð±Ð»Ð°ÑÑÐµÐ¹ Ð¸Ð·Ð¾Ð±ÑÐ°Ð¶ÐµÐ½Ð¸Ñ Ñ ÑÐµÐºÑÑÐ¾Ð¼. ÐÑÐµÐ´Ð¼ÐµÑÐ¾Ð¼ Ð¸ÑÑÐ»ÐµÐ´Ð¾Ð²Ð°Ð½Ð¸Ñ ÑÐ²Ð»ÑÐµÑÑÑ Ð¿ÑÐ¸Ð¼ÐµÐ½Ð¸Ð¼Ð¾ÑÑÑ Ð¸ ÑÑÑÐµÐºÑÐ¸Ð²Ð½Ð¾ÑÑÑ Ð´Ð°Ð½Ð½ÑÑ Ð¼Ð¾Ð´ÐµÐ»ÐµÐ¹ Ð´Ð»Ñ ÑÐµÑÐµÐ½Ð¸Ñ Ð·Ð°Ð´Ð°ÑÐ¸ ÐºÐ»Ð°ÑÑÐ¸ÑÐ¸ÐºÐ°ÑÐ¸Ð¸ ÑÐ¾ÑÐ¾Ð³ÑÐ°ÑÐ¸Ð¹ ÑÐ°ÑÐ°Ð´Ð¾Ð² ÐºÐ¾Ð¼Ð¼ÐµÑÑÐµÑÐºÐ¸Ñ Ð·Ð´Ð°Ð½Ð¸Ð¹ Ð¿Ð¾ ÑÐ¸Ð¿Ñ Ð¿ÑÐµÐ´Ð¾ÑÑÐ°Ð²Ð»ÑÐµÐ¼ÑÑ ÑÑÐ»ÑÐ³. Ð¦ÐµÐ»ÑÑ Ð¸ÑÑÐ»ÐµÐ´Ð¾Ð²Ð°Ð½Ð¸Ñ ÑÐ²Ð»ÑÐµÑÑÑ ÑÐ°Ð·ÑÐ°Ð±Ð¾ÑÐºÐ° ÐºÐ¾Ð¼Ð±Ð¸Ð½Ð¸ÑÐ¾Ð²Ð°Ð½Ð½Ð¾Ð¹ Ð¼Ð½Ð¾Ð³Ð¾ÑÑÐºÐ°Ð²Ð½Ð¾Ð¹ Ð½ÐµÐ¹ÑÐ¾ÑÐµÑÐµÐ²Ð¾Ð¹ Ð¼Ð¾Ð´ÐµÐ»Ð¸ Ð´Ð»Ñ ÑÐµÑÐµÐ½Ð¸Ñ Ð·Ð°Ð´Ð°ÑÐ¸ ÐºÐ»Ð°ÑÑÐ¸ÑÐ¸ÐºÐ°ÑÐ¸Ð¸ ÑÐ°ÑÐ°Ð´Ð¾Ð² ÐºÐ¾Ð¼Ð¼ÐµÑÑÐµÑÐºÐ¸Ñ Ð·Ð´Ð°Ð½Ð¸Ð¹ Ð¿Ð¾ ÑÐ¸Ð¿Ñ Ð¿ÑÐµÐ´Ð¾ÑÑÐ°Ð²Ð»ÑÐµÐ¼ÑÑ ÑÑÐ»ÑÐ³, Ð° ÑÐ°ÐºÐ¶Ðµ Ð¸ÑÑÐ»ÐµÐ´Ð¾Ð²Ð°Ð½Ð¸Ðµ ÐµÑ ÑÑÑÐµÐºÑÐ¸Ð²Ð½Ð¾ÑÑÐ¸. ÐÐ»Ñ Ð´Ð¾ÑÑÐ¸Ð¶ÐµÐ½Ð¸Ñ Ð¿Ð¾ÑÑÐ°Ð²Ð»ÐµÐ½Ð½Ð¾Ð¹ ÑÐµÐ»Ð¸ Ð¿ÑÐµÐ´Ð»Ð°Ð³Ð°ÐµÑÑÑ ÑÐ¿ÑÐ¾ÐµÐºÑÐ¸ÑÐ¾Ð²Ð°ÑÑ Ð¸ ÑÐµÐ°Ð»Ð¸Ð·Ð¾Ð²Ð°ÑÑ ÐºÐ¾Ð¼Ð±Ð¸Ð½Ð¸ÑÐ¾Ð²Ð°Ð½Ð½ÑÑ Ð¼Ð½Ð¾Ð³Ð¾ÑÑÐºÐ°Ð²Ð½ÑÑ Ð½ÐµÐ¹ÑÐ¾ÑÐµÑÐµÐ²ÑÑ Ð°ÑÑÐ¸ÑÐµÐºÑÑÑÑ, ÑÐ°Ð·ÑÐ°Ð±Ð¾ÑÐ°ÑÑ ÑÐ¿ÐµÑÐ¸Ð°Ð»ÑÐ½ÑÐµ Ð¼Ð¾Ð´ÑÐ»Ð¸ Ð² ÑÐ¾ÑÑÐ°Ð²Ðµ Ð°ÑÑÐ¸ÑÐµÐºÑÑÑÑ, Ð¿Ð¾Ð·Ð²Ð¾Ð»ÑÑÑÐ¸Ðµ Ð¸Ð·Ð²Ð»ÐµÐºÐ°ÑÑ Ð´ÐµÑÐºÑÐ¸Ð¿ÑÐ¾ÑÑ Ð¸Ð· ÑÑÐ°ÑÑÐºÐ¾Ð² Ð¸Ð·Ð¾Ð±ÑÐ°Ð¶ÐµÐ½Ð¸Ñ, Ð° ÑÐ°ÐºÐ¶Ðµ Ð¸ÑÑÐ»ÐµÐ´Ð¾Ð²Ð°ÑÑ ÑÑÑÐµÐºÑÐ¸Ð²Ð½Ð¾ÑÑÑ Ð¿ÑÐ¸Ð¼ÐµÐ½ÐµÐ½Ð¸Ñ Ð¿Ð¾Ð»ÑÑÐµÐ½Ð½ÑÑ Ð¼Ð¾Ð´ÑÐ»ÐµÐ¹ Ð² ÑÐ¾ÑÑÐ°Ð²Ðµ ÐºÐ¾Ð¼Ð±Ð¸Ð½Ð¸ÑÐ¾Ð²Ð°Ð½Ð½Ð¾Ð³Ð¾ Ð¼Ð½Ð¾Ð³Ð¾ÑÑÐºÐ°Ð²Ð½Ð¾Ð³Ð¾ Ð½ÐµÐ¹ÑÐ¾ÑÐµÑÐµÐ²Ð¾Ð³Ð¾ ÐºÐ»Ð°ÑÑÐ¸ÑÐ¸ÐºÐ°ÑÐ¾ÑÐ°. Ð ÐºÐ°ÑÐµÑÑÐ²Ðµ ÑÐµÑÐµÐ½Ð¸Ñ Ð¿ÑÐµÐ´Ð»Ð°Ð³Ð°ÐµÑÑÑ Ð°ÑÑÐ¸ÑÐµÐºÑÑÑÐ° ÐºÐ¾Ð¼Ð±Ð¸Ð½Ð¸ÑÐ¾Ð²Ð°Ð½Ð½Ð¾Ð³Ð¾ ÐºÐ»Ð°ÑÑÐ¸ÑÐ¸ÐºÐ°ÑÐ¾ÑÐ°, Ð° ÑÐ°ÐºÐ¶Ðµ ÑÐ°Ð·Ð»Ð¸ÑÐ½ÑÐµ ÑÐ¸Ð¿Ñ Ð´ÐµÑÐºÑÐ¸Ð¿ÑÐ¾ÑÐ¾Ð²: Ð½Ð° Ð¾ÑÐ½Ð¾Ð²Ðµ ÑÐ²ÑÑÑÐ¾ÑÐ½ÑÑ Ð½ÐµÐ¹ÑÐ¾Ð½Ð½ÑÑ ÑÐµÑÐµÐ¹ Ð¸ Ð½Ð° Ð¾ÑÐ½Ð¾Ð²Ðµ Ð¾Ð±ÑÑÐ°ÐµÐ¼ÑÑ Ð¿Ð°ÑÐ°Ð¼ÐµÑÑÐ¸Ð·Ð¾Ð²Ð°Ð½Ð½ÑÑ ÑÑÑÐ°ÑÐµÐ³Ð¸Ð¹ Ð´Ð²Ð¸Ð¶ÐµÐ½Ð¸Ñ Ð°Ð³ÐµÐ½ÑÐ¾Ð². ÐÐ»Ñ ÑÑÐ°Ð²Ð½ÐµÐ½Ð¸Ñ ÑÐ°Ð·ÑÐ°Ð±Ð¾ÑÐ°Ð½Ð½ÑÑ Ð¿Ð¾Ð´ÑÐ¾Ð´Ð¾Ð² Ð¸ Ð¿ÑÐ¾Ð²ÐµÑÐºÐ¸ Ð¸Ñ ÑÑÑÐµÐºÑÐ¸Ð²Ð½Ð¾ÑÑÐ¸ Ð¸ÑÐ¿Ð¾Ð»ÑÐ·ÑÑÑÑÑ Ð´Ð²Ð° Ð½Ð°Ð±Ð¾ÑÐ° Ð´Ð°Ð½Ð½ÑÑ ÑÐ¾ÑÐ¾Ð³ÑÐ°ÑÐ¸Ð¹ ÑÐ°ÑÐ°Ð´Ð¾Ð² ÐºÐ¾Ð¼Ð¼ÐµÑÑÐµÑÐºÐ¸Ñ Ð·Ð´Ð°Ð½Ð¸Ð¹, ÑÐ³ÑÑÐ¿Ð¿Ð¸ÑÐ¾Ð²Ð°Ð½Ð½ÑÑ Ð¿Ð¾ ÑÐ¸Ð¿Ñ Ð¿ÑÐµÐ´ÑÑÐ°Ð²Ð»ÐµÐ½Ð½Ð¾Ð³Ð¾ Ð±Ð¸Ð·Ð½ÐµÑÐ°. Ð ÑÐµÐ·ÑÐ»ÑÑÐ°ÑÐµ ÑÐ°Ð·ÑÐ°Ð±Ð¾ÑÐ°Ð½Ð° ÐºÐ¾Ð¼Ð±Ð¸Ð½Ð¸ÑÐ¾Ð²Ð°Ð½Ð½Ð°Ñ Ð¼Ð½Ð¾Ð³Ð¾ÑÑÐºÐ°Ð²Ð½Ð°Ñ ÑÐ²ÑÑÑÐ¾ÑÐ½Ð°Ñ Ð½ÐµÐ¹ÑÐ¾ÑÐµÑÐµÐ²Ð°Ñ Ð¼Ð¾Ð´ÐµÐ»Ñ, ÐºÐ¾ÑÐ¾ÑÐ°Ñ Ð¿ÑÐ¾Ð´ÐµÐ¼Ð¾Ð½ÑÑÑÐ¸ÑÐ¾Ð²Ð°Ð»Ð° Ð»ÑÑÑÐµÐµ ÐºÐ°ÑÐµÑÑÐ²Ð¾ Ð¿Ð¾ Ð¼ÐµÑÑÐ¸ÐºÐµ F1. ÐÐ°Ð»ÑÐ½ÐµÐ¹ÑÐ¸Ðµ Ð¸ÑÑÐ»ÐµÐ´Ð¾Ð²Ð°Ð½Ð¸Ñ Ð¼Ð¾Ð³ÑÑ Ð±ÑÑÑ Ð½Ð°Ð¿ÑÐ°Ð²Ð»ÐµÐ½Ñ Ð½Ð° Ð¸Ð·ÑÑÐµÐ½Ð¸Ðµ Ð½Ð¾Ð²ÑÑ Ð´ÐµÑÐºÑÐ¸Ð¿ÑÐ¾ÑÐ¾Ð² Ñ ÑÐµÐ»ÑÑ Ð´Ð°Ð»ÑÐ½ÐµÐ¹ÑÐµÐ³Ð¾ ÑÐ»ÑÑÑÐµÐ½Ð¸Ñ Ð´Ð¾ÑÑÐ¸Ð³Ð½ÑÑÑÑ ÑÐµÐ·ÑÐ»ÑÑÐ°ÑÐ¾Ð².
description: This graduate qualification work deals with the problem of classifying images of commercial building facades according to the type of services provided. The object of the research is a combined multi-arm neural network architecture with special descriptors of image areas with text. The subject of the research is the applicability and effectiveness of these models for solving the problem of classifying photos of commercial building facades by type of services provided. The aim of the graduate qualification work is to develop a combined multi-arm neural network model to solve the problem of classifying the facades of commercial buildings by type of services, and research its effectiveness. To achieve this goal, it is proposed to design and implement a combined multi-arm neural network architecture, develop special modules in the architecture, allowing to extract descriptors from image sections, and to study the effectiveness of these modules in the combined multi-arm neural network classifier. As a solution, the architecture of combined classifier is proposed, as well as different types of descriptors: based on convolutional neural networks and on the basis of trainable parametrized strategies of movement of agents. To compare the developed approaches and test their effectiveness, two sets of commercial building images, grouped by the type of business represented, are used. As a result, a combined multi-arm convolutional neural network model is developed, which has demonstrated the best performance on the F1 metric. Further research can be aimed at exploring new descriptors to further improve the results achieved.