title: Ð Ð°Ð·ÑÐ°Ð±Ð¾ÑÐºÐ° Ð¸ Ð¿ÑÐ¸Ð¼ÐµÐ½ÐµÐ½Ð¸Ðµ Ð°Ð»Ð³Ð¾ÑÐ¸ÑÐ¼Ð¾Ð² Ð´ÐµÐ´ÑÐ¿Ð»Ð¸ÐºÐ°ÑÐ¸Ð¸ Ð´Ð»Ñ Ð¿Ð¾Ð²ÑÑÐµÐ½Ð¸Ñ ÐºÐ°ÑÐµÑÑÐ²Ð° Ð¼Ð°ÑÑÐµÑ-Ð´Ð°Ð½Ð½ÑÑ Ð² ÐºÐ¾ÑÐ¿Ð¾ÑÐ°ÑÐ¸Ð²Ð½ÑÑ Ð¸Ð½ÑÐ¾ÑÐ¼Ð°ÑÐ¸Ð¾Ð½Ð½ÑÑ ÑÐ¸ÑÑÐµÐ¼Ð°Ñ
title: Ð²ÑÐ¿ÑÑÐºÐ½Ð°Ñ ÐºÐ²Ð°Ð»Ð¸ÑÐ¸ÐºÐ°ÑÐ¸Ð¾Ð½Ð½Ð°Ñ ÑÐ°Ð±Ð¾ÑÐ° Ð¼Ð°Ð³Ð¸ÑÑÑÐ°
titleType: Subtitle
title: Development and application of deduplication algorithms to improve the quality of master data in corporate information systems
Creators: name: ÐÐ¾ÑÐ°Ð¿Ð¾Ð²Ð°, ÐÐ°ÑÐ¸Ñ
Publisher: name: Ð¡Ð°Ð½ÐºÑ-ÐÐµÑÐµÑÐ±ÑÑÐ³ÑÐºÐ¸Ð¹ Ð¿Ð¾Ð»Ð¸ÑÐµÑÐ½Ð¸ÑÐµÑÐºÐ¸Ð¹ ÑÐ½Ð¸Ð²ÐµÑÑÐ¸ÑÐµÑ ÐÐµÑÑÐ° ÐÐµÐ»Ð¸ÐºÐ¾Ð³Ð¾
Year: 2024
Subjects: ÐÐ½ÑÐ¾ÑÐ¼Ð°ÑÐ¸Ð¾Ð½Ð½ÑÐµ ÑÐ¸ÑÑÐµÐ¼Ñ
Ð´ÐµÐ´ÑÐ¿Ð»Ð¸ÐºÐ°ÑÐ¸Ñ
Ð¼ÐµÑÐ¾Ð´Ñ Ð½ÐµÑÐµÑÐºÐ¾Ð³Ð¾ Ð¿Ð¾Ð¸ÑÐºÐ°
Ð½Ð¾ÑÐ¼Ð°Ð»Ð¸Ð·Ð°ÑÐ¸Ñ
deduplication
fuzzy search methods
normalization

description: ÐÐ°Ð½Ð½Ð°Ñ ÑÐ°Ð±Ð¾ÑÐ° Ð¿Ð¾ÑÐ²ÑÑÐµÐ½Ð° Ð°Ð½Ð°Ð»Ð¸Ð·Ñ Ð¸ Ð¿ÑÐ¸Ð¼ÐµÐ½ÐµÐ½Ð¸Ñ Ð°Ð»Ð³Ð¾ÑÐ¸ÑÐ¼Ð¾Ð² Ð´ÐµÐ´ÑÐ¿Ð»Ð¸ÐºÐ°ÑÐ¸Ð¸ Ð´Ð°Ð½Ð½ÑÑ Ð½Ð° Ð¿ÑÐ¸Ð¼ÐµÑÐµ MDM-ÑÐ¸ÑÑÐµÐ¼Ñ. ÐÐ°Ð´Ð°ÑÐ¸, ÐºÐ¾ÑÐ¾ÑÑÐµ ÑÐµÑÐ°Ð»Ð¸ÑÑ Ð² ÑÐ¾Ð´Ðµ Ð¸ÑÑÐ»ÐµÐ´Ð¾Ð²Ð°Ð½Ð¸Ñ: 1. ÐÐ¿ÑÐµÐ´ÐµÐ»Ð¸ÑÑ Ð¾ÑÐ½Ð¾Ð²Ð½ÑÐµ Ð±Ð¸Ð·Ð½ÐµÑ-Ð¿Ð¾Ð»Ñ; 2. ÐÑÑÐ»ÐµÐ´Ð¾Ð²Ð°ÑÑ ÑÐ¿Ð¾ÑÐ¾Ð±Ñ Ð½Ð¾ÑÐ¼Ð°Ð»Ð¸Ð·Ð°ÑÐ¸Ð¸ Ð´Ð°Ð½Ð½ÑÑ; 3. ÐÑÑÐ»ÐµÐ´Ð¾Ð²Ð°ÑÑ Ð°Ð»Ð³Ð¾ÑÐ¸ÑÐ¼Ñ Ð´Ð»Ñ Ð¿Ð¾Ð¸ÑÐºÐ° Ð´ÑÐ±Ð»Ð¸ÐºÐ°ÑÐ¾Ð² Ð´Ð°Ð½Ð½ÑÑ; 4. ÐÑÐ±ÑÐ°ÑÑ Ð½Ð°Ð¸Ð±Ð¾Ð»ÐµÐµ Ð¿Ð¾Ð´ÑÐ¾Ð´ÑÑÐ¸Ðµ Ð°Ð»Ð³Ð¾ÑÐ¸ÑÐ¼Ñ Ð¿Ð¾Ð¸ÑÐºÐ° Ð´ÑÐ±Ð»Ð¸ÐºÐ°ÑÐ¾Ð²; 5. Ð Ð°Ð·ÑÐ°Ð±Ð¾ÑÐ°ÑÑ ÑÐµÑÐ²Ð¸Ñ Ð´ÐµÐ´ÑÐ¿Ð»Ð¸ÐºÐ°ÑÐ¸Ð¸; 6. ÐÐ¿ÑÐ¾Ð±Ð¸ÑÐ¾Ð²Ð°ÑÑ ÑÐµÑÐ²Ð¸Ñ Ð´ÐµÐ´ÑÐ¿Ð»Ð¸ÐºÐ°ÑÐ¸Ð¸ Ð½Ð° ÑÐµÑÑÐ¾Ð²ÑÑ Ð´Ð°Ð½Ð½ÑÑ. Ð ÐºÐ°ÑÐµÑÑÐ²Ðµ Ð¾ÑÐ½Ð¾Ð²Ð½ÑÑ Ð±Ð¸Ð·Ð½ÐµÑ-Ð¿Ð¾Ð»ÐµÐ¹ Ð²ÑÐ±ÑÐ°Ð½Ñ Ð¾ÑÐ½Ð¾Ð²Ð½ÑÐµ Ð´Ð°Ð½Ð½ÑÐµ, ÐºÐ¾ÑÐ¾ÑÑÐ¼Ð¸ Ð¾Ð¿ÐµÑÐ¸ÑÑÑÑ Ð² ÑÐ¸ÑÑÐµÐ¼Ð°Ñ MDM. ÐÐ°Ð¿ÑÐ¸Ð¼ÐµÑ, Ð¤ÐÐ, ÑÐ¸ÑÑÐ¾Ð²Ð¾Ðµ Ð¿Ð¾Ð»Ðµ Ð¸ Ð´ÑÑÐ³Ð¸Ðµ. ÐÐ»Ñ ÐºÐ°Ð¶Ð´Ð¾Ð³Ð¾ Ð¸Ð· Ð½Ð¸Ñ ÑÐ°Ð·ÑÐ°Ð±Ð¾ÑÐ°Ð½Ñ Ð¿ÑÐ°Ð²Ð¸Ð»Ð° Ð½Ð¾ÑÐ¼Ð°Ð»Ð¸Ð·Ð°ÑÐ¸Ð¸. ÐÐ»Ñ Ð¿Ð¾Ð¸ÑÐºÐ° Ð´ÑÐ±Ð»Ð¸ÐºÐ°ÑÐ¾Ð² Ð²ÑÐ±ÑÐ°Ð½ ÐºÐ»Ð°ÑÑ Ð¼ÐµÑÐ¾Ð´Ð¾Ð² Ð½ÐµÑÐµÑÐºÐ¾Ð³Ð¾ ÑÑÐ°Ð²Ð½ÐµÐ½Ð¸Ñ ÑÑÑÐ¾Ðº, Ð¾ÑÐ¾Ð±ÑÐ°Ð½Ñ 3 Ð¼ÐµÑÐ¾Ð´Ð°, Ð° Ð¸Ð¼ÐµÐ½Ð½Ð¾ ÐÐµÐ²ÐµÐ½ÑÑÐµÐ¹Ð½Ð°, ÐÐ°Ð¼ÐµÑÐ°Ñ-ÐÐµÐ²ÐµÐ½ÑÑÐµÐ¹Ð½Ð° Ð¸ ÐÐ¶Ð°ÑÐ¾-ÐÐ¸Ð½ÐºÐ»ÐµÑÐ°. ÐÑÐ±ÑÐ°Ð½Ð½ÑÐµ Ð¼ÐµÑÐ¾Ð´Ñ ÑÐ°Ð·ÑÐ°Ð±Ð¾ÑÐ°Ð½Ñ Ð¸ Ð¿ÑÐ¸Ð¼ÐµÐ½ÐµÐ½Ñ. Ð ÑÐµÐ·ÑÐ»ÑÑÐ°ÑÐµ Ð¿ÑÐ¾Ð²ÐµÐ´ÐµÐ½Ð½Ð¾Ð¹ ÑÐ°Ð±Ð¾ÑÑ Ð´Ð»Ñ ÐºÐ°Ð¶Ð´Ð¾Ð³Ð¾ Ð¸Ð· Ð±Ð¸Ð·Ð½ÐµÑ-Ð¿Ð¾Ð»ÐµÐ¹ Ð½Ð° Ð¾ÑÐ½Ð¾Ð²Ð°Ð½Ð¸Ð¸ ÑÐ¾ÑÐ½Ð¾ÑÑÐ¸ Ð¾Ð¿ÑÐµÐ´ÐµÐ»ÐµÐ½Ð¸Ñ Ð´ÑÐ±Ð»Ð¸ÐºÐ°ÑÐ¾Ð² Ð¸ Ð²ÑÐµÐ¼ÐµÐ½Ð¸ Ð²ÑÐ¿Ð¾Ð»Ð½ÐµÐ½Ð¸Ñ, Ð²ÑÐ±ÑÐ°Ð½ Ð½Ð°Ð¸Ð±Ð¾Ð»ÐµÐµ Ð¿Ð¾Ð´ÑÐ¾Ð´ÑÑÐ¸Ð¹ Ð¼ÐµÑÐ¾Ð´. Ð¢Ð°ÐºÐ¶Ðµ ÑÐµÐ°Ð»Ð¸Ð·Ð¾Ð²Ð°Ð½ ÑÐ¾Ð±ÑÑÐ²ÐµÐ½Ð½ÑÐ¹ ÑÐµÑÐ²Ð¸Ñ Ð½Ð° Python Ñ Ð¸ÑÐ¿Ð¾Ð»ÑÐ·Ð¾Ð²Ð°Ð½Ð¸ÐµÐ¼ Flask Ð½Ð° Ð¾ÑÐ½Ð¾Ð²Ð°Ð½Ð¸Ð¸ Ð²Ð·Ð°Ð¸Ð¼Ð¾Ð´ÐµÐ¹ÑÑÐ²Ð¸Ñ REST API, ÐºÐ¾ÑÐ¾ÑÑÐ¹ Ð² Ð´Ð°Ð»ÑÐ½ÐµÐ¹ÑÐµÐ¼ Ð¼Ð¾Ð¶ÐµÑ Ð±ÑÑÑ Ð¸Ð½ÑÐµÐ³ÑÐ¸ÑÐ¾Ð²Ð°Ð½ ÐºÐ°Ðº Ð¼Ð¸ÐºÑÐ¾ÑÐµÑÐ²Ð¸Ñ Ð² MDM-ÑÐ¸ÑÑÐµÐ¼Ðµ.
description: This work is devoted to the analysis and application of data deduplication algo-rithms using the example of an MDM system. Tasks that were solved during the research: 1. Identify the main business fields; 2. Explore ways to normalize data; 3. Explore algorithms to find duplicate data; 4. Choose the most suitable algorithms for searching for duplicates; 5. Develop a deduplication service; 6. To test the deduplication service on test data. The basic data used in MDM systems are selected as the main business fields. For example, full name, digital field, and others. Normalization rules have been de-veloped for each of them. The class of fuzzy string comparison methods was selected, 3 methods were selected to search for duplicates, namely Levenstein, Damerau-Levenstein and Jaro-Winkler. These methods have been developed and applied. As a result of the research, the most appropriate method was selected for each of the business fields based on the accuracy of determining duplicates and the execu-tion time. It was implemented as Python service using Flask based on the interaction of the REST API, which can later be integrated as a microservice in an MDM system.