# Flexible Fashion Product Retrieval Using Multimodality-Based Deep Learning
## 1 Introduction  
One of the key objectives of online shopping malls is to maintain efficient interaction and communication between consumers and online vendors for enhancing consumer trust [1]. Many commodities are registered on e-commerce sites or online shopping malls, and consumers try to find desired products through various search methods. Usually, when customers search for products related to fashion and apparel, they are asked to enter or select a search query they want to find. Query information includes the product name, manufacturer name, product type, product category, and/or product image. The online shopping mall then provides product information corresponding to the request of the user. Typically, fashion product search methods use product meta-information that is either entered by sellers or extracted from product images or related information. Images of fashion products may have a higher priority rather than product categories in the search result. The principle of the meta-information-based search methods, including the category-based search, is robust and straightforward. However, there are inherent limitations due to the inaccuracy of input meta-information, imbalance of categories, and misclassification of product images [2]. Therefore, for an online shopping mall to be user-centric, it is necessary to provide correct product information, including relevant images, while retrieving the results that match with the request of the consumer. For this reason, it is essential that the widely-used faceted search methods [3; 4; 5] and image-based search methods [6; 7] must be combined effectively. Figure 1 shows the search result of fashion products conducted in one of the major on-line shopping malls in South Korea. The accuracy of the category-based search, such as women's clothing \(\rightarrow\) one-piece dress \(\rightarrow\) red, was as high as 93.54%, as shown in the upper part of Figure 1. However, the shopping mall system often failed to distinguish between the color of the background and that of the clothing item. That is, the system recognized a specific color in the image but could not determine what it meant. For another test for the category-based search (male clothing \(\rightarrow\) t-shirt \(\rightarrow\) green), we identified not only the aforementioned problem but also another problem: Non-clothing items were included among the retrieved fashion products. In this case, the seller had incorrectly entered the product meta-information, or the automatic category classification or the image search algorithm might not work correctly. For example, a golf tee was mistakenly retrieved for clothes t-shirt (t-shirt is abbreviated as \"T,\" which is pronounced \"tee\"). The incorrect search result likely appeared because the same metadata were applied to different clothes or different items. In the online shopping mall, the seller modifies the metadata of the product slightly and then uploads the same product several times. This is one of the major impediments to the accuracy of the meta-information-based search. In the preliminary study, the accuracy of the category-based search in the case study was about 75%, as explained in Section 4.1. This paper proposes a new fashion product search method using multimodality-based deep learning which can provide flexible and user-friendly retrieval in an online shopping mall. Multimodal learning is a technique for simultaneously processing information with different modalities, such as images and text-based queries. To utilize both metadata-based queries and product image-based features, the proposed approach combines a convolutional neural network (CNN) model and a recurrent neural network (RNN) model. The deep CNN model learns the key features of the fashion product image and generates a unique feature vector of the image. The proposed method also vectorizes the user query through the RNN to match the CNN-based vectorized image.