# Flexible Fashion Product Retrieval Using Multimodality-Based Deep Learning
## 3 Proposed Multimodality-Based Deep Learning for Fashion Product Retrieval
### Pre-Training of the Query Encoder  
When the deep learning-based training was conducted, the learning sometimes did not proceed properly. We assumed that the deep learning model had difficulty in learning the relationship between keywords in the queries. To alleviate this problem, we conducted pre-training of the query encoder, as shown in Figure 3, to make the encoder respond appropriately to the queries. The pre-training process is the same as the primary training process, except that the generated vector is sent to the query decoder instead of the multimodal classifier. The query decoder produces a semantic vector corresponding to the meaning of the user query. An example of a semantic vector is shown in Table 5. The semantic vector is created based on the category query; its element is \(1\) if the corresponding fashion product matches the query and \(0\) otherwise. Table 6 shows the pairs of query words used and their corresponding semantic vectors generated by the query decoder. Because different queries have the same meanings, they can be associated with the same semantic vector. The purpose of pre-training of the encoder is to have good initial weights before main training. In this research, when the accuracy of pre-training reached about 90%, pre-training was stopped. After this pre-training process, the query encoder could generate the query word vectors by reflecting the meaning of the queries. \begin{table}
\begin{tabular}{c c} \hline
**Queries.** & **Meaning** \\ \hline “select * from women” & women fashion products \\ “select * from women where sweater” & women’s sweater \\ “select * from women where red” & women’s red clothes \\ “select * from women where not red” & women’s clothes without the red color \\ \hline \end{tabular}
\end{table} 
Table 4: Query examples using the pseudo SQL form (the syntax can be modified). Figure 3: Pre-training of the query encoder. \begin{table}
\begin{tabular}{c c  
In this study, the total number of queries for pre-training is 10,330: With 164 category-based, 3828 Boolean-based, and 6338 pseudo SQL-based queries. As the pseudo-SQL has the most complex representation, it requires more queries for pre-training compared with the category-based and Boolean-based queries. Important reasons to perform the pre-training are to improve the efficiency of the learning process and to support the flexibility of handling user queries. For example, in contrast to fashion product images, the user query has many different keywords or forms with the same meaning. Therefore, to increase efficiency and flexibility of the multimodal learning, we first pre-trained the encoder with the decoder and then used the encoder with user queries and product images for training the dataset. Therefore, the meaning of the operators such as OR, AND, and NOT were learned in advance. Other query forms were similarly pre-trained. The suggested pre-training process is based on the seq2 seq model proposed by [PERSON] et al. [30]. This is a language generation model that is used for various applications, such as machine translation, speech recognition, and text summarization. To interpret the results of the pre-training, we used the t-distributed Stochastic Neighbor Embedding(t-SNE) visualization technique [31], as shown in Figure 4. t-SNE is a two-dimensional representation of a high-dimensional vector. Figure 4 shows all the query terms used in the pre-training through t-SNE. Figure 4 shows some query examples, which are defined as follows:  
* Example query 1
* \"men \(\rightarrow\) shirt\(\rightarrow\) green\",
* FROM men WHERE shirt AND green\"
* Example query 2
* \"women \(\rightarrow\) dress \(\rightarrow\) red\"
* FROM women WHERE dress AND red\"
* Example query 3
* \"men \(\rightarrow\) coat\"
* \"men AND coat\"
* FROM men WHERE coat\"  
In the results of the pre-training with the foregoing three examples, the queries with similar meanings appear at similar positions, but the queries with different meanings appear at a distance.