FashionKLIP: Enhancing E-Commerce Image-Text Retrieval with Fashion Multi-Modal Conceptual Knowledge Graph
Year: 2023

Abstract:
Image-text retrieval is a core task in the multimodal domain, which arises a lot of attention from both research and industry communities. Recently, the booming of visionlanguage pre-trained (VLP) models has greatly enhanced the performance of cross-modal retrieval. However, the fine-grained interactions between objects from different modalities are far from well-established. This issue becomes more severe in the e-commerce domain, which lacks sufficient training data and fine-grained cross-modal knowledge. To alleviate the problem, this paper proposes a novel e-commerce knowledge-enhanced VLP model FashionKLIP. We first automatically establish a multi-modal conceptual knowledge graph from large-scale e-commerce image-text data, and then inject the prior knowledge into the VLP model to align across modalities at the conceptual level. The experiments conducted on a public benchmark dataset demonstrate that FashionKLIP effectively enhances the performance of e-commerce image-text retrieval upon stateof-the-art VLP models by a large margin. The application of the method in real industrial scenarios also proves the feasibility and efficiency of FashionKLIP. 1

heading: 1 Introduction
text: The explosive growth of multi-modal content on the Web has promoted the research of various crossmodal tasks. Image-text retrieval, which finds correlated texts (or images) for a given image (or text) (Karpathy and Fei-Fei, 2015; Faghri et al., 2017), is a popular cross-modal task with strong practical values in a wide range of industrial applications. Recently, the booming of vision-language
1All the codes and model checkpoints have been released to public in the EasyNLP framework (Wang et al., 2022). URL: https://github.com/alibaba/EasyNLP.
*Corresponding author.
pre-trained (VLP) models (Yao et al., 2021; Zeng et al., 2021; Li et al., 2020c) has greatly improved the representation learning across data of different modalities, leading to significant performance improvement.
However, in the field of e-commerce, the imagetext retrieval task has its own challenges. Here, we suggest that image-text pairs of products have unique characteristics that are different from the general domain (such as MS-COCO (Lin et al., 2014), Flickr30k (Young et al., 2014) and Conceptual Captions (Sharma et al., 2018)), with examples shown in Figure 1. 1) While most texts in the general domain contain descriptions with complete sentence structures, descriptions or queries in e-commerce are usually composed of multiple phrases, describing product details such as materials or styles. 2) Images in the general domain usually have rich backgrounds; in contrast, a product image mainly consists of a large commodity figure in the center without a lot of background objects. These unique domain characteristics make generaldomain models difficult to be directly adopted to the image-text retrieval tasks in e-commerce.
Recently, several domain-specific VLP models including FashionBERT (Gao et al., 2020), Kalei-
149
doBERT (Zhuge et al., 2021), CommerceMM (Yu et al., 2022), EI-CLIP (Ma et al., 2022) and Fashion-ViL (Han et al., 2022) are proposed based on e-commence image-text pairs, which greatly improve the performance of e-commerce imagetext retrieval. Despite the success, the fine-grained cross-modal alignment issue remains unsolved, which may result in the inaccurate matching of details between images and texts. Although some e-commerce VLP models use fine-grained information from either image perspectives (Han et al., 2022) or patch-based image classification (Gao et al., 2020; Yu et al., 2022), they are short of semantic-level alignments across modalities. Some other work (Ma et al., 2022; Zhu et al., 2021) focuses on entities in text modalities, but rarely considers cross-modal interactions. In the general domain, fine-grained interactions could be achieved with object detection (Li et al., 2020c; Tan and Bansal, 2019), scene graph parsing (Cui et al., 2021), or semantic analysis (Yu et al., 2021; Li et al., 2020b). Unfortunately, these tools lose their effectiveness in the e-commerce domain.
To improve the fine-grained alignment between images and texts in e-commerce, this paper proposes an e-commerce knowledge-enhanced VLP model - FashionKLIP. Particularly, we first propose a data-driven strategy to construct a multimodal conceptual knowledge graph in e-commerce (called FashionMMKG) from a large-scale ecommerce image-text corpus, where the fashion concepts are automatically extracted and organized in the form of a semantic hierarchy, each associated with its representative images. The FashionMMKG is later incorporated as the prior crossmodal fashion knowledge in training a CLIP-style model to support e-commerce image-text retrieval. For model training, we learn the representation alignment of image-text pairs across the two modalities by contrastive learning, and further optimize the alignment at the conceptual level. The conceptual alignment is further obtained by matching the text representations with the visual prototype representations of the fashion concepts in FashionMMKG.
Our contributions can be summarized as follows:
• We innovatively propose a data-driven approach to construct a multi-modal conceptual knowledge graph in the e-commerce domain named FashionMMKG without human intervention.
• We construct an e-commerce knowledgeenhanced VLP model called FashionKLIP, which learns conceptual-level alignments based on the prior knowledge in FashionMMKG.
• We conduct experiments on a popular fashion benchmark dataset FashionGen (Rostamzadeh et al., 2018) and show that FashionKLIP outperforms state-of-the-art VLP models in the e-commerce domain.
• We also apply the method to real industrial scenarios and observe significant improvements in image/text-to-product retrieval tasks.

heading: 2 Related Work
text: Vision-Language Pre-training. VLP models can be categorized into single-stream models (Chen et al., 2020; Li et al., 2020a; Gan et al., 2020), which first concatenate multi-modal inputs for interactions, and dual-stream models (Jia et al., 2021; Radford et al., 2021; Yao et al., 2021; Li et al., 2020b), which obtain the representations of the image and text respectively and learn the alignment afterwards. Although single-stream models may lead to high retrieval accuracy due to the early fusion of images and texts, the inference efficiency is sacrificed to a certain extend. Recently, to focus more on fine-grained semantic level interactions of images and texts, some works improve the similarity strategy by calculating between the image patch and the text token (Yao et al., 2021) or leverage fine-grained image information through object detectors (Li et al., 2020c,b; Gan et al., 2020; Zeng et al., 2021). Others introduces structured scene graphs for semantic knowledge (Yu et al., 2021). Despite their success in general domain, such methods are hard to be adopted to e-commerce data. Fashion-based Retrieval. FashionBERT (Gao et al., 2020) first adopts pre-training tasks such as masking strategy to e-commerce images and texts. KaleidoBERT (Zhuge et al., 2021) extracts a series of multi-grained image patches for augmentation to guide masking strategy for fine-grained matching. CommerceMM (Yu et al., 2022) proposes pre-training tasks to align uni-modal with multi-modal features for more consistent alignment. EI-CLIP (Ma et al., 2022) defines the entityaware retrieval task from the linguistic perspective by introducing a causal model to concatenate different meta-data as e-commerce entities. Lately,
Fashion-ViL (Han et al., 2022) designs a flexible architecture for various downstream tasks. However, current methods still suffer from insufficient fine-grained semantic alignment, which may diminish the cross-modal understanding capability of models at semantic level.

heading: 3 Methodology
text: This section introduces how FashionMMKG is constructed and how FashionKLIP incorporates conceptual-level interactions of cross-modal fashion knowledge from FashionMMKG.

heading: 3.1 FashionMMKG Construction
text: Textual Modality. Instead of building an ontologybased knowledge graph (Deng et al., 2022), we automatically construct FashionMMKG to alleviate the gap with real-world user queries. The construction procedures include first determining the concept set through mining massive fashion texts and then matching each concept with its corresponding images. Given a fashion dataset D{T, I} containing N image-text pairs, we first extract all the texts
T . We use the NLP tool spacy2 for sentence components analysis and part-of-speech tagging.We obtain multi-grained concept phrases by concatenating adjective modifiers with the key word. For an input text “Heathered cotton lounge shorts in navy. Elasticized waistband with drawstring closure”, we extract root concepts such as “navy”, “waistband”, “closure” and “heathered”, as well as more detailed phrases: “cotton lounge shorts”, “cotton lounge shorts in navy”, “heathered cotton lounge shorts in navy”, etc. Based on different conceptual hierarchical granularities of extracted results, we build up hypernym-hyponym (“is-a”) relationships between concepts in the form of relation triplets by judging whether two concepts are contained by each other, such as <"cotton lounge shorts in navy", is-a, "cotton lounge shorts">.
After all the relation triplets are extracted, we organize these fashion concepts in a hierarchical structure. A sub-tree with the root node “shorts” is shown in Figure 3. The construction process of the hierarchical structure can be further implemented
2https://spacy.io/usage/linguistic-features
in a dynamic process. When previously unseen concepts appear, we can add these new concepts into existing hierarchical trees, as the newly updated concept “short sleeve t-shirt in white” in Figure 2.
Visual Modality. For the visual modality, we adopt a prompt-based image retrieval method for each concept, and iteratively update the procedure in the subsequent visual-linguistic training process. Utilizing the generalization ability of a pre-trained CLIP-style model, we retrieve product images from the image set I , with the query formulated as "A photo of {concept}" as in (Radford et al., 2021; Yao et al., 2021; Gu et al., 2022). Based on the cosine distance of the image and text features, a naive approach is to select the top k images with the highest similarities as the concept visual prototype.
The retrieval results of some concepts are shown in Figure 4. We can see that the top k images of coarse-grained concepts are usually visually diverse, while images tend to be more semantically consistent when it comes to more specific concepts. To ensure that both similarity and diversity of visual representations for each concept are considered, we slightly expand the range of image candidates (using a larger k), and employ the MMR algorithm (Carbonell and Goldstein, 1998) to improve the diversity of the selected images. It runs in an iterative process until a sufficient number of images are selected from the k candidates. Denote C as the candidate image set and S as the collection of images that have been selected for concept
c. Each time, we choose an image vi by:
MMR(vi) = argmax vi∈C\S [λSim(c, vi)
− (1− λ)max vj∈S
Sim(vi, vj)] (1)
where Sim(·, ·) is the cosine similarity between the corresponding text/image features, and λ is the coefficient to adjust the relevance and diversity of results. Here, we set λ = 0.8 by default.

heading: 3.2 FashionKLIP Training
text: During the model training, as shown in Figure 2, we first extract concepts from the texts. If there are new concepts, FashionMMKG is automatically expanded. For parameter optimization, FashionKLIP consists of two tasks: image-text contrastive learning (ITC) for matching images and texts globally, and concept-visual alignment learning (CVA) for conceptual-level cross-modal alignment. ITC. We train a CLIP-style model to learn the global representations of image-text pairs. For b image-text pairs in each training batch, denote LIk and L T k as the contrastive image-to-text and text-to-image matching loss, respectively. The ITC loss function can be expressed as LITC = 1 2 ∑b k=1(L I k + L T k ), with L T k to be defined as:
LTk (x T k , {xIj}bj=1) = −log exp(sTk,k)∑ j exp(s T j,k)
(2)
where the corresponding text of an image xIk is xTk , and s T j,k is the cosine similarity between the image/text features of xIj and x T k . L I k is defined symmetrically to LTk .
CVA. We further align concepts and visual prototypes from the FashionMMKG. For an input text xTk with image x I k, we obtain a multi-grained concept set Con(xTk ), where hypernym concepts from the tree are also introduced to avoid paying much attention to fine-grained concepts but ignoring the cross-modal understanding of high-level concepts. For a concept ci ∈ Con(xTk ), we denote S(ci) to be the collection of the selected similar yet diverse images to represent the visual characteristics of the concept (as described previously in Section 3.1). We select q images with the highest scores with image xIk in S(ci) for each ci ∈ Con(xTk ), for the model to learn conceptual alignments. We compute the weighted contrastive loss between each ci and any conceptual image xIk̃ ∈ S(ci), together with conceptual images generated from other texts concepts within the same training batch:
LCTk (Con(x T k ), {S(xTj )}bj=1) =
− 1 q
∑
ci
∑
xI j̃ ∈S(ci)
w(xI k̃ , xIk)log
exp(sT k̃,k ) ∑
j exp(s T j̃,k )
(3) Note that w(xI
k̃ , xIk) is the cosine similarity be-
tween concept image xI k̃ and input image xIk, used as the weight for loss calculation. This forces the representation of a concept ci similar to its conceptual images S(ci), but dis-similar to those of conceptual images from other texts. Similarly, by changing the loss function from text-to-image to image-to-text, we have the symmetric loss LCIk . Thus, the loss function of CVA is expressed as:
LCV A = 1
2
b∑
k=1
(LCIk + L CT k ) (4)
Overall Loss. The total loss function is formulated as: L = 12(LITC + LCV A). In addition, as the representations of images are continuously updated during model training, at the end of each epoch, we leverage Faiss (Johnson et al., 2019) to retrieve top-k images to update the visual prototype representations of the matched concepts.

heading: 4 Experiments
text: We conduct comprehensive evaluations on FashionGen (Rostamzadeh et al., 2018) to show that FashionKLIP outperforms SOTA methods.

heading: 4.1 Implementation Details
text: We first construct FashionMMKG with details shown in Appendix A.1.
Model Training. The specific settings of models are described in Appendix A.2. For training, we conduct both domain-specific pre-training and finetuning for base and large versions of FashionKLIP. We initialize FashionKLIP from CLIP pre-trained weights and continually pre-train the model based on our in-house dataset for MMKG construction (as described previously), only using the contrastive learning process over image-text pairs. Specially, the continual pre-training process is conducted with the parameters of the image encoder fixed. Overall, we have four models: FashionKLIP-S (small), FashionKLIP-M (medium), FashionKLIP-B (base) and FashionKLIP-L (large).
Benchmark Dataset. We use a widely-used benchmark dataset (i.e., FashionGen (Rostamzadeh et al., 2018)) for model evaluation. It contains 67,666 fashion items of 293,008 image-text pairs in 121 sub-categories, with 260,480 pairs for training and 32,528 for validation.
Evaluation. For image-text retrieval tasks, based on a text query, we consider two settings for evaluation. 1) Strictly following (Gao et al., 2020; Zhuge et al., 2021; Ma et al., 2022; Yu et al., 2022), the model is required to pick the matched image in 101 samples, including 1 ground-truth image with 100 randomly selected images within the same product sub-category (denoted as “Sample”). 2) As some recently published works (Ma et al., 2022) also consider large-scale candidates on the entire set, each query is compared with every item in the full dataset (denoted as “Full”). The settings for image-to-text matching are likewise. Recall@1/5/10 is regarded as evaluation metrics as previous works (Gao et al., 2020; Zhuge et al., 2021; Yu et al., 2022).

heading: 4.2 Experimental Results
text: Overall Retrieval Results. We conduct both “full” and “sample” evaluation of FashionKLIP-B against existing SOTA models. In addition, we report the results of different FashionKLIP models on FashionGen using the full evaluation criteria, as shown in Table 3. As the main experimental results shown in Table 1, we can see that FashionKLIP model significantly outperforms the existing SOTA models by a large margin. In particular, on the R@1 metric, FashionKLIP-B even greatly surpasses the methods with multi-modal fusion encoders for more unified representation learning such as CommerceMM (Yu et al., 2022). On full evaluation results in Table 2, FashionKLIP-B shows a remarkable increase of 11-15% compared to EI-CLIP (Ma et al., 2022). For smaller settings such as FashionKLIP-M, the retrieval performance is also competitive and closer to CLIP. As the “full” setting is closer to real-world retrieval scenarios and more challenging as it aims to select from a large candidate set, the performance of FashionKLIP is significant, further proving that the framework can be generalized to wider application scenarios. Based on the experimental results on either setting, we can conclude the effects brought by fashion knowledge, and confirm that more attention to cross-modal conceptual-level interactions leads to an increase in e-commerce image-text matching. 3
Ablation Studies. To further analyze the impor-
3Note that a few works (e.g., Fashion-ViL (Han et al., 2022)) employ additional multi-modal fusion encoders and uniform representation learning (that may be not suitable for fast vector retrieval in real-world applications) and evaluate their models on randomly sampled subsets of FashionGen. Hence, their works are not directly comparable.
tance of conceptual-level fashion image-text alignment, we present different variants of FashionKLIP in Table 4 for two evaluation settings. We can see from the results that both CVA and the FDP contribute to performance improvement. Although the retrieval results decrease slightly when not using FDP, the removal of CVA will harm the retrieval performance more heavily. Besides, the introduction of FDP and CVA at the same time boosts the performance as “Full Implement.” shows, proving the necessity to utilize fashion data for pre-training, which helps establish a better mapping between concepts and images as prior knowledge. More importantly, the focus on fashion knowledge better guides conceptual-level interactions and brings a rise to the alignment between images and texts.

heading: 5 Industrial Application
text: In this section, we verify the effectiveness of FashionKLIP on our Alibaba global e-commerce platform. Specifically, we apply it to product search with two specific retrieval tasks including image-toproduct (I2P) and text-to-product (T2P) retrieval, as shown in Figure 5.
For T2P, we employ a weighted scoring function to compute the similarity score between a query text and a product (with a title and an image) as follows: Scoret2p = α∗Scoret2t+(1−α)∗Scoret2i, where 0 < α < 1, Scoret2t and Scoret2i refer to the embedding similarity score between the query text and the product title, together with the query text and the product image. Similarly, for I2P, we have Scorei2p = α∗Scorei2t+(1−α)∗Scorei2i.
In total, the collected dataset contains 58,463 products (with images and titles) and 3,021 queries.
We conduct zero-shot experiments for T2P and I2P on FashionKLIP-B and FashionKLIP-M and compare it with the baseline CLIP (Radford et al., 2021), as shown in Table 6. For models of the same size, we can see that FashionKLIP-B greatly outperforms CLIP on Recall@1-20 and particularly achieves an improvement of 3~5% on both tasks for R@1. For our model in a smaller size, FashionKLIP-M is still comparable, which mainly reflects on the R@1 and R@5 results of I2P task and the R@5 to R@20 results of T2P. However, the inference of FashionKLIP-M is faster. In Table 5, taking text-to-product as an example, we report the Response Time (RT) and Query Per Second (QPS) using different text encoders to encode user queries on a single GPU (NVIDIA V100). We can see that with similar performance (CLIP and FashionKLIPM), our model has much lower RT and higher QPS. Hence, we confirm FashionKLIP’s feasibility on multi-modal tasks in the industrial applications.

heading: 6 Conclusion and Future Work
text: This paper proposes a novel data-driven approach to construct a multi-modal conceptual knowledge graph in e-commerce namely FashionMMKG. An e-commerce knowledge-enhanced VLP model namely FashionKLIP is then constructed by learning the conceptual-level alignments from the prior knowledge in FashionMMKG. Our empirical study
shows that FashionKLIP outperforms state-of-theart VLP models in the e-commerce domain. We conduct experiments under industrial scenarios and verify its practical value in real-world applications and confirm the efficiency of FashionKLIP. In the future, we will apply the knowledge-enhanced strategy for general large-scale pre-training and bring benefit to more multi-modal tasks.

heading: 7 Acknowledgement
text: This work is funded by National Key Research and Development Project (No.2020AAA0109302), National Natural Science Foundation of China (No.62072323), Shanghai Science and Technology Innovation Action Plan (No. 22511104700, 22511105902), Shanghai Municipal Science and Technology Major Project (No.2021SHZDZX0103), Science and Technology Commission of Shanghai Municipality Grant (No. 22511105902) and supported by Alibaba Group through Alibaba Innovative Research Program.

heading: A Appendix
text: A.1 FashionMMKG Full statistics of our FashionMMKG are shown in Table 8, where we give both the total numbers (cnt)
of items such as the number image-text pairs and concepts, and the average of some attributes (avg) such as occurrence and concept length. As for the data source, we extract fashion concepts from titles of 900,000 product image-text pairs collected from our global e-commerce platform. 4
A.2 Model Settings
We release models with various parameter sizes for industrial applications. The specific hyperparameters of different FashionKLIP models are shown in Table 7. Image Encoder We follow Vision Transformer (ViT) (Dosovitskiy et al., 2020) closely as the image encoder and the modifications of different models lie in the number of layer normalization and the width of attention heads. The size of nonoverlapping image patches are also set to be different. FashionKLIP-L adopts the ViT-L/14 as the image encoder with 24 layers, while FashionKLIP-M uses ViT with 12-layer 512 wide in 88M parameter and the patch size is 32. Text Encoder We adopts a Transformer (Vaswani et al., 2017), utilizing the same architecture as described in (Radford et al., 2019) as the text encoder. For models in different sizes, we refer to (Turc et al., 2019) to set the attention width and number of attention heads of the text encoder. Model Input Images are cropped uniformly to 224 × 224 pixels before entering the model. We limit the maximum input length of the text to 77, with a vocabulary of 49,408.
For a fair comparision, we utilize FashionKLIPB model to compare against other baseline models, which uses ViT-B/32 (Dosovitskiy et al., 2020) as the image encoder, and adopts a 12-layer 512 wide Text Transformer as the text encoder as (Radford et al., 2021), in 63M parameter with 8 attention heads each layer.
A.3 Model Training
The batch size of pre-training is 1,024 per GPU with 8 A100 GPUs (80G), for 20 epoches in total. The learning rate is 5e-5. During dataset-specific model fine-tuning, we retrieve top-20 images for each concept in FashionMMKG and then select 5 images as the visual prototype based on the proposed criteria. The batch size of fine-tuning is 32 per GPU, with a learning rate of 1e-5 on two A100 GPUs. As smaller pre-trained CLIP weights
4https://www.alibaba.com/
are not available, we initialize FashionKLIP-M and FashionKLIP-S models from the pre-trained FashionKLIP-B model by truncating the weights of FashionKLIP-B to the size based on the settings of smaller models. After that, we utilize the contrastive learning process for continually pretraining on the e-commerce in-house data. The batch size during pre-training for FashionKLIP-M and FashionKLIP-S is 256 per GPU on 8 GPUs and the learning rate is 5e-5.

heading: Future Work
text: In the future, we will apply the knowledge-enhanced strategy for general large-scale pre-training and bring benefit to more multi-modal tasks