Spatiotemporal Graph Guided Multi-modal Network for Livestreaming Product Retrieval

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-03 07:55:30

Abstract: With the rapid expansion of e-commerce, more consumers have become accustomed to making purchases via livestreaming. Accurately identifying the products being sold by salespeople, i.e., livestreaming product retrieval (LPR), poses a fundamental and daunting challenge. The LPR task encompasses three primary dilemmas in real-world scenarios: 1) the recognition of intended products from distractor products present in the background; 2) the video-image heterogeneity that the appearance of products showcased in live streams often deviates substantially from standardized product images in stores; 3) there are numerous confusing products with subtle visual nuances in the shop. To tackle these challenges, we propose the Spatiotemporal Graphing Multi-modal Network (SGMN). First, we employ a text-guided attention mechanism that leverages the spoken content of salespeople to guide the model to focus toward intended products, emphasizing their salience over cluttered background products. Second, a long-range spatiotemporal graph network is further designed to achieve both instance-level interaction and frame-level matching, solving the misalignment caused by video-image heterogeneity. Third, we propose a multi-modal hard example mining, assisting the model in distinguishing highly similar products with fine-grained features across the video-image-text domain. Through extensive quantitative and qualitative experiments, we demonstrate the superior performance of our proposed SGMN model, surpassing the state-of-the-art methods by a substantial margin. The code and models will be public soon.

Main paper: 
 introduction: 
 Benefiting from the convenience of e-commerce, shopping online become increasingly popular in recent years. The livestreaming product retrieval (LPR) plays a crucial role in accurately matching products shown in live clips with those available in online Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. MM '24, 28 October -1 November 2024, Melbourne, Australia © 2024 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM ISBN 978-1-4503-XXXX-X/18/06 https://doi.org/XXXXXXX.XXXXXXX (a) Cluttered and similar background products. shops, ensuring a seamless shopping experience and excellent marketing ability [11, 13, 25] . Even though various customer-to-shop retrieval methods have great progress [6, 12, 14] , the cross-domain livestreaming-to-shop problem in LPR has few studies yet. Some unresolved challenges are still rooted in real-world applications.First of all, accurately discerning the intended products offered by salespersons during livestreaming is profoundly challenging. It is common practice for salespersons to showcase multiple products but focus on promoting a specific product at a given time, defined as intended product. Some methods have attempted to incorporate an additional product detection module [12, 46] , but they will incur high annotation costs and computational complexity, and also fail to completely eliminate ambiguity of intent. Sometimes, the detection of multiple foreground boxes introduces more uncertainty and erroneous guidance, as shown in Fig. 1(a) .Secondly, the heterogeneity between realistic livestreaming videos and product images further exacerbates the difficulty of crossdomain retrieval. As depicted in Fig. 1 (b), livestreaming introduces variations in viewpoint, illumination, and occlusions, resulting in significant appearance disparities between products in livestreaming and in online store. It is very common in live scenarios, where methods focused on the instance-level matching of entire clips will fail [6, 12, 44] . Coarse-grained instance-level matching makes it hard to track spatial structural deformations. The inevitable motion and occlusions will render products visually obscured or degraded during specific time slots, making frame-level sequential matching with temporal consistency necessary. Moreover, Large domain discrepancy leads to misalignment of intra-domain features. Existing cross-domain retrieval methods [6, 8, 24, 30, 41] often treat different domains equally while ignoring spatiotemporal relations within and between domains, further reducing retrieval accuracy.Another dilemma in real-world applications lies in the abundance of highly similar products in online stores, necessitating that models excel at learning fine-grained representations and discriminating subtle distinctions among similar-looking products. As shown in Fig. 1 (a), there exists a plethora of indistinguishable bracelets. Some methods based on pre-trained CLIP attempt to learn classifiable visual embeddings from large-scale video-text embeddings [4, 8, 21, 35] . However, generic features often fail to capture subtle product characteristics. This deficiency contributes to the suboptimal performance of the latest method, RICE [44] , particularly in the crucial first-ranking metric (R@1).To tackle the real-world challenges mentioned above, we propose the Spatiotemporal Graphing Multi-modal Network (SGMN) for LPR. Our model consists of three key components: 1) We leverage the verbal explanations provided by salespersons in livestreams, which often contain explicit information about the intended products. The texts from Automatic Speech Recognition (ASR) transcriptions and image titles guide the model to focus on products highly relevant to the verbal context, mitigating distractions from background clutter items. 2) We design a Graph-based Cross-domain Interaction (GCI) module to capture spatiotemporal correlations between videos and images. It is the first exploration of using sequence-to-sequence graph learning to model and enhance crossdomain temporal consistency and spatial correlation. As shown in Fig. 1 (c), we establish the intra-domain and inter-domain connection graphs between video and image synchronously. Benefiting from the frame-level connectivity, the model can still accurately localize regions of intended products that encounter appearance distortion due to occlusion, motion, and brightness changes in Fig. 1(b). 3) The Selective Multi-modal Fusion (SMF) module is proposed to assist the model in distinguishing highly similar products. This mechanism selects the top-K hard examples in global ranking, and then fuses their visual and textual representations for implicitly recalibrating ranking and discerning semantic heterogeneity. The fine-grained alignment across the video-image-text domain demonstrates robust potential. Notably, our method adheres to the principle of independently encoding multi-modal inputs during inference, striking a balance between efficiency and accuracy. Our contributions can be summarized as follows:• We introduce the textual information in the detector-free retrieval framework to guide model attention on the intended product. The comprehensive yet unified representation learned from the cross-modal alignment space of video-image-text solves the practical dilemma of LPR.• The spatiotemporal graph learning is first explored for capturing sequential relations, alleviating inter-domain misalignments in both spatial and temporal dimensions. • We selectively fuse multi-modal features to enhance finegrained representations of hard samples, distinguishing products with subtle visual differences. • Our method achieves the best performance on the largescale benchmark dataset. Extensive quantitative and qualitative experiments prove the superiority of SGMN. 
related works 2.1 livestreaming product retrieval: 
 Before the livestreaming became popular, researchers paid more attention to the customer-to-shop clothes retrieval tasks, such as fashion retrieval [10, 11, 13, 15, 25] . Motivated by video-to-shop retrieval, some works are dedicated to providing pair-matching solutions between image and video features, such as DPRNet [46] and SEAM [12] . Such methods adopt a two-stage framework combining detection and retrieval, localizing the products in the video before performing the global similarity match. AsymNet [6] uses a single-stage network that removes object detection to reduce model complexity. RICE [44] proposes a single-stage network without object detection to reduce model complexity. However, these methods have yet to effectively leverage the textual modality to identify the intended products. In our work, we make the first attempt at enhancing visual representations of intended products using the textual features in a one-stage framework.Existing cross-domain retrieval methods usually learn unified representations across different domains. Some methods feed concatenated visual and textual features into classifiers to predict image-totext matching performance [4, 21, 34, 35] . CLIP2Video [8] transfers knowledge from pretrained CLIP to learn video-to-text features.Recent methods with high inference performance use independent encoders to extract global features and compute cross-domain similarity [28, 30, 32, 36, 48] . Methods for cross-domain interaction are also explored. SCAN [19] uses an attention mechanism to fuse across domains, and IMRAM [2] applies an iterative network for interaction. The multi-grained matching mechanism [30] shows limited performance in capturing long-range temporal dependency.Our work employs a spatiotemporal graph to enhance the crossdomain consistency across the video-image-text domain.The graph reasoning network (GCN) has proven effective due to the powerful expressive capability of graph structures [3, 38] . In cross-domain retrieval tasks, using graph reasoning networks [16, 18, 47] to learn the relationship between different domains is a common practice. VSRN [20] models the local visual features of key objects semantically. CGMN [5] and GSMN [22] learn image-text matching using a cross-domain graph matching network. DSRAN [40] adopts the graph attention. Wang et al. used a coarse-to-Fine graph network for video-text retrieval [39] . The HREM [9] learns semantic relationships among image and text via a hierarchical graph relation model. These methods have shown the advantages of graph learning in modeling spatial correlation, but its potential in capturing temporal information within video sequences has yet to be explored. Our work has made preliminary attempts and validations in spatiotemporal sequence-level graphing. 
method: 
 Our SGMN consists of three parts: the Global Representation Alignment (GRA), the Graph-based Cross-domain Interaction (GCI), and the Selective Multi-modal Fusion (SMF), as illustrated in Fig. 2 . Further implementation details are elucidated in subsequent sections.Image Encoder. We use the pretrained ViT-B/32 model from CLIP [29] as the image encoder. We define a batch of 𝑁 images as 𝐼 ∈ R 𝑁 ×𝐻 ×𝑊 , and the encoders extract non-overlapping image patches with the size of 𝑝 × 𝑝. Then we perform a linear projection to project these flattened patches into 1D markers as shown in Fig. 2 . We then use the Multi-Head Attention (MHA) mechanism to interact with each patch of the image for the global aggregated features 𝐼 𝑐𝑙𝑠 and patch-level hidden embedding 𝐼 ℎ𝑖𝑑 :EQUATIONwhere Q, K, and V represent the query, key, and value. The 𝐼 𝑐𝑙𝑠 ∈ R 𝑁 ×𝐷 and 𝐷 is the embedding dimension.Video Encoder. The live streams are divided into clips 𝑉 ∈ R 𝑁 ×𝐿×𝐻 ×𝑊 , where 𝐿 is the length of video frames. We share parameters between the image encoder and the video encoder for global representation alignment, so the sequential features of video clips are encoded as:EQUATIONWe also adopt the cls token 𝑉 𝑐𝑙𝑠 ∈ R 𝑁 ×𝐿×𝐷 as the global representation of video clips. To enhance inter-frame alignment inside videos and extract correlations within each frame, we use a Temporal Motion Compensation (TMC) module. TMC adds the displacement of inter-frame actions to the video sequence to simulate motion changes as Δ𝑓 𝑡 = 𝑓 𝑡 -𝑓 𝑡 -1 , where 𝑓 𝑡 -1 and 𝑓 𝑡 represent two adjacent frames. We then encode the motion of consecutive frames and insert them as a motion compensation token for enhancing differential-level attention. The global video representation after motion-compensated enhancement is:EQUATIONText Encoder. ChineseCLIP [43] is used to extract initial textual representations for video automatic speech recognition (ASR) 𝑇 𝑎𝑠𝑟 and image titles 𝑇 𝑖𝑡𝑒𝑚 . To keep the model lightweight, we pretrained the ChineseCLIP model and fixed the parameters in model training. Since the speech information from the salesperson always contains redundant product-independent information in the real-world livestreaming, we add a filter layer F filter to eliminate irrelevant units while retaining key information.EQUATIONwhere the F clip (•) is the function of ChineseCLIP. Similarity Loss. We use the triplet loss T (•) [7] to measure the similarity of visual and textual embedding. The detailed implementation can be found in the supplementary. Therefore, the optimization goal of GRA is defined as the similarity loss L 𝑠 between video-to-image and text-to-text representations, and 𝜆 is defined as the loss weighting factor. EQUATIONGraphs Reconstruction. We build a video graph, image graph, and video-image graph using global representations, as shown in Fig. 2 . These spatiotemporal graphs can capture temporal consistency and spatial relations synchronously. First, for a batch with 𝑁 video-image pairs, the sequence length of the video is 𝐿, so we can obtain the global video features as 𝑆 𝑉 = 𝑉 𝑣𝑖𝑠𝑢𝑎𝑙 ∈ R 𝑁 ×𝐿×𝐷 . Then we stack images into sequences of the same length as video clips for the sequence-to-sequence alignment, and the stacked global image features as 𝑆 𝐼 = 𝑆𝑡𝑎𝑐𝑘 (𝐼 𝑣𝑖𝑠𝑢𝑎𝑙 ) ∈ R 𝑁 ×𝐿×𝐷 . The construction of the inter-domain spatiotemporal graph is shown in Fig. 3 . The video-to-image similarity matrix H 𝑉 2𝐼 ∈ R 𝑁 ×𝐿×𝐿 is represented as:EQUATIONHowever, fully connecting each <frame, image> pair while ignoring the relative importance is suboptimal [33] . Thus we define two intra-domain relation matrices and two inter-domain relation matrices, which are video spatiotemporal graph 𝑮 𝑉 2𝑉 , image spatial graph 𝑮 𝐼 2𝐼 and video-image cross-domain graph 𝑮 𝑉 2𝐼 and 𝑮 𝐼 2𝑉 . Then we build the entire spatiotemporal graph with intra-domain relations and cross-domain association as follows:EQUATIONGiven 𝑁 image-video pairs, the size of four relation matrices are{𝑮 𝑉 2𝑉 , 𝑮 𝑉 2𝐼 , 𝑮 𝐼 2𝐼 , 𝑮 𝐼 2𝑉 } ∈ R 𝑁 ×𝐿×𝐿 .The matrices 𝑮 𝑉 2𝑉 and 𝑮 𝐼 2𝐼 represent the instance-level correlation of images or videos, while the matrices 𝑮 𝑉 2𝐼 and 𝑮 𝐼 2𝑉 represents the frame-level correlation between each image and each video frame. All nodes and edges in graphs independently and jointly represent objects and relations of multiple domains. Then we refer to the frame-by-frame matched scores to filter out irrelevant connectivity and define two semantic properties: Connection and Relevance.i) Connection. To build an efficient correlation graph and preserve useful spatiotemporal connections, we follow an advanced fine-grained matching scheme [19] . The connection relationships between each video frame and the image depend on the matching relationships of multiple nodes.As shown in Fig. 3 , given the frame-image similarity matrix, each column vector represents the matching relationships between each video frame and a certain image. We first compute the mean of the similarity and then measure the top-K matching pairs. We calculate the difference between the mean value and top-K values as:EQUATIONwhere 𝑖 ∈ [0, 𝐿) and H 𝑖 𝑉 2𝐼 is the 𝑖-th column vector and 𝐾 is set as 𝐿/2. Finally, the connection property between the frame and the image is determined based on the logical values of C. The M ∈ R 𝑁 ×𝐿×𝐿 is the connection mask:EQUATIONii) Relevance. Due to the heterogeneity between videos and images, simply using global embeddings for cross-modal relevance learning is insufficient. So we continue to employ fine-grained matching based on multiple nodes to narrow the cross-domain differences. For the video-to-image associations, we first extract the column vectors of H 𝑖 𝑉 2𝐼 and then simultaneously compute the mean value and remap the top-K matching scores:EQUATIONwhere 𝑖 ∈ [0, 𝐿) is the column index and 𝐾 = 𝐿. Since the relevance learning for different samples within a batch is independent, we use an additional batch linear layer to encourage interactions between different samples and further expand the matching receptive field.EQUATIONwhere the F BLL (•) includes an ReLU activation layer and two linear layers. Then we obtain the relevance matrix R for all pairs of videoto-image instances. Therefore, the cross-domain graph can be constructed from the connection matrix C and relevance matrix R as:EQUATION)Cross-domain Interaction. As shown in Fig. 4 , we design two attention branches to enhance the visual representation. The crossdomain connections in GCI represent the interaction of intra-domain and inter-domain features. We use the constructed graph 𝑮 𝐼 2𝑉 and 𝑮 𝑉 2𝐼 to replace the mask in the MHA function to guide the model attention to local areas with high relevance as: where F LN (•) is the layer norm function. Then we take the graphenhanced feature 𝐼 𝑚 and 𝑉 𝑚 through the MHA module under the guidance of the intra-modal graphs 𝑮 𝐼 2𝐼 and 𝑮 𝑉 2𝑉 to obtain the graph-enhanced visual representation 𝐼 𝑔 and 𝑉 𝑔 , and F FFN (•) is the function of a feed-forward network.EQUATIONEQUATION)Graph Loss. We first use the triplet loss T (•) to compute the cosine similarity of features augmented by spatiotemporal graphs, jointly optimizing the parameters of the image, video, and videoimage graphs. In addition, the cross loss between the enhanced embedding and the original embedding is additionally calculated to narrow the feature domain difference, and the graph loss is:EQUATIONWe use the KL-divergence loss [37] function K (•) to minimize the difference between the distribution of calibrated cross-domain correlations and the global embedding.EQUATION)And the total loss in GCI is defined as L 𝑔 = L 𝑔𝑟 + L 𝑘𝑙 .To improve the discrimination of semantically related and visually similar products, we further perform the hard negative mining operation using the multi-modal features. First, we compute a similarity matrix with visual and textual representations in the GRA module to select hard examples as follows:EQUATIONthen we select top 𝐾 samples ranked in 𝑀 𝑠𝑖𝑚 as the hard samples for further matching. The 𝐾 is set to 4. The visual and textual features of these samples are selected as:EQUATIONBesides, we fuse the textual feature and visual representations of selected samples using a fusion layer F fusion (•). The multi-modal features of video and image are learned.EQUATIONThe fused features V and Î will pass through a perceptual module consisting of a self-attention layer and a cross-attention layer F 𝑐𝑟𝑜𝑠𝑠 to obtain cross-domain features:EQUATIONThen we use the symmetric cross-entropy loss E (•) [8] to calculate the matching loss in hard negative mining.EQUATIONTo learn a comprehensive representation from the video-image-text alignment space, the total loss is optimized as:EQUATIONMentioned that the model only uses the independent embedding in GRA to calculate the global similarity in the inference stage, having high matching accuracy and efficiency.During the inference stage, only the GRA module is utilized to independently encode the global visual and textual representations of the input video and the product gallery images. The matching scores S are calculated as a weighted similarity of the embeddings from the visual and text domain.EQUATION 
experiments 4.1 experimental setup: 
 Datasets. We conduct experiments on the publicly available dataset LPR4M [44] and MovingFashion (MF) [12] . The LPR4M dataset is the largest publicly available multi-modal dataset that covers 34 categories and comprises three modalities (image, video, and text). It includes 3,955,181 pairs of video-images for training and 20,079 pairs for testing, encompassing a diverse range of scenes. The MF dataset contains only one category of fashion clothing with two modalities (image, video), comprising 15,045 pairs in the training set and 1,341 pairs in the test set. These well-stocked commerce datasets facilitate a fair evaluation of our method in real-world scenarios. More dataset analyses are available in the supplements. Implementation Details. We initialize image and video encoders using the pretrained ViT-B/32 model from CLIP [29] with a feature dimension of 512. The text encoder is initialized by the pretrained RoBERTa-wwm-Base model from ChineseCLIP [43] . For video preprocessing, we extract 10 evenly spaced frames from each clip. Each frame in videos has a probability of 0.5 to be randomly masked for data augmentation, and the masking percentage ranges from 0 to 0.9. We set the weight factor 𝜆 = 0.5 in Eq. 5 and Eq. 23. The margin in triplet loss is set to 0.2 for proper discrimination. In Eq. 8, top-K scores serve as the relevance threshold, set to 𝐿/2 to balance performance and complexity. In Eq. 17, higher 𝐾 implies more memory (M) costs, so we set 𝐾=4. The weight factors are 𝛽 1 = 0.7 and 𝛽 2 = 0.3 in Eq. 22. In model optimization, we use the Adam [17] optimizer with a batch size of 256. The learning rate is 3 × 10 -4 , which decays following the cosine schedule [27] . Our models are trained on 8 NVIDIA Tesla V100 GPUs. Following the standard retrieval task [31, 45] , recall at rank K (R@K) is adopted as the metric to evaluate the performance quantitatively. Methods R@1 R@5 R@10 R@mean FashionNet [25] 13. 4 Methods R@1 R@5 R@10 R@mean NVAN [23] 38.0 62.0 70.0 56.7 MGH [42] 40.0 59.0 66.0 55.0 AsymNet [6] 42.0 73.0 86.0 67.0 SEAM [12] 49.0 80.0 89.0 72.7 RICE [44] 76. We quantitatively compare the proposed SGMN with the existing methods on the LPR4M dataset and MF dataset, and the results are shown in Tab. 1 and Tab. 2. On the large-scale dataset LPR4M, the retrieval performance of our method outperforms the state-of-theart RICE [44] by 10.4% in R@1, 3.4% in R@5, and 1.9% in R@10.Existing methods exhibit limited performance on LPR4M with rich categories and diverse scenes. We also verified the performance of SGMN without using textual features (w/o TE). Our SGMN can still achieve a performance gain of 5.7% on R@1 without the textual guidance to the salient regions (38.7% vs 33.0%).Since the MF dataset does not have text modality, we do not fuse the textual information in training. Efficient global cross-domain alignment still allows SGMN to maintain the best performance with significant gains. Compared with the official method of MF dataset SEAM, our method improves R@1 by 28.8% in R@1. Although the MF testset contains only 1,342 videos, our approach still gains a 1.7% advantage over the highest RICE method in the crucial R@1 metric, signifying our superiority in extracting fine-grained discriminative features. We also verified the model trained on LPR4M without finetuning on MF, and the results show that even if the model has not seen the MF data, it can still surpass most trained models, demonstrating robust zero-shot fitting and generalization capability. To summarize, our method benefits from cross-domain connections and fine-grained matching, showing greater superiority on both simple and diverse datasets, particularly in the crucial R@1 metric.Retrieval Results. We randomly select several video sequences from MF dataset and LPR4M dataset, and the top-3 shop images are shown in Fig. 5 . It can be seen that even if the live scene is filled with numerous products of the same type, our SGMN can still find consistently outperforms RICE by a significant margin across various subsets, demonstrating robustness in real-world scenarios.Text Attention Analysis. Embedding text from video ASR and product items can guide the model to focus on the intended item in the cluttered background at a low cost. Therefore, we visualized the attention of the video encoder in GRA equipped with or without (w/o) the TE module. As shown in Fig. 6 , the method with TE is superior to that w/o TE in accurately retrieving the intended product. For example, in the live clip of selling hairpins, multiple similar hairpins appear in view simultaneously, interfering with the identification of intended products. In this case, the keyword 'butterfly' mentioned by the live salesperson played a key role, significantly increasing the probability of matching products that contain both 'butterfly' and 'hairpin' in the title. Similarly, there are a large number of extremely similar products in the vermilion necklace category gallery, and their visual differences are almost negligible in the live domain, where the resolution is degraded. Fortunately, the assistance of text information solves this dilemma very well. The keywords 'Pixiu' and 'necklace' have endowed the product with specificity, freeing the network from the interference of redundant products irrelevant to the keywords. These results show that our SMGN equipped with additional textual representations alleviates the difficulty of product-heavy recognition. Loss Function Analysis. As shown in the Tab. 5, we analyze the weights of different components in the loss function. It can be observed that not using graph loss or hard example loss is unwise. A higher weight for the graph loss brings more performance gain, but it needs to be balanced with the SMF module. The frame-level temporal correlation complements high similarity discrimination, so we set 𝛽 1 and 𝛽 2 in Eq. 22 as 0. aligned domain is more conducive to learning efficient representations. Since products in the live domain often have appearance deformations, capturing enough fine-grained and long-range features to distinguish video-to-image pairs accurately is critical. The jointly modelled temporal consistency and spatial correlation enhance the global embedding of images and videos, achieving a 2.9% accuracy gain in R@1 (41.5% vs 38.6%). Identification of Similar Products. In livestreaming e-commerce, similar products have incredibly similar appearance and visual features and often appear in the same field of view to facilitate sales, further exacerbating recognition accuracy. Therefore, to illustrate that our proposed SMF module strengthens the model in distinguishing hard negative samples, we selected several video sequences with top scores in the 'Liquor' category retrieval for quantitative and qualitative comparisons. As shown in Fig. 7 (a), the product images of the three liquors have extremely slight visual differences, and their corresponding live video sequences are also easily confused. We quantify the similarity scores of the three videos (A, B, and C) and the corresponding product images (I, II, and III) with and without (w/o) the SMF module, and the results are shown in Fig. 7 (b). It can be seen that the model without SMF has poor scoring discrimination for similar products, and the scores of the correct category and the wrong category are very close, which increases the probability of misclassification. On the contrary, the model that further performs hard example mining using SMF has higher discriminative power for similar products. Furthermore, we visualize the visual attention of our SGMN on three video sequences in Fig. 7(c ). The mining of hard samples incorporating multi-modal features captures sufficient fine-grained features to recalibrate model attention on intended products accurately and efficiently.Model Component Ablation. We analyze the performance of each component in our method on the LPR4M dataset and present the quantitative results in Tab. 7. The baseline is a GRA module that independently encodes visual representations. It can be seen that each component we designed is effective. Among them, the TE introducing text attention and GCI learning spatiotemporal cross-domain interaction achieved higher performance gains, which are 4.5% and 2.7% in R@1, respectively. TMC enhances the intraframe correlation of the global embedding of the video, and SMF further distinguishes similar samples, achieving R1 gains of 0.9% and 0.6%. Overall, the GCI module emphasizes frame-level finegrained correlation over global coarse-grained alignment, while the SMF module achieves highly similar semantic discrimination at minimal cost. Our proposed solution aims to comprehensively optimize spatiotemporal consistent multi-modal representations, addressing the application dilemma of LPR step by step. 
conclusion: 
 In this paper, we rethink the LPR task from a more macroscopic and practical point of view and propose a one-stage spatiotemporal graphing network to address the real-world dilemmas of LPR tasks. To the best of our knowledge, this is the first exploration of sequence-to-sequence graph learning in mitigating heterogeneity between videos and images. Existing methods are limited by their narrow focus on coarse-grained matches. But we progressively enhance the fine-grained discrimination by integrating modality-level text embeddings, instance-level similarity mining, and frame-level graph learning. Despite potential distortions in appearance within the complex livestreaming domain, our method adeptly tracks spatial deformations, ensuring precise location of the intended product. Moreover, we have fully harnessed the accessible textual information from live ASR transcripts and product titles, freeing the network from interference over cluttered background products. The further hard negative mining in video-image-text alignment domain improves the ability of our model to distinguish highly similar products. Extensive quantitative experiments show that our method well satisfies the demand for both local fine-grained attention and global spatiotemporal awareness in real-world scenarios.  
 
 Appendix:  


Decision: This paper presents a novel and promising approach to livestreaming product retrieval. The proposed SGMN method demonstrates strong performance on public datasets and is supported by comprehensive experiments. While there are some limitations in terms of literature coverage and experimental details, the overall contribution appears significant. Given the positive aspects highlighted by all reviewers and the addressed concerns in the rebuttal, I recommend acceptance for this paper. I strongly encourage the authors to address the noted limitations in the camera-ready version.

Reviews: review: paper_summary: paper_summary: This paper pays attention to the livestreaming product retrieval (LPR) problem. To provide a solution, a new method called Spatiotemporal Graphing Multi-modal Network (SGMN) is proposed. This method is applied to two publicly available datasets LPR4M and MF.
strength_weakness: strength_weakness: 1. The topic is livestreaming product retrieval, which is definitely interesting and challenging. This paper not only give its insights of this task but also proposes a new method for it.
2. The proposed method SGMN is technically sound. Extensive experiments in section 4 and supplementary have shown its effectiveness with both qualitative and quantitative results.
3. The experimental design is convincing. And the experimental results are sufficient.
4. For good reproducibility, the code of the proposed method has been provided for review. Besides, this paper promises that the code and models will be public.
limitations: limitations: 1. References to literature is inadequate and more closely related works should be included.
paper_summary: paper_summary: The paper proposes an one-stage spatiotemporal graph network for livesreaming product retrieval, called SGMN. Using text information obtained from video ASR and product items to guide model focus on the intended product. Using sequence-to-sequence graph learning to mitigate heterogeneity between videos and images and progessively enhance the fine-grained matching ability. Hard negative minging further improve the fine-grained representations. Quantitative and qualitative experiments demonstrate the superior performance of the proposed SGMN model.
strength_weakness: strength_weakness: 1. The proposed method achieves the best performance on the largescale benchmark dataset. Extensive quantitative and qualitative experiments prove the superiority of SGMN.
2. The visualization part for text guided attention and multi-modal hard negative minging is intersting and convincing.
3. Sufficient ablation experiments verified the effectiveness of the program design.
limitations: limitations: Since the R@mean metric of the LPR4M dataset is only 63.8, it suggests there is significant room for improvement. Please present some failed cases for analysis and provide an explanation of the limitations of this approach.
paper_summary: paper_summary: This paper proposes a novel approach to improve the retrieval of products in livestreaming contexts. The core of the method is the Spatiotemporal Graphing Multi-modal Network (SGMN), which integrates video, image, and text data to enhance the identification and differentiation of products. Experimental results on two public datasets have demonstrated the validity of the model.
strength_weakness: strength_weakness: 1.The content of the article is relatively abundant.  
2.The experiment is relatively adequate.
limitations: limitations: 1.How does livestreaming product retrieval differ from traditional product retrieval and recommendation? Can't the original method be applied to the task? In live streaming, images of products and purchase links are usually given, would it be simpler and more effective to use images directly for retrieval?  
2.The necessity of this task needs to be further emphasized, as well as practical application scenarios. To make the task easier to understand, a task definition is also needed.  
3.Each module needs to be introduced with the reason for its introduction to enhance the logic of the article. For example, why create video graph, image graph, and video-image graph?  
4.Many methodological and experimental details are not given in full, such as exactly how the TMC module is designed.  
5.Several works in related fields should be cited, e.g.,  
[1] Bi-directional Heterogeneous Graph Hashing towards Efficient Outfit Recommendation.  
[2] Contrastive Collaborative Filtering for Cold-Start Item Recommendation.  
6.A single Recall metric is not sufficient. In the retrieval domain, metrics such as accuracy, NDCG, etc., are usually also provided.
paper_summary: paper_summary: To tackle the three challenges, i.e., the recognition of intended products from distractor products, the video-image heterogeneity and the confusing products with subtle visual nuances, in livestreaming product retrieval, this paper proposes a spatiotemporal graphing multi-model network. In the proposed network, a text-guided attention mechanism leverages the spoken content of salespeople to focus on intended products, a spatiotemporal graph network is designed to solving the video-image heterogeneity, and a multi-model hard example mining is proposed to distinguish highly similar products.
strength_weakness: strength_weakness: The novelty of this paper primarily reflect in two main aspects: (a) This paper leverages the textual modality to identify the intended products from distractor product, and (b) this paper uses the spatiotemporal graph learning to alleviate the inter-domain misalignment in both spatial and temporal dimensions. 
Comprehensive experiments are conducted to prove the performance of the proposed method, including the quantitative comparison with other methods, the qualitative analysis of retrieval results, and the ablation study. The experimental results show that the proposed method outperforms the compared methods.
The paper is well wrote.
limitations: limitations: Some problems about experiments are listed:
How many images and videos are used in the training of the proposed method? Is the same images and videos used for the training of the compared methods?
Is the R@mean a popular metric in the livestreaming product retrieval, and how to compute this metric?
It is better to give more explanations about the data partitioning about product scale, visible duration, and number of products in table 4.