Title: PC-Net: Weakly Supervised Compositional Moment Retrieval via Proposal-Centric Network
Venue: NeurIPS 2025
Accepted: yes
Keywords: Video understanding

multimodal alignment

contrastive learning
Abstract: With the exponential growth of video content, aiming at localizing relevant video moments based on natural language queries, video moment retrieval (VMR) has gained significant attention. Existing weakly supervised VMR methods focus on designing various feature modeling and modal interaction modules to alleviate the reliance on precise temporal annotations. However, these methods have poor generalization capabilities on compositional queries with novel syntactic structures or vocabulary in real-world scenarios. To this end, we propose a new task: weakly supervised compositional moment retrieval (WSCMR). This task trains models using only video-query pairs without precise temporal annotations, while enabling generalization to complex compositional queries. Furthermore, a proposal-centric network (PC-Net) is proposed to tackle this challenging task. First, video and query features are extracted through frozen feature extractors, followed by modality interaction to obtain multimodal features. Second, to handle compositional queries with explicit temporal associations, a dual-granularity proposal generator decodes multimodal global and frame-level features to obtain query-relevant proposal boundaries with fine-grained temporal perception. Third, to improve the discrimination of proposal features, a proposal feature aggregator is constructed to conduct semantic alignment of frames and queries, and employ a learnable peak-aware Gaussian distributor to fit the frame weights within the proposals to derive proposal features from the video frame features. Finally, the proposal quality is assessed based on the results of reconstructing the masked query using the obtained proposal features. To further enhance the model's ability to capture semantic associations between proposals and queries, a quality margin regularizer is constructed to dynamically stratify proposals into high and low query-relevance subsets and enhance the association between queries and common elements within proposals, and suppress spurious correlations via inter-subset contrastive learning. Notably, PC-Net achieves superior performance with 54\% fewer parameters than prior works by parameter-efficient design. Experiments on Charades-CG and ActivityNet-CG demonstrate PC-Net’s ability to generalize across diverse compositional queries. Code is available at https://github.com/mingyao1120/PC-Net.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a new task, Weakly Supervised Compositional Moment Retrieval (WSCMR), in which models are trained only with video–query pairs (no temporal annotations) yet are expected to generalize to queries that contain unseen grammatical structures and vocabulary. To tackle this task the authors propose PC-Net, a “proposal-centric network” that (i) fuses global and frame-level features through a dual-granularity proposal generator, (ii) aggregates frame features with a learnable peak-aware Gaussian distributor and cross-modal contrastive alignment, and (iii) applies a quality-margin regularizer that contrasts high- and low-quality proposals. Experiments on Charades-CG and ActivityNet-CG show that PC-Net attains state-of-the-art mIoU while using fewer parameters than prior work.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n1. Solid engineering: each module is motivated by a concrete limitation in prior weak-supervised methods (coarse boundaries, fixed Gaussians, simple negatives).\n2. High-level pipeline is clearly illustrated (Fig. 2) and symbols are defined.\n3. Raises the bar for weak-supervised moment retrieval by adding compositional generalization—a realistic setting absent from prior work.\n\n**Weaknesses:**\n1. While the overall framework is new, the individual components feel like incremental improvements or combinations of existing techniques. The dual-granularity generator's use of global and local features is a common design pattern. The cross-modal semantic alignment (part of PFA) relies on a standard triplet contrastive loss. The \"learnable peak-aware Gaussian distributor\" is a minor modification to the Gaussian weighting used in prior work. The quality margin regularizer applies another layer of contrastive learning, which has become a standard tool. The novelty appears to lie more in the specific combination of these parts rather than in a truly groundbreaking new mechanism.\n2. Some design choices seem heuristic and lack strong theoretical or empirical justification. For instance, the Quality Margin Regularizer contrasts the average reconstruction losses of high- and low-relevance proposal sets. It's unclear why this is superior to methods that contrast individual proposals. Averaging could potentially wash out important learning signals from specific high-quality candidates. The paper does not provide an analysis to justify this specific design choice over simpler alternatives.\n3. The ablation study in Table 3 raises some concerns about the stability and contribution of individual modules. When added in isolation, the learnable peak-aware Gaussian (LPG) in setting (c) and the quality margin loss in setting (e) both lead to a decrease in performance on the Novel-Composition split compared to the baseline (a). The authors briefly mention this but do not provide a satisfying explanation. This suggests that the components are not robustly beneficial on their own and that the overall performance gain is the result of a carefully-tuned combination of modules, rather than a set of independently strong contributions.\n4. Gains on ActivityNet-CG are modest and sometimes below baselines on recall at stricter IoU—contradicting the “superior” claim.\n5. Writing is dense, with long sentences and inconsistent tense; several typos (“quries”) hamper readability.\n\nQUESTIONS:\n1. Fig. 4 only shows successful cases. Please provide examples where PC-Net fails (e.g., long-range compositional queries) and analyse why.\n2. Why is the whole-video reconstruction loss an appropriate decision boundary?\n3. What hyper-parameter budget and feature extractors were used when re-running CPL, QMN, PPS, etc.?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes the new task of weakly supervised compositional moment retrieval (WSCMR), building on weakly supervised video moment retrieval but applying the models to datasets with with novel syntactic structures or vocabulary. A new model, PC-Net, is proposed to tackle this task, which utilises a number of modules. It uses a dual-granularity proposal generator to produce proposals using a combination of global and local knowledge. It has a proposal feature aggregator with semantic alignment which includes the ability to adjust the Gaussian peak to account for diverse durations. Cross-modal semantic alignment is used to relate the video and text features. The proposals are assessed using query mask reconstruction, and it includes a quality margin regularizer to identify subtle semantic differences between query-relevant proposals. Performance is measured on the Charades-CG and ActivityNet-CG datasets. It achieves broadly improved performance over existing weakly supervised methods, albeit PPS is stronger on many metrics on the ActivityNet-CG benchmark.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n- It is beneficial to attempt the Compositional Video Moment Retrieval task in a weakly supervised manner to avoid the need for extensive annotations. Good from a real-world practicality perspective\n- The components generally make sense in accordance with the aim of improving the model's ability to handle diverse moment durations.\n- Performance on Charades-CG is strong across all metrics, while performance on ActivityNet-CG is also quite strong.\n- The ablations are thorough and display the importance of each component.\n- The model shows good efficiency compared with previous methods\n\n**Weaknesses**\n- The method contains many parts and it is a bit difficult to understand parts of the paper\n- For example, equations 5&6 for the redistribution of weights of the Gaussian distribution. I found it difficult to understand why it is formulated this way, or what the motivation for these equations having this form is. Line 194-196 does not make sense to me grammatically or semantically, \"η_i(t) through the distance discrimination mechanism, the peak area and the edge area are divided to achieve dynamic weight allocation\". It doesn't help that σ_i and x_t are also not described and only defined in the appendix. Even in the appendix, the purpose of σ_i is not really explained. \n- In line 154, c and w are not defined. They are defined much later in the paper but at this point it is potentially confusing.\n- I also found the lack of definition of the complement (line 215) and the origin of the two negative proposal reconstruction losses (line 217) to be confusing.\n- PPS consistently outperforms PC-Net on R1@m metrics on ActivityNet-CG, which perhaps indicates that PC-Net is not as good at achieving tight moment localisations on this dataset.\n- (minor) Line 223 it should say Figure 2(c) rather than Figure 2(e) I assume.\n\nQUESTIONS:\n- (line 215) Is the complement just the part of the video outside of the proposal? So is this why there are two parts to the complement (n1 and n2), one for either side of the proposal?\n- Overall, while I think this is a promising paper and a promising task, the main issues which I have are with readability and the difficulty in understanding parts of the paper, along with some concerns over performance on ActivityNet-CG on R1@m compared to PPS.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a proposal-centric network (PC-Nets) for locating video moments in long videos according to natural language queries. For better retrieval of video moments according to queries with novel structure and vocabulary, the PC-Net features a dual-grained proposal generator, a learnable peak-aware Gaussian distributor, and a quality margin regularizer. The proposed model has been extensively evaluated on multiple benchmark datasets across various settings. It is shown that the PC-Net achieves competitive performance to existing solutions with fewer parameters.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n+ The paper is well written and easy to follow\n+ The weakly-supervised compositional moment retrieval studied in this paper is challenging but practical, which requires generalisation to unseen scenarios while avoiding labour-intensive annotations of moment boundary for training\n+ The proposed method is technically solid and well justified. Various components and designs have been made to improve video moment retrieval from different perspectives, from proposal generation to cross-modal feature aggregation and proposal quality assessment.\n+ The experiments are comprehensive with detailed ablation studies.\n\nWeaknesses:\n+ About motivation, the WSCMR task is explicitly targeted in this paper but it seems like most proposed designs are also valid for general video moment retrieval (VMR). I failed to find strong connections between the designs of PC-Net and the unique challenges of WSCMR. It might be helpful if the authors can spell out and briefly summarise how the challenges unique to WSCMR are addressed by the proposed model. Otherwise, it should be ok to expand the scope to general VMR instead of a specific scenario.\n+ About clarity, given that PC-Net is composed of both common components widely adopted in the field and newly proposed ones, I found it hard to identify the latter, given all components are elaborated to a certain extent.\n\nQUESTIONS:\nI am generally happy with this submission. It will be helpful if the authors can clarify why their proposed model is targeting WSCMR rather than general VMR as suggested in the weaknesses section.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a new task—Weakly Supervised Compositional Moment Retrieval (WSCMR)—and proposes a novel architecture, PC-Net, to tackle it. The authors target the generalization challenge in moment retrieval, especially under weak supervision and compositional queries with novel syntactic structures or vocabulary. PC-Net incorporates a dual-granularity proposal generator, a semantic-aligned feature aggregator, and a quality margin regularizer. Experiments on Charades-CG, ActivityNet-CG, TACoS and Ego4D demonstrate strong performance and generalization with high parameter efficiency.\n\nSTRENGTHS AND WEAKNESSES:\nPaper Strengths\n\nNovel Task Definition: Clearly motivated definition of WSCMR addressing practical scalability issues.\nMethodological Innovation: The integration of dual-granularity proposal generation and semantic-aware aggregation is well-executed and contributes to generalization.\n\nPerformance and Efficiency: Achieves state-of-the-art results with significantly fewer parameters than existing baselines.\n\nThorough Evaluation: Demonstrates effectiveness and generalization across multiple datasets (Charades-CG, ActivityNet-CG, TACoS, Ego4D) with thorough ablations and visualizations.\n\nClarity and Structure: The paper is well-organized with sufficient technical detail and intuitive figures.\n\nMajor Weaknesses\n\nLimited Analysis of Weak vs. Full Supervision: While the model demonstrates strong performance, the paper lacks a deeper analysis of how weak supervision compares to full supervision in terms of interpretability, robustness, and training stability. A more thorough exploration of this trade-off would strengthen the contribution.\n\nQUESTIONS:\n1. The paper claims existing methods poorly generalize to compositional queries with novel syntactic structures or vocabulary, but the 'novel queries' in experiments only use simple combinatorial patterns. Queries with higher semantic richness (e.g., diverse sentences generated by LLMs) should be tested.\n\n2. The model structure appears over-engineered and insufficiently concise. How to validate the rationality of these structures? For example, disassemble the modules into existing frameworks to verify their generalizability.\n\n3. The performance improvement seems marginal despite lower parameters. Could it achieve greater gains at the same parameter level?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-05"}