Title: HopaDIFF: Holistic-Partial Aware Fourier Conditioned Diffusion for Referring Human Action Segmentation in Multi-Person Scenarios
Venue: NeurIPS 2025
Accepted: yes
Keywords: Human action segmentation

video understanding
Abstract: Action segmentation is a core challenge in high-level video understanding, aiming to partition untrimmed videos into segments and assign each a label from a predefined action set. Existing methods primarily address single-person activities with fixed action sequences, overlooking multi-person scenarios. In this work, we pioneer textual reference-guided human action segmentation in multi-person settings, where a textual description specifies the target person for segmentation. We introduce the first dataset for Referring Human Action Segmentation, i.e., RHAS133, built from 133 movies and annotated with 137 fine-grained actions with 33h video data, together with textual descriptions for this new task. Benchmarking existing action segmentation methods on RHAS133 using VLM-based feature extractors reveals limited performance and poor aggregation of visual cues for the target person. To address this, we propose a holistic-partial aware Fourier-conditioned diffusion framework, i.e., HopaDIFF, leveraging a novel cross-input gate attentional xLSTM to enhance holistic-partial long-range reasoning and a novel Fourier condition to introduce more fine-grained control to improve the action segmentation generation. HopaDIFF achieves state-of-the-art results on RHAS133 in diverse evaluation settings. The dataset and code are available at https://github.com/KPeng9510/HopaDIFF.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper pioneers a novel task—Referring Human Action Segmentation (RHAS)—which extends traditional single-person temporal action segmentation to multi-person, textual reference-guided scenarios that are inherently more unconstrained and unstructured. The authors introduce RHAS133, the first dataset for this task, comprising 133 untrimmed movies with 542 annotated individuals. Each person is described with textual references and labeled with fine-grained action annotations. Furthermore, the authors propose HopaDIFF, a holistic-partial aware Fourier-conditioned diffusion framework that integrates dual-branch modeling, HP-xLSTM with bidirectional cross-attention (BCA), diffusion-based generation, and frequency-domain conditioning. The proposed approach significantly outperforms both conventional temporal action segmentation and referring action recognition baselines.\n\nSTRENGTHS AND WEAKNESSES:\nQuality: This work initiates the RHAS task, proposes the RHAS133 dataset, and develops the HopaDIFF framework. The technical methodology is sound, and experiments demonstrate state-of-the-art performance in this emerging area. However, the paper lacks sufficient ablation studies and does not provide an in-depth analysis of each innovative component.\nClarity: The motivation and overall structure are clearly presented. However, critical architectural details are missing, limiting reproducibility. For instance, the interaction mechanism between the two branches within HP-XLSTM via BCA, the internal structure and information flow of the denoising model at each diffusion step, and how spatial, temporal, and frequency embedding conditions guide the model are not sufficiently explained.\nSignificance: This work pioneers the RHAS task, expanding action segmentation into a more complex, textual-reference-guided multi-person scenario, and introduces a corresponding dataset and framework. The contribution is of high significance to the field.\nOriginality: The paper introduces a novel research task and a new dataset. While the proposed framework includes several innovative modules, many components (e.g., diffusion model, xLSTM) are adapted or combined from existing works. Thus, while the task-level originality is high, the architectural novelty is relatively limited.\n\nQUESTIONS:\n1.The end of the Introduction section would benefit from a clear summary of the main contributions of this work.\n\n2.Regarding the dataset, there is a significant class imbalance and long-tailed distribution: some action categories contain over 1e+6 frames, while others have fewer than 1e+3. Although such imbalance is common in many datasets, the authors should explain the extent of this imbalance in detail, discuss its impact on model performance, and describe any mitigation strategies adopted—or justify the choice not to address it. This is critical for ensuring transparency and the credibility of the results.\n\n3.The Methodology section lacks sufficient architectural details for several modules. Specific concerns include:\nA. The model appears to follow a two-stage pipeline—first extracting Holistic and Partial features, then performing subsequent modeling—but this is not clearly stated or described in the paper.\nB. For the denoising diffusion model, the internal structure of the denoising function $M_{\\theta}$ is not detailed. The paper does not explain how the model processes the noisy input, timestep, and the spatial, temporal, and frequency embedding conditions during denoising.\nC. The Bidirectional Cross-Attention (BCA) module lacks a thorough explanation of its mechanism and role within the architecture.\n\n4.Concerns of the Innovation:\nA. While the use of existing components (e.g., VLM and Grounding DINO for feature extraction) is technically reasonable, key modules in the proposed framework—such as the diffusion-based segmentation and xLSTM—are borrowed directly from prior work. The paper allocates significant space and equations to these existing methods, while offering limited discussion of the truly novel aspects such as dual-branch interaction and frequency-conditioned denoising, which weakens the perceived originality.\nB. The explanations of the novel components remain superficial. For instance: 1) The rationale for using frequency-based conditioning is not sufficiently elaborated; the current explanation (\"primarily leverage features in the temporal and spatial domains...\") does not justify why frequency information is beneficial for denoising. 2) The mechanism of BCA-based information exchange is not clearly explained.  3) The decision to keep the two branches separate (rather than fuse them) and train them independently is not justified, nor is this design explored through ablation studies.\n\n5.Efficiency Issues: The paper lacks a discussion of efficiency in terms of parameter count, FLOPs, and training/inference time. The use of two VLM backbones, one Grounding DINO, two ASFormer encoders and decoders, and a diffusion model—each computationally heavy—raises concerns about the feasibility and efficiency of the two-stage pipeline.\n\n6.Performance and Fairness Issues: In Table 5, the performance of the model without HCMGB is still lower than that of FACT, despite both relying solely on holistic features. This raises questions about the comparative effectiveness of other modules in HopaDIFF, and an explanation is needed.\n\n7.Experimental Issues: \nA.Why do the test set results exceed those on the validation set in most experiments?\nB. The overall EDIT scores are low; a justification is needed.\nC. Table 5 lacks an ablation study for removing the diffusion model entirely.\nD. There are no component-level ablation studies examining the internal structure or parameters of each individual module.\nE. There is no analysis of training each branch independently or of fusing the branches—such ablations are essential for validating the dual-branch design.\n\n8.Errors and Inconsistencies:\nLine 104: \"state of the art\" should be hyphenated as \"state-of-the-art\".\nLines 85–88: The sentence beginning with “since...” and ending with “which...” is overly long and should be split for clarity.\nLine 283: The reported F1@50 score is inconsistent with the corresponding table.\nTable 5 and Lines 310 & 315 refer to “w/o DCT-cond,” which suggests Discrete Cosine Transform, while the earlier sections describe frequency conditioning based on DFT (Discrete Fourier Transform); this inconsistency needs to be clarified.\n\nThis work presents a task and framework with acceptable levels of innovation and effort. However, there remain multiple critical concerns regarding methodology, clarity, originality, and experimental completeness. I assign a Borderline Accept and will adjust my score based on the authors’ responses and revisions.\n\nLIMITATIONS:\nThe paper discusses dataset limitations and future directions in both the conclusion and supplementary materials, and provides a fairly comprehensive analysis of the societal impact, including both positive and negative aspects. However, it lacks a clear discussion of the limitations of the proposed framework itself.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 4, "ORIGINALITY": 2, "DATE": "2025-06-19"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce Referring Human Action Segmentation (RHAS), a novel task for segmenting the actions of a specific individual in complex, multi-person videos guided by a textual reference. To facilitate research in this new area, the authors present the RHAS133 dataset. The primary technical contribution is HopaDIFF, a diffusion-based framework designed to address this task. The model integrates both holistic (full-frame) and partial (person-specific) information through a dual-branch architecture. Its key innovations include a novel cross-input gate attentional xLSTM to enhance long-range temporal reasoning and a Fourier-based conditioning mechanism to exert more precise control over the generation of action segments.\n\nSTRENGTHS AND WEAKNESSES:\n## Paper Strengths\n- **Well-Defined Motivation**: Most existing human action segmentation methods are tailored for single-person scenarios [1,2,3,4,5], where actions occur in a fixed sequence defined by a specific task. To address the limitations of these approaches in more complex settings, the authors propose a new task called Referring Human Action Segmentation (RHAS), which focuses on multi-person untrimmed videos guided by textual descriptions.\n- **New Dataset Introduction**: The authors present a new dataset named RHAS133, which includes 133 untrimmed videos featuring a total of 542 individuals. Each person of interest is annotated with a corresponding textual reference, supporting the RHAS task.\n- **Introduction of diverse baselines**: To highlight the difficulty of the RHAS task, the authors adapt and evaluate a range of existing state-of-the-art methods for temporal action segmentation, originally designed for single-person videos, within the RHAS setting. These adapted methods serve as a baseline for comparison and demonstrate the novel challenges of the task.\n- **Introduction and Detailed Explanation of HopaDIFF**: The authors propose a novel framework named HopaDIFF (Holistic-Partial aware Fourier-conditioned Diffusion Framework). It includes two branches: a holistic branch that captures the global context from the entire video, and a partial branch that uses GroundingDINO to locate the referred individual, generating a cropped video for fine-grained representation via a Vision-Language Model (VLM). To support efficient reasoning over long videos and integrate holistic and partial information, the authors introduce the HP-xLSTM module, a modified version of the mLSTM [6] module that incorporates a bidirectional cross-attention (BCA) mechanism at the input gate. Uniquely, this framework integrates Fourier conditioning, offering finer control over the generative process by operating in the frequency domain.\n- **Effectiveness of HopaDIFF**: As shown in Section 5 of the paper, HopaDIFF significantly outperforms baseline methods, demonstrating strong performance gains. The authors conduct an ablation study, analyzing the impact of different VLM backbones and showing that while backbone quality matters, HopaDIFF consistently outperforms competitors. Additional ablations highlight the individual contributions of each component within the framework.\n\n## Paper Weaknesses\n- **Absence of Qualitative Results**: The paper's evaluation is entirely quantitative. While the reported metrics are strong, the paper would be significantly improved by including qualitative results. Visual comparisons are essential for building intuition about the model's performance on this complex task. It would be beneficial for the authors to include:\n    - Side-by-side visualizations of the segmentation output from HopaDIFF compared to key baselines on selected video segments.\n    - A dedicated analysis of failure cases to provide insight into the model's current limitations.\n- **Limited Scope of Ablation Studies**: The provided ablation study effectively demonstrates the contribution of each component within the HopaDIFF framework. However, a more compelling analysis would involve testing the modularity of these components by integrating them into the baseline architectures. This would provide stronger, more isolated evidence of their value. Specific suggestions include:\n    - *Integrating the Partial Branch into FACT [1]*: Since the FACT [1] baseline also uses a dual-branch architecture, its action branch could be replaced with the proposed GroundingDINO-based partial reasoning branch. This would directly test the effectiveness of this component outside of the full HopaDIFF model.\n    - *Augmenting ActDiff with Frequency Conditioning*: The authors note that the ActDiff [2] baseline operates on temporal and spatial features. A valuable experiment would be to augment ActDiff [2] with the proposed DFT-based frequency conditions to determine if this mechanism provides a general performance benefit to diffusion-based action segmentation models.\n    - *Comparing HP-xLSTM to Standard Attention*: The ablation study shows a performance drop when removing the HP-xLSTM module. However, to better justify its complexity, it would be insightful to replace HP-xLSTM with a more standard cross-attention mechanism (similar to that used in FACT [1]) to see if the advanced memory and gating features of xLSTM [6] are truly necessary for achieving state-of-the-art results.\n\n$\\underline{\\text{Minor Revision}}$:\nIn Figure 2, there is a typo: Patial -> Partial.\n\n[1] Lu, Zijia, and Ehsan Elhamifar. \"Fact: Frame-action cross-attention temporal modeling for efficient action segmentation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n[2] Liu, Daochang, et al. \"Diffusion action segmentation.\" Proceedings of the IEEE/CVF international conference on computer vision. 2023.\n\n[3] Bahrami, Emad, Gianpiero Francesca, and Juergen Gall. \"How much temporal long-term context is needed for action segmentation?.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.\n\n[4] Gan, Ziliang, et al. \"ASQuery: A Query-based Model for Action Segmentation.\" 2024 IEEE International Conference on Multimedia and Expo (ICME). IEEE, 2024.\n\n[5] Peng, Kunyu, et al. \"Referring atomic video action recognition.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024.\n\n[6] Beck, Maximilian, et al. \"xLSTM: Extended Long Short-Term Memory.\" Advances in Neural Information Processing Systems 37 (2025): 107547-107603.\n\nQUESTIONS:\nThe paper presents a novel and effective framework, but further clarification and analysis would benefit a few points.\n- Could the authors provide qualitative results to offer a more complete picture of the model's performance?\n- Could the authors replace HP-xLSTM with a more standard cross-attention mechanism (similar to that used in FACT [1]) to see if the advanced memory and gating features of xLSTM [2] are truly necessary for achieving state-of-the-art results?\n\nOther details are reported in the weaknesses.\n\n[1] Lu, Zijia, and Ehsan Elhamifar. \"Fact: Frame-action cross-attention temporal modelling for efficient action segmentation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n[2] Beck, Maximilian, et al. \"xLSTM: Extended Long Short-Term Memory.\" Advances in Neural Information Processing Systems 37 (2025): 107547-107603.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces the first work for addressing temporal action segmentation in multi-person scenarios. \nThe authors constructed a new dataset specifically for this new task. \nThey further propose a diffusion-based solution that integrates both global (holistic) and local (partial) image features, with the latter capturing the specific target subject being described. \nThe solution also leverages frequency-domain information to enhance diffusion-based action segmentation.\nExperiments on the proposed dataset show that the proposed solution outperforms baseline methods. \nThe ablation study further verifies the benefits of incorporating partial image features and leveraging frequency-domain information.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**: \n- The paper addresses the important task of multi-person action segmentation, which has not been discussed in previous works.\n- The paper contributes a new dataset designed for this important task.\n- The proposed solution leverages image features of description-conditioned image crops and captures frequency domain information, which are proven to be effective.\n\n**Weakness**:\n- The paper lacks necessary detailed explanations regarding the implementation of existing methods and ablated settings. Regarding this issue, I have further explained in the question session.\n- I consider that the authors should discuss a naive solution that first crops the target subject based on the input textural reference and then solves by applying existing single-person action segmentation frameworks on these image crops.  \n\nTherefore, I vote for \"borderline reject\" for the current version. I hope the authors could address my concerns listed in the weakness and questions sessions.\n\nQUESTIONS:\n1. When training and testing existing methods (i.e., FACT, ActDiff, ASQuery, LTContent, RefAtomNet), what are the inputs for these methods? \n    - Do they take the original images containing multiple people as input, or do they use image crops featuring only the person being described? \n    - Or I would say, what are the performances of existing methods if we first crop the target person based on textural description and then feed these image crops to these existing methods?\n2. As a follow-up to the first question, what if considering an ablated setup of the proposed solution, where this ablated setup does not leverage holistic features? Examining its performance could illustrate the benefits of capturing interactions among people through holistic features, which could further justify the importance of the proposed task.\n3. Regarding the evaluation protocol, if there are multiple ground truth action labels for the same person at the same timestamp (e.g., sitting and watching for person 1, Fig. 1), how do the authors determine whether the action segmentation is correct? Should the method output all ground truth labels, or is it sufficient to output only one of them?\n4. Why does leveraging frequency-domain information benefit action segmentation? It would be beneficial if the authors could explain the motivation for this design choice.\n5. For the ablated setting _w/o HP-xLSTM_ and _w/o BCAF_, compared to the full solution, what are their differences in the detailed framework design?\n\nLIMITATIONS:\nYes, the authors have adequately addressed the limitations and potential negative societal impact in their submission.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces **Referring Human Action Segmentation (RHAS)**, a novel task that focuses on segmenting the actions of specific individuals in multi-person untrimmed videos based on textual references. To support this task, the authors present **RHAS133**, a new dataset comprising 133 movies and 542 annotated individuals. They also propose **HopaDIFF**, a diffusion-based framework that integrates holistic and partial visual reasoning with Fourier-based conditioning to enhance action segmentation performance.\n\nSTRENGTHS AND WEAKNESSES:\n### **Strengths**\n\n1. **Novel and Well-Motivated Task**: The RHAS task addresses a real and underexplored limitation in existing action segmentation methods—namely, the over-reliance on single-person scenarios with fixed protocols. The integration of referring expressions into multi-person action segmentation is novel and practically relevant, making the problem both original and impactful.\n\n2. **Valuable Dataset Contribution**: The RHAS133 dataset is, to the best of our knowledge, the first specifically designed for referring human action segmentation. It provides rich annotations including 137 fine-grained action classes, natural language referring expressions, and multi-label annotations, making it a valuable and reusable benchmark for the community.\n\n3. **Reasonable Technical Approach**: The proposed dual-branch (holistic-partial) architecture is well-motivated, aiming to jointly capture global contextual cues and target-specific fine-grained motion patterns. The use of HP-xLSTM with a cross-input gating mechanism provides a sensible way to facilitate information exchange between branches.\n\n4. **Thorough Experimental Evaluation**: The experimental section is comprehensive, including a wide range of adapted baselines from both action segmentation and referring action recognition literature. In addition, the evaluation under both random partition and cross-movie splits offers useful insights into model generalization across settings.\n\n---\n\n### **Weaknesses**\n\n1. **Limited Technical Novelty**: The overall architecture mainly combines existing components—such as diffusion models, xLSTM, and DFT-based conditioning—rather than proposing fundamentally new algorithmic techniques. As such, the contribution is more integrative than innovative.\n\n2. **Technical and Methodological Concerns**: The model involves several components (dual branches, HP-xLSTM, BCA, and DFT conditioning), but the paper lacks sufficient justification or ablation for each design choice. Furthermore, there is no discussion or analysis of computational complexity, inference latency, or comparison with more lightweight alternatives, which raises concerns about scalability and deployment.\n\nQUESTIONS:\nPlease refer to **Strengths And Weaknesses**.\n\nLIMITATIONS:\nPlease refer to **Strengths And Weaknesses**.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-03"}