C3-OWD: A Curriculum Cross-modal Contrastive Learning Framework for Open-World Detection

## Abstract

Abstract Object detection has advanced significantly in the closed-set setting, but real-world deployment remains limited by two challenges: poor generalization to unseen categories and insufficient robustness under adverse conditions. Prior research has explored these issues separately: visible-infrared detection improves robustness but lacks generalization, while open-world detection leverages vision-language alignment strategy for category diversity but struggles under extreme environments. This trade-off leaves robustness and diversity difficult to achieve simultaneously. To mitigate these issues, we propose C3-OWD , a curriculum cross-modal contrastive learning framework that unifies both strengths. Stage 1 enhances robustness by pretraining with RGBT data, while Stage 2 improves generalization via vision-language alignment. To prevent catastrophic forgetting between two stages, we introduce an Exponential Moving Average (EMA) mechanism that theoretically guarantees preservation of pre-stage performance with bounded parameter lag and function consistency. Experiments on FLIR, OV-COCO, and OV-LVIS demonstrate the effectiveness of our approach: C3-OWD achieves 80.1 80.1 AP 50 on FLIR, 48.6 48.6 AP Novel 50 {}^{50}_{\text{Novel}} on OV-COCO, and 35.7 35.7 mAP r on OV-LVIS, establishing competitive performance across both robustness and diversity evaluations. Code available at: https://github.com/justin-herry/C3-OWD.git .

## Introduction

Object detection (Zong, Song, and Liu 2023; Zhu et al. 2020; Zhao et al. 2023; Lv et al. 2024) is a fundamental task in computer vision and has achieved remarkable progress under the closed-set setting, where models are trained and evaluated on a fixed set of categories.

Despite their efficiency and accuracy in standard benchmarks, traditional detectors face two key challenges when deployed in the real world:

- •
limited generalization to unseen categories
- •
insufficient robustness under adverse environmental conditions, as shown in Fig [1](https://arxiv.org/html/2509.23316v1#Sx1.F1), such as low illumination, fog, etc.

To address these issues, two research directions have recently gained attention. On the one hand, visible–infrared object detection (RGBT-OD) (Shen et al. 2024a; Devaguptapu et al. 2019; Medeiros et al. 2024; Lee, Park, and Park 2024) introduces complementary thermal cues that significantly improve robustness under extreme conditions. However, RGBT models are still restricted to closed-set categories and thus exhibit limited generalization. On the other hand, open-world detection (OWD) (Gu et al. 2021a; Zang et al. 2022; Zhong et al. 2022; Zang et al. 2022; Wu et al. 2024a) extends detectors beyond fixed taxonomies by leveraging vision–language alignment strategy such as CLIP (Radford et al. 2021) and GLIP (Li et al. 2022a), enabling recognition of novel categories in dynamic and evolving environments. Yet, OWD systems are typically trained on natural images and struggle with robustness when applied to challenging conditions. Therefore, current research reveals an inevitable trade-off: methods focusing on robustness often lack diversity, while methods emphasizing diversity compromise robustness.

Can we develop a unified framework that simultaneously achieves both robustness under extreme conditions and open-vocabulary generalization, thereby breaking the current trade-off between diversity and robustness in object detection?

In this paper, we propose a unified paradigm, C3-OWD, to simultaneously address both challenges through a curriculum cross-modal contrastive learning framework. The training is divided into two stages. In Stage 1, we enhance robustness by pretraining the detector on RGBT data such as FLIR dataset (Systems 2018), enabling the model to resist severe environmental degradation. In Stage 2, we inject semantic priors from text via vision–language alignment, which improves generalization to unseen categories and strengthens the grounding of visual regions with textual descriptions.

A key difficulty in curriculum learning is catastrophic forgetting, where Stage 2 training may override the robustness learned in Stage 1. To alleviate this, we introduce an exponential moving average (EMA) mechanism (Karras et al. 2023; Lee et al. 2024; Li et al. 2024). Our theoretical analysis demonstrates that the EMA mechanism preserves pre-stage performance with bounded parameter lag, ensuring function consistency between momentum and online branches and preventing catastrophic forgetting when adapting from small to large datasets.

Our main contributions are summarized as follows:

- •
We present C3-OWD, a unified cross-modal curriculum learning paradigm that integrates RGBT robustness with open-vocabulary generalization capabilities. Unlike traditional methods that suffer from modality bias, our approach dynamically balances multi-modal information through progressive learning, reducing the trade-off between robustness and diversity to achieve better adaptation to diverse environmental conditions.
- •
We provide rigorous theoretical foundations proving that our Exponential Moving Average (EMA) mechanism effectively prevents catastrophic forgetting with bounded parameter lag and guaranteed function consistency, establishing mathematical guarantees for stable knowledge retention in progressive multi-modal learning.
- •
Extensive experiments validate the effectiveness of C3-OWD, C3-OWD achieves $80.1$ AP^50 on FLIR, $48.6$ AP${}_{\text{Novel}}^{50}$ on OV-COCO, and $35.7$ mAPr on OV-LVIS, achieving competitive results compared to prior state-of-the-art methods.

Figure: Figure 1: Visible-light imaging faces challenges like exposure mask, low brightness, and multi-scale target intensity, leading to information loss. Infrared imaging provides robust solutions, highlighting the necessity of multi-modal fusion.
Refer to caption: x1.png

## Related Work

Open-World Detection (OWD) OWD focuses on detecting and learning unknown objects that are not annotated in the training set (Ma et al. 2023; Xi et al. 2024). During inference, detectors identify potential unknowns, which are then annotated and incrementally added as new categories. Early methods, such as ORE (Joseph et al. 2021), enhanced Faster R-CNN with clustering and energy-based classifiers, while transformer-based approaches like OW-DETR (Gupta et al. 2022) adopted pseudo-labeling but often generated noisy labels. Extensions such as PROB (Zohar, Wang, and Yeung 2023) introduced probabilistic objectness modeling, yet calibration issues remained. Recently, large visual models (e.g., SAM (Kirillov et al. 2023)) have further advanced OWOD through pseudo-labeling and knowledge distillation, though challenges in reliable unknown detection still persist.

RGBT Object Detection To address the limitations of relying solely on RGB images for object detection, some studies have introduced a thermal modality, leading to the development of RGBT object detection (Zhou, Chen, and Cao 2020; Sun et al. 2022a). Some methods use uncertainty or confidence metrics to balance RGB and thermal fusion (Kim, Park, and Ro 2021; Li et al. 2023), while others adjust reliance on thermal inputs based on illumination levels (Guan et al. 2019; Li et al. 2019). Attention-based RGBT networks have also been proposed to further enhance modality integration (Shen et al. 2024b; Yuan and Wei 2024). In this work, we employ RWKV as the backbone and design tailored training strategies to fully exploit the advantages of RGBT data.

Receptance Weighted Key Value (RWKV) RWKV (Peng et al. 2023) has emerged as a promising paradigm for sequential modeling, combining the efficiency of recurrent networks with the scalability of transformers (Li et al. 2025; Hou and Yu 2024). By introducing time-shifted receptance gates and exponentially decaying key projections, it achieves linear-time complexity and efficient memory usage (Peng et al. 2025). These advantages have led to extensions in multimodal integration (Yang et al. 2025b; Fei et al. 2024). For example, PointRWKV (He et al. 2025) enhances geometric feature extraction in 3D point clouds, while Vision-RWKV (Duan et al. 2024a) improves high-resolution image understanding beyond traditional ViTs. In this work, we leverage RWKV to efficiently fuse two visual modalities and text, fully exploiting its potential in multimodal learning.

## Method

### Overview

Our goal is to enhance open-vocabulary detection under challenging conditions by introducing C3-OWD, a curriculum cross-modal contrastive learning framework. As illustrated in Fig. [2](https://arxiv.org/html/2509.23316v1#Sx3.F2), our approach is built upon two-stage designed architecture. Stage 1 leverages RGBT (visible–thermal infrared) datasets to strengthen robustness. Stage 2 trains on COCO with Bi-Momentum Contrastive Alignment and Text-Modulated Deformabale Attention to achieve open-vocabulary detection with enhanced semantic grounding.

Figure: Figure 2: Overview of C3-OWD. (a) Two-stage training. Stage 1 leverages RGBT (visible–thermal infrared) datasets to strengthen robustness via RWKV. Stage 2 trains on COCO with Bi-Momentum Contrastive Alignment and Text-Modulated Deformabale Attention to achieve open-vocabulary detection with enhanced semantic grounding. (b) Architecture components. The Robust Image Feature Block performs efficient sequence modeling and multimodal feature fusion with linear-time complexity using the RWKV architecture. The cross-modal exchange mechanism enables bidirectional information flow and adaptive feature recalibration across modalities. The text-modulated sampling module steers deformable sampling locations with text features via attention, injecting stronger textual priors.
Refer to caption: x2.png

### Stage 1 - Multi-modal Robustness Enhancement

As shown in Fig [2](https://arxiv.org/html/2509.23316v1#Sx3.F2) Stage 1, the primary objective of Stage 1 is to establish environmentally robust feature representations through complementary information from RGB and infrared modalities. Unlike traditional methods that employ simple feature concatenation or weighted fusion, we design an adaptive cross-modal interaction mechanism based on RWKV.

Dual-Modal Feature Extraction: Given paired RGB-infrared images $(I_{rgb},I_{ir})\in\mathbb{R}^{H\times W\times 3}$, we extract multi-scale features through a shared-weight CNN backbone:

$$ $\mathcal{F}_{rgb}=\{F_{rgb}^{l}\}_{l=2,3,4},\quad\mathcal{F}_{ir}=\{F_{ir}^{l}\}_{l=2,3,4}.$ (1) $$

where $F_{*}^{l}\in\mathbb{R}^{B\times C_{l}\times H_{l}\times W_{l}}$ denotes the feature map at layer $l$.

VRWKV-based Adaptive Fusion: To achieve efficient cross-modal interaction, we adopt VRWKV (Duan et al. 2024b) as the base architecture and extend it with a bidirectional attention mechanism to capture global dependencies. For the $t$-th token, the attention output is computed as:

$$ $\begin{aligned} \mathrm{Bi\mbox{-}WKV}(K,V)_{t}=\frac{\sum^{T-1}_{i=0,i\neq t}e^{-(|t-i|-1)/T\cdot w+k_{i}}v_{i}+e^{u+k_{t}}v_{t}}{\sum^{T-1}_{i=0,i\neq t}e^{-(|t-i|-1)/T\cdot w+k_{i}}+e^{u+k_{t}}}.\end{aligned}$ (2) $$

where $w$ is a learnable position decay parameter, $u$ is the current token importance weight, and $k_{i}$, $v_{i}$ are the key and value vectors respectively. This bidirectional mechanism allows each token to simultaneously consider both forward and backward contextual information.

RGB and Infrared features interact through the VRWKV-Block (Duan et al. 2024b) to generate modality-aware enhanced representations:

$$ $\mathcal{F}_{robust}=\text{VRWKV-Block}(\mathcal{F}_{rgb},\mathcal{F}_{ir}).$ (3) $$

where the VRWKV-block consists of an encoder-decoder architecture with cross-modal interaction mechanisms. The encoder processes multi-scale RGB-Thermal features, while the decoder refines the representations, enabling effective fusion of complementary information from both modalities.

During this stage, we employ standard detection losses including classification loss $\mathcal{L}_{cls}$ and regression loss $\mathcal{L}_{reg}$, directly supervised on paired RGB-infrared data. This pre-training strategy enables the model to learn robust feature representations across varying illumination conditions, establishing a strong foundation for subsequent open-world adaptation. The full details can be found in Algorithm [1](https://arxiv.org/html/2509.23316v1#alg1).

### Stage 2 - Vision-Language Generalization Alignment

As shown in Fig [2](https://arxiv.org/html/2509.23316v1#Sx3.F2) Stage 2, in the second stage, we perform comprehensive vision-language alignment through three interconnected components: semantic enhancement fusion, text-modulated deformable attention, and bi-momentum contrastive learning. This stage enriches visual representations with semantic priors while maintaining efficient open-vocabulary detection capabilities.

#### Semantic Enhancement Fusion

We begin by enriching visual features with semantic priors through a hierarchical vision-language fusion module. Visual features $\mathbf{C}=\{C_{2},C_{3},C_{4}\}$ from the backbone (where $C_{2}\in\mathbb{R}^{B\times 256\times H\times W}$) are projected into a shared space via linear projection layers:

$$ $C^{\prime}_{i}=\text{Linear}(C_{i}),\quad i\in\{2,3,4\}$ (4) $$

The scaled embeddings are concatenated as $\mathbf{C}=\text{Concat}(C^{\prime}_{2},C^{\prime}_{3},C^{\prime}_{4})\in\mathbb{R}^{B\times L\times D}$, where $L$ is the sequence length and $D$ is the embedding dimension. Simultaneously, fine-grained inputs are extracted from $C_{2}$ through patch embedding:

$$ $\mathbf{M},(H_{p},W_{p})=\text{PatchEmbed}(C_{2}),$ (5) $$

where $\mathbf{M}\in\mathbb{R}^{B\times N\times D}$ are patch tokens and $(H_{p},W_{p})$ is the patch resolution.

We then perform $L$ rounds of bidirectional cross-attention between $\mathbf{M}$ and text embeddings $\mathbf{T_{clip}}\in\mathbb{R}^{B\times CLA\times D}$ (where $CLA$ is the number of classes), followed by cross-modal exchange:

$$ $\mathbf{M}_{out},\mathbf{T}_{out}=\text{CrossModalExchange}(\mathbf{M},\mathbf{T_{clip}}),$ (6) $$

where the CrossModalExchange module performs bidirectional feature fusion using RWKV-based architecture. Specifically, the module computes cross-modal interactions through two parallel pathways:

$$ $\mathbf{V}_{I}=\text{RWKV}_{I\to T}(R=\mathbf{M},K=\mathbf{T_{clip}},V=\mathbf{T_{clip}}),$ (7) $$

$$ $\mathbf{V}_{T}=\text{RWKV}_{T\to I}(R=\mathbf{T_{clip}},K=\mathbf{M},V=\mathbf{M}),$ (8) $$

where $\text{RWKV}_{I\to T}$ denotes Image-to-Text RWKV interaction and $\text{RWKV}_{T\to I}$ denotes Text-to-Image RWKV interaction. Here, $R$ serves as the query-like from one modality while $K$ and $V$ are the key-value pairs from the other modality.

The attended features are then concatenated with the original features and processed through a two-layer MLP to generate gating coefficients:

$$ $\gamma=\text{MLP}([\mathbf{M}\cdot\mathbf{V}_{I}]),\delta=\text{MLP}([\mathbf{T_{clip}}\cdot\mathbf{V}_{T}]),$ (9) $$

where $\cdot$ denotes concatenation. The final outputs incorporate adaptive residual connections:

$$ $\mathbf{M}_{out}=\mathbf{M}+\gamma\otimes\mathbf{V}_{I},\mathbf{T}_{out}=\mathbf{T_{clip}}+\delta\otimes\mathbf{V}_{T}.$ (10) $$

where $\otimes$ represents element-wise multiplication. This design enables adaptive cross-modal enhancement while preserving the original feature representations.

#### Text-Modulated Sampling

To enhance spatial feature sampling with semantic guidance, we introduce a text-modulated sampling mechanism. Given query features $\mathbf{Q}\in\mathbb{R}^{B\times N\times D}$ and text features $\mathbf{T}_{text}\in\mathbb{R}^{B\times N\times D}$, the module first generates base sampling offsets:

$$ $\Delta_{base}=\text{Linear}(\mathbf{Q}),$ (11) $$

where $\Delta_{base}\in\mathbb{R}^{B\times N\times 2K}$ represents $K$ sampling points with 2D offsets.

Simultaneously, text-guided modulation weights are computed through cross-modal attention:

$$ $\mathbf{A}=\text{Softmax}(\mathbf{Q}\cdot\mathbf{T}_{text}^{T}/\sqrt{D}),$ (12) $$

$$ $\mathbf{W}_{mod}=\text{MLP}(\mathbf{A})\in\mathbb{R}^{B\times N\times 2K},$ (13) $$

where $\mathbf{A}\in\mathbb{R}^{B\times N\times C}$ represents attention scores between queries and text features, and the MLP maps the attention-weighted features from dimension $C$ to $2K$ to match the offset dimensions: $\text{MLP}:\mathbb{R}^{B\times N\times C}\rightarrow\mathbb{R}^{B\times N\times 2K}$.

The text-modulated offsets are obtained by combining base offsets with modulation:

$$ $\Delta_{updated}=\Delta_{base}\oplus\mathbf{W}_{mod},$ (14) $$

where $\oplus$ denotes element-wise addition. These updated offsets guide the sampling process:

$$ $\mathbf{F}_{sampled}=\text{Linear}(\text{Sample}(\mathbf{F}_{ref},\mathbf{p}_{ref}+\Delta_{updated})).$ (15) $$

where $\mathbf{F}_{ref}$ represents reference features, $\mathbf{p}_{ref}=(x,y)$ are reference points, and $\text{Sample}(\cdot)$ performs bilinear interpolation at the offset positions. This text-modulated sampling enables semantically-aware spatial attention, improving feature extraction at object boundaries and semantic regions.

#### Bi-Momentum Contrastive Alignment

To address the issue that negative samples are constrained by memory during training, we adopt MoCo (He et al. 2019) paradigm for region-text contrastive learning. Two momentum-updated feature queues are maintained:

$$ $\mathcal{Q}_{region}\in\mathbb{R}^{K\times D_{proj}},\mathcal{Q}_{text}\in\mathbb{R}^{K\times D_{proj}}.$ (16) $$

where $K$ is the queue size and $D_{proj}$ is the projection dimension.

Positive region proposals are selected via IoU threshold $\tau_{IoU}$:

$$ $\mathcal{P}_{pos}=\{p_{i}|\text{IoU}(p_{i},g_{j})\geq\tau_{IoU},\exists g_{j}\in\mathcal{G}_{gt}\}.$ (17) $$

Region features are encoded through RoI extraction and projection:

$$ $\mathbf{r}_{q}=f_{\theta}(\text{RoIExtractor}(\mathbf{F}_{out},\mathcal{P}_{pos})),$ (18) $$

$$ $\mathbf{r}_{k}=f_{\theta_{m}}(\text{RoIExtractor}(\mathbf{F}_{out},\mathcal{P}_{pos})).$ (19) $$

where $f_{\theta}$ is a trainable projection network and $f_{\theta_{m}}$ is its momentum version updated via EMA:

$$ $\theta_{m}\leftarrow m\cdot\theta_{m}+(1-m)\cdot\theta.$ (20) $$

Multi-positive InfoNCE loss is used for image-to-text and text-to-image alignment:

$$ $\mathcal{L}_{i2t}=-\frac{1}{N_{r}}\sum_{i=1}^{N_{r}}\log\frac{\sum_{j\in\mathcal{P}_{i}^{+}}\exp(s_{ij}/\tau)}{\sum_{j=1}^{N_{t}+K}\exp(s_{ij}/\tau)}.$ (21) $$

$$ $\mathcal{L}_{t2i}=-\frac{1}{N_{t}}\sum_{j=1}^{N_{t}}\log\frac{\sum_{i\in\mathcal{P}_{j}^{+}}\exp(s_{ji}/\tau)}{\sum_{i=1}^{N_{r}+K}\exp(s_{ji}/\tau)}.$ (22) $$

where $s_{ij}=\langle\mathbf{r}_{q,i},\mathbf{t}_{k,j}\rangle\cdot\exp(\alpha)$ denotes the similarity between the $i$-th region feature and the $j$-th text feature, $\alpha$ is a learnable temperature parameter, $\tau$ is a fixed temperature hyperparameter, $N_{r}$ is the number of region features in the current batch, $N_{t}$ is the number of text features in the current batch, $K$ is the momentum queue size, $\mathcal{P}_{i}^{+}$ is the set of positive text indices for the $i$-th region, and $\mathcal{P}_{j}^{+}$ is the set of positive region indices for the $j$-th text.
The total contrastive loss:

$$ $\mathcal{L}_{contrast}=\lambda_{i2t}\mathcal{L}_{i2t}+\lambda_{t2i}\mathcal{L}_{t2i}.$ (23) $$

where $\lambda_{i2t}$ and $\lambda_{t2i}$ are loss weights.
The full details can be found in Algorithm [2](https://arxiv.org/html/2509.23316v1#alg2).

Figure: Algorithm 1 Stage 1: RGB-Infrared Fusion for Robust Detection

###### Theorem 1 (EMA preserves pre-stage performance for the momentum branch) .

Let $\{\theta_{t}\}_{t\geq 0}\subset\mathbb{R}^{p}$ be the online-branch parameters during Stage-2 training, and let the momentum (EMA) branch be updated by $\theta_{m,t}\ \leftarrow\ m\,\theta_{m,t-1}+(1-m)\,\theta_{t}$ and $\theta_{m,0}=\theta_{0}$ with $m\in(0,1]$. Denote by $f_{\theta}$ (online) and $f_{\theta_{m}}$ (momentum) the projection networks used to produce region/text embeddings, and let the multi-positive InfoNCE loss be $\mathcal{L}_{\mathrm{contrast}}(\theta)$ (with similarities $s_{ij}=\langle r_{i}(\theta),\,t_{j}(\theta)\rangle$ and temperature $\tau>0$). Here, we assume:

- (A1)
(Bounded per-step motion) $\|\theta_{t}-\theta_{t-1}\|\leq\delta_{t}$ for all $t\geq 1$, and write $\Delta_{t}:=\max_{1\leq j\leq t}\delta_{j}$.
- (A2)
(Parameter-to-feature Lipschitz) For any RoI/text input, there exist $K_{r},K_{t}>0$ such that
$\|r_{i}(\theta)-r_{i}(\theta^{\prime})\|\leq K_{r}\|\theta-\theta^{\prime}\|$ and $\|t_{j}(\theta)-t_{j}(\theta^{\prime})\|\leq K_{t}\|\theta-\theta^{\prime}\|$ for all $\theta,\theta^{\prime}$ in a convex set containing $\{\theta_{s},\theta_{m,s}\}_{s\leq t}$.
- (A3)
(Logit/Loss Lipschitz) With cosine (or normalized) similarity $s_{ij}=\langle r_{i},t_{j}\rangle$ and temperature $\tau$, the InfoNCE per-example loss is $\rho$-Lipschitz w.r.t. the logit vector, with $\rho\leq 1/\tau$.

Then, for every $t\geq 1$, the momentum–online parameter lag satisfies

$$ $\|\theta_{t}-\theta_{m,t}\|\ \leq\ \frac{1-m}{m}\,\Delta_{t},$ (24) $$

and the following bounds hold:
Function-consistency

$$ $\mathbb{E}\big[\|f_{\theta_{t}}(x)-f_{\theta_{m,t}}(x)\|\big]\ \leq\ (K_{r}\!+\!K_{t})\,\frac{1-m}{m}\,\Delta_{t},\\$ (25) $$

Loss preservation

$$ $\big|\mathcal{L}_{\mathrm{contrast}}(\theta_{m,t})-\mathcal{L}_{\mathrm{contrast}}(\theta_{t})\big|\ \leq\ \rho\,(K_{r}\!+\!K_{t})\,\frac{1-m}{m}\,\Delta_{t}.$ (26) $$

In particular, deploying the EMA/momentum parameters $\theta_{m,t}$ at any time $t$ guarantees an $\varepsilon$-tolerance on the reference distribution whenever $\frac{1-m}{m}\,\Delta_{t}\ \leq\ \frac{\varepsilon}{\rho\,(K_{r}+K_{t})}$.

###### Theorem 1 (EMA preserves pre-stage performance for the momentum branch) .

### Two-Stage Training Strategy

Stage 1 - Multi-modal Robustness Training: Using RGBT datasets, we enhance the model’s robustness in extreme environments through visible-thermal infrared fusion. Only a single query head is used in this stage without any auxiliary heads. The loss function is defined as:

$$ $\mathcal{L}_{stage1}=\mathcal{L}_{det}(\mathbf{X}_{vis},\mathbf{X}_{ir}).$ (27) $$

where $\mathcal{L}_{det}$ is the detection loss, including classification loss $\mathcal{L}_{cls}$ and regression loss $\mathcal{L}_{reg}$.

Stage 2 - Open-Vocabulary Training:
Training on the COCO dataset, we incorporate CLIP semantic features and momentum contrastive learning. Inspired by Co-DETR (Zong, Song, and Liu 2023), we introduce multiple auxiliary heads to enhance the training efficiency of both encoder and decoder. The loss function is defined as:

$$ $\mathcal{L}_{stage2}=\mathcal{L}_{det}+\lambda_{c}\mathcal{L}_{contrast}+\lambda_{aux}\mathcal{L}_{aux},$ (28) $$

$$ $\mathcal{L}{aux}=\sum_{i=1}^{K}\mathcal{L}^{enc}_{i}+\lambda_{1}\sum_{i=1}^{K}\sum_{l=1}^{L}\mathcal{L}^{dec}_{i,l}.$ (29) $$

where $\mathcal{L}_{det}$ is the detection loss based on Hungarian matching including classification and regression losses, $\mathcal{L}_{contrast}$ is the contrastive loss for open-vocabulary learning using CLIP semantics, $\mathcal{L}^{enc}_{i}$ is the encoder auxiliary loss for the $i$-th head (e.g., ATSS or Faster R-CNN loss), $\mathcal{L}^{dec}_{i,l}$ is the decoder auxiliary loss for the $i$-th head at the $l$-th layer, $K$ is the number of auxiliary heads, $L$ is the number of decoder layers, $\lambda_{c}$ is the weight of $\mathcal{L}_{contrast}$, $\lambda_{aux}$ is the weight of $\mathcal{L}_{aux}$ and $\lambda_{1}$ is the weight for decoder auxiliary losses.

To prevent catastrophic forgetting that may arise in continual learning, we introduce Exponential Moving Average(EMA) (Karras et al. 2023; Lee et al. 2024; Li et al. 2024).
Intuitively, EMA maintains a momentum branch that smooths the parameter updates, keeping it close to the online branch while preserving stage-1 robustness. This helps ensure consistency of functions and loss stability across stages. A formal derivation [1](https://arxiv.org/html/2509.23316v1#Thmtheorem1) and proof [Two-Stage Training Strategy](https://arxiv.org/html/2509.23316v1#Sx3.SSx4) are provided.

###### Proof.

(EMA lag). From the update $\theta_{m,t}=m\theta_{m,t-1}+(1-m)\theta_{t}$, subtract from $\theta_{t}$: $\theta_{t}-\theta_{m,t}=(1-m)\big(\theta_{t}-\theta_{m,t-1}\big)=(1-m)\big[(\theta_{t}-\theta_{t-1})+(\theta_{t-1}-\theta_{m,t-1})\big]$. Taking norms and using the triangle inequality gives $x_{t}:=\|\theta_{t}-\theta_{m,t}\|\ \leq\ (1-m)\,u_{t}+(1-m)\,x_{t-1}$ and $u_{t}:=\|\theta_{t}-\theta_{t-1}\|\leq\Delta_{t}.$
Unrolling with $x_{0}=0$ yields $x_{t}\ \leq\ (1-m)\sum_{k=0}^{t-1}(1-m)^{k}u_{t-k}\ \leq\ (1-m)\,\Delta_{t}\sum_{k=0}^{\infty}(1-m)^{k}\ =\ \tfrac{1-m}{m}\,\Delta_{t}$, which is Eq. ([24](https://arxiv.org/html/2509.23316v1#Sx3.E24)).

(Function-consistency). By (A2), for any input (RoI/text) $x$, $\|f_{\theta_{t}}(x)-f_{\theta_{m,t}}(x)\|\leq(K_{r}{+}K_{t})\,\|\theta_{t}-\theta_{m,t}\|$. Taking expectation over the reference distribution and using Eq. ([24](https://arxiv.org/html/2509.23316v1#Sx3.E24)) gives Eq. ([25](https://arxiv.org/html/2509.23316v1#Sx3.E25)).

(Loss preservation). With normalized features, the similarity logits enter InfoNCE as $\{s_{ij}/\tau\}$.
By (A3), the per-example loss is $\rho$-Lipschitz in logits, hence the loss difference between $\theta_{t}$ and $\theta_{m,t}$ is bounded by $\rho\cdot\mathbb{E}\big[\|\mathrm{logits}(\theta_{t})-\mathrm{logits}(\theta_{m,t})\|\big]$. Since each logit is an inner product of two unit vectors, its perturbation is bounded by the sum of the feature perturbations; with (A2) and Eq. ([24](https://arxiv.org/html/2509.23316v1#Sx3.E24)) this expectation is at most $(K_{r}{+}K_{t})\frac{1-m}{m}\Delta_{t}$, proving Eq. ([26](https://arxiv.org/html/2509.23316v1#Sx3.E26)).
∎

The two-stage training strategy enables C3-OWD to first learn robust feature representations from multi-modal data, then leverage these representations to achieve superior open-vocabulary detection performance through semantic alignment with CLIP features and momentum contrastive learning.

Figure: Algorithm 2 Stage 2: Vision-Language Generalization Alignment

###### Proof.

## Experiments

### Datasets and Evaluation Metrics

The training process consists of two stages, utilizing the following datasets: the FLIR Dataset (Systems 2018). The dataset provides thermal infrared images, a modality known for its high robustness to challenging conditions like extreme illumination (e.g., strong light, darkness) and noise pollution. They are used in the first training stage to enhance the detector’s robustness and generalization capabilities by leveraging the invariant properties of the infrared spectrum. Subsequently, in the second training stage, we evaluate our approach on two standard open-vocabulary detection benchmarks modified from LVIS (Gupta, Dollar, and Girshick 2019) and COCO (Lin et al. 2014) respectively. LVIS (Gupta, Dollar, and Girshick 2019) contains 100K images with 1,203 classes. The classes are divided into three groups,
namely frequent, common and rare, based on the number of training images.
Following ViLD (Gu et al. 2021b), we treat 337 rare classes as novel classes and use only the frequent and common classes for training. The COCO dataset is a widely used benchmark for object detection, which consists of 80 classes. Following OVR-CNN (Zareian et al. 2021), we divide the classes in COCO into 48 base categories and 17 novel categories, while removing 15 categories without a synset in the WordNet hierarchy. The training set is the same as the full COCO but only images containing at least one base class are used. For FLIR (Systems 2018) dataset, we report $AP^{50}$—the mean Average Precision (mAP) at an IoU threshold of 0.5 for base categories—as the primary metric. For COCO (Lin et al. 2014), we report $AP^{50}_{novel}$—the mean Average Precision (mAP) at an IoU threshold of 0.5 for novel categories—as the primary metric. Additionally, we provide performance on base categories ($AP^{50}_{base}$) and overall performance across all categories ($AP^{50}$). For LVIS (Gupta, Dollar, and Girshick 2019), we report $AP_{r}$, $AP_{c}$, and $AP_{f}$—denoting mAP on rare, common, and frequent categories, respectively—along with the overall $AP$, all computed using standard box-based mAP.

### Implementation Details

Our model is based on a Deformable-DETR (Zhu et al. 2020) architecture, initialized from a pre-trained ResNet-50 backbone. The first-stage training utilizes the FLIR dataset (Systems 2018), while the second-stage is performed on the COCO (Lin et al. 2014) and LVIS (Gupta, Dollar, and Girshick 2019) dataset. Both stage-1 and stage-2 trainings use 36 epochs. We employ the AdamW optimizer with an initial learning rate of $5\times 10^{-5}$ and weight decay of $1\times 10^{-4}$, applying a layer-wise learning rate decay with a multiplier of 0.1 for the backbone. The learning rate is reduced by a factor of 10 at the $12^{th}$ and $24^{th}$ epoch during the 36-epoch training schedule.Following the MoCo (He et al. 2019), We set $K=65536$ and $\tau_{\text{IoU}}=0.3$. All input images are resized to $640\times 640$ pixels with a batch size of 4 per GPU, and we apply gradient clipping with a maximum norm of 0.1. Following Co-DTER (Zong, Song, and Liu 2023), the model incorporates multiple detection heads including RPN, query-based, and ROI heads, with a total of 6 encoder layers and 8 decoder layers, the loss weighting factor $\lambda_{2}$ set to 2.0. We set $\lambda_{c}=0.01$, $\lambda_{\text{aux}}=0.1$, and $\lambda_{1}=1$. The temperature $\tau$ is set to $0.07$. We train and evaluate our model on 8$\times$NVIDIA A100 GPUs.

### Main result

**Table 1: Comparison on FLIR dataset.**
| Method | Backbone | AP^50 (FLIR) |
| --- | --- | --- |
| MMTOD-CG (Devaguptapu et al. 2019) | RN50 | 61.4 |
| MMTOD-UNIT (Devaguptapu et al. 2019) | RN101 | 61.5 |
| CMPD (Li et al. 2022b) | RN50 | 69.4 |
| CFR (Zhang et al. 2020) | VGG-16 | 72.4 |
| GAFF (Zhang et al. 2021) | RN50 | 72.9 |
| BU-ATT (Kieu, Bagdanov, and Bertini 2021) | RN50 | 73.1 |
| BU-LTT (Kieu, Bagdanov, and Bertini 2021) | RN50 | 73.2 |
| UA_CMDet (Sun et al. 2022b) | RN50 | 78.6 |
| CFT (Fang, Han, and Wang 2021) | RN50 | 78.7 |
| CSAA (Cao et al. 2023) | RN50 | 79.2 |
| ICAFusion (Shen et al. 2024a) | RN50 | 79.2 |
| CrossFormer (Lee, Park, and Park 2024) | Swin-B | 79.3 |
| MFPT (Zhu et al. 2023) | RN101 | 80.0 |
| RSDet (Zhao, Yuan, and Wei 2024) | RN50 | 81.1 |
| MiPa (Medeiros et al. 2024) | ViT-B/16 | 81.3 |
| MMFN (Yang et al. 2025a) | RN50 | 81.8 |
| C3-OWD (Ours) | RN50 | 80.1 |

**Table 2: Comparison on OV-COCO benchmark**
| Method | Backbone | AP${}_{\text{Novel}}^{50}$ |
| --- | --- | --- |
| ViLD (Gu et al. 2021a) | RN50 | 27.6 |
| Detic (Zhou et al. 2022) | RN50 | 27.8 |
| OV-DETR (Zang et al. 2022) | RN50 | 29.4 |
| ProxyDet (Jeong et al. 2024) | RN50 | 30.4 |
| RegionCLIP (Zhong et al. 2022) | RN50 | 31.4 |
| RTGen (Chen et al. 2024) | RN50 | 33.6 |
| BARON-KD (Wu et al. 2023a) | RN50 | 34.0 |
| CLIM (Wu et al. 2024b) | RN50 | 36.9 |
| SAS-Det (Zhao et al. 2024) | RN50 | 37.4 |
| RegionCLIP (Zhong et al. 2022) | RN50x4 | 39.3 |
| CORA (Wu et al. 2023b) | RN50x4 | 41.7 |
| PromptDet (Song and Bang 2023) | ViT-B/16 | 30.6 |
| RO-ViT (Kim, Angelova, and Kuo 2023b) | ViT-L/16 | 33.0 |
| CFM-ViT (Kim, Angelova, and Kuo 2023a) | ViT-L/16 | 34.1 |
| BIND (Zhou et al. 2022) | ViT-L/16 | 41.5 |
| CLIPSelf (Wu et al. 2024a) | ViT-L/14 | 44.3 |
| C3-OWD (Ours) | RN50 | 48.6 |

**Table 3: Comparison on OV-LVIS benchmark**
| Method | Backbone | mAPr |
| --- | --- | --- |
| ViLD (Gu et al. 2021a) | RN50 | 16.3 |
| OV-DETR (Zang et al. 2022) | RN50 | 17.4 |
| BARON-KD (Wu et al. 2023a) | RN50 | 22.6 |
| RegionCLIP (Zhong et al. 2022) | RN50x4 | 22.0 |
| CORA (Wu et al. 2023b) | RN50x4 | 28.1 |
| SAS-Det (Zhao et al. 2024) | RN50x4 | 29.1 |
| CLIM (Wu et al. 2024b) | RN50x64 | 32.3 |
| F-VLM (Kuo et al. 2023) | RN50x64 | 32.8 |
| RTGen (Chen et al. 2024) | Swin-B | 30.2 |
| BIND (Zhou et al. 2022) | ViT-L/16 | 32.5 |
| Detic (Zhou et al. 2022) | Swin-B | 33.8 |
| CFM-ViT (Kim, Angelova, and Kuo 2023a) | ViT-L/14 | 33.9 |
| RO-ViT (Kim, Angelova, and Kuo 2023b) | ViT-H/16 | 34.1 |
| CLIPSelf (Wu et al. 2024a) | ViT-L/14 | 34.9 |
| ProxyDet (Jeong et al. 2024) | Swin-B | 36.7 |
| CoDet (Ma et al. 2024) | ViT-L/14 | 37.0 |
| C3-OWD (Ours) | RN50 | 35.7 |

As shown in Table [1](https://arxiv.org/html/2509.23316v1#Sx4.T1), [2](https://arxiv.org/html/2509.23316v1#Sx4.T2), [3](https://arxiv.org/html/2509.23316v1#Sx4.T3). Although C3-OWD does not achieve the absolute state-of-the-art on the FLIR dataset (slightly behind MMFN’s 81.8), it significantly outperforms all existing methods on open-vocabulary detection tasks (OV-COCO and OV-LVIS), notably achieving a remarkable AP${}^{50}_{\text{Novel}}$ of 48.6 on OV-COCO—surpassing the previous best result of 44.3 by CLIPSelf. More importantly, C3-OWD is the method that demonstrates leading performance across both multimodal robustness and open-vocabulary detection tasks, highlighting its superior generality and cross-task adaptability. This indicates that our proposed curriculum cross-modal contrastive learning framework not only enhances robustness in extreme environments but also effectively transfers to open-vocabulary scenarios, enabling broader applicability.

### Ablation Study

**Table 4: Ablation study of C3-OWD components on OV-COCO and OV-LVIS datasets.**
| No. | Ablated Component | OV-COCO (AP${}^{50}_{novel}$) | OV-LVIS (APr) |
| --- | --- | --- | --- |
| 0 | C3-OWD (Full Model) | 48.6 | 35.7 |
| 1 | w/o encoder fusion | 42.1 | 31.2 |
| 2 | static query selection | 44.1 | 32.3 |
| 3 | w/o text cross-attention | 40.9 | 29.8 |
| 4 | w/o MoCo | 39.7 | 27.5 |
| 5 | w/o Bi-MoCo | 46.1 | 33.2 |

Table [4](https://arxiv.org/html/2509.23316v1#Sx4.T4) presents the ablation results of key components in C3-OWD. The full model (No. 0) achieves the best performance on both OV-COCO and OV-LVIS datasets. Removing the encoder fusion module (No. 1) causes significant performance drops (6.5 in AP${}^{50}_{novel}$ and 4.5 in APr), demonstrating the crucial role of hierarchical feature fusion for open-vocabulary detection. Using static query selection (No. 2) instead of deformable attention also leads to considerable performance degradation, validating the effectiveness of dynamic query optimization. The absence of text cross-attention (No. 3) results in the most substantial performance decline, indicating that vision-language interaction is core to cross-modal alignment. Removing MoCo contrastive learning (No. 4) hurts performance the most, highlighting the critical importance of momentum contrast in addressing positive-negative sample imbalance. Finally, using single-queue instead of dual-queue MoCo (No. 5) also reduces performance, proving that the region-text dual-queue design effectively prevents inter-modal interference. All ablation results consistently show that each component in C3-OWD contributes importantly to the final performance.

Figure: Figure 3: Representative examples of failure cases.
Refer to caption: x3.png

**Table 5: Ablation study on Stage-1 modality and Stage-2 initialization.**
| Config. | Stage-1 Modality | Stage-2 Initialization | FLIR (AP^50) | OV-COCO (AP${}^{50}_{novel}$) | OV-LVIS (APr) |
| --- | --- | --- | --- | --- | --- |
| Full Model | RGB + IR | From Stage-1 | 80.1 | 48.6 | 35.7 |
| A | RGB only | From Stage-1 | 75.2 | 43.5 | 32.1 |
| B | IR only | From Stage-1 | 76.8 | 44.1 | 32.8 |
| C | RGB + IR | Random | 80.0 | 41.2 | 30.5 |
| D | RGB only | Random | 75.1 | 39.8 | 29.1 |
| E | IR only | Random | 76.7 | 40.3 | 29.7 |

As shown in Table [5](https://arxiv.org/html/2509.23316v1#Sx4.T5), we conduct a thorough ablation study to validate the necessity of the two-stage curriculum design in C3-OWD. The results clearly demonstrate that both the multimodal training in Stage-1 and the weight inheritance in Stage-2 are critical for achieving optimal performance. Using only a single modality (RGB or IR) in Stage-1 leads to a noticeable drop in robustness, as evidenced by the decreased FLIR AP^50 (Configs A and B vs. Full). This performance degradation propagates to the open-vocabulary detection tasks in Stage-2, underscoring the importance of cross-modal complementary learning for acquiring generalized representations. Initializing Stage-2 with random weights instead of the pre-trained weights from Stage-1 results in a substantial performance collapse on OV-COCO and OV-LVIS (Config C vs. Full), even when the full RGB-IR modality was used previously. This indicates that the robust features learned in Stage-1 are a crucial foundation for the subsequent semantic alignment and contrastive learning. The worst performance is observed when both components are ablated (Configs D and E), reinforcing that the proposed curriculum learning pipeline is holistic and both stages are indispensable.

### Visualization

We conducted a comparative analysis with traditional methods, as illustrated in Fig [4](https://arxiv.org/html/2509.23316v1#Sx4.F4). Conventional RGBT detectors (e.g., ICAFusion (Shen et al. 2024a), MFPT (Zhu et al. 2023), etc.) are constrained by their closed-set nature, limiting their generalization in open-world scenarios with numerous object categories. On the other hand, traditional open-vocabulary object detection (OVOD) methods (e.g., ViLD (Gu et al. 2021a), CORA (Wu et al. 2023b), etc.) rely solely on the RGB modality and lack complementary infrared information, leading to performance degradation under challenging conditions such as occlusion and shadow coverage. In contrast, our proposed detector effectively integrates multi-modal inputs with open-vocabulary recognition, demonstrating superior generalization across diverse complex environments. Moreover, comparisons between RGB and thermal infrared modalities reveal that while all models perform well under infrared imaging, conventional OVOD detectors exhibit significant performance drops in RGB under low-light or over-exposed conditions, further highlighting the enhanced adaptability and generality of our model.

Figure: Figure 4: A Comparative Evaluation of Our Model against Traditional OVOD and Traditional RGBT Detection Models.
Refer to caption: x4.png

### Failure Cases

Despite strong overall performance, C3-OWD occasionally fails in challenging scenarios. As shown in Fig [3](https://arxiv.org/html/2509.23316v1#Sx4.F3).(a), Dense objects without internal heat sources, such as hats, tables, and chairs, have significantly affected the detection accuracy, and our current model struggles to reliably detect them, highlighting the sensitivity of IR-based detection to environmental heat sources. In Fig [3](https://arxiv.org/html/2509.23316v1#Sx4.F3).(b), a highly occluded vehicle in a dense traffic scene is missed due to insufficient visible cues, suggesting limitations in cross-modal inference under severe occlusion. Fig [3](https://arxiv.org/html/2509.23316v1#Sx4.F3).(c) shows a fine-grained recognition error where a ”husky” is misclassified as ”dog”, indicating that CLIP’s semantic prior may not capture subtle inter-class distinctions.

## Limitations

Despite the promising results, our method suffers from several limitations that warrant further investigation.
Computational Overhead. The hierarchical cross-modal fusion mechanism, while effective, introduces additional computational complexity due to the bidirectional RWKV blocks and iterative attention operations. This may hinder real-time deployment on resource-constrained devices such as embedded systems or drones.
Dependency on Multi-modal Data. The first stage relies heavily on aligned RGB-Thermal (RGBT) data for robustness enhancement. Such paired data is scarce and expensive to collect and annotate, limiting the scalability of our approach in domains where thermal imagery is unavailable.
Generalization to Unseen Modalities. Although our method improves open-vocabulary detection, its performance on entirely unseen sensor modalities (e.g., LiDAR, radar) or under extreme domain shifts remains unverified.
Training Complexity. The two-stage training strategy requires carefully designed curricula and hyperparameter tuning, which may increase the risk of suboptimal convergence and complicate reproduction.
Semantic Granularity. While CLIP provides rich semantic prior, its knowledge is constrained by pre-trained concepts. Our method may still struggle with highly fine-grained or domain-specific categories absent in CLIP’s training distribution.
We believe these limitations point to meaningful directions for future work, including efficient fusion design, self-supervised adaptation, and generalized multi-modal pretraining.

## Future Work

Based on the failure cases and limitations identified in Sections [Limitations](https://arxiv.org/html/2509.23316v1#Sx5) and [Failure Cases](https://arxiv.org/html/2509.23316v1#Sx4.SSx6), we outline several promising directions for future research.
Self-Supervised Modality Alignment. Reducing dependency on paired RGBT data requires methods that can learn robust representations from unaligned or weakly-aligned multi-modal streams. Contrastive learning frameworks (Radford et al. 2021) that align modalities in a shared embedding space without strict pixel-level correspondence offer a viable path forward.
Occlusion-Robust Representation Learning. The failure cases under severe occlusion suggest the need for explicit occlusion modeling. Future models could incorporate temporal consistency constraints (Feichtenhofer et al. 2019) or amodal completion networks (Qi et al. 2021) to reason about partially visible objects.
Fine-Grained Semantic Enhancement. To overcome CLIP’s limitations in fine-grained categorization, future work could integrate domain-specific knowledge bases (Marino, Salakhutdinov, and Gupta 2019) or leverage large language models (OpenAI 2023) to enrich semantic priors with detailed attribute descriptions.
Generalized Multi-modal Pretraining. Extending our approach to unseen modalities like LiDAR and radar requires developing modality-agnostic fusion strategies. Unified multi-modal transformers (Akbari et al. 2021) pretrained on diverse sensor data could enhance generalization under domain shift.
Unsupervised Domain Adaptation. To mitigate training complexity, future work should investigate self-supervised domain adaptation techniques (Ganin et al. 2016) that reduce the need for carefully designed curricula and extensive hyperparameter tuning.

## Social Impact

Our method points to a promising approach for Open World Detection, significantly enhancing the reliability of vision systems in safety-critical applications. By leveraging robust multi-modal (RGB-Thermal) fusion, our framework demonstrates superior performance in challenging conditions such as low illumination, fog, and occlusion. This capability is crucial for enabling all-weather autonomous driving systems, ensuring safer navigation where traditional vision fails. Furthermore, it empowers Unmanned Aerial Vehicles (UAVs) for more effective search-and-rescue and disaster monitoring missions in adverse environments. The integration of open-vocabulary detection also allows these systems to dynamically recognize novel objects, expanding their utility in real-world scenarios without the need for costly re-training. We believe our work paves the way for more adaptable and trustworthy AI systems that can operate reliably in the open world, ultimately contributing to enhanced public safety and operational efficiency in fields like transportation, surveillance, and emergency response.

## Conclusion

In this paper, we introduce an innovative curriculum cross-modal contrastive learning framework for open-world detection, C3-OWD. The proposed two-stage training paradigm addresses the fundamental challenge of achieving both environmental robustness and open-world generalization. Our EMA mechanism provides theoretical guarantees for knowledge preservation across training stages, with rigorously proven bounded parameter lag and function consistency. The integration of RGBT multimodal fusion with RWKV-based architecture enables effective capture of complementary information from visible and infrared modalities, while the vision-language alignment stage extends detection capabilities to novel categories without sacrificing the learned robustness. Extensive experiments validate the effectiveness of C3-OWD across OV-COCO,OV-LVIS and FLIR. C3-OWD achieves competitive results compared to prior state-of-the-art methods.

## References

- Akbari et al. (2021)
Akbari, H.; Yuan, L.; Qian, R.; Chuang, W. H.; Chang, S. F.; Cui, Y.; and Gong, B. 2021.
VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text.
In *Advances in Neural Information Processing Systems*, volume 34, 24206–24221.
- Cao et al. (2023)
Cao, Y.; Bin, J.; Hamari, J.; Blasch, E.; and Liu, Z. 2023.
Multimodal object detection by channel switching and spatial attention.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, 403–411.
- Chen et al. (2024)
Chen, F.; Zhang, H.; Yang, Z.; Chen, H.; Hu, K.; and Savvides, M. 2024.
RTGen: Generating region-text pairs for open-vocabulary object detection.
*arXiv preprint arXiv:2405.19854*.
- Devaguptapu et al. (2019)
Devaguptapu, C.; Akolekar, N.; M Sharma, M.; and N Balasubramanian, V. 2019.
Borrow from anywhere: Pseudo multi-modal object detection in thermal imagery.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops*, 1029–1038.
- Duan et al. (2024a)
Duan, Y.; Wang, W.; Chen, Z.; Zhu, X.; Lu, L.; Lu, T.; Qiao, Y.; Li, H.; Dai, J.; and Wang, W. 2024a.
Vision-rwkv: Efficient and scalable visual perception with rwkv-like architectures.
*arXiv preprint arXiv:2403.02308*.
- Duan et al. (2024b)
Duan, Y.; Wang, W.; Chen, Z.; Zhu, X.; Lu, L.; Lu, T.; Qiao, Y.; Li, H.; Dai, J.; and Wang, W. 2024b.
Vision-RWKV: Efficient and Scalable Visual Perception with RWKV-Like Architectures.
*arXiv preprint arXiv:2403.02308*.
- Fang, Han, and Wang (2021)
Fang, Q.; Han, D.; and Wang, Z. 2021.
Cross-modality fusion transformer for multispectral object detection.
*arXiv preprint arXiv:2111.00273*.
- Fei et al. (2024)
Fei, Z.; Fan, M.; Yu, C.; Li, D.; and Huang, J. 2024.
Diffusion-rwkv: Scaling rwkv-like architectures for diffusion models.
*arXiv preprint arXiv:2404.04478*.
- Feichtenhofer et al. (2019)
Feichtenhofer, C.; Fan, H.; Malik, J.; and He, K. 2019.
SlowFast Networks for Video Recognition.
In *Proceedings of the IEEE/CVF International Conference on Computer Vision*, 6202–6211.
- Ganin et al. (2016)
Ganin, Y.; Ustinova, E.; Ajakan, H.; Germain, P.; Larochelle, H.; Laviolette, F.; Marchand, M.; and Lempitsky, V. 2016.
Domain-Adversarial Training of Neural Networks.
*Journal of Machine Learning Research*, 17(59): 1–35.
- Gu et al. (2021a)
Gu, X.; Lin, T.-Y.; Kuo, W.; and Cui, Y. 2021a.
Open-vocabulary object detection via vision and language knowledge distillation.
*arXiv preprint arXiv:2104.13921*.
- Gu et al. (2021b)
Gu, X.; Lin, T.-Y.; Kuo, W.; and Cui, Y. 2021b.
Open-vocabulary object detection via vision and language knowledge distillation.
*arXiv preprint arXiv:2104.13921*.
- Guan et al. (2019)
Guan, D.; Cao, Y.; Yang, J.; Cao, Y.; and Yang, M. Y. 2019.
Fusion of multispectral data through illumination-aware deep neural networks for pedestrian detection.
*Information Fusion*, 50: 148–157.
- Gupta, Dollar, and Girshick (2019)
Gupta, A.; Dollar, P.; and Girshick, R. 2019.
LVIS: A dataset for large vocabulary instance segmentation.
*arXiv preprint arXiv:1908.03195*.
- Gupta et al. (2022)
Gupta, A.; Narayan, S.; Joseph, K.; Khan, S.; Khan, F. S.; and Shah, M. 2022.
OW-DETR: Open-world detection transformer.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, 9235–9244.
- He et al. (2019)
He, K.; Fan, H.; Wu, Y.; Xie, S.; and Girshick, R. 2019.
Momentum Contrast for Unsupervised Visual Representation Learning.
*arXiv preprint arXiv:1911.05722*.
- He et al. (2025)
He, Q.; Zhang, J.; Peng, J.; He, H.; Li, X.; Wang, Y.; and Wang, C. 2025.
Pointrwkv: Efficient rwkv-like model for hierarchical point cloud learning.
In *Proceedings of the AAAI Conference on Artificial Intelligence*, volume 39, 3410–3418.
- Hou and Yu (2024)
Hou, H.; and Yu, F. R. 2024.
Rwkv-ts: Beyond traditional recurrent neural network for time series tasks.
*arXiv preprint arXiv:2401.09093*.
- Jeong et al. (2024)
Jeong, J.; Park, G.; Yoo, J.; Jung, H.; and Kim, H. 2024.
ProxyDet: Synthesizing proxy novel classes via classwise mixup for open-vocabulary object detection.
In *Proceedings of the AAAI Conference on Artificial Intelligence*, volume 38, 2462–2470.
- Joseph et al. (2021)
Joseph, K.; Pal, A.; Rajanala, S.; and Balasubramanian, V. N. 2021.
Towards open world object detection.
*arXiv preprint arXiv:2103.02603*.
- Karras et al. (2023)
Karras, T.; Aittala, M.; Lehtinen, J.; Hellsten, J.; Aila, T.; and Laine, S. 2023.
Analyzing and Improving the Training Dynamics of Diffusion Models.
*ArXiv*, abs/2312.02696.
- Kieu, Bagdanov, and Bertini (2021)
Kieu, M.; Bagdanov, A. D.; and Bertini, M. 2021.
Bottom-up and layerwise domain adaptation for pedestrian detection in thermal images.
*ACM Transactions on Multimedia Computing, Communications, and Applications*, 17(1): 1–19.
- Kim, Angelova, and Kuo (2023a)
Kim, D.; Angelova, A.; and Kuo, W. 2023a.
Contrastive feature masking open-vocabulary vision transformer.
In *2023 IEEE/CVF International Conference on Computer Vision (ICCV)*, 15556–15566.
- Kim, Angelova, and Kuo (2023b)
Kim, D.; Angelova, A.; and Kuo, W. 2023b.
Region-aware pretraining for open vocabulary object detection with vision transformers.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, 11144–11154.
- Kim, Park, and Ro (2021)
Kim, J. U.; Park, S.; and Ro, Y. M. 2021.
Uncertainty-guided cross-modal learning for robust multispectral pedestrian detection.
*IEEE Transactions on Circuits and Systems for Video Technology*, 32(3): 1510–1523.
- Kirillov et al. (2023)
Kirillov, A.; Mintun, E.; Ravi, N.; Mao, H.; Rolland, C.; Gustafson, L.; Xiao, T.; Whitehead, S.; Berg, A. C.; Lo, W.-Y.; et al. 2023.
Segment anything.
In *Proceedings of the IEEE/CVF international conference on computer vision*, 4015–4026.
- Kuo et al. (2023)
Kuo, W.; Cui, Y.; Gu, X.; Piergiovanni, A.; and Angelova, A. 2023.
Open-vocabulary object detection upon frozen vision and language models.
In *The Eleventh International Conference on Learning Representations*.
- Lee et al. (2024)
Lee, H.; Cho, H.; Kim, H.; Kim, D.; Min, D.; Choo, J.; and Lyle, C. 2024.
Slow and Steady Wins the Race: Maintaining Plasticity with Hare and Tortoise Networks.
*ArXiv*, abs/2406.02596.
- Lee, Park, and Park (2024)
Lee, S.; Park, J.; and Park, J. 2024.
CrossFormer: Cross-guided attention for multi-modal object detection.
*Pattern Recognition Letters*, 179: 144–150.
- Li et al. (2019)
Li, C.; Song, D.; Tong, R.; and Tang, M. 2019.
Illumination-aware faster R-CNN for robust multispectral pedestrian detection.
*Pattern Recognition*, 85: 161–171.
- Li et al. (2022a)
Li, L. H.; Zhang, P.; Zhang, H.; Yang, J.; Li, C.; Zhong, Y.; Wang, L.; Yuan, L.; Zhang, L.; Hwang, J.-N.; Chang, K.-W.; and Gao, J. 2022a.
Grounded Language-Image Pre-training.
arXiv:2112.03857.
- Li et al. (2022b)
Li, Q.; Zhang, C.; Hu, Q.; Fu, H.; and Zhu, P. 2022b.
Confidence-aware fusion using dempster-shafer theory for multispectral pedestrian detection.
*IEEE Transactions on Multimedia*, 25: 3420–3431.
- Li et al. (2023)
Li, R.; Xiang, J.; Sun, F.; Yuan, Y.; Yuan, L.; and Gou, S. 2023.
Multiscale cross-modal homogeneity enhancement and confidence-aware fusion for multispectral pedestrian detection.
*IEEE Transactions on Multimedia*, 26: 852–863.
- Li et al. (2024)
Li, S.; Liu, Z.; Tian, J.; Wang, G.; Wang, Z.; Jin, W.; Wu, D.; Tan, C.; Lin, T.; Liu, Y.; Sun, B.; and Li, S. Z. 2024.
Switch EMA: A Free Lunch for Better Flatness and Sharpness.
*ArXiv*, abs/2402.09240.
- Li et al. (2025)
Li, Z.; Xia, T.; Chang, Y.; and Wu, Y. 2025.
A survey of rwkv.
*Neurocomputing*, 130711.
- Lin et al. (2014)
Lin, T.-Y.; Maire, M.; Belongie, S.; Hays, J.; Perona, P.; Ramanan, D.; Dollár, P.; and Zitnick, C. L. 2014.
Microsoft COCO: Common objects in context.
*arXiv preprint arXiv:1405.0312*.
- Lv et al. (2024)
Lv, W.; Zhao, Y.; Chang, Q.; Huang, K.; Wang, G.; and Liu, Y. 2024.
RT-DETRv2: Improved Baseline with Bag-of-Freebies for Real-Time Detection Transformer.
arXiv:2407.17140.
- Ma et al. (2024)
Ma, C.; Jiang, Y.; Wen, X.; Yuan, Z.; and Qi, X. 2024.
CoDet: Co-occurrence guided region-word alignment for open-vocabulary object detection.
*Advances in Neural Information Processing Systems*, 36.
- Ma et al. (2023)
Ma, S.; Wang, Y.; Wei, Y.; Fan, J.; Sun, X.; Chen, P.; and Zhang, E. 2023.
A simple knowledge distillation framework for open-world object detection.
*arXiv preprint arXiv:2312.08653*.
- Marino, Salakhutdinov, and Gupta (2019)
Marino, K.; Salakhutdinov, R.; and Gupta, A. 2019.
Fine-Grained Image Classification with Learnable Semantic Parts.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, 4500–4509.
- Medeiros et al. (2024)
Medeiros, H. R.; Latortue, D.; Granger, E.; and Pedersoli, M. 2024.
MiPa: Mixed Patch Infrared-Visible Modality Agnostic Object Detection.
*arXiv preprint arXiv:2404.18849*.
- OpenAI (2023)
OpenAI. 2023.
GPT-4 Technical Report.
*arXiv preprint arXiv:2303.08774*.
- Peng et al. (2023)
Peng, B.; Alcaide, E.; Anthony, Q.; Albalak, A.; Arcadinho, S.; Biderman, S.; Cao, H.; Cheng, X.; Chung, M.; Grella, M.; et al. 2023.
Rwkv: Reinventing rnns for the transformer era.
*arXiv preprint arXiv:2305.13048*.
- Peng et al. (2025)
Peng, B.; Zhang, R.; Goldstein, D.; Alcaide, E.; Du, X.; Hou, H.; Lin, J.; Liu, J.; Lu, J.; Merrill, W.; et al. 2025.
Rwkv-7” goose” with expressive dynamic state evolution.
*arXiv preprint arXiv:2503.14456*.
- Qi et al. (2021)
Qi, L.; Jiang, L.; Liu, S.; Shen, X.; and Jia, J. 2021.
Amodal Instance Segmentation with KINS Dataset.
In *IEEE Transactions on Pattern Analysis and Machine Intelligence*, volume 44, 707–720.
- Radford et al. (2021)
Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; Krueger, G.; and Sutskever, I. 2021.
Learning transferable visual models from natural language supervision.
In *Proceedings of the 38th International Conference on Machine Learning*, volume 139, 8748–8763. PMLR.
- Shen et al. (2024a)
Shen, J.; Chen, Y.; Liu, Y.; Zuo, X.; Fan, H.; and Yang, W. 2024a.
ICAFusion: Iterative cross-attention guided feature fusion for multispectral object detection.
*Pattern Recognition*, 145: 109913.
- Shen et al. (2024b)
Shen, J.; Chen, Y.; Liu, Y.; Zuo, X.; Fan, H.; and Yang, W. 2024b.
ICAFusion: Iterative cross-attention guided feature fusion for multispectral object detection.
*Pattern Recognition*, 145: 109913.
- Song and Bang (2023)
Song, H.; and Bang, J. 2023.
Prompt-guided transformers for end-to-end open-vocabulary object detection.
*arXiv preprint arXiv:2303.14386*.
- Sun et al. (2022a)
Sun, Y.; Cao, B.; Zhu, P.; and Hu, Q. 2022a.
Drone-based RGB-infrared cross-modality vehicle detection via uncertainty-aware learning.
*IEEE Transactions on Circuits and Systems for Video Technology*, 32(10): 6700–6713.
- Sun et al. (2022b)
Sun, Y.; Cao, B.; Zhu, P.; and Hu, Q. 2022b.
Drone-based RGB-infrared cross-modality vehicle detection via uncertainty-aware learning.
*IEEE Transactions on Circuits and Systems for Video Technology*, 32(10): 6700–6713.
- Systems (2018)
Systems, F. 2018.
FLIR Thermal Dataset for Algorithm Training.
- Wu et al. (2023a)
Wu, S.; Zhang, W.; Jin, S.; Liu, W.; and Loy, C. C. 2023a.
Aligning bag of regions for open-vocabulary object detection.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, 15254–15264.
- Wu et al. (2024a)
Wu, S.; Zhang, W.; Xu, L.; Jin, S.; Li, X.; Liu, W.; and Loy, C. C. 2024a.
CLIPSelf: Vision transformer distills itself for open-vocabulary dense prediction.
In *The Twelfth International Conference on Learning Representations*.
- Wu et al. (2024b)
Wu, S.; Zhang, W.; Xu, L.; Jin, S.; Liu, W.; and Loy, C. C. 2024b.
CLIM: Contrastive language-image mosaic for region representation.
In *Proceedings of the AAAI Conference on Artificial Intelligence*, volume 38, 6117–6125.
- Wu et al. (2023b)
Wu, X.; Zhu, F.; Zhao, R.; and Li, H. 2023b.
CORA: Adapting CLIP for open-vocabulary detection with region prompting and anchor pre-matching.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, 7031–7040.
- Xi et al. (2024)
Xi, X.; Huang, Y.; Lin, J.; and Luo, R. 2024.
Ktcn: Enhancing open-world object detection with knowledge transfer and class-awareness neutralization.
In *Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, IJCAI-24*, 1462–1470.
- Yang et al. (2025a)
Yang, F.; Liang, B.; Li, W.; and Zhang, J. 2025a.
Multidimensional Fusion Network for Multispectral Object Detection.
*IEEE Transactions on Circuits and Systems for Video Technology*, 35(1): 547–560.
- Yang et al. (2025b)
Yang, Z.; Li, J.; Zhang, H.; Zhao, D.; Wei, B.; and Xu, Y. 2025b.
Restore-rwkv: Efficient and effective medical image restoration with rwkv.
*IEEE Journal of Biomedical and Health Informatics*.
- Yuan and Wei (2024)
Yuan, M.; and Wei, X. 2024.
C^2former: Calibrated and complementary transformer for rgb-infrared object detection.
*IEEE Transactions on Geoscience and Remote Sensing*, 62: 1–12.
- Zang et al. (2022)
Zang, Y.; Li, W.; Zhou, K.; Huang, C.; and Loy, C. C. 2022.
Open-vocabulary DETR with conditional matching.
In *European Conference on Computer Vision*, 106–122. Springer.
- Zareian et al. (2021)
Zareian, A.; Rosa, K. D.; Hu, D. H.; and Chang, S.-F. 2021.
Open-Vocabulary Object Detection Using Captions.
arXiv:2011.10678.
- Zhang et al. (2020)
Zhang, H.; Fromont, E.; Lefevre, S.; and Avignon, B. 2020.
Multispectral fusion for object detection with cyclic fuse-and-refine blocks.
In *2020 IEEE International Conference on Image Processing*, 276–280.
- Zhang et al. (2021)
Zhang, H.; Fromont, E.; Lefèvre, S.; and Avignon, B. 2021.
Guided attentive feature fusion for multispectral pedestrian detection.
In *Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision*, 72–80.
- Zhao et al. (2024)
Zhao, S.; Schulter, S.; Zhao, L.; Zhang, Z.; Kumar B G, V.; Suh, Y.; Chandraker, M.; and Metaxas, D. N. 2024.
Taming self-training for open-vocabulary object detection.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, 13938–13947.
- Zhao, Yuan, and Wei (2024)
Zhao, T.; Yuan, M.; and Wei, X. 2024.
Removal and selection: Improving rgb-infrared object detection via coarse-to-fine fusion.
*arXiv preprint arXiv:2401.10731*.
- Zhao et al. (2023)
Zhao, Y.; Lv, W.; Xu, S.; Wei, J.; Wang, G.; Dang, Q.; Liu, Y.; and Chen, J. 2023.
DETRs Beat YOLOs on Real-time Object Detection.
arXiv:2304.08069.
- Zhong et al. (2022)
Zhong, Y.; Yang, J.; Zhang, P.; Li, C.; Codella, N.; Li, L. H.; Zhou, L.; Dai, X.; Yuan, L.; Li, Y.; et al. 2022.
RegionCLIP: Region-based language-image pretraining.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, 16793–16803.
- Zhou, Chen, and Cao (2020)
Zhou, K.; Chen, L.; and Cao, X. 2020.
Improving multispectral pedestrian detection by addressing modality imbalance problems.
In *European conference on computer vision*, 787–803. Springer.
- Zhou et al. (2022)
Zhou, X.; Girdhar, R.; Joulin, A.; Krähenbühl, P.; and Misra, I. 2022.
Detecting twenty-thousand classes using image-level supervision.
In *European Conference on Computer Vision*, 350–368. Springer.
- Zhu et al. (2020)
Zhu, X.; Su, W.; Lu, L.; Li, B.; Wang, X.; and Dai, J. 2020.
Deformable DETR: Deformable Transformers for End-to-End Object Detection.
*arXiv preprint arXiv:2010.04159*.
- Zhu et al. (2023)
Zhu, Y.; Sun, X.; Wang, M.; and Huang, H. 2023.
Multi-modal feature pyramid transformer for rgb-infrared object detection.
*IEEE Transactions on Intelligent Transportation Systems*, 24(9): 9984–9995.
- Zohar, Wang, and Yeung (2023)
Zohar, O.; Wang, K.-C.; and Yeung, S. 2023.
Prob: Probabilistic objectness for open world object detection.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, 11444–11453.
- Zong, Song, and Liu (2023)
Zong, Z.; Song, G.; and Liu, Y. 2023.
Detrs with collaborative hybrid assignments training.
In *Proceedings of the IEEE/CVF international conference on computer vision*, 6748–6758.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: Please refer to the Weaknesses.
soundness: 3
strengths: The authors propose a cross-modal curriculum learning framework that unifies RGBT robustness and open-vocabulary generalization, dynamically balancing multi-modal information through progressive learning.
summary: This paper presents a curriculum cross-modal contrastive learning framework that addresses the dual challenges of poor generalization and limited robustness in real-world object detection. The approach integrates the strengths of visible-infrared and open-world detection through a two-stage training strategy: Stage 1 enhances robustness via RGBT pretraining, while Stage 2 improves generalization through vision–language alignment. An exponential moving average mechanism is introduced to mitigate catastrophic forgetting between stages, theoretically ensuring performance preservation with bounded parameter drift. Experiments demonstrate strong performance in both robustness and diversity benchmarks.
weaknesses: 1.	The proposed framework includes multiple functional modules, but their motivations are not clearly explained. For example, it remains unclear what specific problem the bidirectional attention mechanism in Stage 1 addresses and why it offers advantages over the original version. The description of the method is overly brief and difficult to comprehend. 
2.	In line 325, what are the specific manifestations of catastrophic forgetting? What causes it, and how does the Exponential Moving Average mitigate this issue?
3.	In Table 2, why are the baselines presented inconsistently on both sides, for example, CoDet? For a fairer comparison of robustness and generalization, it would be better to provide experimental results for both datasets.

---

confidence: 3
contribution: 3
initial_rating: 4
presentation: 3
questions: See weaknesses.
soundness: 3
strengths: The paper identifies a meaningful gap between robustness (RGBT-based detection) and open-world generalization (vision-language models), proposing a curriculum-style framework to bridge the two. The method is evaluated on three benchmarks with solid ablation studies, confirming the contribution of each component.
summary: This paper proposes C3-OWD, a curriculum cross-modal contrastive learning framework for open-world object detection. The approach adopts a two-stage training strategy: 1. Multi-modal robustness enhancement by pretraining on RGBT datasets to improve feature stability. 2. Vision-language generalization alignment through semantic enhancement, text-modulated deformable attention, and a bi-momentum contrastive alignment mechanism.
weaknesses: 1.The core ideas are all adaptations of existing paradigms (Deformable-DETR + CLIP + MoCo). The contribution seems incremental, with limited conceptual advancement.
2.The method section is dense and symbol-heavy, especially in Stage 2. Many notations are introduced abruptly without sufficient intuition, making it difficult to follow for readers not already familiar with Deformable-DETR.
3.The training pipeline seems computationally expensive, but no runtime or cost analysis is reported.
4.Several hyperparameters (e.g., IoU threshold = 0.3) are presented without explanation or ablation.

---

confidence: 4
contribution: 3
initial_rating: 4
presentation: 3
questions: Please follow the strengths and weaknesses. 
Also,
How should the ablation with and without the EMA mechanism be seen?
soundness: 3
strengths: The experimental coverage across FLIR (robustness), OV-COCO, and OV-LVIS (generalization) is appropriate and shows an effort to evaluate both aspects of the claimed contribution.

The theoretical analysis of the EMA mechanism provides mathematical grounding, which adds credibility to the catastrophic forgetting mitigation claim.

The model achieves competitive results on OV-COCO (48.6 AP50_Novel), showing improvement over earlier ResNet-50-based approaches like CLIPSelf (44.3).

The inclusion of algorithmic details and ablations in Tables 3 and 4 shows an attempt at transparency and reproducibility, though key omissions remain.
summary: The paper proposes C3-OWD, a two-stage framework that aims to combine environmental robustness (via RGB-Thermal fusion) with open-vocabulary object detection. In Stage 1, visible and thermal features are fused through VRWKV blocks on the FLIR dataset to enhance robustness under varying illumination. Stage 2 integrates CLIP-based vision-language alignment and momentum contrastive learning on COCO and LVIS to support novel category detection. An exponential moving average (EMA) mechanism is introduced to reduce catastrophic forgetting across stages, with accompanying theoretical bounds. Experiments report 80.1 AP50 on FLIR, 48.6 AP50_Novel on OV-COCO, and 35.7 mAPr on OV-LVIS. While the results on OV-COCO outperform some ResNet-50 baselines, the evidence does not fully support the claim of “breaking the robustness–generalization trade-off.” The method performs below MMFN (81.8) on FLIR and CoDet (37.0) on OV-LVIS, suggesting it lies along the same trade-off frontier rather than surpassing it.
weaknesses: DO the (i) where L is the sequence length .... AND (ii) We then perform L rounds .....
Do they have to be the same, or different? Also, what is the value of L?

---

confidence: 3
contribution: 2
initial_rating: 4
presentation: 3
questions: 1.Can the authors supplement more experiments on both robustness and generalizability.  It would be valuable to include experimental results validating the proposed method on object detection datasets under various adverse conditions.Here are some common object detection datasets for extreme environments: DAWN, RTTS, VisDrone, and SUIM Dataset.


2.Can the authors provide detailed evidence showing that the EMA mechanism indeed effectively preserves Stage 1 performance during Stage 2 training? The paper only provides theoretical derivation without actual evidence.
soundness: 2
strengths: 1.The authors provide detailed theoretical analysis for the EMA mechanism with explicit bounds on parameter lag, function consistency, and loss preservation, which strengthen the credibility of the catastrophic forgetting mitigation claim.


2.The authors provide a thorough ablation study dissecting the contribution of each major model component.

3.The results show that the paper' method achieves leading performance in both multimodal robustness and open-vocabulary detection.
summary: The paper proposes C3-OWD, a curriculum cross-modal contrastive learning framework that integrates the advantages of a two-stage paradigm. While evaluated on datasets like FLIR, COCO, and LVIS, the limited scope of the experimental design challenges the broader domain relevance and generalizability of the findings.
weaknesses: 1.The experimental support is insufficient. Although strong performance is reported on COCO and LVIS, these datasets mainly cover standard scenarios and do not verify generalization in extreme scenarios.The current experiments primarily rely on a single robustness dataset, lacking validation on other extreme environment datasets. This makes it difficult to fully demonstrate the method's robustness across different extreme scenarios.

2.The computational complexity of the proposed framework is a concern. The combination of bidirectional RWKV blocks, MoCo-style queues, and multi-stage training likely incurs significant overhead. 

3.The primary contribution of this work lies in the effective integration of several existing techniques(curriculum learning, cross-modal contrastive objectives, RWKV-based fusion, and EMA).