Explore Hybrid Modeling for Moving Infrared Small Target Detection

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-22 03:58:05

Abstract: Moving infrared small target detection, crucial in contexts like traffic management and maritime rescue, encounters challenges from factors such as complex backgrounds, target occlusion, camera shake, and motion blur. Existing algorithms fall short in comprehensively addressing these issues by finding mathematical models, impeding generalization in complex and dynamic motion scenes. In this paper, we propose a method for finding models of moving infrared small target detection via smoothed-particle hydrodynamics (SPH) and Markov decision processes (MDP). SPH can simulate the motion trajectories of targets and background scenes, while MDP can optimize detection system strategies for optimal action selection based on contexts and target states. Specifically, we develop an SPH-inspired image-level enhancement algorithm which models the image sequence of infrared video as a 3D spatiotemporal graph in SPH. In addition, we design an MDP-guided temporal feature perception module. This module selects reference frames, aggregates features from both reference frames and the current frame. The previous and current frames are modeled as an MDP tailored for multi-frame infrared small target detection tasks, aiding in detecting the current frame. Conducted extensive experiments on two public dataset: DAUB and DATR, the proposed STME-Net surpasses the state-of-the-art methods in terms of objective metrics and visual quality.

Main paper: 
 introduction: 
 Identifying moving targets in challenging weather conditions such as fog and heavy rain is often difficult with visible light videos. In contrast, infrared (IR) videos offer more reliable target detection, even in adverse weather, due to their unique imaging mechanism [26, 36, 39] . Accordingly, detecting small moving targets, derived from this unique modality, i.e., IR videos, is a prominent subject in computer vision, widely applied in traffic management and maritime rescue [28, 37] .As shown in Figure 1 , the long-distance nature of infrared imaging results in insufficient information regarding target details such as size, shape, and texture. Challenges arise for moving infrared targets, including complex background, motion blur, interference and camera shake [9, 21] . Addressing these internal and external factors makes the task of detecting moving infrared small targets in video sequences exceptionally challenging.In recent years, numerous algorithms have emerged for infrared small target detection [6, 12, 30, 38] , categorized into single-frame and multi-frame methods. Single-frame methods focus on small target characteristics, utilizing complex nested network structures and attention modules to minimize information loss during pooling and downsampling processes. DNA-Net [20] employs densely nested interactive and spatial attention modules for feature fusion and enhancement, while UIU-Net [31] achieves multi-level learning by embedding a small U-Net into a larger one, yielding promising results in single-frame detection. However, limitations such as occlusion, motion blur, and camera shake hinder single-frame methods' efficacy in capturing moving infrared small targets. Human visual judgment can infer a blurry target's identity by leveraging information from adjacent frames in videos. Utilizing multiple frames provides rich temporal information compared to a single frame, enabling various multi-frame methods like image-level target enhancement and temporal feature perception [4, 25, 32, 45] .For image-level target enhancement, Du et al. [10] enhance small targets through inter-frame alignment, yet overlook spatial information's significance. Zhu et al. [45] leverage optical flow to enhance moving targets, but this method assumes constant target brightness, which may not hold for infrared small targets due to fluctuations caused by factors like temperature changes and occlusions. In addition, direct application of mainstream YOLO series enhancement algorithms [15, 18, 19] like mosaic enhancement to infrared small targets often leads to target loss. In summary, existing image-level enhancement methods have progressed but have limitations on specific modalities like IR videos, thus failing to achieve desired detection effects. Can we view image-level target enhancement from a mathematical modeling perspective? By scientifically modeling the motion state of the target, we can capture more temporal information. Smoothed Particle Hydrodynamics (SPH) simulates fluid behavior by dividing it into particles and simulating their interactions. SPH can model image-level enhancement, simulating information transmission and interaction within IR images, thus facilitating enhancement.For temporal feature perception, existing methods often employ complex temporal feature aggregation networks. For instance, SSTNet [4] leverages LSTM's memory prediction and a cross-slice ConvLSTM structure to aggregate temporal information. Luo et al. [22] utilize dense nested structures and optical flow to design a multi-scale optical flow reconstruction network for capturing moving small targets. The above temporal feature aggregation networks utilize neural networks' nonlinear properties and parameter learning to handle complex temporal data, potentially boosting performance at the cost of increased training expenses. Can we treat temporal feature perception as a prediction task and model state evolution in time-series IR data? The Markov decision process (MDP) can offer a simplified approach to modeling time-series IR data by abstracting it into states and corresponding transition probabilities. This can allow us to move away from complex temporal aggregation networks and focus on prediction instead.In this paper, we propose a method to explore hybrid models for detecting small moving infrared targets using SPH and MDP. Inspired by SPH, we represent motion as fluid dynamics at the image level, with the background and target modeled as stationary and moving particles, respectively. We develop an SPH-inspired image-level enhancement algorithm, using 3D spatiotemporal modeling and SPH Gaussian elliptical kernels for 3D sliding filtering. During sliding, it enhances local contrast and aggregates temporal information, improving efficiency and unifying spatiotemporal dimensions. In addition, we design an MDP-guided temporal feature perception module, comprising a lightweight feature aggregation network and a prediction propagation module. It enriches the temporal information of the target in the current frame by aggregating the reference frame, while reusing detection results from the previous frame and integrating current frame predictions to model the Markov decision. This assists in detecting the current frame across various modeling states and extracting temporal information from multiple frames. Experimental results on two public datasets DAUB [16] and DATR [13] , incorporating multiple metrics, indicate that the proposed method outperforms the state of-the-art (SOTA) methods.•We find hybird models for moving infrared dim-small target detection, with SPH and MDP playing a crucial role. These models describe target motion and background changes, optimize decision strategies, and boost detection system performance and efficiency.Experimental results on the DAUB and DATR datasets show that our method surpasses the SOTA methods.• We pioneer a mathematical approach to image-level target enhancement and design an SPH-inspired image-level enhancement algorithm. Due to SPH's ability to establish strong spatiotemporal relationships, our enhancement algorithm effectively retains details and structure in IR videos, yielding enhanced images with greater accuracy and naturalness.• We make the first attempt to treat temporal feature perception as a prediction task by designing an MDP-guided temporal feature perception module. This tightly connected and hierarchical module fully exploits temporal information and detection results from reference frames. Modeling the motion process as a Markov model follows explicit and interpretable design principles. 
related work 2.1 infrared small target detection in image: 
 Due to the characteristics of infrared imaging, traditional singleframe detection mainly focuses on modeling the relationship among the target, background, and noise. Traditional algorithms include filter-based methods such as maximum median/mean filters [9] and new top-hat filters [1] , human visual system-based local contrast algorithms like LCM [3] and the improved algorithm MPCM [29] based on local contrast, as well as detection algorithms based on sparse representation, such as IPI [14] and improved RIPT [7] , etc. However, these methods often lack balance between background suppression and target enhancement and rely heavily on manually extracted features, resulting in lower accuracy and higher false alarm rates.Influenced by CNN, single-frame infrared small target detection based on deep learning has become mainstream. Dai et al. [8] utilize bottom-up attention modulation, integrating low-level features into deeper high-level features. Zhang et al. [40] design Taylor finite difference-inspired edge blocks and direction attention aggregation blocks, effectively addressing challenges in detecting the shape of infrared small targets. In addition, Zhang et al. [38] try to introduce pruning into small target detection, and used wavelet pruning rules and regularization methods to achieve infrared efficient pruning. Zhu et al. [44] design a group of cross stage partial networks and a spatial attention module with global average contrast to obtain local and global spatial semantics. Jia et al. [17] abandon the global transformer and the convolutional sliding window of CNN, regarde the local area of the image as a graph node, and apply the graph neural network to infrared small target detection. In order to improve the multi-scale perception ability of the network, Fang et al. [12] designed a scale-adaptive feature enhancement mechanism and an attention-guided cross-weighted feature aggregator. While these single-frame detections excel in feature extraction for stationary targets, applying them to moving small targets faces performance limitations due to unique challenges.Multi-frame detection, with its unique and rich temporal information, outperforms single-frame detection in both accuracy and speed. Traditional multi-frame detection algorithms typically employ methods like energy accumulation. For instance, Zhang et al. With the development of deep learning, especially in video object detection, the focus of multi-frame infrared small target detection has shifted towards deep learning. The emergence of two-stage Faster-RCNN algorithms [23] , and the widespread application of one-stage, YOLO series object detection algorithms have driven advancements in video object detection. FGFA [46] distorts and aggregates adjacent frames onto the current frame through optical flow networks, enhancing target information. MEGA [5] utilizes both local and global temporal information to enhance detection in the current frame. YOLOV [24] , based on a one-stage detector, achieves significant success in inference speed by borrowing ideas from region proposals in two-stage detection. TransVOD++ [43] proposes a temporal Transformer to aggregate spatial object queries and feature memories of frames. However, these general detection methods excel in learning capabilities for textured medium or largesized targets but may not universally apply to multi-frame infrared small target detection due to infrared imaging characteristics.T F  t T F  Selection module index k  ' t F ' 1 t F  ' t k F  Conv Conv ' 1 t F  ' t k F  Conv Conv Norm Conv Multi-Head Attention 1 S 2 S 1 2 a 2 1 a 1 1Zhou et al. [42] propose an infrared image preprocessing and enhancement algorithm, using techniques like clahe, histogram stretching, and automatic gamma adjustment to enhance each channel separately and extract abundant feature information. But spatial domain enhancement alone is ineffective for moving target scenarios. Yan et al. [33] design a multiscale spatiotemporal difference attention network to aggregate more temporal information in feature extraction, achieving a good balance between target discovery and background suppression. Similarly, Bai et al. [2] introduce a cross-connected bidirectional pyramid structure and variable ROI pooling to enhance spatiotemporal information. Nevertheless, aggregating temporal information from complex network structures increases training and prediction costs. To mitigate this, Fan et al. [11] combine a lightweight target detection network with target tracking strategies to introduce motion target detection into tracking. Yuan et al. [34] design a dedicated module for infrared small target detection and use prior predictions during inference to guide the final output. However, the proposed method only improves upon CIOU but does not fully utilize temporal information during prediction. In summary, the above mentioned method lacks robust mathematical modeling due to the complexity of scenes and variations in this field, hindering comprehensive description with simple mathematical models. This limitation constrains the development of such detection methods, potentially resulting in subpar performance of existing algorithms in real-world applications. 
methodology 3.1 overall architecture: 
 The proposed method, illustrated in Figure 2 , is based on the YOLOX [15] framework. Given a set of input frames 𝑄 with a range of 2𝑇 + 1, where𝑄 = 𝐼 𝑡 -𝑇 , 𝐼 𝑡 -(𝑇 -1) , • • • , 𝐼 𝑡 , 𝐼 𝑡 +1 , • • • , 𝐼 𝑡 +𝑇 , beforeentering the network, it is first modeled as a 3D spatiotemporal graph using the proposed SPH-inspired image-level enhancement algorithm (Section 3.2) for enhancing targets and suppressing backgrounds. At this point, the current frame 𝐼 𝑡 in 𝑄 is significantly enhanced. Subsequently, the entire set of frames is transmitted to a feature extraction network for feature extraction. The feature extraction network adopts an FPN+PAN structure, with all input frames sharing convolutional weights. Later, the output feature set{𝐹 𝑖 } , 𝑖 = 𝑡 -𝑇 , 𝑡 -𝑇 + 1, • • • 𝑡, 𝑡 + 1, • • • 𝑡 + 𝑇 ,enters the proposed MDP-guided temporal feature perception module (Section 3.3). This module comprises three parts: selection, aggregation, and propagation. Firstly, a selection module picks out a reference frame feature map 𝐹 𝑠 (𝑠 ∈ 𝑄, 𝑠 ≠ 𝑡) more effective for the current frame feature map 𝐹 𝑡 . Then, 𝐹 𝑡 and 𝐹 𝑠 are jointly input into an aggregation network for fusion, based on a multi-head attention mechanism and multi-scale fusion network. Finally, the propagation module transfers detection results from the previous frame 𝐹 𝑡 -1 (𝑡 ≥ 1) to the current frame 𝐹 𝑡 , aiding in current frame detection. The local contrast method is commonly used for enhancing infrared small targets, but it typically employs square-shaped kernels and operates on single-frame images. In this study, we introduce an SPHinspired image-level enhancement algorithm that models sequences as 3D spatiotemporal grids. It replaces square-shaped kernels with Gaussian elliptical kernels from SPH to enhance targets in both temporal and spatial dimensions. Inspired by SPH density fields, this approach is combined with Gaussian difference, as illustrated in Figure 3 .Because of the fixed filter size and variable target sizes, using a square kernel may blend target and background information. Employing a Gaussian elliptical kernel in SPH for local contrast accommodates diverse target sizes. The Gaussian ellipse expression is as follows:EQUATIONwhere 𝐿 𝑚𝑎𝑥 denotes the maximum value among all target sizes. 𝜃 represents the rotation angle of the ellipse, with this paper is 𝜋/4.Inspired by SPH, we regard particles in the fluid as targets and background in the IR video, where the mass of particles corresponds to pixel values. The continuous density field computed by SPH is the ratio of the total mass of particles within a local sampling volume to the volume of the sampling volume. Similarly, we can approximate the pixel value of the central pixel in the elliptical kernel by the ratio of the sum of pixel values within the elliptical kernel to the area of the kernel. The calculation formula is as:EQUATIONwhere 𝑁 represents all the pixels within the elliptical kernel, 𝐼 𝑖 is their corresponding pixel values, 𝑆 Ω denotes the area of the elliptical kernel, and 𝜔 stands for the weighting coefficient, which depends on the distance between the pixel and the central pixel. Subsequently, we divide the elliptical kernel into 9 sub-windows along its major and minor axes. During the sliding process in the spatial dimension, calculate the maximum pixel value 𝐼 𝑚𝑎𝑥 in the central sub-window. Compute the average grayscale value 𝑔 𝑖 and the maximum pixel value 𝑔 𝑚𝑎𝑥 for various sub-windows around the ellipse. The final enhancement for single-image is expressed as:EQUATIONwhere 𝜀 represents the unit step function. As indicated by the formula, when the target is located in the central sub-window, the target is enhanced at that point.Based on the rotational symmetry of the Gaussian ellipse, Gaussian ellipse filtering is performed in the temporal dimension. This approach efficiently achieves edge detection and key point detection, aligning features across frames, and ultimately enhancing the target. The final aggregation enhancement formula is as follows: where 𝑁 denotes normalization, 𝑠𝑢𝑏 represents background subtraction, 𝑓 𝑤𝑎𝑟𝑝 signifies feature alignment with the SIFT algorithm, 𝑖 indicates the reference frame value, and 𝐺 stands for the Gaussian difference function. The expression is as follows:𝐸 𝑓 𝑖𝑛𝑎𝑙 = 𝑁 𝑡 +𝑇 ∑︁ 𝑖=𝑡 -𝑇 𝑠𝑢𝑏 (𝑓 𝑤𝑎𝑟𝑝 (𝐺 𝑡 * 𝐸 𝑡 -𝐺 𝑖 * 𝐸 𝑖 ) , 𝐸 𝑡 ) , (4)𝐺 (𝑥, 𝑦) = 1 √︃ 2𝜋𝜎 2 1 𝑒𝑥𝑝 - 𝑥 2 + 𝑦 2 2𝜎 2.(5)MDP display memorylessness, where the probability of future states depends only on the current state and is unaffected by past states. This property makes them well-suited for handling sequential data like image sequences, effectively capturing relationships between time intervals. Moreover, MDP effectively model the relationship between past data and future decisions, making them suitable for predicting target positions, states, and other information in time sequences. Accordingly, we can treat temporal feature perception as a prediction task by MPD. And we develop an MDP-guided temporal feature perception module to simplify modeling temporal data by abstracting it into states and corresponding transition probabilities. This reduction in complexity cuts computational costs and enhances interpretation of the model's behavior.In fact, the MDP-guided temporal feature perception module is divided into three sub-modules: frame selection module, featurelevel spatiotemporal information aggregation module, and framelevel prediction information propagation module.Frame selection module. According to the backbone structure of YOLOX, the frame set 𝑄 ∈ R 3×𝑇 ×𝑊 ×𝐻 , after passing through the feature extraction network, produces outputs for three scales: ) is chosen to match the image scale required by the similarity calculation algorithm. First, we compute the similarity between frames 𝐹 𝑡 -1 , 𝐹 𝑡 , and 𝐹 𝑡 +1 using the perceptual hash algorithm (𝑝𝐻𝑎𝑠ℎ) as:𝐹 𝑗 𝑖 ∈ R 𝐶 𝑗 × (2𝑇 +1) ×𝑊 𝑗 ×𝐻 𝑗 , 𝑗 = 1,EQUATIONwhere 𝑝𝐻𝑎𝑠ℎ stands for the perceptual hash algorithm. This algorithm takes fixed-size 32×32 inputs and employs Discrete Cosine Transform (DCT) for pairwise image comparison, yielding the similarity value through Hamming distance.After obtaining three sub-similarity values, they are normalized to the range (1,𝑇 ), while acquiring the normalization weight 𝜔 𝑛𝑜𝑟 . The expression for 𝜔 𝑛𝑜𝑟 is:EQUATIONwhere 𝜔 𝑛𝑜𝑟 represents the normalized coefficient weight. The final index value is calculated as:EQUATIONwhere [] represents rounding up, 𝑁 is the total number of obtained similarity values. Frame selection effectively avoids redundancy in spatiotemporal information. High similarity between the current frame and adjacent frames may indicate occlusion, stationary targets, or slow motion. In such cases, the index obtains a larger value, directing attention to more distant and relevant frames. This aligns with human visual perception, extracting richer information from distant frames and eliminating redundant information from similar adjacent frames, facilitating target acquisition. These principles serve as the starting point for designing frame selection modules. Feature aggregation module. After selecting reference frames, the chosen feature set 𝐹 ′ 𝑖 , where 𝑖 = 𝑡 -𝑘, 𝑡 -1, 𝑡, 𝑡 + 1, 𝑡 + 𝑘 (𝑘 = 𝑖𝑛𝑑𝑒𝑥), is fed into a lightweight aggregation network for spatiotemporal fusion. This network aggregates features from adjacent frames 𝐹 ′ 𝑡 +1 and further reference frames 𝐹 ′ 𝑡 +𝑘 , followed by fine aggregation with the current frame 𝐹 ′ 𝑡 . The aggregation process resembles a transformer's encoder-decoder structure, incorporating residual connections and multi-head attention modules to enhance temporal features. Due to network symmetry, the explanation is based on one side's structure. Initially, the spatiotemporal information of the distant reference frame 𝐹 𝑡 +𝑘 is aggregated with 𝐹 𝑡 +1 . The aggregation formula is as follows:EQUATIONwhere 𝜎 represents the sigmoid activation function, [] denotes concatenation, ⊗ is element-wise multiplication, ⊕ indicates elementwise addition, and 𝑓 stands for the convolution operation. Similarly, 𝐹 2 𝐴𝑔𝑔 is derived from 𝐹 𝑡 -𝑘 and 𝐹 𝑡 -1 . 𝐹 1 𝐴𝑔𝑔 and 𝐹 2 𝐴𝑔𝑔 now represent feature maps obtained by fusing temporal features from adjacent and distant reference frames. Subsequently, they are concatenated with the current frame in preparation for the final temporal feature aggregation. The formula is as follows:EQUATION)where 𝑃𝐸 is the added positional encoding, 𝐹 ′ 𝑡 represents the current frame. The input 𝐹 ′ 𝐴𝑔𝑔 undergoes initial feature extraction through normalization and convolutional layers. The formula is:EQUATIONwhere 𝑁𝑜𝑟𝑚 denotes normalization, 𝑓 represents the convolution operation. Following enhancement processing via the multi-head attention module, we obtain the final feature map 𝐹 𝐴𝑔𝑔 , defined as follows:EQUATION)where Φ represents the multi-head attention module. Predictive propagation module. After feature aggregation, the aggregated frame 𝐹 𝐴𝑔𝑔 is fed into the decoupling head for information prediction. The predicted results, including target coordinates, classification information, and object presence, are then decoded. When 𝑡 ≥ 1, we model the motion of small IR targets using an MDP. We utilize the detection results from the previous frame to correct the detection results for the current frame. The MDP consists of quintuplicate variables: 𝑀𝐷𝑃 (𝑆, 𝐴, 𝑃, 𝑅, 𝜋). Here, 𝑆 represents target states, categorized as presence or absence. 𝐴 stands for the actions performed between states, including detecting the IR target in both frames, not detecting the IR target in both frames, detecting the IR target in the previous frame but not in the subsequent frame, and not detecting the IR target in the previous frame but detecting it in the subsequent frame. 𝑃 denotes the set of state transitions, while 𝑅 signifies the reward function, indicating the rewards obtained when different actions are taken in a certain state. The policy 𝜋 defines the actions 𝐴 that the model may take under various states 𝑆, along with their corresponding probabilities. By learning each policy, we obtain locally maximal rewards and ultimately achieve the optimal result for the entire detection process. Additionally, we define a dynamic frame variable 𝑘, storing the most recent IR target detection outcome.If the IR target is detected in the current frame, we match the policy with the result from the previous frame. The reward function at this point is defined as follows:EQUATION)where 𝑖 and 𝑗 are the top five highest-scoring results detected in the current frame and the previous frame, respectively. 𝜀 represents the IR target state of the previous frame, defines as follows:EQUATIONIf the IR target remains undetected in the current frame, the reward function is as follows:EQUATIONwhere 𝜏 𝑘 represents the results detected in the most recent frame 𝑘. Actually, the overall reward function is defined as follows:EQUATION)where 𝑝 and 𝑞 have the same expression as 𝜀, representing the values of the previous frame and the current frame in two different states 𝑆. Finally, the adjusted IR target box and score information are compared with the ground truth for loss calculation, and the ultimate loss expression is as follows:GT SANet UIU-Net YOLOX TransVOD++ YOLOV Ours Faster-RCNN (a) (g) (f) (e) (d) (c) (b) (h)EQUATIONwhere 𝐿 𝑟𝑒𝑔 indicates the regression loss, 𝐿 𝑐𝑙𝑠 is the classification loss, and 𝐿 𝑜𝑏 𝑗 represents the confidence loss.  
experiment 4.1 dataset and implementation details: 
 EQUATIONEQUATIONwhere TP, FP, FN denote the true positive, false positive, false negative, respectively.In addition, we compute the PR curve of the model. The PR curve reflects the relationship between precision and recall at different confidence levels. From this curve, we derive mAP using the following formula:EQUATIONIn this study, 𝑛 denotes the number of target categories, which is 1. Additionally, we use an IOU threshold of 0.5 for computing mAP, referred to as mAP50.We quantitatively compare the proposed method's performance based on mAP50, precision, recall rate, and F1 score, along with Precision-Recall (P-R) curves. Tables 1 and 2 present the metrics for the 9 object detection methods, including UIU-Net, DNA-Net, SANet, Faster-RCNN, YOLOv5, YOLOX, YOLOv8, TransVOD++ and YOLOV. Our method consistently outperforms others across all metrics, showing superior detection performance. For instance, on the DAUB dataset, our method improves precision, recall, F1, and mAP50 by 3.84%, 0.86%, 2.37%, and 4.94% over the secondranked methods(DNA-Net), respectively. Notably, Faster-RCNN performs the poorest, potentially due to excessive candidate box generation. While YOLO series detectors show promising results, image enhancement algorithms in preprocessing may overshadow objects, affecting overall performance. Infrared small target detection algorithms achieve excellent results but are hampered by complex networks, leading to slow training and inference speeds. Video detection algorithms like YOLOV and TransVOD++ exhibit poor performance due to unsuitable temporal feature aggregation networks for infrared small targets. While on DATR dataset, our method likewise achieves the best results on four metrics. It improves precision, recall, F1, and mAP50 by 0.6%, 2.01%, 0.86%, and 2.11% over the second-ranked methods(YOLOX). Compared with the DAUB dataset, most of the methods have achieved better improvement on the DATR dataset, and we believe that the reason is that the small targets in the DATR dataset are relatively larger, The Precision-Recall (P-R) curve, illustrated in Figure 5 and 6 , is a crucial comprehensive metric. It computes precision and recall at various thresholds, revealing the correlation between them and evaluating the relevance of detection results. A curve closer to the upper right corner signifies superior network performance. In Figure 5 , our curve, highlighted in red, notably covers almost all the compared methods in the upper right corner, demonstrating our network's superior balance between accuracy and recall, resulting in the best overall performance. In addition, it can be seen that the curve of Faster-RCNN is the most flat, and the curve of YOLOV and YOLOv8 ends quickly with the increase of recall. The curves of the rest of the methods are not much different, but they are all below our curve as a whole. This is consistent with the above analysis and experimental results.For a more intuitive comparison of contrastive effects, we select six existing methods for visual comparison with our network. And we choosed several typical scenarios in the DAUB dataset, including mountains, clouds, cities, and forests. As depicted in Figure 4 , our method accurately locates small targets without producing missed detections or false positives at the same IOU threshold, and it can be clearly seen that the results of our method are highly consistent with the groundtruth. Faster-RCNN exhibits the highest number of missed detections and false positives. This is in line with the results of quantitative experiments. Among single-frame infrared small target detection networks, SANet achieves the highest accuracy but still encounters false positives and missed detections. In the case of the two video detectors, TransVOD++ demonstrates subpar visual results, possibly due to challenges in training as a transformer detector and limited applicability to multi-frame infrared small target detection as a general detection framework. YOLOV has a good detection effect in simple backgrounds, but missed detections and false detections occur in complex backgrounds and very small target situations. Because of the image preprocessing stage of YOLO causes confusion of complex backgrounds and targets, resulting in submerged targets. YOLOX showed good results, but their detection boxes did not have the best coincidence with the groundtruth.To validate the effectiveness of our proposed modules, we conduct ablation experiments, with results shown in Table 3 . The base network solely utilizes YOLOX as the detector, without integrating the SPH-inspired image-level enhancement algorithm and MDP-guided temporal feature perception module. We then progressively add these components to YOLOX and observed their impact. Results indicate significant enhancements when incorporating both the MDP-guided temporal feature perception module and the SPH-inspired image-level enhancement algorithm into the base framework. Specifically, on both datasets, the baseline with the SPH-inspired image-level enhancement algorithm improves mAP50 by 1.38% and 1.52%, respectively. Similarly, the baseline with the MDP-guided temporal feature perception module improves mAP50 by 3.07% and 1.94%, respectively. Notably, combining both modules Through ablation experiments, we observe a notable trend: the MDP-guided temporal feature perception module outperforms the SPH-inspired image-level enhancement algorithm. Further analysis reveals that while the SPH-inspired algorithm enhances targets and suppresses background using spatiotemporal information, it faces challenges in detecting small, dark infrared targets against bright backgrounds and clutter. In contrast, the subsequent MDP-guided temporal feature perception module leverages Markov modeling to address moving small targets and eliminate static bright backgrounds, resulting in significantly enhanced detection performance. Essentially, the SPH-inspired algorithm provides coarse localization, reducing false negatives, while the MDP-guided module offers fine localization, reducing both false negatives and false positives. Consequently, comprehensive analysis supports the superiority of the MDP-guided temporal feature perception module in enhancing infrared small targets compared to the SPH-inspired image-level enhancement algorithm. 
conclusion: 
 This paper presents a multi-frame infrared small target detection network by finding hybird models: SPH and MDP. SPH simulates fluid behavior by dividing it into particles and modeling their interactions. It can also simulate information propagation and interaction in images, enhancing them. MDP, known for their memorylessness, is effective for handling sequential data like image sequences, capturing relationships between time intervals. Accordingly, the MDP-guided temporal feature perception module effectively addresses challenges posed by complex backgrounds and occlusions, while the SPH-inspired image-level enhancement algorithm tackles issues arising from camera shake and motion blur. Results on the dataset demonstrate improved accuracy in target detection with reduced false positive and false negative rates. Ablation experiments highlight the contributions of the designed modules and algorithms to enhancing network performance. In the future, we can consider integrating data from different sensors (such as infrared and visible light) to enhance the performance and robustness of target detection. Further algorithm optimization is also essential to minimize power usage and enhance real-time capabilities, meeting the requirements of resource-constrained environments and real-time applications. 
 
 Appendix:  


Decision: This paper received contrasting reviews in origin, WR, BA, BA, WR, which became 2 R, WR, and BA after rebuttal.
The main weaknesses raised by the reviewers regard the low novelty of the method, weak motivations for the design of the proposed approach, and the insufficient/incomplete experimental results, and ablations especially. Other issues regard the computational cost and the request of clarification of some parts of the method and of the parameters used within.

The rebuttal is actually addressing these issues in a reasonable manner, the method does have a theoretical ground representing an original contribution, and new results and estimates are provided as requested by the reviewers.
To the AC and SAC opinions, the rebuttal is satisfactorily addressing the raised remarks. 
For these reasons, the paper is considered acceptable for publication at ACM MM 2024 conference.

Reviews: review: paper_summary: paper_summary: The paper addresses the challenges of detecting small moving infrared targets. This paper proposes a hybrid modeling method for moving
infrared small target detection via smoothed-particle hydrodynamics (SPH) and Markov decision processes (MDP). It achieves good performance on SOTA benchmarks.
strength_weakness: strength_weakness: The paper achieves SOTA performance on existing benchmarks.
limitations: limitations: The paper is hard to follow, the Fig3 is confusing.

The cross-frame interaction is quite simple, i.e., a simple convolutional-based attention mechanism. This is also well-explored in the CV community. 

The proposed methods are quite general for generic video applications. The reviewer doesn't see why this is the best choice for infrared images and for small targets. The motivation is not clear

The qualitative is not convincing. The improvement is not visible.

The authors didn't provide the computational cost for the proposed method and for comparisons.
paper_summary: paper_summary: This paper, propose a hybrid modeling method for moving infrared small target detection via smoothed-particle hydrodynamics (SPH) and Markov decision processes (MDP).
strength_weakness: strength_weakness: 1.This paper  develop an SPH-inspired image-level enhancement algorithm which models the image sequence of infrared video as a 3D spatiotemporal graph in SPH. 
2.This paper make the first attempt to treat temporal feature perception as a prediction task by designing an MDP-guided temporal feature perception module.
limitations: limitations: 1. Lack of comparison in terms of parameter count or inference time.
2. Missing analysis on lambda and T.
3.Apart from the YOLO series, there is a lack of analysis on other types of methods, such as transformer-based methods or those based on key points.
paper_summary: paper_summary: The paper proposes a hybrid modeling method for moving infrared small target detection using smoothed-particle hydrodynamics (SPH) and Markov decision processes (MDP). The SPH simulates the motion trajectories of targets and background scenes, while the MDP optimizes detection system strategies for optimal action selection based on contexts and target states. The proposed method includes an SPH-inspired image-level enhancement algorithm and an MDP-guided temporal feature perception module. The experimental results on two public datasets, DAUB and DATR, demonstrate that the proposed method outperforms state-of-the-art methods in terms of objective metrics and visual quality.
strength_weakness: strength_weakness: The paper introduces a novel hybrid modeling approach for moving infrared small target detection, utilizing SPH and MDP, which effectively describes target motion and background changes and optimizes decision strategies.

The proposed SPH-inspired image-level enhancement algorithm effectively retains details and structure in IR videos, resulting in enhanced images with greater accuracy and naturalness.

The paper pioneers a mathematical approach to image-level target enhancement and introduces an MDP-guided temporal feature perception module, which fully exploits temporal information and detection results from reference frames.
limitations: limitations: In Section 3.2, it is mentioned that according to the backbone structure of YOLOX, the frame set 𝑄 ∈ $R^{3×𝑇×𝑊×𝐻}$. It would be beneficial to clarify the significance of "T" in this context and provide details on the number of reference frames input into the module. Additionally, including any ablations conducted would be helpful, as the number of reference frames can reasonably affect the model's performance.

Regarding the SPH-inspired image-level enhancement algorithm, it would be valuable to specify the number of reference frames used in this process and whether the number of references affects the algorithm's performance. If so, providing additional ablations in this area would enhance the understanding of the algorithm's behavior.

In the conclusion part, specifically in line 925-926, the authors mention meeting the requirements of resource-constrained environments and real-time applications. It would be beneficial to provide information on the inference time and resource usage of the proposed method. Additionally, it would be valuable to include comparisons with state-of-the-art methods to demonstrate the effectiveness of the proposed approach in resource-constrained and real-time scenarios.
paper_summary: paper_summary: This paper presents a hybrid modeling approach using Smoothed Particle Hydrodynamics (SPH) and Markov Decision Processes (MDP) for mobile infrared small target detection. SPH simulates the motion trajectories of targets and background scenes, while MDP optimizes detection system strategies based on environmental and target states to achieve optimal action selection guided by context and target conditions.
strength_weakness: strength_weakness: - The MDP-guided temporal feature perception module effectively addresses challenges posed by complex backgrounds and occlusions. The image-level enhancement algorithm inspired by SPH tackles issues arising from camera shake and motion blur.
limitations: limitations: This paper has two issues in the experimental section.

1. Although the DAUB dataset and DATR dataset to some extent reflect the model's good performance, there is a shortage of certain datasets, such as NUDT-SIRST and NUAA-SIRST, in the testing phase. Does this lack of data raise concerns about evaluating the model's generalization and robustness?

2. In the selected models, while the model proposed in this paper demonstrates superior performance in results, YOLO, Faster-RCNN, and TransVOD++ are not tailored for tasks such as detecting small infrared targets. For example, DNANet and ISNet, which are designed for the same task, exhibit excellent performance. Does this indicate potential issues with the selection of models for comparison?