Reviewerly/exHarmony huggingface dataset:
Object Centric Body Part Attention Network for Human-Object Interaction Detection

The current transformer-based human object interaction (HOI) detection methods have achieved great progress, however, these methods adopt the same structre of decoder to detect human and object, which limits the accuracy of object feature extraction, thereby limiting the accuracy of HOI detection. And due to the distribution differences of multi-granularity features between human and object, the key of HOI detection is object centric interaction with the correlative human body parts. To address this issue, we propose an Object Centric Body Part Attention Network for Human-Object Interaction. First, we introduce a dual-branch decoder for human and object detection named Object Centric Decoder (OCD), where one focuses on quering objects and another pay attention to catch human who interacts with them. Secondly, in order to exploit more fine-grained human body information centered around object, we propose a Body Part Attention (BPA) module to obtain the interactive human body part features for HOI detection. We evaluated our proposed OBPA on the HICO-DET and V-COCO datasets, which significantly outperforms existing counterpart (1.7 mAP on V-COCO, and 0.9 mAP on HICO-DET compared to GEN-VLKT). Code will be available on https://github.com/zhuang1iu/OBPA-NET .
Knowledge Transfer via Leveraging Teacher-Student Network with Visual Attention to Enhance Atmospheric Sand Image Restoration

Deep learning-based methods have recently shown superiority in sand dust image restoration tasks. However, most existing learning-based methods do not pay much attention to the gap between the synthetic image and the real image, making the obtained recovery effect difficult to generalize on the real image. To alleviate this problem, we propose a novel Teacher-Student network (TS-Net) for real-world sand dust image restoration that maximizes the transfer of knowledge learned in the Teacher Network. Thereby improving the recovery ability of the Student Network on real sand dust images. Firstly, the Teacher Network extracts the latent feature information of dust images. Furthermore, use transfer learning to transfer the knowledge learned by the Teacher Network to the faster and more efficient Student Network. We present a Visual Attention (VA) mechanism to couple multi-scale features with contextual features with large kernel convolutions to fully utilize these complementary features to correct the color of sand dust images. Due to the large kernel convolution and feature fusion strategy, the fused features can correct the color cast the problem of dust color, which is very important for high-level tasks. Extensive experiments on the synthetic and real datasets demonstrate that the proposed method significantly improves over the state-of-the-art methods.
TPNet: Enhancing Weakly Supervised Polyp Frame Detection with Temporal Encoder and Prototype-Based Memory Bank

Polyp detection plays a crucial role in the early prevention of colorectal cancer. The availability of large-scale polyp video datasets and video-level annotations has spurred research efforts to formulate polyp detection as a weakly-supervised anomaly detection task, which leverages video-level labeled training data for detecting frame-level polyps. However, few studies have investigated the impact of specific properties within polyp videos, including temporal dynamics, ambiguity, and complex noise. In this work, we propose TPNet, a novel framework that addresses several challenges posed by colonoscopy videos, for weakly-supervised polyp frame detection. Specifically, we design a Temporal Encoder that effectively capturing the temporal dynamics and intricate patterns within polyp video segments to foster accuracy. Additionally, we introduce a Prototype-based Memory Bank that facilitates the storage and retrieval of significant discriminative information, which enhance the sensitivity and robustness in ambiguous and complicated conditions. Experiments conducted on one of the largest and most challenging colonoscopy datasets demonstrate that our proposed TPNet achieves state-of-the-art performance, surpassing the latest cutting-edge method with 6.19% in average precision (AP).
Multi-task Self-supervised Few-Shot Detection

Few-shot object detection involves detecting novel objects with only a few training samples. But very few samples are difficult to cover the bias of the new class in the deep model. To address the issue, we use self-supervision to expand the coverage of samples to provide more observation angles for new classes. In this paper, we propose a multi-task approach that combines self-supervision with few-shot learning to exploit the complementarity of these two domains. Specifically, our self-supervision as an auxiliary task to improve the detection performance of the main task of few-shot learning. Moreover, in order to make self-supervision more suitable for few-shot object detection, we introduce the denoising module to expand the positive and negative samples and the team module for precise positioning. The denoising module expands the positive and negative samples and accelerate model convergence using contrastive denoising training methods. The team module utilizes location constraints for precise localization to improve the accuracy of object detection. Our experimental results demonstrate the effectiveness of our method on the Few-shot object detection task on the PASCAL VOC and COCO datasets, achieving promising results. Our results highlight the potential of combining self-supervision with few-shot learning to improve the performance of object detection models in scenarios where annotated data is limited.
Dilemmatic spaces pertaining to digitalisation, equity and increased goal attainment in Swedish schools and preschools

The digitalisation of school systems and teaching often comes with expectations to enhance both equity and student outcomes, particularly as stated in policy documents. When curricular goals relating to student’s proficiency in using digital tools in their daily lives are introduced, teachers bear the responsibility of ensuring that these goals can be achieved by all students. This article analyses teachers’ perspectives on the interplay between digitalisation, digital competence, equity, and students’ increased attainment of curricular goals. The interviewed teachers worked from preschool to upper-secondary school levels in the same municipality in Sweden, and their responses were subjected to a thematic analysis. The analysis shed light on three dilemmatic spaces, namely flexibility in teaching and learning, equity and students’ increased goal attainment, and becoming a citizen. Within these dilemmatic spaces, teachers faced the challenge of making decisions regarding their actions and their alignment with various expectations, including the well-being of their students, institutional rules and regulations, and policy guidelines.
ADORE: Adaptive Diffusion Optimized Restoration for AI-Generated Facial Imagery

We introduce ADORE (Adaptive Diffusion Optimized Restoration), a pioneering solution that addresses facial distortion issues in diffusion-based, language-guided image generation. ADORE enhances facial quality based on image characteristics and style, improving the visual fidelity of AI-generated images. It also mitigates boundary distortions during the face-background fusion process, offering a novel approach to address instability issues by using generative models for image restoration. Rigorous experiments validate ADORE’s proficiency in achieving high-quality, style-consistent facial restorations. ADORE supports text-driven, fine-tuned facial refinement, leveraging the model’s open-domain synthesis capability. As the first method tailored to enhance facial generation quality in text-to-image models, with its versatility and innovative solutions, ADORE successfully addresses a pressing issue and paves new avenues in image generation.
Out-of-Distribution with Text-to-Image Diffusion Models

Out-of-distribution detection, identifying unexpected data from the known concepts, is essential for reliable machine learning. We present a novel method that explores the application of a text-to-image diffusion model for out-of-distribution detection. Our method is motivated by the fact that the text-to-image diffusion model has shown remarkable capability in generating high-quality images with diverse text descriptions. The text description generates a corresponding text embedding and is injected into the diffusion model to affect image generation. This demonstrates that its internal representation contains semantic information and is highly enhanced by text concepts. This inspires us to apply the diffusion model to extract image representations with suitable text embeddings. In addition, we noticed that describing images directly using native text is often vague and lacking in detail. Thus, we propose an implicit captioner to generate text embeddings for the input images. Subsequently, a compression head is introduced to compress the representations, facilitating easy comparison and removal of noise information. We formulate the proposed text-to-image diffusion model, implicit captioner, and compression head into a network, which we call ODDM: Out-of-distribution Detection with Text-to-Image Diffusion Models. Several experiments shows that our method can achieved superior performance.
RTMDet-R2: An Improved Real-Time Rotated Object Detector

Object detection in remote sensing images is challenging due to the absence of visible features and variations in object orientation. Efficient detection of objects in such images can be achieved using rotated object detectors that utilize oriented bounding boxes. However, existing rotated object detectors often struggle to maintain high accuracy while processing high-resolution remote sensing images in real time. In this paper, we present RTMDet-R2, an improved real-time rotated object detector. RTMDet-R2 incorporates an enhanced path PAFPN to effectively fuse multi-level features and employs a task interaction decouple head to alleviate the imbalance between regression and classification tasks. To further enhance performance, we propose the ProbIoU-aware dynamic label assignment strategy, which enables efficient and accurate label assignment during the training. As a result, RTMDet-R2-m and RTMDet-R2-l achieve 79.10% and 79.46% mAP, respectively, on the DOTA 1.0 dataset using single-scale training and testing, outperforming the majority of other rotated object detectors. Moreover, RTMDet-R2-s and RTMDet-R2-t achieve 78.43% and 77.27% mAP, respectively, while achieving inference frame rates of 175 and 181 FPS at a resolution of 1024 × 1024 on an RTX 3090 GPU with TensorRT and FP16-precision. Furthermore, RTMDet-R2-t achieves 90.63/97.44% mAP on the HRSC2016 dataset. The code and models are available at https://github.com/Zeba-Xie/RTMDet-R2 .
JLInst: Boundary-Mask Joint Learning for Instance Segmentation

Lots of methods have been proposed to improve instance segmentation performance. However, the mask produced by state-of-the-art segmentation networks is still coarse and does not completely align with the whole object instance. Moreover, we find that better object boundary information can help instance segmentation network produce more distinct and clear object masks. Therefore, we present a simple yet effective instance segmentation framework, termed JLInst (Boundary-Mask Joint Learning for Instance Segmentation). Our methods can jointly exploit object boundary and mask semantic information in the instance segmentation network, and generate more precise mask prediction. Besides, we propose the Adaptive Gaussian Weighted Binary Cross-Entropy Loss (GW loss), to focus more on uncertain examples in pixel-level classification. Experiments show that JLInst achieves improved performance (+3.0% AP) than Mask R-CNN on COCO test-dev2017 dataset, and outperforms most recent methods in the fair comparison.
AHT: A Novel Aggregation Hyper-transformer for Few-Shot Object Detection

Few-shot object detection aims to detect novel objects with few annotated examples and this task has been extensively investigated by meta-learning-based paradigm. However, most of the previous approaches suffer from: 1) Most of the previous methods only perform two-branch interaction in the detection head which lacks the interaction of low-level semantic features. 2) Traditional method is difficult to capture the fine-grained differences between categories due to fixed weights. To alleviate these issues, we proposed a simple yet effective method, named Aggregation Hyper-Transformer (AHT) framework, which can generate corresponding weights into the primary network by hypernetworks mechanism. In particular, we design a novel Dynamic Aggregation Module and a Conditional Adaptation Hypernetworks, which apply the aggregated category vectors as conditions to dynamically generates class-specific parameters. Benefiting from the above two modules, our method significantly exceeds the previous meta-learning methods and provides new insights for my community.
FRNet: Improving Face De-occlusion via Feature Reconstruction

Face de-occlusion is essential to improve the accuracy of face-related tasks. However, most existing methods only focus on single occlusion scenarios, rendering them sub-optimal for multiple occlusions. To alleviate this problem, we propose a novel framework for face de-occlusion called FRNet, which is based on feature reconstruction. The proposed FRNet can automatically detect and remove single or multiple occlusions through the predict-extract-inpaint approach, making it a universal solution to deal with multiple occlusions. In this paper, we propose a two-stage occlusion extractor and a two-stage face generator. The former utilizes the predicted occlusion positions to get coarse occlusion masks which are subsequently fine-tuned by the refinement module to tackle complex occlusion scenarios in the real world. The latter utilizes the predicted face structures to reconstruct global structures, and then uses information from neighboring areas and corresponding features to refine important areas, so as to address the issues of structural deficiencies and feature disharmony in the generated face images. We also introduce a gender-consistency loss and an identity loss to improve the attribute recovery accuracy of images. Furthermore, to address the limitations of existing datasets for face de-occlusion, we introduce a new synthetic face dataset including both single and multiple occlusions, which effectively facilitates the model training. Extensive experimental results demonstrate the superiority of the proposed FRNet compared to state-of-the-art methods.
Learning Frequency-Based Disentanglement and Filtering for Generalizable Person Re-identification

Domain Generalization (DG) in Person Re-identification (ReID) tackles the task of testing in unseen domains without using target domain data during training. Existing DG ReID methods achieve impressive performance with unified ensemble models or multi-expert hybrid networks. However, as the number of source domains increases, complex relationships between training samples result in domain-invariant characteristics with spurious correlations, impacting further generalization. To address this, we propose a Bilateral Frequency-Aware Network(BFAN) that leverages spectral feature correlation learning for discriminative hybrid features. BFAN includes a Bilateral Frequency Component-guided Attention (BFCA) module to capture semantic information from diverse frequency features and fuse it with spatial features. Additionally, a Fourier Noise Masquerade Filtering (FNMF) module is introduced to suppress non-generalization-supporting components in the frequency domain. Extensive experiments on various datasets demonstrate our method’s notably competitive performance.
Feature Implicit Enhancement via Super-Resolution for Small Object Detection

In recent years, object detection has made significant strides due to advancements in deep convolutional neural networks. However, the detection performance for small objects remains challenging. The visual information of small objects is easily confused with the background and even more likely to get lost in a series of downsampling operations due to the limited number of pixels, resulting in poor representations. In this paper, we propose a novel approach namely Feature Implicit Enhancement via Super-Resolution (FIESR) to learn more robust feature representations for small object detection. Our FIESR consists of two detection branches and requires two steps of training. Firstly, the detector learns the relationship between low-resolution and corresponding original high-resolution images to enhance the representations of small objects by minimizing a super-resolution loss between the two branches. Secondly, the detector is fine-tuned on original resolution images to fit extremely large objects. Additionally, our FIESR could be applied to various popular detectors such as Faster-RCNN, RetinaNet, FCOS, and DyHead. Our FIESR achieves competitive results on COCO dataset and is proved effective and flexible by extensive experiments.
HTNet: A Hybrid Model Boosted by Triple Self-attention for Crowd Counting

The swift development of convolutional neural network (CNN) has enabled significant headway in crowd counting research. However, the fixed-size convolutional kernels of traditional methods make it difficult to handle problems such as drastic scale change and complex background interference. In this regard, we propose a hybrid crowd counting model to tackle existing challenges. Firstly, we leverage a global self-attention module (GAM) after CNN backbone to capture wider contextual information. Secondly, due to the gradual recovery of the feature map size in the decoding stage, the local self-attention module (LAM) is employed to reduce computational complexity. With this design, the model can fuse features from global and local perspectives to better cope with scale change. Additionally, to establish the interdependence between spatial and channel dimensions, we further design a novel channel self-attention module (CAM) and combine it with LAM. Finally, we construct a simple yet useful double head module that outputs a foreground segmentation map in addition to the intermediate density map, which are then multiplied together in a pixel-wise style to suppress background interference. The experimental results on several benchmark datasets demonstrate that our method achieves remarkable improvement.
Robust Degradation Representation via Efficient Diffusion Model for Blind Super-Resolution

Blind super-resolution (SR) is a challenging low-level vision task, dedicated to recovering corrupted details in low-resolution (LR) images with complex unknown degradations. The mainstream blind SR methods mainly adopt the paradigm of capturing the robust degradation representation from the LR images as condition and then perform deep feature reconstruction. However, the manifold degradation factors make it challenging to achieve flexible estimation. In this paper, we propose a residual-guided diffusion degradation representation scheme (Diff-BSR) for blind SR. Specifically, we leverage the powerful generative capability of the diffusion model (DM) to implicitly model the diverse degradations representation, which helps to resist to the disturbance of varied input. Meanwhile, to reduce the expensive computational complexity and training costs, we design a lightweight degradation extractor in the residual domain. It transforms the target residual distribution in a low-dimension feature space. As a result, Diff-BSR requires only about 60 sampling steps and a much smaller scale denoising network. Moreover, we designed the Degradation-Aware Multihead Self-Attention mechanism to effectively fuse the discriminative representations with the intermediate features of the network for robustness enhancement. Extensive experiments on mainstream blind SR benchmarks show that Diff-BSR achieves SOTA or comparable performance compared to existing methods.
SentiImgBank: A Large Scale Visual Repository for Image Sentiment Analysis

In contrast to existing methods which detect sentiment directly from low/high-level features, we construct a large-scale visual repository, namely SentiImgBank, with the aim of providing a benchmarking visual sentiment lexicon. More specially, the SentiImgBank consists of 24 categories, 5,487 adjective-noun pairs (ANPs), in total of 648,946 images that are collected from social media such as Twitter. In view that ANPs might express different sentiments in different contexts, i.e., contextuality, SentiImgBank annotates the discrete sentiment and emotion scores instead of directly defining the golden label. Hence, each image is associated with ten numerical scores, where three of them are sentiment scores, the remaining seven scores denote different emotions. To alleviate the manually annotation cost, a committee of 15 pre-trained language models based classifiers is proposed to automatically produce the sentiment and emotion scores. Finally, the strong baselines are proposed to evaluate the potential of SentiImgBank. We hope this study provides a publicly available resource for visual sentiment analysis. The full dataset will be publicly available for research ( https://github.com/anonymity2024/SentiImgBank ).
A BiGRU Based Adaptive Gain Estimation for Radar Multi-target Tracking

The currently available multi-target tracking algorithms were developed based on ideal tracking settings, which are unsuitable for actual combat situations. To handle the challenge of unknown measurement noise and low detection probability, this paper presents a novel adaptive gain estimation (AGE) approach with a survival likelihood estimation method. The former utilizes the bidirectional gate recurrent unit (BiGRU) to assign weights to both measurement and prediction in the absence of priori information. The latter leverages a binary classification network to determine the likelihood of target survival. Furthermore, the AGE can be seamlessly integrated with prediction and data association modules to form an end-to-end model known as MTT-AGE (Multi-target Tracking with Adaptive Gain Estimation). The results of MIT trajectory dataset, simulated scenarios and real-world data confirm the efficiency and stability of the MTT-AGE. Furthermore, the ablation experiments are conducted to verify the effectiveness of AGE based on MIT trajectory dataset and simulated scenarios where there is only a single objective.
SpectralTracker: Jointly High and Low-Frequency Modeling for Tracking

Recently, a considerable number of top-performing Transformer based trackers have been proposed. However, most of them mainly focus on utilizing low-frequency information from a spatial-spectral analysis perspective, limiting their performance in complicated scenes. To address this problem, we propose a spectral tracker that explores how to capture high and low-frequency information for robust tracking jointly. Specifically, we design a novel dual-spectral information extraction and aggregation module (DSM) consisting of a high and low-frequency branch to capture and combine complementary frequency information of a Transformer effectively. Firstly, we divide the local window in the high-frequency branch to focus on more fine-grained high-frequency information. Then, in the low-frequency branch, we apply AvgPooling with a low-pass effect on a Transformer to amplify its low-frequency information. Furthermore, we design a shared MLP strategy to polarize the dual-frequency branching to high and low-frequency information attention. Finally, we utilize an MLP to complementarily fuse high and low-frequency information for frequency domain modeling. Comprehensive experiments on five tracking benchmarks (i.e., GOT-10k, TrackingNet, LaSOT, UAV123 and TNL2K) show that our spectral tracker achieves better performance than the state-of-the-art trackers.