Reviewerly/exHarmony huggingface dataset:
Few-Shot Object Detection via Transfer Learning and Contrastive Reweighting

In recent years, there has been increasing interest in few-shot object detection (FSOD), which involves detecting novel objects from just a few annotated examples. Transfer learning has been identified as an effective method for solving this task, as it allows the model to learn better feature embedding. In this paper, we utilize the Faster R-CNN framework and its potential to address FSOD tasks. We optimize the fine-tuning process of the detector based on the characteristics of the data distribution in the few-shot scenario. Additionally, we propose a contrastive reweighting module that enhances the classification of detected objects by making the embedding space more discriminative. Our extensive experiments on PASCAL VOC and MS COCO benchmarks demonstrate that our proposed method outperforms existing works and achieves state-of-the-art results.
SDGC-YOLOv5: A More Accurate Model for Small Object Detection

Small objects often suffer from size and resolution limitations, resulting in poor detection performance when employing traditional object detection models. To address these challenges, we propose SDGC-YOLOv5, a novel model based on YOLOv5. Our contributions are as follows: a) Replacement of the original convolution layer with spatial depth convolution (SD) to enable dense feature extraction and improve the detection of small objects. b) Utilization of a lightweight global context network (GC) to enhance small object recognition capabilities. c) Incorporation of the shifted window scheme to enhance the extraction of small target features at the bottom of the pyramid in the YOLOv5 architecture. Experimental results demonstrate the superiority of SDGC-YOLOv5 over existing YOLO algorithms on multiple publicly available datasets, including MS-COCO, Pascal-VOC2012, and DOTA, achieving performance improvements of 3%, 4.4%, and 0.6%, respectively. Moreover, an unprecedented mAP accuracy of 91% is achieved on the self-built dataset PhotovoltaicPanels. In summary, SDGC-YOLOv5 effectively addresses the limitations of traditional YOLO-based object detection methods and exhibits remarkable performance in detecting small objects with low resolution and small size.
Adaptive Token Excitation with Negative Selection for Video-Text Retrieval

Video-text retrieval aims to efficiently retrieve videos from large collections based on the given text, whereas methods based on the large-scale pre-trained model have drawn sustained attention recently. However, existing methods neglect detailed information in video and text, thus failing to align cross-modal semantic features well and leading to performance bottlenecks. Meanwhile, the general training strategy often treats semantically similar pairs as negatives, which provides the model with incorrect supervision. To address these issues, an adaptive token excitation (ATE) model with negative selection is proposed to adaptively refine features encoded by a large-scale pre-trained model to obtain more informative features without introducing additional complexity. In detail, ATE is first advanced to adaptively aggregate and align different events described in text and video using multiple non-linear event blocks. Then a negative selection strategy is exploited to mitigate false negative effects, which stabilizes the training process. Extensive experiments on several datasets demonstrate the feasibility and superiority of the proposed ATE compared to other state-of-the-art methods. The source code of this work can be found in https://mic.tongji.edu.cn .
Aggregate Distillation for Top-K Recommender System

As knowledge distillation techniques have gained prominence in recommender systems (RS), they have proven instrumental in maintaining exceptional recommendation performance while simultaneously reducing model size. Nonetheless, conventional single-teacher bidirectional distillation methods present two significant limitations. First, the restricted knowledge from a solitary teacher hinders comprehensive learning for student model. Second, both teacher and student models are susceptible to overfitting due to the traditional distillation loss function. In response to these challenges, we introduce an innovative Aggregated Distillation (AD) framework explicitly designed for RS. We propose an ensemble-based bidirectional knowledge distillation method, utilizing multiple teacher models to furnish a more diverse knowledge base and capitalizing on their synergistic effects to bolster the student model’s performance. Moreover, we devise a novel regularization module that integrates a high-performance teacher model (supervisor) into the primary model. This module mitigates overfitting by directing the principal model towards the gradient direction of the incorporated high-performance teacher model, diminishing the risk of optimization towards an overfitting gradient direction. Rigorous experiments on two real-world datasets substantiate that our proposed framework outperforms its counterparts, with each component contributing significantly to the superior performance.
ECDet: A Real-Time Vehicle Detection Network for CPU-Only Devices

We study the problem of efficient vehicle detection for low-power devices in complex traffic environments and construct a lightweight real-time vehicle object detection model called Efficient Channel Detection (ECDet) to enhance the detection capability of vehicles at multiple scales. This work has three contributions: (1) the design of an efficient network backbone called Efficient Channel Network (ECNet); (2) the design of a lightweight real-time vehicle object detection model called ECDet; and (3) the design of a lightweight feature fusion structure called Lightweight Path Aggregation Network (LPAN). In addition, we quantitatively evaluated different methods on the UA-DETRAC dataset. The experiment shows that ECDet can quickly and accurately detect vehicles, and can be widely applied on embedded platforms such as intelligent transportation systems and unmanned driving systems.
Transforming Limitations into Advantages: Improving Small Object Detection Accuracy with SC-AttentionIoU Loss Function

Small object detection is widely used in industries, military, autonomous driving and other fields. However, the accuracy of existing detection models in small object detection needs to be improved. This paper proposes the SC-AttentionIoU loss function to stress the issue. Due to the less features of small objects, SC-AttentionIoU introduces attention within the true bounding box, allowing the existing detection models to focus on the critical features of small objects. Besides, considering attention perhaps ignore non-critical features, SC-AttentionIoU proposes an adjustment factor to balance the critical and non-critical feature areas. Using the YOLOv5s model as a baseline, compared with the widely used CIoU, SC-AttentionIoU achieved an average improvement of 1% in mAP@.5 on the SSDD dataset and an average improvement of 1.47% in mAP@.5 on the PCB dataset in this experiment.
MLF-DET: Multi-Level Fusion for Cross-Modal 3D Object Detection

In this paper, we propose a novel and effective Multi-Level Fusion network, named as MLF-DET, for high-performance cross-modal 3D object DETection, which integrates both the feature-level fusion and decision-level fusion to fully utilize the information in the image. For the feature-level fusion, we present the Multi-scale Voxel Image fusion (MVI) module, which densely aligns multi-scale voxel features with image features. For the decision-level fusion, we propose the lightweight Feature-cued Confidence Rectification (FCR) module which further exploits image semantics to rectify the confidence of detection candidates. Besides, we design an effective data augmentation strategy termed Occlusion-aware GT Sampling (OGS) to reserve more sampled objects in the training scenes, so as to reduce overfitting. Extensive experiments on the KITTI dataset demonstrate the effectiveness of our method. Notably, on the extremely competitive KITTI car 3D object detection benchmark, our method reaches 82.89% moderate AP and achieves state-of-the-art performance without bells and whistles.
Visual-Haptic-Kinesthetic Object Recognition with Multimodal Transformer

Humans recognize objects by combining multi-sensory information in a coordinated fashion. However, visual-based and haptic-based object recognition remain two separate research directions in robotics. Visual images and haptic time series have different properties, which can be difficult for robots to fuse for object recognition as humans do. In this work, we propose an architecture to fuse visual, haptic and kinesthetic data for object recognition, based on the multimodal Convolutional Recurrent Neural Networks with Transformer. We use Convolutional Neural Networks (CNNs) to learn spatial representation, Recurrent Neural Networks (RNNs) to model temporal relationships, and Transformer’s self-attention and cross-attention structures to focus on global and cross-modal information. We propose two fusion methods and conduct experiments on the multimodal AU dataset. The results show that our model offers higher accuracy than the latest multimodal object recognition methods. We conduct an ablation study on the individual components of the inputs to demonstrate the importance of multimodal information in object recognition. The codes will be available at https://github.com/SYLan2019/VHKOR .
LGF$$^2$$: Local and Global Feature Fusion for Text-Guided Object Detection

A baby can successfully learn to identify objects in an image with the corresponding text description provided by their parents. This learning process leverages the multimodal information of both the image and text. However, classical object detection approaches only utilize the image modality to distinguish objects, neglecting the text modality. While many Vision-Language Models have been explored in the object detection task, they often require large amounts of pre-training data and can only specify a particular model structure. In this paper, we propose a lightweight and generic Local and Global Feature Fusion (LGF $$^2$$ ) framework for text-guided object detection to enhance the performance of image modality detection models. Our adaptive text-image fusion module is designed to learn optimal fusion rules between image and text features. Additionally, we introduce a word-level contrastive loss to guide the local-focused fusion of multimodal features and a sentence-level alignment loss to drive the global consistent fusion of multimodal features. Our paradigm is highly adaptable and can be easily embedded into existing image-based object detection models without any extra modification. We conduct extensive experiments on two multimodal detection datasets, and the results demonstrate that our LGF $$^2$$ significantly improves their performance.
Dual Attention Feature Fusion for Visible-Infrared Object Detection

Feature fusion is an essential component of multimodal object detection to exploit the complementary information and common information between multi-source images. When it comes to visible-infrared image pairs, however, the visible images are prone to illumination and visibility and there may be a lot of interference information and little useful information. We suggest performing common feature enhancement and spatial cross attention sequentially to solve this problem. For this purpose, a novel Dual Attention Transformer Feature Fusion (DATFF) module which is designed for feature fusion of intermediate feature maps is proposed. We integrate it into two-stream object detectors and achieve state-of-the-art performance on DroneVehicle and FLIR visible-infrared object detection datasets. Our code is available at https://github.com/a21401624/DATFF .
Scenic Spot Recommendation Method Integrating Knowledge Graph and Distance Cost

Aiming at the problem that the traditional collaborative filtering algorithm only considers external ratings and cold start when recommending attractions, this paper proposes an attraction recommendation algorithm integrated with knowledge graph. Firstly, we construct a user rating matrix based on user ratings and the number of reviews, and calculate the similarity of attractions. Then, we use TransR model to train the semantic vector matrix of attractions, and use cosine similarity formula to calculate the semantic similarity of attractions. Finally, the two similarities are fused and applied to ALS matrix factorization. At the same time, in order to make the model pay attention to user preferences and take into account the distance elements in the scenic spots for recommendation, the distance cost is integrated in the loss function. Experimental results on the scenic spots dataset show that the proposed algorithm is better than the traditional method in scenic spots recommendation.
Automatic Driving Scenarios: A Cross-Domain Approach for Object Detection

As autonomous driving technology advances, the need for accurate and robust object detection in various driving environments has become more urgent. However, domain adaptation presents a significant challenge due to the impact of weather, lighting, and scene context on object detection models. To address this issue, we propose a new method that utilizes pseudo-labels. Our approach involves two modules: the Category-Adversarial-Adaptive (CAA) and the Regression-Adversarial-Adaptive (RAA), which generate pseudo-labels. The detector is then trained on both source domain data and the target domain with pseudo-labels, resulting in improved cross-domain performance. The CAA and RAA modules operate independently and complement each other, allowing them to adapt to their respective detection tasks without interference. Furthermore, we demonstrate the effectiveness of loss smoothing in enhancing the model’s generalization performance. Our experimental results indicate that our model outperforms classic models, achieving improvements of 0.9 $$\%$$ , 2 $$\%$$ , and 1.4 $$\%$$ in the cross-domain challenges of SIM10K to Cityscape, KITTI to Cityscape, and Cityscape to foggyCityscape, respectively.
X-shape Feature Expansion Network for Salient Object Detection in Optical Remote Sensing Images

Salient object detection in optical remote sensing images (RSI-SOD) is a valuable and challenging task. Some factors in RSI, such as the extreme complexity of scale and topological structure as well as the uncertainty of location of the salient object, significantly reduce the accuracy and completeness of salient object prediction. To address these issues, we propose a novel X-shape Feature Expansion Network (XFNet). Specifically, XFNet consists of a traditional encoder-decoder network, complemented by a new component called the X-shape Feature Expansion Module (XFEM). In XFEM, from the perspective of receptive field and multi-scale information, we utilize two branches to enhance the model’s receptive field and multi-scale information. Moreover, we design a core component in XFEM to facilitate the fusion of feature of each branch. Extensive experiments conducted on two commonly used datasets demonstrate that our approach outperforms 11 state-of-the-art methods, including NSI-SOD and RSI-SOD methods.
Candidate-Aware Dynamic Representation for News Recommendation

With the application of collaborative filtering and deep neural network in news recommendation, it becomes more feasible and easier to capture users’ preferences of news browsing. However, different from traditional recommendation tasks, the collaborative filtering process of news recommendation requires additional consideration of news content. In addition, to capture users’ real intentions from multi granularity interactive behavior information is still challenging. To address above challenges, we propose a hierarchical structure, namely Candidate-Aware Self-Attention enhanced convolution network (CASA). Specifically, we first devise a hierarchical self-attention networks to simultaneously extract the collaborative filtering signals between users and candidate news and their global correlations in multiple dimensions. Secondly, we employ a convolutional networks to map the keywords and important statements of the relevant news into the high dimensional feature space. By considering the additional content-level information, we can further reinforce the the collaborative filtering signals among news. Moreover, we also incorporate time and location relations during the news representation learning to better capture the user’s contextual information. Extensive experiments on two real-world datasets demonstrates that CASA outperforms all the other state-of-the-art baselines on news recomendation task.
Self-supervised Video Object Segmentation Using Motion Feature Compensation

Video object segmentation is a popular area of research in computer vision. Traditional models are trained using annotated data, which is both time-consuming and expensive. Training models in unsupervised manner has been proposed as a solution to this issue. However, previous works have focused only on spatial features extracted by self-supervised learning method, without considering the temporal information between frames. In this paper, we propose a new video object segmentation model that utilizes self-supervised learning to extract spatial features, and incorporates a motion feature, extracted from optical flow, as compensation of temporal information for the model, namely motion feature compensation (MFC) model. Additionally, we introduce an attention-based fusion method to merge features from both modalities. Notably, for each video used to train models, we only select two consecutive frames at random to train our model. The dataset Youtube-VOS and DAVIS-2017 are adopted as the training dataset and the validation dataset. The experimental results demonstrate that our approach outperforms previous methods, validating our proposed design. The source code is available at: https://github.com/CVisionProcessing/MFC .
Adaptive Dehazing YOLO for Object Detection

While CNN-based object detection methods operate smoothly in normal images, they produce poor detection results under adverse weather conditions due to image degradation. To address this issue, we propose a novel Adaptive Dehazing YOLO (DH-YOLO) framework to reduce the impact of weather information on the detection tasks. DH-YOLO is a multi-task learning paradigm that jointly optimizes object detection and image restoration tasks in an end-to-end fashion. In the image restoration module, the feature extraction network serves as an encoder, and a Feature Filtering Module (FFM) is used to remove redundant features. The FFM contains an Adaptive Dehazing Module for image recovery, whose parameters are quickly calculated using a lightweight Cascaded Partial Decoder. This allows the framework to make use of weather-invariant information in hazy images to extract haze-free features. By sharing three feature layers at different scales between the two subtasks, the performance of the object detection network is improved by the use of clear features. DH-YOLO is based on YOLOv4 and forms a unified, end-to-end model with the above modules. Experimental results show that our method outperforms many advanced detection methods on real-world foggy datasets, demonstrating its effectiveness in object detection under adverse weather conditions.
Learning Video Localization on Segment-Level Video Copy Detection with Transformer

At present, research on segment-level video copy detection algorithms mainly focuses on end-to-end optimization from key frame selection and feature extraction to similarity pattern detection, causing the deployment of such algorithms to be difficult and expensive, and ignoring specific research on optimizing detectors for similarity pattern detection. To address the above issues, we propose the segment-level Video Copy Detection Transformer (VCDT), a transformer-based detector designed for similarity pattern detection. Its main novelty can be summarized by two points: (1) An anchor training strategy that allows the model to use the positional prior information in the anchor boxes to make predictions more precisely, (2) A query adaptation module to fine-tune the anchor boxes dynamically. Our experiments show that, without bells and whistles, VCDT achieves state-of-the-art performance while showing an impressive convergence speed.
Feature Sniffer: A Stealthy Inference Attacks Framework on Split Learning

Split learning, a novel privacy-preserving distributed machine learning framework, is proposed to overcome the resource limitations issue of devices in federated learning. Previous studies have explored the possibility of inference attacks on split learning. However, existing methods suffer from unrealistic threat models and poor robustness against defensive techniques. Remarkably, we propose a novel and general framework to perform inference attacks stealthily and reveal the privacy vulnerability of split learning at the convergence stage. In our framework, the malicious server distills knowledge on an auxiliary dataset and transfers the identity information of clients’ data to the auxiliary feature space to sniff out the private data. The attack is behind the scenes and hard to detect. Empirically, we consider image classification as the desired task in split learning and evaluate the effectiveness of our method on common image classification datasets. Extensive experiments still obtain SOTA results in the face of strict differential privacy. The code is available at https://github.com/Rostar-github/FSA .
Gradient Adjusted and Weight Rectified Mean Teacher for Source-Free Object Detection

Source-free object detection (SFOD) aims at adapting object detectors to the unlabeled target domain without access to the labeled source domain. Recent SFOD methods are developed based on the Mean Teacher framework, which consists of a student and a teacher model for self-training. Despite the great success, existing methods suffer from the challenges of missing detections and fitting to incorrect pseudo labels. To overcome these challenges, we propose a Gradient Adjusted and Weight Rectified Mean Teacher framework with two novel training strategies for SFOD, i.e., Negative Gradient Adjustment (NGA) and Source Weight Rectification (SWR). The proposed Negative Gradient Adjustment suppresses the negative gradients caused by missing detections, while the Source Weight Rectification enhances the robustness by rectifying errors of pseudo labels. Additionally, weak-strong consistency data augmentation is introduced for stronger detector performance. Extensive experiments on four benchmarks demonstrate that our proposed method outperforms the existing works for SFOD.
IMAM: Incorporating Multiple Attention Mechanisms for 3D Object Detection from Point Cloud

Nowadays, 3D object detection technology from point clouds develops rapidly. However, lots of small objects emerge in real point cloud scenes, which are hard to be detected due to few points, hindering overall detection accuracy. To address this issue, we propose a novel two-stage 3D object detection method, which introduces an attention strategy to enhance key structure information of objects, so as to promote overall detection accuracy, especially for small objects. Specifically, in the first stage, we employ the convolutional block attention module on the 3D sparse convolution layer to extract voxel features and further apply the Swin Transformer to enhance Bird’s Eye View (BEV) feature for generating high-quality proposals. Then, in the second stage, we apply a Voxel Set Abstraction (VSA) module to fuse voxel features and BEV features into keypoint features, followed by a Region of Interest (RoI) pooling module to obtain grid features for confidence prediction and box regression. Experiment results on the KITTI dataset prove that our method IMAM achieves excellent detection performance, especially for pedestrians and cyclists with small sizes.