Reviewerly/exHarmony huggingface dataset:
Federated Domain Adaptation via Pseudo-label Refinement

Unsupervised domain adaptation (UDA) methods usually assume data from multiple domains can be put together for centralized adaptation. Unfortunately, this assumption impairs data privacy, which leads to the failure of traditional methods in practical scenarios. To cope with the above issue, we present a novel decentralized domain adaptation approach which conducts target adaptation in an iterative training process during which only models can be delivered across domains. More specifically, to train a promising target model, we leverage Adversarial Examples (AEs) to filter out error prone predictions of source models towards each target sample based on both robustness and confidence, and then treat the most frequent prediction as the pseudo-label. Besides, to improve central model aggregation, we introduce Knowledge Contribution (KC) to compute reasonable aggregation weights. Extensive experiments conducted on several standard datasets verify the superiority of the proposed method.
Cross-slice Context Consistency for Semi-supervised 3D Left Atrium Segmentation

Semi-supervised learning is a promising approach in reducing the requirement to collect large amounts of dense annotations, especially in medical image segmentation. However, most existing semi-supervised 3D medical image segmentation methods tend to ignore the cross-slice context that contains extensive structural information. We believe cross-slice context can help the model capture semantic information complementary to slice context and achieve robust and more accurate segmentation. Therefore, in this paper, we propose a novel cross-slice context consistency framework for 3D left atrium segmentation named CSC <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> -Net. Our method can effectively utilize unlabeled data by encouraging consistent results between slice segmentation and cross-slice inference segmentation. To achieve this, we design a bidirectional gated context inference module (Bi-GCM) to model cross-slice context and predict slice segmentation without direct slice features. Experiments on a public left atrium (LA) databases show that our method achieves higher performance and outperforms state-of-the-art methods by imposing cross-slice consistency constraint.
Self-Supervised Implicit 3D Reconstruction via RGB-D Scans

Recently, 3D reconstruction methods based on the neural radiance fields have demonstrated remarkable generative performance. However, these methods frequently tend to be resource hungry and are challenging to regulate large low-textured regions in typical indoor scenes. In this work, we analyze and integrate inherent semantic geometry cues for self-supervised 3D reconstruction training via a unified framework of volume rendering and signed distance implicit representations. In contrast to previous neural implicit methods, we simultaneously incorporate the pixel-aligned features and image patches for multi-view consistency, thereby enabling us to depict a large indoor scene from challenging scenarios with rich visual details and large smooth backgrounds. Extensive experiments and comparisons demonstrate that our proposed method has achieved state-of-the-art results by a large margin in various tasks (e.g. actual surface reconstruction, novel view synthesis, and learning a universal scheme in occlusion or distorted regions).
Local Consensus Transformer for Correspondence Learning

Correspondence learning is a vital component in multi-view geometry and computer vision. The heavy outliers make the matching problem very challenging. By revisiting the local consensus benefits of traditional feature matching, we introduce the local consensus to design a learnerable neural network to capture underlying correspondences, dubbed the Local Consensus Transformer, for wide-baseline stereo. Specifically, our network architecture consists of three operations. In order to construct the neighbor topology, a dynamic graph-based embedding layer is used first. These local topologies then guide the multihead self-attention layer to mine a greater amount of context through channel attention. After that, order-aware graph pooling is applied to extract global context from the embedded local consensus. Experimentally, the abalation study shows that pointnet-like learning models can benefit from local consensus. The proposed model achieves state-of-the-art performance on both the YFCC100M outdoor and SUN3D indoor challenging scenes with more than 90 percent outliers.
Content-adaptive Adversarial Embedding for Image Steganography Using Deep Reinforcement Learning

Recently, adversarial perturbations have been used to reassign cost which can enhance the security of steganography, called as adversarial embedding. However, existing methods selected costs to be modified by self-defined rules which were hard to achieve the optimal security against steganalyzers. In this paper, we propose an automatic adversarial embedding scheme called RLAE (deep Reinforcement Learning-based content-adaptive Adversarial Embedding). In RLAE, an agent network utilizes a generative network which generates an embedding policy for cost reassignment automatically according to a basic steganography cost map. Then, an environment network employs a steganalyzer as an attack target that offers rewards for optimizing the agent network. To provide more comprehensive information, we design a joint reward by considering both the adversarial perturbations calculated from the environment network and noise residual signal representing image textures. Experimental results show that the security of the proposed RLAE is superior than state-of-the-art works, especially steganography with for the large payloads.
Conditional Video-Text Reconstruction Network with Cauchy Mask for Weakly Supervised Temporal Sentence Grounding

Temporal sentence grounding aims to detect the target segment most related to a given query in an untrimmed video. To alleviate the expensive annotation cost for temporal labels, researchers paid more attention to weakly supervised setting. Prior studies neglected the utilization of video representation reconstruction, which led to an unbalanced alignment learning. Moreover, they used different strategies to generate proposals which ignored the temporal structure in a query. In this paper, we propose a novel Conditional Video-Text Reconstruction Network (CVTRN). It supports conditional reconstruction of video and text representation. Specifically, video and text features are fused to compute semantic alignment, which is the condition of reconstruction. A new mask strategy for mask conditioned sentence reconstruction is also devised. This strategy focuses more on boundary regions than the widely used Gaussian mask in previous methods. Experimental results on two public benchmark datasets show that our CVTRN outperforms the state-of-the-art methods.
Movie Box Office Prediction With Self-Supervised and Visually Grounded Pretraining

Investments in movie production are associated with a high level of risk as movie revenues have long-tailed and bimodal distributions [1]. Accurate prediction of box-office revenue may mitigate the uncertainty and encourage investment. However, learning effective representations for actors, directors, and user-generated content-related keywords remains a challenging open problem. In this work, we investigate the effects of self-supervised pretraining and propose visual grounding of content keywords in objects from movie posters as a pretraining objective. Experiments on a large dataset of 35,794 movies demonstrate significant benefits of self-supervised training and visual grounding. In particular, visual grounding pretraining substantially improves learning on movies with content keywords and achieves 14.5% relative performance gains compared to a finetuned BERT model with identical architecture.
Boundary-aware Shadow Detection via Mask Decoupling and Feature Correction

It is well-recognized that correctly detecting shadow boundaries is much more difficult than shadow interior pixels in real-world situations due to the rapid intensity and color transitions near shadow boundaries and the complex background textures. This paper presents a novel boundary-aware shadow detection network to enhance the overall detection performance, especially on challenging shadow boundaries. To enable the network aware of shadow boundaries and balance the supervision between image pixels on the boundary and inside of shadows, the shadow mask is decoupled into a body mask and boundary mask. The feature correction module is designed after feature interaction to eliminate various shadow inferences further. Experimental results show that the proposed network outperforms the state-of-the-art shadow detection methods on four shadow detection benchmark datasets.
Modeling Inter-Vehicle Occlusion Scenarios in Multi-Camera Traffic Surveillance Systems

In this paper, we present a novel design for a multi-camera tracking system with occlusion-handling capabilities and its application to a highway traffic surveillance system. The fundamental concept follows the tracking-by-detection principle with monocular detectors and an LMB tracker for tracking the objects in the world frame. All data from the multi-view setup is combined into one consistent representation of the real-time traffic situation. In order to assess the inter-target occlusion scenarios in 3D, the vehicles are modeled as cuboids and their extents are estimated from the bounding boxes provided by the detectors. We re-transform the 3D occlusion estimation problem into the 2D camera space and present two methods for quantifying the occlusion state of the objects. Moreover, we propose a modification to the computation of the existence probability of undetected and occluded targets. Based on this, the tracking system is extended by an occlusion-aware detection model. We evaluate our occlusion-handling approach on a real-world traffic dataset from the Providentia++ project and show an improved tracking performance. We find that the number of misdetected targets is reduced and more track identities are preserved.
Boosting Interactive Image Segmentation by Exploiting Semantic Clues

This paper presents a refinement framework for enhancing the accuracy of interactive image segmentation by exploiting all available semantic clues. Interactive image segmentation iteratively improves segmentation masks using an input image and user annotations. The information available in this process ranges from low-level visual features like colors and textures to high-level semantic information, such as user annotations and segmentation results. Despite tremendous efforts to segment the overall object shapes, existing methods underutilize the available semantic clues, causing unsatisfactory boundary quality for segmentation masks. The proposed framework first extracts confidence guidance maps, then suppresses and lifts the predicted probabilities for confident pixels, and finally utilizes color similarities as bases and prediction confidence as guidance to refine the segmentation boundaries. Experimental results demonstrate that the framework has a low computational cost and significantly boosts existing methods on standard benchmarks.
Model-based Deep Learning for Maneuvering Target Tracking

Maneuvering target tracking, where the system undergoes abrupt changes in the underlying motion model, can be challenging. We propose a model-based deep learning approach for prediction of maneuvering targets to exploit partial knowledge of the system physics-based models during training, without requiring an explicit characterization or fine tuning of model parameters. We formulate a supervised training scheme to learn the dynamics of state-space models and capture the jump processes governing model transitions by minimizing the prediction loss of an encoder-decoder network from model-based generated data. The effectiveness of the proposed method is demonstrated in two maneuvering target tracking scenarios using synthetic and real-world test data. The results show that the model-based encoder-decoder network achieves notably improved performance in terms of target prediction compared to conventional multiple-model solutions, especially when facing model inaccuracies, jumps, and dominant nonlinearities during target maneuvers.
GOSPA-Driven Gaussian Bernoulli Sensor Management

This paper presents a multi-target metric driven approach to sensor management for Bernoulli filtering, in which at most one target of interest is present. The metric used is the generalised optimal sub pattern assignment (GOSPA) metric. We consider the problem of having an agile sensor operating in a surveillance area, tracking objects as they appear from a target birth distribution. Only one target of interest can exist at any given time-step and its single-target density is Gaussian. In this scenario, we have a grid of sensors that we can select from, one at a time using myopic planning. We evaluate the proposed sensor management algorithm via simulations.
Towards Discriminative Representations with Contrastive Instances for Real-Time UAV Tracking

Maintaining high efficiency and high precision are two fundamental challenges in UAV tracking due to the constraints of computing resources, battery capacity, and UAV maximum load. Discriminative correlation filters (DCF)-based trackers can yield high efficiency on a single CPU but with inferior precision. Lightweight Deep learning (DL)-based trackers can achieve a good balance between efficiency and precision but performance gains are limited by the compression rate. High compression rate often leads to poor discriminative representations. To this end, this paper aims to enhance the discriminative power of feature representations from a new feature-learning perspective. Specifically, we attempt to learn more disciminative representations with contrastive instances for UAV tracking in a simple yet effective manner, which not only requires no manual annotations but also allows for developing and deploying a lightweight model. We are the first to explore contrastive learning for UAV tracking. Extensive experiments on four UAV benchmarks, including UAV123@10fps, DTB70, UAVDT and VisDrone2018, show that the proposed DRCI tracker significantly outperforms state-of-the-art UAV tracking methods.
A Novel Channel Pruning Approach based on Local Attention and Global Ranking for CNN Model Compression

Channel pruning facilitates the acceleration and deployment of convolutional neural networks on resource-constrained devices. Nevertheless, existing related methods mainly focus on the importance of an individual channel, neglecting the intra-layer relationship and inter-layer influence. In this paper, we propose a novel local attention and global ranking (LAGR) method for channel pruning. Specifically, we first introduce the attention mechanism to explore the local correlation between channels of the intra-layer. On this basis, we evaluate the global ranking of all channels across the network by the normalization operation. Besides, we introduce a noisy training strategy in the pre-training stage to ensure a balanced weight distribution. Extensive experiments conducted on three representative networks, including VGGNet, GoogLeNet, and ResNet, have demonstrated the superior performance of the proposed method in comparison with several state-of-the-art methods.
Mandari: Multi-Modal Temporal Knowledge Graph-aware Sub-graph Embedding for Next-POI Recommendation

Next-POI recommendation aims to explore from user check-in sequence to predict the next possible location to be visited. Existing methods are often difficult to model the implicit association of multi-modal data with user choices. Moreover, traditional methods struggle to fully explore the variation of user preferences at variable time intervals. To tackle these limitations, we propose a Multi-Modal Temporal Knowledge Graph-aware Sub-graph Embedding approach (Mandari). We first construct a novel Multi-Modal Temporal Knowledge Graph. Based on the proposed knowledge graph, we integrate multi-modal information and leverage the graph attention network to calculate sub-graph prediction probability. Next, we implement a temporal knowledge mining method to model the segmentation and periodicity of user check-in and obtain temporal prediction probability. Finally, we fuse temporal prediction probability with the previous sub-graph prediction probability to obtain the final result. Extensive experiments demonstrate that our approach outperforms existing state-of-the-art methods.
Bounding Box Detection in Visual Tracking: Measurement Model Parameter Estimation

Common visual tracking algorithms make use of measurement models whose parameters need to be specified. These are, namely, measurement noise covariance related to spatial error of detections provided by a visual detection algorithm, probability of detection, and expected number of clutter detections. The measurement model parameters are often hand selected, using no data-based knowledge. This paper proposes a technique to estimate the parameters by reliably associating detections to annotations in each video frame. The technique is verified on the publicly available MOT-17 dataset.
Scheduling Condition-based Maintenance: An Explainable Deep Reinforcement Learning Approach via Reward Decomposition

This paper presents an eXplainable Deep Reinforcement Learning (XDRL) based strategy for solving the proposed problem of fleet-level aircraft maintenance scheduling (AMS) optimization. The XDRL-AMS considers various factors such as the aircraft’s initial status, mission requirements, maintenance resource capacity, and operational constraints to create a maintenance schedule for a specified period. The schedule aims to balance both mission readiness and cost reduction. We developed an RL environment, called AMS-Gym, using the OpenAI Gym toolkit specifically designed for this problem. AMS-Gym is highly flexible, allowing for easy extension to more complex scenarios and incorporating additional explanatory capabilities. The explainable RL capability was achieved by utilizing a decomposed reward Deep Q-Network (drDQN) algorithm. In the context of the AMS scenario, the drDQN consists of two parts: (i) a DQN that aims to maximize the mission accomplishment objective, and (ii) a DQN that aims to minimize the maintenance cost objective. As a result, the proposed drDQN strategy can generate real-time aircraft maintenance decisions, explain why those decisions were selected, and present the tradeoffs between the chosen action and non-selected alternatives. Experiment results show that the proposed drDQN performs well, providing an approximate solution to the vanilla DQN with a simpler structure while offering the ability to explain its decisions. In addition, a web-based prototype with an intuitive textual and visual user interface was developed to demonstrate the feasibility of the drDQN approach.
RASNet: A Reinforcement Assistant Network for Frame Selection in Video-based Posture Recognition

Most existing video-based posture recognition methods treat frames equally using unified or random sampling strategies, thus losing the temporal relationship information among frames. To address this problem, we propose a lightweight framework, namely RASNet, to adaptively select informative frames for recognition. Specifically, we design a video-suited exploration environment to guide the agent in learning the selection strategy. We introduce the reparametrization method to convert the discrete action space into a continuous space, making the agent robust and random. For the reward part, we design a multi-factor function to reward the agent keeping a balance between frame usage and accuracy. Extensive experiments on three large-scale datasets prove the effectiveness of RASNet, e.g., achieving 85.9% accuracy with fewer 1.15 frames than other state-of-the-art methods on Kinetics 600.
Association of labelled tracks with low reliability covariance information: a track graph partitioning approach

In the context of single air picture fusion based on tracks produced by several heterogeneous C2 system, this paper presents our original algorithmic approach and results of the decentralized fusion process. In this applicative context, the decentralized fusion node receives potentially very numerous (thousands) tracks from all registered C2. Some registered C2 are inconstant in terms of performance and accuracy, resulting in heterogeneous data and possibly in ill associations or estimations; therefore tracks and covariances are not always reliable especially in regards to consistency and velocity. A fast, parallelizable and robust track association algorithm is presented, which cope with these C2 tracks information. Track association is made by partitioning a track graph. A tactical situation maintenance strategy inspired by the state of the art in MHT (multiple hypothesis tracking) implementation is proposed.
Semi-Supervised Semantic Segmentation With Region Relevance

Semi-supervised semantic segmentation aims to learn from a small amount of labeled data and plenty of unlabeled ones for the segmentation task. The most common approach is to generate pseudo-labels for unlabeled images to augment the training data. However, the noisy pseudo-labels will lead to cumulative classification errors and aggravate the local inconsistency in prediction. This paper proposes a Region Relevance Network (RRN) to alleviate the problem mentioned above. Specifically, we first introduce a local pseudo-label filtering module that leverages discriminator networks to assess the accuracy of the pseudo-label at the region level. A local selection loss is proposed to mitigate the negative impact of wrong pseudo-labels in consistency regularization training. In addition, we propose a dynamic region-loss correction module, which takes the merit of network diversity to further rate the reliability of pseudo-labels and correct the convergence direction of the segmentation network with a dynamic region loss. Extensive experiments are conducted on PASCAL VOC 2012 and Cityscapes datasets with varying amounts of labeled data, demonstrating that our proposed approach achieves state-of-the-art performance compared to current counterparts. Our code is available at: https://github.com/NUST-Machine-Intelligence-Laboratory/TorchSemiSeg2.
Advances in Multi-Target Tracking Performance Evaluation

This paper introduces several advances in methods to evaluate performance of multi-target trackers. First, we modify the classical target and track purity definitions to be more reflective of fragmentation and swap phenomena. Second, we extend performance evaluation to more general track outputs with arbitrary state probability distributions. Finally, we introduce a track-level generalization of the GOSPA metric that offers a scalar performance measure of overall tracking performance.
Weakly Supervised Few-Shot and Zero-Shot Semantic Segmentation with Mean Instance Aware Prompt Learning

The existing Zero-Shot Segmentation (ZSS) and Few-Shot Segmentation (FSS) methods utilize fully supervised pixel-labeled seen classes to segment unseen classes. Pixel-level labels are hard to obtain, and using weak supervision in the form of inexpensive image labels is often more practical. To this end, we propose a novel unified weakly supervised Zero-Shot and Few-Shot semantic segmentation pipeline that can perform ZSS and FSS on novel classes without using pixel-level labels for either the base (seen) or the novel (unseen) classes. We propose Mean Instance Aware Prompt based Network (MIAPNet), a novel language-guided segmentation pipeline that i) learns context vectors with batch aggregates (mean) to map class prompts to image features and ii) decouples weak ZSS/FSS into weak semantic segmentation and Zero-Shot segmentation. MIAPNet beats existing methods for weak generalized ZSS and weak FSS by 39 and 3 mIOU points respectively on PASCAL VOC and weak FSS by 5 mIOU points on MS COCO.