Reviewerly/exHarmony huggingface dataset:
Trajectory Alignment based Multi-Scaled Temporal Attention for Efficient Video Transformer

Although the video transformer gets remarkable accuracy on video recognition tasks, it is hard to be deployed in resource-constrained scenarios due to the high computational cost. A method that dynamically modifies and trains the transformer model, ensuring that the computational cost matches the deployment scenario requirement, would be an effective solution to this challenge. In this paper, we propose a method for modifying large-scale video transformers with trajectory alignment based multi-scaled temporal attention (TAMS) schemes to reduce the computational cost significantly while losing accuracy slightly. In the temporal dimension, we adopt multi-scaled sparsity patterns in hierarchical transformer blocks. In the spatial dimension, we use region selection to force the transformer to focus on high-importance regions while not corrupting the spatial context. Our method reduces up to 40% computational cost of state-of-the-art large-scale video transformers with a slight accuracy drop (~ 7%) on the video recognition task.
Learning continuous piecewise non-linear activation functions for deep neural networks

Activation functions provide the non-linearity to deep neural networks, which are crucial for the optimization and performance improvement. In this paper, we propose a learnable continuous piece-wise nonlinear activation function (or CPN in short), which improves the widely used ReLU from three directions, i.e., finer pieces, non-linear terms and learnable parameterization. CPN is a continuous activation function with multiple pieces and incorporates non-linear terms in every interval. We give a general formulation of CPN and provide different implementations according to three key factors: whether the activation space is divided uniformly or not, whether the non-linear terms exist or not, and whether the activation function is continuous or not. We demonstrate the effectiveness of our method on image classification and single image super-resolution tasks by simply changing the activation function. For example, CPN improves 4.78% / 4.52% top-1 accuracy over ReLU on MobileNetV2_0.25 / MobileNetV2_0.35 for ImageNet classification and achieves better PSNR on several benchmarks for super-resolution. Our implementation is available at https://github.com/xc-G/CPN.
A Design Approach for Minimizing Data Theft

Not a day goes by without some organization in the world being victimized by the theft of private data, resulting in heavy losses to both the organization and the owners of the data. For an organization, these losses include large expenditures to resume normal operation and damages to its reputation. For data owners, the losses may be financial, and may include identity theft. Security controls (e.g., encryption) help to secure vulnerabilities, but they are not foolproof. This paper proposes a design approach that minimizes the theft of private data from an organization by limiting as much as possible the quantity of such data that is stored on-premise or in the cloud, preferring instead to let the data remain on the user's computing device. This paper additionally presents an example of applying the approach to e-commerce.
Semi-Supervised Methods for Detecting Cyber-Physical Attacks in Industrial Control Systems

Detection of cyber-physical attacks typically relies on maintaining a list of known exploits, and examining real time data for evidence of those exploits. However, this is often cost prohibitive, and still remains vulnerable to zero-day attacks. Thus, development of models that can accurately detect attacks without a priori knowledge of what those attacks look like is an important research question. Here we examine a common semi-supervised learning method, and discuss techniques for refining its application to intrusion detection in industrial control systems (ICS) data. Principal component analysis (PCA) has been shown to be effective in this endeavor, although problems can arise in implementation. We discuss a collection of techniques for improving this method. The techniques are applied to the Hardware-in-the-Loop-Based Augmented Industrial Control System (HAI) dataset, and the results compared.
Explainable Unfolding Network For Joint Edge-Preserving Depth Map Super-Resolution

Although color-guided depth map super-resolution methods based on deep learning have achieved great progress, these methods are not explainable and their super-resolution results are suffered from texture-copying and boundary blurring problems. In order to alleviate these problems, we propose an explainable depth map super-resolution network by unfolding edge-constrained optimization model, dubbed Ex-DSRNet. First, we propose an edge reconstruction network to recover accurate edge information, then feed edge information and color information into different depth map reconstruction networks to reconstruct depth information. Finally, a high-fidelity network is proposed to fuse the above two kinds of depth information to obtain high-quality depth map. A large number of experimental results have demonstrated that the proposed Ex-DSRNet can compete against many state-of-the-art depth map super-resolution methods in term of root mean square error.
CS-PCN: Context-Space Progressive Collaborative Network for Image Denoising

Currently, image-denoising methods based on deep learning cannot adequately reconcile contextual semantic information and spatial details. To take these information optimizations into consideration, in this paper, we propose a Context-Space Progressive Collaborative Network (CS-PCN) for image denoising. CS-PCN is a multi-stage hierarchical architecture composed of a context mining siamese sub-network (CM2S) and a space synthesis sub-network (3S). CM2S aims at extracting rich multi-scale contextual information by sequentially connecting multi-layer feature processors (MLFP) for semantic information pre-processing, attention encoder-decoders (AED) for multi-scale information, and multi-conv attention controllers (MCAC) for supervised feature fusion. 3S parallels MLFP and a single-scale cascading block to learn image details, which not only maintains the contextual information but also emphasizes the complementary spatial ones. Experimental results show that CS-PCN achieves significant performance improvement in synthetic and real-world noise removal.
Out-of-distribution Object Detection through Bayesian Uncertainty Estimation

The superior performance of object detectors is often established under the condition that the test samples are in the same distribution as the training data. However, in many practical applications, out-of-distribution (OOD) instances are inevitable and usually lead to uncertainty in the results. In this paper, we propose a novel, intuitive, and scalable probabilistic object detection method for OOD detection. Unlike other uncertainty-modeling methods that either require huge computational costs to infer the weight distributions or rely on model training through synthetic outlier data, our method is able to distinguish between in-distribution (ID) data and OOD data via weight parameter sampling from proposed Gaussian distributions based on pre-trained networks. We demonstrate that our Bayesian object detector can achieve satisfactory OOD identification performance by reducing the FPR95 score by up to 8.19% and increasing the AUROC score by up to 13.94% when trained on BDD100k and VOC datasets as the ID datasets and evaluated on COCO2017 dataset as the OOD dataset.
Know Who You Are: Learning Target-Aware Transformer for Object Tracking

Tracking methods for measuring the similarity between the template and search region have achieved great success in recent years. Although many researchers have made efforts to introduce template annotations into network, inductive bias for trackers is unavoidable due to the inherent disadvantage of box representation. In this work, a novel tracking framework is proposed to eliminate the misguidance of biased prior, based on which, a target-aware Transformer tracker is designed. We use the template annotation as a predicted item in supervised learning, train our model to estimate the same target in template and search frame simultaneously, so that the tracker can learn the target-awareness both in the past and present frame. Our method can be assembled on the vast majority of Transformer-based networks. Sufficient experiments on six datasets verify the correctness of the proposed model. Without the bells and whistles, our tracker achieves the state-of-the-art performance on multiple benchmarks.
Face Poison: Obstructing DeepFakes by Disrupting Face Detection

Recent years have seen fast development in synthesizing realistic human faces using AI-based forgery technique called DeepFake, which can be weaponized to cause negative personal and social impacts. In this work, we develop a defense method, namely FacePosion, to prevent individuals from becoming victims of DeepFake videos by sabotaging would-be training data. This is achieved by disrupting face detection, a prerequisite step to prepare victim faces for training DeepFake model. Once the training faces are wrongly extracted, the DeepFake model can not be well trained. Specifically, we propose a multi-scale feature-level adversarial attack to disrupt the intermediate features of face detectors using different scales. Extensive experiments are conducted on seven various DeepFake models using six face detection methods, empirically showing that disrupting face detectors using our method can effectively obstruct DeepFakes.
Extended target PMBM tracker with a Gaussian Process target model on LiDAR data

In Multiple Extended Object Tracking, the PMBM (Poisson Multi-Bernoulli Mixture) tracker is considered state-of-the-art. Originally, it was presented with the GGIW (Gamma Gaussian Inverse Wishart) target model, which is a random matrix model. When tracking larger objects using LiDAR, measurements are generated by the contour rather than the whole target surface, and it is beneficial to model this with the target model. A target model which has this capability is the Gaussian Process (GP) extent model. This paper presents a PMBM tracker using this target model. We also discuss considerations related to the use of the GP model in the PMBM framework. Secondly, we present improvements in the target model which increases the robustness of the model by dealing with the inherent nonlinearities using the Gauss-Newton method. We also present a comparison with the GGIW-PMBM tracker on simulated and real LiDAR data gathered from maritime vessels.
Large Pose Friendly Face Reenactment using subtle motions

Face reenactment aims to synthesis a photo-realistic video of the source face by imitating the motion and expression of the driving video while keeping the source appearance (i.e. identity). Although good results have achieved recently, most state-of-the-art methods remain vulnerable to extreme conditions, which greatly restricts the application in the real world. Among various extreme conditions, the large pose problem is the most common one. We clarify that the large pose problem is mainly caused by the severe motion change between the source image and the current driving frame. An intuitive solution is to divide the severe motion change into a sequence of subtle motions. Therefore, we propose a new scheme that exploring the temporal coherence between previous neighbor frame and current frame. The smaller motion change between consecutive frames help to solve the large pose problem. Furthermore, a calibration net is designed to eliminate the error accumulation of the previous step. Extensive experiments demonstrate that our method performs better on large pose face reenactment than the state-of-the-art in terms of large pose cases and visual quality.
CPNet: Exploiting CLIP-based Attention Condenser and Probability Map Guidance for High-fidelity Talking Face Generation

Recently, talking face generation has drawn ever-increasing attention from the research community in computer vision due to its arduous challenges and widespread application scenarios, e.g. movie animation and virtual anchor. Although persevering efforts have been undertaken to enhance the fidelity and lip-sync quality of generated talking face videos, there is still large room for further improvements of synthesis quality and efficiency. Actually, these attempts somewhat ignore the explorations of fine-granularity feature extraction/integration and the consistency between probability distributions of landmarks, thereby recurring the issues of local details blurring and degraded fidelity. To mitigate these dilemmas, in this paper, a novel CLIP-based Attention and Probability Map Guided Network (CPNet) is delicately designed for inferring high-fidelity talking face videos. Specifically, considering the demands of fine-grained feature recalibration, a clip-based attention condenser is exploited to transfer knowledge with rich semantic priors from the prevailing CLIP model. Moreover, to guarantee the consistency in probability space and suppress the landmark ambiguity, we creatively propose the density map of facial landmark as auxiliary supervisory signal to guide the landmark distribution learning of generated frame. Extensive experiments on the widely-used benchmark dataset demonstrate the superiority of our CPNet against state of the arts in terms of image and lip-sync quality. In addition, a cohort of studies are also conducted to ablate the impacts of the individual pivotal components.
On the Spherical Laplace Distribution

In directional statistics, the von Mises-Fisher (vMF) distribution has long been a mainstay for inference with data on the unit hypersphere. The performance of statistical inference based on the vMF distribution, however, may suffer when there are significant outliers and noise in the data. Based on an analogy of the median as a robust measure of central tendency and its relationship to the Laplace distribution, we propose the spherical Laplace (SL) distribution, a novel probability measure for modelling directional data. In this paper, we study foundational properties of the distribution such as theoretical results on maximum likelihood estimation and a sampling scheme for probabilistic inference. We derive efficient numerical routines for parameter estimation in the absence of closed-form formula. An application of model-based clustering is considered under the finite mixture model framework. Our numerical methods for parameter estimation and clustering are validated using simulated and real data experiments.
GradSalMix: Gradient Saliency-Based Mix for Image Data Augmentation

The success of CutMix in image classification has sparked interest in saliency-based mix augmentation methods, which refer to detecting saliency regions to generate more valid images. However, existing mix works either require external tools to locate saliency regions, or rely on additional complex optimization policy for generating new images, which limits their application ranges. To address these deficiencies, we propose Gradient Saliency-based Mix (GradSalMix), a simple yet more general mix augmentation, whose operations are all based on the gradients of the training neural network itself. Specifically, we first locate the saliency regions of two images via their gradients of manifolds, and then directly migrate the region, sampled around the center with a large gradient response value, from one image to another. Afterwards, the labels of images are weighted by their accumulated gradient values for new soft labels, which are shown more accurate than the ones weighted by area ratio. The experimental results show that our proposed method outperforms previous works, in terms of accuracy and robustness against adversarial attacks, on four image classification benchmarks. Moreover, extensive experiments on object detection and point cloud classification also verify the superiority and generality of our method.
Track-to-track Association based on Stochastic Optimization

Multi-sensor fusion can improve environment perception, e.g., by increasing the field of view in collective perception, where intelligent vehicles communicate. Track-to-track fusion in a collective perception scenario makes it necessary to associate tracks from multiple sensors. Especially in settings with many sensors that have limited field of view, track-to-track association can be quite challenging. In this work, we develop a stochastic optimization-based approach for an arbitrary number of sensors with a limited field of view, which utilizes a cluster likelihood to sample joint associations. The proposed method clearly outperforms a greedy approach and computes the most likely associations with only a few samples. We evaluate the approaches on simulated data in static and dynamic scenarios.
Towards Terrain-Based Navigation Using Side-Scan Sonar

This paper introduces a statistical model and corresponding sequential Bayesian estimation method for terrain-based navigation using sidescan sonar (SSS) data. The presented approach relies on slant range measurements extracted from the received ping of a SSS. In particular, incorporating slant range measurements to landmarks for navigation constrains the location and altitude error of an autonomous platform in GPS-denied environments. The proposed navigation filter consists of a prediction step based on the unscented transform and an update step that relies on particle filtering. The SSS measurement model aims to capture the highly nonlinear nature of SSS data while maintaining reasonable computational requirements in the particle-based update step. For our numerical results, we assume a scenario with a surface vehicle that performs SSS and compass measurements. The simulated scenario is consistent with our current hardware platform. We also discuss how the proposed method can be extended to autonomous underwater vehicles (AUVs) in a straightforward way and why the combination of SSS sensor and compass is particularly suitable for small autonomous platforms.
L:OMEM - A fast filter to track maneuvering extended objects

In this work a new class of filters, called Lambda:Omicron Multiplicative Error Model (L:OMEM), is introduced with the aim to efficiently address tracking of maneuvering extended objects. In this context, two main challenges have to be tackled: (1) the tracked object moves with unknown time-varying speed and turning rate; (2) the tracked object can produce a large amount of measurements. Closed-form formulas and a novel method to reduce extended object tracking to a conventional point object tracking problem are derived, so that the novel filter results into an accurate and computationally cheap algorithm. Numerical simulations are presented to validate the effectiveness of the proposed approach, where the L:OMEM filter is compared to state-of-the-art filters for extended objects.
Structural Tensor Nonlinear Complex Diffusion-Based Method for Sonar Image Denoising

This paper presents a sonar image denoising method based on structural tensor and nonlinear complex diffusion (STNCD). The complex diffusion in STNCD utilizes the imaginary value generated by the diffusion process as the directional structure indicator, then controls the diffusion coefficient by introducing the eigenvalue of the structural tensor and the spatial variation structure factor related to the eigenvector. As a result, the complex value of STNCD ensures that the phase information of the image is preserved during the noise suppression process. Furthermore, we make use of the difference between the principal eigenvectors of the structural tensor to describe the local coherence of the features, and then solve the problem that the phase angle cannot be accurately estimated when constructing the structural tensor. Experimental results show that STNCD not only suppresses sonar image noise but also enhances the signal of image edge, which has a wide range of applications.
Learning Discrimination from Contaminated Data: Multi-Instance Learning for Unsupervised Anomaly Detection

Anomaly detection aims at identifying deviant samples from the normal data distribution. Much progress has been made in recent years for anomaly detection with self-supervised representation learning. However, most existing approaches assume the training set contains either only clean normal samples or some labeled abnormal samples. With a contaminated unlabeled training set, the performance is degraded with unclear discrimination between normal and abnormal samples. To address the above challenge, in this paper, we propose a novel unsupervised representation learning framework that takes advantage of the extra information provided by the anomalies in the unlabeled contaminated data for anomaly detection. Specifically, anomaly discrimination learning with pseudo normality score generation and multi-instance contrastive learning is proposed for distinguishing abnormal samples from the unlabeled set. Meanwhile, we combine the discrimination learning with mean-shifted contrastive learning and distribution-shifting transformation classification into a multi-task representation learning framework to improve the stability and robustness of the training. Our proposed method achieves state-of-the-art performance on CIFAR-10, ImageNet-10, MNIST and F-MNIST datasets. Extensive ablation studies further demonstrate the effectiveness of different components of the proposed framework.
Online Performance Assessment of Multi-Sensor Kalman Filters Based on Subjective Logic

Operation monitoring for automation systems requires self-assessment of all data processing modules. In this work, we extend our new self-assessment method for linear Kalman filters based on subjective logic to nonlinear Kalman filtering. Furthermore, we propose novel approaches within this subjective logic-based framework to assess the overall filter performance in multi-sensor systems online, i.e., in real-time without ground truth data. The results of the proposed self-assessment method for nonlinear Kalman filtering are demonstrated through simulation studies, showing advantages compared to classical consistency measures, like the normalized innovation squared. In addition, the results of the proposed online overall filtering assessment for multi-sensor systems can even compete with consistency measures based on ground truth data, which cannot be applied in online applications.
Review of Robotics Technologies and Its Applications

Robots are automatic equipment integrating advanced technologies in multiple disciplines such as mechanics, electronics, control, sensors, and artificial intelligence. Based on a brief introduction of the development history of robotics, this paper reviews the classification of the type of robots, the key technologies involved, and the applications in various fields, analyze the development trend of robotics and recent research hotspots, and provides an outlook on the future development of robotics and its applications.
Utility-Maximizing Bidding Strategy for Data Consumers in Auction-Based Federated Learning

Auction-based Federated Learning (AFL) has attracted extensive research interest due to its ability to motivate data owners to join FL through economic means. Existing works assume that only one data consumer and multiple data owners exist in an AFL marketplace (i.e., a monopoly market). Therefore, data owners bid to join the data consumer for FL. However, this assumption is not realistic in practical AFL marketplaces in which multiple data consumers can compete to attract data owners to join their respective FL tasks. In this paper, we bridge this gap by proposing a first-of-its-kind utility-maximizing bidding strategy for data consumers in federated learning (Fed-Bidder). It enables multiple FL data consumers to compete for data owners via AFL effectively and efficiently by providing with utility estimation capabilities which can accommodate diverse forms of winning functions, each reflecting different market dynamics. Extensive experiments based on six commonly adopted benchmark datasets show that FedBidder is significantly more advantageous compared to four state-of-the-art approaches.