Reviewerly/exHarmony huggingface dataset:
Nostrils and Mouth Detection for Drivers Using Convolutional Neural Networks with Automatically Generated Ground Truth Data

Driver monitoring systems are becoming more important in the automotive industry and will be mandatory in the future years to improve driving safety. Nostrils and mouth are the second most important features to be detected on driver faces after the eyes. Training neural networks for detecting facial features requires a manual labelling process to obtain the ground truth data for training and testing. This paper aims to present a method for automatically generation of such ground truth datasets and the results obtained with convolutional neural networks. This process removes the manual labelling effort and provides a simplified way of training neural networks using datasets with real life scenarios.
A Real-Time Blind Quality-of-Experience Assessment Metric for HTTP Adaptive Streaming

In today’s Internet, HTTP Adaptive Streaming (HAS) is the mainstream standard for video streaming, which switches the bitrate of the video content based on an Adaptive BitRate (ABR) algorithm. An effective Quality of Experience (QoE) assessment metric can provide crucial feedback to an ABR algorithm. However, predicting such real-time QoE on the client side is challenging. The QoE prediction requires high consistency with the Human Visual System (HVS), low latency, and blind assessment, which are difficult to realize together. To address this challenge, we analyzed various characteristics of HAS systems and propose a non-uniform sampling metric to reduce time complexity. Furthermore, we design an effective QoE metric that integrates resolution and rebuffering time as the Quality of Service (QoS), as well as spatiotemporal output from a deep neural network and specific switching events as content information. These reward and penalty features are regressed into quality scores with a Support Vector Regression (SVR) model. Experimental results show that the accuracy of our metric outperforms the mainstream blind QoE metrics by 0.3, and its computing time is only 60% of the video playback, indicating that the proposed metric is capable of providing real-time guidance to ABR algorithms and improving the overall performance of HAS. The QoE model is released on https://github.com/lcysyzxdxc/ASPECT.
Robust and Efficient Memory Network for Video Object Segmentation

This paper proposes a Robust and Efficient Memory Network, referred to as REMN, for studying semi-supervised video object segmentation (VOS). Memory-based methods have recently achieved outstanding VOS performance by performing non-local pixel-wise matching between the query and memory. However, these methods have two limitations. 1) Non-local matching could cause distractor objects in the background to be incorrectly segmented. 2) Memory features with high temporal redundancy consume significant computing resources. For limitation 1, we introduce a local attention mechanism that tackles the background distraction by enhancing the features of foreground objects with the previous mask. For limitation 2, we first adaptively decide whether to update the memory features depending on the variation of foreground objects to reduce temporal redundancy. Second, we employ a dynamic memory bank, which uses a lightweight and differentiable soft modulation gate to decide how many memory features need to be removed in the temporal dimension. Experiments demonstrate that our REMN achieves state-of-the-art results on DAVIS 2017, with a $\mathcal{J}\& \mathcal{F}$ score of 86.3% and on YouTube-VOS 2018, with a $\mathcal{G}$ over mean of 85.5%. Furthermore, our network shows a high inference speed of 25+ FPS and uses relatively few computing resources.
Region-Aware Semantic Consistency for Unsupervised Domain-Adaptive Semantic Segmentation

As acquiring pixel-wise labels for semantic segmentation is labor-intensive, unsupervised domain adaptation (UDA) techniques aim to transfer knowledge from synthetic data to real-scene data. To overcome the distribution misalignment between the source domain and the target domain, Teacher-Student (TS) methods are widely-used and promising. In TS methods, the student resorts to the one-hot pseudo labels generated by the teacher. However, the generated one-hot pseudo labels are dubious and ignore the semantic correlation among classes. Besides, in the same position of the same image, the output distributions between the student and the teacher should be consistent. Such prediction consistency is defined as Region-Aware Semantic Consistency (RASC). Correspondingly, we propose an RASC module to assimilate the output distributions of the teacher and the student. Our RASC module is flexible and easily plugged into TS state-of-the-arts (SOTAs) based on either CNNs or Transformers.
Long-Short Temporal Co-Teaching for Weakly Supervised Video Anomaly Detection

Weakly supervised video anomaly detection (WS-VAD) is a challenging problem that aims to learn VAD models only with video-level annotations. In this work, we propose a Long-Short Temporal Co-teaching (LSTC) method to address the WS-VAD problem. It constructs two tubelet-based spatio-temporal transformer networks to learn from short- and long-term video clips respectively. Each network is trained with respect to a multiple instance learning (MIL)-based ranking loss, together with a cross-entropy loss when clip-level pseudo labels are available. A co-teaching strategy is adopted to train the two networks. That is, clip-level pseudo labels generated from each network are used to supervise the other one at the next training round, and the two networks are learned alternatively and iteratively. Our proposed method is able to better deal with the anomalies with varying durations as well as subtle anomalies. Extensive experiments on three public datasets demonstrate that our method outperforms state-of-the-art WS-VAD methods. Code is available at https://github.com/shengyangsun/LSTC_VAD.
Inpainting of Remote Sensing Sea Surface Temperature image with Multi-scale Physical Constraints

Sea Surface Temperature (SST) is a significant environmental factor indicating marine revolutions, which is popularly applied in the meteorological forecasting and fishing industry. Due to the limited sensing ability and occlusion caused by clouds or ice, it is difficult to obtain complete SST data. Compared to traditional interpolation-based methods which refill missed data only referred to current SST data, inpainting-based methods have been carried out with the advantage of using historical SST images to train Generative adversarial Networks (GAN) by terms of considering SST data reconstruction task as an image inpainting task. However, different from common inpainting tasks constrained by semantics, the SST image is a scientific data visualization image without semantics but physical constraints. To address this problem, this paper proposes a multi-scale inpainting GAN-based neural networks to guarantee the physical constraint and realize reasonable SST image reconstruction. The proposed framework mainly contains two modules including the Average Estimation Module (AEM) to realize a global constraint so as not to generate excessive deviation, and the Multi-scale Anomaly Decouple Module (MSADM) to preserve data specificity of current SST image from well-designed multi-scale and decoupled perspectives. Finally, a post-fusion module concatenates the "average" and "specificity" features together to accomplish our multi-scale physical constraints SST image inpainting task. Sufficient experiments have been carried out to verify the effectiveness and physical consistency compared with prior SOTA methods applied to the public AVHRR Pathfinder SST dataset.
Comparison of HDR quality metrics in Per-Clip Lagrangian multiplier optimisation with AV1

The complexity of modern codecs along with the increased need of delivering high-quality videos at low bitrates has reinforced the idea of a per-clip tailoring of parameters for optimised rate-distortion performance. While the objective quality metrics used for Standard Dynamic Range (SDR) videos have been well studied, the transitioning of consumer displays to support High Dynamic Range (HDR) videos, poses a new challenge to rate-distortion optimisation. In this paper, we review the popular HDR metrics DeltaE100 (DE100), PSNRL100, wPSNR, and HDR-VQM. We measure the impact of employing these metrics in per-clip direct search optimisation of the rate-distortion Lagrange multiplier in AV1. We report, on 35 HDR videos, average Bjontegaard Delta Rate (BD-Rate) gains of 4.675%, 2.226%, and 7.253% in terms of DE100, PSNRL100, and HDR-VQM. We also show that the inclusion of chroma in the quality metrics has a significant impact on optimisation, which can only be partially addressed by the use of chroma offsets.
CA-GAN: Object Placement via Coalescing Attention based Generative Adversarial Network

Learning to posit a foreground object over a background scene is an intriguing yet challenging problem, which frequently emerges in applications such as image editing and scene parsing. To date, most existing studies are fed up with knotty issues, including the deficiency of harnessing the interaction between the object and the scene, the astriction of involving little prior knowledge during training, etc. To break the shackles, we propose a novel end-to-end framework dubbed Coalescing Attention based Generative Adversarial Network (CA-GAN). Specifically, in our synthesizer, a feature polymerizer is designed to distill multi-scale information from both background and foreground. On that basis, a dual-branch coalescing attention module is proposed for a better exploration of the global feature-interaction relationships between object and scene. In addition, we add a supervised trail to learn the prior knowledge from the positive composite image, which further guides the synthesizer to discover a credible placement for the foreground object. With extensive experiments conducted on the OPA dataset, our proposal presents superiority in both rationality and diversity compared with other state-of-the-art methods. Our code is available at https://github.com/ZhengJianwei2/CA-GAN.
Adapt then Generalize: A Simple Two-Stage Framework for Semi-Supervised Domain Generalization

Semi-supervised domain generalization (SSDG) focuses on training a model on a single labeled source domain and several unlabeled source domains simultaneously for the purpose of generalizing to out-of-distribution domain. To prevent model overfitting on the unlabeled source domains and enhance model generalization, we propose an effective two-stage framework that disentangle the SSDG task into a burn-in stage and a mutual-training stage. In the burn-in stage, we train a domain adaptation model to reduce domain gap and produce high-accuracy pseudo labels for the unlabeled data. In the second stage, we devise two peer networks equipped with style randomization modules to take advantage of both the style and class information from the pseudo-labeled data. The peer networks are mutually teaching each other, which allows them to avoid overfitting to noisy data and ultimately improves generalization ability. Extensive experiments show that our method achieves state-of-the-art performance on various benchmarks, demonstrating the effectiveness of our method on SSDG.
Collaborative Auto-encoding for Blind Image Quality Assessment

Blind image quality assessment (BIQA) is a challenging problem with important real-world applications. Recent efforts attempting to exploit powerful representations by deep neural networks (DNN) are hindered by the lack of subjectively annotated data. This paper presents a novel BIQA method which overcomes this fundamental obstacle. Specifically, we design a pair of collaborative autoencoders (COAE) consisting of a content autoencoder (CAE) and a distortion autoencoder (DAE) that work together to extract content and distortion representations, which are shown to be highly descriptive of image quality. While the CAE follows a standard codec procedure, we introduce the CAE-encoded feature as an extra input to the DAE's decoder for reconstructing distorted images, thus effectively forcing DAE's encoder to extract distortion representations. The self-supervised learning framework allows the COAE including two feature extractors to be trained by almost unlimited amount of data, thus leaving limited samples with annotations to finetune a BIQA model. We will show that the proposed BIQA method achieves state-of-the-art performance and has superior generalization capability over other learning based models. All the related codes, trained models, and the supplementary material are available at: https://github.com/Macro-Zhou/NRIQA-VISOR/.
Effects of Selection Bias on Online Adversarial Aware SVM When Facing an Evasion Attack

In evasion attacks, the success of machine learning (ML) depends on its ability to detect an attack in an adversarial setting. It is important that these ML models are trained regularly to keep abreast with evolving evasion attacks. We present an investigation of the effects of selection bias and class imbalance in training an Adversary-Aware Online SVM (AAOSVM). We show that incorrect samples can compromise the model's ability to detect attacks. Comparison results showed better True Positive Rates (TPR) performance than the Online Support Vector Machine (OSVM), and that AAOSVM was sensitive to selection bias but not to class imbalance.
Deep Learning-based Synthesized Image Attribution Using Frequency Distribution Information

As the deepfake technology emerges at a breathtaking pace, it threatens to become a destructive political and social force with unpredictable impact on society. Therefore, detecting deepfakes and even figuring out which deep learning generative models (GMs) created such images is of extreme importance. There are already several methods that find and categorize artifacts left by GMs, with the latest efforts focusing on utilizing the frequency domain to achieve these goals. In this paper, we propose a deep learning-based solution with a learnable coefficient layer that highlights GMs' artifacts to achieve high accuracy on the synthesized image attribution task. Evaluation results have shown that our proposed method not only has comparable performance to state-of-the-art methods, but it also outperforms them on unseen image types, showing great generalizability.
Fedward: Flexible Federated Backdoor Defense Framework with Non-IID Data

Federated learning (FL) enables multiple clients to collaboratively train deep learning models while considering sensitive local datasets’ privacy. However, adversaries can manipulate datasets and upload models by injecting triggers for federated backdoor attacks (FBA). Existing defense strategies against FBA consider specific and limited attacker models, and a sufficient amount of noise to be injected only mitigates rather than eliminates FBA. To address these deficiencies, we introduce a Flexible Federated Backdoor Defense Framework (Fedward) to ensure the elimination of adversarial backdoors. We decompose FBA into various attacks, and design amplified magnitude sparsification (AmGrad) and adaptive OPTICS clustering (AutoOPTICS) to address each attack. Meanwhile, Fedward uses the adaptive clipping method by regarding the number of samples in the benign group as constraints on the boundary. This ensures that Fedward can maintain the performance for the Non-IID scenario. We conduct experimental evaluations over three benchmark datasets and thoroughly compare them to state-of-the-art studies. The results demonstrate the promising defense performance from Fedward, moderately improved by 33% ∼ 75% in clustering defense methods, and 96.98%, 90.74%, and 89.8% for Non-IID to the utmost extent for the average FBA success rate over MNIST, FMNIST, and CIFAR10, respectively.
Learning Lossless Compression for High Bit-Depth Medical Imaging

We propose a learned lossless image compression method for high bit-depth medical imaging (up to 16 bit-depths). Instead of compressing a high bit-depth medical image as a whole, we split it into two low bit-depth subimages, i.e., the most significant bytes (MSB) subimage and the least significant bytes (LSB) subimage, respectively. The MSB subimage depicts piece-wise smooth structure information that is relatively easy to compress. We thus use traditional lossless codecs for low complexity. The LSB subimage depicts the complementary texture information that is more challenging to compress. We design an autoregressive entropy model conditioned on the MSB subimage that models the probability distribution of the LSB subimage and effectively reduces the redundancy between the MSB and LSB subimages. We then encode the LSB subimage to bitstreams based on the learned entropy model. The compressed high bit-depth medical image is finally stored including the bitstreams of the MSB and LSB subimages. Experimental results demonstrate the state-of-the-art compression performance of the proposed method on high bit-depth medical images, compared with both existing traditional and learned lossless image codecs.
Fixed-point iterative computation of Gaussian barycenters for some dissimilarity measures

In practical contexts like sensor fusion or computer vision, it is not unusual to deal with a large number of Gaussian densities that encode the available information. In such cases, if the computational capabilities are limited, a data compression is required, often done by finding the barycenter of the set of Gaussians. However, such computation strongly depends on the chosen loss function (dissimilarity measure) to be minimized, and most often it must be performed by means of numerical methods, since the barycenter can rarely be computed analytically. Some constraints, like the covariance matrix symmetry and positive definiteness can make nontrivial the numerical computation of the Gaussian barycenter. In this work, a set of Fixed-Point Iteration algorithms are presented in order to allow for the agile computation of Gaussian barycenters according to several dissimilarity measures.
HOD: Human-Object Decoupling Network for HOI Detection

Single-stage Human-Object Interaction (HOI) detection methods have attracted considerable attention due to their high efficiency. Existing methods tend to concentrate the detection of humans and objects in one decoder without considering the differences between them, which causes tremendous pressure on a single decoder and affects the detection effect. This paper aims to decouple the detection decoder of humans and objects. In particular, we advocate and propose a novel human-object decoupling network (HOD) that divides the decoder into three tasks: human detection, object detection, and action classification. The network uses the random erasure training strategy to improve the model’s generalization ability and introduces pose features to handle the long-tailed problem. In addition, we design a pose fusion branch to alleviate the semantic gap between pose and HOI datasets. The experimental results suggest that our method achieves consistent improvements over the state-of-theart across different datasets, utilizing only image information. Specifically for HICO-Det, our method outperforms existing methods by a large margin, with a significant relative mAP gain of 7.8%. Our source code will be publicly available upon acceptance.
Anchor-Free Action Proposal Network with Uncertainty Estimation

Proposal generation is a fundamental yet challenging task for two-stage temporal action detection pipelines. The task aims at predicting starting and ending boundaries of segments in realistic video sequences and action recognition methods cannot be directly applied to such videos due to their untrimmed nature. Most state-of-the-art models rely on temporal convolutional neural networks with pre-defined anchor segments. By eliminating anchors, we propose a lighter end-to-end trainable Anchor-Free Multiscale Transformer-based Generator (AMTG) model using local clues via video snippets. To improve effectiveness for temporal evaluation, we apply multiscale Transformer encoders to sequences with a bi-directional mask extension that simultaneously predicts boundary distances with uncertainties and various snippet-based local scores. Later, our model integrates local predictions to generate proposal candidates using the proposed scoring function. Experiments on the THUMOS14 and ActivityNet-1.3 benchmarks demonstrate the effectiveness of AMTG for the temporal proposal generation task.
Mixed Poisson process with Stacy mixing variable

AbstractStacy distribution defined for the first time in 1961 provides a flexible framework for modeling of a wide range of real-life behaviors. It appears under different names in the scientific literature and contains many useful particular cases. Homogeneous Poisson processes are appropriate apriori models for the number of renewals up to a given time t > 0. This article mixes them and considers a mixed Poisson process with Stacy mixing variable. We call it a Poisson-Stacy process. The resulting counting process is one of the Generalized Negative Binomial processes, and the distribution of its time-intersections are very-well investigated in the scientific literature. Here we define and investigate their joint probability distributions. Then, the corresponding mixed renewal process is investigated and Exp-Stacy and Erlang-Stacy distributions are defined and partially studied. The article finishes with a simulation study of these stochastic processes. Some plots of the probability density functions, probability mass functions, mean square regressions, and sample paths are drawn together with the corresponding code for the simulations.KEYWORDS: Mixed Poisson processesStacy distributionKratzel inegral Disclosure statementThe authors report there are no competing interests to declare.Notes1 If a r.v. ξ has some probability law, we call l-type the set of distributions of all r.vs. aξ+b, where a∈R and b∈R. l comes from “linear”.Additional informationFundingThe work was supported by the Scientific Research Fund in Konstantin Preslavsky University of Shumen, Bulgaria under Grant RD-08-35/18.01.2023; and ANID Chile – Proyecto Exploracion ANID 2022, No. 13220184.
Multipartite entanglement and quantum error identification in <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mi>D</mml:mi></mml:math> -dimensional cluster states

An entangled state is said to be $m$-uniform if the reduced density matrix of any $m$ qubits is maximally mixed. This is intimately linked to pure quantum error correction codes (QECCs), which allow us not only to correct errors but also to identify their precise nature and location. Here, we show how to create $m$-uniform states using local gates or interactions and elucidate several QECC applications. We first show that $D$-dimensional cluster states are $m$-uniform with $m=2D$. This zero-correlation-length cluster state does not have finite-size corrections to its $m=2D$ uniformity, which is exact both for infinite lattices and for large enough, but finite, lattices. Yet at some finite value of the lattice extension in each of the $D$ dimensions, which we bound, the uniformity is degraded due to finite support operators which wind around the system. We also outline how to achieve larger $m$ values using quasi-$D$-dimensional cluster states. This opens the possibility to use cluster states to benchmark errors on quantum computers. We demonstrate this ability on a superconducting quantum computer, focusing on the one-dimensional cluster state, which, as we show, allows us to detect and identify one-qubit errors, distinguishing $X, Y$, and $Z$ errors.