Reviewerly/exHarmony huggingface dataset:
Adversarial Attack and Defense of YOLO Detectors in Autonomous Driving Scenarios

Visual detection is a key task in autonomous driving, and it serves as a crucial foundation for self-driving planning and control. Deep neural networks have achieved promising results in various visual tasks, but they are known to be vulnerable to adversarial attacks. A comprehensive understanding of deep visual detectors’ vulnerability is required before people can improve their robustness. However, only a few adversarial attack/defense works have focused on object detection, and most of them employed only classification and/or localization losses, ignoring the objectness aspect. In this paper, we identify a serious objectness-related adversarial vulnerability in YOLO detectors and present an effective attack strategy targeting the objectness aspect of visual detection in autonomous vehicles. Furthermore, to address such vulnerability, we propose a new objectness-aware adversarial training approach for visual detection. Experiments show that the proposed attack targeting the objectness aspect is 45.17% and 43.50% more effective than those generated from classification and/or localization losses on the KITTI and COCO_traffic datasets, respectively. Also, the proposed adversarial defense approach can improve the detectors’ robustness against objectness-oriented attacks by up to 21% and 12% mAP on KITTI and COCO_traffic, respectively.
Learning Distinctive Margin toward Active Domain Adaptation

Despite plenty of efforts focusing on improving the domain adaptation ability (DA) under unsupervised or few-shot semi-supervised settings, recently the solution of active learning started to attract more attention due to its suitability in transferring model in a more practical way with limited annotation resource on target data. Nevertheless, most active learning methods are not inherently designed to handle domain gap between data distribution, on the other hand, some active domain adaptation methods (ADA) usually requires complicated query functions, which is vulnerable to overfitting. In this work, we propose a concise but effective ADA method called Select-by-Distinctive-Margin (SDM), which consists of a maximum margin loss and a margin sampling algorithm for data selection. We provide theoretical analysis to show that SDM works like a Support Vector Machine, storing hard examples around decision boundaries and exploiting them to find informative and transferable data. In addition, we propose two variants of our method, one is designed to adaptively adjust the gradient from margin loss, the other boosts the selectivity of margin sampling by taking the gradient direction into account. We benchmark SDM with standard active learning setting, demonstrating our algorithm achieves competitive results with good data scalability. Code is available at https://github.com/TencentYoutuResearch/ActiveLearning-SDM
Improving the Robustness of Reinforcement Learning Policies With ${\mathcal {L}_{1}}$ Adaptive Control

A reinforcement learning (RL) control policy could fail in a new/perturbed environment that is different from the training environment, due to the presence of dynamic variations. For controlling systems with continuous state and action spaces, we propose an add-on approach to robustifying a pre-trained RL policy by augmenting it with an $\mathcal{L}_{1}$ adaptive controller ($\mathcal{L}_{1}$AC). Leveraging the capability of an $\mathcal{L}_{1}$AC for fast estimation and active compensation of dynamic variations, the proposed approach can improve the robustness of an RL policy which is trained either in a simulator or in the real world without consideration of a broad class of dynamic variations. Numerical and real-world experiments empirically demonstrate the efficacy of the proposed approach in robustifying RL policies trained using both model-free and model-based methods.
PolyphonicFormer: Unified Query Learning for Depth-Aware Video Panoptic Segmentation

The Depth-aware Video Panoptic Segmentation (DVPS) is a new challenging vision problem that aims to predict panoptic segmentation and depth in a video simultaneously. The previous work solves this task by extending the existing panoptic segmentation method with an extra dense depth prediction and instance tracking head. However, the relationship between the depth and panoptic segmentation is not well explored – simply combining existing methods leads to competition and needs carefully weight balancing. In this paper, we present PolyphonicFormer, a vision transformer to unify these sub-tasks under the DVPS task and lead to more robust results. Our principal insight is that the depth can be harmonized with the panoptic segmentation with our proposed new paradigm of predicting instance level depth maps with object queries. Then the relationship between the two tasks via query-based learning is explored. From the experiments, we demonstrate the benefits of our design from both depth estimation and panoptic segmentation aspects. Since each thing query also encodes the instance-wise information, it is natural to perform tracking directly with appearance learning. Our method achieves state-of-the-art results on two DVPS datasets (Semantic KITTI, Cityscapes), and ranks 1st on the ICCV-2021 BMTT Challenge video + depth track. Code is available here .
A brief survey on adaptive video streaming quality assessment

Quality of experience (QoE) assessment for adaptive video streaming plays a significant role in advanced network management systems. It is especially challenging in case of dynamic adaptive streaming schemes over HTTP (DASH) which has increasingly complex characteristics including additional playback issues. In this paper, we provide a brief overview of adaptive video streaming quality assessment. Upon our review of related works, we analyze and compare different variations of objective QoE assessment models with or without using machine learning techniques for adaptive video streaming. Through the performance analysis, we observe that hybrid models perform better than both quality-of-service (QoS) driven QoE approaches and signal fidelity measurement. Moreover, the machine learning-based model slightly outperforms the model without using machine learning for the same setting. In addition, we find that existing video streaming QoE assessment models still have limited performance, which makes it difficult to be applied in practical communication systems. Therefore, based on the success of deep learned feature representations for traditional video quality prediction, we also apply the off-the-shelf deep convolutional neural network (DCNN) to evaluate the perceptual quality of streaming videos, where the spatio-temporal properties of streaming videos are taken into consideration. Experiments demonstrate its superiority, which sheds light on the future development of specifically designed deep learning frameworks for adaptive video streaming quality assessment. We believe this survey can serve as a guideline for QoE assessment of adaptive video streaming.
Communication-Efficient Federated Distillation with Active Data Sampling

Federated learning (FL) is a promising paradigm to enable privacy-preserving deep learning from distributed data. Most previous works are based on federated average (FedAvg), which, however, faces several critical issues, including a high communication overhead and the difficulty in dealing with heterogeneous model architectures. Federated Distillation (FD) is a recently proposed alternative to enable communication-efficient and robust FL, which achieves orders of magnitude reduction of the communication overhead compared with FedAvg and is flexible to handle heterogeneous models at the clients. However, so far there is no unified algorithmic framework or theoretical analysis for FD-based methods. In this paper, we first present a generic meta-algorithm for FD and investigate the influence of key parameters through empirical experiments. Then, we verify the empirical observations theoretically. Based on the empirical results and theory, we propose a communication-efficient FD algorithm with active data sampling to improve the model performance and reduce the communication overhead. Empirical simulations on benchmark datasets will demonstrate that our proposed algorithm effectively and significantly reduces the communication overhead while achieving a satisfactory performance.
XGBRS Framework Integrated with Word2Vec Sentiment Analysis for Augmented Drug Recommendation

Machine Learning is revolutionizing the era day by day and the scope is no more limited to computer science as the advancements are evident in the field of healthcare.Disease diagnosis, personalized medicine, and Recommendation system (RS) are among the promising applications that are using Machine Learning (ML) at a higher level.A recommendation system helps inefficient decision-making and suggests personalized recommendations accordingly.Today people share their experiences through reviews and hence designing of recommendation system based on users' sentiments is a challenge.The recommendation system has gained significant attention in different fields but considering healthcare, little is being done from the perspective of drugs, disease, and medical recommendations.This study is engrossed in designing a recommendation system that is based on the fusion of sentiment analysis and radiant boosting.The polarity of the sentiments is analyzed through user reviews and the processed data is fed into the Extreme Gradient Boosting (XGBOOST) framework to generate the drug recommendation.To establish the applicability of the concept a comparative study is performed between the proposed approach and the existing approaches.
Siamese Alignment Network for Weakly Supervised Video Moment Retrieval

Video moment retrieval, i.e., localizing the specific video moments within a video given a description query, has attracted substantial attention over the past several years. Although great progress has been achieved thus far, most of existing methods are supervised, which require moment-level temporal annotation information. In contrast, weakly-supervised methods which only need video-level annotations remain largely unexplored. In this paper, we propose a novel end-to-end Siamese alignment network for weakly-supervised video moment retrieval. To be specific, we design a multi-scale Siamese module, which could progressively reduce the semantic gap between the visual and textual modality with the Siamese structure. In addition, we present a context-aware multiple instance learning module by considering the influence of adjacent contexts, enhancing the moment-query and video-query alignment simultaneously. By promoting the matching of both moment-level and video-level, our model can effectively improve the retrieval performance, even if only having weak video level annotations. Extensive experiments on two benchmark datasets, i.e., ActivityNet-Captions and Charades-STA, verify the superiority of our model compared with several state-of-the-art baselines.
GAN Compression: Efficient Architectures for Interactive Conditional GANs

Conditional Generative Adversarial Networks (cGANs) have enabled controllable image synthesis for many vision and graphics applications. However, recent cGANs are 1-2 orders of magnitude more compute-intensive than modern recognition CNNs. For example, GauGAN consumes 281G MACs per image, compared to 0.44G MACs for MobileNet-v3, making it difficult for interactive deployment. In this work, we propose a general-purpose compression framework for reducing the inference time and model size of the generator in cGANs. Directly applying existing compression methods yields poor performance due to the difficulty of GAN training and the differences in generator architectures. We address these challenges in two ways. First, to stabilize GAN training, we transfer knowledge of multiple intermediate representations of the original model to its compressed model and unify unpaired and paired learning. Second, instead of reusing existing CNN designs, our method finds efficient architectures via neural architecture search. To accelerate the search process, we decouple the model training and search via weight sharing. Experiments demonstrate the effectiveness of our method across different supervision settings, network architectures, and learning methods. Without losing image quality, we reduce the computation of CycleGAN by 21×, Pix2pix by 12×, MUNIT by 29×, and GauGAN by 9×, paving the way for interactive image synthesis.
Efficient Quantum Network Communication Using Optimized Entanglement Swapping Trees

Quantum network communication is challenging, as the No-cloning theorem in quantum regime makes many classical techniques inapplicable. For long-distance communication, the only viable communication approach is teleportation of quantum states, which requires a prior distribution of entangled pairs (EPs) of qubits. Establishment of EPs across remote nodes can incur significant latency due to the low probability of success of the underlying physical processes. The focus of our work is to develop efficient techniques that minimize EP generation latency. Prior works have focused on selecting entanglement paths; in contrast, we select entanglement swapping trees--a more accurate representation of the entanglement generation structure. We develop a dynamic programming algorithm to select an optimal swapping-tree for a single pair of nodes, under the given capacity and fidelity constraints. For the general setting, we develop an efficient iterative algorithm to compute a set of swapping trees. We present simulation results which show that our solutions outperform the prior approaches by an order of magnitude and are viable for long-distance entanglement generation.
All classes of informationally complete symmetric measurements in finite dimensions

A broad class of informationally complete symmetric measurements is introduced. It can be understood as a common generalization of symmetric, informationally complete positive operator-valued measures and mutually unbiased bases. Additionally, it provides a natural way to define two more families of mutually unbiased symmetric measurement operators in any finite dimension. We show a general method of their construction, together with an example of an optimal measurement. Finally, we analyze the properties of symmetric measurements and provide applications in entropic relations and entanglement detection.
A Unified Framework for Bidirectional Prototype Learning From Contaminated Faces Across Heterogeneous Domains

Existing heterogeneous face synthesis (HFS) methods focus on performing accurate image-to-image translation across domains, while they cannot effectively remove the nuisance facial variations such as poses, expressions or occlusions. To address such challenges, this paper studies a new practical heterogeneous prototype learning (HPL) problem. To be specific, given a face image contaminated by facial variations from a source domain, HPL aims to reconstruct the variation-free prototype in a specified target domain. To tackle HPL, we propose a unified and end-to-end framework named bidirectional heterogeneous prototype learning (BHPL). As a bidirectional learning framework, BHPL is able to simultaneously reconstruct the heterogeneous prototypes across <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">source-to-target</i> as well as <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">target-to-source</i> domains. Furthermore, BHPL is capable of learning the identity prototype features for the contaminated face images from both source and target domains in order to perform robust heterogeneous face recognition. BHPL consists of an encoder-decoder structural generator and two dual-task discriminators, which play an adversarial game such that the generator learns the identity prototype feature and generates the cross-domain identity-preserved prototype for each input face image from both domains, and the discriminators accurately predict face identity and distinguish real versus fake prototypes. Empirically studies on multiple heterogeneous face datasets containing facial variations demonstrate the effectiveness of BHPL.
Domain-Agnostic Prior for Transfer Semantic Segmentation

Unsupervised domain adaptation (UDA) is an important topic in the computer vision community. The key difficulty lies in defining a common property between the source and target domains so that the source-domain features can align with the target-domain semantics. In this paper, we present a simple and effective mechanism that regularizes cross-domain representation learning with a domain-agnostic prior (DAP) that constrains the features extracted from source and target domains to align with a domain-agnostic space. In practice, this is easily implemented as an extra loss term that requires a little extra costs. In the standard evaluation protocol of transferring synthesized data to real data, we validate the effectiveness of different types of DAP, especially that borrowed from a text embedding model that shows favorable performance beyond the state-of-the-art UDA approaches in terms of segmentation accuracy. Our research reveals that UDA benefits much from better proxies, possibly from other data modalities.
DeepQGHO: Quantized Greedy Hyperparameter Optimization in Deep Neural Networks for on-the-Fly Learning

On-the-fly learning is unavoidable for applications that demand instantaneous deep neural network (DNN) training or where transferring data to the central system for training is costly. Hyperparameter optimization plays a significant role in the performance and reliability in deep learning. Many hyperparameter optimization algorithms have been developed for obtaining better validation accuracy in DNNs. Most state-of-the-art hyperparameter optimization techniques are computationally expensive due to the focus on validation accuracy. Therefore, they are unsuitable for on-the-fly learning applications that require faster computation on resource constraint devices (e.g., edge devices). In this paper, we develop a novel greedy approach-based hyperparameter optimization (GHO) algorithm enabling faster computing on edge devices for on-the-fly learning applications. In GHO, we obtain the validation accuracy locally for each of the hyperparameter configurations. The GHO algorithm optimizes each hyperparameter while keeping the others constant in order to converge to the locally optimal solution in the expectation that this choice will lead to a globally optimal solution. We perform an empirical study to compute the performance such as computation time and energy consumption of the GHO and compare it with two state-of-the-art hyperparameter optimization algorithms. We also deploy the GHO algorithm in an edge device to validate the performance of our algorithm. We perform post-training quantization on the GHO algorithm to reduce the inference time. Our GHO algorithm is more than <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$3\times $ </tex-math></inline-formula> energy efficient and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$2\times $ </tex-math></inline-formula> faster than two state-of-the-art hyperparameter optimization techniques on both DNNs and datasets studied in this paper.
A Survey on Shape-Constraint Deep Learning for Medical Image Segmentation

Since the advent of U-Net, fully convolutional deep neural networks and its many variants have completely changed the modern landscape of deep-learning based medical image segmentation. However, the over-dependence of these methods on pixel-level classification and regression has been identified early on as a problem. Especially when trained on medical databases with sparse available annotation, these methods are prone to generate segmentation artifacts such as fragmented structures, topological inconsistencies and islands of pixel. These artifacts are especially problematic in medical imaging since segmentation is almost always a pre-processing step for some downstream evaluations like surgical planning, visualization, prognosis, or treatment planning. However, one common thread across all these downstream tasks is the demand of anatomical consistency. To ensure the segmentation result is anatomically consistent, approaches based on Markov/ Conditional Random Fields, Statistical Shape Models, Active Contours are becoming increasingly popular over the past 5 years. In this review paper, a broad overview of recent literature on bringing explicit anatomical constraints for medical image segmentation is given, the shortcomings and opportunities are discussed and the potential shift towards implicit shape modelling is elaborated. We review the most relevant papers published until the submission date and provide a tabulated view with method details for quick access.
Proof of Sense: A Novel Consensus Mechanism for Spectrum Misuse Detection

Optimal use of scarce radio spectrum is essential in the proliferation of beyond 5G networks, and promising blockchain technology offers various benefits for the spectrum management. However, existing blockchain-based solutions are expensive, nonoptimized, and lack spectrum fraud detection. This article proposes a novel consensus mechanism for a blockchain-based dynamic spectrum access (DSA) system. The proposed “ <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Proof-of-Sense</i> ” consensus mechanism operates based on spectrum sensing procedures rather than cryptographic calculations. It is specially designed to address fraudulent/unauthorized access to the spectrum by analyzing the sensed spectrum data. The core of the consensus mechanism is a cryptographic key sharing mechanism inspired by Shamir’s secret sharing scheme. Moreover, the proposed DSA system can enable different microservices, such as automated spectrum auctions, payment and penalty handling, and spectrum fraud detection. A proof of concept based on experimental approaches coupled with Matlab simulations is presented to analyze the performance of the proposed consensus mechanism.
Intelligent multi-agent reinforcement learning model for resources allocation in cloud computing

Now more than ever, optimizing resource allocation in cloud computing is becoming more critical due to the growth of cloud computing consumers and meeting the computing demands of modern technology. Cloud infrastructures typically consist of heterogeneous servers, hosting multiple virtual machines with potentially different specifications, and volatile resource usage. This makes the resource allocation face many issues such as energy conservation, fault tolerance, workload balancing, etc. Finding a comprehensive solution that considers all these issues is one of the essential concerns of cloud service providers. This paper presents a new resource allocation model based on an intelligent multi-agent system and reinforcement learning method (IMARM). It combines the multi-agent characteristics and the Q-learning process to improve the performance of cloud resource allocation. IMARM uses the properties of multi-agent systems to dynamically allocate and release resources, thus responding well to changing consumer demands. Meanwhile, the reinforcement learning policy makes virtual machines move to the best state according to the current state environment. Also, we study the impact of IMARM on execution time. The experimental results showed that our proposed solution performs better than other comparable algorithms regarding energy consumption and fault tolerance, with reasonable load balancing and respectful execution time.
Cyberassured Systems Engineering at Scale

Our team has developed a model-based systems engineering environment that integrates formal methods at all levels of system design. Our methodology and tools enable systems engineers to address cybersecurity concerns early in the development of complex high-assurance systems.