Reviewerly/exHarmony huggingface dataset:
Towards characterizing adversarial defects of deep learning software from the lens of uncertainty

Over the past decade, deep learning (DL) has been successfully applied to many industrial domain-specific tasks. However, the current state-of-the-art DL software still suffers from quality issues, which raises great concern especially in the context of safety- and security-critical scenarios. Adversarial examples (AEs) represent a typical and important type of defects needed to be urgently addressed, on which a DL software makes incorrect decisions. Such defects occur through either intentional attack or physical-world noise perceived by input sensors, potentially hindering further industry deployment. The intrinsic uncertainty nature of deep learning decisions can be a fundamental reason for its incorrect behavior. Although some testing, adversarial attack and defense techniques have been recently proposed, it still lacks a systematic study to uncover the relationship between AEs and DL uncertainty.
Semi-supervised Segmentation Based on Error-Correcting Supervision

Pixel-level classification is an essential part of computer vision. For learning from labeled data, many powerful deep learning models have been developed recently. In this work, we augment such supervised segmentation models by allowing them to learn from unlabeled data. Our semi-supervised approach, termed Error-Correcting Supervision, leverages a collaborative strategy. Apart from the supervised training on the labeled data, the segmentation network is judged by an additional network. The secondary correction network learns on the labeled data to optimally spot correct predictions, as well as to amend incorrect ones. As auxiliary regularization term, the corrector directly influences the supervised training of the segmentation network. On unlabeled data, the output of the correction network is essential to create a proxy for the unknown truth. The corrector’s output is combined with the segmentation network’s prediction to form the new target. We propose a loss function that incorporates both the pseudo-labels as well as the predictive certainty of the correction network. Our approach can easily be added to supervised segmentation models. We show consistent improvements over a supervised baseline on experiments on both the Pascal VOC 2012 and the Cityscapes datasets with varying amounts of labeled data.
SRAF-Net: Shape Robust Anchor-Free Network for Garbage Dumps in Remote Sensing Imagery

The detection of garbage dumps is of great significance for environmental protection. Recently, deep learning algorithms have brought impressive improvements for regular object detection. Different from conventional objects, garbage dumps are more inconspicuous and irregular and have the problem of blurred boundaries. To solve these problems, we propose a shape robust anchor-free network (SRAF-Net) that consists of feature extraction, multitask detection, and postprocessing. First, our network leverages the context-based deformable (CBD) module to combine context attention and deformable convolution. The contextual information obtained by context attention enables the network to focus on objects with inconspicuous appearance, while the deformable convolution enhances the feature representation. Then, we propose a multitask detection head to regress irregular garbage dumps in a more accurate and efficient way. The anchor-based methods need to define some anchors with a fixed shape. However, our detection method is anchor-free that learns the shapes of objects from training data. The detection head adaptively generates various shapes of bounding boxes with their classification confidences and localization confidences. Weighted by the localization confidences, we merge bounding boxes during postprocessing, which alleviates the blurred boundaries. In addition, we build a new public data set named garbage dumps data set (GDD) to verify the effectiveness of our method. Extensive experiments on GDD indicate that our method surpasses the existing detection methods in terms of speed and accuracy for the garbage dumps detection task.
Hierarchical Gumbel Attention Network for Text-based Person Search

Text-based person search aims to retrieve the pedestrian images that best match a given textual description from gallery images. Previous methods utilize the soft-attention mechanism to infer the semantic alignments between the regions of image and the corresponding words in sentence. However, these methods may fuse the irrelevant multi-modality features together which cause matching redundancy problem. In this work, we propose a novel hierarchical Gumbel attention network for text-based person search via Gumbel top-k re-parameterization algorithm. Specifically, it adaptively selects the strong semantically relevant image regions and words/phrases from images and texts for precise alignment and similarity calculation. This hard selection strategy is able to fuse the strong-relevant multi-modality features for alleviating the problem of matching redundancy. Meanwhile, a Gumbel top-k re-parameterization algorithm is designed as a low-variance, unbiased gradient estimator to handle the discreteness problem of hard attention mechanism by an end-to-end manner. Moreover, a hierarchical adaptive matching strategy is employed by the model from three different granularities, i.e., word-level, phrase-level, and sentence-level, towards fine-grained matching. Extensive experimental results demonstrate the state-of-the-art performance. Compared the existed best method, we achieve the 8.24% Rank-1 and 7.6% mAP relative improvements in the text-to-image retrieval task, and 5.58% Rank-1 and 6.3% mAP relative improvements in the image-to-text retrieval task on CUHK-PEDES dataset, respectively.
Design of experiments using artificial neural network ensemble for photovoltaic generation forecasting

In recent years, renewable and sustainable energy sources have attracted the attention of various investors and stakeholders, such as energy sector agents and even consumers. It is perplexing to observe and anticipate the required levels of photovoltaic generation, which are inherent tasks for such rapid insertion into the electric grid. This distributed/renewable generation must be integrated in a coordinated way such that there is no negative impact on the electric performance of the grid, increasing in the complexity of energy management. In this article, a methodology for photovoltaic generation forecasting is addressed for a horizon of one week ahead, using a new approach based on an artificial neural network (ANN) ensemble. Two main questions will be explored with this approach: how to select the ANNs, and how to combine them in the ensemble. The design of experiments (DOE) approach is applied to the photovoltaic time series factors and ANN factors. Then, a cluster analysis is performed to select the best networks. From this point on, a mixture (MDE) is employed to determine the ideal weights for the ensemble formation. The methodology is detailed throughout the paper and, based on the combination of forecasts, the photovoltaic generation was estimated for a specific panel set located in the state of Minas Gerais, Brazil, reaching the value of 4.7% for the weekly mean absolute percentage error. The versatility of the proposed method allowed the change of the number of factors to be used in the experimental arrangement, the forecast model, and the desired forecast horizon, and consequently enhancing the forecasting determination.
Chip-scale nonlinear photonics for quantum light generation

Nonclassical states of light are an essential resource for many emerging quantum technologies and applications ranging from information processing, encrypted communications, and networking to sensing, metrology, and imaging. Nonlinear optical processes in solid-state materials are widely used for generating quantum light, including single photons, entangled-photon pairs, and quadrature-squeezed states. Recent advances in nonlinear photonics have enabled the functionality of benchtop nonlinear instruments to be scaled down to a single chip without sacrificing efficiency or degrading the key performance metrics. The dramatic improvement in the size, weight, power, cost, and stability enabled by photonic integrated circuits has been essential for enabling the chip-scale generation, manipulation, and detection of quantum light at a steadily increasing degree of complexity and scale. Within the last decade, the authors have seen the progression from few-component photonic circuits operating on two photons to arrays of 18 identical heralded single-photon sources and reconfigurable devices operating with more than 650 components for multidimensional entanglement and arbitrary two-photon quantum gates. In this review, the authors summarize the history and recent key technological developments of chip-scale nonlinear quantum light generation based on integrated nonlinear photonics, recent advances in heterogeneous integrated methods, and approaches for system-level integration and demonstrated applications.
Occlusion gesture recognition based on improved <scp>SSD</scp>

Summary Gesture recognition has always been a research hotspot in the field of human‐computer interaction. Its purpose is to realize the natural interaction with the machine by recognizing the semantics expressed by gesture. In the process of gesture recognition, the occlusion of gesture is an inevitable problem. In the process of gesture recognition, some or even all of the gesture features will be lost due to the occlusion of the gesture, resulting in the wrong recognition or even unrecognizability of the gesture. Therefore, it is of great significance to study gesture recognition under occlusion. The single shot multibox detector (SSD) algorithm is analyzed, and the front‐end network is compared. Mobilenets is selected as the front‐end network, and the Mobilenets‐SSD network is improved. In tensorflow environment, based on the improved network model, the self‐occlusion gesture and object occluding gesture are trained in color map, depth map, and color and depth fusion respectively. The recognition models of self‐occlusion gestures and object‐occlusion gestures in color map, depth map, and color and depth fusion are obtained. And compare and analyze the learning rate, loss function, and average accuracy of various models obtained for occlusion gesture recognition.
Self-supervising Fine-Grained Region Similarities for Large-Scale Image Localization

The task of large-scale retrieval-based image localization is to estimate the geographical location of a query image by recognizing its nearest reference images from a city-scale dataset. However, the general public benchmarks only provide noisy GPS labels associated with the training images, which act as weak supervisions for learning image-to-image similarities. Such label noise prevents deep neural networks from learning discriminative features for accurate localization. To tackle this challenge, we propose to self-supervise image-to-region similarities in order to fully explore the potential of difficult positive images alongside their sub-regions. The estimated image-to-region similarities can serve as extra training supervision for improving the network in generations, which could in turn gradually refine the fine-grained similarities to achieve optimal performance. Our proposed self-enhanced image-to-region similarity labels effectively deal with the training bottleneck in the state-of-the-art pipelines without any additional parameters or manual annotations in both training and inference. Our method outperforms state-of-the-arts on the standard localization benchmarks by noticeable margins and shows excellent generalization capability on multiple image retrieval datasets (Code of this work is available at https://github.com/yxgeee/SFRS .).
Detection of calibration drift in clinical prediction models to inform model updating

Model calibration, critical to the success and safety of clinical prediction models, deteriorates over time in response to the dynamic nature of clinical environments. To support informed, data-driven model updating strategies, we present and evaluate a calibration drift detection system. Methods are developed for maintaining dynamic calibration curves with optimized online stochastic gradient descent and for detecting increasing miscalibration with adaptive sliding windows. These methods are generalizable to support diverse prediction models developed using a variety of learning algorithms and customizable to address the unique needs of clinical use cases. In both simulation and case studies, our system accurately detected calibration drift. When drift is detected, our system further provides actionable alerts by including information on a window of recent data that may be appropriate for model updating. Simulations showed these windows were primarily composed of data accruing after drift onset, supporting the potential utility of the windows for model updating. By promoting model updating as calibration deteriorates rather than on pre-determined schedules, implementations of our drift detection system may minimize interim periods of insufficient model accuracy and focus analytic resources on those models most in need of attention.
Building a Manga Dataset “Manga109” With Annotations for Multimedia Applications

Manga, or comics, which are a type of multimodal artwork, have been left behind in the recent trend of deep learning applications because of the lack of a proper dataset.Hence, we built Manga109, a dataset consisting of a variety of 109 Japanese comic books (94 authors and 21,142 pages) and made it publicly available by obtaining author permissions for academic use.We carefully annotated the frames, speech texts, character faces, and character bodies; the total number of annotations exceeds 500k.This dataset provides numerous manga images and annotations, which will be beneficial for use in machine learning algorithms and their evaluation.In addition to academic use, we obtained further permission for a subset of the dataset for industrial use.In this article, we describe the details of the dataset and present a few examples of multimedia processing applications (detection, retrieval, and generation) that apply existing deep learning methods and are made possible by the dataset.
ConfuciuX: Autonomous Hardware Resource Assignment for DNN Accelerators using Reinforcement Learning

DNN accelerators provide efficiency by leveraging reuse of activations/weights/outputs during the DNN computations to reduce data movement from DRAM to the chip. The reuse is captured by the accelerator's dataflow. While there has been significant prior work in exploring and comparing various dataflows, the strategy for assigning on-chip hardware resources (i.e., compute and memory) given a dataflow that can optimize for performance/energy while meeting platform constraints of area/power for DNN(s) of interest is still relatively unexplored. The design-space of choices for balancing compute and memory explodes combinatorially, as we show in this work (e.g., as large as O(10 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">72</sup> ) choices for running MobileNet-V2), making it infeasible to do manual-tuning via exhaustive searches. It is also difficult to come up with a specific heuristic given that different DNNs and layer types exhibit different amounts of reuse.In this paper, we propose an autonomous strategy called Con-fuciuX to find optimized HW resource assignments for a given model and dataflow style. ConfuciuX leverages a reinforcement learning method, REINFORCE, to guide the search process, leveraging a detailed HW performance cost model within the training loop to estimate rewards. We also augment the RL approach with a genetic algorithm for further fine-tuning. Con-fuciuX demonstrates the highest sample-efficiency for training compared to other techniques such as Bayesian optimization, genetic algorithm, simulated annealing, and other RL methods. It converges to the optimized hardware configuration 4.7 to 24 times faster than alternate techniques.
JSTASR: Joint Size and Transparency-Aware Snow Removal Algorithm Based on Modified Partial Convolution and Veiling Effect Removal

Snow removal usually affects the performance of computer vision. Comparing with other atmospheric phenomenon (e.g., haze and rain), snow is more complicated due to its transparency, various size, and accumulation of veiling effect, which make single image de-snowing more challenging. In this paper, first, we reformulate the snow model. Different from that in the previous works, in the proposed snow model, the veiling effect is included. Second, a novel joint size and transparency-aware snow removal algorithm called JSTASR is proposed. It can classify snow particles according to their sizes and conduct snow removal in different scales. Moreover, to remove the snow with different transparency, the transparency-aware snow removal is developed. It can address both transparent and non-transparent snow particles by applying the modified partial convolution. Experiments show that the proposed method achieves significant improvement on both synthetic and real-world datasets and is very helpful for object detection on snow images.
Adaptive Cost-Sensitive Online Classification

Cost-Sensitive Online Classification has drawn extensive attention in recent years, where the main approach is to directly online optimize two well-known cost-sensitive metrics: (i) weighted sum of sensitivity and specificity and (ii) weighted misclassification cost. However, previous existing methods only considered first-order information of data stream. It is insufficient in practice, since many recent studies have proved that incorporating second-order information enhances the prediction performance of classification models. Thus, we propose a family of cost-sensitive online classification algorithms with adaptive regularization in this paper. We theoretically analyze the proposed algorithms and empirically validate their effectiveness and properties in extensive experiments. Then, for better trade off between the performance and efficiency, we further introduce the sketching technique into our algorithms, which significantly accelerates the computational speed with quite slight performance loss. Finally, we apply our algorithms to tackle several online anomaly detection tasks from real world. Promising results prove that the proposed algorithms are effective and efficient in solving cost-sensitive online classification problems in various real-world domains.
A theory of single-shot error correction for adversarial noise

Single-shot error correction is a technique for correcting physical errors using only a single round of noisy check measurements, such that any residual noise affects a small number of qubits. We propose a general theory of single-shot error correction and establish a sufficient condition called good soundness of the code's measurement checks. Good code soundness in topological (or LDPC) codes is shown to entail a macroscopic energy barrier for the associated Hamiltonian. Consequently, 2D topological codes with local checks can not have good soundness. In tension with this, we also show that for any code a specific choice of measurement checks does exist that provides good soundness. In other words, every code can perform single-shot error correction but the required checks may be nonlocal and act on many qubits. If we desire codes with both good soundness and simple measurement checks (the LDPC property) then careful constructions are needed. Finally, we use a double application of the homological product to construct quantum LDPC codes with single-shot error correcting capabilities. Our double homological product codes exploit redundancy in measurements checks through a process we call metachecking.
Normality Learning in Multispace for Video Anomaly Detection

Video anomaly detection is a challenging task owing to the rare and diverse nature of abnormal events. However, most of the existing methods only learn the normality in a single space, focusing on low-level detailed features, which is easily affected by unimportant pixels. To address this issue, in this study, we propose a semi-supervised method based on the generative adversarial network and frame prediction, wherein the normality is learned in both the original image space and latent space, and the events deviating from the normality are detected as anomalies. In particular, given a video clip, we first predict a future frame and minimize the prediction errors between the generated frame and its ground truth. Thereafter, we encode the predicted frames and their ground truths in the latent space and minimize their differences. In the testing phase, we calculate the normal scores of each frame in both the image and latent spaces to obtain a comprehensive evaluation. Utilizing the multispace can capture more normality distribution information of the data, which can benefit anomaly detection. The results of experiments on three benchmark datasets demonstrate the effectiveness of the proposed method.
A Principled Approach to Data Valuation for Federated Learning

Federated learning (FL) is a popular technique to train machine learning (ML) models on decentralized data sources. In order to sustain long-term participation of data owners, it is important to fairly appraise each data source and compensate data owners for their contribution to the training process. The Shapley value (SV) defines a unique payoff scheme that satisfies many desiderata for a data value notion. It has been increasingly used for valuing training data in centralized learning. However, computing the SV requires exhaustively evaluating the model performance on every subset of data sources, which incurs prohibitive communication cost in the federated setting. Besides, the canonical SV ignores the order of data sources during training, which conflicts with the sequential nature of FL. This chapter proposes a variant of the SV amenable to FL, which we call the federated Shapley value. The federated SV preserves the desirable properties of the canonical SV while it can be calculated without incurring extra communication cost and is also able to capture the effect of participation order on data value. We conduct a thorough empirical study of the federated SV on a range of tasks, including noisy label detection, adversarial participant detection, and data summarization on different benchmark datasets, and demonstrate that it can reflect the real utility of data sources for FL and has the potential to enhance system robustness, security, and efficiency. We also report and analyze “failure cases” and hope to stimulate future research.