Reviewerly/exHarmony huggingface dataset:
Graph Attention Networks for Speaker Verification

This work presents a novel back-end framework for speaker verification using graph attention networks. Segment-wise speaker embeddings extracted from multiple crops within an utterance are interpreted as node representations of a graph. The proposed framework inputs segment-wise speaker embeddings from an enrollment and a test utterance and directly outputs a similarity score. We first construct a graph using segment-wise speaker embeddings and then input these to graph attention networks. After a few graph attention layers with residual connections, each node is projected into a one-dimensional space using affine transform, followed by a readout operation resulting in a scalar similarity score. To enable successful adaptation for speaker verification, we propose techniques such as separating trainable weights for attention map calculations between segment-wise speaker embeddings from different utterances. The effectiveness of the proposed framework is validated using three different speaker embedding extractors trained with different architectures and objective functions. Experimental results demonstrate consistent improvement over various baseline back-end classifiers, with an average equal error rate improvement of 20% over the cosine similarity back-end without test time augmentation.
AISpeech-SJTU Accent Identification System for the Accented English Speech Recognition Challenge

This paper describes the AISpeech-SJTU system for the accent identification track of the Interspeech-2020 Accented English Speech Recognition Challenge. In this challenge track, only 160-hour accented English data collected from 8 countries and the auxiliary Librispeech dataset are provided for training. To build an accurate and robust accent identification system, we explore the whole system pipeline in detail. First, we introduce the ASR based phone posteriorgram (PPG) feature to accent identification and verify its efficacy. Then, a novel TTS based approach is carefully designed to augment the very limited accent training data for the first time. Finally, we propose the test time augmentation and embedding fusion schemes to further improve the system performance. Our final system is ranked first in the challenge and outperforms all the other participants by a large margin. The submitted system achieves 83.63% average accuracy on the challenge evaluation data, ahead of the others by more than 10% in absolute terms.
Non-elitist evolutionary algorithms excel in fitness landscapes with sparse deceptive regions and dense valleys

It is largely unknown how the runtime of evolutionary algorithms depends on fitness landscape characteristics for broad classes of problems. Runtime guarantees for complex and multi-modal problems where EAs are typically applied are rarely available.
On the Co-evolution of ML Pipelines and Source Code - Empirical Study of DVC Projects

The growing popularity of machine learning (ML) applications has led to the introduction of software engineering tools such as Data Versioning Control (DVC), MLFlow and Pachyderm that enable versioning ML data, models, pipelines and model evaluation metrics. Since these versioned ML artifacts need to be synchronized not only with each other, but also with the source and test code of the software applications into which the models are integrated, prior findings on co-evolution and coupling between software artifacts might need to be revisited. Hence, in order to understand the degree of coupling between ML-related and other software artifacts, as well as the adoption of ML versioning features, this paper empirically studies the usage of DVC in 391 Github projects, 25 of which in detail. Our results show that more than half of the DVC files in a project are changed at least once every one-tenth of the project's lifetime. Furthermore, we observe a tight coupling between DVC files and other artifacts, with 1/4 pull requests changing source code and 1/2 pull requests changing tests requiring a change to DVC files. As additional evidence of the observed complexity associated with adopting ML-related software engineering tools like DVC, an average of 78% of the studied projects showed a non-constant trend in pipeline complexity.
Can Program Synthesis be Used to Learn Merge Conflict Resolutions? An Empirical Analysis

Forking structure is widespread in the open-source repositories and that causes a significant number of merge conflicts. In this paper, we study the problem of textual merge conflicts from the perspective of Microsoft Edge, a large, highly collaborative fork off the main Chromium branch with significant merge conflicts. Broadly, this study is divided into two sections. First, we empirically evaluate textual merge conflicts in Microsoft Edge and classify them based on the type of files, location of conflicts in a file, and the size of conflicts. We found that ~28% of the merge conflicts are 1-2 line changes, and many resolutions have frequent patterns. Second, driven by these findings, we explore Program Synthesis (for the first time) to learn patterns and resolve structural merge conflicts. We propose a novel domain-specific language (DSL) that captures many of the repetitive merge conflict resolution patterns and learn resolution strategies as programs in this DSL from example resolutions. We found that the learned strategies can resolve 11.4% of the conflicts (~41% of 1-2 line changes) that arise in the C++ files with 93.2% accuracy.
Efficient Data Retrieval Over Encrypted Attribute-Value Type Databases in Cloud-Assisted Ehealth Systems

In cloud-assisted electronic health (eHealth) systems, outsourced electronic health records (EHRs) have attribute-value type formats: an EHR corresponds to an entry with a unique identity and has multiple types of attribute values. Such formatted EHRs form an attribute-value type database, where both the attribute values and unique identities can serve as keywords for searching. Since EHRs are very sensitive, they are always encrypted before being outsourced, which makes retrieval of target EHRs by either identity or attribute value hard. Moreover, the EHRs are encrypted by different doctors, a researcher, who is delegated to research a certain kind of disease, cannot find out all corresponding EHRs due to the difference of encryption keys. In this article, we construct a triple dictionary index structure for the attribute-value type database to allow a researcher to retrieve encrypted EHRs by the identity and attribute value and to perform dynamic operations over them. We employ an identity server to assist doctors in generating encryption keys via an oblivious way. By doing so, the researcher can retrieve encrypted EHRs without leaking any information to the identity server. We analyze the security and evaluate the performance of our scheme to demonstrate that it achieves <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$L$</tex-math></inline-formula> -adaptive security and forward security with high efficiency.
Infrared thermal imaging denoising method based on second-order channel attention mechanism

We propose a deep learning (DL) image denoising method to solve the problem of poor image quality and serious noise interference in the infrared thermal imaging. We develop a DL model to process the image, fit the noise residual of the image and remove the noise from the original image to get a clear image. The model adopts the convolutional neural network (CNN) architecture and uses the second-order attention mechanism and non-local modules at the regional level to improve the extraction of image features and fit the noise residual. We implement experiments on four different datasets to analyze the performance of the algorithm in different noise environments. The experimental results show that the proposed method can effectively remove the noise in the infrared image, performs well in different types of noise, and retain a lot of image details.
Improved LSTM-Based Abnormal Stream Data Detection and Correction System for Internet of Things

The Internet of Things (IoT) is the integration of all information and Internet technology in the information age, which can realize the collection and transmission of intelligent information. A large number of sensors are producing and collecting data involving various industries every day. The amount of stream data generated is huge, and a large number of abnormal data are also generated in the process. Due to the demands of business and life quality improvement, the application of IoT technology to real-time monitoring and correction of massive stream data, especially the correction of abnormal data, is a very valuable research direction, and also the key to ensure the credibility and fidelity of IoT data. This article proposes a recurrent neural network model based on long- and short-term memory network (LSTM) and LSTM+. LSTM+ model not only reduces the regression error compared with the traditional LSTM model, but also can detect abnormal data collected by IoT terminal nodes, and can correct the abnormal data in real time, so as to ensure that the network prediction can have good stability and robustness.
Examining the Intersections of Race, Religion &amp; Community Technologies: A Photovoice Study

Churches have historically played an important role in Black American communities, catalyzing the pursuit of aims such as social justice, community organization, and health promotion. However, researchers have rarely examined how technology can support an assets-based approach to these efforts, nor the implications of race, traditions, and history when creating such systems. Addressing this gap, we conducted research with two predominantly Black churches to explore health promotion design opportunities. We used photovoice, a research method where participants led their own data collection and analysis. Participants provided nuanced descriptions of the racial and ethnic identities of their communities, and how church history and aspirations for the future impacted these identities. Our findings characterize tensions between tradition and 'modernization,' implications for technology design, and the need for a temporal approach to understanding communities. We conclude with broader implications for studying the intersection of race and religion in community technology design.
BDF: A new decision forest algorithm

The foremost requirement for a decision forest to achieve better ensemble accuracy is building a set of accurate and diverse individual decision trees as base classifiers. Existing decision forest algorithms mainly differ from each other on how they induce diversity among the decision trees. At the same time, most of the drawbacks of existing algorithms originate from their induction processes of diversity. In this paper, we propose a new decision forest algorithm that is more balanced through effective synchronization between different sources of diversity. The proposed algorithm is balanced theoretically and empirically. We carried out experiments on 25 well-known data sets that are publicly available from the UCI Machine Learning Repository, to perform an extensive empirical evaluation. The experimental results indicate that the proposed algorithm has the best average ensemble accuracy rank of 1.8 compared to its closest competitor at 3.5. Using the Friedman and Bonferroni-Dunn tests, we also show that such an improvement is indeed statistically significant. In addition, the proposed algorithm is found to be competitive in terms of complexity and other relevant parameters.
Image smoothing based on global sparsity decomposition and a variable parameter

Abstract Smoothing images, especially with rich texture, is an important problem in computer vision. Obtaining an ideal result is difficult due to complexity, irregularity, and anisotropicity of the texture. Besides, some properties are shared by the texture and the structure in an image. It is a hard compromise to retain structure and simultaneously remove texture. To create an ideal algorithm for image smoothing, we face three problems. For images with rich textures, the smoothing effect should be enhanced. We should overcome inconsistency of smoothing results in different parts of the image. It is necessary to create a method to evaluate the smoothing effect. We apply texture pre-removal based on global sparse decomposition with a variable smoothing parameter to solve the first two problems. A parametric surface constructed by an improved Bessel method is used to determine the smoothing parameter. Three evaluation measures: edge integrity rate, texture removal rate, and gradient value distribution are proposed to cope with the third problem. We use the alternating direction method of multipliers to complete the whole algorithm and obtain the results. Experiments show that our algorithm is better than existing algorithms both visually and quantitatively. We also demonstrate our method’s ability in other applications such as clip-art compression artifact removal and content-aware image manipulation.
Adaptive Hypergraph Convolutional Network for No-Reference 360-degree Image Quality Assessment

In no-reference 360-degree image quality assessment (NR 360IQA), graph convolutional networks (GCNs), which model interactions between viewports through graphs, have achieved impressive performance. However, prevailing GCN-based NR 360IQA methods suffer from three main limitations. First, they only use high-level features of the distorted image to regress the quality score, while the human visual system scores the image based on hierarchical features. Second, they simplify complex high-order interactions between viewports in a pairwise fashion through graphs. Third, in the graph construction, they only consider the spatial location of the viewport, ignoring its content characteristics. Accordingly, to address these issues, we propose an adaptive hypergraph convolutional network for NR 360IQA, denoted as AHGCN. Specifically, we first design a multi-level viewport descriptor for extracting hierarchical representations from viewports. Then, we model interactions between viewports through hypergraphs, where each hyperedge connects two or more viewports. In the hypergraph construction, we build a location-based hyperedge and a content-based hyperedge for each viewport. Experimental results on two public 360IQA databases demonstrate that our proposed approach has a clear advantage over state-of-the-art full-reference and no-reference IQA models.
APPL: Adaptive Planner Parameter Learning

While current autonomous navigation systems allow robots to successfully drive themselves from one point to another in specific environments, they typically require extensive manual parameter re-tuning by human robotics experts in order to function in new environments. Furthermore, even for just one complex environment, a single set of fine-tuned parameters may not work well in different regions of that environment. These problems prohibit reliable mobile robot deployment by non-expert users. As a remedy, we propose Adaptive Planner Parameter Learning (appl), a machine learning framework that can leverage non-expert human interaction via several modalities – including teleoperated demonstrations, corrective interventions, and evaluative feedback – and also unsupervised reinforcement learning to learn a parameter policy that can dynamically adjust the parameters of classical navigation systems in response to changes in the environment. appl inherits safety and explainability from classical navigation systems while also enjoying the benefits of machine learning, i.e., the ability to adapt and improve from experience. We present a suite of individual appl methods and also a unifying cycle-of-learning scheme that combines all the proposed methods in a framework that can improve navigation performance through continual, iterative human interaction and simulation training.
Integrating Deep Learning with First-Order Logic Programmed Constraints for Zero-Day Phishing Attack Detection

Considering the fatality of phishing attacks that are emphasized by many organizations, the inductive learning approach using reported malicious URLs has been verified in the field of deep learning. However, the deep learning-based method mainly focused on the fitting of a classification task via historical URL observation shows a limitation of recall due to the characteristics of zero-day attack. In order to model the nature of a zero-day phishing attack in which URL addresses are generated and discarded immediately, an approach that utilizes the expert knowledge is promising. We introduce the integration method of deep learning and logic programmed domain knowledge to inject the real-world constraints. We design neural and logic classifiers and propose the joint learning method of each component based on the traditional neuro-symbolic integration. Extensive experiments on three real-world datasets consisting of 222,541 URLs showed the highest recall among the latest deep learning methods, despite the hostile class-imbalanced condition. We demonstrate that the optimized weighting between neural and logic component has an effect of improving the recall over 3% compared to the existing methods.
Robust Multimodal Representation Learning With Evolutionary Adversarial Attention Networks

Multimodal representation learning is beneficial for many multimedia-oriented applications, such as social image recognition and visual question answering. The different modalities of the same instance (e.g., a social image and its corresponding description) are usually correlational and complementary. Most existing approaches for multimodal representation learning are not effective to model the deep correlation between different modalities. Moreover, it is difficult for these approaches to deal with the noise within social images. In this article, we propose a deep learning-based approach named evolutionary adversarial attention networks (EAANs), which combines the attention mechanism with adversarial networks through evolutionary training, for robust multimodal representation learning. Specifically, a two-branch visual-textual attention model is proposed to correlate visual and textual content for joint representation. Then adversarial networks are employed to impose regularization upon the representation by matching its posterior distribution to the given priors. Finally, the attention model and adversarial networks are integrated into an evolutionary training framework for robust multimodal representation learning. Extensive experiments have been conducted on four real-world datasets, including PASCAL, MIR, CLEF, and NUS-WIDE. Substantial performance improvements on the tasks of image classification and tag recommendation demonstrate the superiority of the proposed approach.
A Study of Multilingual Toxic Text Detection Approaches under Imbalanced Sample Distribution

Multilingual characteristics, lack of annotated data, and imbalanced sample distribution are the three main challenges for toxic comment analysis in a multilingual setting. This paper proposes a multilingual toxic text classifier which adopts a novel fusion strategy that combines different loss functions and multiple pre-training models. Specifically, the proposed learning pipeline starts with a series of pre-processing steps, including translation, word segmentation, purification, text digitization, and vectorization, to convert word tokens to a vectorized form suitable for the downstream tasks. Two models, multilingual bidirectional encoder representation from transformers (MBERT) and XLM-RoBERTa (XLM-R), are employed for pre-training through Masking Language Modeling (MLM) and Translation Language Modeling (TLM), which incorporate semantic and contextual information into the models. We train six base models and fuse them to obtain three fusion models using the F1 scores as the weights. The models are evaluated on the Jigsaw Multilingual Toxic Comment dataset. Experimental results show that the best fusion model outperforms the two state-of-the-art models, MBERT and XLM-R, in F1 score by 5.05% and 0.76%, respectively, verifying the effectiveness and robustness of the proposed fusion strategy.
Representation of Developer Expertise in Open Source Software

Background: Accurate representation of developer expertise has always been an important research problem. While a number of studies proposed novel methods of representing expertise within individual projects, these methods are difficult to apply at an ecosystem level. However, with the focus of software development shifting from monolithic to modular, a method of representing developers' expertise in the context of the entire OSS development becomes necessary when, for example, a project tries to find new maintainers and look for developers with relevant skills. Aim: We aim to address this knowledge gap by proposing and constructing the Skill Space where each API, developer, and project is represented and postulate how the topology of this space should reflect what developers know (and projects need). Method: we use the World of Code infrastructure to extract the complete set of APIs in the files changed by open source developers and, based on that data, employ Doc2Vec embeddings for vector representations of APIs, developers, and projects. We then evaluate if these embeddings reflect the postulated topology of the Skill Space by predicting what new APIs/projects developers use/join, and whether or not their pull requests get accepted. We also check how the developers' representations in the Skill Space align with their self-reported API expertise. Result: Our results suggest that the proposed embeddings in the Skill Space appear to satisfy the postulated topology and we hope that such representations may aid in the construction of signals that increase trust (and efficiency) of open source ecosystems at large and may aid investigations of other phenomena related to developer proficiency and learning.
Adaptive spatial pixel-level feature fusion network for multispectral pedestrian detection

A pedestrian detector that uses visible and thermal infrared image pairs as the input has better detection performance than a detector that uses only visible image under challenging illumination conditions. With the aim to efficiently and effectively fuse complementary information from visible and thermal infrared images, this paper proposes an adaptive spatial pixel-level feature fusion network called the ASPFF Net, which can adaptively extract spatial pixel-level features from visible and infrared images for fusion. Specifically, first, two light networks with different weights are used to extract multi-scale features of visible and infrared images. Next, for features of the same scale but different modalities, the fusion weights of different spatial positions and pixels in the two feature maps are obtained by the spatial attention module (SAM) and pixel attention module (PAM). The original features of visible and infrared images are recalibrated by the fusion weights, and multi-scale fused feature layers are obtained. Finally, different scales of pedestrians are detected on the fused multi-scale feature layers. Compared with the other recent multispectral pedestrian detectors on the reasonable subset of the KAIST multispectral pedestrian detection dataset, the proposed detector is attractive in balancing speed and accuracy. The extensive experiments on the KAIST dataset demonstrate the effectiveness of the proposed method for the fusion of visible and infrared image in multispectral pedestrian detection.