Reviewerly/exHarmony huggingface dataset:
Towards human-centric software complexity metrics: a neuroscience-based approach

Current code complexity metrics used in the software industry do not consider the human factor from an individual perspective (i.e., the programmer, or tester). In software development, the developers play a critical and fundamental role to accomplish each development activity, and the limitations induced by them are inevitable. Therefore, the ability to capture perceived code complexity accurately is very important for the software quality and its adequate maintenance. This Ph.D. thesis focuses on exploring the possible approaches to incorporating a human-centric dimension in software complexity assessment and management. To reach this purpose, we use the information captured from wearable and non-intrusive devices (e.g., ECG and Eye-tracking with Pupillography) to identify the relevant cognitive states (such as mental effort, stress and mental fatigue) of individual developers during the software development process, and use such data to develop new complexity metrics that consider the genuine human perception of the challenges in comprehending code. Expected new human-centric code complexity metrics must consider the developers’ perspective and provide an accurate assessment of software complexity to improve the software’s quality and reduce bugs, time consumed, and unnecessary costs.
Multi-Objective White-Box Test Input Selection for Deep Neural Network Model Enhancement

To reveal incorrect behaviors and improve the quality of DNN models through testing, a commonly used approach is to collect massive test inputs and manually label them for model optimization. However, it is labor-intensive and time-consuming to manually label the whole collected test inputs. To reduce the labeling cost, recent studies propose test input selection approaches to select a subset of test inputs for model retraining, thereby improving the performance of DNN models. Existing approaches primarily rely on output probabilities to select error-revealing test inputs. Nevertheless, such approaches select test inputs without the knowledge of underlying mechanisms that lead to incorrect decisions. Furthermore, they fail to consider the diversity of test inputs, limiting the capability of DNN models to learn more diverse features. To address these limitations, this paper proposes MOON, a white-box test input selection approach based on multi-objective optimization. The neuron spectrum is proposed to localize suspicious neurons that contribute to erroneous decisions made by DNN models. MOON formulates the test input selection method into a search-based testing problem. By tailoring a multi-objective optimization algorithm, it guides the search process towards maximizing the outputs of suspicious neurons while promoting diversity in neuron behaviors. An empirical evaluation on three datasets and six DNN models was conducted. The experimental results demonstrate the effectiveness of MOON in localizing suspicious neurons. In addition, MOON significantly outperforms state-of-the-art test selection approaches, with an accuracy improvement rate of up to 236.1%.
Why Do Deep Learning Projects Differ in Compatible Framework Versions? An Exploratory Study

Deep learning (DL) is becoming increasingly important and widely used in our society. DL projects are mainly built upon DL frameworks, which frequently evolve due to the introduction of new features or bug fixing. Consequently, compatibility issues are commonly seen in DL projects. The compatible framework versions may differ across DL projects, i.e., for a specific framework version, one project runs normally while the other crashes, even if the client code uses the same framework API. Existing studies mainly focus on analyzing the API evolution of Python libraries and the related compatibility issues. However, the difference in framework version compatibility (DFVC) among DL projects has rarely been systematically studied. In this paper, we conduct an empirical study on 90 PyTorch and 50 TensorFlow projects collected from GitHub. By upgrading and downgrading the framework versions, we obtain compatible versions for each project and further investigate the root causes of the different compatible framework versions across projects. We summarize seven root causes: Python version, absence of using the same breaking API, import path, parameter, third-party library, resource, and API usage constraint. We further present six implications based on our empirical findings. Our study can facilitate DL practitioners to gain a better understanding of the DFVC among DL projects.
Reviewing and Proposing Approaches for Self-adaptive Testing in the Field

Software systems are an essential part of our everyday life, and over time, they are becoming larger and more complex. These characteristics make systems even more susceptible to faults. For this reason, there is an increasing need to prolong testing beyond software release, even while the system is operating in the field. This is called testing in the field. One strategy to handle the challenges associated with testing in the field, such as uncertainty, is to employ self-adaptive testing. The self-adaptation of the testing process guarantees that the system is functioning properly, even after the performance of its own adaptation. In this research proposal, we aim at proposing a novel approach for self-adaptive testing in the field. We first review the literature and propose a definition, a taxonomy, and a reference architecture for this type of approach. We also highlight the main challenges faced by these approaches. Finally, we also propose a strategy that combines Combinatorial Testing and Markov Chains to perform Self-Adaptive Testing in the Field.
SAPPX: Securing COTS Binaries with Automatic Program Partitioning for Intel SGX

In the era of cloud computing, many applications are migrated to public servers not fully controlled by users who may fear their critical operations or data from being compromised by attackers. Previous studies have shown that Intel SGX enclaves can improve applications’ security in many market products. Yet they mainly rely on developers to reprogram and recompile the application into an SGX-aware version. To address this problem, we propose SAPPX, an SGX-based program retrofitting method that can automatically partition COTS application binaries into two parts without breaking the original program semantics. The first part of the application runs in user space, while the second part is executed in an SGX enclave to protect the user’s sensitive information. We have implemented a prototype of SAPPX on x86/Linux platforms and evaluated its performance using real-world applications and SPECCPU 2017 benchmarks. The experimental results show that the average overhead of the proposed approach is up to 19%.
A Bayesian Network-Based Analysis of Changing Impact of Organizational Goals on Software Development

Changes in organizational goals in companies affect various metric values related to software development; however, few methods are available to evaluate the impact of such changes. Therefore, we propose a method to divide the collected metric values before and after a change in organizational goals, construct a Bayesian network before and after the change to analyze the causal relationship between the metric values, and compare and evaluate the results before and after the change. In this study, we evaluate changes from 2016 to 2020 in organizational goals and the Bayesian network for person-hours, number of bugs, etc. in projects related to the development of cloud services that support the real estate industry, using data provided by e-Seikatsu Co., Ltd.
Evaluating Ethereum Reentrancy Detection Tools via Mutation Testing

Reentrancy vulnerabilities in Ethereum smart contracts have caused huge financial losses in recent years, promoting the development of reentrancy detection tools. Evaluation of reentrancy detection tools has emerged as an essential research focus consequently. However, despite numerous reentrancy detection evaluation approaches, there is no systematically classified set of reentrancy vulnerabilities in evaluation. This oversight may result in a limited understanding of the actual effectiveness of reentrancy detection tools. This paper employs mutation testing to generate classified sets of reentrancy vulnerabilities and evaluate reentrancy detection tools. To determine the types of reentrancy that can be mutated, we use symbolic execution to identify potential reentrancy paths in smart contracts. For each reentrancy type, we design one to three mutation operators. Then, we employ pattern matching to locate statements where mutation operators can be applied, thus generating sets of reentrancy vulnerabilities of different types. We implemented and compared our mutation testing tool with other tools. Results show that we generated at least 29% more classified reentrancy mutants. Furthermore, we evaluated two state-of-the-art reentrancy detection tools based on the classified reentrancy mutants. The findings provide directions for advancing reentrancy detection tools.
AutoKAD: Empowering KPI Anomaly Detection with Label-Free Deployment

Monitoring Key Performance Indicators (KPIs) and detecting anomalies in online service systems is critical. However, choosing the right KPI anomaly detection algorithm and appropriate hyperparameters presents a challenge. Conventional Automated Machine Learning (AutoML) struggles to address this because the hold-out dataset lacks labels and its loss doesn’t reliably reflect anomaly detection accuracy. To address the above challenges, this paper introduces AutoKAD, an AutoML framework designed to solve the combined algorithm selection and hyperparameter optimization problem for unsupervised KPI Anomaly Detection. We propose a label-free universal objective function, inspired by the Local Outlier Factor (LOF), for evaluating AutoML trials. Additionally, we improve the acquisition function and designs a cluster-based warm start strategy to enhance exploration effectiveness and efficiency. The experimental results on three real-world datasets show that our approach outperforms the SOTA model selection algorithm by 11% in F1-score and achieves comparable performance (99%) with theoretically optimal results. We believe that AutoKAD can greatly improve the deployment feasibility of existing anomaly detection algorithms in real-world systems. Our code is anonymously released at https://github.com/NetManAIOps/AutoKAD.
Cross-Project Aging-Related Bug Prediction Based on Feature Transfer and Class Imbalance Learning

Software aging refers to system performance degradation and failure due to Aging-Related Bugs (ARBs) in long-running software systems. ARB prediction helps identify ARBs to prevent software aging. However, early project stages often lack ARB data for model training. To address this, cross-project ARB prediction (CPAP) is proposed, where a model is trained using labeled source projects to predict the ARB present in the target project. In CPAP, distribution discrepancy and class imbalance pose challenges, impacting model performance. In this paper, a hybrid CPAP combines feature transfer learning with class imbalance learning is proposed to solve the above problems, which we named KDK. In the feature transfer learning stage, KDK leverages Kernel Principal Component Analysis (KPCA) in conjunction with Double Marginalized Denoising Autoencoders (DMDA) to acquire feature representations that reduce distribution disparities among projects. In the class imbalance learning stage, K-means Clustering Cleaning Ensemble (KCE) is employed to rebalance and clean the source project dataset, who has a severe class imbalance and class overlap. Experiments are conducted on three open-source projects with two performance metrics (AUC and Balance). Experiment results demonstrate that our approach enhances Balance values by 9.0%, 32.9%, 4.4%, and 3.9% in comparison to state-of-the-art CPAP methods, namely TLAP, JPKS, SRLA, and JDA-ISDA, respectively.
Lark: Verified Cross-Domain Access Control for Trusted Execution Environments

Trusted Execution Environments (TEEs) play a crucial role in embedded systems, IoT, and cloud computing. However, their security issues are a major concern, particularly related to defects or improper implementations in access control mechanisms. Such issues can result in severe problems like privilege escalation and unintended memory accesses during inter-domain communication. Moreover, employing mathematical methods for rigorous security guarantees is essential.To address these challenges, we propose Lark, a cross-domain access control for TEEs, which is modeled and verified in Isabelle/HOL. Lark applies orthogonal access control attributes on memory to decouple access permissions of different privilege levels. Additionally, it enforces strict access permission checks for inter-domain communications. For a strict security guarantee, Lark is formalized and verified in Isabelle/HOL, with 84 definitions and 35 lemmas containing ∼1,600 lines of code. The machine-checkable proofs demonstrate that Lark ensures memory isolation and information flow security. We identify and resolve an inter-domain communication issue within an open-source TEE, and develop a prototype that implements the access control features of Lark. Exhaustive evaluations on real-world applications demonstrate that Lark introduces less than 5% performance overhead.
Evaluating the Effect of Common Annotation Faults on Object Detection Techniques

Machine learning (ML) is applied in many safety-critical domains such as autonomous driving and medical diagnosis. Many ML applications in such domains require object detection, which includes both classification and localization, to provide additional context. To ensure high accuracy, state-of-the-art object detection (OD) systems require large quantities of correctly annotated images for training. However, creating such datasets is non-trivial, may involve significant human effort, and is hence inevitably prone to annotation faults. We evaluate the effect of such faults on OD applications. We present ODFI, which can inject five different types of common annotation faults into any COCO-formatted dataset. We then use ODFI to inject these faults into two road traffic and one medical X-ray imaging datasets. Finally, using these faulty datasets, we systematically evaluate and compare the efficacy of existing OD techniques that are designed to be robust against such faults. To do so, we introduce a new metric that evaluates the robustness of OD models in the presence of faults. We find that (1) single-stage detectors trained with faulty annotations perform better in scenes with more objects, (2) redundant bounding boxes have the least impact on robustness, and (3) ensembles have the highest overall robustness among the robust OD techniques considered.
Building Resilient ML Applications using Ensembles against Faulty Training Data

Supervised machine learning (ML) applications are data-driven, and thus require large amounts of data for training. Whether the training data is manually collected or automatically generated, it is prone to faults like mislabelling, accidental deletion, or repetition. Our research focuses on developing the most effective and efficient techniques with minimal human effort to improve the resilience of ML applications against faulty training data. We find that ensembles have a higher resilience to faulty training data than individual models, especially when using ensembles with architecturally diverse constituent models. We also find that ensembles are more effective than many existing techniques against mislabelled training data, even in safety-critical domains such as autonomous vehicles and healthcare. Hence, we aim to improve the applicability of ensembles in real-world systems by (1) optimizing the process of searching for resilient ensembles, and (2) reducing the training cost by maximizing reuse of trained weights and (3) examining methods to make ensembles more explainable to stakeholders.
Evaluating and Enhancing the Robustness of Federated Learning System against Realistic Data Corruption

Federated learning (FL) has emerged as a prominent paradigm enabling collaborative model training without transmitting local data, thereby safeguarding data privacy. However, the practical implementation of FL systems on these devices faces a significant challenge: the heterogeneous corruption of data on individual clients, leading to unanticipated accuracy degradation during real-world deployment. In this work, we first introduce a realistic data corruption simulation framework to test the robustness of FL systems. In this framework, an in-depth analysis of potential data corruption patterns occurring on devices is conducted, followed by the construction of individual datasets with varying corruption types and degrees. Such data corruption results in the robustness degradation of conventional FL protocol (FedAVG) significantly higher than centralized learning (CL). Atop this key observation, we propose an adaptive FL protocol that emulates the CL training process. The protocol leverages imbalanced client data sampling to mitigate the negative impact of data corruption. Furthermore, a hybrid aggregation strategy is designed to accelerate model convergence and reduce additional communication overhead. Extensive experiments validate the effectiveness of our approach in enhancing the robustness of FL systems against client data corruption, which achieves up to 12% higher converge accuracy than FedAVG-based systems with acceptable overhead.
Characterizing Reliability of Three-version Traffic Sign Classifier System through Diversity Metrics

The N-version machine learning (ML) system is an architecture approach to enhance the reliability of ML system outputs by exploiting ML model diversity and input data diversity. While existing studies theoretically show the relation between diversity metrics and system reliability, there is a shortage of empirical studies validating reliability models with diversity parameters in real datasets. In this paper, focusing on traffic sign recognition tasks, we empirically analyze the impact of diversity parameter estimations for predicting the reliability of three-version traffic sign classifier systems. Using five real-world traffic sign datasets, we confirm that the three-version architecture effectively enhances system reliability by applying diverse models and diversified input images. Then, we estimate the diversity parameters and apply them to variants of reliability prediction models. The prediction residuals between the observed reliability and the predicted reliability are mostly less than 0.017 across all data sets, which is half of the residual achieved by the conventional prediction model, except for the architecture of a single model with triple input. As the estimated values of diversity parameters tend to be stable with a relatively small number of samples, we consider that the reliability prediction models using diversity parameters are useful in the early-stage design of ML systems.
A Survey of Metrics to Enhance Training Dependability in Large Language Models

The rapidly advancing field of artificial intelligence requires meticulous attention to the training and monitoring of large language models (LLMs). This paper offers a systematic analysis of existing metrics and introduces new ones, focusing on their theoretical underpinnings and practical implementations. We present empirical results and insights into the performance of selected metrics, elucidating the complex interplay of variables in the training process. Our comprehensive approach provides significant insights into LLM training, and promises to improve the dependability and efficiency of future models.
Automatic Static Vulnerability Detection for Machine Learning Libraries: Are We There Yet?

Automatic detection of software security vulnerabilities is critical in software quality assurance. Many static analysis tools that can help detect security vulnerabilities have been proposed. While these static analysis tools are mainly evaluated on general software projects call into question their practical effectiveness and usefulness for Machine Learning (ML) libraries. In this paper, we address this question by analyzing five popular and widely used static analysis tools, i.e., Flawfinder, RATS, Cppcheck, Facebook Infer, and Clang static analyzer, on a curated dataset of software security vulnerabilities gathered from four popular ML libraries, including Mlpack, MXNet, PyTorch, and TensorFlow, with a total of 410 known vulnerabilities. Our research categorizes these tools’ capabilities to understand better the strengths and weaknesses of the tools for detecting software security vulnerabilities in ML libraries. Overall, our study shows that static analysis tools find a negligible amount of all security vulnerabilities accounting for 5/410 unique vulnerabilities (0.01%), Flawfinder and RATS are the most effective static checkers for finding software security vulnerabilities in ML libraries. We further identify and discuss opportunities to make the tools more effective and practical based on our observations.
On the Vulnerabilities of Text-to-SQL Models

Although it has been demonstrated that Natural Language Processing (NLP) algorithms are vulnerable to deliberate attacks, the question of whether such weaknesses can lead to software security threats is under-explored. To bridge this gap, we conducted vulnerability tests on Text-to-SQL systems that are commonly used to create natural language interfaces to databases. We showed that the Text-to-SQL modules within six commercial applications can be manipulated to produce malicious code, potentially leading to data breaches and Denial of Service attacks. <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> This is the first demonstration that NLP models can be exploited as attack vectors in the wild. In addition, experiments using four open-source language models verified that straightforward backdoor attacks on Text-to-SQL systems achieve a 100% success rate without affecting their performance. The aim of this work is to draw the community’s attention to potential software security issues associated with NLP algorithms and encourage exploration of methods to mitigate against them.
FLAGS Simulation Framework for Federated Learning Algorithms

Federated Learning (FL) provides an effective mechanism for distributed learning. However, it is expected to operate in a highly diverse setting with distinct behaviors from the participating nodes as well as dynamic network conditions. The FL performance, therefore, is subject to change due to the highly transitory nature of the overall system. An efficient simulation framework must be flexible to allow a range of participant behaviors, interactions, and environment characteristics. In this demo paper, we present the Federated Learning Algorithm Simulation (FLAGS) framework that we propose as a lightweight FL implementation and testing platform. FLAGS framework allows for a wide range of device behaviors and cooperative mechanisms, enabling rapid testing of multiple FL algorithms.