Reviewerly/exHarmony huggingface dataset:
Code Difference Guided Adversarial Example Generation for Deep Code Models

Adversarial examples are important to test and enhance the robustness of deep code models. As source code is discrete and has to strictly stick to complex grammar and semantics constraints, the adversarial example generation techniques in other domains are hardly applicable. Moreover, the adversarial example generation techniques specific to deep code models still suffer from unsatisfactory effectiveness due to the enormous ingredient search space. In this work, we propose a novel adversarial example generation technique (i.e., CODA) for testing deep code models. Its key idea is to use code differences between the target input (i.e., a given code snippet as the model input) and reference inputs (i.e., the inputs that have small code differences but different prediction results with the target input) to guide the generation of adversarial examples. It considers both structure differences and identifier differences to preserve the original semantics. Hence, the ingredient search space can be largely reduced as the one constituted by the two kinds of code differences, and thus the testing process can be improved by designing and guiding corresponding equivalent structure transformations and identifier renaming transformations. Our experiments on 15 deep code models demonstrate the effective-ness and efficiency of CODA, the naturalness of its generated examples, and its capability of enhancing model robustness after adversarial fine-tuning. For example, CODA reveals 88.05 % and 72.51 % more faults in models than the state-of-the-art techniques (i.e., CARROT and ALERT) on average, respectively.
Neural SZZ Algorithm

The SZZ algorithm has been widely used for identifying bug-inducing commits. However, it suffers from low precision, as not all deletion lines in the bug-fixing commit are related to the bug fix. Previous studies have attempted to address this issue by using static methods to filter out noise, e.g., comments and refactoring operations in the bug-fixing commit. However, these methods have two limitations. First, it is challenging to include all refactoring and non-essential change patterns in a tool, leading to the potential exclusion of relevant lines and the inclusion of irrelevant lines. Second, applying these tools might not always improve performance. In this paper, to address the aforementioned challenges, we propose NEURALSZZ, a deep learning approach for detecting the root cause deletion lines in a bug-fixing commit and using them as input for the SZZ algorithm. NEURALSZZ first constructs a heterogeneous graph attention network model that captures the semantic relationships between each deletion line and the other deletion and addition lines. To pinpoint the root cause of a bug, Neuralszz uses a learning-to-rank technique to rank all deletion lines in the commit. To evaluate the effectiveness of NEURALSZZ, we utilize three datasets containing high-quality bug-fixing and bug-inducing commits. The experiment results show that NEURALSZZ outperforms various baseline methods, e.g., traditional machine learning-based approaches and BI-LSTM in identifying the root cause of bugs. Moreover, by utilizing the top-ranked deletion lines and applying the SZZ algorithm, Neuralszz demonstrates better precision and F1-score compared to previous SZZ algorithms.
Challenges with Passwordless FIDO2 in an Enterprise Setting: A Usability Study

Fast Identity Online 2 (FIDO2), a modern authentication protocol, is gaining popularity as a default strong authentication mechanism. It has been recognized as a leading candidate to overcome limitations (e.g., it is phishing resistant) of existing authentication solutions. However, the task of deprecating weak methods such as password-based authentication is not trivial and requires a comprehensive approach. While security, privacy, and end-user usability of FIDO2 have been addressed in both academic and industry literature, the difficulties associated with its integration with production environments, such as solution completeness or edge-case support, have received little attention. In particular, complex environments such as enterprise identity management pose unique challenges for any authentication system. In this paper, we identify challenging enterprise identity lifecycle use cases (e.g., remote workforce and legacy systems) by conducting a usability study, in which 118 professionals shared their perception of challenges to FIDO2 integration from their hands-on field experience. Our analysis of the user study results revealed serious gaps such as account recovery (selected by over 60% of our respondents), and identify priority development areas for the FIDO2 community.
Friend or Foe Inside? Exploring In-Process Isolation to Maintain Memory Safety for Unsafe Rust

Rust is a popular memory-safe systems programming language. In order to interact with hardware or call into non-Rust libraries, Rust provides unsafe language features that shift responsibility for ensuring memory safety to the developer. Failing to do so, may lead to memory-safety violations in unsafe code which can violate safety of the entire application. In this work we explore in-process isolation with Memory Protection Keys as a mechanism to shield safe program sections from safety violations that may happen in unsafe sections. Our approach is easy to use and comprehensive as it prevents heap and stack-based violations. We further compare process-based and in-process isolation mechanisms and the necessary requirements for data serialization, communication, and context switching. Our results show that in-process isolation can be effective and efficient, permits for a high degree of automation, and also enables a notion of application rewinding where the safe program section may detect and safely handle violations in unsafe code.
MELT: Mining Effective Lightweight Transformations from Pull Requests

Software developers often struggle to update APIs, leading to manual, time-consuming, and error-prone processes. We introduce Melt, a new approach that generates lightweight API migration rules directly from pull requests in popular library repositories. Our key insight is that pull requests merged into open-source libraries are a rich source of information sufficient to mine API migration rules. By leveraging code examples mined from the library source and automatically generated code examples based on the pull requests, we infer transformation rules in Comby, a language for structural code search and replace. Since inferred rules from single code examples may be too specific, we propose a generalization procedure to make the rules more applicable to client projects. Melt rules are syntax-driven, interpretable, and easily adaptable. Moreover, unlike previous work, our approach enables rule inference to seamlessly integrate into the library workflow, removing the need to wait for client code migrations. We evaluated Melt on pull requests from four popular libraries, successfully mining 461 migration rules from code examples in pull requests and 114 rules from auto-generated code examples. Our generalization procedure increases the number of matches for mined rules by 9×. We applied these rules to client projects and ran their tests, which led to an overall decrease in the number of warnings and fixing some test cases demonstrating MELT's effectiveness in real-world scenarios.
Characterizing Static Analysis Alerts for Terraform Manifests: An Experience Report

While Terraform has gained popularity to implement the practice of infrastructure as code (IaC), there is a lack of characterization of static analysis for Terraform manifests. Such lack of characterization hinders practitioners to assess how to use static analysis for their Terraform development process, as it happened for Company A, an organization who uses Terraform to create automated software deployment pipelines. In this experience report, we have investigated 491 static analysis alerts that occur for 10 open source and one proprietary Terraform repositories. From our analysis we observe: (i) 10 categories of static analysis alerts to appear for Terraform manifests, of which five are related to security, (ii) Terraform resources with dependencies to have more static analysis alerts than that of resources with no dependencies, and (iii) practitioner perceptions to vary from one alert category to another while deciding on taking actions for reported alerts. We conclude our paper by providing a list of lessons for practitioners and toolsmiths on how to improve static analysis for Terraform manifests.
CodeGen4Libs: A Two-Stage Approach for Library-Oriented Code Generation

Automated code generation has been extensively studied in recent literature. In this work, we first survey 66 participants to motivate a more pragmatic code generation scenario, i.e., library-oriented code generation, where the generated code should implement the functionally of the natural language query with the given library. We then revisit existing learning-based code generation techniques and find they have limited effectiveness in such a library-oriented code generation scenario. To address this limitation, we propose a novel library-oriented code generation technique, CodeGen4Libs, which incorporates two stages: import generation and code generation. The import generation stage generates import statements for the natural language query with the given third-party libraries, while the code generation stage generates concrete code based on the generated imports and the query. To evaluate the effectiveness of our approach, we conduct extensive experiments on a dataset of 403,780 data items. Our results demonstrate that CodeGen4Libs outperforms baseline models in both import generation and code generation stages, achieving improvements of up to 97.4% on EM (Exact Match), 54.5% on BLEU, and 53.5% on Hit@All. Overall, our proposed CodeGen4Libs approach shows promising results in generating high-quality code with specific third-party libraries, which can improve the efficiency and effectiveness of software development.
ART4FL: An Agent-Based Architectural Approach for Trustworthy Federated Learning in the IoT

The integration of the Internet of Things (IoT) and Machine Learning (ML) technologies has opened up for the development of novel types of systems and services. Federated Learning (FL) has enabled the systems to collaboratively train their ML models while preserving the privacy of the data collected by their IoT devices and objects. Several FL frameworks have been developed, however, they do not enable FL in open, distributed, and heterogeneous IoT environments. Specifically, they do not support systems that collect similar data to dynamically discover each other, communicate, and negotiate about the training terms (e.g., accuracy, communication latency, and cost). Towards bridging this gap, we propose ART4FL, an end-to-end framework that enables FL in open IoT settings. The framework enables systems' users to configure agents that participate in FL on their behalf. Those agents negotiate and make commitments (i.e., contractual agreements) to dynamically form federations. To perform FL, the framework deploys the needed services dynamically, monitors the training rounds, and calculates agents' trust scores based on the established commitments. ART4FL exploits a blockchain network to maintain the trust scores, and it provides those scores to negotiating agents' during the federations' formation phase.
Beyond the Code: Investigating the Effects of Pull Request Conversations on Design Decay

Background: Code development is done collaboratively in platforms such as GitHub and GitLab, following a pull-based development model. In this model, developers actively communicate and share their knowledge through conversations. Pull request conversations are affected by social aspects such as communication dynamics among developers, discussion content, and organizational dynamics. Despite prior studies indicating that social aspects indeed impact software quality, it is still unknown to what extent social aspects influence design decay during software development. Thus, since social aspects are intertwined with design and implementation decisions, there is a need for investigating how social aspects contribute to avoiding, reducing, or accelerating design decay. Aims: To fill this gap, we performed a study aimed at investigating the effects of pull request conversation on design decay. Method: We investigated 10,746 pull request conversations from 11 open-source systems, characterizing in terms of three different social aspects: discussion content, organizational and communication dynamics. We considered 18 social metrics to these three social aspects, and analyzed how they associate with design decay. We used a statistical approach to assess which social metrics are able to discriminate between impactful and unimpactful pull requests. Then, we employed a multiple logistic regression model to evaluate the influence of each social metric per social aspect in the presence of each other on design decay. Finally, we also observed how the combination of all social metrics influences the design decay. Results: Our findings reveal that social metrics related to the size and duration of a discussion, the presence of design-related keywords, the team size, and gender diversity can be used to discriminate between design impactful and unimpactful pull requests. Organizational growth and gender diversity prevent decay. Each software community has its unique aspects that can be used to detect and prevent design decay. Also, design improvements can be accomplished by timely feedback, engaged communication, and design-oriented discussions with the contribution of multiple participants who provide significant comments. Conclusion: The social aspects related to pull request conversations are useful indicators of design decay.
Personalized First Issue Recommender for Newcomers in Open Source Projects

Many open source projects provide good first issues (GFIs) to attract and retain newcomers. Although several automated GFI recommenders have been proposed, existing recommenders are limited to recommending generic GFIs without considering differences between individual newcomers. However, we observe mismatches between generic GFIs and the diverse background of newcomers, resulting in failed attempts, discouraged onboarding, and delayed issue resolution. To address this problem, we assume that personalized first issues (PFIs) for newcomers could help reduce the mismatches. To justify the assumption, we empirically analyze 37 newcomers and their first issues resolved across multiple projects. We find that the first issues resolved by the same newcomer share similarities in task type, programming language, and project domain. These findings underscore the need for a PFI recommender to improve over state-of-the-art approaches. For that purpose, we identify features that influence newcomers' personalized selection of first issues by analyzing the relationship between possible features of the newcomers and the characteristics of the newcomers' chosen first issues. We find that the expertise preference, OSS experience, activeness, and sentiment of newcomers drive their personalized choice of the first issues. Based on these findings, we propose a Personalized First Issue Recommender (PFIRec), which employs LamdaMART to rank candidate issues for a given newcomer by leveraging the identified influential features. We evaluate PFIRec using a dataset of 68,858 issues from 100 GitHub projects. The evaluation results show that PFIRec outperforms existing first issue recommenders, potentially doubling the probability that the top recommended issue is suitable for a specific newcomer and reducing one-third of a newcomer's unsuccessful attempts to identify suitable first issues, in the median. We provide a replication package at https://zenodo.org/record/7915841.
What Makes Good In-Context Demonstrations for Code Intelligence Tasks with LLMs?

Pre-trained models of source code have gained widespread popularity in many code intelligence tasks.Recently, with the scaling of the model and corpus size, large language models have shown the ability of in-context learning (ICL).ICL employs task instructions and a few examples as demonstrations, and then inputs the demonstrations to the language models for making predictions.This new learning paradigm is training-free and has shown impressive performance in various natural language processing and code intelligence tasks.However, the performance of ICL heavily relies on the quality of demonstrations, e.g., the selected examples.It is important to systematically investigate how to construct a good demonstration for code-related tasks.In this paper, we empirically explore the impact of three key factors on the performance of ICL in code intelligence tasks: the selection, order, and number of demonstration examples.We conduct extensive experiments on three code intelligence tasks including code summarization, bug fixing, and program synthesis.Our experimental results demonstrate that all the above three factors dramatically impact the performance of ICL in code intelligence tasks.Additionally, we summarize our findings and provide takeaway suggestions on how to construct effective demonstrations, taking into account these three perspectives.We also show that a carefully-designed demonstration based on our findings can lead to substantial improvements over widely-used demonstration construction methods, e.g., improving BLEU-4, EM, and EM by at least 9.90%, 175.96%, and 50.81% on code summarization, bug fixing, and program synthesis, respectively.
Automated Software Entity Matching Between Successive Versions

Version control systems are widely used to manage the evolution of software applications. However, such version control systems take source code as lines of plain text, and thus they cannot present the evolution of software entities embedded in the source code. To this end, a few approaches have been proposed to match software entities before and after a given commit, known as software entity matching algorithms. However, the accuracy of such algorithms requires further improvement. In this paper, we propose an automated iterative algorithm (called ReMapper) to match software entities between two successive versions. The key insight of ReMapper is that the qualified name, the implementation, and the references of a software entity together can distinguish it from others. It matches software entities iteratively because the mapping depends on the reference-based similarity whereas the reference-based similarity depends on the mapping of entities as well. We evaluated ReMapper on a benchmark consisting of 215 commits from 21 real-world projects. Our evaluation results suggest that ReMapper substantially outperformed the state of the art, reducing the number of mistakes (false positives plus false negatives) substantially by 85.8%. We also evaluated to what extent it may improve the automated refactoring discovery (mining) that relies heavily on automated entity matching. Our evaluation results suggest that it substantially improved the state of the art in refactoring discovery, improving recall by 6.9% and reducing the number of false positives by 72.6%.
To Share, or Not to Share: Exploring Test-Case Reusability in Fork Ecosystems

Code is often reused to facilitate collaborative development, to create software variants, to experiment with new ideas, or to develop new features in isolation. Social-coding platforms, such as GitHub, enable enhanced code reuse with forking, pull requests, and cross-project traceability. With these concepts, forking has become a common strategy to reuse code by creating clones (i.e., forks) of projects. Thereby, forking establishes fork ecosystems of co-existing projects that are similar, but developed in parallel, often with rather sporadic code propagation and synchronization. Consequently, forked projects vary in quality and often involve redundant development efforts. Unfortunately, as we will show, many projects do not benefit from test cases created in other forks, even though those test cases could actually be reused to enhance the quality of other projects. We believe that reusing test cases—in addition to the implementation code—can improve software quality, software maintainability, and coding efficiency in fork ecosystems. While researchers have worked on test-case-reuse techniques, their potential to improve the quality of real fork ecosystems is unknown. To shed light on test-case reusability, we study to what extent test cases can be reused across forked projects. We mined a dataset of test cases from 305 fork ecosystems on GitHub—totaling 1,089 projects—and assessed the potential for reusing these test cases among the forked projects. By performing a manual inspection of the test cases' applicability, by transplanting the test cases, and by analyzing the causes of non-applicability, we contribute an understanding of the benefits (e.g., uncovering bugs) and of the challenges (e.g., automated code transplantation, deciding about applicability) of reusing test cases in fork ecosystems.
Can shallow quantum circuits scramble local noise into global white noise?

Abstract Shallow quantum circuits are believed to be the most promising candidates for achieving early practical quantum advantage—this has motivated the development of a broad range of error mitigation techniques whose performance generally improves when the quantum state is well approximated by a global depolarising (white) noise model. While it has been crucial for demonstrating quantum supremacy that random circuits scramble local noise into global white noise—a property that has been proved rigorously—we investigate to what degree practical shallow quantum circuits scramble local noise into global white noise. We define two key metrics as (a) density matrix eigenvalue uniformity and (b) commutator norm that quantifies stability of the dominant eigenvector. While the former determines the distance from white noise, the latter determines the performance of purification based error mitigation. We derive analytical approximate bounds on their scaling and find in most cases they nicely match numerical results. On the other hand, we simulate a broad class of practical quantum circuits and find that white noise is in certain cases a bad approximation posing significant limitations on the performance of some of the simpler error mitigation schemes. On a positive note, we find in all cases that the commutator norm is sufficiently small guaranteeing a very good performance of purification-based error mitigation. Lastly, we identify techniques that may decrease both metrics, such as increasing the dimensionality of the dynamical Lie algebra by gate insertions or randomised compiling.