Reviewerly/exHarmony huggingface dataset:
RocketHA: A High Availability Design Paradigm for Distributed Log-Based Storage System

As a team from Alibaba Cloud, we have developed and open-sourced RocketMQ, a cloud-native “messaging, eventing, streaming” real-time data processing platform that covers cloud-edge-device collaboration scenarios. During the development of RocketMQ, we also formulated RocketHA, a log-based storage high availability design theory that provides a robust solution for distributed log storage software used in industrial applications. RocketHA comprises six fundamental components that enable automatic cluster recovery from failures such as crashes and partitions. This design paradigm has been successfully implemented in the open-source RocketMQ. Our evaluation demonstrates that RocketHA ensures high availability, fast recovery, high throughput, and data loss prevention. We hope that RocketHA will inspire and guide the development of high-availability solutions for all log-based storage systems.
On Automated Assistants for Software Development: The Role of LLMs

Software developers handle many complex tasks that include gathering and applying domain knowledge, coordinating subtasks, designing interfaces, turning ideas into elegant code, and more. They must switch contexts between these tasks, incurring more cognitive costs. Recent advances in large language models (LLMs) open up new possibilities for moving beyond the support provided by automated assistants (AAs) available today. In this paper, we explore if a human memory model can provide a framework for the systematic investigation of AAs for software development based on LLMs and other new technologies.
Improving Code Extraction from Coding Screencasts Using a Code-Aware Encoder-Decoder Model

Accurate automatic code extraction from tutorial videos is crucial for software developers seeking to reuse the code contained in these videos. Current methods using optical character recognition (OCR) often yield inaccurate results due to code complexity and variations in screencast formats. To address this issue, we introduce CodeT5-OCRfix, an approach that leverages the pre-trained code-aware large language model CodeT5 to enhance code extraction accuracy by post-processing OCRed code. We first collect a large and diverse dataset of source code screenshots captured from more than 10K Java projects from GitHub. We then apply the most widely used OCR engine for the task of code extraction from videos, Tesseract, on these screenshots and collect the OCRed code along with the ground truth code extracted from the Java files. We built a training dataset of more than 585K pairs of OCRed and ground truth code pairs, which we then used to fine-tune CodeT5, obtaining our model CodeT5-OCRfix. An empirical evaluation on both screenshots and screencast frames shows that CodeT5-OCRfix outperforms baseline code extraction models and is also more time-efficient. Our approach therefore improves the state-of-the-art in code extraction techniques from screencasts and images.
Automatic Generation and Reuse of Precise Library Summaries for Object-Sensitive Pointer Analysis

The extensive use of libraries in modern software impedes the scalability of pointer analysis. To address this issue, library summarization can be beneficial, but only if the resulting summary-based pointer analysis is faster without sacrificing much precision in the application code. However, currently, no library summarization approaches exist that meet this design objective. This paper presents a novel approach that solves this problem by using k-object-sensitive pointer analysis, k-obj, for Java. The approach involves applying k-obj, along with a set of summary-based inference rules, to generate a k-object-sensitive library summary. By replacing the program's library with this summary and applying k-obj, the efficiency of the program can be significantly improved while maintaining nearly the same or better precision in the application code. We validate our approach with an implementation in Soot and an evaluation using representative Java programs.
Tutorial: The End of Binary Protocol Parser Vulnerabilities : Using RecordFlux and SPARK to implement formally-verified binary formats and communication protocols.

Implementing binary formats and communication protocols correctly is hard. Consequently, security vulnerabilities in such implementations are frequent and can have catastrophic consequences. In this tutorial, we present the RecordFlux toolset and language to practitioners faced with the implementation of critical protocol stacks. They will use the RecordFlux domain-specific language (DSL) to formally specify their own version of a real-world protocol and then verify the correctness of the generated code using the SPARK toolset.
Live Programming for Finite Model Finders

Finite model finders give users the ability to specify properties of a system in mathematical logic and then automatically find concrete examples, called solutions, that satisfy the properties. These solutions are often viewed as a key benefit of model finders, as they create an exploratory environment for developers to engage with their model. In practice, users find less benefit from these solutions than expected. For years, researchers believed that the problem was that too many solutions are produced. However, a recent user study found that users actually prefer enumerating a broad set of solutions. Inspired by a recent user study on Alloy, a modeling language backed by a finite model finder, we believe that the issue is that solutions are too removed from the logical constraints that generate them to help users build an understanding of the constraints themselves. In this paper, we outline a proof-of-concept for live programming of Alloy models in which writing the model and exploring solutions are intertwined. We highlight how this development environment enables more productive feedback loops between the developer, the model and the solutions.
An Image is Worth a Thousand Toxic Words: A Metamorphic Testing Framework for Content Moderation Software

The exponential growth of social media platforms has brought about a revolution in communication and content dissemination in human society. Nevertheless, these platforms are being increasingly misused to spread toxic content, including hate speech, malicious advertising, and pornography, leading to severe negative consequences such as harm to teenagers' mental health. Despite tremendous efforts in developing and deploying textual and image content moderation methods, malicious users can evade moderation by embedding texts into images, such as screenshots of the text, usually with some interference. We find that modern content moderation software's performance against such malicious inputs remains underexplored. In this work, we propose OASIS, a metamorphic testing framework for content moderation software. OASIS employs 21 transform rules summarized from our pilot study on 5,000 real-world toxic contents collected from 4 popular social media applications, including Twitter, Instagram, Sina Weibo, and Baidu Tieba. Given toxic textual contents, OASIS can generate image test cases, which preserve the toxicity yet are likely to bypass moderation. In the evaluation, we employ OASIS to test five commercial textual content moderation software from famous companies (i.e., Google Cloud, Microsoft Azure, Baidu Cloud, Alibaba Cloud and Tencent Cloud), as well as a state-of-the-art moderation research model. The results show that OASIS achieves up to 100% error finding rates. Moreover, through retraining the models with the test cases generated by OASIS, the robustness of the moderation model can be improved without performance degradation.
Minecraft: Automated Mining of Software Bug Fixes with Precise Code Context

Repository mining of bug fixes from version control systems like GitHub is a challenging problem as far as the precision of the bug context is concerned, i.e., source codes immediately preceding and succeeding the fix location. Coupled with this, identification of the type of the bug fix goes a long way towards creating high quality datasets that can be used for several downstream tasks. However, existing bug fix datasets suffer from the following limitations that dilute the data quality. Firstly, they do not focus on multilingual projects in their entirety given that most open-source projects are now multilingual. Secondly, the granularity of the bug fixes are considered only at the function/method level without specifying line/statement level information. Thirdly, bug fixes lying within the scope of a source file but outside any of its constituent functions have not been examined. In this paper, we propose a solution to overcome the aforementioned limitations by introducing a novel and extensive dataset named Minecraft. With a size of 28.8GB (considering 416 GitHub projects encompassing programming languages such as C, C++, Java, and Python, 2.2M commits, 3.29M bug-fix pairs), Minecraft surpasses the existing datasets by 4-fold enlargement in terms of data availability. We believe Minecraft would serve as a valuable resource for various stakeholders in the software development and research communities, empowering them to improve software quality, develop innovative bug detection and auto-fix techniques, and advance the field of software engineering.
An Empirical Study of Parameter-Efficient Fine-Tuning Methods for Pre-Trained Code Models

Pre-trained code models (e.g. CodeBERT and CodeT5) have demonstrated their code intelligence in various software engineering tasks, such as code summarization. And full fine-tuning has become the typical approach to adapting these models to downstream tasks. However, full fine-tuning these large models can be computationally expensive and memory-intensive, particularly when training for multiple tasks. To alleviate this issue, several parameter-efficient fine-tuning methods (e.g. Adapter and LoRA) have been proposed to only train a small number of additional parameters, while keeping the original pre-trained parameters frozen. Although these methods claim superiority over the prior techniques, they seldom make a comprehensive and fair comparison on multiple software engineering tasks. Moreover, besides their potential in reducing fine-tuning costs and maintaining approximate performance, the effectiveness of these methods in low-resource, cross-language, and cross-project scenarios is inadequately studied. To this end, we first conduct experiments by fine-tuning state-of-the-art code models with these methods on both code understanding tasks and code generation tasks. The results show that, by tuning only 0.5% additional parameters, these methods may achieve comparable or higher performance than full fine-tuning in code understanding tasks, but they may exhibit slightly weaker performance in code generation tasks. We also investigate the impact of these methods with varying numbers of training samples and find that, a considerable number of samples (e.g. 1000 for clone detection) may be required for them to approximate the performance of full fine-tuning. Our experimental results in cross-language and cross-project scenarios demonstrate that by freezing most pre-trained parameters and tuning only 0.5% additional parameters, these methods achieve consistent improvements in models' transfer learning ability in comparison to full fine-tuning. Our code and data are available at https://github.com/anonymous-ase23/ CodeModelParameterEfficientFinetuning.
Comparing The Fine-Tuning and Performance of Whisper Pre-Trained Models for Turkish Speech Recognition Task

Despite the increasing success of Automatic Speech Recognition (ASR) technology, different results can be obtained for different languages. ASR technology has become more efficient due to the continuous development of artificial intelligence and machine learning algorithms. The development of many ready-made ASR systems has been made possible by this. The success rates of ready-made systems vary depending on the language used. The purpose of this study was to evaluate the performance of the Turkish ASR system created by finetuning the Whisper Small model and compare it to the existing Turkish ASR systems. Performance evaluations were conducted using the Word Error Rate (WER) metric. The Turkish ASR performance of the Whisper Small model was impressive on a wide vocabulary (WER %17.98). The fine-tuned model successfully recognized words and expressions belonging to different subject areas and produced accurate text predictions (WER %12.89). The Whisper Small ASR model has demonstrated the advantages of having a large vocabulary compared to the existing Turkish ASR systems. Compared to other systems, it produced more accurate results by working with a lower error rate.
Open Source Software Tools for Data Management and Deep Model Training Automation

Designing and optimizing deep models require managing large datasets and conducting carefully designed controlled experiments that depend on large sets of hyper-parameters and problem dependent software/data configurations. These experiments are executed by training the model under observation with varying configurations. Since executing a typical training run can take days even on proven acceleration fabrics such as Graphics Processing Units (GPU), properly managing training data, avoiding human error in configuration preparations and securing the repeatability of the experiments are of utmost importance. In this paper, we present two open source software tools that aim to achieve these goals, namely, a Dataset Manager (DatumAid) tool and a Training Automation Manager (OrchesTrain) tool. DatumAid is a software tool that integrates with Computer Vision Annotation Tool (CVAT) to facilitate the management of annotated datasets. By adding additional functionality, DatumAid allows users to filter labeled data, manipulate datasets, and export datasets for training purposes. The tool adopts a simple code structure while providing flexibility to users through configuration files. OrchesTrain aims to automate model training process by facilitating rapid preparation and training of models in the desired style for the intended tasks. Users can seamlessly integrate their models prepared in the PyTorch library into the system and leverage the full capabilities of OrchesTrain. It enables the simultaneous or separate usage of Wandb, MLflow, and TensorBoard loggers. To ensure reproducibility of the conducted experiments, all configurations and codes are saved to the selected logger in an appropriate structure within a YAML file along with the serialized model files. Both software tools are publicly available on GitHub.
Delving into Commit-Issue Correlation to Enhance Commit Message Generation Models

Commit message generation (CMG) is a challenging task in automated software engineering that aims to generate natural language descriptions of code changes for commits. Previous methods all start from the modified code snippets, outputting commit messages through template-based, retrieval-based, or learning-based models. While these methods can summarize what is modified from the perspective of code, they struggle to provide reasons for the commit. The correlation between commits and issues that could be a critical factor for generating rational commit messages is still unexplored. In this work, we delve into the correlation between commits and issues from the perspective of dataset and methodology. We construct the first dataset anchored on combining correlated commits and issues. The dataset consists of an unlabeled commit-issue parallel part and a labeled part in which each example is provided with human-annotated rational information in the issue. Furthermore, we propose ExGroFi (Extraction, Grounding, Ene-tuning), a novel paradigm that can introduce the correlation between commits and issues into the training phase of models. To evaluate whether it is effective, we perform comprehensive experiments with various state-of-the-art CMG models. The results show that compared with the original models, the performance of ExGroFi-enhanced models is significantly improved.
An Efficient Algorithm for Clustering Sets

This paper proposes an algorithm, named HWK-Sets, based on K-Means, suited for clustering data which are variable-sized sets of elementary items. Clustering sets is difficult because data objects do not have numerical attributes and it is not possible to use the classical Euclidean distance upon which K-Means is normally based. An adaptation of the Jaccard distance between sets is used, which exploits application-sensitive information. More in particular, the Hartigan and Wong variation of K-Means is adopted which uses medoids as cluster representatives, can work with several seeding methods and can favor the fast attainment of a careful solution. The paper introduces HWK-Sets which is implemented in Java by parallel streams. Then, the efficiency and accuracy of HWK-Sets are demonstrated by simulation experiments.
ACWRecommender: A Tool for Validating Actionable Warnings with Weak Supervision

Static analysis tools have gained popularity among developers for finding potential bugs, but their widespread adoption is hindered by the accomnpanying high false alarm rates (up to 90%). To address this challenge, previous studies proposed the concept of actionable warnings, and apply machine-learning methods to distinguish actionable warnings from false alarms. Despite these efforts, our preliminary study suggests that the current methods used to collect actionable warnings are rather shaky and unreliable, resulting in a large proportion of invalid actionable warnings. In this work, we mined 68,274 reversions from Top-500 Github C repositories to create a substantia actionable warning dataset and assigned weak labels to each warning's likelihood of being a real bug. To automatically identify actionable warnings and recommend those with a high probability of being real bugs (AWHB), we propose a two-stage framework called ACWRecommender. In the first stage, our tool use a pre-trained model, i.e., UniXcoder, to identify actionable warnings from a huge number of SA tool's reported warnings. In the second stage, we rerank valid actionable warnings to the top by using weakly supervised learning. Experimental results showed that our tool outperformed several baselines for actionable warning detection (in terms of F1-score) and performed better for AWHB recommendation (in terms of nDCG and MRR). Additionaly, we also performed an in-the-wild evaluation, we manually validated 24 warnings out of 2,197 reported warnings on 10 randomly selected projects, 22 of which were confirmed by developers as real bugs, demonstrating the practical usage of our tool.
REEF: A Framework for Collecting Real-World Vulnerabilities and Fixes

Software plays a crucial role in our daily lives, and therefore the quality and security of software systems have become increasingly important. However, vulnerabilities in software still pose a significant threat, as they can have serious consequences. Recent advances in automated program repair have sought to automatically detect and fix bugs using data-driven techniques. Sophisticated deep learning methods have been applied to this area and have achieved promising results. However, existing benchmarks for training and evaluating these techniques remain limited, as they tend to focus on a single programming language and have relatively small datasets. Moreover, many benchmarks tend to be outdated and lack diversity, focusing on a specific codebase. Worse still, the quality of bug explanations in existing datasets is low, as they typically use imprecise and uninformative commit messages as explanations. To address these issues, we propose an automated collecting framework REEF to collect REal-world vulnErabilities and Fixes from open-source repositories. We focus on vulnerabilities since they are exploitable and have serious consequences. We develop a multi-language crawler to collect vulnerabilities and their fixes, and design metrics to filter for high-quality vulnerability-fix pairs. Furthermore, we propose a neural language model-based approach to generate high-quality vulnerability explanations, which is key to producing informative fix messages. Through extensive experiments, we demonstrate that our approach can collect high-quality vulnerability-fix pairs and generate strong explanations. The dataset we collect contains 4,466 CVEs with 30,987 patches (including 236 CWE) across 7 programming languages with detailed related information, which is superior to existing benchmarks in scale, coverage, and quality. Evaluations by human experts further confirm that our framework produces high-quality vulnerability explanations.
A Comprehensive Study on Code Clones in Automated Driving Software

With the continuous improvement of artificial intelligence technology, autonomous driving technology has been greatly developed. Hence automated driving software has drawn more and more attention from both researchers and practitioners. Code clone is a commonly used to speed up the development cycle in software development, but many studies have shown that code clones may affect software maintainability. Currently, there is little research investigating code clones in automated driving software. To bridge this gap, we conduct a comprehensive experience study on the code clones in automated driving software. Through the analysis of Apollo and Autoware, we have presented that code clones are prevalent in automated driving software. about 30% of code lines are involved in code clones and more than 50% of files contain code clones. Moreover, a notable portion of these code clones has caused bugs and co-modifications. Due to the high complexity of autonomous driving, the automated driving software is often designed to be modular, with each module responsible for a single task. When considering each module individually, we have found that Perception, Planning, Canbus, and Sensing modules are more likely to encounter code clones, and more likely to have bug-prone and co-modified clones. Finally, we have shown that there exist cross-module clones to propagate bugs and co-modifications in different modules, which undermine the software's modularity.
A Majority Invariant Approach to Patch Robustness Certification for Deep Learning Models

Patch robustness certification ensures no patch within a given bound on a sample can manipulate a deep learning model to predict a different label. However, existing techniques cannot certify samples that cannot meet their strict bars at the classifier level or the patch region level. This paper proposes MajorCert. MajorCert firstly finds all possible label sets manipulatable by the same patch region on the same sample across the underlying classifiers, then enumerates their combinations element-wise, and finally checks whether the majority invariant of all these combinations is intact to certify samples.