Reviewerly/exHarmony huggingface dataset:
A Closer Look at Different Difficulty Levels Code Generation Abilities of ChatGPT

Code generation aims to generate source code implementing human requirements illustrated with natural language specifications. With the rapid development of intelligent software engineering, automated code generation has become a hot research topic in both artificial intelligence and software engineering, and researchers have made significant achievements on code generation. More recently, large language models (LLMs) have demonstrated outstanding performance on code generation tasks, such as ChatGPT released by OpenAI presents the fantastic potential on automated code generation. However, the existing studies are limited to exploring LLMs' ability for generating code snippets to solve simple programming problems, the task of competition-level code generation has never been investigated. The specifications of the programming competition are always complicated and require the specific input/output format as well as the high-level algorithmic reasoning ability. In this study, we conduct the first large empirical study to investigate the zero-shot learning ability of ChatGPT for solving competition programming problems. Specifically, we warm up the design of prompts by using the Human-Eval dataset. Then, we apply the well-designed prompt to the competition-level code generation dataset, namely APPS, to further explore the effectiveness of using ChatGPT for solving competition problems. We collect ChatGPT's outputs on 5,000 code competition problems, the evaluation results show that it can successfully pass 25.4% test cases. By further feeding extra information (e.g, test failed information) to ChatGPT, we observe that ChatGPT has the potential to fix partial pass into a fully pass program. Moreover, we investigate the solutions generated by LLMs and the existing solutions, we find that it prefers to directly copy the code instead of re-write when facing more difficult problems. Finally, we evaluate the code quality generated by ChatGPT in terms of “code cleanness”, we observe that the generated codes are with small functions and file sizes, which are in line with the standard of clean code.
Conversational AI in Academia: A Practical Exploration with ChatGPT

In recent years, the rise of conversational AI and chatbots, like ChatGPT, has begun to revolutionize various sectors. One domain ripe for exploration is academia, where the intersection of vast knowledge bases and real-time, iterative feedback can potentially transform traditional research methodologies. This paper uniquely combines a comprehensive literature review with a firsthand interaction with ChatGPT to demonstrate its utility in academic research. The findings underscore the benefits of immediate access to information, structured guidance on research direction, and the provision of dynamic feedback. The documented live interaction serves as a testament to the value of integrating chatbot interactions into academic processes. While acknowledging the current limitations of these tools, the paper also casts a gaze forward, envisioning a future where chatbot integrations are commonplace, facilitating enhanced research quality and efficiency.
Assessing the Performance of an Architecture-Aware Optimization Tool for Neural Networks

The important growth in the demand for Neural Network solutions has created an urgent need for efficient implementations across a wide array of environments and platforms. As industries increasingly rely on AI-driven technologies, optimizing the performance and effectiveness of these networks has become crucial. While numerous studies have achieved promising results in this field, the process of fine-tuning and identifying optimal architectures for specific problem domains remains a complex and resource-intensive task. As such, there is a pressing need to explore and evaluate techniques that can improve this optimization process, reducing costs and time-to-deployment while maximizing the overall performance of Neural Networks. This work focuses on evaluating the optimization process of NetAdpat for two neural networks on an Nvidia Jetson device. We observe a performance decay for the larger network when the algorithm tries to meet the latency constraint. Furthermore, we propose potential alternatives to optimize this tool. Particularly, we propose an alternative configuration search procedure that allows us to enhance the optimization process, achieving speedups of up to <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\sim 7/\times$</tex> .
BLADE: Towards Scalable Source Code Debloating

Existing source code debloaters fall short due to low scalability and high runtime overhead when applied in dynamic cloud settings, where instances are spun up on the fly. To address this challenge, we propose BLADE that leverages the common coding idioms and language restrictions to build simple yet effective heuristics for faster source-code debloating. For example, usually, coding constructs are defined before used. Thus, the probability of breaking code after the removal of a node reduces with the depth of its position in the syntax tree. Also, while debloating certain functionalities, statements from a basic block have a higher possibility of getting removed together. To utilize these insights, BLADE employs a hierarchical source code reduction, where reduction candidates are chosen with reverse pre-order traversal, so that it removes uses before the definitions. Low runtime overhead makes BLADE practical to apply code debloating to large workloads. Our evaluation shows that BLADE runs faster than existing source code debloating tools. Compared to Chisel, BLADE is, on average, $2.3\times$ faster and provides comparable reductions in the code size and attack surfaces. In comparison to Debop, another source code debloater, BLADE, on average, is $2.75\times$ faster.
Transforming Education in the 21st Century: The Role of PROF-XXI Project in Developing Teaching Competencies

The rapid evolution of digital technologies has changed the educational landscape, providing new opportunities for innovative approaches to learning and challenging traditional teaching methods. In response to this, educators must adapt their practices to ensure that they are effectively using digital tools to support their instruction but also students' learning. This article presents the experience of the implementation of a Massive Open Online Course (MOOC) entitled “The 21st Century Teacher: Learning to Teach with Digital Technologies”. The MOOC was designed taking as a reference the competences defined in the DigCompEdu framework with the aim to train educators with the necessary methodologies, strategies, and best practices to teach effectively using digital technologies. The MOOC was designed to be easily accessible and available on an Open edX instance, with four lessons covering various aspects of digital teaching. The MOOC was prepared collaboratively as part of an international Erasmus+ project named PROF-XXI. The target group of this course was mainly Latin American educators. A total of 5,660 participants took this course in two cohorts, with 681 participants completing it (12% acceptance rate). Results and lessons learned from designing and running this course are presented in this work.
Exploring the Advances in Identifying Useful Code Review Comments

Effective peer code review in collaborative software development necessitates useful reviewer comments and supportive automated tools. Code review comments are a central component of the Modern Code Review process in the industry and open-source development. Therefore, it is important to ensure these comments serve their purposes. This paper reflects the evolution of research on the usefulness of code review comments. It examines papers that define the usefulness of code review comments, mine and annotate datasets, study developers' perceptions, analyze factors from different aspects, and use machine learning classifiers to automatically predict the usefulness of code review comments. Finally, it discusses the open problems and challenges in recognizing useful code review comments for future research.
Who is the Real Hero? Measuring Developer Contribution via Multi-Dimensional Data Integration

Proper incentives are important for motivating developers in open-source communities, which is crucial for maintaining the development of open-source software healthy. To provide such incentives, an accurate and objective developer contribution measurement method is needed. However, existing methods rely heavily on manual peer review, lacking objectivity and transparency. The metrics of some automated works about effort estimation use only syntax-level or even text-level information, such as changed lines of code, which lack robustness. Furthermore, some works about identifying core developers provide only a qualitative understanding without a quantitative score or have some project-specific parameters, which makes them not practical in real-world projects. To this end, we propose CVALUE, a multidimensional information fusion-based approach to measure developer contributions. CVALUE extracts both syntax and semantic information from the source code changes in four dimensions: modification amount, understandability, inter-function and intra-function impact of modification. It fuses the information to produce the contribution score for each of the commits in the projects. Experimental results show that CVALUE outperforms other approaches by 19.59% on 10 real-world projects with manually labeled ground truth. We validated and proved that the performance of CVALUE, which takes 83.39 seconds per commit, is acceptable to be applied in real-world projects. Furthermore, we performed a large-scale experiment on 174 projects and detected 2,282 developers having inflated commits. Of these, 2,050 developers did not make any syntax contribution; and 103 were identified as bots.
Availability and Demand of Jobs from the Perception of Automated Recruitment in Oman

Natural Language Processing has been applied for automation of the recruitment process globally with the increase in volumes of profiles and jobs. This paper discusses certain challenges in the use of cutting-edge technologies for documents in the Arabic language. We analyze the availability and demand of jobs in Oman with reference to that in UAE. As a result, hurdles can be minimized in the job-resume matching process between the employers and prospective candidates.
iASTMapper: An Iterative Similarity-Based Abstract Syntax Tree Mapping Algorithm

Abstract syntax tree (AST) mapping algorithms are widely used to locate the code changes in a file revision by mapping the AST nodes of the source code before and after the code changes. A recent differential testing of three state-of- the-art AST mapping algorithms, i.e., GumTree, MTDiff, and IJM, reveals that the algorithms generate inaccurate mappings for a considerable number of file revisions. We find that the inaccurate mappings could be caused by the mutual influence: the mappings of lower-level AST nodes (e.g., tokens) have impacts on the mappings of higher-level AST nodes (e.g., statements) and vice versa. This mutual influence issue is rarely considered by existing algorithms. In this paper, we propose an algorithm, called iASTMapper, that iteratively map two ASTs based on the similarities between AST nodes. Given a file revision, we extract three types of AST nodes in different levels of program structures (i.e., tokens, statements, and inner-statements) from the ASTs of the two source code files. We first build mappings of the unchanged statements and inner-statements. Then, we use an iterative method to map the rest of the nodes without mapping. For each of the three types of nodes, we iteratively map the nodes based on their similarities measured using heuristic rules. We further use an iterative mechanism to connect the three iterative mapping processes by considering the mutual influence between the mappings of different types of nodes. Finally, a series of code edit actions are generated from the node mappings to help users understand and locate the code changes during revisions. We conduct experiments to compare iASTMapper with three baselines, i.e., GumTree, MTDiff, and IJM, by automatically evaluating 210,997 file revisions from ten Java projects. Furthermore, we manually evaluate the correctness of the code edit actions generated for 200 file revisions with 12 evaluators. The results demonstrate that iASTMapper outperforms the baselines. iASTMapper can generate shorter code edit actions by at least 1.29% than the baselines, with a high accuracy of 96.23%.
An Intentional Forgetting-Driven Self-Healing Method for Deep Reinforcement Learning Systems

Deep reinforcement learning (DRL) is increasingly applied in large-scale productions like Netflix and Facebook. As with most data-driven systems, DRL systems can exhibit undesirable behaviors due to environmental drifts, which often occur in constantly-changing production settings. Continual Learning (CL) is the inherent self-healing approach for adapting the DRL agent in response to the environment's conditions shifts. However, successive shifts of considerable magnitude may cause the production environment to drift from its original state. Recent studies have shown that these environmental drifts tend to drive CL into long, or even unsuccessful, healing cycles, which arise from inefficiencies such as catastrophic forgetting, warm-starting failure, and slow convergence. In this paper, we propose Dr. DRL, an effective self-healing approach for DRL systems that integrates a novel mechanism of intentional forgetting into vanilla CL (i.e., standard CL) to overcome its main issues. Dr. DRL deliberately erases the DRL system's minor behaviors to systematically prioritize the adaptation of the key problem-solving skills. Using well-established DRL algorithms, Dr. DRL is compared with vanilla CL on various drifted environments. Dr. DRL is able to reduce, on average, the healing time and fine-tuning episodes by, respectively, 18.74% and 17.72%. Dr. DRL successfully helps agents to adapt to 19.63% of drifted environments left unsolved by vanilla CL while maintaining and even enhancing by up to 45% the obtained rewards for drifted environments that are resolved by both approaches.
A Preliminary Review of Function as a Service Platform Running with AWS Spot Instances

Cloud computing enabled users to easily implement what was previously a complex data center infrastructure, reducing its maintenance related costs. However, cloud costs can be challenging to predict with all its variety of resources and price schemes. Amazon Web Services offers the spot pricing mechanism for virtual machines (VM), in which an end user buys idle VM capacity for up to 90% cost reduction. But, these machines can be terminated by AWS at any given time, so not every application may be a good fit using them. We here explore the deployment of function as a service (FaaS) platforms in spots by using funcX, a distributed and high performance FaaS project. We chose MASA-OpenMP, a DNA sequence comparison project, to be executed in two different scenarios: one with in-memory execution in spot VMs, orchestrated with HADS, a Hibernation-Aware Dynamic Scheduler for spot instances, and the other using a funcX client deployed in spot instances as well. Our initial results showed that costs using the funcX approach are potentially smaller, and it also offers more implementation flexibility. We also present possible next steps for more experiments and investigation.
The Devil is in the Tails: How Long-Tailed Code Distributions Impact Large Language Models

Learning-based techniques, especially advanced Large Language Models (LLMs) for code, have gained considerable popularity in various software engineering (SE) tasks. However, most existing works focus on designing better learning-based models and pay less attention to the properties of datasets. Learning-based models, including popular LLMs for code, heavily rely on data, and the data's properties (e.g., data distribution) could significantly affect their behavior. We conducted an exploratory study on the distribution of SE data and found that such data usually follows a skewed distribution (i.e., long-tailed distribution) where a small number of classes have an extensive collection of samples, while a large number of classes have very few samples. We investigate three distinct SE tasks and analyze the impacts of long-tailed distribution on the performance of LLMs for code. Our experimental results reveal that the long-tailed distribution has a substantial impact on the effectiveness of LLMs for code. Specifically, LLMs for code perform between 30.0% and 254.0% worse on data samples associated with infrequent labels compared to data samples of frequent labels. Our study provides a better understanding of the effects of long-tailed distributions on popular LLMs for code and insights for the future development of SE automation.
VALAR: Streamlining Alarm Ranking in Static Analysis with Value-Flow Assisted Active Learning

Static analyzers play a critical role in program defects and security vulnerabilities detection. Despite their importance, the widespread adoption of static analysis techniques in industrial development faces numerous obstacles, among which the high rate of false alarms constitutes a significant one. To address this issue, we propose a novel approach called Valar, which performs alarm ranking for advanced value-flow analysis using the active learning technique. Active learning algorithms minimize the manual effort for alarm inspection by maximizing the effect of each user labeling in recognizing true/false alarms. Meanwhile, the value-flows provide Valar with a concise and comprehensive summary of the operational semantics about programs. Based on this, Valar is able to reason about the potential correlations between alarms and prioritize the most profitable unlabeled alarm. Additionally, the accuracy of Valar increases as more user labels are given and Valar's active learning model is further refined. We evaluate Valar on 20 real-world C/C++ programs using three value-flow based checkers. Our experimental results demonstrated that Valar significantly lowers the priorities of false alarms with most true alarms ranked high. Notably, Valar ranked all true alarms in the top 47% in 90% projects and ranked 90% true alarms in the top 22% in 75% projects. Furthermore, Valar has no requirement for pretraining and has a negligible computation time of less than 0.1s for each alarm prioritization.
Identify and Update Test Cases When Production Code Changes: A Transformer-Based Approach

Software testing is one of the most essential parts of the software lifecycle and requires a substantial amount of time and effort. During the software evolution, test cases should co-evolve with the production code. However, the co-evolution of test cases often fails due to tight project schedules and other reasons. Obsolete test cases improve the cost of software maintenance and may fail to reveal faults and even lead to future bugs. Therefore, it is essential to detect and update these obsolete test cases in time. In this paper, we propose a novel approach Ceprot (Co-Evolution of Production-Test Code) to identify outdated test cases and update them automatically according to changes in the production code. Ceprot consists of two stages, i.e., obsolete test identification and updating. Specifically, given a production code change and a corresponding test case, Ceprot first identifies whether the test case should be updated. If the test is identified as obsolete, Ceprot will update it to a new version of test case. To evaluate the effectiveness of the two stages, we construct two datasets. Our dataset focuses on method-level production code changes and updates on their obsolete test cases. The experimental results show that Ceprot can effectively identify obsolete test cases with precision and recall of 98.3% and 90.0%, respectively. In addition, test cases generated by Ceprot are identical to the ground truth for 12.3% of samples that are identified as obsolete by Ceprot. We also conduct dynamic evaluation and human evaluation to measure the effectiveness of the updated test cases by Ceprot. 48.0% of updated test cases can be compiled and the average coverage of updated cases is 34.2% which achieves 89% coverage improvement over the obsolete tests. We believe that this study can motivate the co-evolution of production and test code.
Leakpair: Proactive Repairing of Memory Leaks in Single Page Web Applications

Modern web applications often resort to application development frameworks such as React, Vue.js, and Angular. While the frameworks facilitate the development of web applications with several useful components, they are inevitably vulnerable to unmanaged memory consumption since the frameworks often produce Single Page Applications (SPAs). Web applications can be alive for hours and days with behavior loops, in such cases, even a single memory leak in a SPA app can cause performance degradation on the client side. However, recent debugging techniques for web applications still focus on memory leak detection, which requires manual tasks and produces imprecise results. We propose Leakpair,a technique to repair memory leaks in single page applications. Given the insight that memory leaks are mostly non-functional bugs and fixing them might not change the behavior of an application, the technique is designed to proactively generate patches to fix memory leaks, without leak detection, which is often heavy and tedious. To generate effective patches, Leakpairfollows the idea of pattern-based program repair since the automated repair strategy shows successful results in many recent studies. We evaluate the technique on more than 20 open-source projects without using explicit leak detection. The patches generated by our technique are also submitted to the projects as pull requests. The results show that Leakpaircan generate effective patches to reduce memory consumption that are acceptable to developers. In addition, we execute the test suites given by the projects after applying the patches, and it turns out that the patches do not cause any functionality breakage; this might imply that Leakpaircan generate non-intrusive patches for memory leaks.
Using Deep Learning to Automatically Improve Code Readability

Reading source code occupies most of developer's daily activities. Any maintenance and evolution task requires developers to read and understand the code they are going to modify. For this reason, previous research focused on the definition of techniques to automatically assess the readability of a given snippet. However, when many unreadable code sections are detected, developers might be required to manually modify them all to improve their readability. While existing approaches aim at solving specific readability-related issues, such as improving variable names or fixing styling issues, there is still no approach to automatically suggest which actions should be taken to improve code readability. In this paper, we define the first holistic readability-improving approach. As a first contribution, we introduce a methodology for automatically identifying readability-improving commits, and we use it to build a large dataset of 122k commits by mining the whole revision history of all the projects hosted on GitHub between 2015 and 2022. We show that such a methodology has ~86% accuracy. As a second contribution, we train and test the T5 model to emulate what developers did to improve readability. We show that our model achieves a perfect prediction accuracy between 21% and 28%. The results of a manual evaluation we performed on 500 predictions shows that when the model does not change the behavior of the input and it applies changes (34% of the cases), in the large majority of the cases (79.4%) it allows to improve code readability.