2022 29th Asia-Pacific Software Engineering Conference (APSEC)

Comparison of Developer’s Work Efﬁciency
between Different Editors
Sentaro Onizuka† , Tetsuya Kanda† , Katsuro Inoue‡†

Graduate School of Information Science and Technology, Osaka University, Osaka, Japan
Email: {s-onizuk, t-kanda}@ist.osaka-u.ac.jp‡Faculty of Science and Technology, Nanzan University, Aichi, Japan, Email: inoue599@nanzan-u.ac.jp

Abstract—In this paper, as an example of comparing developer’s work efﬁciency between different editors, we propose amethod to collect and compare self-evaluations and quantitativeevaluations of developer’s work efﬁciency for each editor. We
also practiced the proposed method on Visual Studio Code andEclipse, and conﬁrmed the applicability of the method.
Index Terms—Coding Process, Fine-grained Edit History,
Work Efﬁciency, Editor

I. I NTRODUCTION
Although many studies have analyzed the coding process,most studies have been limited to a single development environment, and few have compared the coding process acrossdifferent development environments. However, when development environments are limited, the proﬁciency of the development environment may affect the results of the analysis. For
this reason, it is important to develop methods to compare
coding processes in different development environments.
However, to compare the developer’s work efﬁciency between different editors, it is necessary to collect the edit history
from each editor under the same conditions. In addition,
more granular histories than those recorded in version control
systems can provide a more detailed understanding of thecoding process [1]. Therefore, we extended a ﬁne-grained edit
history collection platform proposed by Ishida et al. [2]. The
ﬁne-grained edit history in this platform is a record of all edit
operations of the source code by the developer on the editor.
In this platform, the history collection module is divided into alanguage server and editor plug-ins so that only editor plug-ins
need to be newly developed for each editor. The infrastructureof the history analysis part is implemented as a server, andthe server provides an API for retrieving the history, making
it easy to use the history with various tools.
Editors are an essential element in the coding process andare important in inﬂuencing the efﬁciency of development. Inaddition, each developer has subjective evaluations of eacheditor, such as strengths and weaknesses. However, such subjective evaluations do not always match actual work efﬁciency.
Therefore, there might be room for improving developer’s
development efﬁciency by ﬁnding a more efﬁcient editor.
In this paper, as an example of comparing coding processes
between different development environments, we propose amethod to compare developer’s work efﬁciency between different editors by collecting and comparing self-evaluations andquantitative evaluations of developer’s work efﬁciency for each

2640-0715/22/$31.00 ©2022 IEEE
DOI 10.1109/APSEC57359.2022.00086

Fig. 1. An Overview of the Proposed Methodeditor. We practice the method on Visual Studio Code andEclipse, and conﬁrm that the method is applicable in practice.
II. P ROPOSED M ETHOD
A. Overview
We propose a method to collect and compare “selfevaluations” and “quantitative evaluations” of developer’s
work efﬁciency for each editor. The proposed method compares two editors and is applicable to any editor that supports
Language Server Protocol [3], which is used in the history
collection part of the platform. A schematic diagram of theproposed method is shown in Figure 1. Subjects are asked
to perform small tasks of similar estimated working timewith two editors and to answer questionnaires about the selfevaluations. Quantitative evaluations are collected through thetasks and compared with self-evaluations.
B. Self-evaluations
Self-evaluations are collected by taking questionnaires before and after the task. Speciﬁcally, developers are asked toanswer the question, “Which editor do you think is faster atcoding?” on a ﬁve-point scale from “Editor 1 is very fast” to“Editor 2 is very fast”. The survey results are then converted
to a numerical value on a scale of -2 to +2, with the higher
value indicating that Editor 1 is faster and the lower value
indicating that Editor 2 is faster.
C. Quantitative evaluations
Quantitative evaluations are based on the ﬁne-grained edit
history collected through the task process. To collect histories572

Fig. 3. A scatter plot of comparing post-development self-evaluations andquantitative evaluations of a source code transcription task
Fig. 2. Meaning of each area in a scatter plot

collect history from different environments. We also developed
a tool to visualize the time-series change of progress rate
based on the collected history using the API provided by
the platform. Eight participants were asked to do two tasks:
the ﬁrst task required them to answer programming problemsof similar difﬁculty, and the second task required them totranscribe source code. Self-evaluations are collected by taking
questionnaires before and after the task. Finally, we createdscatter plots for comparison based on self-evaluations andquantitative evaluations.
As an example of the output obtained by the method,
Figure 3 shows a scatter plot comparing post-development
self-evaluations and quantitative evaluations of a source codetranscription task. The correlation coefﬁcient calculated fromthe plot is −0.27, indicating that there is no positive correlation. Also, regarding the self-evaluations and quantitativeevaluations, three respondents were in agreement, three wereslightly out of agreement, and two were not in agreement,
indicating that only a small number of respondents were inagreement. These results indicate that there is a gap between
the post-development self-evaluations and the quantitativeevaluations of the source code transcription task.

from multiple editors under the same conditions, the platform
described in section I is used. We extended the platform toprovide a history API so that we can easily access the collected
history thereafter.
The data of the time required for each task in the progress
graph are used for evaluations. Speciﬁcally, the time required
for each task is normalized from a minimum value of 0 to
a maximum value of 1. The difference between the averageof normalized time in editor 1 (avg1) and editor 2 (avg2) iscalculated as “−(avg1 − avg2) × 100”. The range of value
calculated here is −100 to +100, where a larger value meansthat editor 1 is more efﬁcient and a smaller value means that
editor 2 is more efﬁcient.
Using the collected history, we can conduct various visualization of the development process. For example, visualizingchanges in the development progress rate over time based on
the edit distance to the ﬁles in their ﬁnal state at each time.
D. Comparison
Scatter plots are created based on the values of selfevaluations and quantitative evaluations, and the scatter plots
are used to determine whether they are in agreement or not.
Each point on the scatter plot corresponds to a developer’s selfevaluations and quantitative evaluations. The x-axis is the selfevaluation and the y-axis is the quantitative evaluation. Since
the self-evaluation is conducted twice, before and after thetask, we can create two types of scatter plots: “comparison ofpre-development self-evaluations and quantitative evaluations”
and “comparison of post-development self-evaluations andquantitative evaluations”.
Two criteria are used to evaluate the agreement between theself-evaluations and the quantitative evaluations based on thescatter plots: one is the correlation coefﬁcient of the scatterplots, and the other is the classiﬁcation by area. The scatterplots are divided into three areas as shown in Figure 2, andthe agreement is judged by which area each point is located.

IV. C ONCLUSION
We proposed a method to realize the comparison of developer’s work efﬁciency between different editors. We showed
an example output of the proposed method with two editors.
In future work, we need to improve the method of quantitative evaluations. Currently, only information on time spentcoding is used, hence it is planned to consider information on
the coding process to more accurately evaluate work efﬁciency.
ACKNOWLEDGMENT
This work was supported by JSPS KAKENHI Grant Numbers JP19K20239 and JP18H04094.
R EFERENCES[1] S. Negara, M. Vakilian, N. Chen, R. E. Johnson, and D. Dig. “Isit dangerous to use version control histories to study source code
evolution?,” In Proc. of ECOOP2012, pp. 79–103, 2012.
[2] N. Ishida, T. Kanda, K. Shimari, and K. Inoue. “Concept of Fine-Grained
Edit History Collection Platform Using Language Server””, SES2020
Workshop 5, no.5, September 2020 (in Japanese).
[3] “Language Server Protocol,” Microsoft, 2022. Accessed: October 12,2022. Available: https://microsoft.github.io/language-server-protocol/III. M ETHOD E XAMPLE
We conducted a small experiment with the proposed method
on Visual Studio Code and Eclipse. According to the proposed
method, we ﬁrst developed plug-ins for each target editor to573